Multiplayer agent harness for work.
Replace background membership with a single owner record (#2057)
Regan Bell877e872cf3bf
diff --git a/cli/README.md b/cli/README.mdindex f72b9229f..a4728925e 100644--- a/cli/README.md+++ b/cli/README.md@@ -96,66 +96,66 @@ that replaces them cannot finish before their longest in-flight turn endslonger deadline or when no long turn is in flight.AWS deployments can opt into durable background ownership with-`aws.backgroundWorkControl: true`. Deploy protocol-capable core images to every-participating stack before bootstrapping ownership. The CLI allocates a unique-`BACKGROUND_DEPLOYMENT_ID` for each replacement core cohort and records it in the-deployment manifest. A no-op deployment and automatic ECS task replacement keep+`aws.backgroundWorkControl: true`. The CLI allocates a unique+`BACKGROUND_DEPLOYMENT_ID` for each replacement core deployment and records it in+the deployment manifest. A no-op deployment and automatic ECS task replacement keepthat identity; an explicit core restart allocates a new one. Pending preparationis persisted before ECS changes, and an ambiguous previous deployment must bereconciled before another identity can be allocated.Control requests require both `CORE_SIGNING_SECRET` and a distinct`DEPLOYMENT_CONTROL_SECRET` of at least 32 characters. The control secret is-restricted to core. Configuring an identity preserves the legacy background flag-until explicit bootstrap verifies every participating task. The exported-`awsBootstrapBackgroundWork` adapter accepts all peer configurations and an exact-desired deployment identity, or `null` to start paused. It refuses missing cohorts-and inconsistent durable membership.+restricted to core. One shared record names the deployment that owns background+work; a core process with a configured identity admits work only while that+record names its deployment, and `BACKGROUND_WORK_ENABLED` is ignored there.The exported `awsBackgroundWorkBootState` reads the exact manifest core task toreturn its recorded boot flag and optional deployment identity. It returns`undefined` only without a recorded core task, and rejects missing or ambiguous-boot flags or identities. Deployment wrappers can preserve the boot environment-while changing durable ownership independently.+boot flags or identities.++The exported `awsBackgroundWorkStatus(config, configDir, candidatePath?)` proves+the stack's exact healthy core task set and reads the owner record through the+stack's own API, returning the responder's `ownerDeploymentId`, `setAt`, `setBy`,+and whether the responding process is currently active. The exported+`awsSetBackgroundWork(config, configDir, enabled, candidatePath?, expected?)` sets+the owner record to the stack's identity (or clears it when `enabled` is false)+without restarting ECS tasks, then polls until every core task of the stack has+reported matching activity. Passing `expected: { ownerDeploymentId }` refuses the change+if the owner differs from what the caller last observed. A demotion refuses to+clear a different deployment's ownership.The exported `awsBackgroundWorkCapacity(config, configDir, candidatePath?)` provesthat an inactive controlled stack is currently reusable. It requires another-owner, a current membership that has relinquished, stable native deployments and-exact task inventories for every workload, and resolved deployment preparation. It-never changes deployment state. The result binds the manifest and deployment-identities, ownership generation, workload task definitions, native deployment IDs,-and task ARNs.-`awsBackgroundWorkStatus` also returns the current manifest ID for an active-owner-proof. A release coordinator can combine both snapshots with immutable candidate-provenance and compare them again under its production lock before any mutation.-This is a point-in-time check, not a reservation: intervening maintenance or task-replacement invalidates the proof and must block promotion.--Live checks use the active ownership cohort's authenticated canary endpoint to-verify a real model reply, session persistence, generated title, error records,-session cleanup, and database catalog health. The CLI proves the exact healthy-task cohort and ownership generation before and after the check. It requires a-final success bound to the request and responding task; heartbeats alone do not-count. An uncertain result never triggers an automatic replay or fallback.-Legacy deployments retain the Fargate canary. A controlled deployment uses that-same path only when a successful ownership read proves bootstrap is disabled or-another cohort owns background work.--Once bootstrapped, background mode changes use generation-checked ownership-requests without restarting ECS tasks. Activation waits for prior owners to stop-claiming and every expected task to finish activation. Pausing stops new claims;-in-flight turns can continue draining. Replacing or rolling back the active core-cohort requires an explicit pause or handover and proof that every member has-relinquished first; ECS then stops the old tasks, and any turn still running-there resumes on the replacement tasks. A demotion-refuses to pause a different current owner. Unresponsive members are never-assumed dead: `awsRetireBackgroundWorkMembers` requires exact instance, task ARN,-and generation identities plus ECS evidence that each task stopped. This recovery-is also available before bootstrap for stopped legacy members.+owner, stable native deployments and exact task inventories for every workload,+and resolved deployment preparation. It never changes deployment state. The+result binds the manifest and deployment identities, the owner, workload task+definitions, native deployment IDs, and task ARNs. A release coordinator can+combine both snapshots with immutable candidate provenance and compare them+again under its production lock before any mutation. This is a point-in-time+check, not a reservation: intervening maintenance or task replacement+invalidates the proof and must block promotion.++Live checks use the owning deployment's authenticated canary endpoint to verify a+real model reply, session persistence, generated title, error records, session+cleanup, and database catalog health. The CLI confirms the stack owns background+work and is active before and after the check, and requires a final success bound+to the request and responding process; heartbeats alone do not count. An uncertain+result never triggers an automatic replay or fallback. Legacy deployments retain+the Fargate canary, and a controlled stack that does not own background work uses+that same path.++Replacing or rolling back a controlled stack's core tasks requires that the stack+is not the current owner: hand ownership to the other stack first. Processes that+lose ownership stop claiming within their poll interval (at most their ten-second+validity window) and finish admitted work under their leases; ECS task+replacement then drains them through `SIGTERM`. `qm down` is not guarded: if the+record still names a stack that has no tasks, set the live stack active to+recover.Core secret uploads defer activation to a subsequent staged `up --restart core`.The generic upload path refuses changes to either control credential while a-controlled cohort is recorded, because replacing credentials before coordinating+controlled deployment is recorded, because replacing credentials before coordinatingall running processes would break ownership control. Legacy AWS deployments keepthe task replacement path when ownership control is not configured.diff --git a/cli/src/backends/aws.ts b/cli/src/backends/aws.tsindex 57d253547..2fa7cbf5b 100644--- a/cli/src/backends/aws.ts+++ b/cli/src/backends/aws.ts@@ -91,10 +91,8 @@ import {import {awaitBackgroundWork,- backgroundWorkMutation,- mutateBackgroundWork,readBackgroundWork,- type BackgroundWorkMember,+ setBackgroundOwner,type BackgroundWorkStatus,type BackgroundWorkTransport,} from "../background-work.ts";@@ -2431,7 +2429,7 @@ export async function awsUp(config: QmConfig, _configDir: string, opts: AwsUpOptif (!current?.backgroundDeploymentId && !services.includes("core"))throw new CliError("enabling background ownership requires deploying core to allocate its identity");reconcileBackgroundPreparation(aws, current, before);- if (services.includes("core")) await assertBackgroundCohortReplaceable(config, current);+ if (services.includes("core")) await assertNotBackgroundOwner(config, current);}const selected = new Set(services);if (allServices.some((service) => !selected.has(service))) {@@ -2786,7 +2784,7 @@ export async function awsRollback(if (currentManifest?.backgroundDeploymentId || targetManifest.backgroundDeploymentId) {if (!aws.backgroundWorkControl || !targetManifest.backgroundDeploymentId)throw new CliError("controlled rollback requires a protocol-capable target deployment");- await assertBackgroundCohortReplaceable(config, currentManifest);+ await assertNotBackgroundOwner(config, currentManifest);}const missing = services.filter((service) => !targetManifest.tasks[service]);if (missing.length)@@ -3116,22 +3114,13 @@ export async function awsBackgroundWorkStatus(assertAwsCallerAccount(requireAws(config));const cohort = await awsBackgroundCohort(config, configDir, candidatePath);const status = await readBackgroundWork(awsBackgroundWorkTransport(config), cohort.deploymentId);- for (const taskArn of cohort.taskArns) {- if (- !status.members.some(- (member) => member.taskArn === taskArn && member.deploymentId === cohort.deploymentId && !member.retired,- )- )- throw new CliError("a running core task has not enrolled in the ownership protocol");- }return { status, deploymentId: cohort.deploymentId, taskArns: cohort.taskArns, manifestId: cohort.manifest.id };}export interface AwsBackgroundWorkCapacity {manifestId: string;deploymentId: string;- generation: number;- desiredDeploymentId: string;+ ownerDeploymentId: string;workloads: Record<string, { taskDefinition: string; deploymentId: string; desiredCount: number; taskArns: string[] }>;}@@ -3226,22 +3215,8 @@ export async function awsBackgroundWorkCapacity(const cohort = await awsBackgroundCohort(config, configDir, candidatePath, false);const transport = awsBackgroundWorkTransport(config);const ownership = await readBackgroundWork(transport, cohort.deploymentId);- if (!ownership.enabled || !ownership.desiredDeploymentId || ownership.desiredDeploymentId === cohort.deploymentId)- throw new CliError("capacity requires enabled background ownership held by another deployment");- assertBackgroundCohortRelinquished(ownership, cohort.deploymentId);- if (- cohort.taskArns.some(- (taskArn) =>- !ownership.members.some(- (member) =>- member.taskArn === taskArn &&- member.deploymentId === cohort.deploymentId &&- !member.retired &&- member.state !== "admitted",- ),- )- )- throw new CliError("capacity requires every current core task to be enrolled and relinquished");+ if (!ownership.ownerDeploymentId || ownership.ownerDeploymentId === cohort.deploymentId)+ throw new CliError("capacity requires background ownership held by another deployment");const states = describedServices(config, workloads);assertOwnedServices(config, states, workloads);const snapshot = serviceSnapshotFromStates(states, workloads);@@ -3249,8 +3224,7 @@ export async function awsBackgroundWorkCapacity(const proof: AwsBackgroundWorkCapacity = {manifestId: cohort.manifest.id,deploymentId: cohort.deploymentId,- generation: ownership.generation,- desiredDeploymentId: ownership.desiredDeploymentId,+ ownerDeploymentId: ownership.ownerDeploymentId,workloads: {},};for (const workload of [...workloads].sort()) {@@ -3269,179 +3243,16 @@ export async function awsBackgroundWorkCapacity(throw new CliError("deployment manifest changed while checking capacity");assertBackgroundPreparationResolved(aws, finalManifest, snapshot);const finalOwnership = await readBackgroundWork(transport, cohort.deploymentId);- if (- !finalOwnership.enabled ||- finalOwnership.generation !== proof.generation ||- finalOwnership.desiredDeploymentId !== proof.desiredDeploymentId- )+ if (finalOwnership.ownerDeploymentId !== proof.ownerDeploymentId)throw new CliError("background ownership changed while checking capacity");- assertBackgroundCohortRelinquished(finalOwnership, cohort.deploymentId);- if (- cohort.taskArns.some(- (taskArn) =>- !finalOwnership.members.some(- (member) =>- member.taskArn === taskArn &&- member.deploymentId === cohort.deploymentId &&- !member.retired &&- member.state !== "admitted",- ),- )- )- throw new CliError("current core enrollment changed while checking capacity");return proof;}-export interface AwsBackgroundWorkPeer {- config: QmConfig;- configDir: string;- candidatePath?: string;-}--async function withBackgroundPeerLeases<T>(peers: AwsBackgroundWorkPeer[], operation: () => Promise<T>): Promise<T> {- const unique = new Map(- peers.map((peer) => {- const aws = requireAws(peer.config);- return [- `${aws.accountId}:${aws.region}:${deployLocksTable(aws)}:${deploymentStateKey(aws, "deploy")}`,- aws,- ] as const;- }),- );- if (unique.size !== peers.length)- throw new CliError("background ownership peers must have distinct deployment leases");- const leases: Array<{ aws: AwsConfig; lease: ReturnType<typeof acquireLease> }> = [];- try {- for (const [, aws] of [...unique.entries()].sort(([a], [b]) => a.localeCompare(b))) {- assertAwsCallerAccount(aws);- leases.push({ aws, lease: acquireLease(aws) });- }- return await operation();- } finally {- for (const { aws, lease } of leases.reverse()) releaseLease(aws, lease);- }-}--export async function awsBootstrapBackgroundWork(- peers: AwsBackgroundWorkPeer[],- desiredDeploymentId: string | null,-): Promise<BackgroundWorkStatus> {- if (!peers.length) throw new CliError("background ownership bootstrap requires every participating deployment");- return withBackgroundPeerLeases(peers, async () => {- const cohorts = [];- for (const peer of peers)- cohorts.push(await awsBackgroundWorkStatus(peer.config, peer.configDir, peer.candidatePath));- const first = cohorts[0]!;- if (cohorts.some((cohort) => cohort.status.enabled || cohort.status.generation !== 0))- throw new CliError("background ownership bootstrap requires disabled generation zero for every cohort");- if (new Set(cohorts.map((cohort) => cohort.deploymentId)).size !== cohorts.length)- throw new CliError("background ownership bootstrap requires distinct cohort identities");- const shared = (status: BackgroundWorkStatus): string =>- canonicalJson({- generation: status.generation,- desiredDeploymentId: status.desiredDeploymentId,- members: [...status.members].sort((a, b) => a.instanceId.localeCompare(b.instanceId)),- });- if (cohorts.some((cohort) => shared(cohort.status) !== shared(first.status)))- throw new CliError("background ownership peers do not agree on shared durable membership");- const taskArns = cohorts.flatMap((cohort) => cohort.taskArns);- if (- new Set(taskArns).size !== taskArns.length ||- first.status.members.some((member) => !member.retired && (!member.taskArn || !taskArns.includes(member.taskArn)))- )- throw new CliError("bootstrap must include every enrolled non-retired task");- const desired =- desiredDeploymentId === null ? undefined : cohorts.find((cohort) => cohort.deploymentId === desiredDeploymentId);- if (desiredDeploymentId !== null && !desired)- throw new CliError("bootstrap desired owner is not a verified peer cohort");- const transport = awsBackgroundWorkTransport(peers[0]!.config);- const mutation = { ...backgroundWorkMutation(0, desiredDeploymentId), bootstrapTaskArns: taskArns };- const applied = await mutateBackgroundWork(transport, first.deploymentId, mutation);- return awaitBackgroundWork(- transport,- first.deploymentId,- { generation: applied.generation, desiredDeploymentId, taskArns: desired?.taskArns ?? [] },- { timeoutMs: envNum("QM_AWS_ROLLOUT_DEADLINE_MS", 30 * 60_000), pollMs: 1000 },- );- });-}--export async function awsRetireBackgroundWorkMembers(- peers: AwsBackgroundWorkPeer[],- terminatedMembers: Array<Pick<BackgroundWorkMember, "instanceId" | "taskArn" | "generation">>,-): Promise<BackgroundWorkStatus> {- if (!peers.length || !terminatedMembers.length)- throw new CliError("task retirement requires deployment peers and exact member identities");- return withBackgroundPeerLeases(peers, async () => {- const first = await awsBackgroundWorkStatus(peers[0]!.config, peers[0]!.configDir, peers[0]!.candidatePath);- for (const retired of terminatedMembers) {- const member = first.status.members.find(- (item) =>- item.instanceId === retired.instanceId &&- item.taskArn === retired.taskArn &&- item.generation === retired.generation,- );- if (!member?.taskArn || member.retired)- throw new CliError("task retirement does not match a live durable member");- let proved = false;- for (const peer of peers) {- const aws = requireAws(peer.config);- if (!member.deploymentId.startsWith(`${aws.services.core!.ecsService}:`)) continue;- const response = awsJson<{- tasks?: Array<{ taskArn?: string; taskDefinitionArn?: string; lastStatus?: string; group?: string }>;- failures?: unknown[];- }>(aws, ["ecs", "describe-tasks", "--cluster", aws.cluster, "--tasks", member.taskArn]);- const task = response.tasks?.find((item) => item.taskArn === member.taskArn);- if (- response.failures?.length ||- !task ||- task.lastStatus !== "STOPPED" ||- task.group !== `service:${aws.services.core!.ecsService}` ||- !task.taskDefinitionArn- )- continue;- const definition = awsJson<{- taskDefinition?: {- containerDefinitions?: Array<{ name?: string; environment?: Array<{ name: string; value: string }> }>;- };- }>(aws, ["ecs", "describe-task-definition", "--task-definition", task.taskDefinitionArn]).taskDefinition;- const identity = definition?.containerDefinitions- ?.find((container) => container.name === "core")- ?.environment?.find((entry) => entry.name === "BACKGROUND_DEPLOYMENT_ID")?.value;- if (identity === member.deploymentId) proved = true;- }- if (!proved)- throw new CliError(- "task retirement requires ECS STOPPED evidence bound to the exact service, task and deployment identity",- );- }- return mutateBackgroundWork(awsBackgroundWorkTransport(peers[0]!.config), first.deploymentId, {- expectedGeneration: first.status.generation,- requestId: randomUUID(),- terminatedMembers,- });- });-}--async function assertBackgroundCohortReplaceable(- config: QmConfig,- current: DeploymentManifest | undefined,-): Promise<void> {+async function assertNotBackgroundOwner(config: QmConfig, current: DeploymentManifest | undefined): Promise<void> {if (!current?.backgroundDeploymentId) return;const status = await readBackgroundWork(awsBackgroundWorkTransport(config), current.backgroundDeploymentId);- if (status.enabled) assertBackgroundCohortRelinquished(status, current.backgroundDeploymentId);-}--function assertBackgroundCohortRelinquished(status: BackgroundWorkStatus, deploymentId: string): void {- if (- status.desiredDeploymentId === deploymentId ||- status.members.some(- (member) => !member.retired && member.deploymentId === deploymentId && member.state === "admitted",- )- )- throw new CliError(- "pause or hand over background ownership and wait for every member to relinquish before replacing the current core cohort",- );+ if (status.ownerDeploymentId === current.backgroundDeploymentId)+ throw new CliError("hand background ownership to another deployment before replacing the current core tasks");}export async function awsSetBackgroundWork(@@ -3449,7 +3260,7 @@ export async function awsSetBackgroundWork(configDir: string,enabled: boolean,candidatePath?: string,- expectedOwnership?: Pick<BackgroundWorkStatus, "generation" | "lastRequestId">,+ expectedOwnership?: Pick<BackgroundWorkStatus, "ownerDeploymentId">,): Promise<BackgroundWorkStatus | undefined> {const { aws, workloads } = awsTopology(config, configDir);if (!workloads.includes("core")) throw new CliError("background work requires the core workload");@@ -3497,32 +3308,21 @@ export async function awsSetBackgroundWork(if (aws.backgroundWorkControl) {const cohort = await awsBackgroundCohort(config, configDir, candidatePath);const transport = awsBackgroundWorkTransport(config);- let status = await readBackgroundWork(transport, cohort.deploymentId);- if (- expectedOwnership &&- (status.generation !== expectedOwnership.generation || status.lastRequestId !== expectedOwnership.lastRequestId)- )+ const status = await readBackgroundWork(transport, cohort.deploymentId);+ if (expectedOwnership && status.ownerDeploymentId !== expectedOwnership.ownerDeploymentId)throw new CliError("background ownership changed since promotion; refusing automatic compensation");- if (!status.enabled)- throw new CliError("explicitly bootstrap all background ownership cohorts before changing ownership");- if (!enabled && status.desiredDeploymentId !== null && status.desiredDeploymentId !== cohort.deploymentId)+ if (!enabled && status.ownerDeploymentId !== null && status.ownerDeploymentId !== cohort.deploymentId)throw new CliError("cannot disable background work on a different deployment's current owner");- const desiredDeploymentId = enabled ? cohort.deploymentId : null;- if (status.desiredDeploymentId !== desiredDeploymentId) {- status = await mutateBackgroundWork(transport, cohort.deploymentId, {- ...backgroundWorkMutation(status.generation, desiredDeploymentId),- ...(expectedOwnership ? { expectedLastRequestId: expectedOwnership.lastRequestId } : {}),+ const ownerDeploymentId = enabled ? cohort.deploymentId : null;+ if (status.ownerDeploymentId !== ownerDeploymentId)+ await setBackgroundOwner(transport, cohort.deploymentId, {+ ownerDeploymentId,+ expectedOwnerDeploymentId: status.ownerDeploymentId,});- }return awaitBackgroundWork(transport,cohort.deploymentId,- {- generation: status.generation,- desiredDeploymentId,- taskArns: enabled ? cohort.taskArns : [],- lastRequestId: status.lastRequestId,- },+ { ownerDeploymentId, active: enabled, instances: cohort.taskArns.length },{ timeoutMs: envNum("QM_AWS_ROLLOUT_DEADLINE_MS", 30 * 60_000), pollMs: 1000 },);}@@ -3706,7 +3506,7 @@ export async function awsSecretsPush(config: QmConfig, configDir: string, envFil}if (aws.backgroundWorkControl && affected.includes("core")) {note(- "core secret activation deferred: pause or hand over background ownership, then run qm up --restart core to allocate a new cohort",+ "core secret activation deferred: pause or hand over background ownership, then run qm up --restart core to allocate a new deployment identity",);affected.splice(affected.indexOf("core"), 1);}@@ -5005,7 +4805,7 @@ async function checkLive(if (!failures.length) {try {const cohort = aws.backgroundWorkControl ? await awsBackgroundWorkStatus(config, configDir) : undefined;- if (cohort?.status.enabled && cohort.status.desiredDeploymentId === cohort.deploymentId) {+ if (cohort && cohort.status.ownerDeploymentId === cohort.deploymentId) {const transport = awsBackgroundWorkTransport(config, "/v1/deployment/live-session", 600_000, 65_536);await checkControlledLiveSession({before: cohort,diff --git a/cli/src/background-work.ts b/cli/src/background-work.tsindex c65ec6edc..f1bc7e6e5 100644--- a/cli/src/background-work.ts+++ b/cli/src/background-work.ts@@ -1,26 +1,19 @@-import { randomUUID } from "node:crypto";import { CliError } from "./log.ts";import { sleep } from "./util.ts";-export interface BackgroundWorkMember {- instanceId: string;- taskArn: string | null;- deploymentId: string;- generation: number;- state: "admitted" | "relinquished" | "drained";- retired: boolean;- ready: boolean;-}-export interface BackgroundWorkStatus {- protocol: 1;- enabled: boolean;+ protocol: 2;deploymentId: string;instanceId: string;- generation: number;- desiredDeploymentId: string | null;- lastRequestId: string | null;- members: BackgroundWorkMember[];+ ownerDeploymentId: string | null;+ setAt: string | null;+ setBy: string | null;+ active: boolean;+}++export interface BackgroundOwnerChange {+ ownerDeploymentId: string | null;+ expectedOwnerDeploymentId: string | null;}export type BackgroundWorkTransport = (@@ -28,14 +21,6 @@ export type BackgroundWorkTransport = (body?: string,) => Promise<{ status: number; body: string }>;-export type BackgroundWorkMutation = {- expectedGeneration: number;- requestId: string;-} & (- | { desiredDeploymentId: string | null; bootstrapTaskArns?: string[]; expectedLastRequestId?: string | null }- | { terminatedMembers: Array<Pick<BackgroundWorkMember, "instanceId" | "taskArn" | "generation">> }-);-function record(value: unknown): value is Record<string, unknown> {return value !== null && typeof value === "object" && !Array.isArray(value);}@@ -44,8 +29,28 @@ function text(value: unknown): value is string {return typeof value === "string" && value.length > 0;}-function generation(value: unknown): value is number {- return typeof value === "number" && Number.isSafeInteger(value) && value >= 0;+function nullableText(value: unknown): value is string | null {+ return value === null || text(value);+}++function memberProtocolStatus(value: Record<string, unknown>): BackgroundWorkStatus | undefined {+ if (+ typeof value.enabled !== "boolean" ||+ !nullableText(value.desiredDeploymentId) ||+ !Array.isArray(value.members) ||+ !value.members.every(record)+ )+ return undefined;+ const self = value.members.find((member) => member.instanceId === value.instanceId);+ return {+ protocol: 2,+ deploymentId: value.deploymentId as string,+ instanceId: value.instanceId as string,+ ownerDeploymentId: value.enabled ? value.desiredDeploymentId : null,+ setAt: null,+ setBy: null,+ active: self?.state === "admitted" && self.ready === true && self.retired === false,+ };}export function parseBackgroundWorkStatus(body: string, deploymentId: string): BackgroundWorkStatus {@@ -55,45 +60,27 @@ export function parseBackgroundWorkStatus(body: string, deploymentId: string): B} catch {throw new CliError("background ownership returned invalid JSON");}+ if (!record(value) || value.deploymentId !== deploymentId || !text(value.instanceId))+ throw new CliError("background ownership response does not match the requested deployment");+ const legacy = value.protocol === 1 ? memberProtocolStatus(value) : undefined;+ if (legacy) return legacy;if (- !record(value) ||- value.protocol !== 1 ||- typeof value.enabled !== "boolean" ||- value.deploymentId !== deploymentId ||- !text(value.instanceId) ||- !generation(value.generation) ||- !(value.desiredDeploymentId === null || text(value.desiredDeploymentId)) ||- !(value.lastRequestId === null || text(value.lastRequestId)) ||- !Array.isArray(value.members)- ) {- throw new CliError("background ownership response does not match the requested deployment and protocol");- }- const members = value.members;- for (const member of members) {- if (- !record(member) ||- !text(member.instanceId) ||- !(member.taskArn === null || text(member.taskArn)) ||- !text(member.deploymentId) ||- !generation(member.generation) ||- member.generation > value.generation ||- !["admitted", "relinquished", "drained"].includes(String(member.state)) ||- typeof member.retired !== "boolean" ||- typeof member.ready !== "boolean"- ) {- throw new CliError("background ownership returned an invalid member");- }- }- const identities = members.map((member) => member.instanceId);- if (new Set(identities).size !== identities.length)- throw new CliError("background ownership returned duplicate instance identities");- if (- !members.some(- (member) => member.instanceId === value.instanceId && member.deploymentId === deploymentId && !member.retired,- )+ value.protocol !== 2 ||+ !nullableText(value.ownerDeploymentId) ||+ !nullableText(value.setAt) ||+ !nullableText(value.setBy) ||+ typeof value.active !== "boolean")- throw new CliError("background ownership responder is not enrolled in its durable membership");- return value as unknown as BackgroundWorkStatus;+ throw new CliError("background ownership response does not match the single-owner protocol");+ return {+ protocol: 2,+ deploymentId,+ instanceId: value.instanceId,+ ownerDeploymentId: value.ownerDeploymentId,+ setAt: value.setAt,+ setBy: value.setBy,+ active: value.active,+ };}export async function readBackgroundWork(@@ -106,35 +93,12 @@ export async function readBackgroundWork(return parseBackgroundWorkStatus(response.body, deploymentId);}-function mutationCommitted(state: BackgroundWorkStatus, mutation: BackgroundWorkMutation): boolean {- if (state.lastRequestId !== mutation.requestId) return false;- if ("desiredDeploymentId" in mutation) {- return (- state.generation === mutation.expectedGeneration + 1 &&- state.desiredDeploymentId === mutation.desiredDeploymentId &&- state.enabled- );- }- return (- state.generation === mutation.expectedGeneration &&- mutation.terminatedMembers.every((retired) =>- state.members.some(- (member) =>- member.instanceId === retired.instanceId &&- member.taskArn === retired.taskArn &&- member.generation === retired.generation &&- member.retired,- ),- )- );-}--export async function mutateBackgroundWork(+export async function setBackgroundOwner(transport: BackgroundWorkTransport,deploymentId: string,- mutation: BackgroundWorkMutation,+ change: BackgroundOwnerChange,): Promise<BackgroundWorkStatus> {- const body = JSON.stringify(mutation);+ const body = JSON.stringify(change);let response: { status: number; body: string } | undefined;for (let attempt = 0; attempt < 3; attempt++) {try {@@ -145,7 +109,7 @@ export async function mutateBackgroundWork(if (response?.status === 200) {try {const state = parseBackgroundWorkStatus(response.body, deploymentId);- if (mutationCommitted(state, mutation)) return state;+ if (state.ownerDeploymentId === change.ownerDeploymentId) return state;} catch {response = undefined;}@@ -156,70 +120,35 @@ export async function mutateBackgroundWork(} catch {observed = undefined;}- if (observed && mutationCommitted(observed, mutation)) return observed;- if (observed && observed.generation !== mutation.expectedGeneration)- throw new CliError("background ownership changed concurrently; refusing to replace another ownership generation");+ if (observed?.ownerDeploymentId === change.ownerDeploymentId) return observed;+ if (observed && observed.ownerDeploymentId !== change.expectedOwnerDeploymentId)+ throw new CliError("background ownership changed concurrently; refusing to replace another owner");if (response && response.status >= 400 && response.status < 500 && response.status !== 429) break;}throw new CliError(- `background ownership mutation is unconfirmed${response ? ` (HTTP ${response.status})` : ""}; retry the same request ID ${mutation.requestId} after reading ownership; automatic compensation is unsafe`,+ `background ownership change is unconfirmed${response ? ` (HTTP ${response.status})` : ""}; read ownership before retrying`,);}-export function backgroundWorkMutation(- expectedGeneration: number,- desiredDeploymentId: string | null,-): BackgroundWorkMutation {- return { expectedGeneration, desiredDeploymentId, requestId: randomUUID() };-}-export async function awaitBackgroundWork(transport: BackgroundWorkTransport,deploymentId: string,- expected: {- generation: number;- desiredDeploymentId: string | null;- taskArns: string[];- lastRequestId?: string | null;- },+ expected: { ownerDeploymentId: string | null; active: boolean; instances: number },options: { timeoutMs: number; pollMs: number },): Promise<BackgroundWorkStatus> {const deadline = Date.now() + options.timeoutMs;+ const confirmed = new Set<string>();for (;;) {const state = await readBackgroundWork(transport, deploymentId);- if (- !state.enabled ||- state.generation !== expected.generation ||- state.desiredDeploymentId !== expected.desiredDeploymentId ||- (expected.lastRequestId !== undefined && state.lastRequestId !== expected.lastRequestId)- ) {+ if (state.ownerDeploymentId !== expected.ownerDeploymentId)throw new CliError("background ownership changed while awaiting acknowledgment");- }- const priorOwners = state.members.filter(- (member) => !member.retired && member.generation < expected.generation && member.state === "admitted",- );- const ready =- expected.desiredDeploymentId === null- ? state.members.every((member) => member.retired || member.state !== "admitted")- : expected.taskArns.length > 0 &&- expected.taskArns.every((taskArn) =>- state.members.some(- (member) =>- member.taskArn === taskArn &&- member.deploymentId === expected.desiredDeploymentId &&- member.generation === expected.generation &&- member.state === "admitted" &&- member.ready &&- !member.retired,- ),- );- const unexpectedOwner = state.members.some(- (member) =>- !member.retired && member.state === "admitted" && member.deploymentId !== expected.desiredDeploymentId,- );- if (priorOwners.length === 0 && !unexpectedOwner && ready) return state;+ if (state.active === expected.active) confirmed.add(state.instanceId);+ else confirmed.clear();+ if (confirmed.size >= expected.instances) return state;if (Date.now() >= deadline)- throw new CliError("timed out awaiting background ownership acknowledgment; no member was inferred dead");+ throw new CliError(+ `timed out awaiting ${expected.instances} ${deploymentId} process(es) to ${expected.active ? "start" : "stop"} background work; ${confirmed.size} confirmed`,+ );await sleep(options.pollMs);}}diff --git a/cli/src/live-session.ts b/cli/src/live-session.tsindex c224e5e6f..93c89cff6 100644--- a/cli/src/live-session.ts+++ b/cli/src/live-session.ts@@ -2,63 +2,27 @@ import { randomUUID } from "node:crypto";import type { BackgroundWorkStatus } from "./background-work.ts";import { CliError } from "./log.ts";-export interface LiveSessionCohort {+export interface LiveSessionOwner {deploymentId: string;- taskArns: string[];status: BackgroundWorkStatus;}-function assertReady(cohort: LiveSessionCohort): void {- const { deploymentId, taskArns, status } = cohort;- if (- !status.enabled ||- status.deploymentId !== deploymentId ||- status.desiredDeploymentId !== deploymentId ||- taskArns.length === 0 ||- new Set(taskArns).size !== taskArns.length ||- taskArns.some(- (taskArn) =>- !status.members.some(- (member) =>- member.taskArn === taskArn &&- member.deploymentId === deploymentId &&- member.generation === status.generation &&- member.state === "admitted" &&- member.ready &&- !member.retired,- ),- ) ||- status.members.some(- (member) =>- !member.retired &&- member.state === "admitted" &&- (member.deploymentId !== deploymentId ||- member.generation !== status.generation ||- member.taskArn === null ||- !taskArns.includes(member.taskArn)),- )- )- throw new CliError("live session requires the exact ready deployment cohort to own background work");+function assertActiveOwner({ deploymentId, status }: LiveSessionOwner): void {+ if (status.deploymentId !== deploymentId || status.ownerDeploymentId !== deploymentId || !status.active)+ throw new CliError("live session requires an active deployment that owns background work");}export async function checkControlledLiveSession(options: {- before: LiveSessionCohort;- read: () => Promise<LiveSessionCohort>;+ before: LiveSessionOwner;+ read: () => Promise<LiveSessionOwner>;request: (body: string) => Promise<{ status: number; body: string }>;}): Promise<void> {const { before } = options;- assertReady(before);+ assertActiveOwner(before);const requestId = randomUUID();let response: { status: number; body: string };try {- response = await options.request(- JSON.stringify({- requestId,- expectedDeploymentId: before.deploymentId,- expectedGeneration: before.status.generation,- expectedTaskArns: before.taskArns,- }),- );+ response = await options.request(JSON.stringify({ requestId, expectedDeploymentId: before.deploymentId }));} catch {throw new CliError(`live session result is unconfirmed for request ${requestId}; no automatic replay or fallback was attempted`,@@ -78,38 +42,12 @@ export async function checkControlledLiveSession(options: {value.ok !== true ||value.requestId !== requestId ||value.deploymentId !== before.deploymentId ||- value.generation !== before.status.generation ||- !before.status.members.some(- (member) =>- member.instanceId === value.instanceId &&- member.taskArn === value.taskArn &&- before.taskArns.includes(member.taskArn ?? "") &&- member.deploymentId === before.deploymentId &&- member.generation === before.status.generation &&- member.state === "admitted" &&- member.ready &&- !member.retired,- )+ typeof value.instanceId !== "string" ||+ !value.instanceId)- throw new CliError(- `live session did not confirm success for the expected request and deployment cohort (${requestId})`,- );+ throw new CliError(`live session did not confirm success for the expected request and deployment (${requestId})`);const after = await options.read();- assertReady(after);- if (- after.deploymentId !== before.deploymentId ||- after.status.generation !== before.status.generation ||- after.taskArns.length !== before.taskArns.length ||- after.taskArns.some((arn) => !before.taskArns.includes(arn)) ||- !after.status.members.some(- (member) =>- member.instanceId === value.instanceId &&- member.taskArn === value.taskArn &&- member.state === "admitted" &&- member.ready &&- !member.retired &&- member.generation === before.status.generation,- )- )- throw new CliError("live session deployment cohort changed during qualification");+ assertActiveOwner(after);+ if (after.deploymentId !== before.deploymentId)+ throw new CliError("live session deployment changed during qualification");}diff --git a/cli/test/aws.test.ts b/cli/test/aws.test.tsindex c83406c85..d0873f127 100644--- a/cli/test/aws.test.ts+++ b/cli/test/aws.test.ts@@ -18,8 +18,6 @@ import {awsBackgroundWorkStatus,awsBackgroundWorkCapacity,awsBackgroundWorkBootState,- awsBootstrapBackgroundWork,- awsRetireBackgroundWorkMembers,awsDeploymentLayerTransport,awsDown,awsLogs,@@ -425,7 +423,6 @@ else if (a.includes("ecs describe-tasks") && s.taskInventory) {else if (a.includes("ecs list-tasks") && s.taskInventory?.[after("--service-name")]) console.log(JSON.stringify({ taskArns: after("--desired-status") === "STOPPED" ? [] : s.taskInventory[after("--service-name")].map(task => task.taskArn) }));else if (a.includes("ecs describe-tasks") && Object.values(s.taskInventory || {}).flat().some(task => args.includes(task.taskArn))) console.log(JSON.stringify({ tasks: Object.values(s.taskInventory).flat().filter(task => args.includes(task.taskArn)), failures: [] }));else if (a.includes("ecs list-tasks")) console.log(JSON.stringify({ taskArns: process.env.AWS_FAKE_NO_RUNNING_TASK ? [] : [...(process.env.AWS_FAKE_LARGE_ROLLOUT ? Array.from({ length: 100 }, (_, i) => "arn:aws:ecs:us-west-2:123456789012:task/old-core-" + i) : []), "arn:aws:ecs:us-west-2:123456789012:task/live-core"] }));-else if (a.includes("ecs describe-tasks") && s.stoppedTasks?.[after("--tasks")]) console.log(JSON.stringify({tasks: [s.stoppedTasks[after("--tasks")]], failures: []}));else if (a.includes("ecs describe-tasks") && a.includes("task/old-core-")) console.log(JSON.stringify({ tasks: [] }));else if (a.includes("ecs describe-tasks") && a.includes("task/live-core")) console.log(JSON.stringify({ tasks: [{ taskArn: "arn:aws:ecs:us-west-2:123456789012:task/live-core", taskDefinitionArn: process.env.AWS_FAKE_STALE_CORE ? "stale-task-definition" : s.services["acme-core"].taskDefinition, lastStatus: "RUNNING", healthStatus: "HEALTHY", containers: [{ name: "core", networkInterfaces: [{ privateIpv4Address: "10.0.1.8" }] }] }] }));else if (a.includes("ecs run-task")) console.log(JSON.stringify({ tasks: [{ taskArn: "arn:aws:ecs:us-west-2:123456789012:task/canary" }] }));@@ -5737,7 +5734,7 @@ test("AWS deployment progress rejects invalid options before AWS calls", async (}});-test("inactive capacity proves exact relinquished cohorts without mutating deployment state", async () => {+test("inactive capacity proves the stack is not the owner and its tasks are stable without mutating deployment state", async () => {const dir = mkdtempSync(join(tmpdir(), "qm-capacity-"));const dockerBin = join(dir, "docker");writeFileSync(dockerBin, `#!/usr/bin/env node\nconsole.log("Digest: sha256:${"a".repeat(64)}");\n`);@@ -5772,31 +5769,23 @@ test("inactive capacity proves exact relinquished cohorts without mutating deplo);const coreArn = baseline.taskInventory[configured.aws!.services.core!.ecsService][0].taskArn;const ownership = {- protocol: 1,- enabled: true,+ protocol: 2,deploymentId: manifest.backgroundDeploymentId,instanceId: "inactive-instance",- generation: 2,- desiredDeploymentId: "peer:active",- lastRequestId: "handover",- members: [- {- instanceId: "inactive-instance",- deploymentId: manifest.backgroundDeploymentId,- taskArn: coreArn,- generation: 1,- state: "drained",- ready: false,- retired: false,- },- ],+ ownerDeploymentId: "peer:active" as string | null,+ setAt: "2026-10-04T00:00:00.000Z",+ setBy: "peer:active",+ active: false,};let reads = 0;let changeOwnership = false;globalThis.fetch = async () => {reads++;return new Response(- JSON.stringify({ ...ownership, generation: ownership.generation + (changeOwnership && reads > 1 ? 1 : 0) }),+ JSON.stringify({+ ...ownership,+ ownerDeploymentId: changeOwnership && reads > 1 ? "peer:other" : ownership.ownerDeploymentId,+ }),);};const reset = () => {@@ -5808,8 +5797,7 @@ test("inactive capacity proves exact relinquished cohorts without mutating deploconst proof = await awsBackgroundWorkCapacity(configured, dir);assert.equal(proof.manifestId, manifest.id);assert.equal(proof.deploymentId, manifest.backgroundDeploymentId);- assert.equal(proof.generation, 2);- assert.equal(proof.desiredDeploymentId, "peer:active");+ assert.equal(proof.ownerDeploymentId, "peer:active");assert.deepEqual(Object.keys(proof.workloads), ["core", "web-ui"]);assert.deepEqual(proof.workloads.core!.taskArns, [coreArn]);assert.equal((await awsBackgroundWorkStatus(configured, dir)).manifestId, manifest.id);@@ -5820,17 +5808,12 @@ test("inactive capacity proves exact relinquished cohorts without mutating deplo/transact-write|put-item|delete-item|update-item|update-service|register-task-definition|run-task|stop-task/,);- ownership.members[0]!.state = "admitted";- reset();- await assert.rejects(awsBackgroundWorkCapacity(configured, dir), /every member to relinquish/);- ownership.members[0]!.state = "relinquished";- reset();- assert.deepEqual(await awsBackgroundWorkCapacity(configured, dir), proof);- ownership.members[0]!.state = "drained";- ownership.desiredDeploymentId = manifest.backgroundDeploymentId;- reset();- await assert.rejects(awsBackgroundWorkCapacity(configured, dir), /held by another deployment/);- ownership.desiredDeploymentId = "peer:active";+ for (const owner of [manifest.backgroundDeploymentId, null]) {+ ownership.ownerDeploymentId = owner;+ reset();+ await assert.rejects(awsBackgroundWorkCapacity(configured, dir), /held by another deployment/);+ }+ ownership.ownerDeploymentId = "peer:active";reset();const unresolved = structuredClone(baseline);@@ -5943,27 +5926,16 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECSassert.ok(preparedAt >= 0 && preparedAt < preparationCalls.indexOf("ecs register-task-definition"));const taskArn = "arn:aws:ecs:us-west-2:123456789012:task/live-core";const ownership = {- protocol: 1,- enabled: false,+ protocol: 2,deploymentId: manifest.backgroundDeploymentId,instanceId: "instance-one",- generation: 0,- desiredDeploymentId: null as string | null,- lastRequestId: null as string | null,- members: [- {- instanceId: "instance-one",- taskArn,- deploymentId: manifest.backgroundDeploymentId,- generation: 0,- state: "drained",- retired: false,- ready: false,- },- ],+ ownerDeploymentId: null as string | null,+ setAt: null as string | null,+ setBy: null as string | null,+ active: false,};const fetchLayer = globalThis.fetch;- const mutations: unknown[] = [];+ const mutations: Array<Record<string, unknown>> = [];let smokeCalls = 0;let smokeFailure = false;globalThis.fetch = async (url, init) => {@@ -5979,9 +5951,10 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECS.update(`v0:${headers.get("x-timestamp")}:POST\n/v1/deployment/live-session\n${body}`).digest("hex")}`,);- assert.equal(request.expectedDeploymentId, manifest.backgroundDeploymentId);- assert.equal(request.expectedGeneration, ownership.generation);- assert.deepEqual(request.expectedTaskArns, [taskArn]);+ assert.deepEqual(request, {+ requestId: request.requestId,+ expectedDeploymentId: manifest.backgroundDeploymentId,+ });assert.ok(init?.signal instanceof AbortSignal);if (smokeFailure) throw new Error("disconnected with a sensitive credential");return new Response(@@ -5989,9 +5962,7 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECSok: true,requestId: request.requestId,deploymentId: ownership.deploymentId,- generation: ownership.generation,instanceId: "instance-one",- taskArn,})}\n`,);}@@ -6006,23 +5977,11 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECSif (init?.method === "POST") {const mutation = JSON.parse(body);mutations.push(mutation);- assert.equal(mutation.expectedGeneration, ownership.generation);- ownership.lastRequestId = mutation.requestId;- if (mutation.terminatedMembers) {- for (const proof of mutation.terminatedMembers) {- const member = ownership.members.find((item) => item.instanceId === proof.instanceId)!;- member.retired = true;- member.state = "drained";- member.ready = false;- }- } else {- ownership.enabled = true;- ownership.generation++;- ownership.desiredDeploymentId = mutation.desiredDeploymentId;- ownership.members[0]!.generation = ownership.generation;- ownership.members[0]!.state = mutation.desiredDeploymentId ? "admitted" : "relinquished";- ownership.members[0]!.ready = Boolean(mutation.desiredDeploymentId);- }+ assert.equal(mutation.expectedOwnerDeploymentId, ownership.ownerDeploymentId);+ ownership.ownerDeploymentId = mutation.ownerDeploymentId;+ ownership.setAt = new Date().toISOString();+ ownership.setBy = ownership.deploymentId;+ ownership.active = mutation.ownerDeploymentId === ownership.deploymentId;}return new Response(JSON.stringify(ownership), { status: 200 });};@@ -6031,39 +5990,14 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECSawait awsCheckLive(single, { configDir: dir, report: false });assert.equal(smokeCalls, 0);assert.match(readFileSync(fake.log, "utf8"), /ecs run-task/);- await assert.rejects(awsSetBackgroundWork(single, dir, true), /explicitly bootstrap/);- const stoppedLegacyArn = "arn:aws:ecs:us-west-2:123456789012:task/stopped-legacy-core";- ownership.members.push({- ...ownership.members[0]!,- instanceId: "legacy-dead",- taskArn: stoppedLegacyArn,- state: "admitted",- });- const beforeBootstrap = JSON.parse(readFileSync(fake.state, "utf8"));- beforeBootstrap.stoppedTasks = {- [stoppedLegacyArn]: {- taskArn: stoppedLegacyArn,- lastStatus: "STOPPED",- group: "service:acme-core",- taskDefinitionArn: manifest.tasks.core,- },- };- writeFileSync(fake.state, JSON.stringify(beforeBootstrap));- await assert.rejects(- awsBootstrapBackgroundWork([{ config: single, configDir: dir }], manifest.backgroundDeploymentId),- /every enrolled/,- );- const retiredLegacy = await awsRetireBackgroundWorkMembers(- [{ config: single, configDir: dir }],- [{ instanceId: "legacy-dead", taskArn: stoppedLegacyArn, generation: 0 }],- );- assert.equal(retiredLegacy.generation, 0);- assert.equal(retiredLegacy.enabled, false);- const bootstrapped = await awsBootstrapBackgroundWork(- [{ config: single, configDir: dir }],- manifest.backgroundDeploymentId,- );- assert.equal(bootstrapped.generation, 1);+ const activated = await awsSetBackgroundWork(single, dir, true);+ assert.equal(activated?.ownerDeploymentId, manifest.backgroundDeploymentId);+ assert.equal(activated?.active, true);+ assert.deepEqual(mutations, [+ { ownerDeploymentId: manifest.backgroundDeploymentId, expectedOwnerDeploymentId: null },+ ]);+ assert.deepEqual(await awsSetBackgroundWork(single, dir, true), activated);+ assert.equal(mutations.length, 1);writeFileSync(fake.log, "");await awsCheckLive(single, { configDir: dir, report: false });assert.equal(smokeCalls, 1);@@ -6073,80 +6007,39 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECSassert.equal(smokeCalls, 2);assert.doesNotMatch(readFileSync(fake.log, "utf8"), /ecs run-task/);smokeFailure = false;- const activeOwnership = structuredClone(ownership);- ownership.generation++;- ownership.desiredDeploymentId = "other-owner";- ownership.members[0]!.state = "relinquished";- ownership.members[0]!.ready = false;- writeFileSync(fake.log, "");- await awsCheckLive(single, { configDir: dir, report: false });- assert.equal(smokeCalls, 2);- assert.match(readFileSync(fake.log, "utf8"), /ecs run-task/);- Object.assign(ownership, activeOwnership);writeFileSync(fake.log, "");- await assert.rejects(awsUp(single, dir, { yes: true }), /pause or hand over/);+ await assert.rejects(awsUp(single, dir, { yes: true }), /hand background ownership/);+ await assert.rejects(awsRollback(single, manifest.id), /hand background ownership/);assert.doesNotMatch(readFileSync(fake.log, "utf8"),/ecs update-service|register-task-definition|s3api put-object|ecr get-login-password/,);- ownership.desiredDeploymentId = "another-current-owner";- const beforeWrongPause = mutations.length;+ const activeOwnership = structuredClone(ownership);+ ownership.ownerDeploymentId = "other-owner";+ ownership.active = false;+ writeFileSync(fake.log, "");+ await awsCheckLive(single, { configDir: dir, report: false });+ assert.equal(smokeCalls, 2);+ assert.match(readFileSync(fake.log, "utf8"), /ecs run-task/);await assert.rejects(awsSetBackgroundWork(single, dir, false), /different deployment/);- assert.equal(mutations.length, beforeWrongPause);- ownership.desiredDeploymentId = manifest.backgroundDeploymentId;- const beforeCompensation = mutations.length;- for (const expected of [- { generation: ownership.generation - 1, lastRequestId: ownership.lastRequestId },- { generation: ownership.generation, lastRequestId: "different-request" },- ]) {+ assert.equal(mutations.length, 1);+ Object.assign(ownership, activeOwnership);+ for (const expected of [{ ownerDeploymentId: null }, { ownerDeploymentId: "other-owner" }]) {await assert.rejects(awsSetBackgroundWork(single, dir, false, undefined, expected), /changed since promotion/);}- assert.equal(mutations.length, beforeCompensation);- await awsSetBackgroundWork(single, dir, false, undefined, {- generation: ownership.generation,- lastRequestId: ownership.lastRequestId,+ assert.equal(mutations.length, 1);+ const paused = await awsSetBackgroundWork(single, dir, false, undefined, {+ ownerDeploymentId: manifest.backgroundDeploymentId,+ });+ assert.equal(paused?.ownerDeploymentId, null);+ assert.equal(paused?.active, false);+ assert.deepEqual(mutations.at(-1), {+ ownerDeploymentId: null,+ expectedOwnerDeploymentId: manifest.backgroundDeploymentId,});- const confirmed = await awsSetBackgroundWork(single, dir, true);- assert.equal(confirmed?.generation, 3);- assert.equal(ownership.generation, 3);- assert.equal(mutations.length, 4);- assert.doesNotMatch(readFileSync(fake.log, "utf8"), /ecs update-service|register-task-definition|run-task/);+ assert.doesNotMatch(readFileSync(fake.log, "utf8"), /ecs update-service|register-task-definition/);const unchanged = JSON.parse(readFileSync(fake.state, "utf8"));assert.deepEqual(unchanged.dynamo, persisted.dynamo);- await assert.rejects(- awsRetireBackgroundWorkMembers(- [{ config: single, configDir: dir }],- [{ instanceId: "instance-one", taskArn, generation: 3 }],- ),- /STOPPED evidence/,- );- const stoppedArn = "arn:aws:ecs:us-west-2:123456789012:task/stopped-core";- ownership.members.push({- ...ownership.members[0]!,- instanceId: "terminated-instance",- taskArn: stoppedArn,- generation: 1,- });- unchanged.stoppedTasks = {- [stoppedArn]: {- taskArn: stoppedArn,- lastStatus: "STOPPED",- group: "service:acme-core",- taskDefinitionArn: manifest.tasks.core,- },- };- writeFileSync(fake.state, JSON.stringify(unchanged));- const retired = await awsRetireBackgroundWorkMembers(- [{ config: single, configDir: dir }],- [{ instanceId: "terminated-instance", taskArn: stoppedArn, generation: 1 }],- );- assert.equal(retired.generation, 3);- assert.equal(retired.members.at(-1)!.retired, true);- await awsSetBackgroundWork(single, dir, false);- ownership.members[0]!.state = "admitted";- await assert.rejects(awsUp(single, dir, { yes: true, restart: ["core"] }), /every member to relinquish/);- await assert.rejects(awsRollback(single, manifest.id), /every member to relinquish/);- ownership.members[0]!.state = "relinquished";await awsUp(single, dir, { yes: true, restart: ["core"] });const replacement = JSON.parse(readFileSync(fake.state, "utf8"));const replacementManifest = JSON.parse(@@ -6155,7 +6048,6 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECSassert.notEqual(replacementManifest.backgroundDeploymentId, manifest.backgroundDeploymentId);assert.notEqual(replacementManifest.tasks.core, manifest.tasks.core);ownership.deploymentId = replacementManifest.backgroundDeploymentId;- ownership.members[0]!.deploymentId = replacementManifest.backgroundDeploymentId;const controlEnv = join(dir, "control.env");writeFileSync(controlEnv,@@ -6186,7 +6078,7 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECSwriteFileSync(fake.state, JSON.stringify(replacement));ownership.deploymentId = "stale-cohort";await assert.rejects(awsSetBackgroundWork(single, dir, false), /requested deployment/);- assert.equal(mutations.length, 6);+ assert.equal(mutations.length, 2);ownership.deploymentId = replacementManifest.backgroundDeploymentId;await awsRollback(single, manifest.id);const rolledBack = JSON.parse(readFileSync(fake.state, "utf8"));diff --git a/cli/test/background-work.test.ts b/cli/test/background-work.test.tsindex bdc5674f7..3f2446afd 100644--- a/cli/test/background-work.test.ts+++ b/cli/test/background-work.test.ts@@ -2,191 +2,177 @@ import test from "node:test";import assert from "node:assert/strict";import {awaitBackgroundWork,- mutateBackgroundWork,parseBackgroundWorkStatus,+ setBackgroundOwner,type BackgroundWorkStatus,type BackgroundWorkTransport,} from "../src/background-work.ts";-const state = (): BackgroundWorkStatus => ({- protocol: 1,- enabled: true,+const state = (patch: Partial<BackgroundWorkStatus> = {}): BackgroundWorkStatus => ({+ protocol: 2,deploymentId: "new-cohort",instanceId: "new-instance",- generation: 2,- desiredDeploymentId: "new-cohort",- lastRequestId: "our-request",- members: [- {- instanceId: "old-instance",- taskArn: "old-task",- deploymentId: "old-cohort",- generation: 1,- state: "relinquished",- retired: false,- ready: true,- },- {- instanceId: "new-instance",- taskArn: "new-task",- deploymentId: "new-cohort",- generation: 2,- state: "admitted",- retired: false,- ready: true,- },- ],+ ownerDeploymentId: "new-cohort",+ setAt: "2026-10-04T00:00:00.000Z",+ setBy: "new-cohort",+ active: true,+ ...patch,});-const response = (value: BackgroundWorkStatus) => ({ status: 200, body: JSON.stringify(value) });+const response = (value: unknown) => ({ status: 200, body: JSON.stringify(value) });-test("ownership status refuses stale responders and malformed durable membership", () => {- assert.equal(parseBackgroundWorkStatus(JSON.stringify(state()), "new-cohort").generation, 2);+test("ownership status refuses other responders and malformed records", () => {+ assert.deepEqual(parseBackgroundWorkStatus(JSON.stringify(state()), "new-cohort"), state());assert.throws(() => parseBackgroundWorkStatus(JSON.stringify(state()), "old-cohort"), /requested deployment/);- const duplicate = state();- duplicate.members.push(duplicate.members[0]!);- assert.throws(() => parseBackgroundWorkStatus(JSON.stringify(duplicate), "new-cohort"), /duplicate/);- const unenrolled = state();- unenrolled.instanceId = "missing";- assert.throws(() => parseBackgroundWorkStatus(JSON.stringify(unenrolled), "new-cohort"), /not enrolled/);+ assert.throws(() => parseBackgroundWorkStatus("not json", "new-cohort"), /invalid JSON/);+ for (const patch of [+ { protocol: 3 },+ { ownerDeploymentId: 4 },+ { ownerDeploymentId: "" },+ { active: "yes" },+ { setAt: 12 },+ { instanceId: "" },+ ])+ assert.throws(+ () => parseBackgroundWorkStatus(JSON.stringify({ ...state(), ...patch }), "new-cohort"),+ /does not match/,+ );+ const paused = parseBackgroundWorkStatus(+ JSON.stringify(state({ ownerDeploymentId: null, active: false })),+ "new-cohort",+ );+ assert.equal(paused.ownerDeploymentId, null);});-test("lost mutation responses require the exact request ID and generation on durable readback", async () => {+test("a member-protocol responder is read as its enabled desired owner", () => {+ const legacy = {+ protocol: 1,+ enabled: true,+ deploymentId: "old-cohort",+ instanceId: "old-instance",+ generation: 4,+ desiredDeploymentId: "old-cohort",+ lastRequestId: "handover",+ members: [+ {+ instanceId: "old-instance",+ taskArn: "old-task",+ deploymentId: "old-cohort",+ generation: 4,+ state: "admitted",+ retired: false,+ ready: true,+ },+ ],+ };+ const parsed = parseBackgroundWorkStatus(JSON.stringify(legacy), "old-cohort");+ assert.equal(parsed.protocol, 2);+ assert.equal(parsed.ownerDeploymentId, "old-cohort");+ assert.equal(parsed.active, true);+ assert.equal(parsed.setAt, null);+ legacy.members[0]!.state = "relinquished";+ assert.equal(parseBackgroundWorkStatus(JSON.stringify(legacy), "old-cohort").active, false);+ legacy.enabled = false;+ assert.equal(parseBackgroundWorkStatus(JSON.stringify(legacy), "old-cohort").ownerDeploymentId, null);+ const rewritten = { ...legacy, enabled: true, desiredDeploymentId: "new-cohort", members: [] };+ assert.deepEqual(parseBackgroundWorkStatus(JSON.stringify(rewritten), "old-cohort"), {+ protocol: 2,+ deploymentId: "old-cohort",+ instanceId: "old-instance",+ ownerDeploymentId: "new-cohort",+ setAt: null,+ setBy: null,+ active: false,+ });+ assert.throws(+ () => parseBackgroundWorkStatus(JSON.stringify({ ...legacy, members: "none" }), "old-cohort"),+ /does not match/,+ );+});++test("a lost change response is confirmed by reading the owner back", async () => {const calls: string[] = [];- const mutation = { expectedGeneration: 1, desiredDeploymentId: "new-cohort", requestId: "our-request" };+ const change = { ownerDeploymentId: "new-cohort", expectedOwnerDeploymentId: "old-cohort" };let observed = state();const transport: BackgroundWorkTransport = async (method, body) => {calls.push(method);if (method === "POST") {- assert.deepEqual(JSON.parse(body!), mutation);+ assert.deepEqual(JSON.parse(body!), change);throw new Error("connection lost after commit");}return response(observed);};- assert.equal((await mutateBackgroundWork(transport, "new-cohort", mutation)).generation, 2);+ assert.equal((await setBackgroundOwner(transport, "new-cohort", change)).ownerDeploymentId, "new-cohort");assert.deepEqual(calls, ["POST", "GET"]);- observed = { ...state(), lastRequestId: "someone-else" };- await assert.rejects(mutateBackgroundWork(transport, "new-cohort", mutation), /changed concurrently/);-});--test("retirement acknowledgment binds every terminated instance, task and generation", async () => {- const mutation = {- expectedGeneration: 2,- requestId: "retirement",- terminatedMembers: [{ instanceId: "old-instance", taskArn: "old-task", generation: 1 }],- };- const observed = state();- observed.lastRequestId = "retirement";- observed.members[0]!.retired = true;- const transport: BackgroundWorkTransport = async () => response(observed);- assert.equal((await mutateBackgroundWork(transport, "new-cohort", mutation)).generation, 2);- observed.members[0]!.taskArn = "different-task";- await assert.rejects(mutateBackgroundWork(transport, "new-cohort", mutation), /unconfirmed/);-});--test("activation waits for every old owner and every expected new task without waiting for old turns to drain", async () => {- const observed = state();- observed.members[0]!.state = "admitted";- let polls = 0;- const transport: BackgroundWorkTransport = async () => {- polls++;- if (polls === 2) observed.members[0]!.state = "relinquished";- if (polls === 3)- observed.members.push({- ...observed.members[1]!,- instanceId: "second-instance",- taskArn: "second-task",- ready: false,- });- if (polls === 4) observed.members[2]!.ready = true;- return response(observed);- };- const ready = await awaitBackgroundWork(- transport,- "new-cohort",- { generation: 2, desiredDeploymentId: "new-cohort", taskArns: ["new-task", "second-task"] },- { timeoutMs: 10_000, pollMs: 1 },- );- assert.equal(polls, 4);- assert.equal(ready.members[0]!.state, "relinquished");-});--test("pause never infers unacknowledged owners dead and refuses concurrent generations", async () => {- const observed = state();- observed.desiredDeploymentId = null;- observed.members[0]!.state = "admitted";- observed.members[1]!.state = "drained";- const transport: BackgroundWorkTransport = async () => response(observed);- const expected = { generation: 2, desiredDeploymentId: null, taskArns: [] };- await assert.rejects(- awaitBackgroundWork(transport, "new-cohort", expected, { timeoutMs: 0, pollMs: 1 }),- /no member was inferred dead/,- );- observed.members[0]!.state = "relinquished";- assert.equal(- (await awaitBackgroundWork(transport, "new-cohort", expected, { timeoutMs: 0, pollMs: 1 })).generation,- 2,- );- observed.generation = 3;- await assert.rejects(- awaitBackgroundWork(transport, "new-cohort", expected, { timeoutMs: 0, pollMs: 1 }),- /changed while awaiting/,- );+ observed = state({ ownerDeploymentId: "someone-else" });+ await assert.rejects(setBackgroundOwner(transport, "new-cohort", change), /changed concurrently/);});for (const mode of ["late-commit", "unavailable-read", "never-commits", "competitor", "rejected"] as const) {- test(`mutation confirmation handles ${mode} without changing its request`, async () => {- const mutation = { expectedGeneration: 1, desiredDeploymentId: "new-cohort", requestId: "our-request" };+ test(`owner change confirmation handles ${mode} without changing its request`, async () => {+ const change = { ownerDeploymentId: "new-cohort", expectedOwnerDeploymentId: "old-cohort" };let posts = 0;let reads = 0;- const old = {- ...state(),- generation: 1,- desiredDeploymentId: "old-cohort",- lastRequestId: "previous",- members: state().members.map((member) => ({ ...member, generation: 1 })),- };+ const old = state({ ownerDeploymentId: "old-cohort", active: false });const transport: BackgroundWorkTransport = async (method, body) => {if (method === "POST") {posts++;- assert.equal(body, JSON.stringify(mutation));+ assert.equal(body, JSON.stringify(change));if (mode === "rejected") return { status: 403, body: "denied" };if (posts > 1 && (mode === "late-commit" || mode === "unavailable-read")) return response(state());throw new Error("response timed out before commit");}reads++;if (mode === "unavailable-read") throw new Error("temporarily unavailable");- if (mode === "competitor") return response({ ...state(), lastRequestId: "competitor" });+ if (mode === "competitor") return response(state({ ownerDeploymentId: "competitor" }));return response(old);};if (mode === "late-commit" || mode === "unavailable-read") {- assert.equal((await mutateBackgroundWork(transport, "new-cohort", mutation)).generation, 2);+ assert.equal((await setBackgroundOwner(transport, "new-cohort", change)).ownerDeploymentId, "new-cohort");assert.equal(posts, 2);assert.equal(reads, 1);} else {await assert.rejects(- mutateBackgroundWork(transport, "new-cohort", mutation),- mode === "competitor" ? /changed concurrently/ : /unconfirmed.*automatic compensation is unsafe/,+ setBackgroundOwner(transport, "new-cohort", change),+ mode === "competitor" ? /changed concurrently/ : /unconfirmed/,);assert.equal(posts, mode === "never-commits" ? 3 : 1);}});}-test("readiness cannot replace the confirmed mutation identity with a same-generation retirement", async () => {- const observed = { ...state(), lastRequestId: "operator-retirement" };+test("activation waits for every expected process to report activity and refuses a changed owner", async () => {+ let polls = 0;+ const transport: BackgroundWorkTransport = async () => {+ polls++;+ const instanceId = polls % 2 ? "first" : "second";+ if (polls === 4) return response(state({ instanceId, active: false }));+ return response(state({ instanceId, active: polls >= 3 }));+ };+ const ready = await awaitBackgroundWork(+ transport,+ "new-cohort",+ { ownerDeploymentId: "new-cohort", active: true, instances: 2 },+ { timeoutMs: 10_000, pollMs: 1 },+ );+ assert.equal(polls, 6);+ assert.equal(ready.active, true);await assert.rejects(awaitBackgroundWork(- async () => response(observed),+ async () => response(state({ ownerDeploymentId: "other" })),"new-cohort",- {- generation: 2,- desiredDeploymentId: "new-cohort",- taskArns: ["new-task"],- lastRequestId: "our-request",- },+ { ownerDeploymentId: "new-cohort", active: true, instances: 1 },{ timeoutMs: 0, pollMs: 1 },),/changed while awaiting/,);+ await assert.rejects(+ awaitBackgroundWork(+ async () => response(state({ ownerDeploymentId: null, active: true })),+ "new-cohort",+ { ownerDeploymentId: null, active: false, instances: 1 },+ { timeoutMs: 0, pollMs: 1 },+ ),+ /timed out awaiting 1 new-cohort process\(es\) to stop background work; 0 confirmed/,+ );});diff --git a/cli/test/live-session.test.ts b/cli/test/live-session.test.tsindex a26159631..cc2ac2566 100644--- a/cli/test/live-session.test.ts+++ b/cli/test/live-session.test.ts@@ -1,86 +1,59 @@import test from "node:test";import assert from "node:assert/strict";-import { checkControlledLiveSession, type LiveSessionCohort } from "../src/live-session.ts";+import { checkControlledLiveSession, type LiveSessionOwner } from "../src/live-session.ts";-function cohort(): LiveSessionCohort {+function owner(): LiveSessionOwner {return {deploymentId: "core:release",- taskArns: ["task-a", "task-b"],status: {- protocol: 1,- enabled: true,+ protocol: 2,deploymentId: "core:release",instanceId: "instance-a",- generation: 4,- desiredDeploymentId: "core:release",- lastRequestId: "handover",- members: ["a", "b"].map((suffix) => ({- instanceId: `instance-${suffix}`,- taskArn: `task-${suffix}`,- deploymentId: "core:release",- generation: 4,- state: "admitted",- retired: false,- ready: true,- })),+ ownerDeploymentId: "core:release",+ setAt: "2026-10-04T00:00:00.000Z",+ setBy: "core:release",+ active: true,},};}function success(body: string): Record<string, unknown> {const request = JSON.parse(body);+ assert.deepEqual(Object.keys(request).sort(), ["expectedDeploymentId", "requestId"]);assert.equal(request.expectedDeploymentId, "core:release");- assert.equal(request.expectedGeneration, 4);- assert.deepEqual(request.expectedTaskArns, ["task-a", "task-b"]);- return {- ok: true,- requestId: request.requestId,- deploymentId: "core:release",- generation: 4,- instanceId: "instance-a",- taskArn: "task-a",- };+ return { ok: true, requestId: request.requestId, deploymentId: "core:release", instanceId: "instance-a" };}-test("live session accepts whitespace heartbeats and checks the exact cohort afterward", async () => {+test("live session accepts whitespace heartbeats and checks ownership afterward", async () => {let reads = 0;await checkControlledLiveSession({- before: cohort(),+ before: owner(),request: async (body) => ({ status: 200, body: ` \n\t\n${JSON.stringify(success(body))}\n` }),read: async () => {reads++;- return cohort();+ return owner();},});assert.equal(reads, 1);});-test("live session refuses incomplete or competing ownership before requesting a canary", async () => {- const mutations: Array<(value: LiveSessionCohort) => void> = [+test("live session refuses an inactive or non-owning deployment before requesting a canary", async () => {+ const mutations: Array<(value: LiveSessionOwner) => void> = [(value) => {- value.status.enabled = false;+ value.status.ownerDeploymentId = null;},(value) => {- value.status.desiredDeploymentId = "other";+ value.status.ownerDeploymentId = "other";},(value) => {- value.status.members[0]!.ready = false;+ value.status.active = false;},(value) => {- value.status.members[0]!.generation--;- },- (value) => {- value.status.members[0]!.retired = true;- },- (value) => {- value.status.members[0]!.state = "relinquished";- },- (value) => {- value.status.members.push({ ...value.status.members[0]!, instanceId: "other", taskArn: "unexpected" });+ value.status.deploymentId = "other";},];for (const mutate of mutations) {- const before = cohort();+ const before = owner();mutate(before);await assert.rejects(checkControlledLiveSession({@@ -88,32 +61,30 @@ test("live session refuses incomplete or competing ownership before requesting arequest: async () => {assert.fail("no canary before ownership proof");},- read: async () => cohort(),+ read: async () => owner(),}),- /exact ready deployment cohort/,+ /active deployment that owns background work/,);}});-test("live session requires an explicit final success bound to request and instance", async () => {+test("live session requires an explicit final success bound to request and deployment", async () => {for (const patch of [{ ok: false, error: "sensitive server error" },{ ok: undefined },{ requestId: "another-request" },{ deploymentId: "stale" },- { generation: 3 },- { instanceId: "old-instance" },- { taskArn: "old-task" },+ { instanceId: "" },]) {let calls = 0;await assert.rejects(checkControlledLiveSession({- before: cohort(),+ before: owner(),request: async (body) => {calls++;return { status: 200, body: JSON.stringify({ ...success(body), ...patch }) };},- read: async () => cohort(),+ read: async () => owner(),}),(error: Error) => {assert.match(error.message, /did not confirm success/);@@ -130,7 +101,7 @@ test("live session never retries an ambiguous or truncated response", async () =let calls = 0;await assert.rejects(checkControlledLiveSession({- before: cohort(),+ before: owner(),request: async () => {calls++;if (kind === "disconnect") throw new Error("Bearer secret must never escape");@@ -139,7 +110,7 @@ test("live session never retries an ambiguous or truncated response", async () =body: kind === "heartbeat-only" ? " \n\n" : '{"ok":true',};},- read: async () => cohort(),+ read: async () => owner(),}),(error: Error) => {assert.doesNotMatch(error.message, /Bearer secret/);@@ -150,33 +121,30 @@ test("live session never retries an ambiguous or truncated response", async () =}});-test("live session rejects ownership or task replacement during a successful model check", async () => {- const mutations: Array<(value: LiveSessionCohort) => void> = [- (value) => {- value.status.generation++;- value.status.members.forEach((member) => member.generation++);- },+test("live session rejects an ownership change during a successful model check", async () => {+ const mutations: Array<(value: LiveSessionOwner) => void> = [(value) => {- value.taskArns[1] = "replacement";- value.status.members[1]!.taskArn = "replacement";+ value.status.ownerDeploymentId = null;},(value) => {- value.status.members[0]!.instanceId = "replacement-instance";+ value.status.active = false;},(value) => {- value.status.desiredDeploymentId = null;+ value.deploymentId = "core:replacement";+ value.status.deploymentId = "core:replacement";+ value.status.ownerDeploymentId = "core:replacement";},];for (const mutate of mutations) {- const after = cohort();+ const after = owner();mutate(after);await assert.rejects(checkControlledLiveSession({- before: cohort(),+ before: owner(),request: async (body) => ({ status: 200, body: JSON.stringify(success(body)) }),read: async () => after,}),- /cohort/,+ /owns background work|deployment changed/,);}});diff --git a/docs/background-ownership.md b/docs/background-ownership.mdindex fab396345..33b125eb4 100644--- a/docs/background-ownership.md+++ b/docs/background-ownership.md@@ -1,6 +1,6 @@# Background ownership-A core deployment can transfer background work to another deployment without restarting either HTTP process. Existing runs retain their leases and heartbeats while the old deployment stops accepting background work. The deployment controller uses a shared PostgreSQL record to fence generations and record each process's acknowledgment.+Production runs two core deployments (blue and green) against one PostgreSQL database. Exactly one of them may run background work at a time: Slack ingress, run claims, cron callbacks, and periodic maintenance. A single shared owner record names that deployment. Every core process polls the record and admits background work only while it names the process's own deployment.## Enable the capability@@ -13,109 +13,72 @@ Configure every core replica with:| `CORE_SIGNING_SECRET` | The normal source request signing secret. || `DATABASE_URL` | The shared PostgreSQL database. |-On ECS, `ECS_CONTAINER_METADATA_URI_V4` supplies the task ARN used to associate process membership with infrastructure termination evidence. A process without task metadata can enroll, but cannot participate in an ECS cohort bootstrap or task retirement.+With `BACKGROUND_DEPLOYMENT_ID` set, `BACKGROUND_WORK_ENABLED` is ignored: the owner record alone decides. A fresh database has no owner, so no deployment runs background work until an operator sets one. Without `BACKGROUND_DEPLOYMENT_ID`, the process follows `BACKGROUND_WORK_ENABLED` as before and the legacy build-heartbeat supersession applies.-Installing this capability does not activate the protocol. Until explicit bootstrap, each replica follows `BACKGROUND_WORK_ENABLED`. Upgrade both deployments and verify their exact current task cohort before bootstrapping. Do not fall back to boot-flag mutation after the protocol has been enabled.+## The owner record++The record lives in the `background_ownership` durable map under the key `ownership`:++| Field | Meaning |+| ------------------- | ------------------------------------------------------------------ |+| `ownerDeploymentId` | The deployment that may run background work, or `null` for nobody. |+| `setAt` | When the owner last changed. |+| `setBy` | The deployment whose API accepted the change. |++There are no per-process rows. Processes never write the record; only the operator endpoint does. A record written by the earlier member protocol (`enabled`, `desiredDeploymentId`, `members`, …) is read as `enabled ? desiredDeploymentId : null`; reads and no-op sets leave it untouched. The next owner change rewrites it, and the rewritten row also carries `enabled: true`, `desiredDeploymentId` equal to the owner, and the previous `generation` and `members` so that processes and CLIs still running the member protocol keep reading the same owner, acknowledge their relinquish normally, and refuse to admit: a build that predates this record can be replaced through the normal `qm up` path. If that older build's CLI changes `desiredDeploymentId` after the rewrite, the member fields win until the next owner change, so both builds always agree on the owner; do not use the older CLI once this one has set an owner. Ownership cannot be handed back to a color that still runs the member protocol with this CLI (its endpoint rejects the new request body), so the first rollout rolls forward only. The compatibility fields can go once no deployment runs the member protocol.## Operator endpoint`GET /v1/background-work` and `POST /v1/background-work` require both normal source request signing and `Authorization: Bearer <DEPLOYMENT_CONTROL_SECRET>`. Agent capabilities cannot invoke these routes. Portal forwards only those exact methods and path, preserving the supplied credentials.-Status includes `protocol: 1`, the responding process's `deploymentId` and `instanceId`, `enabled`, `generation`, `desiredDeploymentId`, `lastRequestId`, and durable `members`. Every member includes its deployment identity, process identity, task ARN, admission generation, state, readiness, and retirement flag.--To bootstrap, post:+Status is:```json{- "expectedGeneration": 0,- "requestId": "ed73d35e-7dfd-49b9-8550-62955df7df7b",- "desiredDeploymentId": "core:release-a",- "bootstrapTaskArns": ["exact-task-arn-a", "exact-task-arn-b"]+ "protocol": 2,+ "deploymentId": "core:release-b",+ "instanceId": "2b1c…",+ "ownerDeploymentId": "core:release-a",+ "setAt": "2026-10-04T18:02:11.148Z",+ "setBy": "core:release-a",+ "active": false}```-The cohort must match all enrolled, nonretired processes exactly. Later transitions omit `bootstrapTaskArns` and increment the generation using the same compare-and-swap contract. Set `desiredDeploymentId` to `null` to pause with no successor.+`deploymentId` and `instanceId` identify the responding process; `active` says whether that process has finished starting background work and is admitting it. Because requests reach one process behind the load balancer, `active` describes one process; the CLI polls until it has seen every expected process of the deployment report the same state.-Use a fresh UUID for each logical mutation and reuse that UUID only when retrying the same payload. After an uncertain response, inspect `lastRequestId`, generation, desired identity, and the affected member records. A matching desired identity alone does not prove that a particular request committed.+To change the owner, post:-## Admission, readiness, and draining+```json+{ "ownerDeploymentId": "core:release-b", "expectedOwnerDeploymentId": "core:release-a" }+```-A process records `admitted` before starting background resources. The desired deployment cannot admit while any earlier generation remains admitted. All replicas of the desired deployment may admit; existing per-job claims and leases continue to coordinate work within the deployment.+`expectedOwnerDeploymentId` is an optional compare-and-swap guard: the change is refused with `409 background_ownership_conflict` when the current owner is neither the expected one nor already the requested one. Setting the owner to the value it already has is a no-op success, so a lost response is confirmed by re-reading the record or by repeating the same request. Set `ownerDeploymentId` to `null` to pause background work everywhere.-`ready: true` means activation completed. A deployment operator should wait for every expected current task to have an admitted, ready process at the desired generation before considering activation successful.+## What processes do-`relinquished` acknowledges that the process stopped new claims and closed its Slack ingress. Already admitted turns may still run. Cron polling can resume after rollback while earlier callbacks keep their queue connection and heartbeats. Maintenance callbacks that cannot relinquish safely are joined before acknowledgment.+Each process polls the record every second. When the record names its deployment, it starts background work: run claims, the cron scheduler, maintenance, and Slack ingress. While it owns, it keeps a local validity window of ten seconds that each successful read renews; a failed or hanging read lets the window expire, which fences new local claims until the next successful read confirms ownership again.-`drained` means the process's admitted background work has finished. `qm up` replaces a cohort once every member has relinquished. Members still finishing admitted work get the shutdown drain window, after which the platform terminates them; a turn cut off that way releases its lease on exit and resumes on the replacement tasks.+When the record stops naming its deployment, the process stops claiming new work and closes its Slack ingress within one poll interval, or within the ten-second validity window if its reads are failing. The new owner may start inside that window; run claims stay exclusive through the run store's leases, not through ownership, so the overlap costs at most a few redundant poll cycles. Turns and callbacks already admitted continue under their existing leases and heartbeats until they finish. There is no durable acknowledgment: the deploy workflow does not wait for old processes to report anything. It relies on ECS task replacement, which sends `SIGTERM` and allows a drain window, and on the new owner resuming any run whose lease lapses.-Database errors or an expired local validity watchdog fence new local work. They do not establish durable relinquishment or authorize another deployment to bypass an outstanding member.+Synchronous turns and manually started cron callbacks are admitted work too. A deployment that is not the owner refuses new synchronous execution while still accepting durable asynchronous submissions for the owner to pick up.-## Terminated processes+## Deploying-When a process cannot acknowledge, the trusted operator must independently prove that its exact infrastructure task has stopped. It can then post:+`qm up` and `qm rollback` refuse to replace a controlled stack's core tasks while the owner record names that stack. The deploy workflow therefore:-```json-{- "expectedGeneration": 3,- "requestId": "e59f2af3-1d57-49a4-9453-91a391540c17",- "terminatedMembers": [{ "instanceId": "exact-process-id", "taskArn": "exact-stopped-task-arn", "generation": 2 }]-}-```+1. Deploys the new release to the stack that is not the owner.+2. Verifies it, then sets the owner record to the new stack (`awsSetBackgroundWork(..., true)`), which waits until every one of the new stack's core tasks has reported `active`.+3. Switches public routing. The previous owner stops claiming within a second of the change and drains under `SIGTERM` when its tasks are later replaced.-Retirement preserves the ownership generation, updates `lastRequestId`, and marks only the matching members retired and drained. Retired task identities cannot enroll again. An unreachable HTTP endpoint, old heartbeat, or elapsed timeout is not termination evidence.+Rolling back ownership is the same operation in the other direction. Nothing needs to be retired, proven stopped, or cleaned up: a hard-killed process leaves nothing behind in the record. If the record names a deployment that no longer has tasks (for example after `qm down`), no background work runs until an operator sets the owner again; `awsSetBackgroundWork(..., true)` on the live stack does that, since its compare-and-swap expects whatever owner it just read.## Live deployment session check-A ready, active deployment accepts `POST /v1/deployment/live-session` with the-same source signature and distinct deployment bearer credential. The exact body-contains `requestId` (a fresh UUID), `expectedDeploymentId`, `expectedGeneration`,-and `expectedTaskArns` (the exact healthy task cohort). The endpoint rejects-unknown fields, inactive ownership, incomplete readiness, stale generations,-and mismatched membership before starting work.--The check runs the same fixed session command used by the standalone deployment-smoke: a real model reply, persisted turns, generated title, session error log,-session archival, then the configured PostgreSQL catalog checks. The request-cannot select a principal, URL, model, prompt, or command. Model HTTP requests-have a five-minute limit; other HTTP requests and database operations have-30-second limits. These are failure bounds, not a release-duration claim.--The response is newline-delimited JSON with an initial newline and five-second-whitespace heartbeats, followed by one final object containing `ok`, `requestId`,-`deploymentId`, `instanceId`, `taskArn`, and `generation`. A failed result adds a-fixed error code without raw model, database, or credential details. Ownership-and cohort readiness are checked again before success. Heartbeats alone never-prove success; the caller must receive and verify the complete final object.--Request IDs are durably consumed across replicas and cannot be replayed. Each-process permits one check at a time, including its cleanup. Disconnecting the-caller does not cancel the check or release that guard before cleanup finishes.-Singleflight is per process, and durable request IDs prevent replay. Requests with distinct IDs can run concurrently on different replicas. The release workflow serializes its own requests.-An uncertain result must fail the release; do not retry automatically or fall-back to a second canary execution.--## Enrolling an active legacy deployment--Before bootstrap, a controlled process with its legacy boot flag enabled publishes-an enrollment-specific legacy build heartbeat only after its generation-zero-membership is admitted and ready. Older legacy workers then stop claiming new-runs even when both processes use the same image. Controlled processes ignore-legacy supersession results and continue using durable ownership admission.-Inactive, unready, or fenced processes do not publish this compatibility heartbeat;-publishing also stops once durable ownership is enabled.--This bridge preserves the legacy worker drain behavior. It does not add missing-lifecycle controls to older binaries: their Slack ingress, cron callbacks, and-inline HTTP execution can remain active.-Never treat the heartbeat as a deployment relinquishment acknowledgment or as-proof that a legacy task is safe to terminate. Bootstrap still requires explicit-infrastructure proof that every legacy task has retired, including pending tasks.-If enrollment is abandoned, the legacy heartbeat expiry permits older workers-to resume their existing claim loop.--Synchronous turns and manually started cron callbacks are admitted work too. A-paused deployment refuses new synchronous execution while still accepting durable-asynchronous submissions for the active workers. Accepted turns, scheduled-callbacks, and their nested work keep running with their existing leases; their-completion is part of the deployment's drain acknowledgment. Resuming ownership-restores synchronous admission without restarting or canceling those calls.+A deployment that owns background work accepts `POST /v1/deployment/live-session` with the same source signature and distinct deployment bearer credential. The body contains exactly `requestId` (a fresh UUID) and `expectedDeploymentId`. The endpoint rejects unknown fields, a mismatched deployment, a non-owning deployment, and an inactive responder before starting work.++The check runs the same fixed session command used by the standalone deployment smoke: a real model reply, persisted turns, generated title, session error log, session archival, then the configured PostgreSQL catalog checks. The request cannot select a principal, URL, model, prompt, or command. Model HTTP requests have a five-minute limit; other HTTP requests and database operations have 30-second limits.++The response is newline-delimited JSON with an initial newline and five-second whitespace heartbeats, followed by one final object containing `ok`, `requestId`, `deploymentId`, and `instanceId`. A failed result adds a fixed error code without raw model, database, or credential details. Ownership is checked again before success. Heartbeats alone never prove success; the caller must receive and verify the complete final object.++Request IDs are durably consumed across replicas and cannot be replayed. Each process permits one check at a time, including its cleanup. Disconnecting the caller does not cancel the check or release that guard before cleanup finishes. An uncertain result must fail the release; do not retry automatically or fall back to a second canary execution.diff --git a/src/api/deps.ts b/src/api/deps.tsindex c4b7d3d87..4261b0e11 100644--- a/src/api/deps.ts+++ b/src/api/deps.ts@@ -2,7 +2,7 @@ import type { ExternalSlackPolicies } from "../resolution/external-slack.ts";import type { ProviderKeys } from "../harness/pi-harness.ts";import type { ModelGatewayTransportConfig } from "../model/provider-endpoints.ts";import type { DurableMap } from "../persistence/durable-map.ts";-import type { BackgroundOwnershipStore } from "../runs/background-ownership.ts";+import type { BackgroundOwnershipControl } from "../runs/background-ownership.ts";import type { LoopIngressService } from "../loops/ingress.ts";import type { createSuggestedActivityService } from "../suggestions/activities.ts";import type { ManagedSlack } from "../surfaces/slack-managed.ts";@@ -182,7 +182,7 @@ export interface ServerDeps {sessionShareBytes?: DurableByteStore;environments?: EnvironmentStore;deploymentLayer?: DeploymentLayerStore;- backgroundOwnership?: { store: BackgroundOwnershipStore; instanceId: string; deploymentId: string };+ backgroundOwnership?: BackgroundOwnershipControl;deploymentControlSecret?: string;credentialServices?: () => readonly string[];brokeredServices?: () => readonly string[];diff --git a/src/api/routes/background-work.ts b/src/api/routes/background-work.tsindex 7d4a8dbaf..3f1bb97f3 100644--- a/src/api/routes/background-work.ts+++ b/src/api/routes/background-work.ts@@ -9,25 +9,16 @@ function identity(value: unknown): value is string {return typeof value === "string" && value.length > 0 && value.length <= 2048;}-function generation(value: unknown): value is number {- return typeof value === "number" && Number.isSafeInteger(value) && value >= 0;-}--function keys(value: Record<string, unknown>, allowed: string[]): boolean {- return Object.keys(value).every((key) => allowed.includes(key));-}-function status(ctx: ApiCtx, state: BackgroundOwnership): void {const control = ctx.deps.backgroundOwnership!;sendJson(ctx.res, 200, {- protocol: 1,+ protocol: 2,deploymentId: control.deploymentId,instanceId: control.instanceId,- enabled: state.enabled,- generation: state.generation,- desiredDeploymentId: state.desiredDeploymentId,- lastRequestId: state.lastRequestId,- members: state.members,+ ownerDeploymentId: state.ownerDeploymentId,+ setAt: state.setAt,+ setBy: state.setBy,+ active: control.active(),});}@@ -55,74 +46,25 @@ async function backgroundWork(ctx: ApiCtx): Promise<void> {const body = ctx.body;if (!isObj(body) ||- !generation(body.expectedGeneration) ||- typeof body.requestId !== "string" ||- !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(body.requestId)+ !Object.keys(body).every((key) => ["ownerDeploymentId", "expectedOwnerDeploymentId"].includes(key)) ||+ !(body.ownerDeploymentId === null || identity(body.ownerDeploymentId)) ||+ !(+ body.expectedOwnerDeploymentId === undefined ||+ body.expectedOwnerDeploymentId === null ||+ identity(body.expectedOwnerDeploymentId)+ )) {return sendJson(ctx.res, 400, { error: "invalid_background_request" });}try {- if (body.terminatedMembers !== undefined) {- if (- !keys(body, ["expectedGeneration", "requestId", "terminatedMembers"]) ||- !Array.isArray(body.terminatedMembers) ||- body.terminatedMembers.length < 1 ||- body.terminatedMembers.length > 1000 ||- !body.terminatedMembers.every(- (member) =>- isObj(member) &&- keys(member, ["instanceId", "taskArn", "generation"]) &&- identity(member.instanceId) &&- identity(member.taskArn) &&- generation(member.generation),- )- ) {- return sendJson(ctx.res, 400, { error: "invalid_background_retirement" });- }- const terminatedMembers = body.terminatedMembers.map((member) => ({- instanceId: member.instanceId as string,- taskArn: member.taskArn as string,- generation: member.generation as number,- }));- return status(- ctx,- await control.store.retire({- expectedGeneration: body.expectedGeneration,- requestId: body.requestId,- terminatedMembers,- }),- );- }- if (- !keys(body, [- "expectedGeneration",- "requestId",- "desiredDeploymentId",- "bootstrapTaskArns",- "expectedLastRequestId",- ]) ||- !(body.desiredDeploymentId === null || identity(body.desiredDeploymentId)) ||- !(- body.expectedLastRequestId === undefined ||- body.expectedLastRequestId === null ||- identity(body.expectedLastRequestId)- ) ||- (body.bootstrapTaskArns !== undefined &&- (!Array.isArray(body.bootstrapTaskArns) ||- body.bootstrapTaskArns.length < 1 ||- body.bootstrapTaskArns.length > 1000 ||- !body.bootstrapTaskArns.every(identity)))- ) {- return sendJson(ctx.res, 400, { error: "invalid_background_transition" });- }return status(ctx,- await control.store.transition({- expectedGeneration: body.expectedGeneration,- requestId: body.requestId,- desiredDeploymentId: body.desiredDeploymentId,- ...(body.expectedLastRequestId !== undefined ? { expectedLastRequestId: body.expectedLastRequestId } : {}),- ...(body.bootstrapTaskArns ? { bootstrapTaskArns: body.bootstrapTaskArns as string[] } : {}),+ await control.store.set({+ ownerDeploymentId: body.ownerDeploymentId,+ setBy: control.deploymentId,+ ...(body.expectedOwnerDeploymentId !== undefined+ ? { expectedOwnerDeploymentId: body.expectedOwnerDeploymentId }+ : {}),}),);} catch (error) {diff --git a/src/api/routes/deployment-live-smoke.ts b/src/api/routes/deployment-live-smoke.tsindex 0bc9b33f0..703f0d061 100644--- a/src/api/routes/deployment-live-smoke.ts+++ b/src/api/routes/deployment-live-smoke.ts@@ -16,60 +16,24 @@ async function deploymentLiveSmoke(ctx: ApiCtx): Promise<void> {const body = ctx.body;if (!isObj(body) ||- !Object.keys(body).every((key) =>- ["requestId", "expectedDeploymentId", "expectedGeneration", "expectedTaskArns"].includes(key),- ) ||+ !Object.keys(body).every((key) => ["requestId", "expectedDeploymentId"].includes(key)) ||typeof body.requestId !== "string" ||!/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(body.requestId) ||- typeof body.expectedDeploymentId !== "string" ||- body.expectedDeploymentId !== control.deploymentId ||- !Number.isSafeInteger(body.expectedGeneration) ||- (body.expectedGeneration as number) < 1 ||- !Array.isArray(body.expectedTaskArns) ||- !body.expectedTaskArns.length ||- body.expectedTaskArns.length > 1000 ||- !body.expectedTaskArns.every((arn) => typeof arn === "string" && arn.length > 0 && arn.length <= 2048) ||- new Set(body.expectedTaskArns).size !== body.expectedTaskArns.length+ body.expectedDeploymentId !== control.deploymentId)return sendJson(ctx.res, 400, { error: "invalid_deployment_smoke_request" });if (running.has(control)) return sendJson(ctx.res, 409, { error: "deployment_smoke_running" });- const expectedTasks = new Set(body.expectedTaskArns as string[]);- const valid = (state: BackgroundOwnership): boolean => {- const members = state.members.filter((member) => !member.retired && member.state === "admitted");- return (- state.enabled &&- state.generation === body.expectedGeneration &&- state.desiredDeploymentId === control.deploymentId &&- members.some((member) => member.instanceId === control.instanceId) &&- members.length === expectedTasks.size &&- new Set(members.map((member) => member.taskArn)).size === expectedTasks.size &&- members.every(- (member) =>- member.deploymentId === control.deploymentId &&- member.generation === state.generation &&- member.state === "admitted" &&- member.ready &&- member.taskArn !== null &&- expectedTasks.has(member.taskArn),- )- );- };+ const valid = (state: BackgroundOwnership): boolean =>+ state.ownerDeploymentId === control.deploymentId && control.active();running.add(control);let heartbeat: ReturnType<typeof setInterval> | undefined;try {- const state = await control.store.get();- if (!valid(state)) return sendJson(ctx.res, 409, { error: "deployment_smoke_ownership_conflict" });+ if (!valid(await control.store.get()))+ return sendJson(ctx.res, 409, { error: "deployment_smoke_ownership_conflict" });if (!(await replay.claim(`deployment-live-smoke:${body.requestId.toLowerCase()}`, REQUEST_EXPIRY))) {return sendJson(ctx.res, 409, { error: "deployment_smoke_replay" });}- const member = state.members.find((entry) => entry.instanceId === control.instanceId)!;- const identity = {- requestId: body.requestId,- deploymentId: control.deploymentId,- instanceId: control.instanceId,- taskArn: member.taskArn,- generation: state.generation,- };+ const identity = { requestId: body.requestId, deploymentId: control.deploymentId, instanceId: control.instanceId };ctx.res.writeHead(200, {"content-type": "application/x-ndjson","cache-control": "no-store, no-transform",diff --git a/src/index.ts b/src/index.tsindex 91a7b4c03..670d718c7 100644--- a/src/index.ts+++ b/src/index.ts@@ -1,6 +1,5 @@import "./instrument.ts";import { createBackgroundController } from "./runs/background-controller.ts";-import { backgroundTaskArn } from "./runs/background-task-identity.ts";import { createManagedSlack } from "./surfaces/slack-managed.ts";import { randomBytes } from "node:crypto";import { lookup } from "node:dns/promises";@@ -130,10 +129,6 @@ if (config.backgroundWorkEnabled && !config.backgroundDeploymentId)let backgroundController: ReturnType<typeof createBackgroundController> | undefined;if (built.backgroundOwnership) {- const identity = {- ...built.backgroundOwnership,- taskArn: await backgroundTaskArn(process.env.ECS_CONTAINER_METADATA_URI_V4),- };let periodicStop: Promise<void> = Promise.resolve();let activationEpoch = 0;const stopPeriodic = () => {@@ -149,9 +144,8 @@ if (built.backgroundOwnership) {void runtime.stop().catch((error) => console.error("[qm] Slack background stop failed:", errMessage(error)));};backgroundController = createBackgroundController({- store: identity.store,- identity: { deploymentId: identity.deploymentId, instanceId: identity.instanceId, taskArn: identity.taskArn },- legacyEnabled: config.backgroundWorkEnabled,+ store: built.backgroundOwnership.store,+ deploymentId: built.backgroundOwnership.deploymentId,async start(signal) {const epoch = ++activationEpoch;if (signal.aborted) return;@@ -181,7 +175,7 @@ if (built.backgroundOwnership) {},onError: reportFailureAs("background ownership", undefined),});- built.runtime.setBackgroundAdmission(backgroundController.canClaim);+ built.runtime.setBackgroundAdmission(backgroundController.canClaim, backgroundController.active);backgroundController.start();}diff --git a/src/runs/background-controller.ts b/src/runs/background-controller.tsindex 4df7e3dc3..7ee7955d2 100644--- a/src/runs/background-controller.ts+++ b/src/runs/background-controller.ts@@ -1,10 +1,8 @@-import type { BackgroundMember, BackgroundOwnershipStore } from "./background-ownership.ts";-import { BackgroundOwnershipConflict } from "./background-ownership.ts";+import type { BackgroundOwnershipStore } from "./background-ownership.ts";export interface BackgroundControllerDeps {- store: BackgroundOwnershipStore;- identity: Pick<BackgroundMember, "instanceId" | "deploymentId" | "taskArn">;- legacyEnabled: boolean;+ store: Pick<BackgroundOwnershipStore, "get">;+ deploymentId: string;start(signal: AbortSignal): Promise<void>;fence(): void;relinquish(): Promise<void>;@@ -17,10 +15,8 @@ export interface BackgroundControllerDeps {export function createBackgroundController(deps: BackgroundControllerDeps) {let running = false;- let registered = false;- let admission: number | null = null;- let admissionEpoch = 0;let activation: AbortController | null = null;+ let started = false;let validUntil = 0;let watchdog: ReturnType<typeof setTimeout> | null = null;let poller: ReturnType<typeof setInterval> | null = null;@@ -31,6 +27,7 @@ export function createBackgroundController(deps: BackgroundControllerDeps) {const validityMs = deps.validityMs ?? 10_000;const fence = (): void => {validUntil = 0;+ started = false;activation?.abort();deps.fence();if (watchdog) clearTimeout(watchdog);@@ -38,17 +35,10 @@ export function createBackgroundController(deps: BackgroundControllerDeps) {};const release = async (): Promise<void> => {fence();- if (admission === null) return;+ if (!activation) return;await deps.relinquish();- const generation = admission;- const epoch = admissionEpoch;- await deps.store.acknowledge(deps.identity.instanceId, generation, "relinquished");- admission = null;activation = null;- draining = deps.drained().then(async () => {- if (admission === null && epoch === admissionEpoch)- await deps.store.acknowledge(deps.identity.instanceId, generation, "drained");- });+ draining = deps.drained();void draining.catch(deps.onError);};const renew = (): void => {@@ -57,30 +47,19 @@ export function createBackgroundController(deps: BackgroundControllerDeps) {watchdog = setTimeout(fence, validityMs);watchdog.unref?.();};+ const owned = async (): Promise<boolean> => (await deps.store.get()).ownerDeploymentId === deps.deploymentId;const refreshStartup = (): Promise<void> => {if (!starting || !running || !activation || activation.signal.aborted) return Promise.resolve();if (refreshing) return refreshing;- const currentActivation = activation;- const generation = admission;+ const current = activation;refreshing = (async () => {try {- const state = await deps.store.get();- if (!starting || !running || activation !== currentActivation || currentActivation?.signal.aborted) return;- const member = state.members.find((entry) => entry.instanceId === deps.identity.instanceId);- if (- state.generation !== generation ||- !member ||- member.retired ||- member.state !== "admitted" ||- member.generation !== generation ||- !(state.enabled ? state.desiredDeploymentId === deps.identity.deploymentId : deps.legacyEnabled)- ) {- fence();- return;- }- renew();+ const stillOwned = await owned();+ if (!starting || !running || activation !== current || current.signal.aborted) return;+ if (stillOwned) renew();+ else fence();} catch (error) {- if (starting && activation === currentActivation) {+ if (starting && activation === current) {fence();deps.onError(error);}@@ -97,47 +76,31 @@ export function createBackgroundController(deps: BackgroundControllerDeps) {}pending = (async () => {try {- if (!registered) {- await deps.store.register(deps.identity);- registered = true;- }- const state = await deps.store.get();- const member = state.members.find((entry) => entry.instanceId === deps.identity.instanceId);- const desired =- running &&- member !== undefined &&- !member.retired &&- (state.enabled ? state.desiredDeploymentId === deps.identity.deploymentId : deps.legacyEnabled);- if (admission !== null && (!desired || state.generation !== admission || activation?.signal.aborted))- await release();+ const owner = await owned();+ const desired = running && owner;+ if (activation && (!desired || activation.signal.aborted)) await release();if (!desired) return;- if (admission === null) {- await deps.store.admit(deps.identity.instanceId, state.generation, deps.legacyEnabled);- admission = state.generation;- admissionEpoch++;- activation = new AbortController();- renew();- if (!running) {- await release();- return;- }- starting = true;- const startupDeadline = setTimeout(() => {- fence();- deps.onError(new Error("Background startup timed out; waiting for activation cleanup before retrying"));- }, deps.startupTimeoutMs ?? 120_000);- startupDeadline.unref?.();- try {- await deps.start(activation.signal);- } finally {- clearTimeout(startupDeadline);- starting = false;- }- if (activation.signal.aborted || !running) await release();- else await deps.store.markReady(deps.identity.instanceId, state.generation);- } else {+ if (activation) {renew();+ return;+ }+ const current = new AbortController();+ activation = current;+ renew();+ starting = true;+ const startupDeadline = setTimeout(() => {+ fence();+ deps.onError(new Error("Background startup timed out; waiting for activation cleanup before retrying"));+ }, deps.startupTimeoutMs ?? 120_000);+ startupDeadline.unref?.();+ try {+ await deps.start(current.signal);+ } finally {+ clearTimeout(startupDeadline);+ starting = false;}+ if (current.signal.aborted || !running) await release();+ else started = true;} catch (error) {fence();try {@@ -145,15 +108,17 @@ export function createBackgroundController(deps: BackgroundControllerDeps) {} catch (releaseError) {deps.onError(releaseError);}- if (!(error instanceof BackgroundOwnershipConflict)) deps.onError(error);+ deps.onError(error);}})().finally(() => {pending = null;});return pending;};+ const canClaim = () => running && activation !== null && !activation.signal.aborted && Date.now() < validUntil;return {- canClaim: () => running && admission !== null && !activation?.signal.aborted && Date.now() < validUntil,+ canClaim,+ active: () => started && canClaim(),reconcile,start() {if (running) return;diff --git a/src/runs/background-ownership.ts b/src/runs/background-ownership.tsindex 80f487aae..2603afbf2 100644--- a/src/runs/background-ownership.ts+++ b/src/runs/background-ownership.ts@@ -1,211 +1,88 @@import type { DurableMap } from "../persistence/durable-map.ts";-export interface BackgroundMember {- instanceId: string;- deploymentId: string;- taskArn: string | null;- generation: number;- state: "admitted" | "relinquished" | "drained";- retired: boolean;- ready: boolean;+export interface BackgroundOwnership {+ ownerDeploymentId: string | null;+ setAt: string | null;+ setBy: string | null;}-export interface BackgroundOwnership {+export interface BackgroundOwnershipChange {+ ownerDeploymentId: string | null;+ expectedOwnerDeploymentId?: string | null;+ setBy: string;+}++export class BackgroundOwnershipConflict extends Error {}++interface MemberProtocolRecord {enabled: boolean;generation: number;desiredDeploymentId: string | null;- lastRequestId: string | null;- lastRequest: string | null;- members: BackgroundMember[];+ members: unknown[];}-export interface BackgroundTransition {- expectedGeneration: number;- requestId: string;- desiredDeploymentId: string | null;- expectedLastRequestId?: string | null;- bootstrapTaskArns?: string[];+type StoredOwnership = BackgroundOwnership | MemberProtocolRecord | (BackgroundOwnership & MemberProtocolRecord);++function memberProtocolOwner(value: MemberProtocolRecord): BackgroundOwnership {+ return { ownerDeploymentId: value.enabled ? value.desiredDeploymentId : null, setAt: null, setBy: null };}-export class BackgroundOwnershipConflict extends Error {}+function fromStored(value: StoredOwnership): BackgroundOwnership {+ if (!("ownerDeploymentId" in value)) return memberProtocolOwner(value);+ if ("desiredDeploymentId" in value && value.desiredDeploymentId !== value.ownerDeploymentId)+ return memberProtocolOwner(value);+ return { ownerDeploymentId: value.ownerDeploymentId, setAt: value.setAt, setBy: value.setBy };+}++function readableByMemberProtocol(state: BackgroundOwnership, previous?: StoredOwnership): BackgroundOwnership {+ const members = previous && "members" in previous ? previous : { generation: 0, members: [] };+ return {+ ...state,+ enabled: true,+ generation: members.generation,+ desiredDeploymentId: state.ownerDeploymentId,+ lastRequestId: null,+ lastRequest: null,+ members: members.members,+ } as BackgroundOwnership;+}export function createBackgroundOwnershipStore(map: DurableMap<BackgroundOwnership>) {if (!map.update) throw new Error("Background ownership requires atomic updates");const key = "ownership";- const initial: BackgroundOwnership = {- enabled: false,- generation: 0,- desiredDeploymentId: null,- lastRequestId: null,- lastRequest: null,- members: [],- };- const ready = () => map.putIfAbsent(key, structuredClone(initial));- const update = async (change: (state: BackgroundOwnership) => void): Promise<BackgroundOwnership> => {- await ready();- const result = await map.update!(key, (current) => {- const next = structuredClone(current);- change(next);- return next;- });- if (!result) throw new Error("Background ownership disappeared");- return result;- };- const memberOf = (state: BackgroundOwnership, instanceId: string): BackgroundMember => {- const member = state.members.find((entry) => entry.instanceId === instanceId);- if (!member || member.retired) throw new BackgroundOwnershipConflict("Instance is missing or retired");- return member;- };- const expectGeneration = (state: BackgroundOwnership, expected: number): void => {- if (!Number.isSafeInteger(expected) || expected < 0 || state.generation !== expected)- throw new BackgroundOwnershipConflict("Background ownership generation changed");- };+ const initial: BackgroundOwnership = { ownerDeploymentId: null, setAt: null, setBy: null };+ const ready = () => map.putIfAbsent(key, readableByMemberProtocol(initial));return {async get(): Promise<BackgroundOwnership> {- return structuredClone((await map.get(key)) ?? (await ready()));- },- register(identity: Pick<BackgroundMember, "instanceId" | "deploymentId" | "taskArn">) {- return update((state) => {- if (identity.taskArn && state.members.some((member) => member.taskArn === identity.taskArn && member.retired))- throw new BackgroundOwnershipConflict("Task has already been retired");- const previous = state.members.find((member) => member.instanceId === identity.instanceId);- if (previous) {- if (- previous.deploymentId !== identity.deploymentId ||- previous.taskArn !== identity.taskArn ||- previous.retired- )- throw new BackgroundOwnershipConflict("Instance identity cannot change");- return;- }- state.members.push({- ...identity,- generation: state.generation,- state: "drained",- retired: false,- ready: false,- });- });- },- transition(request: BackgroundTransition) {- const fingerprint = JSON.stringify([- "transition",- request.expectedGeneration,- request.desiredDeploymentId,- request.bootstrapTaskArns?.slice().sort() ?? null,- ...(request.expectedLastRequestId !== undefined ? [request.expectedLastRequestId] : []),- ]);- return update((state) => {- if (state.lastRequestId === request.requestId) {- if (- state.lastRequest !== fingerprint ||- state.generation !== request.expectedGeneration + 1 ||- state.desiredDeploymentId !== request.desiredDeploymentId- )- throw new BackgroundOwnershipConflict("Request identity reused for a different transition");- return;- }- expectGeneration(state, request.expectedGeneration);- if (request.expectedLastRequestId !== undefined && state.lastRequestId !== request.expectedLastRequestId)- throw new BackgroundOwnershipConflict("Background ownership request changed");- if (!state.enabled) {- const expected = request.bootstrapTaskArns;- if (!expected?.length || new Set(expected).size !== expected.length)- throw new BackgroundOwnershipConflict("Bootstrap requires an exact nonempty task cohort");- const enrolled = state.members.filter((member) => !member.retired);- if (- enrolled.some((member) => !member.taskArn || !expected.includes(member.taskArn)) ||- expected.some((taskArn) => !enrolled.some((member) => member.taskArn === taskArn))- )- throw new BackgroundOwnershipConflict("Bootstrap task cohort does not match enrolled instances");- } else if (request.bootstrapTaskArns) {- throw new BackgroundOwnershipConflict("Ownership is already enabled");- }- if (- request.desiredDeploymentId !== null &&- !state.members.some((member) => !member.retired && member.deploymentId === request.desiredDeploymentId)- )- throw new BackgroundOwnershipConflict("Desired deployment has no enrolled instances");- state.enabled = true;- state.generation++;- state.desiredDeploymentId = request.desiredDeploymentId;- state.lastRequestId = request.requestId;- state.lastRequest = fingerprint;- });+ return fromStored((await map.get(key)) ?? (await ready()));},- admit(instanceId: string, expectedGeneration: number, legacyEnabled: boolean) {- return update((state) => {- expectGeneration(state, expectedGeneration);- const member = memberOf(state, instanceId);- if (state.enabled) {- if (state.desiredDeploymentId !== member.deploymentId)- throw new BackgroundOwnershipConflict("Deployment is not the desired owner");- if (state.members.some((other) => other.state === "admitted" && other.generation !== state.generation))- throw new BackgroundOwnershipConflict("Previous owners have not relinquished work");- } else if (!legacyEnabled) {- throw new BackgroundOwnershipConflict("Legacy background work is disabled");- }- member.generation = state.generation;- member.state = "admitted";- member.ready = false;- });- },- markReady(instanceId: string, generation: number) {- return update((state) => {- expectGeneration(state, generation);- const member = memberOf(state, instanceId);+ async set(change: BackgroundOwnershipChange): Promise<BackgroundOwnership> {+ await ready();+ const result = await map.update!(key, (stored) => {+ const current = fromStored(stored);if (- member.generation !== generation ||- member.state !== "admitted" ||- (state.enabled && state.desiredDeploymentId !== member.deploymentId)+ change.expectedOwnerDeploymentId !== undefined &&+ current.ownerDeploymentId !== change.expectedOwnerDeploymentId &&+ current.ownerDeploymentId !== change.ownerDeploymentId)- throw new BackgroundOwnershipConflict("Instance admission changed");- member.ready = true;- });- },- acknowledge(instanceId: string, generation: number, next: "relinquished" | "drained") {- return update((state) => {- const member = memberOf(state, instanceId);- if (member.generation !== generation) throw new BackgroundOwnershipConflict("Instance admission changed");- if (member.state === "drained" || member.state === next) return;- if (next === "drained" && member.state !== "relinquished")- throw new BackgroundOwnershipConflict("Instance must relinquish before reporting drained");- member.state = next;- member.ready = false;- });- },- retire(request: {- expectedGeneration: number;- requestId: string;- terminatedMembers: Array<Pick<BackgroundMember, "instanceId" | "taskArn" | "generation">>;- }) {- const fingerprint = JSON.stringify([- "retire",- request.expectedGeneration,- request.terminatedMembers.map((member) => [member.instanceId, member.taskArn, member.generation]).sort(),- ]);- return update((state) => {- expectGeneration(state, request.expectedGeneration);- if (state.lastRequestId === request.requestId) {- if (state.lastRequest !== fingerprint)- throw new BackgroundOwnershipConflict("Request identity reused for a different retirement");- return;- }- if (!request.terminatedMembers.length)- throw new BackgroundOwnershipConflict("Retirement requires exact terminated members");- for (const proof of request.terminatedMembers) {- const member = state.members.find((entry) => entry.instanceId === proof.instanceId);- if (!member || !proof.taskArn || member.taskArn !== proof.taskArn || member.generation !== proof.generation)- throw new BackgroundOwnershipConflict("Termination evidence does not match the enrolled instance");- member.state = "drained";- member.retired = true;- member.ready = false;- }- state.lastRequestId = request.requestId;- state.lastRequest = fingerprint;+ throw new BackgroundOwnershipConflict("Background owner changed");+ if (current.ownerDeploymentId === change.ownerDeploymentId) return stored;+ return readableByMemberProtocol(+ { ownerDeploymentId: change.ownerDeploymentId, setAt: new Date().toISOString(), setBy: change.setBy },+ stored,+ );});+ if (!result) throw new Error("Background ownership disappeared");+ return fromStored(result);},};}export type BackgroundOwnershipStore = ReturnType<typeof createBackgroundOwnershipStore>;++export interface BackgroundOwnershipControl {+ store: BackgroundOwnershipStore;+ instanceId: string;+ deploymentId: string;+ active: () => boolean;+}diff --git a/src/runs/background-task-identity.ts b/src/runs/background-task-identity.tsdeleted file mode 100644index 24b63ff96..000000000--- a/src/runs/background-task-identity.ts+++ /dev/null@@ -1,9 +0,0 @@-export async function backgroundTaskArn(metadataUri: string | undefined): Promise<string | null> {- if (!metadataUri) return null;- const response = await fetch(`${metadataUri.replace(/\/$/, "")}/task`, { signal: AbortSignal.timeout(5_000) });- if (!response.ok) throw new Error(`Task metadata returned ${response.status}`);- const data = (await response.json()) as { TaskARN?: unknown };- if (typeof data.TaskARN !== "string" || !data.TaskARN.startsWith("arn:") || !data.TaskARN.includes(":task/"))- throw new Error("Task metadata did not contain a task ARN");- return data.TaskARN;-}diff --git a/src/runs/instance-registry.ts b/src/runs/instance-registry.tsindex b076f728b..cf96237a6 100644--- a/src/runs/instance-registry.ts+++ b/src/runs/instance-registry.ts@@ -57,15 +57,3 @@ export function createPostgresInstanceRegistry(},};}--export function createLegacyEnrollmentBridge(- registry: InstanceRegistry,- eligible: () => Promise<boolean>,-): InstanceRegistry {- return {- async beat() {- if (await eligible()) await registry.beat();- return false;- },- };-}diff --git a/src/wiring.ts b/src/wiring.tsindex 8e9205521..424dc2b98 100644--- a/src/wiring.ts+++ b/src/wiring.ts@@ -11,7 +11,7 @@ import { createAdmittedWork } from "./util/admitted-work.ts";import { runSessionSmoke } from "./deployment/postdeploy-smoke.ts";import {createBackgroundOwnershipStore,- type BackgroundOwnershipStore,+ type BackgroundOwnershipControl,type BackgroundOwnership,} from "./runs/background-ownership.ts";import { loadConnectorSdk } from "./sandbox/connector-sdk.ts";@@ -338,7 +338,6 @@ import { isTerminal, type Run, type RunStore } from "./runs/run-store.ts";import { createWorker, type Worker } from "./runs/worker.ts";import {createNoopInstanceRegistry,- createLegacyEnrollmentBridge,createPostgresInstanceRegistry,type InstanceRegistry,} from "./runs/instance-registry.ts";@@ -428,7 +427,7 @@ export interface Runtime {start(): void;startBackground(): void;stopBackgroundClaims(): Promise<void>;- setBackgroundAdmission(check: () => boolean): void;+ setBackgroundAdmission(canClaim: () => boolean, active: () => boolean): void;stopBackground(): Promise<void>;backgroundDrained(): Promise<void>;stop(): Promise<void>;@@ -469,7 +468,7 @@ export function stopWithBackstop(export interface BuiltApp {checkReadiness: (signal: AbortSignal) => Promise<void>;- backgroundOwnership?: { store: BackgroundOwnershipStore; instanceId: string; deploymentId: string };+ backgroundOwnership?: BackgroundOwnershipControl;suggestedActivityMaintenance: Sweeper;suggestedActivities?: ReturnType<typeof createSuggestedActivityService>;app: App;@@ -575,6 +574,7 @@ export function buildApp(} = {},): BuiltApp {let backgroundAdmission = () => !config.backgroundDeploymentId;+ let backgroundActive = () => false;const admittedWork = createAdmittedWork({canStart: () => !config.backgroundDeploymentId || backgroundAdmission(),});@@ -2540,37 +2540,22 @@ export function buildApp(directory,currentScopeMembers,});- const backgroundOwnership = config.backgroundDeploymentId+ const backgroundOwnership: BackgroundOwnershipControl | undefined = config.backgroundDeploymentId? {store: createBackgroundOwnershipStore(artifactMap<BackgroundOwnership>("background_ownership")),instanceId: randomUUID(),deploymentId: config.backgroundDeploymentId,+ active: () => backgroundActive(),}: undefined;- const legacyRegistry =- pgArtifactMap && (backgroundOwnership || (config.buildSha && config.backgroundWorkEnabled))+ const instanceRegistry: InstanceRegistry =+ pgArtifactMap && !backgroundOwnership && config.buildSha && config.backgroundWorkEnabled? createPostgresInstanceRegistry(pgArtifactMap.pool, {instanceId: randomUUID(),- buildSha: backgroundOwnership ? `enrollment:${backgroundOwnership.deploymentId}` : config.buildSha!,+ buildSha: config.buildSha,startedAt: Date.now(),}): createNoopInstanceRegistry();- const instanceRegistry: InstanceRegistry = backgroundOwnership- ? createLegacyEnrollmentBridge(legacyRegistry, async () => {- if (!config.backgroundWorkEnabled || !backgroundAdmission()) return false;- const state = await backgroundOwnership.store.get();- const member = state.members.find((entry) => entry.instanceId === backgroundOwnership.instanceId);- return (- !state.enabled &&- state.generation === 0 &&- member?.generation === 0 &&- !member.retired &&- member.state === "admitted" &&- member.ready &&- backgroundAdmission()- );- })- : legacyRegistry;const drain: DrainController = createDrainController({ registry: instanceRegistry });const workers: Worker[] = Array.from({ length: Math.max(1, config.workers) }, () =>createWorker({@@ -2742,8 +2727,9 @@ export function buildApp(if (config.backgroundWorkEnabled && !config.backgroundDeploymentId) startBackground();},startBackground,- setBackgroundAdmission(check) {- backgroundAdmission = check;+ setBackgroundAdmission(canClaim, active) {+ backgroundAdmission = canClaim;+ backgroundActive = active;},async stopBackgroundClaims() {void stopBackground().catch(swallowAs("wiring: background drain failed", undefined));diff --git a/test/background-controller.test.ts b/test/background-controller.test.tsindex f1be31399..623b291e4 100644--- a/test/background-controller.test.ts+++ b/test/background-controller.test.ts@@ -4,18 +4,17 @@ import { createMemoryMap } from "../src/persistence/durable-map.ts";import { createBackgroundOwnershipStore, type BackgroundOwnership } from "../src/runs/background-ownership.ts";import { createBackgroundController } from "../src/runs/background-controller.ts";-async function setup() {+function setup() {const store = createBackgroundOwnershipStore(createMemoryMap<BackgroundOwnership>());const errors: unknown[] = [];- function replica(instanceId: string, deploymentId: string, legacyEnabled = false) {+ function replica(deploymentId: string) {let starts = 0;let stops = 0;let failStart = false;const drain = Promise.withResolvers<void>();const controller = createBackgroundController({store,- identity: { instanceId, deploymentId, taskArn: `task:${instanceId}` },- legacyEnabled,+ deploymentId,start: async () => {starts++;if (failStart) throw new Error("activation failed");@@ -39,85 +38,94 @@ async function setup() {},};}- return { store, replica, errors };+ const own = (deploymentId: string | null) => store.set({ ownerDeploymentId: deploymentId, setBy: "test" });+ return { store, replica, errors, own };}-test("two deployments hand over all replicas while previous work remains draining", async () => {- const { store, replica } = await setup();- const a = replica("a1", "a", true);- const a2 = replica("a2", "a", true);- const b = replica("b1", "b");+test("only the owning deployment's replicas run background work and handover is reversible", async () => {+ const { replica, own } = setup();+ const a = replica("a");+ const a2 = replica("a");+ const b = replica("b");for (const r of [a, a2, b]) {r.controller.start();await r.controller.reconcile();}- assert.equal(a.starts(), 1);- assert.equal(b.starts(), 0);- await store.transition({- expectedGeneration: 0,- requestId: "bootstrap",- desiredDeploymentId: "b",- bootstrapTaskArns: ["task:a1", "task:a2", "task:b1"],- });- await b.controller.reconcile();+ assert.equal(a.starts(), 0);assert.equal(b.starts(), 0);- await a.controller.reconcile();- await b.controller.reconcile();+ assert.equal(a.controller.canClaim(), false);+ await own("a");+ for (const r of [a, a2, b]) await r.controller.reconcile();+ assert.equal(a.starts(), 1);+ assert.equal(a2.starts(), 1);assert.equal(b.starts(), 0);- await a2.controller.reconcile();- await b.controller.reconcile();- assert.equal(b.starts(), 1);+ assert.equal(a.controller.canClaim(), true);+ assert.equal(a.controller.active(), true);+ assert.equal(b.controller.canClaim(), false);+ await own("b");+ for (const r of [a, a2, b]) await r.controller.reconcile();+ assert.equal(a.stops(), 1);+ assert.equal(a2.stops(), 1);assert.equal(a.controller.canClaim(), false);- assert.equal((await store.get()).members.find((m) => m.instanceId === "a1")?.state, "relinquished");- await store.transition({ expectedGeneration: 1, requestId: "rollback", desiredDeploymentId: "a" });- await b.controller.reconcile();- await a.controller.reconcile();+ assert.equal(b.starts(), 1);+ assert.equal(b.controller.canClaim(), true);+ await own("a");+ for (const r of [a, a2, b]) await r.controller.reconcile();assert.equal(a.starts(), 2);+ assert.equal(b.stops(), 1);+ assert.equal(b.controller.canClaim(), false);+ await own(null);+ for (const r of [a, a2, b]) await r.controller.reconcile();+ assert.equal(a.stops(), 2);+ assert.equal(a.controller.canClaim(), false);a.drain.resolve();await a.controller.drained();- assert.equal((await store.get()).members.find((m) => m.instanceId === "a1")?.state, "admitted");for (const r of [a, a2, b]) {r.drain.resolve();await r.controller.stop();}});-test("failed activation relinquishes membership and allows a reversible successor", async () => {- const { store, replica, errors } = await setup();- const a = replica("a", "a", true);+test("failed activation relinquishes and the next poll retries", async () => {+ const { replica, errors, own } = setup();+ await own("a");+ const a = replica("a");a.fail();a.controller.start();await a.controller.reconcile();assert.equal(a.controller.canClaim(), false);- assert.equal((await store.get()).members[0]?.state, "relinquished");+ assert.equal(a.stops(), 1);assert.equal(errors.length, 1);+ await a.controller.reconcile();+ assert.equal(a.starts(), 2);a.drain.resolve();await a.controller.stop();});-test("database failure fences locally without inventing a durable relinquishment", async () => {- const { store } = await setup();+test("database failure fences locally and work resumes once the owner record is readable again", async () => {+ const { store, own } = setup();+ await own("a");let disconnected = false;let fenced = 0;+ let starts = 0;+ let stops = 0;const controller = createBackgroundController({store: {- ...store,get: async () => {if (disconnected) throw new Error("offline");return store.get();},- acknowledge: async (...args) => {- if (disconnected) throw new Error("offline");- return store.acknowledge(...args);- },},- identity: { instanceId: "a", deploymentId: "a", taskArn: "task:a" },- legacyEnabled: true,- start: async () => {},+ deploymentId: "a",+ start: async () => {+ starts++;+ },fence: () => {fenced++;},- relinquish: async () => {},+ relinquish: async () => {+ stops++;+ },drained: async () => {},onError() {},pollMs: 60_000,@@ -129,28 +137,30 @@ test("database failure fences locally without inventing a durable relinquishmentawait controller.reconcile();assert.equal(controller.canClaim(), false);assert.ok(fenced > 0);- assert.equal((await store.get()).members[0]?.state, "admitted");+ assert.equal(stops, 1);disconnected = false;+ await controller.reconcile();+ assert.equal(starts, 2);+ assert.equal(controller.canClaim(), true);await controller.stop();+ assert.equal(stops, 2);await controller.drained();- assert.equal((await store.get()).members[0]?.state, "drained");});-test("a hanging database read expires local admission and cannot resurrect it after stop", async () => {- const { store } = await setup();+test("a hanging owner read expires local admission and cannot resurrect it after stop", async () => {+ const { store, own } = setup();+ await own("a");const waiting = Promise.withResolvers<void>();let hang = false;let starts = 0;const controller = createBackgroundController({store: {- ...store,get: async () => {if (hang) await waiting.promise;return store.get();},},- identity: { instanceId: "a", deploymentId: "a", taskArn: "task:a" },- legacyEnabled: true,+ deploymentId: "a",start: async () => {starts++;},@@ -173,25 +183,27 @@ test("a hanging database read expires local admission and cannot resurrect it afassert.equal(starts, 1);assert.equal(controller.canClaim(), false);await controller.drained();- assert.equal((await store.get()).members[0]?.state, "drained");});-test("activation is not ready until startup completes and expiry fences a late startup", async () => {- const { store } = await setup();+test("a process is active only after startup completes, and expiry during a slow startup aborts it", async () => {+ const { store, own } = setup();+ await own("a");const started = Promise.withResolvers<void>();const finish = Promise.withResolvers<void>();let signal: AbortSignal | undefined;+ let stops = 0;const controller = createBackgroundController({store,- identity: { instanceId: "a", deploymentId: "a", taskArn: "task:a" },- legacyEnabled: true,+ deploymentId: "a",start: async (value) => {signal = value;started.resolve();await finish.promise;},fence() {},- relinquish: async () => {},+ relinquish: async () => {+ stops++;+ },drained: async () => {},onError() {},pollMs: 60_000,@@ -199,19 +211,22 @@ test("activation is not ready until startup completes and expiry fences a late s});controller.start();await started.promise;- assert.equal((await store.get()).members[0]?.ready, false);+ assert.equal(controller.canClaim(), true);+ assert.equal(controller.active(), false);await new Promise((resolve) => setTimeout(resolve, 40));assert.equal(signal?.aborted, true);finish.resolve();await controller.reconcile();- assert.equal((await store.get()).members[0]?.ready, false);+ assert.equal(stops, 1);+ assert.equal(controller.active(), false);await controller.stop();});-for (const outcome of ["complete", "transition", "read-failure", "read-stall", "stop", "startup-timeout"] as const) {+for (const outcome of ["complete", "handover", "read-failure", "read-stall", "stop", "startup-timeout"] as const) {test(`pending startup renews verified ownership and fences on ${outcome}`, async (t) => {t.mock.timers.enable({ apis: ["Date", "setTimeout", "setInterval"] });- const { store } = await setup();+ const { store, own } = setup();+ await own("a");const started = Promise.withResolvers<void>();const finish = Promise.withResolvers<void>();const read = Promise.withResolvers<void>();@@ -225,7 +240,6 @@ for (const outcome of ["complete", "transition", "read-failure", "read-stall", "const errors: unknown[] = [];const controller = createBackgroundController({store: {- ...store,get: async () => {reads++;maxReads = Math.max(maxReads, reads);@@ -238,8 +252,7 @@ for (const outcome of ["complete", "transition", "read-failure", "read-stall", "}},},- identity: { instanceId: "a", deploymentId: "a", taskArn: "task:a" },- legacyEnabled: true,+ deploymentId: "a",start: async (value) => {starts++;starting = true;@@ -267,18 +280,10 @@ for (const outcome of ["complete", "transition", "read-failure", "read-stall", "await flush();assert.equal(controller.canClaim(), true);assert.equal(signal?.aborted, false);- assert.equal((await store.get()).members[0]?.ready, false);}assert.equal(starts, 1);assert.equal(stops, 0);- if (outcome === "transition") {- await store.transition({- expectedGeneration: 0,- requestId: "switch",- desiredDeploymentId: null,- bootstrapTaskArns: ["task:a"],- });- }+ if (outcome === "handover") await own("b");if (outcome === "read-failure") readMode = "fail";if (outcome === "read-stall" || outcome === "stop") readMode = "stall";t.mock.timers.tick(10);@@ -291,7 +296,6 @@ for (const outcome of ["complete", "transition", "read-failure", "read-stall", "}assert.equal(starts, 1);assert.equal(stops, 0);- assert.equal((await store.get()).members[0]?.ready, false);}if (outcome === "read-stall" || outcome === "stop") {t.mock.timers.tick(110);@@ -307,7 +311,6 @@ for (const outcome of ["complete", "transition", "read-failure", "read-stall", "finish.resolve();await reconciled;await stopping;- assert.equal((await store.get()).members[0]?.ready, outcome === "complete");assert.equal(starts, 1);assert.equal(maxReads, 1);assert.equal(errors.length, outcome === "read-failure" || outcome === "startup-timeout" ? 1 : 0);diff --git a/test/background-ownership.test.ts b/test/background-ownership.test.tsindex 2bd1a26cd..9baba717d 100644--- a/test/background-ownership.test.ts+++ b/test/background-ownership.test.ts@@ -3,120 +3,88 @@ import test from "node:test";import { createMemoryMap } from "../src/persistence/durable-map.ts";import { createBackgroundOwnershipStore, type BackgroundOwnership } from "../src/runs/background-ownership.ts";-async function fixture() {+test("a fresh record has no owner and the first set records who set it and when", async () => {const store = createBackgroundOwnershipStore(createMemoryMap<BackgroundOwnership>());- for (const [instanceId, deploymentId] of [- ["a1", "a"],- ["a2", "a"],- ["b1", "b"],- ["b2", "b"],- ])- await store.register({ instanceId: instanceId!, deploymentId: deploymentId!, taskArn: `task:${instanceId}` });- await store.admit("a1", 0, true);- await store.admit("a2", 0, true);- const bootstrap = {- expectedGeneration: 0,- requestId: "bootstrap",- desiredDeploymentId: "a",- bootstrapTaskArns: ["task:a1", "task:a2", "task:b1", "task:b2"],- };- return { store, bootstrap };-}--test("bootstrap preserves legacy work until every old replica acknowledges relinquishment", async () => {- const { store, bootstrap } = await fixture();- assert.equal((await store.get()).enabled, false);- await assert.rejects(store.admit("b1", 0, false), /disabled/);- await assert.rejects(store.transition({ ...bootstrap, bootstrapTaskArns: ["task:a1"] }), /cohort/);- await store.transition(bootstrap);- await assert.rejects(store.admit("a1", 1, true), /Previous owners/);- await store.acknowledge("a1", 0, "relinquished");- await assert.rejects(store.admit("a1", 1, true), /Previous owners/);- await store.acknowledge("a2", 0, "relinquished");- await Promise.all([store.admit("a1", 1, true), store.admit("a2", 1, true)]);- assert.equal((await store.get()).members.filter((member) => member.state === "admitted").length, 2);-});--test("generation CAS and replay identity fence stale transitions and admissions", async () => {- const { store, bootstrap } = await fixture();- await store.transition(bootstrap);- assert.equal((await store.transition(bootstrap)).generation, 1);- await assert.rejects(store.transition({ ...bootstrap, requestId: "different" }), /generation changed/);- await assert.rejects(store.transition({ ...bootstrap, desiredDeploymentId: "b" }), /identity reused/);- await assert.rejects(store.admit("a1", 0, true), /generation changed/);- await assert.rejects(store.admit("b1", 1, true), /not the desired/);-});--test("handover permits previous turns to drain after new deployment starts and supports rollback", async () => {- const { store, bootstrap } = await fixture();- await store.transition(bootstrap);- for (const id of ["a1", "a2"]) await store.acknowledge(id, 0, "relinquished");- for (const id of ["a1", "a2"]) await store.admit(id, 1, true);- await store.transition({ expectedGeneration: 1, requestId: "to-b", desiredDeploymentId: "b" });- for (const id of ["a1", "a2"]) await store.acknowledge(id, 1, "relinquished");- for (const id of ["b1", "b2"]) await store.admit(id, 2, false);- assert.equal((await store.get()).members.find((member) => member.instanceId === "a1")?.state, "relinquished");- await store.transition({ expectedGeneration: 2, requestId: "rollback", desiredDeploymentId: "a" });- await assert.rejects(store.admit("a1", 3, true), /Previous owners/);- for (const id of ["b1", "b2"]) await store.acknowledge(id, 2, "relinquished");- await store.admit("a1", 3, true);- await assert.rejects(store.acknowledge("a1", 1, "drained"), /admission changed/);- assert.equal((await store.get()).members.find((member) => member.instanceId === "a1")?.state, "admitted");+ assert.deepEqual(await store.get(), { ownerDeploymentId: null, setAt: null, setBy: null });+ const before = Date.now();+ const state = await store.set({ ownerDeploymentId: "a", setBy: "a" });+ assert.equal(state.ownerDeploymentId, "a");+ assert.equal(state.setBy, "a");+ assert.ok(Date.parse(state.setAt!) >= before);+ assert.deepEqual(await store.get(), state);});-test("only exact explicit termination evidence retires disconnected members and blocks task resurrection", async () => {- const { store, bootstrap } = await fixture();- await store.transition(bootstrap);- await store.acknowledge("a1", 0, "relinquished");- await assert.rejects(store.admit("a1", 1, true), /Previous owners/);- const proof = { instanceId: "a2", taskArn: "task:a2", generation: 0 };- for (const wrong of [- { ...proof, taskArn: "other" },- { ...proof, generation: 1 },- { ...proof, instanceId: "other" },- ])- await assert.rejects(- store.retire({ expectedGeneration: 1, requestId: "retire", terminatedMembers: [wrong] }),- /evidence/,- );- await store.retire({ expectedGeneration: 1, requestId: "retire", terminatedMembers: [proof] });- await store.admit("a1", 1, true);- await assert.rejects(store.register({ instanceId: "new-process", deploymentId: "a", taskArn: "task:a2" }), /retired/);- await assert.rejects(store.admit("a2", 1, true), /retired/);-});--test("pause has no successor and failed admissions never mutate ownership", async () => {- const { store, bootstrap } = await fixture();- await store.transition({ ...bootstrap, desiredDeploymentId: null });- const before = await store.get();- await assert.rejects(store.admit("a1", 1, true), /not the desired/);- assert.deepEqual(await store.get(), before);- for (const id of ["a1", "a2"]) {- await store.acknowledge(id, 0, "relinquished");- await store.acknowledge(id, 0, "drained");- }- assert.ok((await store.get()).members.every((member) => member.state === "drained"));+test("compare-and-swap refuses a stale expectation but accepts an idempotent retry", async () => {+ const store = createBackgroundOwnershipStore(createMemoryMap<BackgroundOwnership>());+ await store.set({ ownerDeploymentId: "a", setBy: "a" });+ await assert.rejects(store.set({ ownerDeploymentId: "b", expectedOwnerDeploymentId: null, setBy: "b" }), /changed/);+ assert.equal((await store.get()).ownerDeploymentId, "a");+ const handover = await store.set({ ownerDeploymentId: "b", expectedOwnerDeploymentId: "a", setBy: "b" });+ assert.equal(handover.ownerDeploymentId, "b");+ const retried = await store.set({ ownerDeploymentId: "b", expectedOwnerDeploymentId: "a", setBy: "b" });+ assert.deepEqual(retried, handover);+ const paused = await store.set({ ownerDeploymentId: null, expectedOwnerDeploymentId: "b", setBy: "b" });+ assert.equal(paused.ownerDeploymentId, null);+ assert.equal(paused.setBy, "b");});-test("compensation fences a same-generation retirement and replays only the exact precondition", async () => {- const { store, bootstrap } = await fixture();- await store.transition(bootstrap);- const observed = await store.get();- const compensation = {- expectedGeneration: observed.generation,- expectedLastRequestId: observed.lastRequestId,- requestId: "compensation",- desiredDeploymentId: "b",+test("a member-protocol record is read as its enabled desired owner and stays readable by that protocol", async () => {+ const map = createMemoryMap<BackgroundOwnership>();+ const legacy = {+ enabled: true,+ generation: 7,+ desiredDeploymentId: "core:blue",+ lastRequestId: "request",+ lastRequest: "fingerprint",+ members: [{ instanceId: "i", deploymentId: "core:blue", state: "admitted" }],};- await store.retire({- expectedGeneration: observed.generation,- requestId: "operator-retirement",- terminatedMembers: [{ instanceId: "a2", taskArn: "task:a2", generation: 0 }],+ await map.put("ownership", structuredClone(legacy) as unknown as BackgroundOwnership);+ const store = createBackgroundOwnershipStore(map);+ assert.deepEqual(await store.get(), { ownerDeploymentId: "core:blue", setAt: null, setBy: null });+ assert.deepEqual(await map.get("ownership"), legacy);+ await store.set({ ownerDeploymentId: "core:blue", setBy: "core:blue" });+ assert.deepEqual(await map.get("ownership"), legacy);+ await assert.rejects(+ store.set({ ownerDeploymentId: "core:green", expectedOwnerDeploymentId: null, setBy: "core:green" }),+ /changed/,+ );+ const migrated = await store.set({+ ownerDeploymentId: "core:green",+ expectedOwnerDeploymentId: "core:blue",+ setBy: "core:green",});- await assert.rejects(store.transition(compensation), /request changed/);- assert.equal((await store.get()).generation, 1);- const current = { ...compensation, expectedLastRequestId: "operator-retirement" };- assert.equal((await store.transition(current)).generation, 2);- assert.equal((await store.transition(current)).generation, 2);- await assert.rejects(store.transition(compensation), /identity reused/);+ assert.equal(migrated.ownerDeploymentId, "core:green");+ assert.deepEqual(await map.get("ownership"), {+ ...migrated,+ enabled: true,+ generation: 7,+ desiredDeploymentId: "core:green",+ lastRequestId: null,+ lastRequest: null,+ members: legacy.members,+ });+ await map.put("ownership", { ...legacy, enabled: false } as unknown as BackgroundOwnership);+ assert.equal((await store.get()).ownerDeploymentId, null);+});++test("a member-protocol transition written after the owner record wins until the next owner change", async () => {+ const map = createMemoryMap<BackgroundOwnership>();+ const store = createBackgroundOwnershipStore(map);+ await store.set({ ownerDeploymentId: "core:green", setBy: "core:green" });+ const stored = (await map.get("ownership")) as unknown as Record<string, unknown>;+ await map.put("ownership", {+ ...stored,+ generation: 1,+ desiredDeploymentId: "core:blue",+ } as unknown as BackgroundOwnership);+ assert.equal((await store.get()).ownerDeploymentId, "core:blue");+ await map.put("ownership", { ...stored, generation: 1, desiredDeploymentId: null } as unknown as BackgroundOwnership);+ assert.equal((await store.get()).ownerDeploymentId, null);+ await assert.rejects(+ store.set({ ownerDeploymentId: "core:green", expectedOwnerDeploymentId: "core:green", setBy: "core:green" }),+ /changed/,+ );+ const reclaimed = await store.set({ ownerDeploymentId: "core:green", expectedOwnerDeploymentId: null, setBy: "x" });+ assert.equal(reclaimed.ownerDeploymentId, "core:green");+ assert.equal(((await map.get("ownership")) as unknown as Record<string, unknown>).desiredDeploymentId, "core:green");});diff --git a/test/background-work-routes.test.ts b/test/background-work-routes.test.tsindex 78658cab2..8bf77897a 100644--- a/test/background-work-routes.test.ts+++ b/test/background-work-routes.test.ts@@ -1,6 +1,5 @@import "./support/auto-fake-sprites.ts";import assert from "node:assert/strict";-import { randomUUID } from "node:crypto";import type { AddressInfo } from "node:net";import test from "node:test";import { createServer } from "../src/api/server.ts";@@ -18,16 +17,15 @@ const path = "/v1/background-work";async function fixture(secret: string | undefined = controlSecret) {const store = createBackgroundOwnershipStore(createMemoryMap<BackgroundOwnership>());- await store.register({ instanceId: "instance-a", deploymentId: "cohort-a", taskArn: "task-a" });- await store.register({ instanceId: "instance-b", deploymentId: "cohort-b", taskArn: "task-b" });- await store.admit("instance-a", 0, true);+ await store.set({ ownerDeploymentId: "cohort-a", setBy: "cohort-a" });+ let active = false;const built = buildApp(testConfig({ signingSecret: sourceSecret }));const server = createServer(built.app, {signingSecret: sourceSecret,portalIdentitySecret: "portal-identity-secret".repeat(3),capabilitySecret: "capability-only-secret".repeat(3),requireSignedPortalIdentity: true,- backgroundOwnership: { store, instanceId: "instance-b", deploymentId: "cohort-b" },+ backgroundOwnership: { store, instanceId: "instance-b", deploymentId: "cohort-b", active: () => active },deploymentControlSecret: secret,});server.listen(0);@@ -48,15 +46,17 @@ async function fixture(secret: string | undefined = controlSecret) {...(raw ? { body: raw } : {}),});};- return { store, request, close: () => new Promise<void>((resolve) => server.close(() => resolve())) };+ return {+ store,+ request,+ activate: () => {+ active = true;+ },+ close: () => new Promise<void>((resolve) => server.close(() => resolve())),+ };}-const transition = () => ({- expectedGeneration: 0,- requestId: randomUUID(),- desiredDeploymentId: "cohort-b",- bootstrapTaskArns: ["task-a", "task-b"],-});+const handover = () => ({ ownerDeploymentId: "cohort-b", expectedOwnerDeploymentId: "cohort-a" });test("ownership controls require both source and distinct deployment credentials", async () => {const srv = await fixture();@@ -67,19 +67,19 @@ test("ownership controls require both source and distinct deployment credentials{ "x-signature": "invalid" },];for (const credentials of invalidCredentials) {- assert.equal((await srv.request("POST", transition(), credentials)).status, 401);+ assert.equal((await srv.request("POST", handover(), credentials)).status, 401);}const capability = await mintCapabilityToken({ actorId: "U1", scopeId: scopeId("personal", "U1"), exp: Date.now() + CAPABILITY_TTL_MS },"capability-only-secret".repeat(3),);for (const method of ["GET", "POST"]) {- const response = await srv.request(method, method === "POST" ? transition() : undefined, {+ const response = await srv.request(method, method === "POST" ? handover() : undefined, {"x-agent-capability": capability,});assert.equal(response.status, 403);}- assert.equal((await srv.store.get()).generation, 0);+ assert.equal((await srv.store.get()).ownerDeploymentId, "cohort-a");assert.equal((await srv.request("GET")).status, 200);} finally {await srv.close();@@ -94,94 +94,71 @@ test("ownership controls require both source and distinct deployment credentials}});-test("authenticated transitions report exact responder identity, preserve retry identity, and reject stale generations", async () => {+test("status reports the responder, the owner record, and the responder's activity", async () => {const srv = await fixture();try {- const change = transition();- const response = await srv.request("POST", change);+ const before = (await (await srv.request("GET")).json()) as Record<string, unknown>;+ assert.equal(before.protocol, 2);+ assert.equal(before.instanceId, "instance-b");+ assert.equal(before.deploymentId, "cohort-b");+ assert.equal(before.ownerDeploymentId, "cohort-a");+ assert.equal(before.setBy, "cohort-a");+ assert.equal(typeof before.setAt, "string");+ assert.equal(before.active, false);+ srv.activate();+ const response = await srv.request("POST", handover());assert.equal(response.status, 200);const status = (await response.json()) as Record<string, unknown>;- assert.equal(status.protocol, 1);- assert.equal(status.instanceId, "instance-b");- assert.equal(status.deploymentId, "cohort-b");- assert.equal(status.generation, 1);- assert.equal(status.lastRequestId, change.requestId);- assert.equal(status.lastRequest, undefined);- assert.equal((await srv.request("POST", change)).status, 200);- assert.equal((await srv.request("POST", transition())).status, 409);- const retired = await srv.request("POST", {- expectedGeneration: 1,- requestId: randomUUID(),- terminatedMembers: [{ instanceId: "instance-a", taskArn: "wrong-task", generation: 0 }],- });- assert.equal(retired.status, 409);- const pause = await srv.request("POST", {- expectedGeneration: 1,- requestId: randomUUID(),- desiredDeploymentId: null,- });- assert.equal(pause.status, 200);- assert.equal((await srv.store.get()).desiredDeploymentId, null);+ assert.equal(status.ownerDeploymentId, "cohort-b");+ assert.equal(status.setBy, "cohort-b");+ assert.equal(status.active, true);+ assert.deepEqual(Object.keys(status).sort(), [+ "active",+ "deploymentId",+ "instanceId",+ "ownerDeploymentId",+ "protocol",+ "setAt",+ "setBy",+ ]);} finally {await srv.close();}});-test("malformed or mixed mutations cannot change ownership", async () => {+test("compare-and-swap rejects stale expectations while retries and pauses succeed", async () => {const srv = await fixture();try {- for (const body of [- null,- { ...transition(), expectedGeneration: -1 },- { ...transition(), requestId: "invalid" },- { ...transition(), desiredDeploymentId: 3 },- { ...transition(), extra: true },- { ...transition(), bootstrapTaskArns: [] },- { ...transition(), terminatedMembers: [] },- {- expectedGeneration: 0,- requestId: randomUUID(),- terminatedMembers: [{ instanceId: "instance-a", taskArn: "task-a", generation: -1 }],- },- { expectedGeneration: 0, requestId: randomUUID() },- ])- assert.equal((await srv.request("POST", body)).status, 400);- assert.equal((await srv.store.get()).generation, 0);- const retired = await srv.request("POST", {- expectedGeneration: 0,- requestId: randomUUID(),- terminatedMembers: [{ instanceId: "instance-a", taskArn: "task-a", generation: 0 }],- });- assert.equal(retired.status, 200);- assert.equal((await srv.store.get()).members.find((member) => member.instanceId === "instance-a")?.retired, true);+ assert.equal((await srv.request("POST", { ...handover(), expectedOwnerDeploymentId: null })).status, 409);+ assert.equal((await srv.store.get()).ownerDeploymentId, "cohort-a");+ assert.equal((await srv.request("POST", handover())).status, 200);+ assert.equal((await srv.request("POST", handover())).status, 200);+ assert.equal(+ (await srv.request("POST", { ownerDeploymentId: "cohort-a", expectedOwnerDeploymentId: "cohort-c" })).status,+ 409,+ );+ assert.equal((await srv.request("POST", { ownerDeploymentId: null })).status, 200);+ assert.equal((await srv.store.get()).ownerDeploymentId, null);} finally {await srv.close();}});-test("compensation request preconditions are enforced atomically through the authenticated route", async () => {+test("malformed mutations cannot change ownership", async () => {const srv = await fixture();try {- const promotion = transition();- assert.equal((await srv.request("POST", promotion)).status, 200);- const compensation = {- expectedGeneration: 1,- expectedLastRequestId: promotion.requestId,- requestId: randomUUID(),- desiredDeploymentId: null,- };- const retirement = {- expectedGeneration: 1,- requestId: randomUUID(),- terminatedMembers: [{ instanceId: "instance-a", taskArn: "task-a", generation: 0 }],- };- assert.equal((await srv.request("POST", retirement)).status, 200);- assert.equal((await srv.request("POST", compensation)).status, 409);- assert.equal((await srv.request("POST", { ...compensation, expectedLastRequestId: 123 })).status, 400);- const current = { ...compensation, expectedLastRequestId: retirement.requestId };- assert.equal((await srv.request("POST", current)).status, 200);- assert.equal((await srv.request("POST", current)).status, 200);- assert.equal((await srv.store.get()).generation, 2);+ for (const body of [+ null,+ {},+ { ownerDeploymentId: 3 },+ { ownerDeploymentId: "" },+ { ownerDeploymentId: "x".repeat(2049) },+ { ...handover(), extra: true },+ { ...handover(), expectedOwnerDeploymentId: 7 },+ { expectedGeneration: 0, requestId: "legacy", desiredDeploymentId: "cohort-b" },+ ])+ assert.equal((await srv.request("POST", body)).status, 400);+ assert.equal((await srv.store.get()).ownerDeploymentId, "cohort-a");} finally {await srv.close();}diff --git a/test/deployment-live-smoke-routes.test.ts b/test/deployment-live-smoke-routes.test.tsindex 8d517c42c..31b9d4d3f 100644--- a/test/deployment-live-smoke-routes.test.ts+++ b/test/deployment-live-smoke-routes.test.ts@@ -19,15 +19,8 @@ const path = "/v1/deployment/live-session";async function fixture(run: () => Promise<void> = async () => {}) {const store = createBackgroundOwnershipStore(createMemoryMap<BackgroundOwnership>());- await store.register({ instanceId: "instance-a", deploymentId: "cohort-a", taskArn: "task-a" });- await store.transition({- expectedGeneration: 0,- requestId: randomUUID(),- desiredDeploymentId: "cohort-a",- bootstrapTaskArns: ["task-a"],- });- await store.admit("instance-a", 1, false);- await store.markReady("instance-a", 1);+ await store.set({ ownerDeploymentId: "cohort-a", setBy: "cohort-a" });+ let active = true;const dedupe = createMemoryReplayDedupe();const built = buildApp(testConfig({ signingSecret: sourceSecret }));const server = createServer(built.app, {@@ -35,7 +28,7 @@ async function fixture(run: () => Promise<void> = async () => {}) {portalIdentitySecret: "portal-identity-secret".repeat(3),capabilitySecret: "capability-only-secret".repeat(3),requireSignedPortalIdentity: true,- backgroundOwnership: { store, instanceId: "instance-a", deploymentId: "cohort-a" },+ backgroundOwnership: { store, instanceId: "instance-a", deploymentId: "cohort-a", active: () => active },deploymentControlSecret: controlSecret,deploymentLiveSmoke: run,replayDedupe: { durable: true, claim: (...args) => dedupe.claim(...args) },@@ -58,15 +51,17 @@ async function fixture(run: () => Promise<void> = async () => {}) {...(raw ? { body: raw } : {}),});};- return { store, request, close: () => new Promise<void>((resolve) => server.close(() => resolve())) };+ return {+ store,+ request,+ deactivate: () => {+ active = false;+ },+ close: () => new Promise<void>((resolve) => server.close(() => resolve())),+ };}-const body = () => ({- requestId: randomUUID(),- expectedDeploymentId: "cohort-a",- expectedGeneration: 1,- expectedTaskArns: ["task-a"],-});+const body = () => ({ requestId: randomUUID(), expectedDeploymentId: "cohort-a" });test("live smoke requires distinct credentials, rejects capabilities and arbitrary inputs", async () => {let calls = 0;@@ -90,18 +85,17 @@ test("live smoke requires distinct credentials, rejects capabilities and arbitrafor (const request of [null,{ ...body(), url: "https://untrusted.invalid" },- { ...body(), expectedTaskArns: [] },- { ...body(), expectedTaskArns: ["task-a", "task-a"] },+ { ...body(), expectedGeneration: 1 },{ ...body(), expectedDeploymentId: "other" },+ { ...body(), requestId: "not-a-uuid" },]) {assert.equal((await srv.request("POST", request)).status, 400);}- for (const request of [- { ...body(), expectedGeneration: 2 },- { ...body(), expectedTaskArns: ["different"] },- ]) {- assert.equal((await srv.request("POST", request)).status, 409);- }+ await srv.store.set({ ownerDeploymentId: "cohort-b", setBy: "cohort-b" });+ assert.equal((await srv.request("POST", body())).status, 409);+ await srv.store.set({ ownerDeploymentId: "cohort-a", setBy: "cohort-a" });+ srv.deactivate();+ assert.equal((await srv.request("POST", body())).status, 409);assert.equal(calls, 0);} finally {await srv.close();@@ -123,8 +117,6 @@ test("live smoke returns exact identity and rejects request replay", async () =>requestId: request.requestId,deploymentId: "cohort-a",instanceId: "instance-a",- taskArn: "task-a",- generation: 1,});assert.equal((await srv.request("POST", request)).status, 409);assert.equal(calls, 1);@@ -165,7 +157,7 @@ test("live smoke keeps singleflight through client disconnect and cleanup", asyntest("live smoke fails when ownership changes and never returns raw errors", async () => {const srv = await fixture(async () => {- await srv.store.transition({ expectedGeneration: 1, requestId: randomUUID(), desiredDeploymentId: null });+ await srv.store.set({ ownerDeploymentId: null, setBy: "operator" });});try {const result = (await (await srv.request("POST", body())).json()) as { ok: boolean; error: string };diff --git a/test/legacy-enrollment-bridge.test.ts b/test/legacy-enrollment-bridge.test.tsdeleted file mode 100644index b388db37e..000000000--- a/test/legacy-enrollment-bridge.test.ts+++ /dev/null@@ -1,42 +0,0 @@-import assert from "node:assert/strict";-import test from "node:test";-import { createLegacyEnrollmentBridge } from "../src/runs/instance-registry.ts";--test("legacy enrollment publishes only when admitted and ignores legacy supersession", async () => {- let eligible = false;- let beats = 0;- const bridge = createLegacyEnrollmentBridge(- {- beat: async () => {- beats++;- return true;- },- },- async () => eligible,- );- assert.equal(await bridge.beat(), false);- assert.equal(beats, 0);- eligible = true;- assert.equal(await bridge.beat(), false);- assert.equal(beats, 1);- eligible = false;- assert.equal(await bridge.beat(), false);- assert.equal(beats, 1);-});--test("legacy enrollment propagates failed eligibility reads without publishing", async () => {- let beats = 0;- const bridge = createLegacyEnrollmentBridge(- {- beat: async () => {- beats++;- return false;- },- },- async () => {- throw new Error("database unavailable");- },- );- await assert.rejects(bridge.beat(), /database unavailable/);- assert.equal(beats, 0);-});diff --git a/test/postgres-background-ownership.test.ts b/test/postgres-background-ownership.test.tsindex cf05b0134..09202af55 100644--- a/test/postgres-background-ownership.test.ts+++ b/test/postgres-background-ownership.test.ts@@ -5,39 +5,56 @@ import { createPostgresMapFactory } from "../src/persistence/durable-map.ts";import { createBackgroundOwnershipStore, type BackgroundOwnership } from "../src/runs/background-ownership.ts";const databaseUrl = process.env.DATABASE_URL;-test("Postgres serializes concurrent ownership transitions and replica admission", { skip: !databaseUrl }, async () => {- const table = `background_test_${randomUUID().replaceAll("-", "")}`;- const factories = [createPostgresMapFactory(databaseUrl!), createPostgresMapFactory(databaseUrl!)];- try {- const stores = factories.map((factory) => createBackgroundOwnershipStore(factory.map<BackgroundOwnership>(table)));- await Promise.all(- stores.map((store, i) => store.register({ instanceId: `a${i}`, deploymentId: "a", taskArn: `task:a${i}` })),- );- await Promise.all(stores.map((store, i) => store.admit(`a${i}`, 0, true)));- const transitions = await Promise.allSettled(- stores.map((store) =>- store.transition({- expectedGeneration: 0,- requestId: randomUUID(),- desiredDeploymentId: "a",- bootstrapTaskArns: ["task:a0", "task:a1"],- }),- ),- );- assert.equal(transitions.filter((result) => result.status === "fulfilled").length, 1);- assert.equal(transitions.filter((result) => result.status === "rejected").length, 1);- await assert.rejects(stores[0]!.admit("a0", 1, true), /Previous owners/);- await Promise.all(stores.map((store, i) => store.acknowledge(`a${i}`, 0, "relinquished")));- await Promise.all(stores.map((store, i) => store.admit(`a${i}`, 1, true)));- const state = await stores[0]!.get();- assert.equal(state.members.filter((member) => member.state === "admitted" && member.generation === 1).length, 2);- assert.deepEqual(await stores[1]!.get(), state);- } finally {- const pool = await factories[0]!.pool.pool();- await pool.query(`DROP TABLE IF EXISTS ${table}`);- await Promise.all(factories.map((factory) => factory.pool.close()));- }-});+test(+ "Postgres serializes concurrent owner changes and migrates a member-protocol record",+ { skip: !databaseUrl },+ async () => {+ const table = `background_test_${randomUUID().replaceAll("-", "")}`;+ const factories = [createPostgresMapFactory(databaseUrl!), createPostgresMapFactory(databaseUrl!)];+ try {+ const maps = factories.map((factory) => factory.map<BackgroundOwnership>(table));+ const stores = maps.map((map) => createBackgroundOwnershipStore(map));+ const claims = await Promise.allSettled(+ stores.map((store, i) =>+ store.set({ ownerDeploymentId: `d${i}`, expectedOwnerDeploymentId: null, setBy: `d${i}` }),+ ),+ );+ assert.equal(claims.filter((result) => result.status === "fulfilled").length, 1);+ assert.equal(claims.filter((result) => result.status === "rejected").length, 1);+ const state = await stores[0]!.get();+ assert.ok(["d0", "d1"].includes(state.ownerDeploymentId!));+ assert.deepEqual(await stores[1]!.get(), state);+ await maps[0]!.put("ownership", {+ enabled: true,+ generation: 3,+ desiredDeploymentId: "core:blue",+ lastRequestId: null,+ lastRequest: null,+ members: [{ instanceId: "old", deploymentId: "core:blue", state: "admitted" }],+ } as unknown as BackgroundOwnership);+ assert.equal((await stores[1]!.get()).ownerDeploymentId, "core:blue");+ const migrated = await stores[1]!.set({+ ownerDeploymentId: "core:green",+ expectedOwnerDeploymentId: "core:blue",+ setBy: "core:green",+ });+ assert.equal(migrated.ownerDeploymentId, "core:green");+ assert.deepEqual(await maps[0]!.get("ownership"), {+ ...migrated,+ enabled: true,+ generation: 3,+ desiredDeploymentId: "core:green",+ lastRequestId: null,+ lastRequest: null,+ members: [{ instanceId: "old", deploymentId: "core:blue", state: "admitted" }],+ });+ } finally {+ const pool = await factories[0]!.pool.pool();+ await pool.query(`DROP TABLE IF EXISTS ${table}`);+ await Promise.all(factories.map((factory) => factory.pool.close()));+ }+ },+);test("Postgres queue resumes polling without terminating a prior generation callback",@@ -85,73 +102,52 @@ test();test(- "delayed ownership commit after an old read is confirmed by replaying the same mutation",+ "a delayed owner change committed after the client gave up is confirmed by the readback",{ skip: !databaseUrl, timeout: 20_000 },async () => {- const { mutateBackgroundWork } = await import("../cli/src/background-work.ts");+ const { setBackgroundOwner } = await import("../cli/src/background-work.ts");const table = `background_test_${randomUUID().replaceAll("-", "")}`;const factory = createPostgresMapFactory(databaseUrl!);const store = createBackgroundOwnershipStore(factory.map<BackgroundOwnership>(table));const pool = await factory.pool.pool();const blocker = await pool.connect();let first: Promise<BackgroundOwnership> | undefined;+ const status = (state: BackgroundOwnership) =>+ JSON.stringify({ ...state, protocol: 2, deploymentId: "b", instanceId: "b-1", active: false });try {- await store.register({ instanceId: "a", deploymentId: "a", taskArn: "task:a" });- await store.transition({- expectedGeneration: 0,- requestId: randomUUID(),- desiredDeploymentId: "a",- bootstrapTaskArns: ["task:a"],- });+ await store.set({ ownerDeploymentId: "a", setBy: "a" });await blocker.query("BEGIN");await blocker.query("SELECT * FROM durable_map_versions WHERE tbl = $1 FOR UPDATE", [table]);- const mutation = { expectedGeneration: 1, requestId: randomUUID(), desiredDeploymentId: "a" };+ const change = { ownerDeploymentId: "b", expectedOwnerDeploymentId: "a" };let posts = 0;- let oldReads = 0;- const result = await mutateBackgroundWork(+ let reads = 0;+ const result = await setBackgroundOwner(async (method, body) => {if (method === "POST") {posts++;- assert.equal(body, JSON.stringify(mutation));+ assert.equal(body, JSON.stringify(change));if (posts === 1) {- first = store.transition(mutation);+ first = store.set({ ...change, setBy: "b" });throw new Error("client timeout while server transaction is pending");}await blocker.query("COMMIT");await first;- const committed = await store.transition(mutation);- return {- status: 200,- body: JSON.stringify({ ...committed, protocol: 1, deploymentId: "a", instanceId: "a" }),- };+ return { status: 200, body: status(await store.set({ ...change, setBy: "b" })) };}- const old = await store.get();- assert.equal(old.generation, 1);- oldReads++;- return { status: 200, body: JSON.stringify({ ...old, protocol: 1, deploymentId: "a", instanceId: "a" }) };+ reads++;+ const current = await store.get();+ if (reads === 1) assert.equal(current.ownerDeploymentId, "a");+ return { status: 200, body: status(current) };},- "a",- mutation,+ "b",+ change,);- assert.equal(result.generation, 2);+ assert.equal(result.ownerDeploymentId, "b");assert.equal(posts, 2);- assert.equal(oldReads, 1);- assert.equal((await store.get()).generation, 2);- const precondition = {- expectedGeneration: 2,- expectedLastRequestId: mutation.requestId,- requestId: randomUUID(),- desiredDeploymentId: null,- };- await store.retire({- expectedGeneration: 2,- requestId: randomUUID(),- terminatedMembers: [{ instanceId: "a", taskArn: "task:a", generation: 0 }],- });- await assert.rejects(store.transition(precondition), /request changed/);- assert.equal((await store.get()).generation, 2);+ assert.equal(reads, 1);+ assert.equal((await store.get()).ownerDeploymentId, "b");} finally {- await blocker.query("ROLLBACK");+ await blocker.query("ROLLBACK").catch(() => {});blocker.release();await first?.catch(() => {});await pool.query(`DROP TABLE IF EXISTS ${table}`);diff --git a/test/postgres-instance-registry.test.ts b/test/postgres-instance-registry.test.tsindex 01ca1e004..fd3e4cdb0 100644--- a/test/postgres-instance-registry.test.ts+++ b/test/postgres-instance-registry.test.ts@@ -1,7 +1,7 @@import { test, before } from "node:test";import assert from "node:assert/strict";import { createPostgresMapFactory } from "../src/persistence/durable-map.ts";-import { createPostgresInstanceRegistry, createLegacyEnrollmentBridge } from "../src/runs/instance-registry.ts";+import { createPostgresInstanceRegistry } from "../src/runs/instance-registry.ts";import { settle } from "./support/settle.ts";const URL = process.env.DATABASE_URL;@@ -53,47 +53,6 @@ test(},);-test("controlled enrollment drains same-image legacy workers without self-supersession", { skip }, async () => {- const factory = createPostgresMapFactory(URL!);- const old = createPostgresInstanceRegistry(factory.pool, {- instanceId: "migration-old",- buildSha: "same-image",- startedAt: 10_000,- livenessMs: 200,- });- let eligible = false;- const bridge = createLegacyEnrollmentBridge(- createPostgresInstanceRegistry(factory.pool, {- instanceId: "migration-new",- buildSha: "enrollment:cohort-a",- startedAt: 20_000,- livenessMs: 200,- }),- async () => eligible,- );- try {- assert.equal(await bridge.beat(), false);- assert.equal(await old.beat(), false);- eligible = true;- assert.equal(await bridge.beat(), false);- assert.equal(await old.beat(), true);- const newer = createPostgresInstanceRegistry(factory.pool, {- instanceId: "migration-peer",- buildSha: "another-image",- startedAt: 30_000,- livenessMs: 200,- });- await newer.beat();- assert.equal(await bridge.beat(), false);- eligible = false;- await new Promise((resolve) => setTimeout(resolve, 250));- assert.equal(await bridge.beat(), false);- assert.equal(await old.beat(), false);- } finally {- await factory.pool.close();- }-});-test("a core with background work disabled never heartbeats, so it cannot drain a peer that does",{ skip },diff --git a/test/worker-reaper.test.ts b/test/worker-reaper.test.tsindex 0e377c7fa..e62740ac2 100644--- a/test/worker-reaper.test.ts+++ b/test/worker-reaper.test.ts@@ -685,7 +685,10 @@ test("inline turns remain admitted through pause and queued intake survives roll}),);let admitted = true;- built.runtime.setBackgroundAdmission(() => admitted);+ built.runtime.setBackgroundAdmission(+ () => admitted,+ () => admitted,+ );const completing = Promise.withResolvers<void>();const release = Promise.withResolvers<void>();const complete = built.runs.complete.bind(built.runs);