Multiplayer agent harness for work.
Transfer background ownership between live deployment revisions (#1291)
Regan Bell7480e83e7d48
diff --git a/docs/background-ownership.md b/docs/background-ownership.mdnew file mode 100644index 000000000..d9dc16d97--- /dev/null+++ b/docs/background-ownership.md@@ -0,0 +1,65 @@+# Background ownership++A core deployment can transfer background work to another deployment without restarting either HTTP process. Existing runs retain their leases and heartbeats while the old deployment stops accepting background work. The deployment controller uses a shared PostgreSQL record to fence generations and record each process's acknowledgment.++## Enable the capability++Configure every core replica with:++| Variable | Requirement |+| --------------------------- | ------------------------------------------------------------------------------------------------------------------------- |+| `BACKGROUND_DEPLOYMENT_ID` | An immutable identity shared by replicas of one deployment revision, distinct from every other deployment revision. |+| `DEPLOYMENT_CONTROL_SECRET` | At least 32 characters, distinct from `CORE_SIGNING_SECRET`; distribute only to core and the trusted deployment operator. |+| `CORE_SIGNING_SECRET` | The normal source request signing secret. |+| `DATABASE_URL` | The shared PostgreSQL database. |++On ECS, `ECS_CONTAINER_METADATA_URI_V4` supplies the task ARN used to associate process membership with infrastructure termination evidence. A process without task metadata can enroll, but cannot participate in an ECS cohort bootstrap or task retirement.++Installing this capability does not activate the protocol. Until explicit bootstrap, each replica follows `BACKGROUND_WORK_ENABLED`. Upgrade both deployments and verify their exact current task cohort before bootstrapping. Do not fall back to boot-flag mutation after the protocol has been enabled.++## Operator endpoint++`GET /v1/background-work` and `POST /v1/background-work` require both normal source request signing and `Authorization: Bearer <DEPLOYMENT_CONTROL_SECRET>`. Agent capabilities cannot invoke these routes. Portal forwards only those exact methods and path, preserving the supplied credentials.++Status includes `protocol: 1`, the responding process's `deploymentId` and `instanceId`, `enabled`, `generation`, `desiredDeploymentId`, `lastRequestId`, and durable `members`. Every member includes its deployment identity, process identity, task ARN, admission generation, state, readiness, and retirement flag.++To bootstrap, post:++```json+{+ "expectedGeneration": 0,+ "requestId": "ed73d35e-7dfd-49b9-8550-62955df7df7b",+ "desiredDeploymentId": "core:release-a",+ "bootstrapTaskArns": ["exact-task-arn-a", "exact-task-arn-b"]+}+```++The cohort must match all enrolled, nonretired processes exactly. Later transitions omit `bootstrapTaskArns` and increment the generation using the same compare-and-swap contract. Set `desiredDeploymentId` to `null` to pause with no successor.++Use a fresh UUID for each logical mutation and reuse that UUID only when retrying the same payload. After an uncertain response, inspect `lastRequestId`, generation, desired identity, and the affected member records. A matching desired identity alone does not prove that a particular request committed.++## Admission, readiness, and draining++A process records `admitted` before starting background resources. The desired deployment cannot admit while any earlier generation remains admitted. All replicas of the desired deployment may admit; existing per-job claims and leases continue to coordinate work within the deployment.++`ready: true` means activation completed. A deployment operator should wait for every expected current task to have an admitted, ready process at the desired generation before considering activation successful.++`relinquished` acknowledges that the process stopped new claims and closed its Slack ingress. Already admitted turns may still run. Cron polling can resume after rollback while earlier callbacks keep their queue connection and heartbeats. Maintenance callbacks that cannot relinquish safely are joined before acknowledgment.++`drained` means the process's admitted background work has finished. Releasing ownership and making a deployment safe to replace are separate gates. Do not terminate a relinquished but undrained process merely to meet a rollout time target.++Database errors or an expired local validity watchdog fence new local work. They do not establish durable relinquishment or authorize another deployment to bypass an outstanding member.++## Terminated processes++When a process cannot acknowledge, the trusted operator must independently prove that its exact infrastructure task has stopped. It can then post:++```json+{+ "expectedGeneration": 3,+ "requestId": "e59f2af3-1d57-49a4-9453-91a391540c17",+ "terminatedMembers": [{ "instanceId": "exact-process-id", "taskArn": "exact-stopped-task-arn", "generation": 2 }]+}+```++Retirement preserves the ownership generation, updates `lastRequestId`, and marks only the matching members retired and drained. Retired task identities cannot enroll again. An unreachable HTTP endpoint, old heartbeat, or elapsed timeout is not termination evidence.diff --git a/package.json b/package.jsonindex af3f5e9d3..3c4300574 100644--- a/package.json+++ b/package.json@@ -45,7 +45,7 @@"sandbox:local:build": "bash scripts/local-sandbox-build.sh","sandbox:porter:build": "docker build --platform linux/amd64 -t \"${PORTER_SANDBOX_IMAGE:?Set PORTER_SANDBOX_IMAGE to the registry tag to publish}\" porter && docker push \"$PORTER_SANDBOX_IMAGE\"","deploy:porter-runner-image": "docker build --platform linux/amd64 -t \"${PORTER_DEPLOY_RUNNER_IMAGE:?Set PORTER_DEPLOY_RUNNER_IMAGE to the registry tag to publish}\" -f deploy/porter-runner/Dockerfile . && docker push \"$PORTER_DEPLOY_RUNNER_IMAGE\"",- "test:pg": "node --test --test-concurrency=1 test/postgres-swarms.test.ts test/postgres-store.test.ts test/postgres-grant-store.test.ts test/postgres-admin-grants.test.ts test/postgres-file-artifact-store.test.ts test/postgres-file-upload-store.test.ts test/postgres-directory-store.test.ts test/postgres-map.test.ts test/postgres-resource-search.test.ts test/deploy-store-pg.test.ts test/postgres-cron-fires.test.ts test/postgres-migration-adoption.test.ts test/postgres-migrations.test.ts test/postgres-migration-startup.test.ts test/cron-queue.test.ts test/postgres-metrics-sink.test.ts test/postgres-error-log.test.ts test/postgres-audit-log.test.ts test/postgres-budget.test.ts test/postgres-rate-limiter.test.ts test/postgres-credential-usage-sink.test.ts test/postgres-config-store.test.ts test/postgres-delivery-store.test.ts test/postgres-replay-dedupe.test.ts test/postgres-broker-sessions.test.ts test/postgres-egress-audit-sink.test.ts test/postgres-memory-service.test.ts test/run-signal-store.test.ts test/postgres-listener.test.ts test/postgres-run-activity-store.test.ts test/leader-lease.test.ts test/advisory-lock.test.ts test/persistence-init-retry.test.ts test/migrate-principals.test.ts test/postgres-surface-cache.test.ts test/slack-mirror-fidelity.test.ts test/postgres-instance-registry.test.ts test/run-retry-delay.test.ts test/postgres-run-stream.test.ts test/migrate-main.test.ts",+ "test:pg": "node --test --test-concurrency=1 test/postgres-swarms.test.ts test/postgres-store.test.ts test/postgres-grant-store.test.ts test/postgres-admin-grants.test.ts test/postgres-file-artifact-store.test.ts test/postgres-file-upload-store.test.ts test/postgres-directory-store.test.ts test/postgres-map.test.ts test/postgres-background-ownership.test.ts test/postgres-resource-search.test.ts test/deploy-store-pg.test.ts test/postgres-cron-fires.test.ts test/postgres-migration-adoption.test.ts test/postgres-migrations.test.ts test/postgres-migration-startup.test.ts test/cron-queue.test.ts test/postgres-metrics-sink.test.ts test/postgres-error-log.test.ts test/postgres-audit-log.test.ts test/postgres-budget.test.ts test/postgres-rate-limiter.test.ts test/postgres-credential-usage-sink.test.ts test/postgres-config-store.test.ts test/postgres-delivery-store.test.ts test/postgres-replay-dedupe.test.ts test/postgres-broker-sessions.test.ts test/postgres-egress-audit-sink.test.ts test/postgres-memory-service.test.ts test/run-signal-store.test.ts test/postgres-listener.test.ts test/postgres-run-activity-store.test.ts test/leader-lease.test.ts test/advisory-lock.test.ts test/persistence-init-retry.test.ts test/migrate-principals.test.ts test/postgres-surface-cache.test.ts test/slack-mirror-fidelity.test.ts test/postgres-instance-registry.test.ts test/run-retry-delay.test.ts test/postgres-run-stream.test.ts test/migrate-main.test.ts","livetest": "node scripts/api-livetest.ts","bench:memory": "node --env-file-if-exists=.env scripts/memory-bench.ts","prepare": "node .husky/install.mjs",diff --git a/plugins/portal/src/index.ts b/plugins/portal/src/index.tsindex 9b062744d..da32b1f3d 100644--- a/plugins/portal/src/index.ts+++ b/plugins/portal/src/index.ts@@ -1132,6 +1132,13 @@ async function handle(req: IncomingMessage, res: ServerResponse): Promise<void>});}+ if ((method === "GET" || method === "POST") && pathname === "/v1/background-work") {+ return proxyToUpstream(req, res, { baseUrl: CORE, path: pathname, search: url.search }, [+ ...FORWARD_DEPLOYMENT_LAYER_HEADERS,+ "authorization",+ ]);+ }+if (isDeploymentLayerPassthrough(method, pathname)) {return proxyToUpstream(req,diff --git a/plugins/portal/test/router.test.ts b/plugins/portal/test/router.test.tsindex 8f1e0b434..063105a76 100644--- a/plugins/portal/test/router.test.ts+++ b/plugins/portal/test/router.test.ts@@ -12,7 +12,7 @@ let deploymentLayerRequests = 0;const VALID_SOURCE_SIGNATURE = "v0=valid-source-signature";const upstream = createServer((req: IncomingMessage, res) => {- if (req.url?.startsWith("/v1/deployment-layer")) {+ if (req.url?.startsWith("/v1/deployment-layer") || req.url?.startsWith("/v1/background-work")) {deploymentLayerRequests++;if (req.headers["x-timestamp"] !== "123" || req.headers["x-signature"] !== VALID_SOURCE_SIGNATURE) {res.writeHead(401, { "content-type": "application/json" });@@ -629,3 +629,41 @@ test("impersonate: an admin starts it; the web-ui hop carries target + impersonaassert.equal(stop.status, 200);assert.match(stop.headers.get("set-cookie") ?? "", /portal_impersonate=;[^,]*Max-Age=0/);});++test("background ownership forwards both credentials only on its exact control routes", async () => {+ for (const method of ["GET", "POST"]) {+ const response = await fetch(`${base}/v1/background-work`, {+ method,+ headers: {+ "x-timestamp": "123",+ "x-signature": VALID_SOURCE_SIGNATURE,+ authorization: "Bearer deployment-only-secret",+ "x-as-principal": "must-not-cross",+ cookie: "portal_session=must-not-cross",+ ...(method === "POST" ? { "content-type": "application/json" } : {}),+ },+ ...(method === "POST" ? { body: '{"expectedGeneration":0}' } : {}),+ });+ assert.equal(response.status, 200);+ const payload = (await response.json()) as { headers: Record<string, string>; body: string };+ assert.equal(payload.headers.authorization, "Bearer deployment-only-secret");+ assert.equal(payload.headers["x-signature"], VALID_SOURCE_SIGNATURE);+ assert.equal(payload.headers.cookie, undefined);+ assert.equal(payload.headers["x-as-principal"], undefined);+ if (method === "POST") assert.equal(payload.body, '{"expectedGeneration":0}');+ }+ for (const [method, path] of [+ ["PUT", "/v1/background-work"],+ ["GET", "/v1/background-work/nearby"],+ ]) {+ const response = await fetch(`${base}${path}`, {+ method,+ headers: {+ authorization: "Bearer deployment-only-secret",+ "x-timestamp": "123",+ "x-signature": VALID_SOURCE_SIGNATURE,+ },+ });+ assert.equal(response.status, 404);+ }+});diff --git a/src/api/deps.ts b/src/api/deps.tsindex 6f3daa416..ce88b5fdc 100644--- a/src/api/deps.ts+++ b/src/api/deps.ts@@ -1,3 +1,4 @@+import type { BackgroundOwnershipStore } from "../runs/background-ownership.ts";import type { createSuggestedActivityService } from "../suggestions/activities.ts";import type { ManagedSlack } from "../surfaces/slack-managed.ts";import type { BrokerSessionStore } from "../auth/broker-sessions.ts";@@ -163,6 +164,8 @@ export interface ServerDeps {sessionShareBytes?: DurableByteStore;environments?: EnvironmentStore;deploymentLayer?: DeploymentLayerStore;+ backgroundOwnership?: { store: BackgroundOwnershipStore; instanceId: string; deploymentId: string };+ deploymentControlSecret?: string;credentialServices?: () => readonly string[];brokeredServices?: () => readonly string[];deployDialTimeoutMs?: number;diff --git a/src/api/routes/background-work.ts b/src/api/routes/background-work.tsnew file mode 100644index 000000000..eaab81115--- /dev/null+++ b/src/api/routes/background-work.ts@@ -0,0 +1,119 @@+import { isStrongSigningSecret } from "../../auth/source-auth.ts";+import { timingSafeEqual } from "node:crypto";+import { BackgroundOwnershipConflict, type BackgroundOwnership } from "../../runs/background-ownership.ts";+import { sendJson } from "../http.ts";+import { isObj } from "./shared.ts";+import type { ApiCtx, Route } from "./route.ts";++function identity(value: unknown): value is string {+ return typeof value === "string" && value.length > 0 && value.length <= 2048;+}++function generation(value: unknown): value is number {+ return typeof value === "number" && Number.isSafeInteger(value) && value >= 0;+}++function keys(value: Record<string, unknown>, allowed: string[]): boolean {+ return Object.keys(value).every((key) => allowed.includes(key));+}++function status(ctx: ApiCtx, state: BackgroundOwnership): void {+ const control = ctx.deps.backgroundOwnership!;+ sendJson(ctx.res, 200, {+ protocol: 1,+ deploymentId: control.deploymentId,+ instanceId: control.instanceId,+ enabled: state.enabled,+ generation: state.generation,+ desiredDeploymentId: state.desiredDeploymentId,+ lastRequestId: state.lastRequestId,+ members: state.members,+ });+}++async function backgroundWork(ctx: ApiCtx): Promise<void> {+ const control = ctx.deps.backgroundOwnership;+ const secret = ctx.deps.deploymentControlSecret;+ if (!control || !ctx.secret || !ctx.auth || !isStrongSigningSecret(secret) || secret === ctx.secret) {+ return sendJson(ctx.res, 503, { error: "background_control_unavailable" });+ }+ const bearer = ctx.req.headers.authorization;+ const expected = Buffer.from(`Bearer ${secret}`);+ const supplied = Buffer.from(typeof bearer === "string" ? bearer : "");+ if (supplied.length !== expected.length || !timingSafeEqual(supplied, expected)) {+ return sendJson(ctx.res, 401, { error: "unauthorized" });+ }+ if (ctx.method === "GET") return status(ctx, await control.store.get());+ const body = ctx.body;+ if (+ !isObj(body) ||+ !generation(body.expectedGeneration) ||+ typeof body.requestId !== "string" ||+ !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(body.requestId)+ ) {+ return sendJson(ctx.res, 400, { error: "invalid_background_request" });+ }+ try {+ if (body.terminatedMembers !== undefined) {+ if (+ !keys(body, ["expectedGeneration", "requestId", "terminatedMembers"]) ||+ !Array.isArray(body.terminatedMembers) ||+ body.terminatedMembers.length < 1 ||+ body.terminatedMembers.length > 1000 ||+ !body.terminatedMembers.every(+ (member) =>+ isObj(member) &&+ keys(member, ["instanceId", "taskArn", "generation"]) &&+ identity(member.instanceId) &&+ identity(member.taskArn) &&+ generation(member.generation),+ )+ ) {+ return sendJson(ctx.res, 400, { error: "invalid_background_retirement" });+ }+ const terminatedMembers = body.terminatedMembers.map((member) => ({+ instanceId: member.instanceId as string,+ taskArn: member.taskArn as string,+ generation: member.generation as number,+ }));+ return status(+ ctx,+ await control.store.retire({+ expectedGeneration: body.expectedGeneration,+ requestId: body.requestId,+ terminatedMembers,+ }),+ );+ }+ if (+ !keys(body, ["expectedGeneration", "requestId", "desiredDeploymentId", "bootstrapTaskArns"]) ||+ !(body.desiredDeploymentId === null || identity(body.desiredDeploymentId)) ||+ (body.bootstrapTaskArns !== undefined &&+ (!Array.isArray(body.bootstrapTaskArns) ||+ body.bootstrapTaskArns.length < 1 ||+ body.bootstrapTaskArns.length > 1000 ||+ !body.bootstrapTaskArns.every(identity)))+ ) {+ return sendJson(ctx.res, 400, { error: "invalid_background_transition" });+ }+ return status(+ ctx,+ await control.store.transition({+ expectedGeneration: body.expectedGeneration,+ requestId: body.requestId,+ desiredDeploymentId: body.desiredDeploymentId,+ ...(body.bootstrapTaskArns ? { bootstrapTaskArns: body.bootstrapTaskArns as string[] } : {}),+ }),+ );+ } catch (error) {+ if (error instanceof BackgroundOwnershipConflict) {+ return sendJson(ctx.res, 409, { error: "background_ownership_conflict", message: error.message });+ }+ throw error;+ }+}++export const backgroundWorkRoutes: ReadonlyArray<Route<ApiCtx>> = [+ { method: "GET", path: "/v1/background-work", auth: "source", handle: backgroundWork },+ { method: "POST", path: "/v1/background-work", auth: "source", handle: backgroundWork },+];diff --git a/src/api/routes/index.ts b/src/api/routes/index.tsindex 2bb8c191d..6407d4925 100644--- a/src/api/routes/index.ts+++ b/src/api/routes/index.ts@@ -1,3 +1,4 @@+import { backgroundWorkRoutes } from "./background-work.ts";import { composioRoutes } from "./composio.ts";import { sendJson } from "../http.ts";import { type ApiCtx, type BaseCtx, type Route } from "./route.ts";@@ -58,6 +59,7 @@ export const apiRoutes: ReadonlyArray<Route<ApiCtx>> = [...swarmRoutes,...searchRoutes,...deploymentLayerRoutes,+ ...backgroundWorkRoutes,...turnRoutes,...runEventRoutes,...credentialRoutes,diff --git a/src/api/user-scoped-routes.ts b/src/api/user-scoped-routes.tsindex b4ccf412c..d649a0ac6 100644--- a/src/api/user-scoped-routes.ts+++ b/src/api/user-scoped-routes.ts@@ -139,6 +139,7 @@ const SYSTEM: Rule[] = [pat("POST", "/v1/auth/broker/sessions"),pat("POST", "/v1/auth/broker/sessions/use"),pat("PUT", "/v1/deployment-layer"),+ pat("POST", "/v1/background-work"),pat("POST", "/v1/session-cap"),pat("POST", "/v1/keychain/drops/:id"),pat("POST", "/v1/keychain/asks"),diff --git a/src/config.ts b/src/config.tsindex 2aeade5dd..1c22a37ad 100644--- a/src/config.ts+++ b/src/config.ts@@ -1,3 +1,4 @@+import { isStrongSigningSecret } from "./auth/source-auth.ts";import { parseScopeId } from "./types.ts";import type { SandboxScopeDefaults } from "./sandbox/sandbox-routing.ts";import { existsSync, readdirSync } from "node:fs";@@ -113,6 +114,8 @@ export interface Config {backgroundJobTtlMs: number;backgroundJobTtlMaxMs: number;backgroundWorkEnabled: boolean;+ backgroundDeploymentId?: string;+ deploymentControlSecret?: string;buildSha?: string;ecsTaskProtection: boolean;ecsAgentUri?: string;@@ -1023,6 +1026,19 @@ function modelProviderEnvStrict(env: NodeJS.ProcessEnv): ModelProvider | undefin}export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {+ if (env.BACKGROUND_DEPLOYMENT_ID !== undefined) {+ if (!env.BACKGROUND_DEPLOYMENT_ID.trim() || env.BACKGROUND_DEPLOYMENT_ID.length > 256)+ throw new Error("BACKGROUND_DEPLOYMENT_ID must be nonempty and at most 256 characters");+ if (!env.DATABASE_URL) throw new Error("Background ownership requires DATABASE_URL");+ if (+ !isStrongSigningSecret(env.CORE_SIGNING_SECRET) ||+ !isStrongSigningSecret(env.DEPLOYMENT_CONTROL_SECRET) ||+ env.DEPLOYMENT_CONTROL_SECRET === env.CORE_SIGNING_SECRET+ )+ throw new Error(+ "Background ownership requires a distinct DEPLOYMENT_CONTROL_SECRET of at least 32 characters and CORE_SIGNING_SECRET",+ );+ }const swarmDefaults = resolveSwarmSettings(env.SWARM_DEFAULTS === undefined ? undefined : JSON.parse(env.SWARM_DEFAULTS),);@@ -1371,6 +1387,9 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {CONFIG_DEFAULTS.backgroundJobTtlMaxSec) * 1000,backgroundWorkEnabled:boolEnvStrict("BACKGROUND_WORK_ENABLED", env.BACKGROUND_WORK_ENABLED) ?? CONFIG_DEFAULTS.backgroundWorkEnabled,+ ...(env.BACKGROUND_DEPLOYMENT_ID+ ? { backgroundDeploymentId: env.BACKGROUND_DEPLOYMENT_ID, deploymentControlSecret: env.DEPLOYMENT_CONTROL_SECRET }+ : {}),...(env.GIT_SHA ? { buildSha: env.GIT_SHA } : {}),ecsTaskProtection: boolEnvStrict("ECS_TASK_PROTECTION", env.ECS_TASK_PROTECTION) ?? true,...(env.ECS_AGENT_URI ? { ecsAgentUri: env.ECS_AGENT_URI } : {}),diff --git a/src/cron/job-queue.ts b/src/cron/job-queue.tsindex b279d10bc..78b674574 100644--- a/src/cron/job-queue.ts+++ b/src/cron/job-queue.ts@@ -18,6 +18,7 @@ export interface CronJobQueue {start(handlers: CronQueueHandlers, tickIntervalMs: number): Promise<void>;enqueueFire(job: CronFireJob): Promise<void>;healthy(): boolean;+ stopClaims?(): Promise<void>;stop(): Promise<void>;}@@ -50,12 +51,17 @@ export function createPgBossCronQueue(boss.on("error", (e) => console.error("[cron-queue] pg-boss error:", errMessage(e)));let ticker: Sweeper | null = null;let started = false;+ let initialized = false;let lastSendOkAt = 0;return {async start(handlers, tickIntervalMs) {+ if (started) return;pg ??= createPgPool(databaseUrl, []);try {- await boss.start();+ if (!initialized) {+ await boss.start();+ initialized = true;+ }await boss.createQueue(FIRE_QUEUE, { policy: "short", notify: true });await boss.createQueue(TICK_QUEUE, { policy: "short", notify: true });const localConcurrency = Math.min(32, Math.max(1, Math.trunc(fireConcurrency)));@@ -68,8 +74,15 @@ export function createPgBossCronQueue();await boss.work(TICK_QUEUE, { pollingIntervalSeconds: 1 }, () => handlers.onTick());} catch (e) {- await boss.stop({ close: true, graceful: false }).catch(() => {});- await closePool();+ if (initialized) {+ await Promise.all([+ boss.offWork(FIRE_QUEUE, { wait: false }),+ boss.offWork(TICK_QUEUE, { wait: false }),+ ]).catch(() => {});+ } else {+ await boss.stop({ close: true, graceful: false }).catch(() => {});+ await closePool();+ }throw e;}started = true;@@ -97,12 +110,18 @@ export function createPgBossCronQueue(healthy() {return started && Date.now() - lastSendOkAt < HEALTHY_SEND_MAX_AGE_MS;},+ async stopClaims() {+ started = false;+ void ticker?.stop();+ await Promise.all([boss.offWork(FIRE_QUEUE, { wait: false }), boss.offWork(TICK_QUEUE, { wait: false })]);+ },async stop() {started = false;await ticker?.stop();try {await boss.stop({ close: true, graceful: false });} finally {+ initialized = false;await closePool();}},diff --git a/src/cron/scheduler.ts b/src/cron/scheduler.tsindex 270b9d800..d99c4921e 100644--- a/src/cron/scheduler.ts+++ b/src/cron/scheduler.ts@@ -59,6 +59,9 @@ export interface Scheduler {runNow(cronId: string): Promise<RunNowResult>;notifyChanged(cronId: string): void;start(intervalMs: number): void;+ ready(): Promise<void>;+ stopClaims(): Promise<void>;+ drained(): Promise<void>;stop(): Promise<void>;}@@ -78,6 +81,7 @@ export interface SchedulerDeps {};sweepAsks?: (now: number) => Promise<void>;jobQueue?: CronJobQueue;+ requireQueueStart?: boolean;sessions?: TriggerDeps["sessions"];fireLoop?: (loopId: string, fireKey: string) => Promise<{ status?: TurnResult["status"]; note?: string }>;}@@ -324,7 +328,7 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {}};- const fireDue = async (t: number): Promise<void> => {+ const fireDue = async (t: number, observed: number): Promise<void> => {const due = await deps.crons.due(t);let batch = due;if (due.length > maxFiresPerTick) {@@ -342,6 +346,7 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {console.warn(`[scheduler] fan-out capped: firing ${batch.length}/${due.length} due crons this tick`);}for (const cron of batch) {+ if (stopped || observed !== epoch) break;try {const { authzFailed, deferred } = await fire(cron, t, `cron:${cron.id}:${cron.scheduledAt}`, cron.scheduledAt);if (!authzFailed && !deferred) await deps.crons.markFired(cron.id, t, cron.scheduledAt);@@ -353,19 +358,21 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {const tick = async (nowArg?: number): Promise<void> => {const t = nowArg ?? now();+ const observed = epoch;await leaderLease.hold(TICK_LEASE_KEY, async () => {- await fireDue(t);+ await fireDue(t, observed);await sweepStranded(t);await gcFires(t);await deps.sweepAsks?.(t).catch((e: unknown) => console.error("[scheduler] ask sweep failed:", errMessage(e)));});};- const sweeper = createSweeper(- () => tick().catch((e: unknown) => console.error("[scheduler] tick failed:", errMessage(e))),- 1000,- { label: "scheduler" },- );+ const makeSweeper = () =>+ createSweeper(() => tick().catch((e: unknown) => console.error("[scheduler] tick failed:", errMessage(e))), 1000, {+ label: "scheduler",+ });++ let sweeper = makeSweeper();function nextSlot(cron: Cron): number | undefined {return recoverNextFireAt(cron.schedule, cron.createdAt, cron.lastFiredAt, cron.nextFireAt);@@ -389,6 +396,7 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {}async function fireJob(job: CronFireJob): Promise<void> {+ const observed = epoch;const cron = await deps.crons.get(job.cronId);if (!cron || cron.archived || !cron.enabled) return;const slot = nextSlot(cron);@@ -402,7 +410,12 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {await deps.jobQueue!.enqueueFire({ ...job, notBefore: cron.deferUntil! });return;}+ if (stopped || observed !== epoch) return;if (!(await deps.crons.claimSlot(job.cronId, slot, t))) return;+ if (stopped || observed !== epoch) {+ await deps.crons.unclaimSlot(job.cronId, slot, t, cron.lastFiredAt);+ return;+ }try {const { authzFailed, deferred } = await fire(cron, t, `cron:${cron.id}:${slot}`, slot);if (authzFailed || deferred) {@@ -421,6 +434,7 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {async function reconcile(): Promise<void> {try {for (const cron of await deps.crons.list()) {+ if (stopped) break;if (cron.archived || !cron.enabled) continue;const job = nextJob(cron);if (job) await deps.jobQueue!.enqueueFire(job);@@ -449,10 +463,17 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {);let stopped = false;+ let epoch = 0;let started = false;let stopFailed = false;let starting: Promise<void> | null = null;let stopping: Promise<void> | null = null;+ let pausing: Promise<void> | null = null;+ const oldSweeps = new Set<Promise<void>>();+ const retireSweep = (work: Promise<void>) => {+ oldSweeps.add(work);+ void work.finally(() => oldSweeps.delete(work));+ };const pending = new Set<Promise<void>>();async function runQueueTask(work: () => Promise<void>): Promise<void> {if (stopped) return;@@ -492,7 +513,7 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {void enqueueNext(cronId).catch((e: unknown) => console.error("[scheduler] cron enqueue failed:", errMessage(e)));},start(intervalMs) {- if (started || stopping || stopFailed) return;+ if (started || stopping || pausing || stopFailed) return;started = true;stopped = false;stopSignal = Promise.withResolvers<void>();@@ -500,11 +521,12 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {sweeper.start(intervalMs);return;}+ const observed = epoch;starting = deps.jobQueue.start({- onFire: (job) => runQueueTask(() => fireJob(job)),- onTick: () => runQueueTask(reconcile),+ onFire: (job) => (observed === epoch ? runQueueTask(() => fireJob(job)) : Promise.resolve()),+ onTick: () => (observed === epoch ? runQueueTask(reconcile) : Promise.resolve()),},intervalMs,)@@ -513,22 +535,58 @@ export function createScheduler(deps: SchedulerDeps): Scheduler {if (!stopped) leaseGuard.start();},(e: unknown) => {+ if (deps.requireQueueStart) throw e;console.error("[scheduler] cron job queue failed to start; falling back to interval ticks:", errMessage(e));if (!stopped) sweeper.start(intervalMs);},);},+ async ready() {+ await starting;+ },+ stopClaims() {+ if (pausing) return pausing;+ stopped = true;+ started = false;+ epoch++;+ stopSignal.resolve();+ retireSweep(sweeper.stop());+ sweeper = makeSweeper();+ const guard = leaseGuard.stop();+ pausing = (async () => {+ await starting?.catch(() => {});+ if (deps.jobQueue?.stopClaims) await deps.jobQueue.stopClaims();+ else await deps.jobQueue?.stop();+ await guard;+ stopFailed = false;+ })()+ .catch((error) => {+ stopFailed = true;+ throw error;+ })+ .finally(() => {+ pausing = null;+ });+ return pausing;+ },+ async drained() {+ await Promise.all([...oldSweeps, ...pending]);+ },stop() {if (stopping) return stopping;stopped = true;+ epoch++;stopSignal.resolve();started = false;const sweeps = [sweeper.stop(), leaseGuard.stop()];stopping = (async () => {- await starting;- await deps.jobQueue?.stop();+ await starting?.catch(() => {});+ if (deps.jobQueue?.stopClaims) await deps.jobQueue.stopClaims();+ await pausing;await Promise.all(sweeps);+ await Promise.all(oldSweeps);await Promise.allSettled(pending);+ await deps.jobQueue?.stop();stopFailed = false;})().catch((error: unknown) => {diff --git a/src/index.ts b/src/index.tsindex 335ab20e9..6482da0b5 100644--- a/src/index.ts+++ b/src/index.ts@@ -1,3 +1,5 @@+import { createBackgroundController } from "./runs/background-controller.ts";+import { backgroundTaskArn } from "./runs/background-task-identity.ts";import { createManagedSlack } from "./surfaces/slack-managed.ts";import { randomBytes } from "node:crypto";import { lookup } from "node:dns/promises";@@ -30,7 +32,8 @@ const managedSlack = process.env.QM_SLACK_SERVICE_URLtoken: process.env.QM_SLACK_SERVICE_TOKEN ?? "",appId: process.env.QM_SLACK_APP_ID ?? "",store: built.slackInstallation,- reconcile: config.backgroundWorkEnabled ? () => slackRuntime.reconcile() : undefined,+ reconcile:+ config.backgroundWorkEnabled || config.backgroundDeploymentId ? () => slackRuntime.reconcile() : undefined,}): undefined;const server = createServer(built.app, {@@ -78,14 +81,15 @@ if (config.deployProvider === "docker") {});}-if (config.backgroundWorkEnabled) {+if (config.backgroundWorkEnabled && !config.backgroundDeploymentId) {built.scheduler.start(1000);built.suggestedActivityMaintenance.start();-} else {+} else if (!config.backgroundDeploymentId) {console.log("[qm] background work disabled; scheduler and runtime loops will not start");}const slackRuntime = createSlackRuntimeReconciler({+ startPaused: Boolean(config.backgroundDeploymentId),load: async () => {const status = await built.slackInstallation.status();const stored = await built.slackInstallation.get();@@ -110,17 +114,76 @@ const slackRuntime = createSlackRuntimeReconciler({startPlugin: (desired) => startSlackPlugin(desired, built.slackCore),onError: (error) => console.error(`[qm] slack plugin reconciliation failed: ${errMessage(error)}`),});-if (config.backgroundWorkEnabled) slackRuntime.start();+if (config.backgroundWorkEnabled && !config.backgroundDeploymentId) slackRuntime.start();const slackAccountRuntimes = slackAccountConfigsFromEnv(process.env).map((account) =>createSlackRuntimeReconciler({+ startPaused: Boolean(config.backgroundDeploymentId),load: () => Promise.resolve({ version: `environment:${account.accountId}`, config: account }),startPlugin: (desired) => startSlackPlugin(desired, built.slackCore),onError: (error) =>console.error(`[qm] slack account "${account.accountId}" reconciliation failed: ${errMessage(error)}`),}),);-if (config.backgroundWorkEnabled) for (const runtime of slackAccountRuntimes) runtime.start();+if (config.backgroundWorkEnabled && !config.backgroundDeploymentId)+ for (const runtime of slackAccountRuntimes) runtime.start();++let backgroundController: ReturnType<typeof createBackgroundController> | undefined;+if (built.backgroundOwnership) {+ const identity = {+ ...built.backgroundOwnership,+ taskArn: await backgroundTaskArn(process.env.ECS_CONTAINER_METADATA_URI_V4),+ };+ let periodicStop: Promise<void> = Promise.resolve();+ let activationEpoch = 0;+ const stopPeriodic = () => {+ activationEpoch++;+ periodicStop = Promise.all([built.scheduler.stopClaims(), built.suggestedActivityMaintenance.stop()]).then(+ () => {},+ );+ void periodicStop.catch((error) => console.error("[qm] periodic background stop failed:", errMessage(error)));+ void built.runtime+ .stopBackgroundClaims()+ .catch((error) => console.error("[qm] background claim stop failed:", errMessage(error)));+ for (const runtime of [slackRuntime, ...slackAccountRuntimes])+ void runtime.stop().catch((error) => console.error("[qm] Slack background stop failed:", errMessage(error)));+ };+ backgroundController = createBackgroundController({+ store: identity.store,+ identity: { deploymentId: identity.deploymentId, instanceId: identity.instanceId, taskArn: identity.taskArn },+ legacyEnabled: config.backgroundWorkEnabled,+ async start(signal) {+ const epoch = ++activationEpoch;+ if (signal.aborted) return;+ built.runtime.startBackground();+ await periodicStop;+ if (signal.aborted || epoch !== activationEpoch) return;+ built.scheduler.start(1000);+ await built.scheduler.ready();+ if (signal.aborted || epoch !== activationEpoch) return;+ built.suggestedActivityMaintenance.start();+ for (const runtime of [slackRuntime, ...slackAccountRuntimes]) {+ if (signal.aborted) return;+ runtime.start();+ await runtime.reconcile();+ }+ },+ fence: stopPeriodic,+ async relinquish() {+ await Promise.all([+ built.runtime.stopBackgroundClaims(),+ built.scheduler.stopClaims(),+ ...[slackRuntime, ...slackAccountRuntimes].map((runtime) => runtime.stop()),+ ]);+ },+ async drained() {+ await Promise.all([built.runtime.backgroundDrained(), built.scheduler.drained(), periodicStop]);+ },+ onError: (error) => console.error("[qm] background ownership failed:", errMessage(error)),+ });+ built.runtime.setBackgroundAdmission(backgroundController.canClaim);+ backgroundController.start();+}let shuttingDown = false;function shutdown(signal: string): void {@@ -135,7 +198,18 @@ function shutdown(signal: string): void {built.deploymentLayerRefresh.stop();server.close();server.closeIdleConnections();- stopWithBackstop(built.runtime, config.shutdownDrainMs, "qm", () => server.closeAllConnections());+ stopWithBackstop(+ {+ async stop() {+ await backgroundController?.stop();+ await built.runtime.stop();+ },+ releaseInFlightRuns: () => built.runtime.releaseInFlightRuns(),+ },+ config.shutdownDrainMs,+ "qm",+ () => server.closeAllConnections(),+ );}process.on("SIGINT", () => shutdown("SIGINT"));process.on("SIGTERM", () => shutdown("SIGTERM"));diff --git a/src/monitors/monitor-poller.ts b/src/monitors/monitor-poller.tsindex 865f9c55b..639d4f020 100644--- a/src/monitors/monitor-poller.ts+++ b/src/monitors/monitor-poller.ts@@ -121,6 +121,8 @@ function renderEvent(m: Monitor, output: string, ev: MonitorEvent): { input: str}export function createMonitorPoller(deps: MonitorPollerDeps): MonitorPoller {+ let stopped = false;+ let epoch = 0;const now = deps.now ?? (() => Date.now());const maxFiresPerTick = deps.maxFiresPerTick ?? 20;const heartbeatMs = deps.heartbeatMs ?? DEFAULT_HEARTBEAT_MS;@@ -251,7 +253,7 @@ export function createMonitorPoller(deps: MonitorPollerDeps): MonitorPoller {return true;}- async function pollAll(t: number): Promise<void> {+ async function pollAll(t: number, observed: number): Promise<void> {if (!supportsProcessSessions(deps.sandbox)) return;const sandbox = deps.sandbox;const enabled = (await deps.monitors.enabled()).sort((a, b) => a.createdAt - b.createdAt);@@ -261,6 +263,7 @@ export function createMonitorPoller(deps: MonitorPollerDeps): MonitorPoller {let fires = 0;try {for (const m of enabled) {+ if (stopped || observed !== epoch) break;if (fires >= maxFiresPerTick) {console.warn(`[monitor] fan-out capped: fired ${fires}/${enabled.length} watched jobs this tick`);break;@@ -303,7 +306,8 @@ export function createMonitorPoller(deps: MonitorPollerDeps): MonitorPoller {const tick = async (nowArg?: number): Promise<void> => {const t = nowArg ?? now();- await leaderLease.hold(TICK_LEASE_KEY, () => pollAll(t));+ const observed = epoch;+ await leaderLease.hold(TICK_LEASE_KEY, () => pollAll(t, observed));};const sweeper = createSweeper(@@ -313,7 +317,14 @@ export function createMonitorPoller(deps: MonitorPollerDeps): MonitorPoller {);return {tick,- start: sweeper.start,- stop: sweeper.stop,+ start(intervalMs) {+ stopped = false;+ sweeper.start(intervalMs);+ },+ stop() {+ stopped = true;+ epoch++;+ return sweeper.stop();+ },};}diff --git a/src/runs/background-controller.ts b/src/runs/background-controller.tsnew file mode 100644index 000000000..897bb14b1--- /dev/null+++ b/src/runs/background-controller.ts@@ -0,0 +1,125 @@+import type { BackgroundMember, BackgroundOwnershipStore } from "./background-ownership.ts";+import { BackgroundOwnershipConflict } from "./background-ownership.ts";++export interface BackgroundControllerDeps {+ store: BackgroundOwnershipStore;+ identity: Pick<BackgroundMember, "instanceId" | "deploymentId" | "taskArn">;+ legacyEnabled: boolean;+ start(signal: AbortSignal): Promise<void>;+ fence(): void;+ relinquish(): Promise<void>;+ drained(): Promise<void>;+ onError(error: unknown): void;+ validityMs?: number;+ pollMs?: number;+}++export function createBackgroundController(deps: BackgroundControllerDeps) {+ let running = false;+ let registered = false;+ let admission: number | null = null;+ let admissionEpoch = 0;+ let activation: AbortController | null = null;+ let validUntil = 0;+ let watchdog: ReturnType<typeof setTimeout> | null = null;+ let poller: ReturnType<typeof setInterval> | null = null;+ let pending: Promise<void> | null = null;+ let draining: Promise<void> = Promise.resolve();+ const validityMs = deps.validityMs ?? 10_000;+ const fence = (): void => {+ validUntil = 0;+ activation?.abort();+ deps.fence();+ if (watchdog) clearTimeout(watchdog);+ watchdog = null;+ };+ const release = async (): Promise<void> => {+ fence();+ if (admission === null) return;+ await deps.relinquish();+ const generation = admission;+ const epoch = admissionEpoch;+ await deps.store.acknowledge(deps.identity.instanceId, generation, "relinquished");+ admission = null;+ activation = null;+ draining = deps.drained().then(async () => {+ if (admission === null && epoch === admissionEpoch)+ await deps.store.acknowledge(deps.identity.instanceId, generation, "drained");+ });+ void draining.catch(deps.onError);+ };+ const renew = (): void => {+ validUntil = Date.now() + validityMs;+ if (watchdog) clearTimeout(watchdog);+ watchdog = setTimeout(fence, validityMs);+ watchdog.unref?.();+ };+ const reconcile = (): Promise<void> => {+ if (pending) return pending;+ pending = (async () => {+ try {+ if (!registered) {+ await deps.store.register(deps.identity);+ registered = true;+ }+ const state = await deps.store.get();+ const member = state.members.find((entry) => entry.instanceId === deps.identity.instanceId);+ const desired =+ running &&+ member !== undefined &&+ !member.retired &&+ (state.enabled ? state.desiredDeploymentId === deps.identity.deploymentId : deps.legacyEnabled);+ if (admission !== null && (!desired || state.generation !== admission || activation?.signal.aborted))+ await release();+ if (!desired) return;+ if (admission === null) {+ await deps.store.admit(deps.identity.instanceId, state.generation, deps.legacyEnabled);+ admission = state.generation;+ admissionEpoch++;+ activation = new AbortController();+ renew();+ if (!running) {+ await release();+ return;+ }+ await deps.start(activation.signal);+ if (activation.signal.aborted || !running) await release();+ else await deps.store.markReady(deps.identity.instanceId, state.generation);+ } else {+ renew();+ }+ } catch (error) {+ fence();+ try {+ await release();+ } catch (releaseError) {+ deps.onError(releaseError);+ }+ if (!(error instanceof BackgroundOwnershipConflict)) deps.onError(error);+ }+ })().finally(() => {+ pending = null;+ });+ return pending;+ };+ return {+ canClaim: () => running && admission !== null && !activation?.signal.aborted && Date.now() < validUntil,+ reconcile,+ start() {+ if (running) return;+ running = true;+ poller = setInterval(() => void reconcile(), deps.pollMs ?? 1_000);+ poller.unref?.();+ void reconcile();+ },+ async stop() {+ running = false;+ fence();+ if (poller) clearInterval(poller);+ poller = null;+ await pending;+ await release();+ },+ drained: () => draining,+ };+}diff --git a/src/runs/background-ownership.ts b/src/runs/background-ownership.tsnew file mode 100644index 000000000..0dedd5ba9--- /dev/null+++ b/src/runs/background-ownership.ts@@ -0,0 +1,207 @@+import type { DurableMap } from "../persistence/durable-map.ts";++export interface BackgroundMember {+ instanceId: string;+ deploymentId: string;+ taskArn: string | null;+ generation: number;+ state: "admitted" | "relinquished" | "drained";+ retired: boolean;+ ready: boolean;+}++export interface BackgroundOwnership {+ enabled: boolean;+ generation: number;+ desiredDeploymentId: string | null;+ lastRequestId: string | null;+ lastRequest: string | null;+ members: BackgroundMember[];+}++export interface BackgroundTransition {+ expectedGeneration: number;+ requestId: string;+ desiredDeploymentId: string | null;+ bootstrapTaskArns?: string[];+}++export class BackgroundOwnershipConflict extends Error {}++export function createBackgroundOwnershipStore(map: DurableMap<BackgroundOwnership>) {+ if (!map.update) throw new Error("Background ownership requires atomic updates");+ const key = "ownership";+ const initial: BackgroundOwnership = {+ enabled: false,+ generation: 0,+ desiredDeploymentId: null,+ lastRequestId: null,+ lastRequest: null,+ members: [],+ };+ const ready = () => map.putIfAbsent(key, structuredClone(initial));+ const update = async (change: (state: BackgroundOwnership) => void): Promise<BackgroundOwnership> => {+ await ready();+ const result = await map.update!(key, (current) => {+ const next = structuredClone(current);+ change(next);+ return next;+ });+ if (!result) throw new Error("Background ownership disappeared");+ return result;+ };+ const memberOf = (state: BackgroundOwnership, instanceId: string): BackgroundMember => {+ const member = state.members.find((entry) => entry.instanceId === instanceId);+ if (!member || member.retired) throw new BackgroundOwnershipConflict("Instance is missing or retired");+ return member;+ };+ const expectGeneration = (state: BackgroundOwnership, expected: number): void => {+ if (!Number.isSafeInteger(expected) || expected < 0 || state.generation !== expected)+ throw new BackgroundOwnershipConflict("Background ownership generation changed");+ };+ return {+ async get(): Promise<BackgroundOwnership> {+ return structuredClone((await map.get(key)) ?? (await ready()));+ },+ register(identity: Pick<BackgroundMember, "instanceId" | "deploymentId" | "taskArn">) {+ return update((state) => {+ if (identity.taskArn && state.members.some((member) => member.taskArn === identity.taskArn && member.retired))+ throw new BackgroundOwnershipConflict("Task has already been retired");+ const previous = state.members.find((member) => member.instanceId === identity.instanceId);+ if (previous) {+ if (+ previous.deploymentId !== identity.deploymentId ||+ previous.taskArn !== identity.taskArn ||+ previous.retired+ )+ throw new BackgroundOwnershipConflict("Instance identity cannot change");+ return;+ }+ state.members.push({+ ...identity,+ generation: state.generation,+ state: "drained",+ retired: false,+ ready: false,+ });+ });+ },+ transition(request: BackgroundTransition) {+ const fingerprint = JSON.stringify([+ "transition",+ request.expectedGeneration,+ request.desiredDeploymentId,+ request.bootstrapTaskArns?.slice().sort() ?? null,+ ]);+ return update((state) => {+ if (state.lastRequestId === request.requestId) {+ if (+ state.lastRequest !== fingerprint ||+ state.generation !== request.expectedGeneration + 1 ||+ state.desiredDeploymentId !== request.desiredDeploymentId+ )+ throw new BackgroundOwnershipConflict("Request identity reused for a different transition");+ return;+ }+ expectGeneration(state, request.expectedGeneration);+ if (!state.enabled) {+ const expected = request.bootstrapTaskArns;+ if (!expected?.length || new Set(expected).size !== expected.length)+ throw new BackgroundOwnershipConflict("Bootstrap requires an exact nonempty task cohort");+ const enrolled = state.members.filter((member) => !member.retired);+ if (+ enrolled.some((member) => !member.taskArn || !expected.includes(member.taskArn)) ||+ expected.some((taskArn) => !enrolled.some((member) => member.taskArn === taskArn))+ )+ throw new BackgroundOwnershipConflict("Bootstrap task cohort does not match enrolled instances");+ } else if (request.bootstrapTaskArns) {+ throw new BackgroundOwnershipConflict("Ownership is already enabled");+ }+ if (+ request.desiredDeploymentId !== null &&+ !state.members.some((member) => !member.retired && member.deploymentId === request.desiredDeploymentId)+ )+ throw new BackgroundOwnershipConflict("Desired deployment has no enrolled instances");+ state.enabled = true;+ state.generation++;+ state.desiredDeploymentId = request.desiredDeploymentId;+ state.lastRequestId = request.requestId;+ state.lastRequest = fingerprint;+ });+ },+ admit(instanceId: string, expectedGeneration: number, legacyEnabled: boolean) {+ return update((state) => {+ expectGeneration(state, expectedGeneration);+ const member = memberOf(state, instanceId);+ if (state.enabled) {+ if (state.desiredDeploymentId !== member.deploymentId)+ throw new BackgroundOwnershipConflict("Deployment is not the desired owner");+ if (state.members.some((other) => other.state === "admitted" && other.generation !== state.generation))+ throw new BackgroundOwnershipConflict("Previous owners have not relinquished work");+ } else if (!legacyEnabled) {+ throw new BackgroundOwnershipConflict("Legacy background work is disabled");+ }+ member.generation = state.generation;+ member.state = "admitted";+ member.ready = false;+ });+ },+ markReady(instanceId: string, generation: number) {+ return update((state) => {+ expectGeneration(state, generation);+ const member = memberOf(state, instanceId);+ if (+ member.generation !== generation ||+ member.state !== "admitted" ||+ (state.enabled && state.desiredDeploymentId !== member.deploymentId)+ )+ throw new BackgroundOwnershipConflict("Instance admission changed");+ member.ready = true;+ });+ },+ acknowledge(instanceId: string, generation: number, next: "relinquished" | "drained") {+ return update((state) => {+ const member = memberOf(state, instanceId);+ if (member.generation !== generation) throw new BackgroundOwnershipConflict("Instance admission changed");+ if (member.state === "drained" || member.state === next) return;+ if (next === "drained" && member.state !== "relinquished")+ throw new BackgroundOwnershipConflict("Instance must relinquish before reporting drained");+ member.state = next;+ member.ready = false;+ });+ },+ retire(request: {+ expectedGeneration: number;+ requestId: string;+ terminatedMembers: Array<Pick<BackgroundMember, "instanceId" | "taskArn" | "generation">>;+ }) {+ const fingerprint = JSON.stringify([+ "retire",+ request.expectedGeneration,+ request.terminatedMembers.map((member) => [member.instanceId, member.taskArn, member.generation]).sort(),+ ]);+ return update((state) => {+ expectGeneration(state, request.expectedGeneration);+ if (state.lastRequestId === request.requestId) {+ if (state.lastRequest !== fingerprint)+ throw new BackgroundOwnershipConflict("Request identity reused for a different retirement");+ return;+ }+ if (!request.terminatedMembers.length)+ throw new BackgroundOwnershipConflict("Retirement requires exact terminated members");+ for (const proof of request.terminatedMembers) {+ const member = state.members.find((entry) => entry.instanceId === proof.instanceId);+ if (!member || !proof.taskArn || member.taskArn !== proof.taskArn || member.generation !== proof.generation)+ throw new BackgroundOwnershipConflict("Termination evidence does not match the enrolled instance");+ member.state = "drained";+ member.retired = true;+ member.ready = false;+ }+ state.lastRequestId = request.requestId;+ state.lastRequest = fingerprint;+ });+ },+ };+}++export type BackgroundOwnershipStore = ReturnType<typeof createBackgroundOwnershipStore>;diff --git a/src/runs/background-task-identity.ts b/src/runs/background-task-identity.tsnew file mode 100644index 000000000..24b63ff96--- /dev/null+++ b/src/runs/background-task-identity.ts@@ -0,0 +1,9 @@+export async function backgroundTaskArn(metadataUri: string | undefined): Promise<string | null> {+ if (!metadataUri) return null;+ const response = await fetch(`${metadataUri.replace(/\/$/, "")}/task`, { signal: AbortSignal.timeout(5_000) });+ if (!response.ok) throw new Error(`Task metadata returned ${response.status}`);+ const data = (await response.json()) as { TaskARN?: unknown };+ if (typeof data.TaskARN !== "string" || !data.TaskARN.startsWith("arn:") || !data.TaskARN.includes(":task/"))+ throw new Error("Task metadata did not contain a task ARN");+ return data.TaskARN;+}diff --git a/src/runs/worker.ts b/src/runs/worker.tsindex 4c7ddc9e3..1b7b84526 100644--- a/src/runs/worker.ts+++ b/src/runs/worker.ts@@ -188,7 +188,7 @@ export function createWorker(deps: WorkerDeps): Worker {await waitForWork(observed);continue;}- if (stopped) {+ if (stopped || (deps.canClaim && !deps.canClaim())) {if (run.leaseToken !== null)await deps.runs.releaseLease(run.id, run.leaseToken)diff --git a/src/surfaces/slack-runtime.ts b/src/surfaces/slack-runtime.tsindex 12058b7be..0876ad0dc 100644--- a/src/surfaces/slack-runtime.ts+++ b/src/surfaces/slack-runtime.ts@@ -16,13 +16,14 @@ export function createSlackRuntimeReconciler<Config>(opts: {load: () => Promise<ReloadableSlackConfig<Config> | null>;startPlugin: (config: Config) => Promise<{ stop(): Promise<void> }>;intervalMs?: number;+ startPaused?: boolean;onError?: (error: unknown) => void;}) {let active: { plugin: { stop(): Promise<void> }; version: string; config: Config } | null = null;let timer: NodeJS.Timeout | null = null;let inFlight: Promise<void> | null = null;let stopping: Promise<void> | null = null;- let stopped = false;+ let stopped = opts.startPaused ?? false;let pendingCleanup: (() => Promise<void>) | null = null;const reconcile = async (): Promise<void> => {diff --git a/src/wiring.ts b/src/wiring.tsindex 8c576602c..18c015316 100644--- a/src/wiring.ts+++ b/src/wiring.ts@@ -1,3 +1,8 @@+import {+ createBackgroundOwnershipStore,+ type BackgroundOwnershipStore,+ type BackgroundOwnership,+} from "./runs/background-ownership.ts";import { loadConnectorSdk } from "./sandbox/connector-sdk.ts";import { createFlyTunnelManager, parseFlyWireguardPeers } from "./deploy/fly-tunnel-manager.ts";import type { FlyPeerClaim } from "./deploy/fly-peer-claims.ts";@@ -385,6 +390,8 @@ import { createSlackInstallationStore, type SlackInstallationStore } from "./surexport interface Runtime {start(): void;startBackground(): void;+ stopBackgroundClaims(): Promise<void>;+ setBackgroundAdmission(check: () => boolean): void;stopBackground(): Promise<void>;backgroundDrained(): Promise<void>;stop(): Promise<void>;@@ -417,6 +424,7 @@ export function stopWithBackstop(}export interface BuiltApp {+ backgroundOwnership?: { store: BackgroundOwnershipStore; instanceId: string; deploymentId: string };suggestedActivityMaintenance: Sweeper;suggestedActivities?: ReturnType<typeof createSuggestedActivityService>;app: App;@@ -2109,6 +2117,7 @@ export function buildApp(const sweepAsks =keychain && askResolution ? createAskExpirySweep({ keychain, fire: askResolution, auditLog }) : undefined;const scheduler = createScheduler({+ requireQueueStart: Boolean(config.backgroundDeploymentId),crons,deliveries,idempotency,@@ -2187,8 +2196,16 @@ export function buildApp(directory,currentScopeMembers,});+ let backgroundAdmission = () => !config.backgroundDeploymentId;+ const backgroundOwnership = config.backgroundDeploymentId+ ? {+ store: createBackgroundOwnershipStore(artifactMap<BackgroundOwnership>("background_ownership")),+ instanceId: randomUUID(),+ deploymentId: config.backgroundDeploymentId,+ }+ : undefined;const instanceRegistry: InstanceRegistry =- config.buildSha && pgArtifactMap+ config.buildSha && pgArtifactMap && !config.backgroundDeploymentId? createPostgresInstanceRegistry(pgArtifactMap.pool, {instanceId: randomUUID(),buildSha: config.buildSha,@@ -2211,7 +2228,7 @@ export function buildApp(heartbeatIntervalMs: config.heartbeatIntervalMs,errors,pollMs: 250,- canClaim: () => drain.canClaim(),+ canClaim: () => backgroundAdmission() && drain.canClaim(),onClaimed: () => drain.noteBusy(),}),);@@ -2261,9 +2278,11 @@ export function buildApp(: null;let backgroundRunning = false;let backgroundStopping: Promise<void> | null = null;+ let backgroundClaimsStopping: Promise<void> = Promise.resolve();+ let monitorDrained: Promise<void> = Promise.resolve();let backgroundGeneration = 0;function startBackground(): void {- if (backgroundRunning || backgroundStopping) return;+ if (backgroundRunning) return;backgroundRunning = true;drain.start();const generation = ++backgroundGeneration;@@ -2276,29 +2295,41 @@ export function buildApp(}).catch(swallowAs("wiring: worker resume failed", undefined));}- reaper.start();- processReaper?.start();- monitorPoller?.start(config.monitorPollMs);- monitorRetentionSweeper.start();- if (config.skillSyncPollMs > 0) skillSyncEngine.start(config.skillSyncPollMs);- blobSweeper.start();- fileUploads?.start();- idleSweeper?.start();- keepWarmSweeper.start();- deepIdleSweeper?.start();- wakeSweep.start();- swarms?.start();- orphanedSignalSweeper.start();- sessionReturnSweeper.start();+ const startPeriodic = () => {+ if (!backgroundRunning || generation !== backgroundGeneration) return;+ reaper.start();+ processReaper?.start();+ monitorPoller?.start(config.monitorPollMs);+ void monitorDrained+ .then(() => {+ if (backgroundRunning && generation === backgroundGeneration) monitorPoller?.start(config.monitorPollMs);+ })+ .catch(swallowAs("wiring: monitor resume failed", undefined));+ monitorRetentionSweeper.start();+ if (config.skillSyncPollMs > 0) skillSyncEngine.start(config.skillSyncPollMs);+ blobSweeper.start();+ fileUploads?.start();+ idleSweeper?.start();+ keepWarmSweeper.start();+ deepIdleSweeper?.start();+ wakeSweep.start();+ swarms?.start();+ orphanedSignalSweeper.start();+ sessionReturnSweeper.start();+ };+ if (backgroundStopping)+ void backgroundClaimsStopping.then(startPeriodic).catch(swallowAs("wiring: periodic resume failed", undefined));+ else startPeriodic();}function stopBackground(): Promise<void> {- if (backgroundStopping) return backgroundStopping;backgroundRunning = false;+ const previous = backgroundStopping;backgroundGeneration++;+ const monitorStopping = monitorPoller?.stop();+ monitorDrained = monitorStopping ?? Promise.resolve();const stopping = [reaper.stop(),processReaper?.stop(),- monitorPoller?.stop(),monitorRetentionSweeper.stop(),skillSyncEngine.stop(),idleSweeper?.stop(),@@ -2312,19 +2343,28 @@ export function buildApp(sessionReturnSweeper.stop(),...workers.map((worker) => worker.stopClaims()),];- backgroundStopping = Promise.all(stopping)+ backgroundClaimsStopping = Promise.all(stopping).then(() => {});+ const draining = Promise.all([previous, backgroundClaimsStopping, monitorStopping]).then(() => {}).finally(() => {- backgroundStopping = null;+ if (backgroundStopping === draining) backgroundStopping = null;});- return backgroundStopping;+ backgroundStopping = draining;+ return draining;}const runtime: Runtime = {start() {flyTunnel?.monitor();- if (config.backgroundWorkEnabled) startBackground();+ if (config.backgroundWorkEnabled && !config.backgroundDeploymentId) startBackground();},startBackground,+ setBackgroundAdmission(check) {+ backgroundAdmission = check;+ },+ async stopBackgroundClaims() {+ void stopBackground().catch(swallowAs("wiring: background drain failed", undefined));+ await Promise.all([backgroundClaimsStopping, ...workers.map((worker) => worker.stopClaims())]);+ },stopBackground,async backgroundDrained() {await backgroundStopping;@@ -2432,7 +2472,10 @@ export function buildApp(...(ambientJudgments ? { ambientJudgments } : {}),...(ackEmojiPicks ? { ackEmojiPicks } : {}),channelPolicy,- ...(config.suggestedActivitiesEnabled && config.backgroundWorkEnabled ? { suggestedActivities } : {}),+ ...(config.suggestedActivitiesEnabled && (config.backgroundWorkEnabled || config.backgroundDeploymentId)+ ? { suggestedActivities }+ : {}),+ ...(backgroundOwnership ? { backgroundOwnership } : {}),suggestedActivityMaintenance,uiState: artifactMap<PersistedUiState>("web_ui_state"),sessionShares: artifactMap<SessionShare>("session_shares"),@@ -2459,6 +2502,9 @@ export function serverDeps(const carriedModelAuth = harnessCarriedModelAuth(config);return {production: config.production,+ ...(built.backgroundOwnership+ ? { backgroundOwnership: built.backgroundOwnership, deploymentControlSecret: config.deploymentControlSecret }+ : {}),allowUnauthenticatedCore: config.allowUnauthenticatedCore,...(config.signingSecret ? { signingSecret: config.signingSecret } : {}),...(config.capabilitySecret ? { capabilitySecret: config.capabilitySecret } : {}),diff --git a/test/background-controller.test.ts b/test/background-controller.test.tsnew file mode 100644index 000000000..7e2e53871--- /dev/null+++ b/test/background-controller.test.ts@@ -0,0 +1,209 @@+import assert from "node:assert/strict";+import test from "node:test";+import { createMemoryMap } from "../src/persistence/durable-map.ts";+import { createBackgroundOwnershipStore, type BackgroundOwnership } from "../src/runs/background-ownership.ts";+import { createBackgroundController } from "../src/runs/background-controller.ts";++async function setup() {+ const store = createBackgroundOwnershipStore(createMemoryMap<BackgroundOwnership>());+ const errors: unknown[] = [];+ function replica(instanceId: string, deploymentId: string, legacyEnabled = false) {+ let starts = 0;+ let stops = 0;+ let failStart = false;+ const drain = Promise.withResolvers<void>();+ const controller = createBackgroundController({+ store,+ identity: { instanceId, deploymentId, taskArn: `task:${instanceId}` },+ legacyEnabled,+ start: async () => {+ starts++;+ if (failStart) throw new Error("activation failed");+ },+ fence() {},+ relinquish: async () => {+ stops++;+ },+ drained: () => drain.promise,+ onError: (error) => errors.push(error),+ pollMs: 60_000,+ validityMs: 60_000,+ });+ return {+ controller,+ drain,+ starts: () => starts,+ stops: () => stops,+ fail: () => {+ failStart = true;+ },+ };+ }+ return { store, replica, errors };+}++test("two deployments hand over all replicas while previous work remains draining", async () => {+ const { store, replica } = await setup();+ const a = replica("a1", "a", true);+ const a2 = replica("a2", "a", true);+ const b = replica("b1", "b");+ for (const r of [a, a2, b]) {+ r.controller.start();+ await r.controller.reconcile();+ }+ assert.equal(a.starts(), 1);+ assert.equal(b.starts(), 0);+ await store.transition({+ expectedGeneration: 0,+ requestId: "bootstrap",+ desiredDeploymentId: "b",+ bootstrapTaskArns: ["task:a1", "task:a2", "task:b1"],+ });+ await b.controller.reconcile();+ assert.equal(b.starts(), 0);+ await a.controller.reconcile();+ await b.controller.reconcile();+ assert.equal(b.starts(), 0);+ await a2.controller.reconcile();+ await b.controller.reconcile();+ assert.equal(b.starts(), 1);+ assert.equal(a.controller.canClaim(), false);+ assert.equal((await store.get()).members.find((m) => m.instanceId === "a1")?.state, "relinquished");+ await store.transition({ expectedGeneration: 1, requestId: "rollback", desiredDeploymentId: "a" });+ await b.controller.reconcile();+ await a.controller.reconcile();+ assert.equal(a.starts(), 2);+ a.drain.resolve();+ await a.controller.drained();+ assert.equal((await store.get()).members.find((m) => m.instanceId === "a1")?.state, "admitted");+ for (const r of [a, a2, b]) {+ r.drain.resolve();+ await r.controller.stop();+ }+});++test("failed activation relinquishes membership and allows a reversible successor", async () => {+ const { store, replica, errors } = await setup();+ const a = replica("a", "a", true);+ a.fail();+ a.controller.start();+ await a.controller.reconcile();+ assert.equal(a.controller.canClaim(), false);+ assert.equal((await store.get()).members[0]?.state, "relinquished");+ assert.equal(errors.length, 1);+ a.drain.resolve();+ await a.controller.stop();+});++test("database failure fences locally without inventing a durable relinquishment", async () => {+ const { store } = await setup();+ let disconnected = false;+ let fenced = 0;+ const controller = createBackgroundController({+ store: {+ ...store,+ get: async () => {+ if (disconnected) throw new Error("offline");+ return store.get();+ },+ acknowledge: async (...args) => {+ if (disconnected) throw new Error("offline");+ return store.acknowledge(...args);+ },+ },+ identity: { instanceId: "a", deploymentId: "a", taskArn: "task:a" },+ legacyEnabled: true,+ start: async () => {},+ fence: () => {+ fenced++;+ },+ relinquish: async () => {},+ drained: async () => {},+ onError() {},+ pollMs: 60_000,+ });+ controller.start();+ await controller.reconcile();+ assert.equal(controller.canClaim(), true);+ disconnected = true;+ await controller.reconcile();+ assert.equal(controller.canClaim(), false);+ assert.ok(fenced > 0);+ assert.equal((await store.get()).members[0]?.state, "admitted");+ disconnected = false;+ await controller.stop();+ await controller.drained();+ assert.equal((await store.get()).members[0]?.state, "drained");+});++test("a hanging database read expires local admission and cannot resurrect it after stop", async () => {+ const { store } = await setup();+ const waiting = Promise.withResolvers<void>();+ let hang = false;+ let starts = 0;+ const controller = createBackgroundController({+ store: {+ ...store,+ get: async () => {+ if (hang) await waiting.promise;+ return store.get();+ },+ },+ identity: { instanceId: "a", deploymentId: "a", taskArn: "task:a" },+ legacyEnabled: true,+ start: async () => {+ starts++;+ },+ fence() {},+ relinquish: async () => {},+ drained: async () => {},+ onError() {},+ pollMs: 60_000,+ validityMs: 20,+ });+ controller.start();+ await controller.reconcile();+ hang = true;+ const reconcile = controller.reconcile();+ await new Promise((resolve) => setTimeout(resolve, 40));+ assert.equal(controller.canClaim(), false);+ const stop = controller.stop();+ waiting.resolve();+ await Promise.all([reconcile, stop]);+ assert.equal(starts, 1);+ assert.equal(controller.canClaim(), false);+ await controller.drained();+ assert.equal((await store.get()).members[0]?.state, "drained");+});++test("activation is not ready until startup completes and expiry fences a late startup", async () => {+ const { store } = await setup();+ const started = Promise.withResolvers<void>();+ const finish = Promise.withResolvers<void>();+ let signal: AbortSignal | undefined;+ const controller = createBackgroundController({+ store,+ identity: { instanceId: "a", deploymentId: "a", taskArn: "task:a" },+ legacyEnabled: true,+ start: async (value) => {+ signal = value;+ started.resolve();+ await finish.promise;+ },+ fence() {},+ relinquish: async () => {},+ drained: async () => {},+ onError() {},+ pollMs: 60_000,+ validityMs: 20,+ });+ controller.start();+ await started.promise;+ assert.equal((await store.get()).members[0]?.ready, false);+ await new Promise((resolve) => setTimeout(resolve, 40));+ assert.equal(signal?.aborted, true);+ finish.resolve();+ await controller.reconcile();+ assert.equal((await store.get()).members[0]?.ready, false);+ await controller.stop();+});diff --git a/test/background-ownership.test.ts b/test/background-ownership.test.tsnew file mode 100644index 000000000..f1e6879bc--- /dev/null+++ b/test/background-ownership.test.ts@@ -0,0 +1,99 @@+import assert from "node:assert/strict";+import test from "node:test";+import { createMemoryMap } from "../src/persistence/durable-map.ts";+import { createBackgroundOwnershipStore, type BackgroundOwnership } from "../src/runs/background-ownership.ts";++async function fixture() {+ const store = createBackgroundOwnershipStore(createMemoryMap<BackgroundOwnership>());+ for (const [instanceId, deploymentId] of [+ ["a1", "a"],+ ["a2", "a"],+ ["b1", "b"],+ ["b2", "b"],+ ])+ await store.register({ instanceId: instanceId!, deploymentId: deploymentId!, taskArn: `task:${instanceId}` });+ await store.admit("a1", 0, true);+ await store.admit("a2", 0, true);+ const bootstrap = {+ expectedGeneration: 0,+ requestId: "bootstrap",+ desiredDeploymentId: "a",+ bootstrapTaskArns: ["task:a1", "task:a2", "task:b1", "task:b2"],+ };+ return { store, bootstrap };+}++test("bootstrap preserves legacy work until every old replica acknowledges relinquishment", async () => {+ const { store, bootstrap } = await fixture();+ assert.equal((await store.get()).enabled, false);+ await assert.rejects(store.admit("b1", 0, false), /disabled/);+ await assert.rejects(store.transition({ ...bootstrap, bootstrapTaskArns: ["task:a1"] }), /cohort/);+ await store.transition(bootstrap);+ await assert.rejects(store.admit("a1", 1, true), /Previous owners/);+ await store.acknowledge("a1", 0, "relinquished");+ await assert.rejects(store.admit("a1", 1, true), /Previous owners/);+ await store.acknowledge("a2", 0, "relinquished");+ await Promise.all([store.admit("a1", 1, true), store.admit("a2", 1, true)]);+ assert.equal((await store.get()).members.filter((member) => member.state === "admitted").length, 2);+});++test("generation CAS and replay identity fence stale transitions and admissions", async () => {+ const { store, bootstrap } = await fixture();+ await store.transition(bootstrap);+ assert.equal((await store.transition(bootstrap)).generation, 1);+ await assert.rejects(store.transition({ ...bootstrap, requestId: "different" }), /generation changed/);+ await assert.rejects(store.transition({ ...bootstrap, desiredDeploymentId: "b" }), /identity reused/);+ await assert.rejects(store.admit("a1", 0, true), /generation changed/);+ await assert.rejects(store.admit("b1", 1, true), /not the desired/);+});++test("handover permits previous turns to drain after new deployment starts and supports rollback", async () => {+ const { store, bootstrap } = await fixture();+ await store.transition(bootstrap);+ for (const id of ["a1", "a2"]) await store.acknowledge(id, 0, "relinquished");+ for (const id of ["a1", "a2"]) await store.admit(id, 1, true);+ await store.transition({ expectedGeneration: 1, requestId: "to-b", desiredDeploymentId: "b" });+ for (const id of ["a1", "a2"]) await store.acknowledge(id, 1, "relinquished");+ for (const id of ["b1", "b2"]) await store.admit(id, 2, false);+ assert.equal((await store.get()).members.find((member) => member.instanceId === "a1")?.state, "relinquished");+ await store.transition({ expectedGeneration: 2, requestId: "rollback", desiredDeploymentId: "a" });+ await assert.rejects(store.admit("a1", 3, true), /Previous owners/);+ for (const id of ["b1", "b2"]) await store.acknowledge(id, 2, "relinquished");+ await store.admit("a1", 3, true);+ await assert.rejects(store.acknowledge("a1", 1, "drained"), /admission changed/);+ assert.equal((await store.get()).members.find((member) => member.instanceId === "a1")?.state, "admitted");+});++test("only exact explicit termination evidence retires disconnected members and blocks task resurrection", async () => {+ const { store, bootstrap } = await fixture();+ await store.transition(bootstrap);+ await store.acknowledge("a1", 0, "relinquished");+ await assert.rejects(store.admit("a1", 1, true), /Previous owners/);+ const proof = { instanceId: "a2", taskArn: "task:a2", generation: 0 };+ for (const wrong of [+ { ...proof, taskArn: "other" },+ { ...proof, generation: 1 },+ { ...proof, instanceId: "other" },+ ])+ await assert.rejects(+ store.retire({ expectedGeneration: 1, requestId: "retire", terminatedMembers: [wrong] }),+ /evidence/,+ );+ await store.retire({ expectedGeneration: 1, requestId: "retire", terminatedMembers: [proof] });+ await store.admit("a1", 1, true);+ await assert.rejects(store.register({ instanceId: "new-process", deploymentId: "a", taskArn: "task:a2" }), /retired/);+ await assert.rejects(store.admit("a2", 1, true), /retired/);+});++test("pause has no successor and failed admissions never mutate ownership", async () => {+ const { store, bootstrap } = await fixture();+ await store.transition({ ...bootstrap, desiredDeploymentId: null });+ const before = await store.get();+ await assert.rejects(store.admit("a1", 1, true), /not the desired/);+ assert.deepEqual(await store.get(), before);+ for (const id of ["a1", "a2"]) {+ await store.acknowledge(id, 0, "relinquished");+ await store.acknowledge(id, 0, "drained");+ }+ assert.ok((await store.get()).members.every((member) => member.state === "drained"));+});diff --git a/test/background-work-routes.test.ts b/test/background-work-routes.test.tsnew file mode 100644index 000000000..d785b3d4f--- /dev/null+++ b/test/background-work-routes.test.ts@@ -0,0 +1,160 @@+import "./support/auto-fake-sprites.ts";+import assert from "node:assert/strict";+import { randomUUID } from "node:crypto";+import type { AddressInfo } from "node:net";+import test from "node:test";+import { createServer } from "../src/api/server.ts";+import { mintCapabilityToken, CAPABILITY_TTL_MS } from "../src/auth/capability-token.ts";+import { scopeId } from "../src/types.ts";+import { signRequest } from "../src/auth/source-auth.ts";+import { buildApp } from "../src/wiring.ts";+import { createMemoryMap } from "../src/persistence/durable-map.ts";+import { createBackgroundOwnershipStore, type BackgroundOwnership } from "../src/runs/background-ownership.ts";+import { testConfig } from "./support/test-config.ts";++const sourceSecret = "source-only-secret".repeat(3);+const controlSecret = "deployment-only-secret".repeat(3);+const path = "/v1/background-work";++async function fixture(secret: string | undefined = controlSecret) {+ const store = createBackgroundOwnershipStore(createMemoryMap<BackgroundOwnership>());+ await store.register({ instanceId: "instance-a", deploymentId: "cohort-a", taskArn: "task-a" });+ await store.register({ instanceId: "instance-b", deploymentId: "cohort-b", taskArn: "task-b" });+ await store.admit("instance-a", 0, true);+ const built = buildApp(testConfig({ signingSecret: sourceSecret }));+ const server = createServer(built.app, {+ signingSecret: sourceSecret,+ portalIdentitySecret: "portal-identity-secret".repeat(3),+ capabilitySecret: "capability-only-secret".repeat(3),+ requireSignedPortalIdentity: true,+ backgroundOwnership: { store, instanceId: "instance-b", deploymentId: "cohort-b" },+ deploymentControlSecret: secret,+ });+ server.listen(0);+ const url = `http://localhost:${(server.address() as AddressInfo).port}${path}`;+ let nonce = 0;+ const request = async (method: string, body?: unknown, overrides: Record<string, string> = {}) => {+ const raw = body === undefined ? "" : JSON.stringify(body);+ const ts = Math.floor(Date.now() / 1000) + nonce++;+ return fetch(url, {+ method,+ headers: {+ "content-type": "application/json",+ "x-timestamp": String(ts),+ "x-signature": signRequest(sourceSecret, ts, `${method}\n${path}\n${raw}`),+ authorization: `Bearer ${controlSecret}`,+ ...overrides,+ },+ ...(raw ? { body: raw } : {}),+ });+ };+ return { store, request, close: () => new Promise<void>((resolve) => server.close(() => resolve())) };+}++const transition = () => ({+ expectedGeneration: 0,+ requestId: randomUUID(),+ desiredDeploymentId: "cohort-b",+ bootstrapTaskArns: ["task-a", "task-b"],+});++test("ownership controls require both source and distinct deployment credentials", async () => {+ const srv = await fixture();+ try {+ const invalidCredentials: Record<string, string>[] = [+ { authorization: "" },+ { authorization: `Bearer ${sourceSecret}` },+ { "x-signature": "invalid" },+ ];+ for (const credentials of invalidCredentials) {+ assert.equal((await srv.request("POST", transition(), credentials)).status, 401);+ }+ const capability = await mintCapabilityToken(+ { actorId: "U1", scopeId: scopeId("personal", "U1"), exp: Date.now() + CAPABILITY_TTL_MS },+ "capability-only-secret".repeat(3),+ );+ for (const method of ["GET", "POST"]) {+ const response = await srv.request(method, method === "POST" ? transition() : undefined, {+ "x-agent-capability": capability,+ });+ assert.equal(response.status, 403);+ }+ assert.equal((await srv.store.get()).generation, 0);+ assert.equal((await srv.request("GET")).status, 200);+ } finally {+ await srv.close();+ }+ for (const secret of ["short", sourceSecret, "", " ".repeat(32)]) {+ const disabled = await fixture(secret);+ try {+ assert.equal((await disabled.request("GET")).status, 503);+ } finally {+ await disabled.close();+ }+ }+});++test("authenticated transitions report exact responder identity, preserve retry identity, and reject stale generations", async () => {+ const srv = await fixture();+ try {+ const change = transition();+ const response = await srv.request("POST", change);+ assert.equal(response.status, 200);+ const status = (await response.json()) as Record<string, unknown>;+ assert.equal(status.protocol, 1);+ assert.equal(status.instanceId, "instance-b");+ assert.equal(status.deploymentId, "cohort-b");+ assert.equal(status.generation, 1);+ assert.equal(status.lastRequestId, change.requestId);+ assert.equal(status.lastRequest, undefined);+ assert.equal((await srv.request("POST", change)).status, 200);+ assert.equal((await srv.request("POST", transition())).status, 409);+ const retired = await srv.request("POST", {+ expectedGeneration: 1,+ requestId: randomUUID(),+ terminatedMembers: [{ instanceId: "instance-a", taskArn: "wrong-task", generation: 0 }],+ });+ assert.equal(retired.status, 409);+ const pause = await srv.request("POST", {+ expectedGeneration: 1,+ requestId: randomUUID(),+ desiredDeploymentId: null,+ });+ assert.equal(pause.status, 200);+ assert.equal((await srv.store.get()).desiredDeploymentId, null);+ } finally {+ await srv.close();+ }+});++test("malformed or mixed mutations cannot change ownership", async () => {+ const srv = await fixture();+ try {+ for (const body of [+ null,+ { ...transition(), expectedGeneration: -1 },+ { ...transition(), requestId: "invalid" },+ { ...transition(), desiredDeploymentId: 3 },+ { ...transition(), extra: true },+ { ...transition(), bootstrapTaskArns: [] },+ { ...transition(), terminatedMembers: [] },+ {+ expectedGeneration: 0,+ requestId: randomUUID(),+ terminatedMembers: [{ instanceId: "instance-a", taskArn: "task-a", generation: -1 }],+ },+ { expectedGeneration: 0, requestId: randomUUID() },+ ])+ assert.equal((await srv.request("POST", body)).status, 400);+ assert.equal((await srv.store.get()).generation, 0);+ const retired = await srv.request("POST", {+ expectedGeneration: 0,+ requestId: randomUUID(),+ terminatedMembers: [{ instanceId: "instance-a", taskArn: "task-a", generation: 0 }],+ });+ assert.equal(retired.status, 200);+ assert.equal((await srv.store.get()).members.find((member) => member.instanceId === "instance-a")?.retired, true);+ } finally {+ await srv.close();+ }+});diff --git a/test/config.test.ts b/test/config.test.tsindex 198819b3a..aadb10398 100644--- a/test/config.test.ts+++ b/test/config.test.ts@@ -764,3 +764,26 @@ test("Fly shared application name is passed to the deployment provider", () => {assert.equal(config.flyDeploy.sharedAppName, "qm-example-apps");assert.equal(config.flyDeploy.dataVolumeSizeGb, 1);});++test("background ownership requires durable storage and an independent deployment authority", () => {+ const env = {+ BACKGROUND_DEPLOYMENT_ID: "core:release-a",+ DATABASE_URL: "postgres://localhost/test",+ CORE_SIGNING_SECRET: "source-signing-secret-0123456789abcdef",+ DEPLOYMENT_CONTROL_SECRET: "deployment-control-secret-0123456789abcdef",+ };+ const config = loadConfig(env);+ assert.equal(config.backgroundDeploymentId, env.BACKGROUND_DEPLOYMENT_ID);+ assert.equal(config.deploymentControlSecret, env.DEPLOYMENT_CONTROL_SECRET);+ assert.equal(config.backgroundWorkEnabled, true);+ assert.equal(loadConfig({ ...env, BACKGROUND_WORK_ENABLED: "false" }).backgroundWorkEnabled, false);+ assert.throws(() => loadConfig({ ...env, DATABASE_URL: "" }), /DATABASE_URL/);+ assert.throws(() => loadConfig({ ...env, DEPLOYMENT_CONTROL_SECRET: "short" }), /distinct DEPLOYMENT_CONTROL_SECRET/);+ assert.throws(+ () => loadConfig({ ...env, DEPLOYMENT_CONTROL_SECRET: env.CORE_SIGNING_SECRET }),+ /distinct DEPLOYMENT_CONTROL_SECRET/,+ );+ assert.throws(() => loadConfig({ ...env, BACKGROUND_DEPLOYMENT_ID: " " }), /BACKGROUND_DEPLOYMENT_ID/);+ assert.throws(() => loadConfig({ ...env, CORE_SIGNING_SECRET: "short" }), /CORE_SIGNING_SECRET/);+ assert.throws(() => loadConfig({ ...env, DEPLOYMENT_CONTROL_SECRET: " ".repeat(32) }), /DEPLOYMENT_CONTROL_SECRET/);+});diff --git a/test/cron-queue-lifecycle.test.ts b/test/cron-queue-lifecycle.test.tsindex d0a5abbd0..c20710865 100644--- a/test/cron-queue-lifecycle.test.ts+++ b/test/cron-queue-lifecycle.test.ts@@ -5,6 +5,7 @@ type Adapter = { executeSql(text: string, values?: unknown[]): Promise<unknown>const pools: { closed: boolean }[] = [];let adapter: Adapter;let failStart = false;+let failWork = false;mock.module("../src/persistence/pg-pool.ts", {namedExports: {@@ -36,7 +37,10 @@ mock.module("pg-boss", {if (failStart) throw new Error("startup failed");}async createQueue() {}- async work() {}+ async work() {+ if (failWork) throw new Error("polling failed");+ }+ async offWork() {}async send() {await adapter.executeSql("SELECT 1");}@@ -85,3 +89,27 @@ test("startup failure closes the database and permits a fresh start", async () =await queue.stop();}});++test("resuming polling failure preserves the database for admitted callbacks", async () => {+ const queue = createPgBossCronQueue("postgres://unused");+ await queue.start(handlers, 60_000);+ await queue.stopClaims!();+ const count = pools.length;+ failWork = true;+ try {+ await assert.rejects(queue.start(handlers, 60_000), /polling failed/);+ assert.equal(queue.healthy(), false);+ assert.equal(pools.at(-1)?.closed, false);+ await adapter.executeSql("SELECT 1");+ } finally {+ failWork = false;+ }+ try {+ await queue.start(handlers, 60_000);+ assert.equal(pools.length, count);+ assert.ok(queue.healthy());+ } finally {+ await queue.stop();+ }+ assert.ok(pools.at(-1)?.closed);+});diff --git a/test/cron-scheduler.test.ts b/test/cron-scheduler.test.tsindex 575a6ac98..57c565338 100644--- a/test/cron-scheduler.test.ts+++ b/test/cron-scheduler.test.ts@@ -1498,3 +1498,151 @@ test("scheduler cannot resume after uncertain queue shutdown until a stop retryawait scheduler.stop();assert.equal(starts, 2);});++test("scheduler relinquishes queue claims before a long turn drains and preserves its queue connection", async () => {+ const crons = createCronStore();+ const entered = Promise.withResolvers<void>();+ const finish = Promise.withResolvers<void>();+ let onFire: ((job: { cronId: string; scheduledAt: number }) => Promise<void>) | undefined;+ let claimsStopped = false;+ let queueClosed = false;+ const scheduler = createScheduler({+ crons,+ deliveries: createDeliveryStore(),+ idempotency: createIdempotencyStore(),+ identity: createIdentityService(),+ now: () => 1000,+ run: async () => {+ entered.resolve();+ await finish.promise;+ assert.equal(queueClosed, false);+ return { status: "ok", reply: "done" };+ },+ jobQueue: {+ start: async (handlers) => {+ onFire = handlers.onFire;+ },+ enqueueFire: async () => {},+ healthy: () => true,+ stopClaims: async () => {+ claimsStopped = true;+ },+ stop: async () => {+ queueClosed = true;+ },+ },+ });+ const cron = await crons.create({+ schedule: { firstFireAt: 1 },+ action: "work",+ owner: "U1",+ createdBy: "U1",+ ownerScopeId: scopeId("personal", "U1"),+ });+ scheduler.start(1000);+ const work = onFire!({ cronId: cron.id, scheduledAt: 1 });+ await entered.promise;+ const stopped = scheduler.stop();+ await scheduler.stopClaims();+ assert.equal(claimsStopped, true);+ assert.equal(queueClosed, false);+ let drained = false;+ const draining = scheduler.drained().then(() => {+ drained = true;+ });+ await new Promise((resolve) => setImmediate(resolve));+ assert.equal(drained, false);+ finish.resolve();+ await Promise.all([work, stopped, draining]);+ assert.equal(queueClosed, true);+});++test("scheduler pause fences the next cron in an already admitted polling batch", async () => {+ const entered = Promise.withResolvers<void>();+ const finish = Promise.withResolvers<void>();+ const { crons, calls, scheduler } = harness(async () => {+ entered.resolve();+ await finish.promise;+ return { status: "ok", reply: "done" };+ });+ for (let i = 0; i < 2; i++)+ await crons.create({+ schedule: { firstFireAt: 1 },+ action: `work ${i}`,+ owner: "U1",+ createdBy: "U1",+ ownerScopeId: scopeId("personal", "U1"),+ });+ const tick = scheduler.tick(1000);+ await entered.promise;+ await scheduler.stopClaims();+ finish.resolve();+ await tick;+ assert.equal(calls.length, 1);+ await scheduler.stop();+});++test("scheduler resumes new queue claims while an older admitted turn remains alive", async () => {+ const crons = createCronStore();+ const entered = Promise.withResolvers<void>();+ const finish = Promise.withResolvers<void>();+ let onFire: ((job: { cronId: string; scheduledAt: number }) => Promise<void>) | undefined;+ let calls = 0;+ let starts = 0;+ let closed = false;+ const scheduler = createScheduler({+ crons,+ deliveries: createDeliveryStore(),+ idempotency: createIdempotencyStore(),+ identity: createIdentityService(),+ now: () => 1000,+ run: async () => {+ if (++calls === 1) {+ entered.resolve();+ await finish.promise;+ }+ assert.equal(closed, false);+ return { status: "ok", reply: "done" };+ },+ jobQueue: {+ start: async (handlers) => {+ starts++;+ onFire = handlers.onFire;+ },+ enqueueFire: async () => {},+ healthy: () => true,+ stopClaims: async () => {},+ stop: async () => {+ closed = true;+ },+ },+ });+ const jobs = [];+ for (let i = 0; i < 2; i++)+ jobs.push(+ await crons.create({+ schedule: { firstFireAt: 1 },+ action: `work ${i}`,+ owner: "U1",+ createdBy: "U1",+ ownerScopeId: scopeId("personal", "U1"),+ }),+ );+ scheduler.start(1000);+ await scheduler.ready();+ const oldHandler = onFire!;+ const oldWork = oldHandler({ cronId: jobs[0]!.id, scheduledAt: 1 });+ await entered.promise;+ await scheduler.stopClaims();+ scheduler.start(1000);+ await scheduler.ready();+ assert.equal(starts, 2);+ await oldHandler({ cronId: jobs[1]!.id, scheduledAt: 1 });+ assert.equal(calls, 1);+ await onFire!({ cronId: jobs[1]!.id, scheduledAt: 1 });+ assert.equal(calls, 2);+ assert.equal(closed, false);+ finish.resolve();+ await oldWork;+ await scheduler.stop();+});diff --git a/test/postgres-background-ownership.test.ts b/test/postgres-background-ownership.test.tsnew file mode 100644index 000000000..130f7a4bc--- /dev/null+++ b/test/postgres-background-ownership.test.ts@@ -0,0 +1,93 @@+import assert from "node:assert/strict";+import { randomUUID } from "node:crypto";+import test from "node:test";+import { createPostgresMapFactory } from "../src/persistence/durable-map.ts";+import { createBackgroundOwnershipStore, type BackgroundOwnership } from "../src/runs/background-ownership.ts";++const databaseUrl = process.env.DATABASE_URL;+test("Postgres serializes concurrent ownership transitions and replica admission", { skip: !databaseUrl }, async () => {+ const table = `background_test_${randomUUID().replaceAll("-", "")}`;+ const factories = [createPostgresMapFactory(databaseUrl!), createPostgresMapFactory(databaseUrl!)];+ try {+ const stores = factories.map((factory) => createBackgroundOwnershipStore(factory.map<BackgroundOwnership>(table)));+ await Promise.all(+ stores.map((store, i) => store.register({ instanceId: `a${i}`, deploymentId: "a", taskArn: `task:a${i}` })),+ );+ await Promise.all(stores.map((store, i) => store.admit(`a${i}`, 0, true)));+ const transitions = await Promise.allSettled(+ stores.map((store) =>+ store.transition({+ expectedGeneration: 0,+ requestId: randomUUID(),+ desiredDeploymentId: "a",+ bootstrapTaskArns: ["task:a0", "task:a1"],+ }),+ ),+ );+ assert.equal(transitions.filter((result) => result.status === "fulfilled").length, 1);+ assert.equal(transitions.filter((result) => result.status === "rejected").length, 1);+ await assert.rejects(stores[0]!.admit("a0", 1, true), /Previous owners/);+ await Promise.all(stores.map((store, i) => store.acknowledge(`a${i}`, 0, "relinquished")));+ await Promise.all(stores.map((store, i) => store.admit(`a${i}`, 1, true)));+ const state = await stores[0]!.get();+ assert.equal(state.members.filter((member) => member.state === "admitted" && member.generation === 1).length, 2);+ assert.deepEqual(await stores[1]!.get(), state);+ } finally {+ const pool = await factories[0]!.pool.pool();+ await pool.query(`DROP TABLE IF EXISTS ${table}`);+ await Promise.all(factories.map((factory) => factory.pool.close()));+ }+});++test(+ "Postgres queue resumes polling without terminating a prior generation callback",+ { skip: !databaseUrl, timeout: 20_000 },+ async () => {+ const { createPgBossCronQueue } = await import("../src/cron/job-queue.ts");+ const schema = `handover_${randomUUID().replaceAll("-", "")}`;+ const queue = createPgBossCronQueue(databaseUrl!, schema);+ const oldEntered = Promise.withResolvers<void>();+ const oldFinish = Promise.withResolvers<void>();+ const newEntered = Promise.withResolvers<void>();+ let oldFinished = false;+ try {+ await queue.start(+ {+ onTick: async () => {},+ onFire: async () => {+ oldEntered.resolve();+ await oldFinish.promise;+ oldFinished = true;+ },+ },+ 60_000,+ );+ await queue.enqueueFire({ cronId: "old", scheduledAt: Date.now() });+ await oldEntered.promise;+ await queue.stopClaims!();+ assert.equal(oldFinished, false);+ await queue.start(+ {+ onTick: async () => {},+ onFire: async () => {+ newEntered.resolve();+ },+ },+ 60_000,+ );+ await queue.enqueueFire({ cronId: "new", scheduledAt: Date.now() });+ await newEntered.promise;+ assert.equal(oldFinished, false);+ oldFinish.resolve();+ await new Promise((resolve) => setImmediate(resolve));+ assert.equal(oldFinished, true);+ } finally {+ oldFinish.resolve();+ await queue.stop();+ const { default: pg } = await import("pg");+ const pool = new pg.Pool({ connectionString: databaseUrl! });+ await pool.query(`DROP SCHEMA IF EXISTS ${schema} CASCADE`);+ await pool.end();+ }+ },+);diff --git a/test/slack-runtime.test.ts b/test/slack-runtime.test.tsindex 079e7e788..51088be1a 100644--- a/test/slack-runtime.test.ts+++ b/test/slack-runtime.test.ts@@ -292,3 +292,24 @@ test("a rollback startup with incomplete cleanup also blocks relinquishment", ascleanupAllowed = true;await runtime.stop();});++test("a controlled inactive Slack runtime cannot open a socket through configuration reconciliation", async () => {+ let starts = 0;+ const runtime = createSlackRuntimeReconciler({+ startPaused: true,+ load: async () => ({ version: "installed", config: {} }),+ startPlugin: async () => {+ starts++;+ return { stop: async () => {} };+ },+ });+ await runtime.reconcile();+ await runtime.reconcile();+ assert.equal(starts, 0);+ runtime.start();+ await runtime.reconcile();+ assert.equal(starts, 1);+ await runtime.stop();+ await runtime.reconcile();+ assert.equal(starts, 1);+});