The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
fix: allow concurrent agent runs on one OpenAI or xAI subscription connection (#13452)
Nicky Leach9ed55f6931a5
diff --git a/doc/connections/AI-CONNECTIONS.md b/doc/connections/AI-CONNECTIONS.mdindex 626baf504b..6f9967f723 100644--- a/doc/connections/AI-CONNECTIONS.md+++ b/doc/connections/AI-CONNECTIONS.md@@ -108,26 +108,23 @@ grant's credentials. Inherited credential variables are cleared. Conflictingproject authentication and provider-routing overrides are rejected. Managedfailure cannot reactivate host or legacy credentials.-A subscription invocation takes a grant-scoped transaction advisory lease only-when it writes a provider authentication file back to the grant. OpenAI and xAI-subscriptions do this, because their refresh tokens are single use. An-Anthropic subscription invocation writes no file back, so it takes no lease;-two Anthropic invocations of one grant run at the same time. The reserved-database client keeps one transaction open until cleanup, including on-transaction-pooling proxies such as PgBouncer. Session-level advisory locks must-not be used here: a pooled connection can return to a different backend for-cleanup and leave the original lock behind. The lease transaction disables its-idle timeout and contains no application data writes; cleanup rolls it back.-Two-different users' grants can run concurrently; a second invocation of the same-file-backed subscription receives a retryable busy response while it is in-use. Refreshes are merged only into the originating active grant, with-reconnect/revocation version checks. Temporary homes are removed on normal-completion or failure.--For a fresh task execution, subscription contention creates a durable scheduled-retry checked every 60–120 seconds. The task shows “Waiting for AI subscription”-and does not request a reconnect or consume its provider-failure retry allowance.+A subscription invocation takes no lease. Two invocations of one grant, from+the same or a different provider account, run at the same time. At cleanup,+each invocation re-reads the credential stored at that moment under a row+lock on the grant, then compares it against its own refreshed copy using the+provider's own freshness field: Codex compares `last_refresh` and bounds it+against the host clock; Grok compares `expires_at`. The newer credential+persists; a tie or an unparseable freshness value keeps the stored+credential, so a spent single-use refresh token never overwrites a good one.+Refreshes are merged only into the originating active grant, with a+revocation check. Temporary homes are removed on normal completion or+failure.++A fresh task execution cannot enter subscription contention. The freshest-write+rule above resolves the conflict instead. A run that already entered this wait+keeps a durable scheduled retry, checked every 60–120 seconds. The task shows+“Waiting for AI subscription”. It does not request a reconnect, and it does not+consume its provider-failure retry allowance.Each attempt rechecks task eligibility, ownership, budget, and current credentialaccess. Revocation and other configuration failures still require user action.Authorized comment wakes that started as non-assignee runs can resume without@@ -215,8 +212,8 @@ unmanaged legacy agents retain their existing authentication paths.`server/src/__tests__/ai-connections.test.ts` exercises storage, isolation,defaults, human audiences, agent access, reconnect races, refresh ownership,-subscription locking, migration replay, and redacted API failures against a real-embedded database. Existing login, adapter, tool, and channel suites cover their+concurrent subscription write-backs, migration replay, and redacted API+failures against a real embedded database. Existing login, adapter, tool, and channel suites cover theirshared integration paths. The onboarding tests cover managed reuse and keeping asuccessfully connected account after failed agent creation.@@ -291,15 +288,18 @@ inside the task. Connecting installs access for that agent and resumes the pendiwork automatically. Explicit incompatible bindings and shared-account permissiondenials still fail; hiring never expands a restricted shared account's audience.-Concurrent runs using the same subscription wait through scheduled retries while-the credential lease is held. They do not request new credentials or consume the-provider-failure retry allowance. Each retry revalidates the account and existing-run-dispatch rules still suppress cancelled, reassigned, or otherwise ineligible work.-Assignee retries must retain execution-lock ownership at scheduling, promotion, and dispatch.+Concurrent runs of one subscription do not wait for each other. No credential+lease exists to hold them, so a fresh task execution cannot enter a contention+wait. A run that already entered this wait keeps its scheduled retries. It does+not request new credentials, and it does not consume the provider-failure retry+allowance. Each retry revalidates the account, and existing run-dispatch rules+still suppress cancelled, reassigned, or otherwise ineligible work. An assignee+retry must still keep execution-lock ownership at scheduling, promotion, and+dispatch.`server/src/__tests__/agent-hire-ai-connections.test.ts` covers both creation routes,both providers and methods, approval gates, native provider mapping, shared access-boundaries, and subscription contention. The opt-in+boundaries, and concurrent runs of one subscription for both providers. The opt-in[`tests/hiring-ai-connections/README.md`](../../tests/hiring-ai-connections/README.md)describes real browser hiring, subtask, connection, and automatic-resume checks onlocal and Daytona environments, plus the production component Storybook checks.diff --git a/doc/run-log-events.md b/doc/run-log-events.mdindex 6c47dd1932..6a99289429 100644--- a/doc/run-log-events.md+++ b/doc/run-log-events.md@@ -179,9 +179,10 @@ identity checks remain in force.## AI subscription contention-A run that waits for a managed subscription lease writes an informational-`lifecycle` event to the local run log. Its payload contains only-`retryScheduled`, a boolean that reports whether the scheduler created a retry.+A fresh task execution cannot enter this wait. A run that already entered this+wait writes an informational `lifecycle` event to the local run log. Its+payload contains only `retryScheduled`, a boolean that reports+whether the scheduler created a retry.The message distinguishes an automatic retry from work that is no longer eligible.This pre-provider wait records `ai_connection_busy` on the cancelled run and doesnot consume the provider-failure retry allowance. The event contains no credentialsdiff --git a/packages/adapters/codex-local/src/server/codex-auth-copyback.ts b/packages/adapters/codex-local/src/server/codex-auth-copyback.tsindex 96d29f3e20..f22ae0369a 100644--- a/packages/adapters/codex-local/src/server/codex-auth-copyback.ts+++ b/packages/adapters/codex-local/src/server/codex-auth-copyback.ts@@ -7,7 +7,11 @@ import {readSubscriptionAccountId,writeCodexAuthCacheEntry,} from "./codex-auth-cache.js";-import { USE_SOURCE_EXIT, decideCodexAuthMerge } from "./codex-auth-merge-decision.js";+import {+ IMPLAUSIBLE_LAST_REFRESH_EXIT,+ USE_SOURCE_EXIT,+ decideCodexAuthMerge,+} from "./codex-auth-merge-decision.js";// The outbound copy-back reuses the exact same direction-agnostic decision// predicate the inbound restore runs, through the shared `decideCodexAuthMerge`@@ -122,6 +126,17 @@ export async function copyBackCodexAuth(input: CopyBackCodexAuthInput): Promise<return "copied";}+ // Make a clock-bound rejection visible. A host with a wrong clock would+ // otherwise discard an honest refresh with no signal. Log only the exit+ // code and this caller's own label — never a timestamp read from either+ // file, and never credential bytes.+ if (decision === IMPLAUSIBLE_LAST_REFRESH_EXIT) {+ await log(+ `[paperclip] Codex auth copy-back: WARNING host credential kept (decision exit ${IMPLAUSIBLE_LAST_REFRESH_EXIT}, codex auth copy-back) — the sandbox copy's last_refresh sat further ahead of the host clock than the plausible skew allowance. Check the host clock if this is unexpected.`,+ );+ return "kept-host";+ }+await log("[paperclip] Codex auth copy-back: host credential kept (sandbox copy is not a strictly-newer same-identity subscription credential).",);diff --git a/packages/adapters/codex-local/src/server/codex-auth-merge-decision.cjs b/packages/adapters/codex-local/src/server/codex-auth-merge-decision.cjsindex fe557795a2..01822d2e39 100644--- a/packages/adapters/codex-local/src/server/codex-auth-merge-decision.cjs+++ b/packages/adapters/codex-local/src/server/codex-auth-merge-decision.cjs@@ -64,54 +64,107 @@ function parseAuth(filePath) {// A leading positional flag (not an environment variable) keeps the mode// explicit per call, so a host-default two-path call can never enter seed mode.//-// Exit 10 = use source; exit 20 = keep destination. The predicate only ever-// reads the two files and exits with a code — it never prints token bytes.+// Exit contract. Exit 10 = use source; exit 20 = keep destination; exit 22 =+// keep destination, the source `last_refresh` sat further ahead of the host+// clock than the plausible skew allowance. The predicate only ever reads the+// two files and exits with a code — it never prints token bytes.const USE_SOURCE = 10;const KEEP_DESTINATION = 20;+const IMPLAUSIBLE_LAST_REFRESH = 22;const SEED_IF_DEST_ABSENT_FLAG = "--seed-if-dest-absent";-const rawArgs = process.argv.slice(2);-const seedIfDestAbsent = rawArgs[0] === SEED_IF_DEST_ABSENT_FLAG;-const [sourceAuthPath, destinationAuthPath] = seedIfDestAbsent ? rawArgs.slice(1) : rawArgs;-const sourceAuth = parseAuth(sourceAuthPath);-const destinationAuth = parseAuth(destinationAuthPath);--// Seed mode only: fill an ABSENT (unusable) destination slot from a usable-// subscription source. A subscription-kind source is guaranteed usable and to-// carry a real account_id (parseAuth returns "subscription" only then), so this-// is never a random pick. This branch changes ONLY the destination-unusable-// case; the api-key and unusable-source guards below still keep the destination.-if (- seedIfDestAbsent &&- destinationAuth.kind === "unusable" &&- sourceAuth.kind === "subscription"-) {- process.exit(USE_SOURCE);-}+// A `last_refresh` records an event that already happened, so an honest value+// always sits at or before the host clock. Only clock skew between a sandbox+// and the host explains a small future value. Five minutes is larger than+// real skew on a time-synchronised host, and it is short enough that a source+// which claims a `last_refresh` further ahead than this cannot be trusted.+const MAX_FUTURE_LAST_REFRESH_SKEW_MS = 5 * 60 * 1000;++// `decide` takes already-parsed `{ kind, accountId, lastRefresh }` shapes plus+// a caller-supplied `nowMs` and the `seedIfDestAbsent` flag, so a test can+// drive the exact skew bound without spawning a process and racing wall-clock+// drift. Guard order, first match wins:+// 1. Seed mode, the destination is unusable, and the source is a usable+// subscription credential -> USE_SOURCE, unless the source fails the+// skew bound in step 3 below, in which case the absent slot stays+// absent (IMPLAUSIBLE_LAST_REFRESH). A poisoned seed would out-live+// every honest refresh, since nothing could ever compare as "strictly+// newer" than an unbounded future value, so the bound applies here too.+// 2. Either side unusable, a kind mismatch, the destination is an api-key+// credential, or the two sides carry a different account_id ->+// KEEP_DESTINATION.+// 3. The source `last_refresh` sits further ahead of `nowMs` than+// MAX_FUTURE_LAST_REFRESH_SKEW_MS -> IMPLAUSIBLE_LAST_REFRESH. Only the+// source is bounded, and only against the caller's clock: the source is+// the sandbox-supplied side, so it must never supply the reference time.+// 4. The source `last_refresh` is strictly greater than the destination's+// -> USE_SOURCE.+// 5. Otherwise (a tie, a null value on either side, or an older source) ->+// KEEP_DESTINATION.+function decide(source, destination, nowMs, seedIfDestAbsent) {+ const sourceIsImplausible =+ source.lastRefresh !== null &&+ source.lastRefresh - nowMs > MAX_FUTURE_LAST_REFRESH_SKEW_MS;++ // Seed mode only: fill an ABSENT (unusable) destination slot from a usable+ // subscription source. A subscription-kind source is guaranteed usable and to+ // carry a real account_id (parseAuth returns "subscription" only then), so this+ // is never a random pick. This branch changes ONLY the destination-unusable+ // case; the api-key and unusable-source guards below still keep the destination.+ if (+ seedIfDestAbsent &&+ destination.kind === "unusable" &&+ source.kind === "subscription"+ ) {+ return sourceIsImplausible ? IMPLAUSIBLE_LAST_REFRESH : USE_SOURCE;+ }++ // Fail closed to the destination unless both sides are the same usable,+ // subscription-kind identity — an unusable side, an api-key credential, a kind+ // mismatch, or a different account_id all keep the destination copy.+ if (+ destination.kind === "unusable" ||+ source.kind === "unusable" ||+ source.kind !== destination.kind ||+ destination.kind === "apikey" ||+ source.accountId !== destination.accountId+ ) {+ return KEEP_DESTINATION;+ }++ if (sourceIsImplausible) {+ return IMPLAUSIBLE_LAST_REFRESH;+ }++ // Use the source credential only when it is strictly fresher: both sides must+ // carry a parseable last_refresh and the source one must be strictly greater.+ // Ties and null/unparseable freshness keep the destination copy so a spent+ // single-use refresh token is never written over a good one.+ if (+ source.lastRefresh !== null &&+ destination.lastRefresh !== null &&+ source.lastRefresh > destination.lastRefresh+ ) {+ return USE_SOURCE;+ }-// Fail closed to the destination unless both sides are the same usable,-// subscription-kind identity — an unusable side, an api-key credential, a kind-// mismatch, or a different account_id all keep the destination copy.-if (- destinationAuth.kind === "unusable" ||- sourceAuth.kind === "unusable" ||- sourceAuth.kind !== destinationAuth.kind ||- destinationAuth.kind === "apikey" ||- sourceAuth.accountId !== destinationAuth.accountId-) {- process.exit(KEEP_DESTINATION);+ return KEEP_DESTINATION;}-// Use the source credential only when it is strictly fresher: both sides must-// carry a parseable last_refresh and the source one must be strictly greater.-// Ties and null/unparseable freshness keep the destination copy so a spent-// single-use refresh token is never written over a good one.-if (- sourceAuth.lastRefresh !== null &&- destinationAuth.lastRefresh !== null &&- sourceAuth.lastRefresh > destinationAuth.lastRefresh-) {- process.exit(USE_SOURCE);+if (require.main === module) {+ const rawArgs = process.argv.slice(2);+ const seedIfDestAbsent = rawArgs[0] === SEED_IF_DEST_ABSENT_FLAG;+ const [sourceAuthPath, destinationAuthPath] = seedIfDestAbsent ? rawArgs.slice(1) : rawArgs;+ const sourceAuth = parseAuth(sourceAuthPath);+ const destinationAuth = parseAuth(destinationAuthPath);+ process.exit(decide(sourceAuth, destinationAuth, Date.now(), seedIfDestAbsent));}-process.exit(KEEP_DESTINATION);+module.exports = {+ decide,+ parseAuth,+ USE_SOURCE,+ KEEP_DESTINATION,+ IMPLAUSIBLE_LAST_REFRESH,+ MAX_FUTURE_LAST_REFRESH_SKEW_MS,+};diff --git a/packages/adapters/codex-local/src/server/codex-auth-merge-decision.test.ts b/packages/adapters/codex-local/src/server/codex-auth-merge-decision.test.tsindex 5c05352cda..40bc772851 100644--- a/packages/adapters/codex-local/src/server/codex-auth-merge-decision.test.ts+++ b/packages/adapters/codex-local/src/server/codex-auth-merge-decision.test.ts@@ -1,5 +1,6 @@import { execFile as execFileCallback } from "node:child_process";import { mkdtemp, rm, writeFile } from "node:fs/promises";+import { createRequire } from "node:module";import os from "node:os";import path from "node:path";import { fileURLToPath } from "node:url";@@ -8,6 +9,28 @@ import { afterEach, describe, expect, it } from "vitest";const execFile = promisify(execFileCallback);+// The exact-boundary cases below drive the predicate's pure `decide` function+// directly (via `require`, never spawning a process), so the skew bound can be+// tested to the exact millisecond without racing subprocess-spawn wall-clock+// drift. Every other case in this file drives the REAL `.cjs` through a+// spawned `node` process (no stub), matching how the wrapper invokes it in+// production.+const decisionModule = createRequire(import.meta.url)(+ fileURLToPath(new URL("./codex-auth-merge-decision.cjs", import.meta.url)),+) as {+ decide: (+ source: { kind: "subscription" | "apikey" | "unusable"; accountId?: string; lastRefresh: number | null },+ destination: { kind: "subscription" | "apikey" | "unusable"; accountId?: string; lastRefresh: number | null },+ nowMs: number,+ seedIfDestAbsent?: boolean,+ ) => number;+ USE_SOURCE: number;+ KEEP_DESTINATION: number;+ IMPLAUSIBLE_LAST_REFRESH: number;+ MAX_FUTURE_LAST_REFRESH_SKEW_MS: number;+};+const { decide, IMPLAUSIBLE_LAST_REFRESH, MAX_FUTURE_LAST_REFRESH_SKEW_MS } = decisionModule;+// This suite pins the opt-in seed mode of the single decision predicate. The// default (no-flag) call keeps the fail-closed host-default contract unchanged.// The leading positional `--seed-if-dest-absent` flag adds one behaviour: fill@@ -165,3 +188,94 @@ describe("codex-auth-merge-decision predicate seed mode", () => {expect(seedCode).toBe(USE_SOURCE);});});++// This suite pins the host-clock bound on the source `last_refresh`. A+// `last_refresh` records an event that already happened, so an honest value+// never sits far ahead of the host clock; only clock skew explains a small+// future value. A source that claims a value further ahead than+// `MAX_FUTURE_LAST_REFRESH_SKEW_MS` cannot be trusted, since a same-account+// sandbox that controls its own `auth.json` could otherwise pin an unbounded+// future timestamp that every later honest refresh compares as older than.+// These cases drive `decide` directly with an explicit `nowMs`, so the bound+// is proven to the exact millisecond and never depends on the real clock.+describe("codex-auth-merge-decision predicate: host-clock bound on last_refresh", () => {+ const USE_SOURCE = 10;+ const KEEP_DESTINATION = 20;++ function subscription(accountId: string, lastRefresh: number | null) {+ return { kind: "subscription" as const, accountId, lastRefresh };+ }++ it("keeps the destination when the source last_refresh sits one millisecond beyond the bound", () => {+ const nowMs = Date.now();+ const source = subscription("acct", nowMs + MAX_FUTURE_LAST_REFRESH_SKEW_MS + 1);+ const destination = subscription("acct", nowMs - 60_000);+ expect(decide(source, destination, nowMs, false)).toBe(IMPLAUSIBLE_LAST_REFRESH);+ });++ it("uses the source when the source last_refresh sits exactly at the bound", () => {+ const nowMs = Date.now();+ const source = subscription("acct", nowMs + MAX_FUTURE_LAST_REFRESH_SKEW_MS);+ const destination = subscription("acct", nowMs - 60_000);+ expect(decide(source, destination, nowMs, false)).toBe(USE_SOURCE);+ });++ it("still uses the source for an ordinary same-account, strictly-newer, past timestamp", () => {+ const nowMs = Date.now();+ const source = subscription("acct", nowMs - 60_000);+ const destination = subscription("acct", nowMs - 120_000);+ expect(decide(source, destination, nowMs, false)).toBe(USE_SOURCE);+ });++ it("measures the bound against the host clock and not against a value embedded in either payload", () => {+ // Fixed instants; only the caller-supplied `nowMs` moves between the two+ // assertions, so a passing/failing bound can only be explained by the+ // caller's clock, never by a value embedded in either payload.+ const sourceLastRefresh = 2_000_000_000_000;+ const destinationLastRefresh = 1_000_000_000_000;+ const source = subscription("acct", sourceLastRefresh);+ const destination = subscription("acct", destinationLastRefresh);++ expect(decide(source, destination, sourceLastRefresh - MAX_FUTURE_LAST_REFRESH_SKEW_MS, false)).toBe(USE_SOURCE);+ expect(decide(source, destination, sourceLastRefresh - MAX_FUTURE_LAST_REFRESH_SKEW_MS - 1, false)).toBe(+ IMPLAUSIBLE_LAST_REFRESH,+ );+ });++ it("keeps the destination for a same-identity source whose kind or account guard would otherwise fire, regardless of the bound", () => {+ // A kind mismatch or a different account_id already keeps the destination+ // before the bound runs, so an implausible source never surfaces as+ // IMPLAUSIBLE_LAST_REFRESH when a different guard already applies.+ const nowMs = Date.now();+ const implausibleLastRefresh = nowMs + MAX_FUTURE_LAST_REFRESH_SKEW_MS + 1;+ const differentAccount = decide(+ subscription("acct-x", implausibleLastRefresh),+ subscription("acct-y", nowMs - 60_000),+ nowMs,+ false,+ );+ expect(differentAccount).toBe(KEEP_DESTINATION);+ });++ // Seed mode fills an ABSENT destination slot from a usable subscription+ // source. The bound applies there too: an absent destination that got+ // seeded with an implausible future last_refresh would never be+ // refreshable again, since no later honest refresh could ever compare as+ // "strictly newer" than an unbounded future value. The seed-mode source in+ // production is the sandbox's own `auth.json`+ // (see `writeCodexAuthCacheEntry` in codex-auth-cache.ts), so this is the+ // same untrusted input the write-back guard defends against.+ it("seed mode keeps an absent destination slot absent when the source last_refresh is implausibly far in the future", () => {+ const nowMs = Date.now();+ const source = subscription("acct", nowMs + MAX_FUTURE_LAST_REFRESH_SKEW_MS + 1);+ const destination = { kind: "unusable" as const, lastRefresh: null };+ expect(decide(source, destination, nowMs, true)).toBe(IMPLAUSIBLE_LAST_REFRESH);+ });++ it("seed mode still fills an absent destination slot when the source last_refresh sits at or before the bound", () => {+ const nowMs = Date.now();+ const source = subscription("acct", nowMs + MAX_FUTURE_LAST_REFRESH_SKEW_MS);+ const destination = { kind: "unusable" as const, lastRefresh: null };+ expect(decide(source, destination, nowMs, true)).toBe(USE_SOURCE);+ });+});diff --git a/packages/adapters/codex-local/src/server/codex-auth-merge-decision.ts b/packages/adapters/codex-local/src/server/codex-auth-merge-decision.tsindex 0ecd228a73..bbcfddfcce 100644--- a/packages/adapters/codex-local/src/server/codex-auth-merge-decision.ts+++ b/packages/adapters/codex-local/src/server/codex-auth-merge-decision.ts@@ -11,10 +11,12 @@ const execFile = promisify(execFileCallback);// `codex-auth-merge-decision.cjs`. It reads only the two files, and it exits with// a code; it never prints token bytes. Argument order sets source and// destination (first = source, second = destination), so the caller frames the-// direction. Exit 10 = use source; exit 20 = keep destination. The leading-// `--seed-if-dest-absent` flag adds one behavior: fill an absent destination slot-// from a usable subscription source. This module gives every caller one shared-// entry point, so the predicate contract can never drift between callers.+// direction. Exit 10 = use source; exit 20 = keep destination; exit 22 = keep+// destination, the source `last_refresh` sat further ahead of the host clock+// than the plausible skew allowance. The leading `--seed-if-dest-absent` flag+// adds one behavior: fill an absent destination slot from a usable+// subscription source. This module gives every caller one shared entry point,+// so the predicate contract can never drift between callers.const DECISION_SCRIPT_PATH = fileURLToPath(new URL("./codex-auth-merge-decision.cjs", import.meta.url),@@ -25,6 +27,10 @@ const SEED_IF_DEST_ABSENT_FLAG = "--seed-if-dest-absent";export const USE_SOURCE_EXIT = 10;/** Exit code: keep the destination credential. */export const KEEP_DESTINATION_EXIT = 20;+/** Exit code: keep the destination; the source `last_refresh` was implausibly far in the future. */+export const IMPLAUSIBLE_LAST_REFRESH_EXIT = 22;++const KNOWN_EXIT_CODES = new Set([USE_SOURCE_EXIT, KEEP_DESTINATION_EXIT, IMPLAUSIBLE_LAST_REFRESH_EXIT]);export interface DecideCodexAuthMergeOptions {/** Opt in to the cache-slot seed mode: fill an absent destination from a@@ -36,9 +42,10 @@ export interface DecideCodexAuthMergeOptions {}/**- * Runs the shared decision predicate and returns its exit code (10 or 20). A- * non-10/20 exit, or a failure to run `node`, is a hard failure: this throws so a- * broken predicate is never mistaken for a "keep destination" decision.+ * Runs the shared decision predicate and returns its exit code (10, 20, or+ * 22). Any other exit, or a failure to run `node`, is a hard failure: this+ * throws so a broken predicate is never mistaken for a "keep destination"+ * decision.*/export async function decideCodexAuthMerge(sourcePath: string,@@ -52,7 +59,7 @@ export async function decideCodexAuthMerge(await execFile("node", args);} catch (error) {const code = (error as { code?: unknown }).code;- if (code === USE_SOURCE_EXIT || code === KEEP_DESTINATION_EXIT) {+ if (typeof code === "number" && KNOWN_EXIT_CODES.has(code)) {return code;}const detail =@@ -65,9 +72,9 @@ export async function decideCodexAuthMerge(: String(error);throw new Error(`${options.errorLabel} decision predicate failed: ${detail}`);}- // `execFile` resolved, so the predicate exited 0. The predicate always exits 10- // or 20, so a clean exit 0 is unexpected; fail loud.+ // `execFile` resolved, so the predicate exited 0. The predicate always exits+ // 10, 20, or 22, so a clean exit 0 is unexpected; fail loud.throw new Error(- `${options.errorLabel} decision predicate exited 0 (expected 10 or 20)`,+ `${options.errorLabel} decision predicate exited 0 (expected 10, 20, or 22)`,);}diff --git a/packages/adapters/codex-local/src/server/codex-auth-merge.test.ts b/packages/adapters/codex-local/src/server/codex-auth-merge.test.tsindex 502c069e94..3d46666793 100644--- a/packages/adapters/codex-local/src/server/codex-auth-merge.test.ts+++ b/packages/adapters/codex-local/src/server/codex-auth-merge.test.ts@@ -596,6 +596,7 @@ describe("codex-auth-merge-decision predicate (source/destination)", () => {const KEEP_DESTINATION = 20;const USE_SOURCE = 10;+ const IMPLAUSIBLE_LAST_REFRESH = 22;function subscriptionAuth(input: {accountId: string;@@ -737,6 +738,18 @@ describe("codex-auth-merge-decision predicate (source/destination)", () => {destinationAuth: "{not valid json",expected: KEEP_DESTINATION,},+ {+ // 400 days ahead is comfortably beyond the 5-minute skew allowance and+ // any subprocess-spawn scheduling delay, so this integration-level+ // check never depends on millisecond timing.+ name: "source last_refresh implausibly far in the future → keep destination",+ sourceAuth: subscriptionAuth({+ accountId: "acct",+ lastRefresh: new Date(Date.now() + 400 * 24 * 60 * 60 * 1000).toISOString(),+ }),+ destinationAuth: subscriptionAuth({ accountId: "acct", lastRefresh: OLDER }),+ expected: IMPLAUSIBLE_LAST_REFRESH,+ },];for (const entry of cases) {@@ -761,4 +774,17 @@ describe("codex-auth-merge-decision predicate (source/destination)", () => {expect(result.code).toBe(USE_SOURCE);expect(result.output).not.toContain("SENTINEL");});++ it("never emits source token bytes for an implausibly-future last_refresh", async () => {+ const result = await runDecision({+ sourceAuth: subscriptionAuth({+ accountId: "acct",+ lastRefresh: new Date(Date.now() + 400 * 24 * 60 * 60 * 1000).toISOString(),+ marker: "SECRET-SENTINEL",+ }),+ destinationAuth: subscriptionAuth({ accountId: "acct", lastRefresh: OLDER }),+ });+ expect(result.code).toBe(IMPLAUSIBLE_LAST_REFRESH);+ expect(result.output).not.toContain("SENTINEL");+ });});diff --git a/server/src/__tests__/agent-hire-ai-connections.test.ts b/server/src/__tests__/agent-hire-ai-connections.test.tsindex c7645acc3c..db0533ad65 100644--- a/server/src/__tests__/agent-hire-ai-connections.test.ts+++ b/server/src/__tests__/agent-hire-ai-connections.test.ts@@ -6,7 +6,7 @@ import express from "express";import request from "supertest";import { and, eq } from "drizzle-orm";import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";-import { agents, companies, companyMemberships, createDb, heartbeatRuns, issues, issueThreadInteractions, principalPermissionGrants, toolConnectionInstalls } from "@paperclipai/db";+import { agents, companies, companyMemberships, createDb, heartbeatRuns, issues, principalPermissionGrants, toolConnectionInstalls } from "@paperclipai/db";import { type AiConnectionBinding } from "@paperclipai/shared";import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";import { agentRoutes } from "../routes/agents.js";@@ -202,47 +202,7 @@ describe("agent-created hires use managed AI connections", () => {});describe("hired agents sharing a subscription", () => {- it("waits for the openai parent and resumes without asking for a new connection", async () => {- const provider = "openai";- const f = await fixture(provider, "subscription");- const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Waiting teammate", role: "engineer", adapterType: f.adapterType, reportsTo: f.agentId, adapterConfig: { cwd: home, engine: "cli" }, runtimeConfig: { heartbeat: { enabled: false } } }));- const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Subscription child task", status: "todo", assigneeAgentId: agent.id, responsibleUserId: f.userId, createdByUserId: f.userId }).returning();- const parentRuntime = await prepareManagedAiRuntime(db, { companyId: f.companyId, agentId: f.agentId, responsibleUserId: f.userId, adapterType: f.adapterType, binding: f.binding, config: { cwd: home } });- const execute = vi.fn(async () => {- await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, issue.id));- return { exitCode: 0, signal: null, timedOut: false, resultJson: {} };- });- registerServerAdapter({ ...getServerAdapter(f.adapterType), execute });- const heartbeat = heartbeatService(db);- let parentReleased = false;- try {- const run = await heartbeat.invoke(agent.id, "assignment", { issueId: issue.id, wakeReason: "issue_assigned", responsibleUserId: f.userId }, "system");- expect(run).not.toBeNull();- await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).not.toBe("running");- expect(await heartbeat.getRun(run!.id)).toMatchObject({ status: "cancelled", errorCode: "ai_connection_busy" });- let retry: typeof heartbeatRuns.$inferSelect | undefined;- await expect.poll(async () => { [retry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run!.id)); return retry?.status; }, { timeout: 10_000 }).toBe("scheduled_retry");- expect(execute).not.toHaveBeenCalled();- expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id))).toHaveLength(0);- expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0].executionRunId).toBe(retry!.id);- parentReleased = true;- await parentRuntime.cleanup();- const promoted = await heartbeat.promoteDueScheduledRetries(new Date(retry!.scheduledRetryAt!.getTime() + 1000));- expect(promoted.runIds).toContain(retry!.id);- await heartbeat.resumeQueuedRuns();- await expect.poll(async () => (await heartbeat.getRun(retry!.id))?.status, { timeout: 20_000 }).toBe("succeeded");- expect(execute).toHaveBeenCalledTimes(1);- expect((await heartbeat.getRun(retry!.id))?.contextSnapshot?.aiConnection).toMatchObject({ connectionId: f.account.connectionId, responsibleUserId: f.userId, method: "subscription" });- } finally {- if (!parentReleased) await parentRuntime.cleanup();- await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));- await heartbeat.drainActiveRunExecutions();- unregisterServerAdapter(f.adapterType);- }- });-- it("runs the anthropic child alongside a live parent and inherits its connection", async () => {- const provider = "anthropic";+ it.each(["openai", "anthropic"] as const)("runs the %s child alongside a live parent and inherits its connection", async (provider) => {const f = await fixture(provider, "subscription");const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Concurrent teammate", role: "engineer", adapterType: f.adapterType, reportsTo: f.agentId, adapterConfig: { cwd: home, engine: "cli" }, runtimeConfig: { heartbeat: { enabled: false } } }));const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Subscription child task", status: "todo", assigneeAgentId: agent.id, responsibleUserId: f.userId, createdByUserId: f.userId }).returning();diff --git a/server/src/__tests__/ai-connections.test.ts b/server/src/__tests__/ai-connections.test.tsindex c36e35ef58..f9d836f8f4 100644--- a/server/src/__tests__/ai-connections.test.ts+++ b/server/src/__tests__/ai-connections.test.ts@@ -8,7 +8,7 @@ import { mkdtemp, rm, access, readFile, writeFile } from "node:fs/promises";import os from "node:os";import path from "node:path";import { and, eq, sql } from "drizzle-orm";-import { createDb, companies, agents, heartbeatRuns, companyMemberships, connectionGrants, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests } from "@paperclipai/db";+import { createDb, companies, agents, heartbeatRuns, companyMemberships, connectionGrants, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets } from "@paperclipai/db";import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";import { aiConnectionService } from "../services/ai-connections.js";import * as executionTarget from "@paperclipai/adapter-utils/execution-target";@@ -282,36 +282,6 @@ describe("managed AI connections", () => {const [agent] = await db.select().from(agents).where(eq(agents.id, agentId));expect(agent.runtimeConfig.aiConnection).toBeUndefined();});- it("serializes OpenAI subscription refresh and releases the lease after execution", async () => {- // OpenAI writes its rotated refresh token back to a shared auth file, so- // two runs against the same grant must not overlap.- const userId = "openai-lease-user";- await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });- const token = JSON.stringify({ tokens: { access_token: "fixture-lease-access", refresh_token: "fixture-lease-refresh", id_token: "fixture-lease-id", account_id: "fixture-lease-account" } });- await service.save(companyId, userId, { provider: "openai", method: "subscription", ownership: "personal", name: "Lease subscription", loginSessionId: "fixture", allAgents: true, agentIds: [] }, token);- const subscription = { ...input, adapterType: "codex_local", binding: { provider: "openai", method: "subscription", mode: "responsible_user" } as const, responsibleUserId: userId, config: { model: "same-model" } };- const first = await prepareManagedAiRuntime(db, subscription);- const selected = await service.select({ ...subscription, userId });- const lockKey = `ai-runtime:${selected.grant.id}`;- const held = await db.execute(sql`- select activity.state, activity.xact_start- from pg_locks locks join pg_stat_activity activity on activity.pid = locks.pid- where locks.locktype = 'advisory' and locks.granted- and locks.classid = (hashtextextended(${lockKey}, 0) >> 32)::int::oid- and locks.objid = (hashtextextended(${lockKey}, 0) & 4294967295)::oid- and locks.objsubid = 1- `);- // A transaction-pooling proxy may move an idle, unpinned client to a- // different backend. The lock must hold a transaction for its lifetime.- expect(held).toHaveLength(1);- expect(held[0].state).toBe("idle in transaction");- expect(held[0].xact_start).not.toBeNull();- await expect(prepareManagedAiRuntime(db, subscription)).rejects.toThrow("in use");- await first.cleanup();- const next = await prepareManagedAiRuntime(db, subscription);- expect(next.identity).toBe(first.identity);- await next.cleanup();- });it("runs two Claude subscription executions for the same grant at the same time", async () => {// Claude writes no auth file back to the grant, so two runs share no// mutable state and must not wait for each other.@@ -325,7 +295,7 @@ describe("managed AI connections", () => {await Promise.all([first.cleanup(), second.cleanup()]);}});- it("reproduces same-agent OpenAI subscription contention and resumes without reconnecting", async () => {+ it("runs a same-agent OpenAI subscription child alongside a still-open parent", async () => {const userId = "subscription-contention-user";await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });const credential = JSON.stringify({ tokens: { access_token: "fixture-access", refresh_token: "fixture-refresh", id_token: "fixture-id", account_id: "fixture-account" } });@@ -341,29 +311,15 @@ describe("managed AI connections", () => {// A parent can create and assign a child before its own execution ends.// Both runs select the same personal subscription, even on the same agent.const parent = await prepareManagedAiRuntime(db, runInput);- try {- await expect(prepareManagedAiRuntime(db, runInput)).rejects.toMatchObject({- status: 422,- message: "This subscription is in use. Retry when its current execution finishes.",- details: { code: "ai_connection_busy" },- });- const selected = await service.select({ ...runInput, userId });- expect(selected.grant).toMatchObject({ id: account.grantId, status: "active" });- expect(await service.credential(selected)).toBe(credential);- } finally {- await parent.cleanup();- }- // Releasing the parent alone is sufficient: no reconnect, credential- // rotation, account switch, or agent configuration change is required.const child = await prepareManagedAiRuntime(db, runInput);try {expect(child.identity).toBe(parent.identity);expect(child.attribution.grantId).toBe(account.grantId);} finally {- await child.cleanup();+ await Promise.all([parent.cleanup(), child.cleanup()]);}});- it("persists refreshed credentials only to their original grant and fences reconnects", async () => {+ it("persists the freshest refreshed credential to its original grant across a same-account reconnect", async () => {const auth = (marker: string, hour: number) => JSON.stringify({ tokens: { account_id: "fixture-account", id_token: `id-${marker}`, access_token: `access-${marker}`, refresh_token: `refresh-${marker}` }, last_refresh: `2026-09-10T${hour}:00:00Z` });const intent = { provider: "openai" as const, method: "subscription" as const, name: "Refresh test", ownership: "personal" as const, agentIds: [], allAgents: true, loginSessionId: "fixture" };const saved = await service.save(companyId, "alice", intent, auth("first", 10));@@ -375,10 +331,125 @@ describe("managed AI connections", () => {expect(await service.credential(selected)).toBe(auth("refreshed", 11));const second = await prepareManagedAiRuntime(db, runInput);expect(second.identity).not.toBe(first.identity);+ // A same-account reconnect writes an older last_refresh than the run+ // that is still open.await service.save(companyId, "alice", { ...intent, connectionId: saved.connectionId }, auth("reconnect", 12));- await writeFile(path.join(String(second.config.env.CODEX_HOME), "auth.json"), auth("stale-process", 13));+ await writeFile(path.join(String(second.config.env.CODEX_HOME), "auth.json"), auth("later-refresh", 13));await second.cleanup();- expect(await service.credential(await service.select({ ...runInput, userId: "alice" }))).toBe(auth("reconnect", 12));+ // The newer refresh persists to the grant it started from.+ expect(await service.credential(await service.select({ ...runInput, userId: "alice" }))).toBe(auth("later-refresh", 13));+ });+ it("resolves two concurrent OpenAI subscription write-backs by freshness, not by order", async () => {+ const userId = "concurrent-freshness-user";+ await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });+ const auth = (marker: string, hour: number) => JSON.stringify({ tokens: { account_id: "fixture-account", id_token: `id-${marker}`, access_token: `access-${marker}`, refresh_token: `refresh-${marker}` }, last_refresh: `2026-09-10T${hour}:00:00Z` });+ await service.save(companyId, userId, { provider: "openai", method: "subscription", ownership: "personal", name: "Freshness fixture", loginSessionId: "fixture", allAgents: true, agentIds: [] }, auth("start", 10));+ const runInput = { ...input, adapterType: "codex_local", responsibleUserId: userId, binding: { provider: "openai", method: "subscription", mode: "responsible_user" } as const, config: { model: "same-model" } };+ // Two runs use the same OpenAI subscription grant at the same time.+ // Neither call below throws ai_connection_busy.+ const older = await prepareManagedAiRuntime(db, runInput);+ const newer = await prepareManagedAiRuntime(db, runInput);+ await writeFile(path.join(String(older.config.env.CODEX_HOME), "auth.json"), auth("older", 11));+ await writeFile(path.join(String(newer.config.env.CODEX_HOME), "auth.json"), auth("newer", 12));+ // The run with the newer last_refresh writes back first. The run with+ // the older last_refresh writes back last and must not overwrite it.+ await newer.cleanup();+ await older.cleanup();+ const stored = await service.credential(await service.select({ ...runInput, userId }));+ expect(stored).toBe(auth("newer", 12));+ });+ it("resolves two concurrent OpenAI subscription write-backs by freshness in reverse arrival order", async () => {+ const userId = "concurrent-freshness-reverse-user";+ await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });+ const auth = (marker: string, hour: number) => JSON.stringify({ tokens: { account_id: "fixture-account", id_token: `id-${marker}`, access_token: `access-${marker}`, refresh_token: `refresh-${marker}` }, last_refresh: `2026-09-10T${hour}:00:00Z` });+ await service.save(companyId, userId, { provider: "openai", method: "subscription", ownership: "personal", name: "Reverse freshness fixture", loginSessionId: "fixture", allAgents: true, agentIds: [] }, auth("start", 10));+ const runInput = { ...input, adapterType: "codex_local", responsibleUserId: userId, binding: { provider: "openai", method: "subscription", mode: "responsible_user" } as const, config: { model: "same-model" } };+ // Two runs use the same OpenAI subscription grant at the same time.+ // Neither call below throws ai_connection_busy.+ const older = await prepareManagedAiRuntime(db, runInput);+ const newer = await prepareManagedAiRuntime(db, runInput);+ await writeFile(path.join(String(older.config.env.CODEX_HOME), "auth.json"), auth("older", 11));+ await writeFile(path.join(String(newer.config.env.CODEX_HOME), "auth.json"), auth("newer", 12));+ // The run with the older last_refresh writes back first. The run with+ // the newer last_refresh writes back last and must win.+ await older.cleanup();+ await newer.cleanup();+ const stored = await service.credential(await service.select({ ...runInput, userId }));+ expect(stored).toBe(auth("newer", 12));+ });+ it("resolves two concurrent xAI subscription write-backs by freshness, not by order", async () => {+ const userId = "concurrent-freshness-xai-user";+ await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });+ const identityKey = "https://auth.x.ai::33333333-3333-3333-3333-333333333333";+ const auth = (marker: string, expiresAtMs: number) => JSON.stringify({ [identityKey]: { key: `key-${marker}`, refresh_token: `refresh-${marker}`, expires_at: new Date(expiresAtMs).toISOString() } });+ const now = Date.now();+ await service.save(companyId, userId, { provider: "xai", method: "subscription", ownership: "personal", name: "Grok freshness fixture", loginSessionId: "fixture", allAgents: true, agentIds: [] }, auth("start", now));+ const runInput = { ...input, adapterType: "grok_local", responsibleUserId: userId, binding: { provider: "xai", method: "subscription", mode: "responsible_user" } as const, config: { model: "same-model" } };+ // Two runs use the same xAI subscription grant at the same time.+ // Neither call below throws ai_connection_busy.+ const older = await prepareManagedAiRuntime(db, runInput);+ const newer = await prepareManagedAiRuntime(db, runInput);+ await writeFile(path.join(String(older.config.env.GROK_HOME), "auth.json"), auth("older", now + 60 * 60 * 1000));+ await writeFile(path.join(String(newer.config.env.GROK_HOME), "auth.json"), auth("newer", now + 2 * 60 * 60 * 1000));+ // The run with the older expiry writes back first. The run with the+ // newer expiry writes back last and must win.+ await older.cleanup();+ await newer.cleanup();+ const stored = await service.credential(await service.select({ ...runInput, userId }));+ expect(stored).toBe(auth("newer", now + 2 * 60 * 60 * 1000));+ });+ it("discards a credential write-back when the grant is revoked while the run is open", async () => {+ const userId = "revoked-write-back-user";+ await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });+ const auth = (marker: string, hour: number) => JSON.stringify({ tokens: { account_id: "fixture-account", id_token: `id-${marker}`, access_token: `access-${marker}`, refresh_token: `refresh-${marker}` }, last_refresh: `2026-09-10T${hour}:00:00Z` });+ const saved = await service.save(companyId, userId, { provider: "openai", method: "subscription", ownership: "personal", name: "Revocation fixture", loginSessionId: "fixture", allAgents: true, agentIds: [] }, auth("start", 10));+ const runInput = { ...input, adapterType: "codex_local", responsibleUserId: userId, binding: { provider: "openai", method: "subscription", mode: "responsible_user" } as const, config: { model: "same-model" } };+ const run = await prepareManagedAiRuntime(db, runInput);+ const [grantBeforeCleanup] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, saved.grantId));+ const ref = grantBeforeCleanup.credentialSecretRefs.find(r => r.configPath === "ai.credential")!;+ const [secretBefore] = await db.select().from(companySecrets).where(eq(companySecrets.id, ref.secretId));+ // A newer last_refresh would win the freshness merge if the grant stayed+ // active. The revoked grant must discard the write-back before that merge+ // decides anything.+ await writeFile(path.join(String(run.config.env.CODEX_HOME), "auth.json"), auth("revoked-run", 11));+ await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, saved.grantId));+ await run.cleanup();+ const [secretAfter] = await db.select().from(companySecrets).where(eq(companySecrets.id, ref.secretId));+ // service.select rejects a revoked grant, so it cannot read the stored+ // credential here. Compare the stored secret version directly instead.+ expect(secretAfter.latestVersion).toBe(secretBefore.latestVersion);+ });+ it("does not let a stale write-back overwrite an authorized secret rotation that commits while cleanup waits on the credential lock", async () => {+ const userId = "credential-lock-race-user";+ await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });+ const auth = (marker: string, hour: number) => JSON.stringify({ tokens: { account_id: "fixture-account", id_token: `id-${marker}`, access_token: `access-${marker}`, refresh_token: `refresh-${marker}` }, last_refresh: `2026-09-10T${hour}:00:00Z` });+ const saved = await service.save(companyId, userId, { provider: "openai", method: "subscription", ownership: "personal", name: "Credential lock race fixture", loginSessionId: "fixture", allAgents: true, agentIds: [] }, auth("start", 10));+ const runInput = { ...input, adapterType: "codex_local", responsibleUserId: userId, binding: { provider: "openai", method: "subscription", mode: "responsible_user" } as const, config: { model: "same-model" } };+ const run = await prepareManagedAiRuntime(db, runInput);+ // The run's own refresh looks newer than the value it started with, but+ // it must lose to a company-authorized rotation that commits while+ // cleanup is still waiting on the credential secret's row lock.+ await writeFile(path.join(String(run.config.env.CODEX_HOME), "auth.json"), auth("run-refresh", 11));+ const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, saved.grantId));+ const ref = grant.credentialSecretRefs.find(r => r.configPath === "ai.credential")!;+ let holdAcquired!: () => void;+ const holdAcquiredPromise = new Promise<void>(resolve => { holdAcquired = resolve; });+ let releaseHold!: () => void;+ const holdReleased = new Promise<void>(resolve => { releaseHold = resolve; });+ // An authorized rotation writes the new credential inside its own open+ // transaction, so it still holds the secret row's lock when signaled.+ const holder = db.transaction(async tx => {+ await secretService(tx).rotate(ref.secretId, { value: auth("authorized-rotation", 12) }, { userId });+ holdAcquired();+ await holdReleased;+ });+ await holdAcquiredPromise;+ const cleanupPromise = run.cleanup();+ releaseHold();+ await holder;+ await cleanupPromise;+ const stored = await service.credential(await service.select({ ...runInput, userId }));+ expect(stored).toBe(auth("authorized-rotation", 12));});it("enforces the shared transport discriminator and existing harness compatibility", () => {expect(connectionPurposeTransportSchema.safeParse({ connectionPurpose: "ai", transport: "mcp_remote" }).success).toBe(false);diff --git a/server/src/__tests__/heartbeat-ai-subscription-contention.test.ts b/server/src/__tests__/heartbeat-ai-subscription-contention.test.tsdeleted file mode 100644index 570506083f..0000000000--- a/server/src/__tests__/heartbeat-ai-subscription-contention.test.ts+++ /dev/null@@ -1,250 +0,0 @@-import { randomUUID } from "node:crypto";-import { mkdtemp, rm } from "node:fs/promises";-import os from "node:os";-import path from "node:path";-import { eq } from "drizzle-orm";-import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";-import {- agents, companies, companyMemberships, connectionGrants, createDb, heartbeatRuns,- issueComments, issueRecoveryActions, issueThreadInteractions, issues,-} from "@paperclipai/db";-import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";-import { heartbeatService } from "../services/heartbeat.js";-import { aiConnectionService } from "../services/ai-connections.js";-import { prepareManagedAiRuntime } from "../services/ai-connection-runtime.js";-import { executionFailureRetryCount } from "../services/execution-recovery-attempt.js";-import { executionProjectionsForRuns } from "../services/execution-projection.js";-import type { AdapterExecutionContext } from "../adapters/index.js";--const execute = vi.hoisted(() => vi.fn(async (_input: AdapterExecutionContext) => ({- exitCode: 0, signal: null, timedOut: false, resultJson: {}, summary: "Fixture work completed.",-})));-vi.mock("../adapters/index.js", async () => ({- ...await vi.importActual<typeof import("../adapters/index.js")>("../adapters/index.js"),- getServerAdapter: () => ({ supportsLocalAgentJwt: false, execute }),-}));--const afterCheckout = vi.hoisted(() => vi.fn(async () => {}));-vi.mock("../services/issues.js", async () => {- const actual = await vi.importActual<typeof import("../services/issues.js")>("../services/issues.js");- return { ...actual, issueService: (...args: Parameters<typeof actual.issueService>) => {- const service = actual.issueService(...args);- return { ...service, checkout: async (...checkoutArgs: Parameters<typeof service.checkout>) => {- const result = await service.checkout(...checkoutArgs);- await afterCheckout();- return result;- } };- } };-});--describe("heartbeat AI subscription contention", () => {- let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;- let db: ReturnType<typeof createDb>;- let heartbeat: ReturnType<typeof heartbeatService>;- let home: string;-- beforeAll(async () => {- home = await mkdtemp(path.join(os.tmpdir(), "paperclip-subscription-contention-"));- vi.stubEnv("PAPERCLIP_HOME", home);- vi.stubEnv("PAPERCLIP_INSTANCE_ID", "subscription-contention-tests");- database = await startEmbeddedPostgresTestDatabase("paperclip-subscription-contention-db-");- db = createDb(database.connectionString);- heartbeat = heartbeatService(db);- execute.mockImplementation(async (input) => {- await db.update(issues).set({ status: "done" }).where(eq(issues.id, String(input.context.issueId)));- return { exitCode: 0, signal: null, timedOut: false, resultJson: {}, summary: "Fixture work completed." };- });- }, 90_000);- afterAll(async () => {- await heartbeat?.drainActiveRunExecutions();- await database?.cleanup();- vi.unstubAllEnvs();- if (home) await rm(home, { recursive: true, force: true });- });-- async function fixture() {- execute.mockClear();- const companyId = randomUUID();- const agentId = randomUUID();- const issueId = randomUUID();- const userId = randomUUID();- await db.insert(companies).values({ id: companyId, name: "Contention fixture", issuePrefix: `S${companyId.slice(0, 8).toUpperCase()}`, defaultResponsibleUserId: userId });- await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });- const binding = { provider: "openai", method: "subscription", mode: "responsible_user" } as const;- await db.insert(agents).values({ id: agentId, companyId, name: "CEO", adapterType: "codex_local", adapterConfig: { cwd: home }, runtimeConfig: { aiConnection: binding, heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 20 } } });- await db.insert(issues).values({ id: issueId, companyId, title: "Child task", status: "todo", assigneeAgentId: agentId, responsibleUserId: userId });- const account = await aiConnectionService(db).save(companyId, userId, { provider: "openai", method: "subscription", ownership: "personal", name: "Fixture subscription", loginSessionId: "fixture", allAgents: true, agentIds: [] }, JSON.stringify({ tokens: { access_token: "fixture-access", refresh_token: "fixture-refresh", id_token: "fixture-id", account_id: "fixture-account" } }));- const parent = await prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: userId, adapterType: "codex_local", binding, config: { cwd: home } });- let released = false;- const release = async () => {- if (released) return;- released = true;- await parent.cleanup();- };- return { companyId, agentId, issueId, userId, account, parent, release };- }-- async function defer(f: Awaited<ReturnType<typeof fixture>>, context: Record<string, unknown> = {}) {- const run = await heartbeat.invoke(f.agentId, "assignment", {- issueId: f.issueId, wakeReason: "issue_assigned", ...context,- }, "system");- expect(run).not.toBeNull();- await heartbeat.drainActiveRunExecutions();- expect(await heartbeat.getRun(run!.id)).toMatchObject({ status: "cancelled", errorCode: "ai_connection_busy" });- const [retry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run!.id));- expect(retry).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy" });- return retry!;- }-- async function dispatch(retry: typeof heartbeatRuns.$inferSelect, service = heartbeat) {- await service.promoteDueScheduledRetries(new Date(retry.scheduledRetryAt!.getTime() + 1));- await service.resumeQueuedRuns();- await service.drainActiveRunExecutions();- }-- it("defers a child without a reconnect blocker and executes after its parent's subscription is released", async () => {- const f = await fixture();- try {- const run = await heartbeat.invoke(f.agentId, "assignment", { issueId: f.issueId, wakeReason: "issue_assigned" }, "system");- expect(run).not.toBeNull();- await heartbeat.drainActiveRunExecutions();- const deferred = await heartbeat.getRun(run!.id);- expect(deferred).toMatchObject({ status: "cancelled", errorCode: "ai_connection_busy" });- expect(execute).not.toHaveBeenCalled();- const [retry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run!.id));- expect(retry).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy" });- expect(executionFailureRetryCount(retry)).toBe(0);- const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));- expect(issue.status).not.toBe("blocked");- expect(issue.executionRunId).toBe(retry.id);- expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId))).toHaveLength(0);- expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.companyId, f.companyId))).toHaveLength(0);- expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issueId))).toHaveLength(0);- await f.release();- await heartbeat.promoteDueScheduledRetries(new Date(retry.scheduledRetryAt!.getTime() + 1));- await heartbeat.resumeQueuedRuns();- await heartbeat.drainActiveRunExecutions();- const completed = await heartbeat.getRun(retry.id);- expect(completed).toMatchObject({ status: "succeeded" });- expect(execute).toHaveBeenCalledTimes(1);- expect(completed?.contextSnapshot?.aiConnection).toMatchObject({ grantId: f.account.grantId, identity: f.parent.identity });- } finally {- await f.release();- }- });-- it("keeps waiting beyond the failure-attempt limit and survives a new service instance", async () => {- const f = await fixture();- try {- let retry = await defer(f, {- failureRetriesBeforeAiConnectionWait: 99, aiConnectionBusyDeferredWhileAssignee: false,- });- expect(retry.contextSnapshot?.aiConnectionBusyDeferredWhileAssignee).toBe(true);- for (let count = 0; count < 3; count++) {- await dispatch(retry);- const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, retry.id));- expect(successor).toMatchObject({ status: "scheduled_retry", scheduledRetryAttempt: count + 2 });- expect(executionFailureRetryCount(successor)).toBe(0);- retry = successor!;- const projections = await executionProjectionsForRuns(db, f.companyId, [retry.id]);- expect(projections.get(retry.id)).toMatchObject({- phase: "retry_scheduled", label: "Waiting for AI subscription", attempt: 1,- });- }- expect(execute).not.toHaveBeenCalled();- await f.release();- await dispatch(retry, heartbeatService(db));- expect(await heartbeat.getRun(retry.id)).toMatchObject({ status: "succeeded" });- expect(execute).toHaveBeenCalledTimes(1);- } finally {- await f.release();- }- });-- it.each([- ["cancelled", "issue_cancelled"],- ["done", "issue_terminal_status"],- ["reassigned", "issue_reassigned"],- ["agent_paused", "agent_not_invokable"],- ])("does not dispatch after the waiting task is %s", async (change, errorCode) => {- const f = await fixture();- try {- const retry = await defer(f);- if (change === "agent_paused") {- await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));- } else {- await db.update(issues).set(change === "reassigned"- ? { assigneeAgentId: null, assigneeUserId: f.userId }- : { status: change }).where(eq(issues.id, f.issueId));- }- await f.release();- await dispatch(retry);- expect(await heartbeat.getRun(retry.id)).toMatchObject({ status: "cancelled", errorCode });- expect(execute).not.toHaveBeenCalled();- const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));- expect(issue.executionRunId).toBeNull();- } finally {- await f.release();- }- });-- it.each(["issue_assigned", "issue_commented"])("does not turn an assignee %s wake into a non-assignee retry after reassignment during preflight", async (wakeReason) => {- const f = await fixture();- try {- afterCheckout.mockImplementationOnce(async () => {- await db.update(issues).set({- assigneeAgentId: null, assigneeUserId: f.userId, executionRunId: null,- }).where(eq(issues.id, f.issueId));- });- const commentId = randomUUID();- await db.insert(issueComments).values({ id: commentId, companyId: f.companyId, issueId: f.issueId,- authorType: "user", authorUserId: f.userId, body: "Continue this task." });- const run = await heartbeat.invoke(f.agentId, "assignment", {- issueId: f.issueId, wakeReason, commentId, aiConnectionBusyDeferredWhileAssignee: false,- }, "system");- await heartbeat.drainActiveRunExecutions();- expect(await heartbeat.getRun(run!.id)).toMatchObject({ status: "cancelled", errorCode: "ai_connection_busy" });- expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run!.id))).toHaveLength(0);- expect(execute).not.toHaveBeenCalled();- } finally {- afterCheckout.mockReset();- await f.release();- }- });-- it("resumes an authorized comment wake without stealing the assignee's task lock", async () => {- const f = await fixture();- try {- await db.update(issues).set({ assigneeAgentId: null, assigneeUserId: f.userId }).where(eq(issues.id, f.issueId));- const retry = await defer(f, { wakeReason: "issue_comment_mentioned", commentId: randomUUID() });- expect(retry.contextSnapshot?.aiConnectionBusyDeferredWhileAssignee).toBe(false);- await dispatch(retry);- const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, retry.id));- expect(successor).toMatchObject({ status: "scheduled_retry" });- expect(successor.contextSnapshot?.aiConnectionBusyDeferredWhileAssignee).toBe(false);- const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));- expect(issue.executionRunId).toBeNull();- await f.release();- await dispatch(successor);- expect(await heartbeat.getRun(successor.id)).toMatchObject({ status: "succeeded" });- expect(execute).toHaveBeenCalledTimes(1);- } finally {- await f.release();- }- });-- it("rechecks revoked credentials after waiting and retains the configuration failure", async () => {- const f = await fixture();- try {- const retry = await defer(f);- await f.release();- await db.update(connectionGrants).set({ status: "revoked", revokedAt: new Date() }).where(eq(connectionGrants.id, f.account.grantId));- await dispatch(retry);- expect(await heartbeat.getRun(retry.id)).toMatchObject({ status: "failed", errorCode: "configuration_incomplete" });- expect(execute).not.toHaveBeenCalled();- expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, retry.id))).toHaveLength(0);- } finally {- await f.release();- }- });-});diff --git a/server/src/__tests__/secrets-service.test.ts b/server/src/__tests__/secrets-service.test.tsindex dbc6050fdf..186af5a993 100644--- a/server/src/__tests__/secrets-service.test.ts+++ b/server/src/__tests__/secrets-service.test.ts@@ -5,7 +5,7 @@ import os from "node:os";import path from "node:path";import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";import type { MockInstance } from "vitest";-import { and, eq } from "drizzle-orm";+import { and, eq, sql } from "drizzle-orm";import { resolveCodexAuthCacheDir, withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server";import {activityLog,@@ -674,6 +674,79 @@ describeEmbeddedPostgres("secretService", () => {expect(events).toEqual(["rotate-provider-enter", "rotate-provider-exit", "create-provider-enter"]);});+ it("locks the parent secret row before its version rows during rotate, so a concurrent credential write-back cannot deadlock with it", async () => {+ // Inserting the rotate's new version row needs a read lock on the+ // parent secret row to satisfy the version table's foreign key, so a+ // holder that locks the parent row first would block that insert and+ // never reach the statements this test distinguishes. Lock the secret's+ // existing version row instead: that does not touch the parent row, so+ // the rotate call's insert proceeds and the call reaches its final+ // transaction, where it then queues behind this same held row for its+ // "mark the old version previous" update.+ const companyId = await seedCompany();+ const svc = secretService(db);+ const secret = await svc.create(companyId, {+ name: `lock-order-${randomUUID()}`,+ provider: "local_encrypted",+ value: "original-value",+ });++ const holderPidReady = deferred<number>();+ let releaseHolder!: () => void;+ const holderReleased = new Promise<void>((resolve) => {+ releaseHolder = resolve;+ });+ const holder = db.transaction(async (tx) => {+ await tx+ .select({ id: companySecretVersions.id })+ .from(companySecretVersions)+ .where(and(eq(companySecretVersions.secretId, secret.id), eq(companySecretVersions.version, 1)))+ .for("update");+ const [backend] = (await tx.execute(sql`select pg_backend_pid() as pid`)) as unknown as Array<{ pid: number }>;+ holderPidReady.resolve(backend.pid);+ await holderReleased;+ });+ const holderPid = await holderPidReady.promise;++ const rotateCall = svc.rotate(secret.id, { value: "rotated-value" });++ // Poll until Postgres reports the rotate call's own backend blocked+ // behind the lock holder above, instead of guessing how long that+ // takes.+ let rotateBlocked = false;+ for (let attempt = 0; attempt < 200; attempt++) {+ const rows = (await db.execute(+ sql`select 1 from pg_stat_activity where ${holderPid} = any(pg_blocking_pids(pid))`,+ )) as unknown as Array<unknown>;+ if (rows[0]) {+ rotateBlocked = true;+ break;+ }+ await new Promise((resolve) => setTimeout(resolve, 10));+ }+ try {+ expect(rotateBlocked).toBe(true);+ // The rotate call is confirmed blocked on the existing version row.+ // If it had updated that row before the parent secret row — the+ // opposite order this fix removes — the parent row would still be+ // free here. A `nowait` probe from a third transaction settles which+ // is true: it fails the instant the parent row already carries an+ // uncommitted write from the blocked rotate call, and this fix makes+ // that write happen first.+ const probe = await db+ .transaction(async (tx) => {+ await tx.execute(sql`select 1 from company_secrets where id = ${secret.id} for update nowait`);+ })+ .then(() => null)+ .catch((error: unknown) => error);+ expect(String((probe as { cause?: unknown })?.cause ?? probe)).toMatch(/could not obtain lock/i);+ } finally {+ releaseHolder();+ }+ await holder;+ await expect(rotateCall).resolves.toMatchObject({ latestVersion: 2 });+ });+it("fails a queued local_encrypted create when an account-home cleanup removes its directory first", async () => {// The mutation lock alone stops a write and an account-home cleanup's// check-and-delete from interleaving; it does not stop them from runningdiff --git a/server/src/services/ai-connection-runtime.ts b/server/src/services/ai-connection-runtime.tsindex 0f304a9118..1ee19dcf08 100644--- a/server/src/services/ai-connection-runtime.ts+++ b/server/src/services/ai-connection-runtime.ts@@ -4,7 +4,7 @@ import { mkdtemp, mkdir, writeFile, readFile, rm } from "node:fs/promises";import os from "node:os";import path from "node:path";import { and, eq } from "drizzle-orm";-import { type Db, connectionGrants } from "@paperclipai/db";+import { type Db, companySecrets, connectionGrants } from "@paperclipai/db";import {AI_CONNECTION_CAPABILITIES,type AiConnectionBinding,@@ -166,41 +166,6 @@ done`,}}-async function acquireCredentialLease(db: Db, grantId: string) {- const client = await db.$client.reserve();- try {- // A reserved client pins our connection to PgBouncer, not its backend.- // Keep the lease in one transaction so transaction-pooling deployments- // cannot acquire and release it on different PostgreSQL sessions.- await client`begin`;- await client`set local idle_in_transaction_session_timeout = 0`;- const [result] =- await client`select pg_try_advisory_xact_lock(hashtextextended(${`ai-runtime:${grantId}`}, 0)) as acquired`;- if (!result.acquired)- throw unprocessable(- "This subscription is in use. Retry when its current execution finishes.",- { code: "ai_connection_busy" },- );- } catch (error) {- try {- await client`rollback`;- } finally {- client.release();- }- throw error;- }- let released = false;- return async () => {- if (released) return;- released = true;- try {- await client`rollback`;- } finally {- client.release();- }- };-}-export async function prepareManagedAiRuntime(db: Db,input: {@@ -246,12 +211,6 @@ export async function prepareManagedAiRuntime(const subscriptionFile =selection.attribution.method === "subscription" &&input.binding.provider !== "anthropic";- // A file-backed subscription runs one credential rotation at a time, so it- // needs the lease. Anthropic writes no file back, so two runs share no- // mutable state and can run at the same time without the lease.- const release = subscriptionFile- ? await acquireCredentialLease(db, selection.grant.id)- : async () => {};let home: string | undefined;try {const selectedGrantId = selection.grant.id;@@ -344,18 +303,32 @@ export async function prepareManagedAiRuntime(),).for("update");- // Reconnect/revocation wins over a process holding an older credential.- if (- !grant ||- grant.status !== "active" ||- grant.updatedAt.getTime() !==- selection.grant.updatedAt.getTime()- )- return;- const current = await service.credential({- ...selection,- grant,- });+ // A missing or revoked grant blocks the write-back. Among+ // active copies, the merge decision below keeps the+ // credential with the newest provider freshness field.+ if (!grant || grant.status !== "active") return;+ const ref = grant.credentialSecretRefs.find(+ (r) => r.configPath === "ai.credential",+ );+ if (!ref) return;+ // Lock the referenced secret row for the rest of this+ // transaction. The grant-row lock above does not cover it,+ // so an authorized rotation of this secret could otherwise+ // land between the read and the write below and be+ // overwritten by this stale write-back.+ await tx+ .select({ id: companySecrets.id })+ .from(companySecrets)+ .where(+ and(+ eq(companySecrets.id, ref.secretId),+ eq(companySecrets.companyId, input.companyId),+ ),+ )+ .for("update");+ const current = await aiConnectionService(+ tx as unknown as Db,+ ).credential({ ...selection, grant });const destination = path.join(providerHome,"current-auth.json",@@ -370,9 +343,6 @@ export async function prepareManagedAiRuntime(errorLabel: "AI account refresh",});if (decision !== 10) return;- const ref = grant.credentialSecretRefs.find(- (r) => r.configPath === "ai.credential",- )!;await secretService(tx).rotate(ref.secretId,{ value: refreshed },@@ -385,20 +355,12 @@ export async function prepareManagedAiRuntime(});}} finally {- try {- if (home) await rm(home, { recursive: true, force: true });- } finally {- await release();- }+ if (home) await rm(home, { recursive: true, force: true });}},};} catch (error) {- try {- if (home) await rm(home, { recursive: true, force: true });- } finally {- await release();- }+ if (home) await rm(home, { recursive: true, force: true });throw error;}}diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.tsindex b985426deb..58cab31a15 100644--- a/server/src/services/heartbeat.ts+++ b/server/src/services/heartbeat.ts@@ -16037,10 +16037,9 @@ export function heartbeatService(.then((rows) => rows[0] ?? null);}- // Subscription homes are serialized so refresh tokens cannot be overwritten- // by another run. Contention is a resource wait, not broken authentication.- // The database lease is released on completion/disconnect; retrying remains- // safe across processes and each attempt revalidates the selected account.+ // No code path raises the `ai_connection_busy` error any more. This+ // function stays because a stored run row can still carry that error code+ // from an earlier release.async function finalizeAiConnectionBusyDeferral(run: typeof heartbeatRuns.$inferSelect,error: HttpError,diff --git a/server/src/services/secrets.ts b/server/src/services/secrets.tsindex c5facc4964..143275d094 100644--- a/server/src/services/secrets.ts+++ b/server/src/services/secrets.ts@@ -2522,22 +2522,14 @@ export function secretService(db: Db | DbTransaction) {}try {+ // Lock and update the parent secret row before touching its version+ // rows. A credential write-back locks the parent secret row first, then+ // calls this function. If this transaction updated the version rows+ // first instead, the two transactions would take their two row locks+ // in opposite order and could deadlock. Matching the order here removes+ // that risk: every caller now locks the parent row before the version+ // rows, so a lock cycle between the two tables cannot form.return await db.transaction(async (tx) => {- await tx- .update(companySecretVersions)- .set({ status: "previous" })- .where(and(- eq(companySecretVersions.secretId, secret.id),- ne(companySecretVersions.version, nextVersion),- ));- await tx- .update(companySecretVersions)- .set({ status: "current" })- .where(and(- eq(companySecretVersions.secretId, secret.id),- eq(companySecretVersions.version, nextVersion),- ));-const updated = await tx.update(companySecrets).set({@@ -2561,11 +2553,28 @@ export function secretService(db: Db | DbTransaction) {if (!updated) {// The predicate matched no row. A supplied expected version means a// concurrent rotation won the race; return the stale conflict. An- // unguarded rotation means the secret is gone.+ // unguarded rotation means the secret is gone. Neither version row+ // has been touched yet, so there is nothing to undo here.throw input.expectedLatestVersion === undefined? notFound("Secret not found"): conflict(SECRET_VERSION_STALE_CONFLICT);}++ await tx+ .update(companySecretVersions)+ .set({ status: "previous" })+ .where(and(+ eq(companySecretVersions.secretId, secret.id),+ ne(companySecretVersions.version, nextVersion),+ ));+ await tx+ .update(companySecretVersions)+ .set({ status: "current" })+ .where(and(+ eq(companySecretVersions.secretId, secret.id),+ eq(companySecretVersions.version, nextVersion),+ ));+return updated;});} catch (error) {