Enhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active
TypeScript aianthropicartifactsawsazurechatgptchatgpt-cloneclaudeclonedeepseekgeminigooglegpt-5librechatmcpo1openairesponses-apivisionwebui
🪧 fix: Advertise Detached Event Actor Support from the Generation Store (#15322)
Danny Avila77c2a51cf34e
diff --git a/.env.example b/.env.exampleindex 1c21df2889..9b9cea96ed 100644--- a/.env.example+++ b/.env.example@@ -197,10 +197,6 @@ LOG_TO_FILE=true# Set to true to enable agent debug loggingAGENT_DEBUG_LOGGING=false-# Rolling-deployment fence for durable detached Event Actor completions. Requires Redis generation-# streams. Leave disabled while deploying consumers, then enable after every replica is updated.-# AGENT_TRIGGERS_DETACHED_ACTIONS_PRODUCER_ENABLED=false-# Enable memory diagnostics (logs heap/RSS snapshots every 60s, auto-enabled with --inspect)# MEM_DIAG=truediff --git a/CONTEXT.md b/CONTEXT.mdindex 6bac598b1d..71c192613a 100644--- a/CONTEXT.md+++ b/CONTEXT.md@@ -15,4 +15,5 @@- **Event actor invocation fork**: a delivery-owned checkpoint namespace copied from the event actor head. A warm invocation receives only the new trusted event, then commits its terminal checkpoint when the expected action is observed or deletes the fork otherwise. When the invocation pauses for approval or Ask User, the SDK emits signed, versioned suspension evidence. The child Conversation is the canonical one-shot suspension authority; the generation job carries only a versioned projection for UI, rolling-deploy routing, and the existing resume endpoint. Shared-store deployments publish new suspensions only under generation protocol v2, after the homogeneous-fleet cutover; protocol v1 keeps pause-capable actors on the legacy history path so an old resume consumer cannot consume unknown evidence. A resume shares one identity between its Conversation claim and provider-owner CAS, clears the predecessor projection, and publishes a successor only after a re-pause is canonical. A pending interrupt takes precedence over expected-action evidence from the same segment; if that segment already applied the expected action, publishing its successor pause cold-marks the prior head until a later applied commit replaces it. An ambiguous projection write is accepted only after reading back the exact generation, action, and suspension. The provider-start CAS is written only after client reconstruction and immediately before the continuation gate opens, then retains its exact execution identity after drain, so terminal recovery can compensate a projected claim only when that identity proves execution never began. Durable approval projection is exposed before the persistence barrier opens, preventing a resolved action from being announced afterward. Terminal no-action retirement cancels or settles the exact suspension and releases its delivery-side action admission before public settlement; if retention already removed the child Conversation, the delivery remains authoritative for its exact admission identity. Resume, re-pause, cancellation, and expiry claim or replace that exact suspension before touching its job projection, so later mailbox deliveries stay blocked until terminal history and handling evidence settle.- **Event actor receipt**: the private, terminal proof stored on the authoritative `AgentTriggerDelivery` row for one bound actor invocation. Its unique delivery identity, terminal resolution, exact checkpoint, and bounded action identity provide replay and recovery for the retention window without storing prompts, events, tool arguments, tool output, or conversation history. It does not own the actor checkpoint; the conversation keeps only the actor head and any active unresolved reconciliation until this receipt is durable.- **Agent event actor mailbox**: the automatic durable delivery-ordering lane for one authenticated source binding. It keeps later deliveries queued after transport admission until the current child turn records an authoritative terminal handling outcome. It serializes existing coalesced batches and individual events without becoming a second execution controller or actor checkpoint store.+- **Agent trigger capability shield**: the durable mixed-version representation for internal trigger work that only a capability-aware worker may execute. Mongo uses an old-publishable `staging` shell; a queued `leased` shell without an owner or deadline, which old workers cannot claim but can use for bounded lane rechecks; a private lease only during execution; and a legacy-terminal `capability_dead` shell once dead. Private capability fields own current claiming, retry, and dead-letter truth. Redis uses a versioned fail-closed terminal status and recovery index that old replacement scripts and sweepers cannot consume. The shield is an implementation detail at the storage seam, never a deployment switch or user-configured product mode.- **Theme definition**: a versioned, data-only description of LibreChat semantic colors and shared appearance roles, optionally specialized by light or dark mode. The theme module validates and resolves partial definitions against bundled defaults before adapters apply them. A theme definition does not contain arbitrary CSS, application behavior, or alternate feature layouts.diff --git a/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js b/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.jsindex d93eb82f68..ae4bb202b8 100644--- a/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js+++ b/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js@@ -9,6 +9,8 @@ const mockLogger = {const mockGenerationJobManager = {isRedis: false,+ detachedAgentEventActionStoreMode: 'process_local',+ supportsDetachedAgentEventActions: true,createJob: jest.fn(),getJob: jest.fn(),emitError: jest.fn(),@@ -267,7 +269,6 @@ jest.mock('@librechat/api', () => ({resumeAgentEventActor: (...args) => mockResumeAgentEventActor(...args),createAgentEventActorDetachedActionLifecycle: (...args) =>mockCreateAgentEventActorDetachedActionLifecycle(...args),- isAgentEventActorDetachedActionProducerEnabled: () => true,parseAgentEventActorDetachedCompletion: (value) => (value?.version === 1 ? value : undefined),EVENT_ACTOR_DETACHED_COMPLETION_SOURCE: 'librechat-event-actor',EVENT_ACTOR_DETACHED_COMPLETION_TYPE: 'librechat.event_actor.detached_completion',@@ -395,6 +396,8 @@ describe('ResumableAgentController resume metadata', () => {mockGenerationJobManager.getJob.mockResolvedValue(undefined);mockGenerationJobManager.updateMetadata.mockResolvedValue(undefined);mockGenerationJobManager.isRedis = false;+ mockGenerationJobManager.detachedAgentEventActionStoreMode = 'process_local';+ mockGenerationJobManager.supportsDetachedAgentEventActions = true;mockGenerationJobManager.persistAgentEventDetachedTerminalEvidence.mockResolvedValue(true);mockGenerationJobManager.emitChunk.mockResolvedValue(undefined);mockGenerationJobManager.emitDone.mockResolvedValue(undefined);@@ -4253,6 +4256,7 @@ describe('ResumableAgentController resume metadata', () => {it('resumes the original actor suspension from exact detached terminal evidence', async () => {mockGenerationJobManager.isRedis = true;+ mockGenerationJobManager.detachedAgentEventActionStoreMode = 'distributed';mockGenerationJobManager.createJob.mockResolvedValue({createdAt: 2000,metadata: {@@ -4385,9 +4389,9 @@ describe('ResumableAgentController resume metadata', () => {);const lifecycleDependencies =mockCreateAgentEventActorDetachedActionLifecycle.mock.calls.at(-1)[1];- expect(lifecycleDependencies.producerEnabled()).toBe(true);- mockGenerationJobManager.isRedis = false;- expect(lifecycleDependencies.producerEnabled()).toBe(false);+ expect(lifecycleDependencies.storeMode()).toBe('distributed');+ mockGenerationJobManager.detachedAgentEventActionStoreMode = 'process_local';+ expect(lifecycleDependencies.storeMode()).toBe('process_local');expect(mockGenerationJobManager.createJob).toHaveBeenCalledWith('child-conversation','user-123',diff --git a/api/server/controllers/agents/__tests__/resume.spec.js b/api/server/controllers/agents/__tests__/resume.spec.jsindex 3d8646ccec..5f236036ab 100644--- a/api/server/controllers/agents/__tests__/resume.spec.js+++ b/api/server/controllers/agents/__tests__/resume.spec.js@@ -47,6 +47,8 @@ const mockJobStore = {};const mockGenerationJobManager = {+ detachedAgentEventActionStoreMode: 'distributed',+ supportsDetachedAgentEventActions: true,getJob: jest.fn(),getJobStore: jest.fn(() => mockJobStore),getResumeState: jest.fn(),@@ -112,7 +114,6 @@ const mockSettleAgentEventActorSuspension = jest.fn();const mockRecordAgentEventActorReconciliation = jest.fn();const mockResumeAgentEventActor = jest.fn();const mockCreateAgentEventActorDetachedActionLifecycle = jest.fn();-const mockIsAgentEventActorDetachedActionProducerEnabled = jest.fn();const mockGetAgentTriggerDelivery = jest.fn();const mockReserveAgentEventActorDetachedAction = jest.fn();const mockMarkAgentEventActorDetachedActionRunning = jest.fn();@@ -141,8 +142,6 @@ jest.mock('@librechat/api', () => ({resumeAgentEventActor: (...args) => mockResumeAgentEventActor(...args),createAgentEventActorDetachedActionLifecycle: (...args) =>mockCreateAgentEventActorDetachedActionLifecycle(...args),- isAgentEventActorDetachedActionProducerEnabled: () =>- mockIsAgentEventActorDetachedActionProducerEnabled(),}));jest.mock('~/models', () => ({@@ -372,6 +371,8 @@ describe('ResumeAgentController (POST /agents/chat/resume)', () => {mockGenerationJobManager.markProviderExecutionDrained.mockResolvedValue(true);mockGenerationJobManager.failPausePersistence.mockResolvedValue(true);mockGenerationJobManager.isRedis = true;+ mockGenerationJobManager.detachedAgentEventActionStoreMode = 'distributed';+ mockGenerationJobManager.supportsDetachedAgentEventActions = true;mockGenerationJobManager.persistAgentEventDetachedTerminalEvidence = jest.fn().mockResolvedValue(true);@@ -399,7 +400,6 @@ describe('ResumeAgentController (POST /agents/chat/resume)', () => {mockClaimAgentEventActorSuspension.mockResolvedValue({ status: 'claimed' });mockSettleAgentEventActorSuspension.mockResolvedValue({ status: 'settled' });mockRecordAgentEventActorReconciliation.mockResolvedValue(true);- mockIsAgentEventActorDetachedActionProducerEnabled.mockReturnValue(true);mockGetAgentTriggerDelivery.mockResolvedValue(undefined);mockCreateAgentEventActorDetachedActionLifecycle.mockReturnValue(undefined);endpointAgent = {@@ -678,9 +678,9 @@ describe('ResumeAgentController (POST /agents/chat/resume)', () => {).toBeLessThan(mockInitializeClient.mock.invocationCallOrder[0]);const lifecycleDependencies =mockCreateAgentEventActorDetachedActionLifecycle.mock.calls[0][1];- expect(lifecycleDependencies.producerEnabled()).toBe(true);- mockGenerationJobManager.isRedis = false;- expect(lifecycleDependencies.producerEnabled()).toBe(false);+ expect(lifecycleDependencies.storeMode()).toBe('distributed');+ mockGenerationJobManager.detachedAgentEventActionStoreMode = 'process_local';+ expect(lifecycleDependencies.storeMode()).toBe('process_local');expect(resumedClient.resumeCompletion).toHaveBeenCalledTimes(1);expect(mockGenerationJobManager.updateMetadata).toHaveBeenCalledWith(CONVO_ID,@@ -745,7 +745,8 @@ describe('ResumeAgentController (POST /agents/chat/resume)', () => {status: 'pending',},});- mockIsAgentEventActorDetachedActionProducerEnabled.mockReturnValue(false);+ mockGenerationJobManager.isRedis = false;+ mockGenerationJobManager.supportsDetachedAgentEventActions = false;const res = await post(approveBody());@@ -826,7 +827,7 @@ describe('ResumeAgentController (POST /agents/chat/resume)', () => {status: 'pending',},});- mockIsAgentEventActorDetachedActionProducerEnabled.mockReturnValue(false);+ mockGenerationJobManager.isRedis = false;mockInitializeClient.mockRejectedValue(new Error('client reconstruction failed'));mockResumeAgentEventActor.mockImplementation(async (input) => {expect(await input.claimProjection()).toBe(true);diff --git a/api/server/controllers/agents/request.js b/api/server/controllers/agents/request.jsindex 7cc2613c5c..eea985c77f 100644--- a/api/server/controllers/agents/request.js+++ b/api/server/controllers/agents/request.js@@ -33,7 +33,6 @@ const {executeAgentEventActor,resumeAgentEventActor,createAgentEventActorDetachedActionLifecycle,- isAgentEventActorDetachedActionProducerEnabled,parseAgentEventActorDetachedCompletion,EVENT_ACTOR_DETACHED_COMPLETION_SOURCE,EVENT_ACTOR_DETACHED_COMPLETION_TYPE,@@ -1398,10 +1397,9 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTitagentEventBindingId: boundEventBindingId,...(agentEventDelivery.expectedAction != null && {agentEventExpectedAction: agentEventDelivery.expectedAction,- ...(GenerationJobManager.isRedis &&- isAgentEventActorDetachedActionProducerEnabled() && {- agentEventDetachedActionProducerRequired: true,- }),+ ...(GenerationJobManager.isRedis && {+ agentEventDetachedActionProducerRequired: true,+ }),}),}),...(isRegenerate && { isRegenerate: true }),@@ -2104,9 +2102,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTitreserveAgentEventActorDetachedAction,markAgentEventActorDetachedActionRunning,settleAgentEventActorDetachedAction,- producerEnabled: () =>- GenerationJobManager.isRedis &&- isAgentEventActorDetachedActionProducerEnabled(),+ storeMode: () => GenerationJobManager.detachedAgentEventActionStoreMode,persistTerminalEvidence: async (evidence) => {const persisted =await GenerationJobManager.persistAgentEventDetachedTerminalEvidence(diff --git a/api/server/controllers/agents/resume.js b/api/server/controllers/agents/resume.jsindex c7fe2d697e..d1f0176131 100644--- a/api/server/controllers/agents/resume.js+++ b/api/server/controllers/agents/resume.js@@ -43,7 +43,6 @@ const {resumeAgentEventActor,createAgentEventActionRecorder,createAgentEventActorDetachedActionLifecycle,- isAgentEventActorDetachedActionProducerEnabled,findAgentEventAppliedAction,} = require('@librechat/api');const { disposeClient } = require('~/server/cleanup');@@ -1386,7 +1385,7 @@ const ResumeAgentController = async (req, res, next, initializeClient, addTitle)if (durableEventActorSuspension != null &&durableEventActorRequiresDetachedProducer &&- (!GenerationJobManager.isRedis || !isAgentEventActorDetachedActionProducerEnabled())+ !GenerationJobManager.supportsDetachedAgentEventActions) {const currentJob = await GenerationJobManager.getJob(streamId).catch(() => null);await rollbackUnconsumedScheduleClaim(currentJob);@@ -1398,7 +1397,7 @@ const ResumeAgentController = async (req, res, next, initializeClient, addTitle)503,{code: 'EVENT_ACTOR_RESUME_CAPABILITY_UNAVAILABLE',- error: 'A durable Event Actor resume worker is temporarily unavailable',+ error: 'A compatible Event Actor resume worker is temporarily unavailable',},generationProtocolVersion,);@@ -1475,8 +1474,7 @@ const ResumeAgentController = async (req, res, next, initializeClient, addTitle)reserveAgentEventActorDetachedAction,markAgentEventActorDetachedActionRunning,settleAgentEventActorDetachedAction,- producerEnabled: () =>- GenerationJobManager.isRedis && isAgentEventActorDetachedActionProducerEnabled(),+ storeMode: () => GenerationJobManager.detachedAgentEventActionStoreMode,persistTerminalEvidence: async (evidence) => {const persisted =await GenerationJobManager.persistAgentEventDetachedTerminalEvidence(diff --git a/api/server/services/Agents/triggers.js b/api/server/services/Agents/triggers.jsindex f6269e2cba..406d43553a 100644--- a/api/server/services/Agents/triggers.js+++ b/api/server/services/Agents/triggers.js@@ -3,7 +3,6 @@ const {createAgentEventContinueResolver,createSubagentCompletionWakeupResolver,GenerationJobManager,- isAgentEventActorDetachedActionProducerEnabled,} = require('@librechat/api');const methods = require('~/models');@@ -15,8 +14,7 @@ const completionResolver = createSubagentCompletionWakeupResolver({const service = createAgentTriggerService({methods,isPrincipalActive: methods.isAgentTriggerPrincipalActive,- supportsDetachedActionCompletion: () =>- GenerationJobManager.isRedis && isAgentEventActorDetachedActionProducerEnabled(),+ supportsDetachedActionCompletion: () => GenerationJobManager.supportsDetachedAgentEventActions,prepareContinue: createAgentEventContinueResolver({methods,getGenerationJob: (conversationId) => GenerationJobManager.getJob(conversationId),diff --git a/api/server/services/Agents/triggers.spec.js b/api/server/services/Agents/triggers.spec.jsindex d824e68182..6eeba98996 100644--- a/api/server/services/Agents/triggers.spec.js+++ b/api/server/services/Agents/triggers.spec.js@@ -1,7 +1,6 @@const mockCreateAgentTriggerService = jest.fn();-const mockIsProducerEnabled = jest.fn();const mockGenerationJobManager = {- isRedis: true,+ supportsDetachedAgentEventActions: true,getJob: jest.fn(),};@@ -10,7 +9,6 @@ jest.mock('@librechat/api', () => ({createAgentEventContinueResolver: jest.fn(() => jest.fn()),createSubagentCompletionWakeupResolver: jest.fn(() => jest.fn()),GenerationJobManager: mockGenerationJobManager,- isAgentEventActorDetachedActionProducerEnabled: () => mockIsProducerEnabled(),}));jest.mock('~/models', () => ({@@ -21,8 +19,7 @@ describe('agent trigger service composition', () => {beforeEach(() => {jest.resetModules();jest.clearAllMocks();- mockGenerationJobManager.isRedis = true;- mockIsProducerEnabled.mockReturnValue(true);+ mockGenerationJobManager.supportsDetachedAgentEventActions = true;mockCreateAgentTriggerService.mockReturnValue({initialize: jest.fn(),stop: jest.fn(),@@ -39,16 +36,13 @@ describe('agent trigger service composition', () => {});});- it('advertises detached completion capability only after Redis and producer activation', () => {+ it('advertises detached completion capability for every compatible generation store', () => {require('./triggers');const supportsDetachedActionCompletion =mockCreateAgentTriggerService.mock.calls[0][0].supportsDetachedActionCompletion;expect(supportsDetachedActionCompletion()).toBe(true);- mockIsProducerEnabled.mockReturnValue(false);- expect(supportsDetachedActionCompletion()).toBe(false);- mockIsProducerEnabled.mockReturnValue(true);- mockGenerationJobManager.isRedis = false;+ mockGenerationJobManager.supportsDetachedAgentEventActions = false;expect(supportsDetachedActionCompletion()).toBe(false);});});diff --git a/packages/api/src/agents/triggers/README.md b/packages/api/src/agents/triggers/README.mdindex b18f4108ce..93abb3f4bb 100644--- a/packages/api/src/agents/triggers/README.md+++ b/packages/api/src/agents/triggers/README.md@@ -127,16 +127,14 @@ Bound child continuations are automatic after authentication and binding authori`endpoints.agents.eventDriven.selfUrl`; most deployments should omit both and use the boundlistener.-Detached Event Actor completion production has an explicit rolling-deployment fence. It remains-fail-closed until Redis generation streams are active and-`AGENT_TRIGGERS_DETACHED_ACTIONS_PRODUCER_ENABLED=true` is set. Deploy the new version to every-replica first, verify that no old API or worker replica remains, and only then enable the producer-across the Redis-backed fleet. This ordering ensures Redis recovery, HTTP completion consumers,-account-deletion drains, lane maintenance, and operational requeue/dead-letter paths all understand-capability-fenced completion work before any is created. For rollback, disable the producer, drain-all detached completion work, and only then start an older version. Internal detached completions-always target the current replica's bound listener; `AGENT_TRIGGERS_SELF_URL` remains available for-ordinary trigger dispatch but cannot route capability-owned completion work to another replica.+Detached Event Actor completion is automatic for every built-in generation store. The in-memory+adapter preserves the lifecycle while its process remains alive; Redis adds restart recovery and+replica handoff without changing the Event Actor interface.+Completion work is stored behind a mixed-version compatibility shield: older replicas retain lane+and account-deletion safety but cannot claim, recover, requeue, or interpret the new work. Internal+detached completions always target the capable worker's bound listener;+`AGENT_TRIGGERS_SELF_URL` remains available for ordinary trigger dispatch but cannot route+capability-owned completion work to another replica.```httpPOST /api/agents/v1/events/bindingsdiff --git a/packages/api/src/agents/triggers/detachedAction.spec.ts b/packages/api/src/agents/triggers/detachedAction.spec.tsindex c5df8e949a..02f2539458 100644--- a/packages/api/src/agents/triggers/detachedAction.spec.ts+++ b/packages/api/src/agents/triggers/detachedAction.spec.ts@@ -3,31 +3,8 @@ import type { AgentTriggerEnvelope } from './envelope';import {createAgentEventActorDetachedActionLifecycle,createAgentEventDetachedResumeHandler,- isAgentEventActorDetachedActionProducerEnabled,} from './detachedAction';-describe('isAgentEventActorDetachedActionProducerEnabled', () => {- const key = 'AGENT_TRIGGERS_DETACHED_ACTIONS_PRODUCER_ENABLED';- const original = process.env[key];-- afterEach(() => {- if (original == null) {- delete process.env[key];- return;- }- process.env[key] = original;- });-- it('requires an explicit true activation', () => {- delete process.env[key];- expect(isAgentEventActorDetachedActionProducerEnabled()).toBe(false);- process.env[key] = 'false';- expect(isAgentEventActorDetachedActionProducerEnabled()).toBe(false);- process.env[key] = ' TRUE ';- expect(isAgentEventActorDetachedActionProducerEnabled()).toBe(true);- });-});-describe('createAgentEventActorDetachedActionLifecycle', () => {it('owns only the exact expected action and exposes a suspension after launch', async () => {const action = {@@ -75,7 +52,7 @@ describe('createAgentEventActorDetachedActionLifecycle', () => {persistTerminalEvidence,onTerminal: wake,waitForTerminalPersistenceRetry,- producerEnabled: () => true,+ storeMode: () => 'distributed',now: () => new Date('2026-08-28T12:00:00.000Z'),},);@@ -151,7 +128,7 @@ describe('createAgentEventActorDetachedActionLifecycle', () => {});});- it('fails closed before durable reservation until detached production is activated', async () => {+ it('fails closed before reservation when the generation store lacks detached-action support', async () => {const reserve = jest.fn();const lifecycle = createAgentEventActorDetachedActionLifecycle({@@ -169,7 +146,7 @@ describe('createAgentEventActorDetachedActionLifecycle', () => {settleAgentEventActorDetachedAction: jest.fn(),persistTerminalEvidence: jest.fn(),onTerminal: jest.fn(),- producerEnabled: () => false,+ storeMode: () => undefined,},);@@ -182,7 +159,7 @@ describe('createAgentEventActorDetachedActionLifecycle', () => {}),).resolves.toEqual({status: 'conflict',- error: expect.stringContaining('not activated'),+ error: expect.stringContaining('requires a compatible generation store'),});expect(reserve).not.toHaveBeenCalled();expect(lifecycle.readSuspension()).toBeUndefined();@@ -225,7 +202,7 @@ describe('createAgentEventActorDetachedActionLifecycle', () => {settleAgentEventActorDetachedAction: jest.fn(),persistTerminalEvidence: jest.fn(),onTerminal: jest.fn(),- producerEnabled: () => true,+ storeMode: () => 'distributed',},);@@ -290,7 +267,7 @@ describe('createAgentEventActorDetachedActionLifecycle', () => {persistTerminalEvidence,onTerminal: jest.fn(),waitForTerminalPersistenceRetry,- producerEnabled: () => true,+ storeMode: () => 'distributed',},);const reservation = await lifecycle.reserve({@@ -357,7 +334,7 @@ describe('createAgentEventActorDetachedActionLifecycle', () => {settleAgentEventActorDetachedAction: jest.fn(),persistTerminalEvidence: jest.fn(),onTerminal: jest.fn(),- producerEnabled: () => true,+ storeMode: () => 'distributed',},);const input = {@@ -408,7 +385,7 @@ describe('createAgentEventActorDetachedActionLifecycle', () => {settleAgentEventActorDetachedAction: jest.fn(),persistTerminalEvidence: jest.fn(),onTerminal: jest.fn(),- producerEnabled: () => true,+ storeMode: () => 'distributed' as const,now: () => new Date('2026-08-28T12:00:00.000Z'),};const owner = {diff --git a/packages/api/src/agents/triggers/detachedAction.ts b/packages/api/src/agents/triggers/detachedAction.tsindex 1aee9e2ce5..61c2adf907 100644--- a/packages/api/src/agents/triggers/detachedAction.ts+++ b/packages/api/src/agents/triggers/detachedAction.ts@@ -14,9 +14,9 @@ import type {AgentTriggerEnvelope,AgentTriggerExpectedAction,} from './envelope';+import type { DetachedAgentEventActionStoreMode, SerializableJobData } from '~/stream';import type { EventActorDetachedActionLifecycle } from '../handlers';import type { AgentEventDetachedTerminalEvidence } from './types';-import type { SerializableJobData } from '~/stream';import { matchesExpectedAction } from './expectedAction';import { createAgentTriggerEnvelope } from './envelope';@@ -25,14 +25,9 @@ const TERMINAL_PERSIST_RETRY_INITIAL_MS = 100;const TERMINAL_PERSIST_RETRY_MAX_MS = 30_000;const RESERVATION_RECOVERY_MS = 60_000;const RUNNING_RECOVERY_MS = 30 * 60_000;-const DETACHED_ACTION_PRODUCER_ENV = 'AGENT_TRIGGERS_DETACHED_ACTIONS_PRODUCER_ENABLED';export const EVENT_ACTOR_DETACHED_COMPLETION_TYPE = 'librechat.event_actor.detached_completion';export const EVENT_ACTOR_DETACHED_COMPLETION_SOURCE = 'librechat-event-actor';-export function isAgentEventActorDetachedActionProducerEnabled(): boolean {- return process.env[DETACHED_ACTION_PRODUCER_ENV]?.trim().toLowerCase() === 'true';-}-export function parseAgentEventDetachedTerminalEvidence(value: unknown,): AgentEventDetachedTerminalEvidence | undefined {@@ -141,7 +136,7 @@ interface DetachedActionDependencies {persistTerminalEvidence(input: AgentEventDetachedTerminalEvidence): Promise<void>;onTerminal(input: { taskId: string; idempotencyKey: string }): Promise<void>;waitForTerminalPersistenceRetry?(delayMs: number): Promise<void>;- producerEnabled(): boolean;+ storeMode(): DetachedAgentEventActionStoreMode | undefined;now?(): Date;}@@ -279,7 +274,7 @@ export function createAgentEventActorDetachedActionLifecycle(deps: DetachedActionDependencies,): AgentEventActorDetachedActionLifecycle {const now = deps.now ?? (() => new Date());- const producerEnabled = deps.producerEnabled();+ const storeMode = deps.storeMode();let current: { taskId: string; idempotencyKey: string; launchAcknowledged: boolean } | undefined;const scope = {deliveryKey: owner.invocationId,@@ -306,11 +301,11 @@ export function createAgentEventActorDetachedActionLifecycle() {return { status: 'ignored' };}- if (!producerEnabled) {+ if (storeMode == null) {return {status: 'conflict',error:- 'Detached Event Actor production is not activated for this deployment; no external action was launched',+ 'Detached Event Actor production requires a compatible generation store; no external action was launched',};}const reservedAt = now();diff --git a/packages/api/src/stream/GenerationJobManager.ts b/packages/api/src/stream/GenerationJobManager.tsindex fb35be40e9..699300e92b 100644--- a/packages/api/src/stream/GenerationJobManager.ts+++ b/packages/api/src/stream/GenerationJobManager.ts@@ -30,6 +30,7 @@ import type {IdempotencyClaimValue,PreemptMessage,SteerQueueItem,+ DetachedAgentEventActionStoreMode,} from './interfaces/IJobStore';import type { AgentStartupTelemetry } from '~/agents/startup';import type { RecoveredSteerPayload } from './SteerRecovery';@@ -904,6 +905,19 @@ class GenerationJobManagerClass {return this._isRedis;}+ /**+ * Store-selected detached Event Actor execution guarantee. In-memory stores+ * support the same lifecycle within one process; Redis additionally makes it+ * recoverable across restarts and replica handoffs.+ */+ get detachedAgentEventActionStoreMode(): DetachedAgentEventActionStoreMode | undefined {+ return this.jobStore.detachedAgentEventActionStoreMode;+ }++ get supportsDetachedAgentEventActions(): boolean {+ return this.detachedAgentEventActionStoreMode != null;+ }+/** Installs the application-owned approval-expiry hook without coupling the* stream package to any particular trigger/scheduler implementation. */setApprovalExpiredHandler(handler?: ApprovalExpiredHandler): void {@@ -7612,6 +7626,22 @@ class GenerationJobManagerClass {logger.error('[GenerationJobManager] Failed to enumerate pending host actions', err);}}+ // Detached completion generations live in a versioned recovery lane. An+ // older replica knows only the ordinary terminal-host-action index and+ // therefore cannot deserialize away the original invocation identity or+ // acknowledge the completion against the wrong mailbox delivery.+ if (this.jobStore.getDetachedAgentEventTerminalHostActionJobs) {+ try {+ for (const job of await this.jobStore.getDetachedAgentEventTerminalHostActionJobs()) {+ candidates.set(job.streamId, job);+ }+ } catch (err) {+ logger.error(+ '[GenerationJobManager] Failed to enumerate detached Event Actor host actions',+ err,+ );+ }+ }const streamIds = new Set([...this.runtimeState.keys(), ...candidates.keys()]);for (const streamId of streamIds) {let job: SerializableJobData | null;diff --git a/packages/api/src/stream/__tests__/RedisJobStore.spec.ts b/packages/api/src/stream/__tests__/RedisJobStore.spec.tsindex bc70dba462..de54ce5911 100644--- a/packages/api/src/stream/__tests__/RedisJobStore.spec.ts+++ b/packages/api/src/stream/__tests__/RedisJobStore.spec.ts@@ -721,6 +721,7 @@ describe('RedisJobStore', () => {const runningMembership = createDeferred<number>();const requiresActionRemoval = createDeferred<number>();const terminalHostActionRemoval = createDeferred<number>();+ const detachedTerminalHostActionRemoval = createDeferred<number>();const userMembership = createDeferred<number>();const userExpiry = createDeferred<number>();const started: string[] = [];@@ -749,6 +750,10 @@ describe('RedisJobStore', () => {started.push('requires_action');return requiresActionRemoval.promise;}+ if (key === 'stream:agent_event_detached:terminal_host_action:v1') {+ started.push('detached_terminal_host_action');+ return detachedTerminalHostActionRemoval.promise;+ }started.push('terminal_host_action');return terminalHostActionRemoval.promise;}),@@ -765,9 +770,16 @@ describe('RedisJobStore', () => {expect(started).toEqual(['job']);evalResult.resolve(1);- await waitFor(() => started.length === 5);+ await waitFor(() => started.length === 6);- expect(started).toEqual(['job', 'running', 'requires_action', 'terminal_host_action', 'user']);+ expect(started).toEqual([+ 'job',+ 'running',+ 'requires_action',+ 'terminal_host_action',+ 'detached_terminal_host_action',+ 'user',+ ]);expect(settled).toBe(false);expect(expire).not.toHaveBeenCalled();@@ -779,6 +791,7 @@ describe('RedisJobStore', () => {'running','requires_action','terminal_host_action',+ 'detached_terminal_host_action','user','user_expiry',]);@@ -797,6 +810,10 @@ describe('RedisJobStore', () => {await Promise.resolve();expect(settled).toBe(false);+ detachedTerminalHostActionRemoval.resolve(1);+ await Promise.resolve();+ expect(settled).toBe(false);+requiresActionRemoval.resolve(1);await expect(creating).resolves.toMatchObject({streamId: 'stream-1',@@ -968,9 +985,10 @@ describe('RedisJobStore', () => {expect(saveCall[0]).toContain('currentCreatedAt ~= ARGV[3]');expect(saveCall[0]).toContain('currentStatus ~= "running"');expect(saveCall[0]).toContain('currentStatus ~= "requires_action"');- expect(saveCall[0]).toContain('terminalHostActionPending ~= "1"');+ expect(saveCall[0]).toContain('detachedAgentEventTerminalHostActionPending');+ expect(saveCall[0]).toContain('and not terminalHostActionPending');expect(saveCall[0]).toContain(- 'currentStatus == "requires_action" or terminalHostActionPending == "1"',+ 'currentStatus == "requires_action" or terminalHostActionPending',);expect(saveCall.slice(1)).toEqual([2,@@ -1034,6 +1052,118 @@ describe('RedisJobStore', () => {expect(transitionCall[19]).toBe('86400');});+ test('pre-arms detached completion recovery outside the legacy terminal index', async () => {+ const evalTransition = jest.fn().mockResolvedValue(1);+ const sadd = jest.fn().mockResolvedValue(1);+ const detachedIdentity = {+ agentEventDeliveryKey: 'completion-delivery',+ agentEventInvocationKey: 'original-invocation',+ agentEventInvocationGenerationCreatedAt: '90',+ };+ const redis = {+ isCluster: true,+ eval: evalTransition,+ sadd,+ srem: jest.fn().mockResolvedValue(1),+ expire: jest.fn().mockResolvedValue(1),+ hgetall: jest+ .fn()+ .mockResolvedValueOnce({+ streamId: 'stream-detached-host-action',+ userId: 'user-1',+ status: 'running',+ createdAt: '100',+ syncSent: '0',+ ...detachedIdentity,+ })+ .mockResolvedValue({+ streamId: 'stream-detached-host-action',+ userId: 'user-1',+ status: 'complete',+ createdAt: '100',+ completedAt: '200',+ detachedAgentEventTerminalHostActionPending: '1',+ syncSent: '0',+ ...detachedIdentity,+ }),+ } as unknown as Cluster;+ const store = new RedisJobStore(redis);++ await expect(+ store.transitionStatus('stream-detached-host-action', {+ from: 'running',+ to: 'complete',+ expectCreatedAt: 100,+ patch: { completedAt: 200, terminalHostActionPending: true },+ }),+ ).resolves.toBe(true);++ expect(sadd.mock.calls[0]).toEqual([+ 'stream:agent_event_detached:terminal_host_action:v1',+ '["stream-detached-host-action",100]',+ ]);+ expect(sadd.mock.calls).not.toContainEqual([+ 'stream:terminal_host_action',+ '["stream-detached-host-action",100]',+ ]);+ const transitionCall = evalTransition.mock.calls[0];+ expect(transitionCall).toContain('status');+ expect(transitionCall).toContain('detached_terminal_pending_v1');+ expect(transitionCall).toContain('detachedAgentEventTerminalStatus');+ expect(transitionCall).toContain('complete');+ });++ test('retains an exact detached recovery hint until its terminal CAS is visible', async () => {+ const member = '["stream-detached-prearm-race",100]';+ const detachedIdentity = {+ agentEventDeliveryKey: 'completion-delivery',+ agentEventInvocationKey: 'original-invocation',+ agentEventInvocationGenerationCreatedAt: '90',+ };+ const srem = jest.fn().mockResolvedValue(1);+ const redis = {+ isCluster: true,+ smembers: jest.fn().mockResolvedValue([member]),+ srem,+ sadd: jest.fn().mockResolvedValue(1),+ expire: jest.fn().mockResolvedValue(1),+ hgetall: jest+ .fn()+ .mockResolvedValueOnce({+ streamId: 'stream-detached-prearm-race',+ userId: 'user-1',+ status: 'running',+ createdAt: '100',+ syncSent: '0',+ ...detachedIdentity,+ })+ .mockResolvedValueOnce({+ streamId: 'stream-detached-prearm-race',+ userId: 'user-1',+ status: 'detached_terminal_pending_v1',+ detachedAgentEventTerminalStatus: 'complete',+ detachedAgentEventTerminalHostActionPending: '1',+ createdAt: '100',+ completedAt: '200',+ syncSent: '0',+ ...detachedIdentity,+ }),+ } as unknown as Cluster;+ const store = new RedisJobStore(redis);++ await expect(store.getDetachedAgentEventTerminalHostActionJobs()).resolves.toEqual([]);+ expect(srem).not.toHaveBeenCalledWith(+ 'stream:agent_event_detached:terminal_host_action:v1',+ member,+ );+ await expect(store.getDetachedAgentEventTerminalHostActionJobs()).resolves.toEqual([+ expect.objectContaining({+ streamId: 'stream-detached-prearm-race',+ terminalHostActionPending: true,+ }),+ ]);+ });+test('acknowledges evidence TTLs without removing a successor retry member', async () => {const evalClear = jest.fn().mockResolvedValue(1);const srem = jest.fn().mockResolvedValue(1);@@ -1063,6 +1193,9 @@ describe('RedisJobStore', () => {'7','11',);+ expect(evalClear.mock.calls[0][0]).toContain(+ 'redis.call("HSET", KEYS[1], "status", detachedStatus)',+ );expect(srem).toHaveBeenCalledWith('stream:terminal_host_action','["stream-host-action-clear",100]',diff --git a/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts b/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.tsindex 03f9e03fad..cc2497ff17 100644--- a/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts+++ b/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts@@ -1184,6 +1184,80 @@ describe('RedisJobStore Integration Tests', () => {await store.destroy();});+ test('isolates detached Event Actor completion recovery from the legacy terminal lane', async () => {+ if (!ioredisClient) {+ return;+ }++ const { RedisJobStore } = await import('../implementations/RedisJobStore');+ const store = new RedisJobStore(ioredisClient, { runningTtl: 60 });+ await store.initialize();++ const streamId = `terminal-detached-event-${Date.now()}`;+ const invocationKey = 'event-invocation-original';+ const completionDeliveryKey = 'event-completion-delivery';+ const invocationGenerationCreatedAt = Date.now() - 1_000;+ const job = await store.createJob(streamId, 'user-1', streamId, undefined, {+ agentEventDeliveryKey: completionDeliveryKey,+ agentEventInvocationKey: invocationKey,+ agentEventInvocationGenerationCreatedAt: invocationGenerationCreatedAt,+ });+ const member = JSON.stringify([streamId, job.createdAt]);++ await expect(+ store.transitionStatus(streamId, {+ from: 'running',+ to: 'complete',+ expectCreatedAt: job.createdAt,+ patch: { completedAt: Date.now(), terminalHostActionPending: true },+ }),+ ).resolves.toBe(true);++ // A pre-detached replica scans only this legacy key. The completion must+ // never become visible there, because it would deserialize only the+ // completion delivery and lose the original invocation identity.+ await expect(ioredisClient.sismember('stream:terminal_host_action', member)).resolves.toBe(0);+ await expect(ioredisClient.hgetall(`stream:{${streamId}}:job`)).resolves.toMatchObject({+ status: 'detached_terminal_pending_v1',+ detachedAgentEventTerminalStatus: 'complete',+ detachedAgentEventTerminalHostActionPending: '1',+ });+ await expect(store.getJob(streamId)).resolves.toMatchObject({+ status: 'complete',+ terminalHostActionPending: true,+ });+ await expect(store.getTerminalHostActionJobs()).resolves.not.toEqual(+ expect.arrayContaining([expect.objectContaining({ streamId })]),+ );+ await expect(store.getDetachedAgentEventTerminalHostActionJobs()).resolves.toEqual([+ expect.objectContaining({+ streamId,+ agentEventDeliveryKey: completionDeliveryKey,+ agentEventInvocationKey: invocationKey,+ agentEventInvocationGenerationCreatedAt: invocationGenerationCreatedAt,+ }),+ ]);++ // The current creator reports an ordinary predecessor conflict, while a+ // pre-detached creator rejects the raw versioned status as corrupt. Both+ // outcomes fence replacement even when its caller permits active replacement.+ await expect(store.createJob(streamId, 'user-1', streamId)).rejects.toMatchObject({+ name: 'JobPredecessorMismatchError',+ });++ await store.clearTerminalHostAction(streamId, job.createdAt);+ await expect(ioredisClient.hgetall(`stream:{${streamId}}:job`)).resolves.toMatchObject({+ status: 'complete',+ });+ await expect(+ ioredisClient.hget(`stream:{${streamId}}:job`, 'detachedAgentEventTerminalStatus'),+ ).resolves.toBeNull();+ await expect(+ ioredisClient.sismember('stream:agent_event_detached:terminal_host_action:v1', member),+ ).resolves.toBe(0);+ await store.destroy();+ });+test('terminal host acknowledgement deletes a zero-TTL job after settlement', async () => {if (!ioredisClient) {return;diff --git a/packages/api/src/stream/__tests__/terminalHostAction.spec.ts b/packages/api/src/stream/__tests__/terminalHostAction.spec.tsindex a30785acdc..2d68b2a8e1 100644--- a/packages/api/src/stream/__tests__/terminalHostAction.spec.ts+++ b/packages/api/src/stream/__tests__/terminalHostAction.spec.ts@@ -1,3 +1,8 @@+import type { AgentEventActorDetachedAction } from '@librechat/data-schemas';+import {+ createAgentEventActorDetachedActionLifecycle,+ createAgentEventDetachedResumeHandler,+} from '~/agents/triggers/detachedAction';import { InMemoryEventTransport } from '~/stream/implementations/InMemoryEventTransport';import { buildPendingAction, buildToolApprovalPayload } from '~/agents/hitl/policy';import { InMemoryJobStore } from '~/stream/implementations/InMemoryJobStore';@@ -53,6 +58,155 @@ describe('GenerationJobManager terminal host actions', () => {);});+ it('runs detached launch, completion, and continuation through the in-memory adapter', async () => {+ const streamId = 'conversation-memory-detached-action';+ const invocationId = 'trigger-memory-detached-action';+ const enqueueContinuation = jest.fn().mockResolvedValue(undefined);+ let action: AgentEventActorDetachedAction = {+ version: 1 as const,+ invocationId,+ expectedToolName: 'submit_move',+ toolName: 'submit_move',+ toolCallId: 'call-memory-detached-action',+ turnId: 'response-memory:0',+ taskId: `event_actor_${'a'.repeat(64)}`,+ idempotencyKey: 'a'.repeat(64),+ launchAttempt: 1,+ status: 'reserved',+ reservedAt: new Date(),+ observedAt: new Date(),+ recoveryAfter: new Date(Date.now() + 60_000),+ };+ const resumeDetachedAction = createAgentEventDetachedResumeHandler({+ getAgentTriggerDelivery: jest.fn().mockResolvedValue({+ user: 'user-1',+ envelope: {+ mode: 'continue',+ principal: { userId: 'user-1' },+ target: {+ agentId: 'agent-1',+ bindingId: 'binding-1',+ conversationId: streamId,+ parentMessageId: 'message-1',+ sourceKeyId: 'source-key-1',+ },+ expectedAction: { toolName: 'submit_move' },+ },+ }),+ enqueueAgentTrigger: enqueueContinuation,+ requestId: () => 'memory-detached-continuation',+ });+ manager.setTerminalHostActionHandler(async (_streamId, job) => {+ if (action.status !== 'succeeded') {+ throw new Error('detached action is still running');+ }+ await resumeDetachedAction({+ streamId,+ job,+ handlingGenerationCreatedAt: job.createdAt,+ suspension: {} as never,+ action,+ });+ });+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: {+ agentEventDeliveryKey: invocationId,+ agentEventBindingId: 'binding-1',+ agentEventExpectedAction: { toolName: 'submit_move' },+ },+ });+ const lifecycle = createAgentEventActorDetachedActionLifecycle(+ {+ user: 'user-1',+ bindingId: 'binding-1',+ conversationId: streamId,+ generationCreatedAt: job.createdAt,+ turnCreatedAt: job.createdAt,+ invocationId,+ expectedAction: { toolName: 'submit_move' },+ },+ {+ storeMode: () => manager.detachedAgentEventActionStoreMode,+ reserveAgentEventActorDetachedAction: jest.fn(async () => ({+ status: 'reserved' as const,+ action,+ })),+ markAgentEventActorDetachedActionRunning: jest.fn(async () => {+ action = { ...action, status: 'running', observedAt: new Date() };+ return { status: 'applied' as const, action };+ }),+ settleAgentEventActorDetachedAction: jest.fn(async (input) => {+ action = {+ ...action,+ status: 'succeeded',+ result: input.result,+ observedAt: input.observedAt,+ };+ return { status: 'applied' as const, action };+ }),+ persistTerminalEvidence: async (evidence) => {+ const persisted = await manager.persistAgentEventDetachedTerminalEvidence(+ streamId,+ job.createdAt,+ evidence,+ );+ if (!persisted) {+ throw new Error('failed to stage detached terminal evidence');+ }+ },+ onTerminal: async () => {+ await manager.retryTerminalHostAction(streamId, job.createdAt);+ },+ },+ );++ expect(manager.detachedAgentEventActionStoreMode).toBe('process_local');+ await expect(+ lifecycle.reserve({+ toolName: 'submit_move',+ toolCallId: action.toolCallId,+ turnId: action.turnId,+ arguments: {},+ }),+ ).resolves.toMatchObject({ status: 'reserved', taskId: action.taskId });+ await expect(+ lifecycle.markRunning({ taskId: action.taskId, idempotencyKey: action.idempotencyKey }),+ ).resolves.toBe(true);+ expect(lifecycle.readSuspension()).toMatchObject({+ kind: 'internal_completion',+ actionId: action.taskId,+ });++ await expect(manager.completeJob(streamId, undefined, job.createdAt)).resolves.toBe(true);+ expect(enqueueContinuation).not.toHaveBeenCalled();+ await expect(store.getJob(streamId)).resolves.toMatchObject({+ terminalHostActionPending: true,+ });++ await expect(+ lifecycle.settle({+ taskId: action.taskId,+ idempotencyKey: action.idempotencyKey,+ status: 'succeeded',+ result: 'move accepted',+ }),+ ).resolves.toBe(true);+ await lifecycle.wake({ taskId: action.taskId, idempotencyKey: action.idempotencyKey });++ expect(enqueueContinuation).toHaveBeenCalledWith(+ expect.objectContaining({+ mode: 'continue',+ deliveryId: `detached_completion:${action.taskId}`,+ event: expect.objectContaining({+ type: 'librechat.event_actor.detached_completion',+ payload: expect.objectContaining({ invocationId, taskId: action.taskId }),+ }),+ }),+ { requiredWorkerCapability: 'event_actor_detached_action_v1' },+ );+ await expect(store.getJob(streamId)).resolves.not.toHaveProperty('terminalHostActionPending');+ });+it('runs and acknowledges a generation-fenced host action before terminal cleanup', async () => {const handler = jest.fn().mockResolvedValue(undefined);manager.setTerminalHostActionHandler(handler);@@ -394,6 +548,47 @@ describe('GenerationJobManager terminal host actions', () => {);});+ it('recovers detached completion jobs from the capability-isolated terminal lane', async () => {+ const streamId = 'conversation-detached-terminal-lane';+ const invocationKey = 'trigger-original-invocation';+ const completionDeliveryKey = 'trigger-internal-completion';+ const invocationGenerationCreatedAt = 42;+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: {+ agentEventDeliveryKey: completionDeliveryKey,+ agentEventInvocationKey: invocationKey,+ agentEventInvocationGenerationCreatedAt: invocationGenerationCreatedAt,+ },+ });+ await store.transitionStatus(streamId, {+ from: 'running',+ to: 'complete',+ expectCreatedAt: job.createdAt,+ patch: { completedAt: Date.now(), terminalHostActionPending: true },+ });+ const terminalJob = (await store.getJob(streamId))!;+ jest.spyOn(store, 'getTerminalHostActionJobs').mockResolvedValue([]);+ Object.assign(store, {+ getDetachedAgentEventTerminalHostActionJobs: jest.fn().mockResolvedValue([terminalJob]),+ });+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);++ await (manager as unknown as { cleanup: () => Promise<void> }).cleanup();++ expect(handler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({+ agentEventDeliveryKey: completionDeliveryKey,+ agentEventInvocationKey: invocationKey,+ agentEventInvocationGenerationCreatedAt: invocationGenerationCreatedAt,+ }),+ expect.any(Array),+ expect.any(Array),+ );+ await expect(store.getJob(streamId)).resolves.not.toHaveProperty('terminalHostActionPending');+ });+it('retains completed run-step evidence through repeated host-action failures', async () => {const handler = jest.fn()diff --git a/packages/api/src/stream/implementations/InMemoryJobStore.ts b/packages/api/src/stream/implementations/InMemoryJobStore.tsindex b886b05058..f0ba55cfb7 100644--- a/packages/api/src/stream/implementations/InMemoryJobStore.ts+++ b/packages/api/src/stream/implementations/InMemoryJobStore.ts@@ -141,6 +141,8 @@ interface ContentState {* - No chunk persistence needed - same instance handles generation and reconnects*/export class InMemoryJobStore implements IJobStoreV2 {+ readonly detachedAgentEventActionStoreMode = 'process_local' as const;+private jobs = new Map<string, SerializableJobData>();private contentState = new Map<string, ContentState>();private cleanupInterval: NodeJS.Timeout | null = null;diff --git a/packages/api/src/stream/implementations/RedisJobStore.ts b/packages/api/src/stream/implementations/RedisJobStore.tsindex a987a78350..e3dcf3389a 100644--- a/packages/api/src/stream/implementations/RedisJobStore.ts+++ b/packages/api/src/stream/implementations/RedisJobStore.ts@@ -364,14 +364,18 @@ const JOB_CREATE_LUA ='local replacedProviderDrained = redis.call("HGET", KEYS[1], "providerDrained") ' +'local replacedTerminalPersistencePending = redis.call("HGET", KEYS[1], "terminalPersistencePending") ' +'local replacedTerminalHostActionPending = redis.call("HGET", KEYS[1], "terminalHostActionPending") ' ++ 'local replacedDetachedTerminalHostActionPending = redis.call("HGET", KEYS[1], "detachedAgentEventTerminalHostActionPending") ' +'local replacedProtocol = redis.call("HGET", KEYS[1], "generationProtocolVersion") ' +'local MAX_SAFE_EPOCH = 9007199254740991 ' +'local function isSafeEpoch(value) return type(value) == "number" and value >= 0 ' +'and value <= MAX_SAFE_EPOCH and value == math.floor(value) end ' +'local function isValidJobStatus(value) return value == "running" or value == "requires_action" ' +'or value == "complete" or value == "error" or value == "aborted" end ' ++ 'local detachedTerminalShield = replacedStatus == "detached_terminal_pending_v1" ' ++ 'and replacedDetachedTerminalHostActionPending == "1" ' +'local replacedEpoch = tonumber(replacedCreatedAt) local previousCreatedAt = replacedEpoch ' +- 'if previousJobExists == 1 and (not isSafeEpoch(replacedEpoch) or not isValidJobStatus(replacedStatus)) then ' ++ 'if previousJobExists == 1 and (not isSafeEpoch(replacedEpoch) ' ++ 'or (not isValidJobStatus(replacedStatus) and not detachedTerminalShield)) then ' +'return { "", "", "0", "replacement_receipt_corrupt" } end ' +'if (replacedProviderExecutionId and not replacedProviderDrained) ' +'or (replacedProviderDrained and not replacedProviderExecutionId) ' +@@ -386,11 +390,11 @@ const JOB_CREATE_LUA ='local observedConversationId = replacedConversationId ' +'local observedActive = previousJobExists == 1 and ' +'(replacedStatus == "running" or replacedStatus == "requires_action" ' +- 'or replacedTerminalPersistencePending == "1" or replacedTerminalHostActionPending == "1") ' ++ 'or replacedTerminalPersistencePending == "1" or replacedTerminalHostActionPending == "1" or replacedDetachedTerminalHostActionPending == "1") ' +'if retainedEpoch and (not previousCreatedAt or retainedEpoch > previousCreatedAt) then ' +'previousCreatedAt = retainedEpoch observedCreatedAt = retainedEpochRaw ' +'observedStatus = nil observedConversationId = nil observedActive = false end ' +- 'if observedActive and replacedTerminalHostActionPending == "1" then ' ++ 'if observedActive and (replacedTerminalHostActionPending == "1" or replacedDetachedTerminalHostActionPending == "1") then ' +'return { previousUserId or "", previousTenantId or "", "0", "predecessor_mismatch", ' +'observedCreatedAt, observedStatus or "", observedConversationId or "", "1", "1" } end ' +'if ARGV[13] == "1" and observedActive then ' +@@ -623,7 +627,7 @@ const PROVIDER_DRAIN_LUA =* the deadline cannot be extended by retry enumeration. */const RECOVER_TERMINAL_PROVIDER_DRAIN_LUA ='if redis.call("HGET", KEYS[1], "createdAt") ~= ARGV[1] then return 0 end ' +- 'if redis.call("HGET", KEYS[1], "terminalHostActionPending") ~= "1" then return 0 end ' ++ 'if redis.call("HGET", KEYS[1], "terminalHostActionPending") ~= "1" and redis.call("HGET", KEYS[1], "detachedAgentEventTerminalHostActionPending") ~= "1" then return 0 end ' +'if redis.call("HGET", KEYS[1], "providerDrained") ~= "0" then return 0 end ' +'local completedAt = tonumber(redis.call("HGET", KEYS[1], "completedAt") or "") ' +'if not completedAt or completedAt > tonumber(ARGV[2]) then return 0 end ' +@@ -905,12 +909,12 @@ const RUNSTEPS_SAVE_LUA ='if not currentCreatedAt then return 0 end ' +'if ARGV[3] ~= "" and currentCreatedAt ~= ARGV[3] then return 0 end ' +'local currentStatus = redis.call("HGET", KEYS[2], "status") ' +- 'local terminalHostActionPending = redis.call("HGET", KEYS[2], "terminalHostActionPending") ' +- 'if currentStatus ~= "running" and currentStatus ~= "requires_action" and terminalHostActionPending ~= "1" then return 0 end ' ++ 'local terminalHostActionPending = redis.call("HGET", KEYS[2], "terminalHostActionPending") == "1" or redis.call("HGET", KEYS[2], "detachedAgentEventTerminalHostActionPending") == "1" ' ++ 'if currentStatus ~= "running" and currentStatus ~= "requires_action" and not terminalHostActionPending then return 0 end ' +'redis.call("SET", KEYS[1], ARGV[1]) ' +'local run = tonumber(ARGV[2]) ' +'local target = run ' +- 'if currentStatus == "requires_action" or terminalHostActionPending == "1" then ' ++ 'if currentStatus == "requires_action" or terminalHostActionPending then ' +'local jt = redis.call("TTL", KEYS[2]) ' +'if jt > target then target = jt end ' +'end ' +@@ -1586,6 +1590,10 @@ const KEYS = {/** Terminal jobs that still owe a durable host lifecycle hook (global set). Retains* the aborted approval-expiry job for cross-replica / post-restart hook retry. */terminalHostActionJobs: 'stream:terminal_host_action',+ /** Versioned recovery lane for detached Event Actor completion generations.+ * Pre-detached replicas only scan `terminalHostActionJobs`, so they cannot+ * claim a generation whose host hook requires both invocation identities. */+ detachedAgentEventTerminalHostActionJobsV1: 'stream:agent_event_detached:terminal_host_action:v1',/** User's active jobs set, tenant-qualified when tenantId is available */userJobs: (userId: string, tenantId?: string) =>tenantId ? `stream:user:{${tenantId}:${userId}}:jobs` : `stream:user:{${userId}}:jobs`,@@ -1593,6 +1601,10 @@ const KEYS = {idempotency: (key: string) => `stream:idem:${key}`,};+/** Pre-detached creation scripts reject an unknown status before replacement,+ * while capable readers project the private logical terminal status. */+const DETACHED_TERMINAL_LEGACY_STATUS = 'detached_terminal_pending_v1';+interface TerminalHostActionMember {streamId: string;createdAt?: number;@@ -1605,6 +1617,13 @@ function terminalHostActionMember(streamId: string, createdAt: number): string {return JSON.stringify([streamId, createdAt]);}+function isDetachedAgentEventCompletionJob(job: SerializableJobData): boolean {+ // The invocation key alone is enough to make legacy recovery unsafe. Keep a+ // malformed completion with a missing generation timestamp in the capable+ // lane so it fails closed there instead of exposing it to an old consumer.+ return job.agentEventInvocationKey != null;+}+function parseTerminalHostActionMember(member: string): TerminalHostActionMember {try {const parsed = JSON.parse(member) as unknown;@@ -1705,6 +1724,8 @@ interface PendingChunkAppendBatch {}export class RedisJobStore implements IJobStoreV2 {+ readonly detachedAgentEventActionStoreMode = 'distributed' as const;+private redis: Redis | Cluster;private cleanupInterval: NodeJS.Timeout | null = null;private ttl: typeof DEFAULT_TTL;@@ -2316,6 +2337,14 @@ export class RedisJobStore implements IJobStoreV2 {? KEYS.userJobs(job.userId, job.tenantId): null;const terminalMember = job == null ? null : terminalHostActionMember(streamId, job.createdAt);+ const terminalHostActionIndex =+ job != null && isDetachedAgentEventCompletionJob(job)+ ? KEYS.detachedAgentEventTerminalHostActionJobsV1+ : KEYS.terminalHostActionJobs;+ const otherTerminalHostActionIndex =+ terminalHostActionIndex === KEYS.terminalHostActionJobs+ ? KEYS.detachedAgentEventTerminalHostActionJobsV1+ : KEYS.terminalHostActionJobs;if (this.isCluster) {const operations: Promise<unknown>[] = [@@ -2327,11 +2356,23 @@ export class RedisJobStore implements IJobStoreV2 {: this.redis.srem(KEYS.requiresActionJobs, streamId),];if (job?.terminalHostActionPending === true) {- operations.push(this.redis.sadd(KEYS.terminalHostActionJobs, terminalMember!));+ operations.push(this.redis.sadd(terminalHostActionIndex, terminalMember!));+ operations.push(this.redis.srem(otherTerminalHostActionIndex, terminalMember!));+ if (terminalHostActionIndex !== KEYS.terminalHostActionJobs) {+ operations.push(this.redis.srem(KEYS.terminalHostActionJobs, streamId));+ }} else if (terminalMember == null) {operations.push(this.redis.srem(KEYS.terminalHostActionJobs, streamId));+ operations.push(this.redis.srem(KEYS.detachedAgentEventTerminalHostActionJobsV1, streamId));} else {operations.push(this.redis.srem(KEYS.terminalHostActionJobs, streamId, terminalMember));+ operations.push(+ this.redis.srem(+ KEYS.detachedAgentEventTerminalHostActionJobsV1,+ streamId,+ terminalMember,+ ),+ );}// Terminal host-action membership follows the durable hash field, not status, so// an aborted approval-expiry job stays enumerable for hook retry until acked.@@ -2366,12 +2407,15 @@ export class RedisJobStore implements IJobStoreV2 {pipeline.srem(KEYS.requiresActionJobs, streamId);}if (job?.terminalHostActionPending === true) {- pipeline.sadd(KEYS.terminalHostActionJobs, terminalMember!);+ pipeline.sadd(terminalHostActionIndex, terminalMember!);+ pipeline.srem(otherTerminalHostActionIndex, terminalMember!);pipeline.srem(KEYS.terminalHostActionJobs, streamId);} else if (terminalMember != null) {pipeline.srem(KEYS.terminalHostActionJobs, streamId, terminalMember);+ pipeline.srem(KEYS.detachedAgentEventTerminalHostActionJobsV1, streamId, terminalMember);} else {pipeline.srem(KEYS.terminalHostActionJobs, streamId);+ pipeline.srem(KEYS.detachedAgentEventTerminalHostActionJobsV1, streamId);}for (const userJobsKey of observedUserKeys) {if (userJobsKey !== activeUserKey) {@@ -2521,15 +2565,33 @@ export class RedisJobStore implements IJobStoreV2 {): Promise<true | SteerQueueItem[] | null> {const { from, to, patch, clear, expectActionId, expectCreatedAt, notAfterMs } = args;const key = KEYS.job(streamId);+ const terminal = this.statusSetKey(to) === null;+ const terminalJob = terminal ? await this.getJob(streamId) : null;+ const detachedTerminalHostActionPending =+ terminal &&+ patch?.terminalHostActionPending === true &&+ terminalJob != null &&+ isDetachedAgentEventCompletionJob(terminalJob);+ const persistedPatch = detachedTerminalHostActionPending+ ? {+ ...patch,+ status: DETACHED_TERMINAL_LEGACY_STATUS,+ terminalHostActionPending: undefined,+ detachedAgentEventTerminalHostActionPending: true,+ detachedAgentEventTerminalStatus: to as Extract<+ JobStatus,+ 'complete' | 'aborted' | 'error'+ >,+ }+ : patch;// status + patch become HSET pairs; serializeJob skips undefined, so// cleared fields go through HDEL (`clear`) instead.const fields = Object.entries(- this.serializeJob({ status: to, ...(patch ?? {}) } as SerializableJobData),+ this.serializeJob({ status: to, ...(persistedPatch ?? {}) } as SerializableJobData),).flat();const clearFields = (clear ?? []).map(String);- const terminal = this.statusSetKey(to) === null;let ttl = terminal ? this.ttl.completed : this.runningStorageTtlSeconds();if (terminal && patch?.terminalPersistencePending === true) {ttl = Math.max(ttl, TERMINAL_PERSISTENCE_RETENTION_TTL_S);@@ -2546,15 +2608,18 @@ export class RedisJobStore implements IJobStoreV2 {// decision can resume it.ttl = this.pauseTtlSeconds(patch?.pendingAction);}- const terminalJob = terminal ? await this.getJob(streamId) : null;// Redis Cluster cannot atomically update the same-slot job hash and this// global retry index. Arm a generation-scoped retry hint before the// terminal CAS. A predecessor acknowledgement can remove only its own// member, regardless of how a successor's SADD and CAS interleave.const terminalMemberCreatedAt = expectCreatedAt ?? terminalJob?.createdAt;if (terminal && patch?.terminalHostActionPending === true && terminalMemberCreatedAt != null) {+ const terminalHostActionIndex =+ terminalJob != null && isDetachedAgentEventCompletionJob(terminalJob)+ ? KEYS.detachedAgentEventTerminalHostActionJobsV1+ : KEYS.terminalHostActionJobs;await this.redis.sadd(- KEYS.terminalHostActionJobs,+ terminalHostActionIndex,terminalHostActionMember(streamId, terminalMemberCreatedAt),);}@@ -2816,7 +2881,21 @@ export class RedisJobStore implements IJobStoreV2 {}async getTerminalHostActionJobs(): Promise<SerializableJobData[]> {- const members = await this.redis.smembers(KEYS.terminalHostActionJobs);+ return this.getIndexedTerminalHostActionJobs(KEYS.terminalHostActionJobs, false);+ }++ async getDetachedAgentEventTerminalHostActionJobs(): Promise<SerializableJobData[]> {+ return this.getIndexedTerminalHostActionJobs(+ KEYS.detachedAgentEventTerminalHostActionJobsV1,+ true,+ );+ }++ private async getIndexedTerminalHostActionJobs(+ indexKey: string,+ detachedAgentEventCompletion: boolean,+ ): Promise<SerializableJobData[]> {+ const members = await this.redis.smembers(indexKey);if (members.length === 0) {return [];}@@ -2828,6 +2907,8 @@ export class RedisJobStore implements IJobStoreV2 {const stale: string[] = [];const legacy: string[] = [];const migrate: string[] = [];+ const rerouteToLegacy: string[] = [];+ const rerouteToDetached: string[] = [];const heldByGeneration = new Map<string, SerializableJobData>();const readyByGeneration = new Map<string, SerializableJobData>();const providerLossCutoff = Date.now() - PROVIDER_DRAIN_TIMEOUT_MS;@@ -2839,6 +2920,17 @@ export class RedisJobStore implements IJobStoreV2 {job.terminalHostActionPending === true &&(indexedMember.createdAt == null || indexedMember.createdAt === job.createdAt)) {+ const jobIsDetachedCompletion = isDetachedAgentEventCompletionJob(job);+ if (jobIsDetachedCompletion !== detachedAgentEventCompletion) {+ const generationMember = terminalHostActionMember(job.streamId, job.createdAt);+ if (jobIsDetachedCompletion) {+ rerouteToDetached.push(generationMember);+ } else {+ rerouteToLegacy.push(generationMember);+ }+ stale.push(indexedMember.member);+ continue;+ }if (indexedMember.createdAt == null) {legacy.push(indexedMember.member);migrate.push(terminalHostActionMember(job.streamId, job.createdAt));@@ -2870,21 +2962,41 @@ export class RedisJobStore implements IJobStoreV2 {if (job.providerDrained !== false) {readyByGeneration.set(generationKey, job);}+ } else if (+ job != null &&+ indexedMember.createdAt != null &&+ indexedMember.createdAt === job.createdAt+ ) {+ /** The terminal transition pre-arms this exact generation before its+ * hash CAS. Do not delete that hint merely because the CAS has not+ * become visible yet: the producer can commit and die before its+ * post-CAS reconciliation. The hint becomes removable once the exact+ * generation hash is replaced or reaped. */+ continue;} else {stale.push(indexedMember.member);}}+ // Repair a hint written by an earlier capable build before removing it+ // from the wrong lane. In particular, this drains detached completions out+ // of the legacy set without ever returning them to a capable claimant.+ if (rerouteToLegacy.length > 0) {+ await this.redis.sadd(KEYS.terminalHostActionJobs, ...rerouteToLegacy);+ }+ if (rerouteToDetached.length > 0) {+ await this.redis.sadd(KEYS.detachedAgentEventTerminalHostActionJobsV1, ...rerouteToDetached);+ }if (migrate.length > 0) {try {- await this.redis.sadd(KEYS.terminalHostActionJobs, ...migrate);- await this.redis.srem(KEYS.terminalHostActionJobs, ...legacy);+ await this.redis.sadd(indexKey, ...migrate);+ await this.redis.srem(indexKey, ...legacy);} catch {// Preserve the legacy hint if migration cannot prove the replacement// member was written. A duplicate hint is safer than lost discovery.}}if (stale.length > 0) {- await this.redis.srem(KEYS.terminalHostActionJobs, ...stale).catch(() => undefined);+ await this.redis.srem(indexKey, ...stale).catch(() => undefined);}// Enumerating IS the retry attempt: extend each pending job's TTL so unacknowledged// host-action evidence outlives a host dependency (e.g. Mongo) that stays unreachable@@ -2909,7 +3021,10 @@ export class RedisJobStore implements IJobStoreV2 {// member includes this generation, so removing it cannot affect a successor.const cleared = (await this.redis.eval('if redis.call("HGET", KEYS[1], "createdAt") ~= ARGV[1] then return 0 end ' +- 'redis.call("HDEL", KEYS[1], "terminalHostActionPending") ' ++ 'local detachedStatus = redis.call("HGET", KEYS[1], "detachedAgentEventTerminalStatus") ' ++ 'if detachedStatus then redis.call("HSET", KEYS[1], "status", detachedStatus) end ' ++ 'redis.call("HDEL", KEYS[1], "terminalHostActionPending", "detachedAgentEventTerminalHostActionPending", "detachedAgentEventTerminalStatus") ' ++ 'if detachedStatus then redis.call("HDEL", KEYS[1], "lastActiveAt") end ' +'if tonumber(ARGV[2]) > 0 then redis.call("EXPIRE", KEYS[1], ARGV[2]) else redis.call("DEL", KEYS[1]) end ' +'if tonumber(ARGV[3]) > 0 then redis.call("EXPIRE", KEYS[2], ARGV[3]) else redis.call("DEL", KEYS[2]) end ' +'if tonumber(ARGV[4]) > 0 then redis.call("EXPIRE", KEYS[3], ARGV[4]) else redis.call("DEL", KEYS[3]) end ' +@@ -2924,9 +3039,13 @@ export class RedisJobStore implements IJobStoreV2 {String(this.ttl.runStepsAfterComplete),)) as number;if (cleared === 1 && expectedCreatedAt != null) {- await this.redis- .srem(KEYS.terminalHostActionJobs, terminalHostActionMember(streamId, expectedCreatedAt))- .catch(() => undefined);+ const member = terminalHostActionMember(streamId, expectedCreatedAt);+ await Promise.all([+ this.redis.srem(KEYS.terminalHostActionJobs, member).catch(() => undefined),+ this.redis+ .srem(KEYS.detachedAgentEventTerminalHostActionJobsV1, member)+ .catch(() => undefined),+ ]);}}@@ -4699,11 +4818,17 @@ export class RedisJobStore implements IJobStoreV2 {* Deserialize job data from Redis hash.*/private deserializeJob(data: Record<string, string>): SerializableJobData {+ const detachedAgentEventTerminalStatus =+ data.detachedAgentEventTerminalStatus === 'complete' ||+ data.detachedAgentEventTerminalStatus === 'aborted' ||+ data.detachedAgentEventTerminalStatus === 'error'+ ? data.detachedAgentEventTerminalStatus+ : undefined;const job: CreatedJobData = {streamId: data.streamId,userId: data.userId,tenantId: data.tenantId || undefined,- status: data.status as JobStatus,+ status: detachedAgentEventTerminalStatus ?? (data.status as JobStatus),createdAt: parseInt(data.createdAt, 10),generationProtocolVersion: data.generationProtocolVersion === '2' ? 2 : 1,checkpointNamespace: data.checkpointNamespace || undefined,@@ -4776,7 +4901,15 @@ export class RedisJobStore implements IJobStoreV2 {? data.preserveForScheduleReconcile === '1': undefined,terminalHostActionPending:- data.terminalHostActionPending != null ? data.terminalHostActionPending === '1' : undefined,+ data.terminalHostActionPending === '1' ||+ data.detachedAgentEventTerminalHostActionPending === '1'+ ? true+ : undefined,+ detachedAgentEventTerminalHostActionPending:+ data.detachedAgentEventTerminalHostActionPending != null+ ? data.detachedAgentEventTerminalHostActionPending === '1'+ : undefined,+ detachedAgentEventTerminalStatus,// Deferred tools discovered before a HITL pause; replayed into createRun on resume.discoveredTools: data.discoveredTools ? JSON.parse(data.discoveredTools) : undefined,activityPhaseSnapshot: data.activityPhaseSnapshotdiff --git a/packages/api/src/stream/index.ts b/packages/api/src/stream/index.tsindex 3c745f8f10..d7b04fb3cb 100644--- a/packages/api/src/stream/index.ts+++ b/packages/api/src/stream/index.ts@@ -20,6 +20,7 @@ export type {ParkedSteerClaim,IJobStore,IJobStoreV2,+ DetachedAgentEventActionStoreMode,} from './interfaces/IJobStore';// Canonical "is this approval live?" predicate — one definition shared by the// stores, the approval lifecycle, and the status route / message middleware.diff --git a/packages/api/src/stream/interfaces/IJobStore.ts b/packages/api/src/stream/interfaces/IJobStore.tsindex c545bf162e..e23238b483 100644--- a/packages/api/src/stream/interfaces/IJobStore.ts+++ b/packages/api/src/stream/interfaces/IJobStore.ts@@ -11,6 +11,15 @@ import type { ResolvedAskUserQuestion } from '~/agents/hitl/resume';import type { RecoveredSteerPayload } from '../SteerRecovery';import type { MCPRuntimeRequestBody } from '~/mcp/types';+/**+ * Detached Event Actor execution guarantee advertised by a generation store.+ *+ * `process_local` keeps the lifecycle coherent while this process is alive.+ * `distributed` additionally permits restart recovery and replica handoff.+ * Absence means the store cannot host detached Event Actor actions.+ */+export type DetachedAgentEventActionStoreMode = 'process_local' | 'distributed';+/*** Rewrites string-enum members to their literal values, recursively. The SDK and* data-provider declare nominally distinct enums (`ContentTypes`, `StepTypes`, ...)@@ -234,6 +243,14 @@ export interface SerializableJobData {* no action clears it immediately on its no-op success, so nothing accumulates.*/terminalHostActionPending?: boolean;+ /** Redis-only durable marker for a detached Event Actor completion hook.+ * Capable stores expose it through `terminalHostActionPending` as well, but+ * keep the persisted field distinct so legacy reconciliation cannot index or+ * claim the completion through the ordinary terminal-action lane. */+ detachedAgentEventTerminalHostActionPending?: boolean;+ /** Logical terminal state hidden behind a versioned fail-closed shell while+ * a detached Event Actor host action remains unacknowledged. */+ detachedAgentEventTerminalStatus?: Extract<JobStatus, 'complete' | 'aborted' | 'error'>;/*** Last time a cleanup pass enumerated this pending host action for retry. Retention is* measured from this rather than `completedAt`, so evidence survives as long as some@@ -841,6 +858,8 @@ export interface ResumeState {* store at runtime.*/export interface IJobStore {+ readonly detachedAgentEventActionStoreMode?: DetachedAgentEventActionStoreMode;+initialize(): Promise<void>;createJob(@@ -877,6 +896,11 @@ export interface IJobStore {* retry the host adapter after a restart / on another replica, even though the job is* no longer in the requires_action index. */getTerminalHostActionJobs?(): Promise<SerializableJobData[]>;+ /** Enumerates detached Event Actor completion generations from a versioned+ * retry lane known only to capable consumers. Redis keeps this lane separate+ * from `getTerminalHostActionJobs` so a rolling-deployment replica that only+ * understands the legacy job identity can never claim it. */+ getDetachedAgentEventTerminalHostActionJobs?(): Promise<SerializableJobData[]>;/** Clears the pending-host-action marker once the adapter acknowledges success.* Identity-fenced on `expectedCreatedAt` so a replacement generation at the same* streamId is never cleared through its predecessor. */diff --git a/packages/data-schemas/src/methods/triggerDelivery.spec.ts b/packages/data-schemas/src/methods/triggerDelivery.spec.tsindex bbb3b5c5ea..c58a033878 100644--- a/packages/data-schemas/src/methods/triggerDelivery.spec.ts+++ b/packages/data-schemas/src/methods/triggerDelivery.spec.ts@@ -238,6 +238,122 @@ describe('agent trigger delivery methods', () => {expect(claims.filter((claim) => claim != null)).toHaveLength(1);});+ it('keeps capability work visible but nonclaimable to pre-capability consumers', async () => {+ const user = new mongoose.Types.ObjectId();+ const queued = await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ user,+ orderingKey: 'legacy-visible-capability-lane',+ requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,+ }),+ );+ const stored = await Delivery.findById(queued.delivery.id).lean();+ expect(stored).toMatchObject({+ status: 'leased',+ capabilityStatus: 'pending',+ claimAvailableAt: START,+ requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,+ });+ expect(stored!.availableAt).toEqual(START);+ expect(stored!.leaseUntil).toBeUndefined();++ /** Exact pre-shield claim semantics: ordinary pending work or an expired+ * lease. The inert outer lease has no deadline, so it is never claimable. */+ const oldClaim = await Delivery.findOneAndUpdate(+ {+ $or: [+ { status: 'pending', availableAt: { $lte: START } },+ {+ status: 'leased',+ leaseUntil: { $lte: new Date(START.getTime() - 30_000) },+ },+ ],+ },+ {+ $set: {+ status: 'leased',+ leaseBy: 'old-worker',+ leaseUntil: new Date(START.getTime() + 60_000),+ claimToken: 'old-claim',+ },+ },+ { new: true },+ ).lean();+ expect(oldClaim).toBeNull();++ const successor = await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ user,+ orderingKey: 'legacy-visible-capability-lane',+ deliveryKey: 'ordinary-after-shielded-capability',+ }),+ );+ const oldSuccessorClaim = await methods.claimNextAgentTriggerDelivery({+ workerId: 'old-worker',+ claimToken: 'old-successor-claim',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ });+ expect(oldSuccessorClaim).toMatchObject({ id: successor.delivery.id });+ /** Exact pre-shield ordering semantics see the inert lease, but its real+ * due time and missing lease deadline produce a bounded recheck instead of+ * copying the year-9999 claim fence onto the successor. */+ const oldEarlier = await Delivery.findOne({+ orderingKey: 'legacy-visible-capability-lane',+ laneSequence: { $lt: successor.delivery.laneSequence },+ status: { $in: ['pending', 'capability_pending', 'leased', 'capability_leased'] },+ }).lean();+ expect(oldEarlier).toMatchObject({ status: 'leased', availableAt: START });+ expect(oldEarlier!.leaseUntil).toBeUndefined();+ const successorRecheckAt = new Date(START.getTime() + 5_000);+ await expect(+ methods.releaseAgentTriggerDelivery({+ id: oldSuccessorClaim!.id,+ workerId: 'old-worker',+ claimToken: 'old-successor-claim',+ availableAt: successorRecheckAt,+ }),+ ).resolves.toBe(true);+ await expect(+ Delivery.countDocuments({ user, status: 'leased', leaseUntil: { $gt: START } }),+ ).resolves.toBe(0);++ const capabilityClaim = await methods.claimNextAgentTriggerDelivery({+ workerId: 'capable-worker',+ claimToken: 'capable-claim',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1],+ });+ expect(capabilityClaim).toMatchObject({ id: queued.delivery.id });+ await expect(+ Delivery.countDocuments({ user, status: 'leased', leaseUntil: { $gt: START } }),+ ).resolves.toBe(1);+ const attempt = await methods.beginAgentTriggerDeliveryAttempt({+ id: capabilityClaim!.id,+ workerId: 'capable-worker',+ claimToken: 'capable-claim',+ now: START,+ });+ await methods.deadLetterAgentTriggerDelivery({+ id: capabilityClaim!.id,+ workerId: 'capable-worker',+ claimToken: 'capable-claim',+ attempt: attempt!,+ error: transientFailure(),+ settledAt: START,+ });+ await expect(+ methods.claimNextAgentTriggerDelivery({+ workerId: 'new-worker',+ claimToken: 'successor-after-capability',+ now: successorRecheckAt,+ leaseUntil: new Date(successorRecheckAt.getTime() + 60_000),+ workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1],+ }),+ ).resolves.toMatchObject({ id: successor.delivery.id });+ });+it('keeps capability-fenced work limited to capable workers through lease recovery', async () => {const queued = await methods.enqueueAgentTriggerDelivery(enqueueInput({@@ -270,7 +386,7 @@ describe('agent trigger delivery methods', () => {await Delivery.updateOne({ _id: capable!.id },- { $set: { leaseUntil: new Date(START.getTime() - 60_000) } },+ { $set: { capabilityLeaseUntil: new Date(START.getTime() - 60_000) } },);const recoveryNow = new Date(START.getTime() + 60_000);await expect(@@ -309,6 +425,7 @@ describe('agent trigger delivery methods', () => {).resolves.toBe(true);await expect(Delivery.findById(recovered!.id).lean()).resolves.toMatchObject({status: 'capability_dead',+ capabilityStatus: 'dead',});const [deadLetter] = await methods.getAgentTriggerDeadLetters();@@ -374,6 +491,307 @@ describe('agent trigger delivery methods', () => {).resolves.toMatchObject({ id: ordinary.delivery.id, status: 'leased' });});+ it('claims older capability work before newer ordinary work', async () => {+ const capability = await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ deliveryKey: 'capability-older',+ availableAt: new Date(START.getTime() - 2_000),+ requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,+ }),+ );+ await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ deliveryKey: 'ordinary-newer',+ availableAt: new Date(START.getTime() - 1_000),+ }),+ );++ await expect(+ methods.claimNextAgentTriggerDelivery({+ workerId: 'capable-worker',+ claimToken: 'claim-oldest-capability',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1],+ }),+ ).resolves.toMatchObject({ id: capability.delivery.id, status: 'capability_leased' });+ });++ it('adopts capability publication completed by a legacy lane publisher', async () => {+ const user = new mongoose.Types.ObjectId();+ const orderingKey = 'legacy-capability-publication';+ const capability = await Delivery.create({+ ...enqueueInput({+ user,+ orderingKey,+ requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,+ }),+ status: 'staging',+ capabilityStatus: 'publishing',+ claimAvailableAt: START,+ availableAt: new Date('9999-12-31T23:59:59.999Z'),+ leaseUntil: new Date('9999-12-31T23:59:59.999Z'),+ laneSequence: 0,+ attempts: 0,+ requeueCount: 0,+ stagingRecoveryAt: START,+ });+ await LaneSequence.create({+ _id: orderingKey,+ value: 1,+ user,+ publisherDeliveryId: capability._id,+ publisherStartedAt: START,+ });++ /** Exact pre-capability publication writes: the old replica assigns the+ * reserved sequence and releases the publisher without touching private fields. */+ await Delivery.updateOne(+ { _id: capability._id, orderingKey, status: 'staging' },+ {+ $set: { status: 'capability_pending', laneSequence: 1 },+ $unset: { stagingRecoveryAt: 1 },+ },+ );+ await LaneSequence.updateOne(+ { _id: orderingKey, value: 1, publisherDeliveryId: capability._id },+ {+ $set: { tailDeliveryId: capability._id },+ $unset: { publisherDeliveryId: 1, publisherStartedAt: 1 },+ },+ );++ const later = await methods.enqueueAgentTriggerDelivery(+ enqueueInput({ user, orderingKey, deliveryKey: 'ordinary-after-capability' }),+ );++ await expect(Delivery.findById(capability._id).lean()).resolves.toMatchObject({+ status: 'leased',+ capabilityStatus: 'pending',+ laneSequence: 1,+ });+ expect(later.delivery).toMatchObject({ laneSequence: 2 });+ });++ it('keeps a privately dead capability delivery nonblocking to old and new workers', async () => {+ const orderingKey = 'dead-capability-lane';+ await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ orderingKey,+ requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,+ }),+ );+ const claim = await methods.claimNextAgentTriggerDelivery({+ workerId: 'capable-worker',+ claimToken: 'capable-dead-claim',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1],+ });+ await methods.beginAgentTriggerDeliveryAttempt({+ id: claim!.id,+ workerId: 'capable-worker',+ claimToken: 'capable-dead-claim',+ now: START,+ });+ await methods.deadLetterAgentTriggerDelivery({+ id: claim!.id,+ workerId: 'capable-worker',+ claimToken: 'capable-dead-claim',+ attempt: 1,+ error: transientFailure(),+ settledAt: START,+ });+ const successor = await methods.enqueueAgentTriggerDelivery(+ enqueueInput({ orderingKey, deliveryKey: 'successor-after-dead-capability' }),+ );++ await expect(Delivery.findById(claim!.id).lean()).resolves.toMatchObject({+ status: 'capability_dead',+ capabilityStatus: 'dead',+ });+ /** Exact pre-shield ordering query. A `pending` shell here would make an+ * old worker defer the successor until the year-9999 availability fence. */+ const oldEarlier = await Delivery.findOne({+ orderingKey,+ laneSequence: { $lt: successor.delivery.laneSequence },+ $or: [+ {+ status: {+ $in: ['pending', 'capability_pending', 'leased', 'capability_leased'],+ },+ },+ {+ status: 'succeeded',+ batchRootId: { $exists: false },+ awaitTerminalHandling: true,+ 'handling.status': 'started',+ },+ ],+ }).lean();+ expect(oldEarlier).toBeNull();+ await expect(methods.findEarlierAgentTriggerDelivery(successor.delivery)).resolves.toBeNull();+ await expect(+ methods.claimNextAgentTriggerDelivery({+ workerId: 'new-worker',+ claimToken: 'new-worker-successor-claim',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1],+ }),+ ).resolves.toMatchObject({ id: successor.delivery.id });+ });++ it('adopts a private dead delivery explicitly requeued by a pre-shield replica', async () => {+ await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ orderingKey: 'legacy-requeued-dead-capability',+ requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,+ }),+ );+ const claim = await methods.claimNextAgentTriggerDelivery({+ workerId: 'capable-worker',+ claimToken: 'claim-before-legacy-requeue',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1],+ });+ await methods.beginAgentTriggerDeliveryAttempt({+ id: claim!.id,+ workerId: 'capable-worker',+ claimToken: 'claim-before-legacy-requeue',+ now: START,+ });+ await methods.deadLetterAgentTriggerDelivery({+ id: claim!.id,+ workerId: 'capable-worker',+ claimToken: 'claim-before-legacy-requeue',+ attempt: 1,+ error: transientFailure(),+ settledAt: START,+ });++ const requeueAt = new Date(START.getTime() + 1_000);+ /** Final state produced by the pre-shield requeue implementation: it+ * recognizes `capability_dead`, republishes `capability_pending`, and does+ * not know that the private lifecycle field must also be reset. */+ await Delivery.updateOne(+ { _id: claim!.id, status: 'capability_dead', capabilityStatus: 'dead' },+ {+ $set: { status: 'capability_pending', availableAt: requeueAt },+ $unset: { settledAt: 1, lastError: 1 },+ $inc: { requeueCount: 1 },+ },+ );++ const reclaimed = await methods.claimNextAgentTriggerDelivery({+ workerId: 'new-worker',+ claimToken: 'claim-after-legacy-requeue',+ now: requeueAt,+ leaseUntil: new Date(requeueAt.getTime() + 60_000),+ workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1],+ });+ expect(reclaimed).toMatchObject({ id: claim!.id, status: 'capability_leased' });+ await expect(Delivery.findById(claim!.id).lean()).resolves.toMatchObject({+ status: 'leased',+ capabilityStatus: 'leased',+ claimAvailableAt: requeueAt,+ });+ });++ it('accepts legacy success as terminal over stale private dead state', async () => {+ const queued = await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ orderingKey: 'legacy-completed-dead-capability',+ requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,+ }),+ );+ const claim = await methods.claimNextAgentTriggerDelivery({+ workerId: 'capable-worker',+ claimToken: 'claim-before-legacy-success',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1],+ });+ await methods.beginAgentTriggerDeliveryAttempt({+ id: claim!.id,+ workerId: 'capable-worker',+ claimToken: 'claim-before-legacy-success',+ now: START,+ });+ await methods.deadLetterAgentTriggerDelivery({+ id: claim!.id,+ workerId: 'capable-worker',+ claimToken: 'claim-before-legacy-success',+ attempt: 1,+ error: transientFailure(),+ settledAt: START,+ });++ const requeueAt = new Date(START.getTime() + 1_000);+ await Delivery.updateOne(+ { _id: claim!.id, status: 'capability_dead', capabilityStatus: 'dead' },+ {+ $set: { status: 'capability_pending', availableAt: requeueAt },+ $unset: { settledAt: 1, lastError: 1 },+ $inc: { requeueCount: 1 },+ },+ );+ const legacyLeaseUntil = new Date(requeueAt.getTime() + 60_000);+ const legacyClaim = await Delivery.findOneAndUpdate(+ { _id: claim!.id, status: 'capability_pending', availableAt: { $lte: requeueAt } },+ {+ $set: {+ status: 'capability_leased',+ leaseBy: 'old-worker',+ leaseUntil: legacyLeaseUntil,+ claimToken: 'old-success-claim',+ },+ },+ { new: true },+ ).lean();+ expect(legacyClaim).toMatchObject({+ status: 'capability_leased',+ capabilityStatus: 'dead',+ });++ const completedAt = new Date(requeueAt.getTime() + 1_000);+ /** Exact terminal write left by the pre-shield worker: the outer lifecycle+ * succeeds while the private lifecycle remains unknown and stale. */+ await Delivery.updateOne(+ {+ _id: claim!.id,+ status: 'capability_leased',+ leaseBy: 'old-worker',+ claimToken: 'old-success-claim',+ },+ {+ $set: {+ status: 'succeeded',+ attempts: 2,+ result: { status: 'legacy-complete' },+ settledAt: completedAt,+ },+ $unset: {+ leaseBy: 1,+ leaseUntil: 1,+ claimToken: 1,+ lastError: 1,+ },+ },+ );++ await expect(+ methods.getAgentTriggerDelivery(queued.delivery.deliveryKey),+ ).resolves.toMatchObject({+ status: 'succeeded',+ result: { status: 'legacy-complete' },+ });+ await expect(methods.getAgentTriggerDeadLetters()).resolves.not.toEqual(+ expect.arrayContaining([expect.objectContaining({ id: claim!.id })]),+ );+ });+it('allocates a replica-stable monotonic sequence within an ordering lane', async () => {const user = new mongoose.Types.ObjectId();const replicas = [@@ -409,7 +827,10 @@ describe('agent trigger delivery methods', () => {const updateSpy = jest.spyOn(Delivery.collection, 'updateOne').mockImplementation(async (filter, update, options) => {- const status = filter.status as { $in?: string[] } | undefined;+ const status = (filter.status ??+ (filter.$or as Array<{ status?: { $in?: string[] } }> | undefined)?.find(+ (condition) => condition.status != null,+ )?.status) as { $in?: string[] } | undefined;if (!paused &&filter.orderingKey === orderingKey &&diff --git a/packages/data-schemas/src/methods/triggerDelivery.ts b/packages/data-schemas/src/methods/triggerDelivery.tsindex c507276d1a..a9ae5add5f 100644--- a/packages/data-schemas/src/methods/triggerDelivery.ts+++ b/packages/data-schemas/src/methods/triggerDelivery.ts@@ -31,11 +31,32 @@ const MAX_PURGE_RECOVERY_LIMIT = 200;const HISTORY_LIMIT = 64;const MAX_BATCH_SIZE = 8;const MAX_BATCH_BYTES = 512 * 1024;+/** Capability work is inert to legacy claimers while preserving their lane+ * behavior: publishing is `staging`; queued work is `leased` without a lease+ * owner/deadline; execution adds a private lease; dead work is terminal. */+const LEGACY_CAPABILITY_SHIELD_AT = new Date('9999-12-31T23:59:59.999Z');+const LEGACY_CAPABILITY_SHIELD_OWNER = 'librechat-capability-shield';function isStagingStatus(status: IAgentTriggerDelivery['status']): boolean {return status === 'staging' || status === 'capability_staging';}+function isStagingDelivery(+ delivery: Pick<IAgentTriggerDelivery, 'status' | 'capabilityStatus'>,+): boolean {+ return isStagingStatus(delivery.status) || delivery.capabilityStatus === 'publishing';+}++const stagingDeliveryScope = {+ $or: [{ status: { $in: ['staging', 'capability_staging'] } }, { capabilityStatus: 'publishing' }],+};+const capabilityStatusProjection = {+ publishing: 'capability_staging',+ pending: 'capability_pending',+ leased: 'capability_leased',+ dead: 'capability_dead',+} as const;+type DuplicateKeyError = { code?: number };export type AgentEventActorReceiptMetric = {@@ -310,6 +331,15 @@ function toRecord(delivery: IAgentTriggerDelivery): AgentTriggerDeliveryRecord {if (delivery._id == null || delivery.createdAt == null) {throw new Error('Persisted agent trigger delivery is missing its identity or creation time');}+ /** A pre-shield worker can requeue and finish a capability delivery without+ * knowing to update its private lifecycle. Its terminal success is exact+ * execution evidence, so stale private metadata cannot override it. */+ const shieldedCapability =+ delivery.requiredWorkerCapability != null && delivery.status !== 'succeeded'+ ? delivery.capabilityStatus+ : undefined;+ const projectedStatus =+ shieldedCapability == null ? delivery.status : capabilityStatusProjection[shieldedCapability];return {id: String(delivery._id),deliveryKey: delivery.deliveryKey,@@ -318,17 +348,25 @@ function toRecord(delivery: IAgentTriggerDelivery): AgentTriggerDeliveryRecord {laneSequence: delivery.laneSequence,envelope: delivery.envelope,user: delivery.user,- status: delivery.status,+ status: projectedStatus,attempts: delivery.attempts,- availableAt: delivery.availableAt,+ availableAt: shieldedCapability+ ? (delivery.claimAvailableAt ?? delivery.availableAt)+ : delivery.availableAt,createdAt: delivery.createdAt,...(delivery.tenantId != null && { tenantId: delivery.tenantId }),...(delivery.requiredWorkerCapability != null && {requiredWorkerCapability: delivery.requiredWorkerCapability,}),- ...(delivery.leaseBy != null && { leaseBy: delivery.leaseBy }),- ...(delivery.leaseUntil != null && { leaseUntil: delivery.leaseUntil }),- ...(delivery.claimToken != null && { claimToken: delivery.claimToken }),+ ...((shieldedCapability ? delivery.capabilityLeaseBy : delivery.leaseBy) != null && {+ leaseBy: shieldedCapability ? delivery.capabilityLeaseBy : delivery.leaseBy,+ }),+ ...((shieldedCapability ? delivery.capabilityLeaseUntil : delivery.leaseUntil) != null && {+ leaseUntil: shieldedCapability ? delivery.capabilityLeaseUntil : delivery.leaseUntil,+ }),+ ...((shieldedCapability ? delivery.capabilityClaimToken : delivery.claimToken) != null && {+ claimToken: shieldedCapability ? delivery.capabilityClaimToken : delivery.claimToken,+ }),...(delivery.lastError != null && { lastError: delivery.lastError }),...(delivery.result !== undefined && { result: delivery.result }),...(delivery.history != null && { history: delivery.history }),@@ -524,6 +562,8 @@ export function createAgentTriggerDeliveryMethods(let publishedStatus: IAgentTriggerDelivery['status'] = 'pending';if (batchRoot != null) {publishedStatus = 'batched';+ } else if (staged?.capabilityStatus === 'publishing') {+ publishedStatus = 'leased';} else if (staged?.requiredWorkerCapability != null) {publishedStatus = 'capability_pending';}@@ -531,37 +571,53 @@ export function createAgentTriggerDeliveryMethods({_id: publisherDeliveryId,orderingKey: lane._id,- status: { $in: ['staging', 'capability_staging'] },+ ...stagingDeliveryScope,requeueCount: publicationGeneration,},{$set: {status: publishedStatus,laneSequence: lane.value,+ ...(staged?.capabilityStatus === 'publishing' && {+ capabilityStatus: 'pending',+ availableAt: staged.claimAvailableAt ?? staged.availableAt,+ }),...(batchRoot?._id != null && {batchRootId: batchRoot._id,batchRootRequeueCount: batchRoot.requeueCount ?? 0,}),},- $unset: { stagingRecoveryAt: 1 },+ $unset: {+ stagingRecoveryAt: 1,+ ...(staged?.capabilityStatus === 'publishing' && {+ leaseBy: 1,+ leaseUntil: 1,+ claimToken: 1,+ }),+ },},);let publicationCommitted = published.modifiedCount === 1;if (published.modifiedCount === 0) {let current = await Delivery().findById(publisherDeliveryId)- .select('orderingKey laneSequence status batchRootId requeueCount')+ .select('orderingKey laneSequence status capabilityStatus batchRootId requeueCount').lean<Pick<IAgentTriggerDelivery,- 'orderingKey' | 'laneSequence' | 'status' | 'batchRootId' | 'requeueCount'+ | 'orderingKey'+ | 'laneSequence'+ | 'status'+ | 'capabilityStatus'+ | 'batchRootId'+ | 'requeueCount'>>();if (batchRoot?._id != null &¤t != null &¤t.orderingKey === lane._id &&- !isStagingStatus(current.status) &&+ !isStagingDelivery(current) &¤t.laneSequence !== lane.value) {await Delivery().updateOne(@@ -581,21 +637,26 @@ export function createAgentTriggerDeliveryMethods();current = await Delivery().findById(publisherDeliveryId)- .select('orderingKey laneSequence status batchRootId requeueCount')+ .select('orderingKey laneSequence status capabilityStatus batchRootId requeueCount').lean<Pick<IAgentTriggerDelivery,- 'orderingKey' | 'laneSequence' | 'status' | 'batchRootId' | 'requeueCount'+ | 'orderingKey'+ | 'laneSequence'+ | 'status'+ | 'capabilityStatus'+ | 'batchRootId'+ | 'requeueCount'>>();}publicationCommitted =current != null &¤t.orderingKey === lane._id &&- !isStagingStatus(current.status) &&+ !isStagingDelivery(current) &¤t.requeueCount === publicationGeneration &¤t.laneSequence === lane.value;- if (current?.orderingKey === lane._id && isStagingStatus(current.status)) {+ if (current?.orderingKey === lane._id && isStagingDelivery(current)) {if (current.requeueCount !== publicationGeneration) {return abandonLanePublisher(publicationLane);}@@ -683,14 +744,65 @@ export function createAgentTriggerDeliveryMethods(const deliveryId = delivery._id;const orderingKey = delivery.orderingKey;+ const adoptLegacyPublishedCapability = async (+ candidate: IAgentTriggerDelivery,+ ): Promise<IAgentTriggerDelivery | null> => {+ if (+ candidate._id == null ||+ candidate.capabilityStatus !== 'publishing' ||+ !['pending', 'capability_pending'].includes(candidate.status) ||+ candidate.laneSequence <= 0+ ) {+ return null;+ }+ const adopted = await Delivery()+ .findOneAndUpdate(+ {+ _id: candidate._id,+ status: { $in: ['pending', 'capability_pending'] },+ capabilityStatus: 'publishing',+ laneSequence: candidate.laneSequence,+ },+ {+ $set: {+ status: 'leased',+ capabilityStatus: 'pending',+ availableAt: candidate.claimAvailableAt ?? candidate.availableAt,+ },+ $unset: {+ stagingRecoveryAt: 1,+ leaseBy: 1,+ leaseUntil: 1,+ claimToken: 1,+ },+ },+ { new: true },+ )+ .lean<IAgentTriggerDelivery>();+ if (adopted == null) {+ return null;+ }+ const adoptedLane = await LaneSequence()+ .findOne({ _id: orderingKey, publisherDeliveryId: candidate._id })+ .lean<IAgentTriggerLaneSequence>();+ if (adoptedLane != null) {+ await recoverLanePublisher(adoptedLane);+ }+ return adopted;+ };+for (;;) {const current = await Delivery().findById(deliveryId).lean<IAgentTriggerDelivery>();if (current == null) {throw new Error('Staged agent trigger delivery disappeared before publication');}- if (!isStagingStatus(current.status)) {+ if (!isStagingDelivery(current)) {return current;}+ const adoptedCurrent = await adoptLegacyPublishedCapability(current);+ if (adoptedCurrent != null) {+ return adoptedCurrent;+ }if ((await UserPurge().exists({ _id: current.user })) != null) {return current;}@@ -705,11 +817,25 @@ export function createAgentTriggerDeliveryMethods(continue;}+ const legacyPublished = await Delivery()+ .findOne({+ orderingKey,+ status: { $in: ['pending', 'capability_pending'] },+ capabilityStatus: 'publishing',+ laneSequence: { $gt: 0 },+ })+ .sort({ laneSequence: 1, _id: 1 })+ .lean<IAgentTriggerDelivery>();+ if (legacyPublished != null) {+ await adoptLegacyPublishedCapability(legacyPublished);+ continue;+ }+// The durable staging row exists before sequence allocation. Publishing// the oldest visible row first lets any replica repair a writer that// stopped in that gap without allowing a later enqueue to overtake it.const next = await Delivery()- .findOne({ orderingKey, status: { $in: ['staging', 'capability_staging'] } })+ .findOne({ orderingKey, ...stagingDeliveryScope })// updatedAt is the current staging admission time: initial enqueue// sets it on insert, while explicit requeue refreshes it so an old// dead letter is admitted behind staging work that already exists.@@ -778,6 +904,13 @@ export function createAgentTriggerDeliveryMethods(try {const created = await Delivery().create({...input,+ ...(input.requiredWorkerCapability == null+ ? {}+ : {+ capabilityStatus: 'publishing',+ claimAvailableAt: input.availableAt,+ leaseUntil: LEGACY_CAPABILITY_SHIELD_AT,+ }),...(input.coalesceKey == null? {}: {@@ -786,7 +919,10 @@ export function createAgentTriggerDeliveryMethods(batchMemberIds: [],}),laneSequence: 0,- status: input.requiredWorkerCapability == null ? 'staging' : 'capability_staging',+ status: 'staging',+ claimAvailableAt: input.availableAt,+ availableAt:+ input.requiredWorkerCapability == null ? input.availableAt : LEGACY_CAPABILITY_SHIELD_AT,attempts: 0,requeueCount: 0,stagingRecoveryAt: new Date(),@@ -805,7 +941,7 @@ export function createAgentTriggerDeliveryMethods(if (existing.fingerprint !== input.fingerprint) {throw new AgentTriggerDeliveryConflictError(input.deliveryKey);}- if (!isStagingStatus(existing.status)) {+ if (!isStagingDelivery(existing)) {return { delivery: toRecord(existing), replayed: true };}staged = existing;@@ -866,7 +1002,7 @@ export function createAgentTriggerDeliveryMethods(* index. Backfill them in bounded batches before using the indexed scan. */const legacyStaged = await Delivery().find({- status: { $in: ['staging', 'capability_staging'] },+ ...stagingDeliveryScope,stagingRecoveryAt: { $exists: false },}).sort({ updatedAt: 1, _id: 1 })@@ -878,7 +1014,7 @@ export function createAgentTriggerDeliveryMethods(await Delivery().updateOne({_id: delivery._id,- status: { $in: ['staging', 'capability_staging'] },+ ...stagingDeliveryScope,stagingRecoveryAt: { $exists: false },},{ $set: { stagingRecoveryAt: legacyCursor } },@@ -894,7 +1030,7 @@ export function createAgentTriggerDeliveryMethods(remaining > 0? await Delivery().find({- status: { $in: ['staging', 'capability_staging'] },+ ...stagingDeliveryScope,stagingRecoveryAt: { $exists: true },...(legacyIds.length > 0 && { _id: { $nin: legacyIds } }),})@@ -914,7 +1050,7 @@ export function createAgentTriggerDeliveryMethods(await Delivery().updateOne({_id: delivery._id,- status: { $in: ['staging', 'capability_staging'] },+ ...stagingDeliveryScope,stagingRecoveryAt: delivery.stagingRecoveryAt,},{ $set: { stagingRecoveryAt: stagingRecoveryCursor } },@@ -924,7 +1060,7 @@ export function createAgentTriggerDeliveryMethods(for (const delivery of staged) {try {const published = await publishStagedDelivery(delivery);- if (!isStagingStatus(published.status)) {+ if (!isStagingDelivery(published)) {recovered += 1;} else {await rotateStagingRecovery(delivery);@@ -1069,6 +1205,18 @@ export function createAgentTriggerDeliveryMethods(workerCapabilities.length === 0? []: [+ {+ requiredWorkerCapability: { $in: workerCapabilities },+ capabilityStatus: 'pending',+ claimAvailableAt: { $lte: input.now },+ status: { $in: ['pending', 'leased'] },+ },+ {+ requiredWorkerCapability: { $in: workerCapabilities },+ capabilityStatus: 'leased',+ capabilityLeaseUntil: { $lte: expiredBefore },+ status: 'leased',+ },{requiredWorkerCapability: { $in: workerCapabilities },status: 'capability_pending',@@ -1079,13 +1227,31 @@ export function createAgentTriggerDeliveryMethods(status: 'capability_leased',leaseUntil: { $lte: expiredBefore },},+ {+ /** A pre-shield replica can explicitly requeue the legacy-visible+ * dead shell. Its publication leaves the private lifecycle at+ * `dead`; the capable claimant atomically adopts that requeue. */+ requiredWorkerCapability: { $in: workerCapabilities },+ status: 'capability_pending',+ capabilityStatus: 'dead',+ settledAt: { $exists: false },+ availableAt: { $lte: input.now },+ },];const claimed = await Delivery().findOneAndUpdate({$or: [- { status: 'pending', availableAt: { $lte: input.now } },- { status: 'leased', leaseUntil: { $lte: expiredBefore } },+ {+ requiredWorkerCapability: { $exists: false },+ status: 'pending',+ availableAt: { $lte: input.now },+ },+ {+ requiredWorkerCapability: { $exists: false },+ status: 'leased',+ leaseUntil: { $lte: expiredBefore },+ },...capabilityConditions,],},@@ -1094,21 +1260,81 @@ export function createAgentTriggerDeliveryMethods($set: {status: {$cond: [- { $in: ['$status', ['capability_pending', 'capability_leased']] },- 'capability_leased',+ { $eq: [{ $type: '$capabilityStatus' }, 'missing'] },+ {+ $cond: [+ { $in: ['$status', ['capability_pending', 'capability_leased']] },+ 'capability_leased',+ 'leased',+ ],+ },'leased',],},- leaseBy: input.workerId,- leaseUntil: input.leaseUntil,- claimToken: input.claimToken,+ claimAvailableAt: {+ $cond: [+ {+ $and: [+ { $eq: ['$status', 'capability_pending'] },+ { $eq: ['$capabilityStatus', 'dead'] },+ ],+ },+ '$availableAt',+ '$claimAvailableAt',+ ],+ },+ capabilityStatus: {+ $cond: [{ $eq: [{ $type: '$capabilityStatus' }, 'missing'] }, '$$REMOVE', 'leased'],+ },+ capabilityLeaseBy: {+ $cond: [+ { $eq: [{ $type: '$capabilityStatus' }, 'missing'] },+ '$$REMOVE',+ input.workerId,+ ],+ },+ capabilityLeaseUntil: {+ $cond: [+ { $eq: [{ $type: '$capabilityStatus' }, 'missing'] },+ '$$REMOVE',+ input.leaseUntil,+ ],+ },+ capabilityClaimToken: {+ $cond: [+ { $eq: [{ $type: '$capabilityStatus' }, 'missing'] },+ '$$REMOVE',+ input.claimToken,+ ],+ },+ leaseBy: {+ $cond: [+ { $eq: [{ $type: '$capabilityStatus' }, 'missing'] },+ input.workerId,+ LEGACY_CAPABILITY_SHIELD_OWNER,+ ],+ },+ leaseUntil: {+ $cond: [+ { $eq: [{ $type: '$capabilityStatus' }, 'missing'] },+ input.leaseUntil,+ LEGACY_CAPABILITY_SHIELD_AT,+ ],+ },+ claimToken: {+ $cond: [+ { $eq: [{ $type: '$capabilityStatus' }, 'missing'] },+ input.claimToken,+ LEGACY_CAPABILITY_SHIELD_OWNER,+ ],+ },settledAt: '$$REMOVE',expiresAt: '$$REMOVE',updatedAt: input.now,},},],- { new: true, sort: { availableAt: 1, createdAt: 1, _id: 1 } },+ { new: true, sort: { claimAvailableAt: 1, availableAt: 1, createdAt: 1, _id: 1 } },).lean<IAgentTriggerDelivery>();return requireClaim(claimed);@@ -1123,9 +1349,15 @@ export function createAgentTriggerDeliveryMethods(laneSequence: { $lt: delivery.laneSequence },$or: [{- status: {- $in: ['pending', 'capability_pending', 'leased', 'capability_leased'],- },+ requiredWorkerCapability: { $exists: false },+ status: { $in: ['pending', 'leased'] },+ },+ {+ status: { $in: ['capability_pending', 'capability_leased'] },+ },+ {+ requiredWorkerCapability: { $exists: true },+ capabilityStatus: { $in: ['publishing', 'pending', 'leased'] },},{status: 'succeeded',@@ -1136,14 +1368,31 @@ export function createAgentTriggerDeliveryMethods(],}).sort({ laneSequence: 1 })- .select('availableAt leaseUntil status handling.status')- .lean<Pick<IAgentTriggerDelivery, 'availableAt' | 'leaseUntil' | 'status' | 'handling'>>();+ .select(+ 'availableAt claimAvailableAt leaseUntil capabilityLeaseUntil status capabilityStatus handling.status',+ )+ .lean<+ Pick<+ IAgentTriggerDelivery,+ | 'availableAt'+ | 'claimAvailableAt'+ | 'leaseUntil'+ | 'capabilityLeaseUntil'+ | 'status'+ | 'capabilityStatus'+ | 'handling'+ >+ >();if (earlier == null) {return null;}+ const shieldedCapability = earlier.capabilityStatus != null;+ const leaseUntil = shieldedCapability ? earlier.capabilityLeaseUntil : earlier.leaseUntil;return {- availableAt: earlier.availableAt,- ...(earlier.leaseUntil != null && { leaseUntil: earlier.leaseUntil }),+ availableAt: shieldedCapability+ ? (earlier.claimAvailableAt ?? earlier.availableAt)+ : earlier.availableAt,+ ...(leaseUntil != null && { leaseUntil }),...(earlier.status === 'succeeded' &&earlier.handling?.status === 'started' && { reason: 'active_handling' as const }),};@@ -1388,33 +1637,83 @@ export function createAgentTriggerDeliveryMethods(return recovered;}- const fence = (input: AgentTriggerDeliveryFence) => ({- _id: input.id,- status: { $in: ['leased', 'capability_leased'] },+ const ordinaryFence = (input: AgentTriggerDeliveryFence) => ({+ status: 'leased' as const,+ requiredWorkerCapability: { $exists: false },leaseBy: input.workerId,claimToken: input.claimToken,});- const capabilityFence = (input: AgentTriggerDeliveryFence) => ({- ...fence(input),+ const legacyCapabilityFence = (input: AgentTriggerDeliveryFence) => ({status: 'capability_leased',requiredWorkerCapability: { $exists: true },+ leaseBy: input.workerId,+ claimToken: input.claimToken,+ });++ const shieldCapabilityFence = (input: AgentTriggerDeliveryFence) => ({+ status: 'leased' as const,+ requiredWorkerCapability: { $exists: true },+ capabilityStatus: 'leased' as const,+ capabilityLeaseBy: input.workerId,+ capabilityClaimToken: input.claimToken,+ });++ const fence = (input: AgentTriggerDeliveryFence) => ({+ _id: input.id,+ $or: [ordinaryFence(input), legacyCapabilityFence(input), shieldCapabilityFence(input)],});async function releaseAgentTriggerDelivery(input: AgentTriggerDeliveryFence & { availableAt: Date },): Promise<boolean> {- const capabilityResult = await Delivery().updateOne(capabilityFence(input), {- $set: { status: 'capability_pending', availableAt: input.availableAt },- $unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1 },- });+ const shieldResult = await Delivery().updateOne(+ { _id: input.id, ...shieldCapabilityFence(input) },+ {+ $set: {+ status: 'leased',+ availableAt: input.availableAt,+ capabilityStatus: 'pending',+ claimAvailableAt: input.availableAt,+ },+ $unset: {+ leaseBy: 1,+ leaseUntil: 1,+ claimToken: 1,+ capabilityLeaseBy: 1,+ capabilityLeaseUntil: 1,+ capabilityClaimToken: 1,+ },+ },+ );+ if (shieldResult.modifiedCount === 1) {+ return true;+ }+ const capabilityResult = await Delivery().updateOne(+ { _id: input.id, ...legacyCapabilityFence(input) },+ {+ $set: {+ status: 'capability_pending',+ availableAt: input.availableAt,+ claimAvailableAt: input.availableAt,+ },+ $unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1 },+ },+ );if (capabilityResult.modifiedCount === 1) {return true;}- const result = await Delivery().updateOne(fence(input), {- $set: { status: 'pending', availableAt: input.availableAt },- $unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1 },- });+ const result = await Delivery().updateOne(+ { _id: input.id, ...ordinaryFence(input) },+ {+ $set: {+ status: 'pending',+ availableAt: input.availableAt,+ claimAvailableAt: input.availableAt,+ },+ $unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1 },+ },+ );return result.modifiedCount === 1;}@@ -1423,7 +1722,17 @@ export function createAgentTriggerDeliveryMethods(): Promise<number | null> {const updated = await Delivery().findOneAndUpdate(- { ...fence(input), leaseUntil: { $gt: input.now } },+ {+ _id: input.id,+ $or: [+ { ...ordinaryFence(input), leaseUntil: { $gt: input.now } },+ { ...legacyCapabilityFence(input), leaseUntil: { $gt: input.now } },+ {+ ...shieldCapabilityFence(input),+ capabilityLeaseUntil: { $gt: input.now },+ },+ ],+ },{ $inc: { attempts: 1 } },{ new: true },)@@ -1441,11 +1750,34 @@ export function createAgentTriggerDeliveryMethods(}const update = {$inc: { attempts: -1 },- $set: { availableAt: input.availableAt },+ $set: { availableAt: input.availableAt, claimAvailableAt: input.availableAt },$unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1 },};+ const shieldResult = await Delivery().updateOne(+ { _id: input.id, ...shieldCapabilityFence(input), attempts: input.attempt },+ {+ $inc: update.$inc,+ $set: {+ status: 'leased',+ availableAt: input.availableAt,+ capabilityStatus: 'pending',+ claimAvailableAt: input.availableAt,+ },+ $unset: {+ leaseBy: 1,+ leaseUntil: 1,+ claimToken: 1,+ capabilityLeaseBy: 1,+ capabilityLeaseUntil: 1,+ capabilityClaimToken: 1,+ },+ },+ );+ if (shieldResult.modifiedCount === 1) {+ return true;+ }const capabilityResult = await Delivery().updateOne(- { ...capabilityFence(input), attempts: input.attempt },+ { _id: input.id, ...legacyCapabilityFence(input), attempts: input.attempt },{...update,$set: { ...update.$set, status: 'capability_pending' },@@ -1455,7 +1787,7 @@ export function createAgentTriggerDeliveryMethods(return true;}const result = await Delivery().updateOne(- { ...fence(input), attempts: input.attempt },+ { _id: input.id, ...ordinaryFence(input), attempts: input.attempt },{ ...update, $set: { ...update.$set, status: 'pending' } },);return result.modifiedCount === 1;@@ -1495,6 +1827,11 @@ export function createAgentTriggerDeliveryMethods(leaseBy: 1,leaseUntil: 1,claimToken: 1,+ capabilityStatus: 1,+ claimAvailableAt: 1,+ capabilityLeaseBy: 1,+ capabilityLeaseUntil: 1,+ capabilityClaimToken: 1,lastError: 1,...(awaitsTerminalHandling && { expiresAt: 1 }),},@@ -2400,14 +2737,20 @@ export function createAgentTriggerDeliveryMethods(]),Delivery().countDocuments({ actorReceipt: { $exists: true }, expiresAt: { $lte: now } }),Delivery().countDocuments({- status: { $in: ['pending', 'capability_pending'] },+ $or: [+ { status: { $in: ['pending', 'capability_pending'] } },+ { status: 'leased', capabilityStatus: 'pending' },+ ],attempts: { $gt: 0 },'envelope.target.bindingId': { $exists: true },'handling.status': 'started',actorReceipt: { $exists: false },}),Delivery().countDocuments({- status: { $in: ['dead', 'capability_dead'] },+ $or: [+ { status: { $in: ['dead', 'capability_dead'] } },+ { status: 'leased', capabilityStatus: 'dead' },+ ],'envelope.target.bindingId': { $exists: true },'handling.status': 'started',actorReceipt: { $exists: false },@@ -2435,7 +2778,12 @@ export function createAgentTriggerDeliveryMethods(): Promise<boolean> {const error = normalizeFailure(input.error);const retryUpdate = (status: 'pending' | 'capability_pending') => ({- $set: { status, availableAt: input.availableAt, lastError: error },+ $set: {+ status,+ availableAt: input.availableAt,+ claimAvailableAt: input.availableAt,+ lastError: error,+ },$unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1, settledAt: 1, expiresAt: 1 },$push: {history: {@@ -2452,14 +2800,43 @@ export function createAgentTriggerDeliveryMethods(},},});+ const shieldResult = await Delivery().updateOne(+ { _id: input.id, ...shieldCapabilityFence(input) },+ {+ $set: {+ status: 'leased',+ availableAt: input.availableAt,+ capabilityStatus: 'pending',+ claimAvailableAt: input.availableAt,+ lastError: error,+ },+ $unset: {+ leaseBy: 1,+ leaseUntil: 1,+ claimToken: 1,+ capabilityLeaseBy: 1,+ capabilityLeaseUntil: 1,+ capabilityClaimToken: 1,+ settledAt: 1,+ expiresAt: 1,+ },+ $push: retryUpdate('pending').$push,+ },+ );+ if (shieldResult.modifiedCount === 1) {+ return true;+ }const capabilityResult = await Delivery().updateOne(- capabilityFence(input),+ { _id: input.id, ...legacyCapabilityFence(input) },retryUpdate('capability_pending'),);if (capabilityResult.modifiedCount === 1) {return true;}- const result = await Delivery().updateOne(fence(input), retryUpdate('pending'));+ const result = await Delivery().updateOne(+ { _id: input.id, ...ordinaryFence(input) },+ retryUpdate('pending'),+ );return result.modifiedCount === 1;}@@ -2489,15 +2866,44 @@ export function createAgentTriggerDeliveryMethods(},},});+ const shieldDead = await Delivery().updateOne(+ { _id: input.id, ...shieldCapabilityFence(input) },+ {+ $set: {+ // The pre-capability runtime already treats `capability_dead` as a+ // terminal, requeueable state, so it cannot block a lane successor.+ status: 'capability_dead',+ availableAt: LEGACY_CAPABILITY_SHIELD_AT,+ capabilityStatus: 'dead',+ settledAt: input.settledAt,+ lastError: error,+ },+ $unset: {+ leaseBy: 1,+ leaseUntil: 1,+ claimToken: 1,+ capabilityLeaseBy: 1,+ capabilityLeaseUntil: 1,+ capabilityClaimToken: 1,+ expiresAt: 1,+ },+ $push: deadUpdate('dead').$push,+ },+ );+ if (shieldDead.modifiedCount === 1) {+ return true;+ }const capabilityDead = await Delivery().updateOne(- capabilityFence(input),+ { _id: input.id, ...legacyCapabilityFence(input) },deadUpdate('capability_dead'),);if (capabilityDead.modifiedCount === 1) {return true;}const dead = await Delivery()- .findOneAndUpdate(fence(input), deadUpdate('dead'), { new: true })+ .findOneAndUpdate({ _id: input.id, ...ordinaryFence(input) }, deadUpdate('dead'), {+ new: true,+ }).select('_id orderingKey batchMemberIds batchMembersSettledAt requeueCount').lean<Pick<@@ -2598,7 +3004,13 @@ export function createAgentTriggerDeliveryMethods(}const boundedLimit = Math.min(limit, MAX_DEAD_LETTER_LIMIT);const deliveries = await Delivery()- .find({ status: { $in: ['dead', 'capability_dead'] }, batchRootId: { $exists: false } })+ .find({+ $or: [+ { status: { $in: ['dead', 'capability_dead'] } },+ { status: { $ne: 'succeeded' }, capabilityStatus: 'dead' },+ ],+ batchRootId: { $exists: false },+ }).sort({ updatedAt: -1, _id: -1 }).limit(boundedLimit).lean<IAgentTriggerDelivery[]>();@@ -2612,7 +3024,14 @@ export function createAgentTriggerDeliveryMethods(const candidate = await Delivery().findOne({_id: id,- status: { $in: ['dead', 'capability_dead'] },+ $or: [+ { status: { $in: ['dead', 'capability_dead'] } },+ {+ status: { $in: ['pending', 'leased'] },+ requiredWorkerCapability: { $exists: true },+ capabilityStatus: 'dead',+ },+ ],batchRootId: { $exists: false },actorReceipt: { $exists: false },actorActionAdmittedAt: { $exists: false },@@ -2623,6 +3042,11 @@ export function createAgentTriggerDeliveryMethods(return null;}const previousRequeueCount = candidate.requeueCount ?? 0;+ const shieldedCapability = candidate.capabilityStatus === 'dead';+ const stagedStatus: IAgentTriggerDelivery['status'] =+ candidate.requiredWorkerCapability == null || shieldedCapability+ ? 'staging'+ : 'capability_staging';/** Claim the root before touching members. A terminal receipt and requeue* now serialize on this CAS; staging recovery finishes member preparation* if the process exits before publication. */@@ -2631,6 +3055,7 @@ export function createAgentTriggerDeliveryMethods({_id: candidate._id,status: candidate.status,+ ...(shieldedCapability && { capabilityStatus: 'dead' }),batchRootId: { $exists: false },actorReceipt: { $exists: false },actorActionAdmittedAt: { $exists: false },@@ -2639,16 +3064,24 @@ export function createAgentTriggerDeliveryMethods(},{$set: {- status: candidate.requiredWorkerCapability == null ? 'staging' : 'capability_staging',+ status: stagedStatus,laneSequence: 0,attempts: 0,availableAt,+ claimAvailableAt: availableAt,+ ...(shieldedCapability && {+ capabilityStatus: 'publishing',+ availableAt: LEGACY_CAPABILITY_SHIELD_AT,+ }),stagingRecoveryAt: new Date(),},$unset: {leaseBy: 1,leaseUntil: 1,claimToken: 1,+ capabilityLeaseBy: 1,+ capabilityLeaseUntil: 1,+ capabilityClaimToken: 1,lastError: 1,result: 1,settledAt: 1,@@ -2683,8 +3116,21 @@ export function createAgentTriggerDeliveryMethods(actorReceipt: { $exists: false },},{- status: { $in: ['leased', 'capability_leased'] },- leaseUntil: { $gt: now },+ $or: [+ {+ status: 'leased',+ requiredWorkerCapability: { $exists: false },+ leaseUntil: { $gt: now },+ },+ {+ status: 'capability_leased',+ leaseUntil: { $gt: now },+ },+ {+ status: 'leased',+ capabilityStatus: 'leased',+ },+ ],},{'actorDetachedAction.status': {diff --git a/packages/data-schemas/src/schema/triggerDelivery.ts b/packages/data-schemas/src/schema/triggerDelivery.tsindex c3dc166c40..52e0894778 100644--- a/packages/data-schemas/src/schema/triggerDelivery.ts+++ b/packages/data-schemas/src/schema/triggerDelivery.ts@@ -140,6 +140,11 @@ const triggerDeliverySchema: Schema<IAgentTriggerDeliveryDocument> = new Schema(default: 'pending',},requiredWorkerCapability: { type: String, maxlength: 128 },+ capabilityStatus: { type: String, enum: ['publishing', 'pending', 'leased', 'dead'] },+ claimAvailableAt: { type: Date },+ capabilityLeaseBy: { type: String },+ capabilityLeaseUntil: { type: Date },+ capabilityClaimToken: { type: String },attempts: { type: Number, required: true, default: 0, min: 0 },availableAt: { type: Date, required: true },envelopeBytes: { type: Number, min: 0 },@@ -185,6 +190,19 @@ const triggerDeliverySchema: Schema<IAgentTriggerDeliveryDocument> = new Schema(triggerDeliverySchema.index({ deliveryKey: 1 }, { unique: true });triggerDeliverySchema.index({ status: 1, availableAt: 1, createdAt: 1 });triggerDeliverySchema.index({ status: 1, leaseUntil: 1, createdAt: 1 });+triggerDeliverySchema.index({ status: 1, claimAvailableAt: 1, createdAt: 1 });+triggerDeliverySchema.index({+ requiredWorkerCapability: 1,+ capabilityStatus: 1,+ claimAvailableAt: 1,+ createdAt: 1,+});+triggerDeliverySchema.index({+ requiredWorkerCapability: 1,+ capabilityStatus: 1,+ capabilityLeaseUntil: 1,+ createdAt: 1,+});triggerDeliverySchema.index({ orderingKey: 1, status: 1, laneSequence: 1 });triggerDeliverySchema.index({orderingKey: 1,diff --git a/packages/data-schemas/src/types/triggerDelivery.ts b/packages/data-schemas/src/types/triggerDelivery.tsindex 671e6c63c7..e6cb71cd0f 100644--- a/packages/data-schemas/src/types/triggerDelivery.ts+++ b/packages/data-schemas/src/types/triggerDelivery.ts@@ -95,6 +95,14 @@ export interface IAgentTriggerDelivery {status: AgentTriggerDeliveryStatus;/** Keeps a delivery invisible to pre-capability workers during rolling deploys. */requiredWorkerCapability?: string;+ /** Private lifecycle for capability-owned work. The outer delivery status+ * remains a legacy-known, nonclaimable compatibility shield. */+ capabilityStatus?: 'publishing' | 'pending' | 'leased' | 'dead';+ /** Canonical claim ordering timestamp. Old rows omit it and sort first. */+ claimAvailableAt?: Date;+ capabilityLeaseBy?: string;+ capabilityLeaseUntil?: Date;+ capabilityClaimToken?: string;attempts: number;availableAt: Date;envelopeBytes?: number;