Enhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active
TypeScript aianthropicartifactsawsazurechatgptchatgpt-cloneclaudeclonedeepseekgeminigooglegpt-5librechatmcpo1openairesponses-apivisionwebui
🧾 feat: Track Authoritative Agent Event Outcomes (#15213)
Danny Avila290b8664d918
diff --git a/CONTEXT.md b/CONTEXT.mdindex 49ebc71484..d2d1ae0d58 100644--- a/CONTEXT.md+++ b/CONTEXT.md@@ -7,4 +7,6 @@- **Live subagent task owner**: the one API process holding a detached child execution, its abort controller, and its bounded control queue. Redis may route trusted poll/control envelopes to that owner, but it does not migrate or persist the executor; Mongo persists only the logical child thread and its continuation fence.- **Subagent completion wakeup**: a durable internal `continue` trigger pre-registered before detached child execution so a process crash cannot lose the wakeup. Delivery defers until the child's terminal transcript is persisted, targets the initiating agent and exact parent response branch, carries task metadata rather than child output, waits for the parent generation to settle, and starts the parent turn that collects the result through the existing task store.- **Subagent activity stream**: an observational, task-scoped live projection of bounded child progress for the currently open private panel. It may cross API replicas through Redis, never carries hidden reasoning text, and never controls or settles execution. The durable child thread remains canonical and its existing polling view is the fallback for missed or unavailable live events.+- **Agent event handling outcome**: the durable, generation-fenced result of a previously accepted event delivery. `started` proves generation admission; terminal states distinguish verified tool application, clean completion without action, failure, and cancellation. Transport success remains separate so an accepted event cannot masquerade as completed work.+- **Agent event expected action**: an optional source-declared tool name and bounded argument subset evaluated against host-observed completed run steps. It is evidence policy, not authorization and not a model-authored success claim.- **Theme definition**: a versioned, data-only description of LibreChat semantic colors and shared appearance roles, optionally specialized by light or dark mode. The theme module validates and resolves partial definitions against bundled defaults before adapters apply them. A theme definition does not contain arbitrary CSS, application behavior, or alternate feature layouts.diff --git a/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js b/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.jsindex 02d35d6643..a53f30351e 100644--- a/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js+++ b/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js@@ -3888,11 +3888,20 @@ describe('ResumableAgentController resume metadata', () => {body: {text: 'Continue from event.',messageId: 'user-msg',+ parentMessageId: 'response-previous',clientRequestId: 'req-event',+ agentEventDelivery: {+ deliveryKey: 'req-event',+ expectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ },+ },conversationId: 'child-conversation',endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },},config: {},+ _isAgentTrigger: true,_agentEventBindingParentConversationId: 'parent-conversation',_agentEventBindingParentAgentId: 'parent-agent',_agentEventBindingTenantId: 'tenant-1',@@ -3915,6 +3924,11 @@ describe('ResumableAgentController resume metadata', () => {expect.objectContaining({responseMessageId: 'req-event:assistant',userMessage: expect.objectContaining({ messageId: 'req-event:user' }),+ agentEventDeliveryKey: 'req-event',+ agentEventExpectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ },}),);});diff --git a/api/server/controllers/agents/request.js b/api/server/controllers/agents/request.jsindex 55ac564bb4..67ec98a15e 100644--- a/api/server/controllers/agents/request.js+++ b/api/server/controllers/agents/request.js@@ -1243,6 +1243,17 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit}return recorded;};+ /** The loopback trigger host binds this lifecycle identity to the same+ * idempotency key that owns generation admission. Ignore mismatched or+ * direct-chat metadata rather than letting callers relabel another run. */+ const rawAgentEventDelivery = req.body?.agentEventDelivery;+ const agentEventDelivery =+ isTriggerContinuation &&+ rawAgentEventDelivery != null &&+ typeof rawAgentEventDelivery === 'object' &&+ rawAgentEventDelivery.deliveryKey === clientRequestId+ ? rawAgentEventDelivery+ : undefined;try {logger.debug(`[ResumableAgentController] Creating job`, {@@ -1288,6 +1299,12 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit// Persist temporary-chat state so a HITL resume keeps the resumed response// non-persisted instead of trusting the resume request to re-send the flag.isTemporary: req._agentEventBindingRetention?.isTemporary ?? req.body?.isTemporary,+ ...(agentEventDelivery != null && {+ agentEventDeliveryKey: agentEventDelivery.deliveryKey,+ ...(agentEventDelivery.expectedAction != null && {+ agentEventExpectedAction: agentEventDelivery.expectedAction,+ }),+ }),...(isRegenerate && { isRegenerate: true }),...(scheduleId? {diff --git a/api/server/experimental.js b/api/server/experimental.jsindex 165a17d205..4b961cc5f2 100644--- a/api/server/experimental.js+++ b/api/server/experimental.js@@ -34,6 +34,7 @@ const {configureMessageFilterRegexValidator,configureFileConfigRegexEngine,GenerationJobManager,+ createAgentEventTerminalHandler,waitForKeyvRedisClient,} = require('@librechat/api');const { connectDb, indexSync } = require('~/db');@@ -65,6 +66,7 @@ const staticCache = require('./utils/staticCache');const optionalJwtAuth = require('./middleware/optionalJwtAuth');const noIndex = require('./middleware/noIndex');const routes = require('./routes');+const agentEventMethods = require('~/models');/** Route admin file-config MIME patterns through a linear-time engine (ReDoS-safe) on upload. */configureFileConfigRegexEngine();@@ -290,6 +292,9 @@ if (cluster.isMaster) {// but an already-fired scheduled generation can still reach HITL here. Settle// its durable run when the generic approval runtime expires it.GenerationJobManager.setApprovalExpiredHandler(recordExpiredScheduleApproval);+ GenerationJobManager.setTerminalHostActionHandler(+ createAgentEventTerminalHandler(agentEventMethods),+ );GenerationJobManager.initialize();/*** The master may assign the sweep worker before or after this worker hasdiff --git a/api/server/index.js b/api/server/index.jsindex 056f45922c..656f2a817b 100644--- a/api/server/index.js+++ b/api/server/index.js@@ -49,6 +49,7 @@ const {configureMessageFilterRegexValidator,configureFileConfigRegexEngine,configureAgentEventRuntime,+ createAgentEventTerminalHandler,createScheduleWriteGate,waitForKeyvRedisClient,} = require('@librechat/api');@@ -77,6 +78,7 @@ const { getAppConfig } = require('./services/Config');const staticCache = require('./utils/staticCache');const noIndex = require('./middleware/noIndex');const routes = require('./routes');+const agentEventMethods = require('~/models');/** Route admin file-config MIME patterns through a linear-time engine (ReDoS-safe) on upload. */configureFileConfigRegexEngine();@@ -123,6 +125,9 @@ const configureGenerationStreams = () => {cleanupOnComplete: !isEnabled(process.env.STREAM_KEEP_COMPLETED_JOBS),});GenerationJobManager.setApprovalExpiredHandler(recordExpiredScheduleApproval);+ GenerationJobManager.setTerminalHostActionHandler(+ createAgentEventTerminalHandler(agentEventMethods),+ );GenerationJobManager.initialize();// Stop active generations and close their SSE streams while the HTTP server drains.registerShutdownTask(diff --git a/e2e/specs/mock/deferred-tools-hitl.spec.ts b/e2e/specs/mock/deferred-tools-hitl.spec.tsindex 3b993eebda..e3c81920c0 100644--- a/e2e/specs/mock/deferred-tools-hitl.spec.ts+++ b/e2e/specs/mock/deferred-tools-hitl.spec.ts@@ -127,7 +127,8 @@ test.describe('deferred tools across HITL resume', () => {const response = await sendMessage(page, `${PROMPT_MARKER}${label}`);expect(response.ok()).toBeTruthy();await expect(page).toHaveURL(/\/c\/(?!new)/, { timeout: 15000 });- await expect(page.getByText(question, { exact: true })).toBeVisible({ timeout: 30000 });+ const questionCard = page.getByRole('paragraph').filter({ hasText: question });+ await expect(questionCard).toHaveText(question, { timeout: 30000 });/** Reload the public conversation route while the graph is paused. This* proves the browser reconstructs the real persisted pending action,@@ -135,7 +136,7 @@ test.describe('deferred tools across HITL resume', () => {const conversationPath = new URL(page.url()).pathname;await page.reload({ waitUntil: 'domcontentloaded' });await expect(page).toHaveURL(conversationPath);- await expect(page.getByText(question, { exact: true })).toBeVisible({ timeout: 30000 });+ await expect(questionCard).toHaveText(question, { timeout: 30000 });const option = page.getByRole('button', {name: new RegExp(`${escapeRegExp(optionLabel)}$`),diff --git a/packages/api/src/agents/triggers/README.md b/packages/api/src/agents/triggers/README.mdindex 247713ac19..5502d8af7e 100644--- a/packages/api/src/agents/triggers/README.md+++ b/packages/api/src/agents/triggers/README.md@@ -108,6 +108,14 @@ results include the conversation and generation identity needed for a later `steresponses never expose the stored source payload, ordering key, retry history, or worker identity.Callers must sanitize `event.payload`; credentials and transport secrets must not be persisted.+For a bound `continue`, `succeeded` means generation admission succeeded, not that the requested+work finished. The status response therefore also exposes a durable `handling` lifecycle:+`started`, followed by exactly one of `applied`, `completed_no_action`, `failed`, or `cancelled`.+Action-aware bound-child sources may send an `expectedAction` containing a tool name and optional+argument subset. LibreChat reports `applied` only when the exact generation completes with+host-observed tool evidence matching that contract; model-authored prose is never accepted as+proof. Fire, steer, and unbound continue deliveries reject this contract.+### Event-driven child actorsRegister a direct child agent once under the same Remote Agents API key that will deliver events.@@ -201,4 +209,6 @@ the batch or create another branch.Coalescing is intentionally accepted only for authenticated bound-child `continue` deliveries.The source must not set `coalesce` for a player turn, command, fence, approval, HITL request, or anyevent whose individual timing or acknowledgment is actionable. `fire`, `steer`, and unbound-`continue` deliveries reject the option instead of silently weakening their semantics.+`continue` deliveries reject the option instead of silently weakening their semantics. Deliveries+with `expectedAction` also reject coalescing because one generation cannot prove several distinct+action fences.diff --git a/packages/api/src/agents/triggers/delivery.spec.ts b/packages/api/src/agents/triggers/delivery.spec.tsindex 0051513f18..ed27115399 100644--- a/packages/api/src/agents/triggers/delivery.spec.ts+++ b/packages/api/src/agents/triggers/delivery.spec.ts@@ -144,6 +144,33 @@ describe('prepareAgentTriggerDelivery', () => {);});+ it('rejects action evidence contracts on observational batches', () => {+ const bound = envelope({+ mode: 'continue',+ target: {+ agentId: 'agent-1',+ conversationId: 'child-thread',+ parentMessageId: 'placeholder',+ bindingId: `evtbind_${'a'.repeat(48)}`,+ sourceKeyId: 'source-key',+ },+ expectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ },+ });++ expect(() => prepareAgentTriggerDelivery(bound, { coalesce: { key: 'commentary' } })).toThrow(+ 'Expected actions cannot be coalesced',+ );+ });++ it('accepts action evidence contracts only for bound child continuations', () => {+ expect(() =>+ prepareAgentTriggerDelivery(envelope({ expectedAction: { toolName: 'submit_move' } })),+ ).toThrow('Expected actions require an authenticated bound-child continue event');+ });+it('rejects invalid scheduling metadata and oversized envelopes', () => {expect(() =>prepareAgentTriggerDelivery(envelope(), { availableAt: new Date(Number.NaN) }),diff --git a/packages/api/src/agents/triggers/delivery.ts b/packages/api/src/agents/triggers/delivery.tsindex e3824af1c9..9e77e912a9 100644--- a/packages/api/src/agents/triggers/delivery.ts+++ b/packages/api/src/agents/triggers/delivery.ts@@ -153,7 +153,20 @@ export function prepareAgentTriggerDelivery(// a fresh request trace without weakening content-conflict detection.const { requestId: _requestId, receivedAt: _receivedAt, ...durableIdentity } = envelope;const requestedAvailableAt = requireAvailableAt(options.availableAt);+ if (+ envelope.expectedAction != null &&+ (envelope.mode !== 'continue' ||+ envelope.target.bindingId == null ||+ envelope.target.sourceKeyId == null)+ ) {+ throw new AgentTriggerDeliveryError(+ 'Expected actions require an authenticated bound-child continue event',+ );+ }const coalesceKey = coalescingIdentity(envelope, options.coalesce);+ if (coalesceKey != null && envelope.expectedAction != null) {+ throw new AgentTriggerDeliveryError('Expected actions cannot be coalesced');+ }if (coalesceKey != null && bytes > MAX_AGENT_TRIGGER_BATCH_BYTES) {throw new AgentTriggerDeliveryError(`Coalesced agent trigger envelope exceeds ${MAX_AGENT_TRIGGER_BATCH_BYTES} bytes`,diff --git a/packages/api/src/agents/triggers/engine.spec.ts b/packages/api/src/agents/triggers/engine.spec.tsindex bc846984eb..f30e89f9bf 100644--- a/packages/api/src/agents/triggers/engine.spec.ts+++ b/packages/api/src/agents/triggers/engine.spec.ts@@ -84,6 +84,56 @@ describe('createAgentTriggerDeliveryEngine', () => {});});+ it('persists generation identity when a bound continuation starts', async () => {+ const envelope = createAgentTriggerEnvelope({+ mode: 'continue',+ requestId: 'request-1',+ deliveryId: 'delivery-1',+ receivedAt: 10,+ principal: { id: 'user-1' },+ target: {+ agentId: 'agent-1',+ conversationId: 'conversation-1',+ parentMessageId: 'response-1',+ bindingId: `evtbind_${'a'.repeat(48)}`,+ sourceKeyId: 'source-key',+ },+ event: {+ id: 'event-1',+ type: 'turn.ready',+ occurredAt: 9,+ source: { id: 'source-key', type: 'remote_api_key' },+ },+ input: 'Take the turn.',+ });+ const store = storeWith({ claimNext: jest.fn(async () => delivery({ envelope })) });+ const result: AgentTriggerExecutionResult = {+ mode: 'continue',+ status: 'started',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ };+ const engine = createAgentTriggerDeliveryEngine(+ { store, dispatch: jest.fn(async () => result), now: () => START, workerId: 'worker-1' },+ { concurrency: 1 },+ );++ await engine.runTick();++ expect(store.complete).toHaveBeenCalledWith(+ expect.objectContaining({+ handling: {+ status: 'started',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ startedAt: START,+ },+ }),+ );+ });+it('dispatches one structured invocation for every member of a claimed batch', async () => {const envelope = createAgentTriggerEnvelope({mode: 'continue',diff --git a/packages/api/src/agents/triggers/engine.ts b/packages/api/src/agents/triggers/engine.tsindex 523db9e6c4..77e46c515a 100644--- a/packages/api/src/agents/triggers/engine.ts+++ b/packages/api/src/agents/triggers/engine.ts@@ -16,6 +16,37 @@ const ORDERING_RECHECK_MS = 250;const DEFAULT_DEFER_MS = 5_000;const MAX_RETRY_AFTER_MS = 24 * 60 * 60_000;+function startedHandling(+ delivery: Pick<AgentTriggerDeliveryRecord, 'envelope'>,+ result: AgentTriggerExecutionResult,+ startedAt: Date,+): AgentTriggerDeliveryRecord['handling'] | undefined {+ const envelope = delivery.envelope;+ if (+ envelope == null ||+ typeof envelope !== 'object' ||+ !('mode' in envelope) ||+ envelope.mode !== 'continue' ||+ !('target' in envelope) ||+ envelope.target == null ||+ typeof envelope.target !== 'object' ||+ !('bindingId' in envelope.target) ||+ result.mode !== 'continue' ||+ result.status === 'settled' ||+ result.streamId == null ||+ result.generationCreatedAt == null+ ) {+ return undefined;+ }+ return {+ status: 'started',+ conversationId: result.conversationId,+ streamId: result.streamId,+ generationCreatedAt: result.generationCreatedAt,+ startedAt,+ };+}+/** A pre-dispatch condition that must not consume the delivery's retry budget. */export class AgentTriggerDeliveryDeferredError extends Error {readonly delayMs: number;@@ -69,6 +100,16 @@ export interface AgentTriggerDeliveryRecord {leaseBy?: string;leaseUntil?: Date;lastError?: AgentTriggerDeliveryFailure;+ handling?: {+ status: 'started' | 'applied' | 'completed_no_action' | 'failed' | 'cancelled';+ conversationId: string;+ streamId: string;+ generationCreatedAt: number;+ startedAt: Date;+ settledAt?: Date;+ error?: string;+ action?: { toolName: string; toolCallId?: string };+ };}export interface AgentTriggerOrderingBlock {@@ -115,6 +156,7 @@ export interface AgentTriggerDeliveryStore {attempt: number;result: AgentTriggerExecutionResult;settledAt: Date;+ handling?: AgentTriggerDeliveryRecord['handling'];}) => Promise<boolean>;retry: (input: {id: string;@@ -456,6 +498,7 @@ export function createAgentTriggerDeliveryEngine(const settledAt = now();try {+ const handling = startedHandling(delivery, result, settledAt);await deps.store.complete({id: delivery.id,workerId,@@ -463,6 +506,7 @@ export function createAgentTriggerDeliveryEngine(attempt,result,settledAt,+ ...(handling != null && { handling }),});} catch (error) {const recorded: AgentTriggerDeliveryFailure = {diff --git a/packages/api/src/agents/triggers/envelope.spec.ts b/packages/api/src/agents/triggers/envelope.spec.tsindex 69afb73154..e23ec152fa 100644--- a/packages/api/src/agents/triggers/envelope.spec.ts+++ b/packages/api/src/agents/triggers/envelope.spec.ts@@ -153,6 +153,35 @@ describe('createAgentTriggerEnvelope', () => {).toThrow('target.bindingId and target.sourceKeyId must be provided together');});+ it('validates and detaches an expected tool-action fence', () => {+ const expectedAction = {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ };+ const envelope = createAgentTriggerEnvelope({+ ...createFireInput(),+ expectedAction,+ });+ expectedAction.argumentSubset = { gameId: 'mutated', expectedPly: 8 };++ expect(envelope.expectedAction).toEqual({+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ });+ expect(() =>+ createAgentTriggerEnvelope({+ ...createFireInput(),+ expectedAction: { toolName: 'submit_move', argumentSubset: [] as never },+ }),+ ).toThrow('expectedAction.argumentSubset must be an object');+ expect(() =>+ createAgentTriggerEnvelope({+ ...createFireInput(),+ expectedAction: { toolName: 'x'.repeat(257) },+ }),+ ).toThrow('expectedAction.toolName must not exceed 256 characters');+ });+it('builds a stable generation-compatible idempotency key per delivery target', () => {const first = createAgentTriggerEnvelope(createFireInput());const retry = createAgentTriggerEnvelope({diff --git a/packages/api/src/agents/triggers/envelope.ts b/packages/api/src/agents/triggers/envelope.tsindex ee3607b497..dfd8a329d2 100644--- a/packages/api/src/agents/triggers/envelope.ts+++ b/packages/api/src/agents/triggers/envelope.ts@@ -1,10 +1,14 @@import { createHash } from 'node:crypto';import type { AgentRunPrincipal, AgentRunPrincipalInput } from '../envelope';+import type { AgentTriggerExpectedAction } from './types';import type { JsonValue } from '../json';import { cloneJsonValue } from '../json';+export type { AgentTriggerExpectedAction } from './types';+export const AGENT_TRIGGER_ENVELOPE_VERSION = 1 as const;export const AGENT_TRIGGER_IDEMPOTENCY_PREFIX = 'trigger_';+const MAX_EXPECTED_ACTION_TOOL_NAME_LENGTH = 256;export type AgentTriggerMode = 'continue' | 'fire' | 'steer';@@ -84,6 +88,7 @@ interface AgentTriggerEnvelopeBase {event: AgentTriggerEvent;/** Host-rendered agent input; source payload remains available in `event.payload`. */input: string;+ expectedAction?: AgentTriggerExpectedAction;}export interface AgentFireTriggerEnvelope extends AgentTriggerEnvelopeBase {@@ -114,6 +119,7 @@ interface CreateAgentTriggerEnvelopeBase {principal: AgentRunPrincipalInput | null | undefined;event: AgentTriggerEvent;input: string;+ expectedAction?: AgentTriggerExpectedAction;}export type CreateAgentTriggerEnvelopeInput =@@ -191,6 +197,33 @@ function createEvent(input: AgentTriggerEvent | null | undefined): AgentTriggerE};}+function createExpectedAction(+ input: AgentTriggerExpectedAction | null | undefined,+): AgentTriggerExpectedAction | undefined {+ if (input == null) {+ return undefined;+ }+ const action = requireRecord(input, 'expectedAction');+ const argumentSubset =+ action.argumentSubset == null+ ? undefined+ : (cloneJsonValue(+ requireRecord(action.argumentSubset, 'expectedAction.argumentSubset'),+ 'expectedAction.argumentSubset',+ error,+ ) as Record<string, JsonValue>);+ const toolName = requireString(action.toolName, 'expectedAction.toolName');+ if (toolName.length > MAX_EXPECTED_ACTION_TOOL_NAME_LENGTH) {+ throw error(+ `expectedAction.toolName must not exceed ${MAX_EXPECTED_ACTION_TOOL_NAME_LENGTH} characters`,+ );+ }+ return {+ toolName,+ ...(argumentSubset != null && { argumentSubset }),+ };+}+function createFireRunContext(input: AgentFireRunContext | null | undefined,): AgentFireRunContext | undefined {@@ -224,6 +257,7 @@ export function createAgentTriggerEnvelope(input: CreateAgentTriggerEnvelopeInput,): AgentTriggerEnvelope {const receivedMode: string = input.mode;+ const expectedAction = createExpectedAction(input.expectedAction);const base: AgentTriggerEnvelopeBase = {version: AGENT_TRIGGER_ENVELOPE_VERSION,requestId: requireString(input.requestId, 'requestId'),@@ -232,6 +266,7 @@ export function createAgentTriggerEnvelope(principal: createPrincipal(input.principal),event: createEvent(input.event),input: requireString(input.input, 'input'),+ ...(expectedAction != null && { expectedAction }),};if (input.mode === 'fire') {@@ -329,6 +364,10 @@ export function parseAgentTriggerEnvelope(input: unknown): AgentTriggerEnvelope...(eventInput.payload !== undefined && { payload: eventInput.payload as JsonValue }),};+ const expectedAction = createExpectedAction(+ envelope.expectedAction as AgentTriggerExpectedAction | null | undefined,+ );+const base: AgentTriggerEnvelopeBase = {version: AGENT_TRIGGER_ENVELOPE_VERSION,requestId: requireString(envelope.requestId, 'requestId'),@@ -337,6 +376,7 @@ export function parseAgentTriggerEnvelope(input: unknown): AgentTriggerEnvelopeprincipal,event,input: requireString(envelope.input, 'input'),+ ...(expectedAction != null && { expectedAction }),};const target = requireRecord(envelope.target, 'target');diff --git a/packages/api/src/agents/triggers/host.spec.ts b/packages/api/src/agents/triggers/host.spec.tsindex 66a5c500aa..b72586eccc 100644--- a/packages/api/src/agents/triggers/host.spec.ts+++ b/packages/api/src/agents/triggers/host.spec.ts@@ -577,8 +577,15 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {it('carries server-resolved binding identity only on bound child continuations', async () => {const base = createContinueEnvelope();+ if (base.mode !== 'continue') {+ throw new Error('Expected a continue envelope');+ }const envelope = {...base,+ expectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ },target: {...base.target,bindingId: `evtbind_${'a'.repeat(48)}`,@@ -598,6 +605,12 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {const headers = fetcher.mock.calls[0][1]?.headers as Record<string, string>;expect(headers['x-lc-agent-event-binding']).toBe(`evtbind_${'a'.repeat(48)}`);expect(headers['x-lc-agent-event-source-key']).toBe('source-key');+ expect(JSON.parse(String(fetcher.mock.calls[0][1]?.body))).toMatchObject({+ agentEventDelivery: {+ deliveryKey: getAgentTriggerIdempotencyKey(envelope),+ expectedAction: envelope.expectedAction,+ },+ });});it.each(['PARENT_NOT_READY', 'EVENT_ACTOR_NOT_READY'])(diff --git a/packages/api/src/agents/triggers/host.ts b/packages/api/src/agents/triggers/host.tsindex 342d919738..4bbd176c8c 100644--- a/packages/api/src/agents/triggers/host.ts+++ b/packages/api/src/agents/triggers/host.ts@@ -615,6 +615,15 @@ async function startRun(isRegenerate: false,clientRequestId: context.idempotencyKey,generationProtocolVersion: 2,+ ...(envelope.mode === 'continue' &&+ envelope.target.bindingId != null && {+ agentEventDelivery: {+ deliveryKey: context.idempotencyKey,+ ...(envelope.expectedAction != null && {+ expectedAction: envelope.expectedAction,+ }),+ },+ }),...(envelope.mode === 'fire' && {agentTrigger: {version: envelope.version,diff --git a/packages/api/src/agents/triggers/index.ts b/packages/api/src/agents/triggers/index.tsindex 92ad867e8f..a1dc7db842 100644--- a/packages/api/src/agents/triggers/index.ts+++ b/packages/api/src/agents/triggers/index.ts@@ -6,5 +6,6 @@ export * from './envelope';export * from './host';export * from './ingress';export * from './lease';+export * from './outcome';export * from './service';export * from './engine';diff --git a/packages/api/src/agents/triggers/ingress.spec.ts b/packages/api/src/agents/triggers/ingress.spec.tsindex 3ccfc4e293..fcb26753d4 100644--- a/packages/api/src/agents/triggers/ingress.spec.ts+++ b/packages/api/src/agents/triggers/ingress.spec.ts@@ -317,16 +317,17 @@ describe('agent trigger event ingress', () => {const deps = dependencies({getDeliveryStatus: jest.fn(async () =>delivery({- status: 'dead',+ status: 'succeeded',attempts: 3,settledAt,- lastError: {- code: 'FORBIDDEN',- message: 'Agent access was revoked',- certainty: 'definite',- retryable: false,- attemptedAt,- status: 403,+ handling: {+ status: 'failed',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ startedAt: attemptedAt,+ settledAt,+ error: 'Agent access was revoked',},}),),@@ -337,18 +338,19 @@ describe('agent trigger event ingress', () => {expect(response.status).toBe(200);expect(response.body).toEqual({id: DELIVERY_KEY,- status: 'dead',+ status: 'succeeded',attempts: 3,availableAt: AVAILABLE_AT.toISOString(),createdAt: CREATED_AT.toISOString(),settledAt: settledAt.toISOString(),- error: {- code: 'FORBIDDEN',- message: 'Agent access was revoked',- certainty: 'definite',- retryable: false,- attemptedAt: attemptedAt.toISOString(),- status: 403,+ handling: {+ status: 'failed',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ startedAt: attemptedAt.toISOString(),+ settledAt: settledAt.toISOString(),+ error: 'Agent access was revoked',},});expect(response.body).not.toHaveProperty('envelope');diff --git a/packages/api/src/agents/triggers/ingress.ts b/packages/api/src/agents/triggers/ingress.tsindex 33544fee30..e09a02217d 100644--- a/packages/api/src/agents/triggers/ingress.ts+++ b/packages/api/src/agents/triggers/ingress.ts@@ -6,6 +6,7 @@ import type {AgentContinueTarget,AgentSteerTarget,AgentTriggerEvent,+ AgentTriggerExpectedAction,AgentTriggerMode,} from './envelope';import type { AgentTriggerEnqueueOptions } from './delivery';@@ -40,6 +41,7 @@ interface AgentTriggerIngressBody {input?: string;orderingKey?: string;coalesce?: { key?: string };+ expectedAction?: AgentTriggerExpectedAction;}export interface AgentTriggerIngressDependencies {@@ -180,6 +182,15 @@ function toPublicDelivery(delivery: Awaited<ReturnType<AgentTriggerService['getD...(delivery.lastError.status != null && { status: delivery.lastError.status }),},}),+ ...(delivery.handling != null && {+ handling: {+ ...delivery.handling,+ startedAt: delivery.handling.startedAt.toISOString(),+ ...(delivery.handling.settledAt != null && {+ settledAt: delivery.handling.settledAt.toISOString(),+ }),+ },+ }),};}@@ -241,6 +252,7 @@ export function createAgentTriggerIngressHandlers(deps: AgentTriggerIngressDepensource: { id: sourceKeyId, type: 'remote_api_key' },},input: body.input as string,+ ...(body.expectedAction != null && { expectedAction: body.expectedAction }),};if (body.mode === 'continue' && req._agentEventBindingResolved !== true) {throw new AgentTriggerIngressError(diff --git a/packages/api/src/agents/triggers/outcome.spec.ts b/packages/api/src/agents/triggers/outcome.spec.tsnew file mode 100644index 0000000000..52cce32e2c--- /dev/null+++ b/packages/api/src/agents/triggers/outcome.spec.ts@@ -0,0 +1,405 @@+import { StepTypes } from 'librechat-data-provider';+import type { Agents } from 'librechat-data-provider';+import type { SerializableJobData } from '~/stream';+import { createAgentEventTerminalHandler } from './outcome';++function job(overrides: Partial<SerializableJobData> = {}): SerializableJobData {+ return {+ streamId: 'conversation-1',+ conversationId: 'conversation-1',+ userId: 'user-1',+ status: 'complete',+ createdAt: 1_787_000_000_000,+ completedAt: 1_787_000_001_000,+ syncSent: false,+ agentEventDeliveryKey: 'trigger_1',+ ...overrides,+ };+}++function completedToolStep(): Agents.RunStep {+ return {+ id: 'step-1',+ index: 0,+ type: StepTypes.TOOL_CALLS,+ status: 'completed',+ stepDetails: {+ type: StepTypes.TOOL_CALLS,+ tool_calls: [+ {+ id: 'call-1',+ name: 'submit_move_mcp_speed-chess',+ args: { gameId: 'game-1', expectedPly: 7 },+ output: '{"accepted":true}',+ },+ ],+ },+ };+}++describe('agent event terminal outcomes', () => {+ it('records applied only from completed tool evidence matching the expected fence', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });++ await handler(+ 'conversation-1',+ job({+ agentEventExpectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ },+ }),+ [completedToolStep()],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({+ deliveryKey: 'trigger_1',+ generationCreatedAt: 1_787_000_000_000,+ status: 'applied',+ action: { toolName: 'submit_move_mcp_speed-chess', toolCallId: 'call-1' },+ }),+ );+ });++ it('reports a clean terminal generation without matching evidence as completed_no_action', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });++ await handler(+ 'conversation-1',+ job({+ agentEventExpectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'another-game' },+ },+ }),+ [completedToolStep()],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'completed_no_action' }),+ );+ });++ it('matches nested action arrays structurally', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });+ const step = completedToolStep();+ if (step.stepDetails.type !== 'tool_calls' || !step.stepDetails.tool_calls?.[0]) {+ throw new Error('Expected tool evidence');+ }+ const call = step.stepDetails.tool_calls[0];+ if ('function' in call) {+ throw new Error('Expected legacy tool evidence');+ }+ call.args = {+ gameId: 'game-1',+ moves: ['e4', { replies: ['c5', 'Nf3'] }],+ };++ await handler(+ 'conversation-1',+ job({+ agentEventExpectedAction: {+ toolName: 'submit_move',+ argumentSubset: { moves: ['e4', { replies: ['c5', 'Nf3'] }] },+ },+ }),+ [step],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'applied' }),+ );+ });++ it('does not treat a background launch handle as applied work', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });+ const step = completedToolStep();+ if (step.stepDetails.type !== 'tool_calls' || !step.stepDetails.tool_calls?.[0]) {+ throw new Error('Expected tool evidence');+ }+ const call = step.stepDetails.tool_calls[0];+ if ('function' in call) {+ throw new Error('Expected legacy tool evidence');+ }+ call.args = { gameId: 'game-1', expectedPly: 7, run_in_background: true };+ call.output = JSON.stringify({+ status: 'running',+ background_task_id: 'task-1',+ });++ await handler(+ 'conversation-1',+ job({ agentEventExpectedAction: { toolName: 'submit_move' } }),+ [step],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'completed_no_action' }),+ );+ });++ it('does not treat a rejected background dispatch as applied work', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });+ const step = completedToolStep();+ if (step.stepDetails.type !== 'tool_calls' || !step.stepDetails.tool_calls?.[0]) {+ throw new Error('Expected tool evidence');+ }+ const call = step.stepDetails.tool_calls[0];+ if ('function' in call) {+ throw new Error('Expected legacy tool evidence');+ }+ call.args = { gameId: 'game-1', expectedPly: 7, run_in_background: true };+ call.output = JSON.stringify({ status: 'rejected', tool: 'submit_move' });++ await handler(+ 'conversation-1',+ job({ agentEventExpectedAction: { toolName: 'submit_move' } }),+ [step],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'completed_no_action' }),+ );+ });++ it('retains a foreground tool-authored rejected status as execution evidence', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });+ const step = completedToolStep();+ if (step.stepDetails.type !== 'tool_calls' || !step.stepDetails.tool_calls?.[0]) {+ throw new Error('Expected tool evidence');+ }+ const call = step.stepDetails.tool_calls[0];+ if ('function' in call) {+ throw new Error('Expected legacy tool evidence');+ }+ call.output = JSON.stringify({ status: 'rejected' });++ await handler(+ 'conversation-1',+ job({ agentEventExpectedAction: { toolName: 'submit_move' } }),+ [step],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'applied' }),+ );+ });++ it.each(['decision_response', 'decision_reason'] as const)(+ 'does not treat a human-authored %s output as executed tool evidence',+ async (field) => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });+ const step = completedToolStep();+ await handler(+ 'conversation-1',+ job({+ agentEventExpectedAction: { toolName: 'submit_move' },+ userSubmittedMessageFieldPaths: [{ path: '/content/2/tool_call/output', field }],+ }),+ [step],+ [+ { type: 'text', text: 'before' },+ { type: 'text', text: 'approval' },+ {+ type: 'tool_call',+ tool_call: {+ id: 'call-1',+ name: 'submit_move_mcp_speed-chess',+ output: 'human supplied output',+ },+ },+ ],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'completed_no_action' }),+ );+ },+ );++ it('does not treat a function-shaped call rejected by input validation as applied', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });+ const step = completedToolStep();+ if (step.stepDetails.type !== 'tool_calls') {+ throw new Error('Expected tool evidence');+ }+ step.stepDetails.tool_calls = [+ {+ id: 'call-invalid',+ type: 'function',+ function: {+ name: 'submit_move_mcp_speed-chess',+ arguments: { gameId: 'game-1', expectedPly: 7 },+ output: '{"rejected":true}',+ },+ inputValidationError: true,+ } as Agents.AgentToolCall,+ ];++ await handler(+ 'conversation-1',+ job({+ agentEventExpectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ },+ }),+ [step],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'completed_no_action' }),+ );+ });++ it.each(['legacy', 'function'] as const)(+ 'does not treat a failed %s foreground call as applied',+ async (shape) => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });+ const step = completedToolStep();+ step.status = 'failed';+ if (shape === 'function') {+ if (step.stepDetails.type !== 'tool_calls') {+ throw new Error('Expected tool evidence');+ }+ step.stepDetails.tool_calls = [+ {+ id: 'call-failed',+ type: 'function',+ function: {+ name: 'submit_move_mcp_speed-chess',+ arguments: { gameId: 'game-1' },+ output: 'Error: [submit_move] tool call failed: unavailable',+ },+ } as Agents.AgentToolCall,+ ];+ } else if (step.stepDetails.type === 'tool_calls' && step.stepDetails.tool_calls?.[0]) {+ const call = step.stepDetails.tool_calls[0];+ if ('function' in call) {+ throw new Error('Expected legacy tool evidence');+ }+ call.output = 'Error: [submit_move] tool call failed: unavailable';+ }++ await handler(+ 'conversation-1',+ job({ agentEventExpectedAction: { toolName: 'submit_move' } }),+ [step],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'completed_no_action' }),+ );+ },+ );++ it('preserves an applied call when a sibling makes the enclosing step fail', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });+ const step = completedToolStep();+ step.status = 'failed';+ if (step.stepDetails.type !== 'tool_calls') {+ throw new Error('Expected tool evidence');+ }+ step.stepDetails.tool_calls = [+ {+ id: 'call-applied',+ name: 'submit_move_mcp_speed-chess',+ args: { gameId: 'game-1' },+ output: '{"accepted":true}',+ executionStatus: 'success',+ } as Agents.AgentToolCall,+ {+ id: 'call-failed',+ name: 'notify_spectators',+ args: {},+ output: 'Error: [notify_spectators] tool call failed: unavailable',+ executionStatus: 'error',+ } as Agents.AgentToolCall,+ ];++ await handler(+ 'conversation-1',+ job({ agentEventExpectedAction: { toolName: 'submit_move' } }),+ [step],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({+ status: 'applied',+ action: { toolName: 'submit_move_mcp_speed-chess', toolCallId: 'call-applied' },+ }),+ );+ });++ it('does not infer that a source action was applied without an explicit evidence contract', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });++ await handler('conversation-1', job(), [completedToolStep()]);++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'completed_no_action' }),+ );+ expect(settleAgentTriggerHandlingOutcome.mock.calls[0][0]).not.toHaveProperty('action');+ });++ it('preserves terminal generation failures without relying on tool evidence', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });++ await handler('conversation-1', job({ status: 'error', error: 'provider unavailable' }), []);++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'failed', error: 'provider unavailable' }),+ );+ });++ it('keeps a verified side effect authoritative when later generation work fails', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });++ await handler(+ 'conversation-1',+ job({+ status: 'error',+ error: 'follow-up model call failed',+ agentEventExpectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ },+ }),+ [completedToolStep()],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'applied' }),+ );+ expect(settleAgentTriggerHandlingOutcome.mock.calls[0][0]).not.toHaveProperty('error');+ });++ it('records cancellation when the generation stops before applying the expected action', async () => {+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);+ const handler = createAgentEventTerminalHandler({ settleAgentTriggerHandlingOutcome });++ await handler(+ 'conversation-1',+ job({ status: 'aborted', agentEventExpectedAction: { toolName: 'submit_move' } }),+ [],+ );++ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith(+ expect.objectContaining({ status: 'cancelled' }),+ );+ });+});diff --git a/packages/api/src/agents/triggers/outcome.ts b/packages/api/src/agents/triggers/outcome.tsnew file mode 100644index 0000000000..5c61e76832--- /dev/null+++ b/packages/api/src/agents/triggers/outcome.ts@@ -0,0 +1,219 @@+import type { Agents } from 'librechat-data-provider';+import type { AgentTriggerExpectedAction } from './envelope';+import type { SerializableJobData } from '~/stream';++interface SettleAgentTriggerHandlingOutcomeInput {+ deliveryKey: string;+ conversationId: string;+ generationCreatedAt: number;+ status: 'applied' | 'completed_no_action' | 'failed' | 'cancelled';+ settledAt: Date;+ error?: string;+ action?: { toolName: string; toolCallId?: string };+}++interface CompletedToolEvidence {+ toolName: string;+ toolCallId?: string;+ arguments?: unknown;+}++const MAX_RECEIPT_ID_LENGTH = 256;++function isBackgroundNonExecutionReceipt(value: unknown, argumentsValue: unknown): boolean {+ const parsedArguments = parseArguments(argumentsValue);+ if (+ parsedArguments == null ||+ typeof parsedArguments !== 'object' ||+ Array.isArray(parsedArguments) ||+ (parsedArguments as Record<string, unknown>).run_in_background !== true+ ) {+ return false;+ }+ const parsed = parseArguments(value);+ if (parsed == null || typeof parsed !== 'object' || Array.isArray(parsed)) {+ return false;+ }+ const receipt = parsed as Record<string, unknown>;+ return (+ (receipt.status === 'running' && typeof receipt.background_task_id === 'string') ||+ receipt.status === 'rejected'+ );+}++function toolEvidence(+ step: Agents.RunStep,+ nonExecutedToolCallIds: ReadonlySet<string>,+): CompletedToolEvidence[] {+ if (step.status === 'in_progress' || step.stepDetails?.type !== 'tool_calls') {+ return [];+ }+ return (step.stepDetails.tool_calls ?? []).flatMap((call) => {+ const executionStatus = (+ call as Agents.AgentToolCall & {+ executionStatus?: 'success' | 'error' | 'cancelled';+ }+ ).executionStatus;+ if (+ executionStatus === 'error' ||+ executionStatus === 'cancelled' ||+ (step.status !== 'completed' && executionStatus !== 'success')+ ) {+ return [];+ }+ if ('inputValidationError' in call && call.inputValidationError === true) {+ return [];+ }+ if (call.id != null && nonExecutedToolCallIds.has(call.id)) {+ return [];+ }+ if ('function' in call) {+ if (+ call.function.output == null ||+ isBackgroundNonExecutionReceipt(call.function.output, call.function.arguments)+ ) {+ return [];+ }+ return [+ {+ toolName: call.function.name,+ toolCallId: call.id,+ arguments: call.function.arguments,+ },+ ];+ }+ if (call.output == null || isBackgroundNonExecutionReceipt(call.output, call.args)) {+ return [];+ }+ return [+ {+ toolName: call.name,+ ...(call.id != null && { toolCallId: call.id }),+ arguments: call.args,+ },+ ];+ });+}++function nonExecutedHITLToolCallIds(+ job: SerializableJobData,+ content: Agents.MessageContentComplex[],+): Set<string> {+ const ids = new Set<string>();+ for (const provenance of job.userSubmittedMessageFieldPaths ?? []) {+ if (provenance.field !== 'decision_response' && provenance.field !== 'decision_reason') {+ continue;+ }+ const match = /^\/content\/(\d+)\/tool_call\/output$/.exec(provenance.path);+ const index = match == null ? Number.NaN : Number(match[1]);+ const part = Number.isSafeInteger(index) ? content[index] : undefined;+ if (part == null || typeof part !== 'object' || !('tool_call' in part)) {+ continue;+ }+ const toolCall = part.tool_call;+ if (toolCall?.id != null && toolCall.id.length > 0) {+ ids.add(toolCall.id);+ }+ }+ return ids;+}++function parseArguments(value: unknown): unknown {+ if (typeof value !== 'string') {+ return value;+ }+ try {+ return JSON.parse(value) as unknown;+ } catch {+ return value;+ }+}++function containsSubset(value: unknown, subset: unknown): boolean {+ if (Array.isArray(subset)) {+ return (+ Array.isArray(value) &&+ value.length === subset.length &&+ subset.every((expected, index) => containsSubset(value[index], expected))+ );+ }+ if (subset == null || typeof subset !== 'object') {+ return Object.is(value, subset);+ }+ if (value == null || typeof value !== 'object' || Array.isArray(value)) {+ return false;+ }+ return Object.entries(subset).every(([key, expected]) =>+ containsSubset((value as Record<string, unknown>)[key], expected),+ );+}++function matchesExpectedAction(+ evidence: CompletedToolEvidence,+ expected: AgentTriggerExpectedAction,+): boolean {+ const nameMatches =+ evidence.toolName === expected.toolName ||+ evidence.toolName.startsWith(`${expected.toolName}_mcp_`);+ return (+ nameMatches &&+ (expected.argumentSubset == null ||+ containsSubset(parseArguments(evidence.arguments), expected.argumentSubset))+ );+}++export function createAgentEventTerminalHandler(methods: {+ settleAgentTriggerHandlingOutcome: (+ input: SettleAgentTriggerHandlingOutcomeInput,+ ) => Promise<boolean>;+}): (+ streamId: string,+ job: SerializableJobData,+ runSteps: Agents.RunStep[],+ content?: Agents.MessageContentComplex[],+) => Promise<void> {+ return async (+ streamId: string,+ job: SerializableJobData,+ runSteps: Agents.RunStep[],+ content: Agents.MessageContentComplex[] = [],+ ) => {+ if (job.agentEventDeliveryKey == null) {+ return;+ }+ const nonExecutedToolCallIds = nonExecutedHITLToolCallIds(job, content);+ const evidence = runSteps.flatMap((step) => toolEvidence(step, nonExecutedToolCallIds));+ const action =+ job.agentEventExpectedAction == null+ ? undefined+ : evidence.find((item) => matchesExpectedAction(item, job.agentEventExpectedAction!));+ const settledAt = new Date(job.completedAt ?? Date.now());+ let status: SettleAgentTriggerHandlingOutcomeInput['status'] = 'completed_no_action';+ if (action != null) {+ status = 'applied';+ } else if (job.status === 'error') {+ status = 'failed';+ } else if (job.status === 'aborted') {+ status = 'cancelled';+ }+ const settled = await methods.settleAgentTriggerHandlingOutcome({+ deliveryKey: job.agentEventDeliveryKey,+ conversationId: job.conversationId ?? streamId,+ generationCreatedAt: job.createdAt,+ status,+ settledAt,+ ...(status === 'failed' && { error: job.error ?? 'Generation failed' }),+ ...(action != null && {+ action: {+ toolName: action.toolName.slice(0, MAX_RECEIPT_ID_LENGTH),+ ...(action.toolCallId != null && {+ toolCallId: action.toolCallId.slice(0, MAX_RECEIPT_ID_LENGTH),+ }),+ },+ }),+ });+ if (!settled) {+ throw new Error(`Failed to settle agent event delivery ${job.agentEventDeliveryKey}`);+ }+ };+}diff --git a/packages/api/src/agents/triggers/types.ts b/packages/api/src/agents/triggers/types.tsnew file mode 100644index 0000000000..eada583069--- /dev/null+++ b/packages/api/src/agents/triggers/types.ts@@ -0,0 +1,9 @@+import type { JsonValue } from '../json';++/** Optional source-declared proof the generation is expected to produce. The+ * host evaluates this against completed tool evidence; it never trusts a model+ * assertion that work happened. */+export interface AgentTriggerExpectedAction {+ toolName: string;+ argumentSubset?: Record<string, JsonValue>;+}diff --git a/packages/api/src/stream/ApprovalLifecycle.ts b/packages/api/src/stream/ApprovalLifecycle.tsindex fb5e1b03f2..75cbd6dd29 100644--- a/packages/api/src/stream/ApprovalLifecycle.ts+++ b/packages/api/src/stream/ApprovalLifecycle.ts@@ -258,6 +258,7 @@ export class ApprovalLifecycle {patch: {completedAt,error: PAUSE_PERSISTENCE_TIMEOUT_ERROR,+ ...(job.agentEventDeliveryKey != null && { terminalHostActionPending: true }),},clear: ['pendingAction',@@ -394,10 +395,10 @@ export class ApprovalLifecycle {}/**- * Expires the observed approval and returns the winning pre-transition job.- * Callers that need to notify runtime-local subscribers or run host lifecycle- * hooks get both the exact generation identity and its trusted metadata without- * a racy post-transition read (the terminal TTL may already remove the hash).+ * Expires the observed approval and returns the winning terminal snapshot.+ * Callers that run host lifecycle hooks get the trusted pre-CAS metadata plus+ * the exact status/error/completion fields committed by this transition,+ * without a racy post-transition read (the terminal TTL may remove the hash).*/async expireWithIdentity(streamId: string,@@ -413,11 +414,18 @@ export class ApprovalLifecycle {) {return null;}- // Built-in in-memory stores may mutate the returned job object in place during- // transitionStatus. Snapshot it before the CAS so the host hook receives the- // observed generation metadata even when terminal TTL removes or rewrites it.- const expiredJob = { ...job };const createdAt = job.createdAt;+ const completedAt = Date.now();+ const error = 'Approval expired before a decision was made';+ const expiredJob: SerializableJobData = {+ ...job,+ status: 'aborted',+ error,+ completedAt,+ ...(options?.markHostActionPending === true && { terminalHostActionPending: true }),+ };+ delete expiredJob.pendingAction;+ delete expiredJob.pendingActionId;const ok = await this.store.transitionStatus(streamId, {from: 'requires_action',to: 'aborted',@@ -428,8 +436,8 @@ export class ApprovalLifecycle {// enumerable) until the adapter acknowledges — set ATOMICALLY here so a crash right// after this CAS still leaves the durable retry evidence.patch: {- error: 'Approval expired before a decision was made',- completedAt: Date.now(),+ error,+ completedAt,...(options?.markHostActionPending === true && { terminalHostActionPending: true }),},expectActionId: expectedActionId,diff --git a/packages/api/src/stream/GenerationJobManager.ts b/packages/api/src/stream/GenerationJobManager.tsindex 24e9b21006..4c60a8ae27 100644--- a/packages/api/src/stream/GenerationJobManager.ts+++ b/packages/api/src/stream/GenerationJobManager.ts@@ -3,6 +3,7 @@ import { logger, getTenantId, SYSTEM_TENANT_ID } from '@librechat/data-schemas';import {Constants,ContentTypes,+ StepTypes,UsageEvents,ApprovalEvents,SteerEvents,@@ -35,21 +36,22 @@ import type { RecoveredSteerPayload } from './SteerRecovery';import type { SteerContentView } from './SteeringLifecycle';import type { GenerationJobStore } from '~/app/metrics';import type * as t from '~/types';-import {- recordGenerationStreamEarlyBufferOverflow,- recordGenerationStreamResumePendingEvents,- recordGenerationStreamSubscription,- setGenerationJobsInFlight,- recordGenerationJob,-} from '~/app/metrics';import {JobCreationSupersededError,JobPredecessorMismatchError,isPendingActionStale,isPendingActionExpired,PAUSE_PERSISTENCE_TIMEOUT_ERROR,+ PROVIDER_DRAIN_TIMEOUT_MS,STEER_QUEUE_MAX_DEPTH,} from './interfaces/IJobStore';+import {+ recordGenerationStreamEarlyBufferOverflow,+ recordGenerationStreamResumePendingEvents,+ recordGenerationStreamSubscription,+ setGenerationJobsInFlight,+ recordGenerationJob,+} from '~/app/metrics';import { isRecoveredSteerPayload, RecoveredSteerPayloadMismatchError } from './SteerRecovery';import { assertJobStoreV2 } from './jobStoreCapabilities';@@ -91,8 +93,28 @@ const REAPED_JOB_ERROR = 'Generation timed out';* digits; this trips only when Redis stalls, bounding buffered batches and* queued commands by pacing the producer instead of growing without limit. */const MAX_OUTSTANDING_COALESCED_RECEIPTS = 256;-const PROVIDER_DRAIN_TIMEOUT_MS = 30_000;const PROVIDER_DRAIN_POLL_MS = 50;+type ToolExecutionStatus = 'success' | 'error' | 'cancelled';+type ToolCallWithExecutionStatus = Agents.AgentToolCall & {+ executionStatus?: ToolExecutionStatus;+};++/** Current agents SDK releases serialize tool failures into one of these+ * host-authored prefixes but do not yet carry ToolMessage.status on the wire.+ * Stamp the result on the exact call while the completion envelope still has+ * that identity; never infer it later from the aggregate step close. */+function completedToolExecutionStatus(call: Agents.ToolCall): ToolExecutionStatus {+ if (call.inputValidationError === true) {+ return 'error';+ }+ const output = call.output;+ return typeof output === 'string' &&+ (/^Error:\s*(\[.*?\]\s*)*tool call failed:/i.test(output) ||+ /^Error processing tool(?::|$)/i.test(output) ||+ /^Error:[\s\S]*\n Please fix your mistakes\.$/i.test(output))+ ? 'error'+ : 'success';+}/** Bounded completed-request replay horizon. It exceeds the default 24-hour* approval window; if a custom/live job outlasts it, `resumeClaimedGeneration`@@ -488,6 +510,13 @@ export type ApprovalExpiredHandler = (job: SerializableJobData,) => void | Promise<void>;+export type TerminalHostActionHandler = (+ streamId: string,+ job: SerializableJobData,+ runSteps: Agents.RunStep[],+ content: Agents.MessageContentComplex[],+) => void | Promise<void>;+export interface CreateGenerationJobOptions {startupTelemetry?: AgentStartupTelemetry;initialMetadata?: Partial<t.GenerationJobMetadata>;@@ -696,6 +725,9 @@ class GenerationJobManagerClass {/** Serializes token-usage read/modify/write updates per stream. */private tokenUsageWriteQueues = new Map<string, Promise<void>>();+ /** Serializes whole-array run-step snapshots so an older save cannot overwrite completion. */+ private runStepWriteQueues = new Map<string, Promise<void>>();+/** Partial-response and disconnect-state writes still draining during shutdown. */private subscriberCleanupPromises = new Set<Promise<void>>();@@ -741,6 +773,7 @@ class GenerationJobManagerClass {/** Optional host hook; the generic stream runtime does not know what external* durable work (scheduled chats, webhooks, etc.) a generation represents. */private approvalExpiredHandler: ApprovalExpiredHandler | undefined;+ private terminalHostActionHandler: TerminalHostActionHandler | undefined;constructor(options?: GenerationJobManagerOptions) {const jobStore =@@ -833,6 +866,7 @@ class GenerationJobManagerClass {this.runStepBuffers?.clear();this.replayEventWriteQueues.clear();this.tokenUsageWriteQueues.clear();+ this.runStepWriteQueues.clear();}this.ownedJobs.clear();@@ -876,6 +910,11 @@ class GenerationJobManagerClass {this.approvalExpiredHandler = handler;}+ /** Installs a durable, generation-fenced terminal lifecycle adapter. */+ setTerminalHostActionHandler(handler?: TerminalHostActionHandler): void {+ this.terminalHostActionHandler = handler;+ }+private get storeLabel(): GenerationJobStore {return this._isRedis ? 'redis' : 'memory';}@@ -1582,6 +1621,7 @@ class GenerationJobManagerClass {this.runStepBuffers?.delete(streamId);this.replayEventWriteQueues.delete(streamId);this.tokenUsageWriteQueues.delete(streamId);+ this.runStepWriteQueues.delete(streamId);this.jobStore.clearContentState(streamId, predecessor.createdAt);try {// The replacement cannot be exposed, so neither generation can@@ -1923,7 +1963,10 @@ class GenerationJobManagerClass {* terminal hash that makes every duplicate retry wait until Redis TTL expiry. */private async terminalizeUnexposedGeneration(streamId: string,- job: Pick<SerializableJobData, 'createdAt' | 'conversationId' | 'providerExecutionId'>,+ job: Pick<+ SerializableJobData,+ 'createdAt' | 'conversationId' | 'providerExecutionId' | 'agentEventDeliveryKey'+ >,message: string,): Promise<boolean> {try {@@ -1946,6 +1989,10 @@ class GenerationJobManagerClass {completedAt: Date.now(),error: message,finalEvent: JSON.stringify(finalEvent),+ ...(job.agentEventDeliveryKey != null &&+ this.terminalHostActionHandler != null && {+ terminalHostActionPending: true,+ }),},expectCreatedAt: job.createdAt,})@@ -2235,12 +2282,7 @@ class GenerationJobManagerClass {return false;},);- await this.jobStore.transitionStatus(streamId, {- from: 'running',- to: 'error',- patch: { completedAt: Date.now(), error: SHUTDOWN_JOB_ERROR },- expectCreatedAt: jobData.createdAt,- });+ await this.completeJob(streamId, SHUTDOWN_JOB_ERROR, jobData.createdAt);if (jobData.providerExecutionId) {await this.markProviderExecutionDrained(streamId,@@ -3262,6 +3304,21 @@ class GenerationJobManagerClass {]);}+ private async persistAgentEventRunStepEvidence(+ streamId: string,+ job: Pick<SerializableJobData, 'createdAt' | 'agentEventDeliveryKey'>,+ ): Promise<void> {+ if (job.agentEventDeliveryKey == null) {+ return;+ }+ const buffered = this.runStepBuffers?.get(streamId);+ if (buffered?.createdAt === job.createdAt && this.jobStore.saveRunSteps != null) {+ await this.queueJobWrite(this.runStepWriteQueues, streamId, () =>+ this.jobStore.saveRunSteps!(streamId, [...buffered.steps], job.createdAt),+ );+ }+ }+async claimTerminalJob(streamId: string,status: TerminalJobClaim['status'],@@ -3327,6 +3384,7 @@ class GenerationJobManagerClass {const runtime = observedRuntime?.createdAt === createdAt ? observedRuntime : undefined;const terminalError = status === 'error' ? (error ?? 'Generation failed') : undefined;await this.flushCoalescedStreamBuffers(streamId);+ await this.persistAgentEventRunStepEvidence(streamId, jobData);const completedAt = Date.now();const drainedSteers = await this.jobStore.transitionStatusAndDrainSteers(streamId, {from: sourceStatus,@@ -3340,6 +3398,10 @@ class GenerationJobManagerClass {patch: {completedAt,...(terminalError != null && { error: terminalError }),+ ...(jobData.agentEventDeliveryKey != null &&+ this.terminalHostActionHandler != null && {+ terminalHostActionPending: true,+ }),...(options.persistencePending === true && {terminalPersistencePending: true,terminalPersistenceStartedAt: completedAt,@@ -3535,10 +3597,20 @@ class GenerationJobManagerClass {? claimedRuntime: undefined;let cleanupError: unknown;+ let retainTerminalHostEvidence = false;// Error jobs stay durable long enough for late subscribers to receive the// stored error. A publication failure must never bypass the finally cleanup.try {+ const claimedTerminalJob = await this.jobStore.getJob(streamId);+ if (+ claimedTerminalJob?.createdAt === createdAt &&+ claimedTerminalJob.terminalHostActionPending === true+ ) {+ retainTerminalHostEvidence =+ claimedTerminalJob.providerDrained === false ||+ !(await this.runTerminalHostActionHandler(streamId, claimedTerminalJob));+ }if (status === 'error' && !this.terminalErrorPublicationSuppressions.has(claim)) {const terminalError = error ?? 'Generation failed';if (runtime) {@@ -3578,6 +3650,7 @@ class GenerationJobManagerClass {!retainForTerminalReplay &&terminalJob?.providerDrained !== false &&terminalJob?.preserveForScheduleReconcile !== true &&+ terminalJob?.terminalHostActionPending !== true &&(terminalJob?.createdAt !== createdAt || terminalJob.terminalPersistencePending !== true)) {// A same-stream replacement created after the claim makes this a safe@@ -3587,6 +3660,7 @@ class GenerationJobManagerClass {}} catch (err) {cleanupError = err;+ retainTerminalHostEvidence = true;} finally {if (runtime && this.runtimeState.get(streamId) === runtime) {this.releaseAbortSubscription(runtime);@@ -3599,10 +3673,13 @@ class GenerationJobManagerClass {runtime.startupTelemetry?.end('completed_without_delta');}runtime.startupTelemetry = undefined;- this.jobStore.clearContentState(streamId, createdAt);- this.runStepBuffers?.delete(streamId);+ if (!retainTerminalHostEvidence) {+ this.jobStore.clearContentState(streamId, createdAt);+ this.runStepBuffers?.delete(streamId);+ }this.replayEventWriteQueues.delete(streamId);this.tokenUsageWriteQueues.delete(streamId);+ this.runStepWriteQueues.delete(streamId);if (status !== 'error' && this._cleanupOnComplete) {this.runtimeState.delete(streamId);}@@ -3860,6 +3937,7 @@ class GenerationJobManagerClass {* claimTerminalJob), so it must drain the coalescers itself — and ahead of* the content snapshot, so a chunk-log reconstruction sees the window tail. */await this.flushCoalescedStreamBuffers(streamId);+ await this.persistAgentEventRunStepEvidence(streamId, jobData);/** Snapshot before claiming terminal state. This is non-destructive: if a* same-epoch approval resume wins the later CAS, its content and steer* queue remain fully owned by that resumed run. */@@ -3901,6 +3979,10 @@ class GenerationJobManagerClass {completedAt: terminalPersistenceStartedAt,terminalPersistencePending: true,terminalPersistenceStartedAt,+ ...(jobData.agentEventDeliveryKey != null &&+ this.terminalHostActionHandler != null && {+ terminalHostActionPending: true,+ }),},});if (drainedSteers != null) {@@ -4111,8 +4193,10 @@ class GenerationJobManagerClass {};} finally {try {- if (options?.awaitProviderDrain) {+ const mustDrainForTerminalEvidence = jobData.agentEventDeliveryKey != null;+ if (options?.awaitProviderDrain || mustDrainForTerminalEvidence) {await this.waitForProviderDrainIfRequired(streamId, jobData);+ await this.persistAgentEventRunStepEvidence(streamId, jobData);}} finally {await this.finishTerminalJob(terminalClaim);@@ -5538,11 +5622,18 @@ class GenerationJobManagerClass {const eventType = eventObj.event as string | undefined;const eventData = eventObj.data;if (- (eventType === 'on_run_step' || eventType === 'on_run_step_completed') &&+ (eventType === 'on_run_step' ||+ eventType === 'on_run_step_completed' ||+ eventType === 'on_run_step_closed') &&eventData != null &&typeof eventData === 'object') {- this.saveRunStepFromEvent(streamId, eventData as Record<string, unknown>, runtime.createdAt);+ this.saveRunStepFromEvent(+ streamId,+ eventType,+ eventData as Record<string, unknown>,+ runtime.createdAt,+ );}/**@@ -6021,6 +6112,7 @@ class GenerationJobManagerClass {this.runStepBuffers?.delete(streamId);this.replayEventWriteQueues.delete(streamId);this.tokenUsageWriteQueues.delete(streamId);+ this.runStepWriteQueues.delete(streamId);}}@@ -6130,22 +6222,147 @@ class GenerationJobManagerClass {}/**- * Extract and save run step from event data.- * The data is already the run step object from the event payload.+ * Extract and save a run step from its wire envelope. Completed events wrap+ * the authoritative step under `result`; live and closed events carry their+ * payload directly. The terminal close is the authoritative execution status:+ * a tool-end payload may contain an error output even though the step closes+ * as failed.*/private saveRunStepFromEvent(streamId: string,+ eventType: 'on_run_step' | 'on_run_step_completed' | 'on_run_step_closed',data: Record<string, unknown>,expectedCreatedAt: number,): void {- // The data IS the run step object- const runStep = data as Agents.RunStep;- if (!runStep.id) {+ const candidate = eventType === 'on_run_step_completed' ? data.result : data;+ if (+ candidate == null ||+ typeof candidate !== 'object' ||+ !('id' in candidate) ||+ typeof candidate.id !== 'string' ||+ candidate.id.length === 0+ ) {return;}- // Fire and forget - accumulate run steps- this.accumulateRunStep(streamId, runStep, expectedCreatedAt);+ if (eventType === 'on_run_step') {+ this.accumulateRunStep(streamId, candidate as Agents.RunStep, expectedCreatedAt);+ return;+ }++ if (eventType === 'on_run_step_closed') {+ const closed = candidate as Agents.RunStepClosedEvent;+ if (+ closed.status !== 'completed' &&+ closed.status !== 'cancelled' &&+ closed.status !== 'failed'+ ) {+ return;+ }+ const bufferState = this.runStepBuffers?.get(streamId);+ const existingStep =+ bufferState?.createdAt === expectedCreatedAt+ ? bufferState.steps.find((step) => step.id === closed.id)+ : undefined;+ if (existingStep == null) {+ return;+ }+ const terminalCalls =+ existingStep.stepDetails.type === StepTypes.TOOL_CALLS+ ? ((existingStep.stepDetails.tool_calls ?? []) as ToolCallWithExecutionStatus[])+ : [];+ const unresolvedCalls = terminalCalls.filter((call) => call.executionStatus == null);+ const calls: Agents.AgentToolCall[] =+ closed.status !== 'completed' && unresolvedCalls.length === 1+ ? terminalCalls.map((call) =>+ call === unresolvedCalls[0]+ ? {+ ...call,+ executionStatus: closed.status === 'failed' ? 'error' : 'cancelled',+ }+ : call,+ )+ : terminalCalls;+ this.accumulateRunStep(+ streamId,+ {+ ...existingStep,+ status: closed.status,+ ...(existingStep.stepDetails.type === StepTypes.TOOL_CALLS && {+ stepDetails: { ...existingStep.stepDetails, tool_calls: calls },+ }),+ },+ expectedCreatedAt,+ );+ return;+ }++ if ('stepDetails' in candidate) {+ this.accumulateRunStep(+ streamId,+ { ...(candidate as Agents.RunStep), status: 'completed' },+ expectedCreatedAt,+ );+ return;+ }++ const completion = candidate as Agents.ToolEndEvent;+ const completedToolCall = completion.tool_call;+ if (completedToolCall == null) {+ return;+ }+ const bufferState = this.runStepBuffers?.get(streamId);+ const existingStep =+ bufferState?.createdAt === expectedCreatedAt+ ? bufferState.steps.find((step) => step.id === completion.id)+ : undefined;+ const existingCalls =+ existingStep?.stepDetails?.type === StepTypes.TOOL_CALLS+ ? (existingStep.stepDetails.tool_calls ?? [])+ : [];+ const completedCallId = completedToolCall.id;+ const executionStatus = completedToolExecutionStatus(completedToolCall);+ const existingCallIndex =+ completedCallId == null ? -1 : existingCalls.findIndex((call) => call.id === completedCallId);+ const completedCalls = [...existingCalls];+ if (existingCallIndex >= 0) {+ const existingCall = completedCalls[existingCallIndex];+ completedCalls[existingCallIndex] =+ 'function' in existingCall+ ? ({+ id: existingCall.id,+ type: 'function',+ executionStatus,+ function: {+ ...existingCall.function,+ output: completedToolCall.output,+ },+ ...('inputValidationError' in completedToolCall &&+ completedToolCall.inputValidationError === true && {+ inputValidationError: true,+ }),+ } as ToolCallWithExecutionStatus)+ : { ...existingCall, ...completedToolCall, executionStatus };+ } else {+ completedCalls.push({ ...completedToolCall, executionStatus } as ToolCallWithExecutionStatus);+ }++ this.accumulateRunStep(+ streamId,+ {+ ...(existingStep ?? {+ id: completion.id,+ index: completion.index,+ type: StepTypes.TOOL_CALLS,+ }),+ status: 'completed',+ stepDetails: {+ type: StepTypes.TOOL_CALLS,+ tool_calls: completedCalls,+ },+ },+ expectedCreatedAt,+ );}/**@@ -6182,7 +6399,10 @@ class GenerationJobManagerClass {// Save to Redisif (this.jobStore.saveRunSteps) {- this.jobStore.saveRunSteps(streamId, buffer, expectedCreatedAt).catch((err) => {+ const snapshot = [...buffer];+ void this.queueJobWrite(this.runStepWriteQueues, streamId, () =>+ this.jobStore.saveRunSteps!(streamId, snapshot, expectedCreatedAt),+ ).catch((err) => {logger.error(`[GenerationJobManager] Failed to save run steps:`, err);});}@@ -6485,6 +6705,17 @@ class GenerationJobManagerClass {return false;}+ const observedJob = await this.jobStore.getJob(streamId);+ if (+ observedJob?.createdAt === expectedCreatedAt &&+ observedJob.providerExecutionId === providerExecutionId+ ) {+ // `providerDrained` is also the cross-replica host-settlement readiness+ // fence. Flush the complete generation-fenced run-step snapshot before+ // advertising that no more provider evidence can arrive.+ await this.persistAgentEventRunStepEvidence(streamId, observedJob);+ }+const [marked, recorded] = await Promise.all([this.jobStore.markProviderExecutionDrained?.(streamId,@@ -6495,6 +6726,19 @@ class GenerationJobManagerClass {Promise.resolve(false),]);const job = await this.jobStore.getJob(streamId);+ if (+ marked &&+ job?.createdAt === expectedCreatedAt &&+ job.providerExecutionId === providerExecutionId &&+ job.providerDrained === true &&+ job.terminalHostActionPending === true+ ) {+ if (job.status === 'aborted' && job.error === APPROVAL_EXPIRED_ERROR) {+ await this.runApprovalExpiredHandler(streamId, job);+ } else {+ await this.runTerminalHostActionHandler(streamId, job);+ }+ }if (marked &&recorded &&@@ -6504,6 +6748,7 @@ class GenerationJobManagerClass {job.providerDrained === true &&job.preserveForScheduleReconcile !== true &&job.terminalPersistencePending !== true &&+ job.terminalHostActionPending !== true &&(job.status === 'complete' || job.status === 'aborted')) {await this.jobStore.deleteJob(streamId, expectedCreatedAt);@@ -7155,7 +7400,10 @@ class GenerationJobManagerClass {expectedCreatedAt ?? observedJob?.createdAt,// Only retain a durable host-action marker when a host adapter is installed to// consume it — a store with no handler owes no action and accumulates nothing.- { markHostActionPending: this.approvalExpiredHandler != null },+ {+ markHostActionPending:+ this.approvalExpiredHandler != null || this.terminalHostActionHandler != null,+ },);if (expiredJob == null) {return false;@@ -7172,6 +7420,14 @@ class GenerationJobManagerClass {): Promise<void> {try {await this.approvalExpiredHandler?.(streamId, job);+ if (job.agentEventDeliveryKey != null) {+ await this.terminalHostActionHandler?.(+ streamId,+ job,+ await this.getTerminalRunSteps(streamId, job),+ (await this.jobStore.getContentParts(streamId, job.createdAt))?.content ?? [],+ );+ }// Success: the durable host action is settled. Clear its pending marker, fenced to// this exact generation so a replacement at the same streamId keeps its own state.// A no-op handler (or none) clears harmlessly, so non-scheduled jobs never linger.@@ -7185,6 +7441,52 @@ class GenerationJobManagerClass {}}+ private async runTerminalHostActionHandler(+ streamId: string,+ job: SerializableJobData,+ ): Promise<boolean> {+ try {+ if (job.agentEventDeliveryKey != null) {+ await this.terminalHostActionHandler?.(+ streamId,+ job,+ await this.getTerminalRunSteps(streamId, job),+ (await this.jobStore.getContentParts(streamId, job.createdAt))?.content ?? [],+ );+ }+ await this.jobStore.clearTerminalHostAction?.(streamId, job.createdAt);+ this.jobStore.clearContentState(streamId, job.createdAt);+ const buffered = this.runStepBuffers?.get(streamId);+ if (buffered?.createdAt === job.createdAt) {+ this.runStepBuffers?.delete(streamId);+ }+ return true;+ } catch (err) {+ logger.error(`[GenerationJobManager] Terminal host hook failed: ${streamId}`, err);+ return false;+ }+ }++ private async getTerminalRunSteps(+ streamId: string,+ job: Pick<SerializableJobData, 'createdAt'>,+ ): Promise<Agents.RunStep[]> {+ const persistedRunSteps = await this.jobStore.getRunSteps(streamId, job.createdAt);+ const buffered = this.runStepBuffers?.get(streamId);+ const runStepsById = new Map(persistedRunSteps.map((step) => [step.id, step]));+ if (buffered?.createdAt === job.createdAt) {+ for (const step of buffered.steps) {+ const persisted = runStepsById.get(step.id);+ // A stale/in-progress event must never erase authoritative completed+ // evidence recovered from the durable owner.+ if (persisted?.status !== 'completed' || step.status === 'completed') {+ runStepsById.set(step.id, step);+ }+ }+ }+ return [...runStepsById.values()].sort((left, right) => left.index - right.index);+ }+private async notifyApprovalExpiredRuntime(streamId: string,createdAt: number,@@ -7277,13 +7579,24 @@ class GenerationJobManagerClass {// successful ack prevents duplicate work. The terminal SSE relay below is// separately idempotent (emitError's errorEvent flag) and always runs so a// loser-replica subscriber still gets a terminal event.- if (job.terminalHostActionPending === true) {+ if (job.terminalHostActionPending === true && job.providerDrained !== false) {await this.runApprovalExpiredHandler(streamId, job);}await this.notifyApprovalExpiredRuntime(streamId, job.createdAt, runtime);changed = this.releaseJobOwnership(streamId, job.createdAt) || changed;continue;}+ if (+ job != null &&+ job.status !== 'running' &&+ job.status !== 'requires_action' &&+ job.terminalHostActionPending === true &&+ job.providerDrained !== false+ ) {+ await this.runTerminalHostActionHandler(streamId, job);+ changed = this.releaseJobOwnership(streamId, job.createdAt) || changed;+ continue;+ }if (!job || job.status !== 'requires_action' || !isPendingActionExpired(job)) {continue;}@@ -7362,6 +7675,14 @@ class GenerationJobManagerClass {}const currentJob = await this.jobStore.getJob(streamId);+ if (+ currentJob?.createdAt === observedRuntime.createdAt &&+ currentJob.terminalHostActionPending === true+ ) {+ // The callback retry still owns this generation's evidence. Retain+ // runtime buffers until it acknowledges and clears the durable marker.+ continue;+ }const isRetainedTerminal =currentJob?.createdAt === observedRuntime.createdAt &¤tJob.status !== 'running' &&@@ -7380,6 +7701,7 @@ class GenerationJobManagerClass {this.runStepBuffers?.delete(streamId);this.replayEventWriteQueues.delete(streamId);this.tokenUsageWriteQueues.delete(streamId);+ this.runStepWriteQueues.delete(streamId);this.jobStore.clearContentState(streamId, observedRuntime.createdAt);this.eventTransport.cleanup(streamId);continue;@@ -7555,7 +7877,6 @@ class GenerationJobManagerClass {return;}- const completedAt = Date.now();const results = await Promise.allSettled(ownedJobs.map(async ([streamId, createdAt]) => {const job = await this.jobStore.getJob(streamId);@@ -7571,35 +7892,11 @@ class GenerationJobManagerClass {runtime.lastSubscriberCleanupGeneration = runtime.attachmentGeneration;await this.persistSubscriberCleanup(streamId, runtime);}- /** Shutdown interrupts live emitters, so their coalesced window must- * drain before the terminal CAS — same rule as claim/abort. */- await this.flushCoalescedStreamBuffers(streamId);- const finalized = await this.jobStore.transitionStatus(streamId, {- from: 'running',- to: 'error',- expectCreatedAt: createdAt,- patch: { completedAt, error: SHUTDOWN_JOB_ERROR },- });- if (!finalized) {+ const claim = await this.claimTerminalJob(streamId, 'error', SHUTDOWN_JOB_ERROR, createdAt);+ if (claim == null) {return;}-- if (runtime?.createdAt === createdAt) {- runtime.errorEvent = SHUTDOWN_JOB_ERROR;- this.releaseAbortSubscription(runtime);- }- try {- await this.eventTransport.emitError(streamId, SHUTDOWN_JOB_ERROR, createdAt);- } catch (err) {- logger.error(- `[GenerationJobManager] Failed to publish shutdown error for ${streamId}:`,- err,- );- }- if (this.ownedJobs.get(streamId) === createdAt) {- this.ownedJobs.delete(streamId);- }- recordGenerationJob(this.storeLabel, 'error');+ await this.finishTerminalJob(claim);}),);@@ -7672,6 +7969,7 @@ class GenerationJobManagerClass {this.runStepBuffers?.clear();this.replayEventWriteQueues.clear();this.tokenUsageWriteQueues.clear();+ this.runStepWriteQueues.clear();logger.debug('[GenerationJobManager] Destroyed');}diff --git a/packages/api/src/stream/__tests__/RedisJobStore.spec.ts b/packages/api/src/stream/__tests__/RedisJobStore.spec.tsindex dc85ee363e..80ddae19eb 100644--- a/packages/api/src/stream/__tests__/RedisJobStore.spec.ts+++ b/packages/api/src/stream/__tests__/RedisJobStore.spec.ts@@ -915,7 +915,7 @@ describe('RedisJobStore', () => {]);});- test('guards run-step saves with the expected epoch and active status inside Redis Lua', async () => {+ test('guards run-step saves with the expected epoch and pending host-action window inside Redis Lua', async () => {const evalCommand = jest.fn().mockResolvedValue(0);const redis = {isCluster: true,@@ -930,6 +930,10 @@ describe('RedisJobStore', () => {expect(saveCall[0]).toContain('currentCreatedAt ~= ARGV[3]');expect(saveCall[0]).toContain('currentStatus ~= "running"');expect(saveCall[0]).toContain('currentStatus ~= "requires_action"');+ expect(saveCall[0]).toContain('terminalHostActionPending ~= "1"');+ expect(saveCall[0]).toContain(+ 'currentStatus == "requires_action" or terminalHostActionPending == "1"',+ );expect(saveCall.slice(1)).toEqual([2,'stream:{stream-runstep-guarded}:runsteps',@@ -940,6 +944,97 @@ describe('RedisJobStore', () => {]);});+ test('arms terminal host-action discovery before committing the terminal hash', async () => {+ const evalTransition = jest.fn().mockResolvedValue(1);+ const sadd = jest.fn().mockResolvedValue(1);+ const redis = {+ isCluster: true,+ eval: evalTransition,+ sadd,+ srem: jest.fn().mockResolvedValue(1),+ expire: jest.fn().mockResolvedValue(1),+ hgetall: jest+ .fn()+ .mockResolvedValueOnce({+ streamId: 'stream-host-action-prearm',+ userId: 'user-1',+ status: 'running',+ createdAt: '100',+ syncSent: '0',+ })+ .mockResolvedValue({+ streamId: 'stream-host-action-prearm',+ userId: 'user-1',+ status: 'complete',+ createdAt: '100',+ completedAt: '200',+ terminalHostActionPending: '1',+ syncSent: '0',+ }),+ } as unknown as Cluster;+ const store = new RedisJobStore(redis);++ await expect(+ store.transitionStatus('stream-host-action-prearm', {+ from: 'running',+ to: 'complete',+ expectCreatedAt: 100,+ patch: { completedAt: 200, terminalHostActionPending: true },+ }),+ ).resolves.toBe(true);++ expect(sadd.mock.calls[0]).toEqual([+ 'stream:terminal_host_action',+ '["stream-host-action-prearm",100]',+ ]);+ expect(sadd.mock.invocationCallOrder[0]).toBeLessThan(+ evalTransition.mock.invocationCallOrder[0],+ );+ const transitionCall = evalTransition.mock.calls[0];+ expect(transitionCall[15]).toBe('86400');+ expect(transitionCall[17]).toBe('86400');+ expect(transitionCall[18]).toBe('86400');+ });++ test('acknowledges evidence TTLs without removing a successor retry member', async () => {+ const evalClear = jest.fn().mockResolvedValue(1);+ const srem = jest.fn().mockResolvedValue(1);+ const redis = {+ isCluster: true,+ eval: evalClear,+ srem,+ } as unknown as Cluster;+ const store = new RedisJobStore(redis, {+ completedTtl: 300,+ chunksAfterCompleteTtl: 7,+ runStepsAfterCompleteTtl: 11,+ });++ await store.clearTerminalHostAction('stream-host-action-clear', 100);++ expect(evalClear).toHaveBeenCalledWith(+ expect.stringContaining(+ 'if tonumber(ARGV[2]) > 0 then redis.call("EXPIRE", KEYS[1], ARGV[2]) else redis.call("DEL", KEYS[1]) end',+ ),+ 3,+ 'stream:{stream-host-action-clear}:job',+ 'stream:{stream-host-action-clear}:chunks',+ 'stream:{stream-host-action-clear}:runsteps',+ '100',+ '300',+ '7',+ '11',+ );+ expect(srem).toHaveBeenCalledWith(+ 'stream:terminal_host_action',+ '["stream-host-action-clear",100]',+ );+ expect(srem).not.toHaveBeenCalledWith(+ 'stream:terminal_host_action',+ '["stream-host-action-clear",101]',+ );+ });+test('guards asynchronous content cleanup against a replacement epoch', async () => {const evalCommand = jest.fn().mockResolvedValue(0);const redis = {diff --git a/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts b/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.tsindex 98fa0deb54..c7458dec3b 100644--- a/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts+++ b/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts@@ -1069,6 +1069,124 @@ describe('RedisJobStore Integration Tests', () => {await store.destroy();});+ test('terminal host settlement retains evidence and waits for the provider drain fence', async () => {+ if (!ioredisClient) {+ return;+ }++ const { RedisJobStore } = await import('../implementations/RedisJobStore');+ const store = new RedisJobStore(ioredisClient, { runningTtl: 60 });+ await store.initialize();++ const streamId = `terminal-runsteps-${Date.now()}`;+ const providerExecutionId = 'terminal-runsteps-provider';+ const job = await store.createJob(streamId, 'user-1', streamId, undefined, {+ providerExecutionId,+ });+ await expect(+ store.beginProviderExecution(streamId, job.createdAt, providerExecutionId),+ ).resolves.toBe(true);+ const completedStep = {+ id: 'step-terminal',+ index: 0,+ type: StepTypes.TOOL_CALLS,+ status: 'completed',+ stepDetails: { type: StepTypes.TOOL_CALLS, tool_calls: [] },+ } as Agents.RunStep;+ await store.saveRunSteps(streamId, [completedStep], job.createdAt);+ await expect(+ store.transitionStatus(streamId, {+ from: 'running',+ to: 'aborted',+ expectCreatedAt: job.createdAt,+ patch: { completedAt: Date.now(), terminalHostActionPending: true },+ }),+ ).resolves.toBe(true);+ await expect(store.getTerminalHostActionJobs?.()).resolves.toEqual([]);+ await expect(store.getRunSteps(streamId, job.createdAt)).resolves.toEqual([completedStep]);++ await expect(+ store.markProviderExecutionDrained(streamId, job.createdAt, providerExecutionId),+ ).resolves.toBe(true);+ await expect(store.getTerminalHostActionJobs?.()).resolves.toEqual([+ expect.objectContaining({ streamId, providerDrained: true }),+ ]);++ await store.clearTerminalHostAction?.(streamId, job.createdAt);+ const lateStep = { ...completedStep, id: 'step-too-late' };+ await store.saveRunSteps(streamId, [lateStep], job.createdAt);+ await expect(store.getRunSteps(streamId, job.createdAt)).resolves.toEqual([]);++ await store.destroy();+ });++ test('terminal host settlement recovers a provider owner lost after the terminal CAS', async () => {+ if (!ioredisClient) {+ return;+ }++ const { RedisJobStore } = await import('../implementations/RedisJobStore');+ const store = new RedisJobStore(ioredisClient, { runningTtl: 60 });+ await store.initialize();++ const streamId = `terminal-lost-provider-${Date.now()}`;+ const providerExecutionId = 'terminal-lost-provider';+ const job = await store.createJob(streamId, 'user-1', streamId, undefined, {+ providerExecutionId,+ });+ await store.beginProviderExecution(streamId, job.createdAt, providerExecutionId);+ await expect(+ store.transitionStatus(streamId, {+ from: 'running',+ to: 'complete',+ expectCreatedAt: job.createdAt,+ patch: {+ completedAt: Date.now() - 30_001,+ terminalHostActionPending: true,+ },+ }),+ ).resolves.toBe(true);++ await expect(store.getTerminalHostActionJobs?.()).resolves.toEqual([+ expect.objectContaining({ streamId, providerDrained: true }),+ ]);+ await expect(store.getJob(streamId)).resolves.toMatchObject({ providerDrained: true });++ await store.clearTerminalHostAction(streamId, job.createdAt);+ await store.destroy();+ });++ test('terminal host acknowledgement deletes a zero-TTL job after settlement', async () => {+ if (!ioredisClient) {+ return;+ }++ const { RedisJobStore } = await import('../implementations/RedisJobStore');+ const store = new RedisJobStore(ioredisClient, { completedTtl: 0 });+ await store.initialize();++ const streamId = `terminal-host-zero-ttl-${Date.now()}`;+ const job = await store.createJob(streamId, 'user-1', streamId);+ await expect(+ store.transitionStatus(streamId, {+ from: 'running',+ to: 'aborted',+ expectCreatedAt: job.createdAt,+ patch: { completedAt: Date.now(), terminalHostActionPending: true },+ }),+ ).resolves.toBe(true);+ await expect(store.getJob(streamId)).resolves.toMatchObject({+ status: 'aborted',+ terminalHostActionPending: true,+ });++ await expect(store.clearTerminalHostAction(streamId, job.createdAt)).resolves.toBeUndefined();+ await expect(store.getJob(streamId)).resolves.toBeNull();+ await expect(store.getTerminalHostActionJobs()).resolves.toEqual([]);++ await store.destroy();+ });+test('appendChunk gives the approval TTL when the chunk key did not exist at pause time', async () => {if (!ioredisClient) {return;@@ -1180,6 +1298,11 @@ describe('RedisJobStore Integration Tests', () => {await store.createJob(streamId, 'user-1', streamId, undefined, {agent_id: 'saved-agent-1',isTemporary: true,+ agentEventDeliveryKey: 'trigger_1',+ agentEventExpectedAction: {+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ },discoveredTools: ['deep_tool'],userSubmittedPaths: ['/content/0/tool_call/args'],userSubmittedMessageFieldPaths: [{ path: '/content/0/tool_call/output', field: 'answer' }],@@ -1187,6 +1310,11 @@ describe('RedisJobStore Integration Tests', () => {const turn1 = await store.getJob(streamId);expect(turn1?.agent_id).toBe('saved-agent-1');expect(turn1?.isTemporary).toBe(true);+ expect(turn1?.agentEventDeliveryKey).toBe('trigger_1');+ expect(turn1?.agentEventExpectedAction).toEqual({+ toolName: 'submit_move',+ argumentSubset: { gameId: 'game-1', expectedPly: 7 },+ });expect(turn1?.discoveredTools).toEqual(['deep_tool']);expect(turn1?.userSubmittedPaths).toEqual(['/content/0/tool_call/args']);expect(turn1?.userSubmittedMessageFieldPaths).toEqual([@@ -1202,6 +1330,8 @@ describe('RedisJobStore Integration Tests', () => {const turn2 = await store.getJob(streamId);expect(turn2?.agent_id).toBeUndefined();expect(turn2?.isTemporary).toBeUndefined();+ expect(turn2?.agentEventDeliveryKey).toBeUndefined();+ expect(turn2?.agentEventExpectedAction).toBeUndefined();expect(turn2?.discoveredTools).toBeUndefined();expect(turn2?.userSubmittedPaths).toBeUndefined();expect(turn2?.userSubmittedMessageFieldPaths).toBeUndefined();@@ -3309,6 +3439,11 @@ describe('RedisJobStore Integration Tests', () => {try {const streamId = 'pause-barrier-expiry-cleanup';const job = await store.createJob(streamId, 'steer-user', streamId, 'tenant-1');+ await store.updateJob(+ streamId,+ { agentEventDeliveryKey: 'trigger-pause-barrier-expiry' },+ job.createdAt,+ );await store.enqueueSteer(streamId,buildSteer('pause-barrier-steer', 'frozen while pause persists'),@@ -3341,6 +3476,7 @@ describe('RedisJobStore Integration Tests', () => {await expect(store.getJob(streamId)).resolves.toMatchObject({status: 'error',error: PAUSE_PERSISTENCE_TIMEOUT_ERROR,+ terminalHostActionPending: true,});const failedJob = await store.getJob(streamId);expect(failedJob?.pendingAction).toBeUndefined();diff --git a/packages/api/src/stream/__tests__/deltaCoalescing.stream_integration.spec.ts b/packages/api/src/stream/__tests__/deltaCoalescing.stream_integration.spec.tsindex ded9014a87..5db02086cc 100644--- a/packages/api/src/stream/__tests__/deltaCoalescing.stream_integration.spec.ts+++ b/packages/api/src/stream/__tests__/deltaCoalescing.stream_integration.spec.ts@@ -506,11 +506,12 @@ describe('Delta coalescing integration', () => {await manager.emitChunk(streamId, {event: 'on_run_step_completed',data: {- id: 'step-1',- runId: 'run-1',- index: 0,- stepDetails: { type: 'message_creation' },- result: { id: 'step-1' },+ result: {+ id: 'step-1',+ runId: 'run-1',+ index: 0,+ stepDetails: { type: 'message_creation' },+ },},});diff --git a/packages/api/src/stream/__tests__/pendingAction.spec.ts b/packages/api/src/stream/__tests__/pendingAction.spec.tsindex cdca10c1e1..a4612c869a 100644--- a/packages/api/src/stream/__tests__/pendingAction.spec.ts+++ b/packages/api/src/stream/__tests__/pendingAction.spec.ts@@ -799,7 +799,7 @@ describe('ApprovalLifecycle via GenerationJobManager.approvals (in-memory)', ()streamId,expect.objectContaining({createdAt: job.createdAt,- status: 'requires_action',+ status: 'aborted',scheduleId: 'schedule-1',scheduledFor: '2026-08-17T12:00:00.000Z',}),@@ -826,7 +826,7 @@ describe('ApprovalLifecycle via GenerationJobManager.approvals (in-memory)', ()expect.objectContaining({createdAt: job.createdAt,scheduleId: 'schedule-ownerless',- status: 'requires_action',+ status: 'aborted',}),);await expect(jobStore.getJob(streamId)).resolves.toMatchObject({ status: 'aborted' });@@ -1059,6 +1059,28 @@ describe('ApprovalLifecycle via GenerationJobManager.approvals (in-memory)', ()expect(handler).toHaveBeenCalledTimes(1); // no duplicate invocation});+ test('passes the committed aborted state to an event outcome handler', async () => {+ const streamId = 'stream-host-event-expiry-state';+ await manager.createJob(streamId, 'user-1');+ await manager.updateMetadata(streamId, { agentEventDeliveryKey: 'delivery-expired' });+ await manager.approvals.pause(streamId, buildAction(streamId));+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);++ expect(await manager.expireApproval(streamId)).toBe(true);++ expect(handler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({+ status: 'aborted',+ error: 'Approval expired before a decision was made',+ terminalHostActionPending: true,+ }),+ expect.any(Array),+ expect.any(Array),+ );+ });+test('clearTerminalHostAction is identity-fenced against a replacement generation', async () => {const streamId = 'stream-host-fence';const job = await pauseScheduled(streamId);@@ -1265,6 +1287,11 @@ describe('InMemoryJobStore — approval expiry cleanup', () => {const store = new InMemoryJobStore({ ttlAfterComplete: 60_000 });try {const job = await store.createJob('stale-pause-barrier', 'u1');+ await store.updateJob(+ 'stale-pause-barrier',+ { agentEventDeliveryKey: 'trigger-stale-pause' },+ job.createdAt,+ );await store.enqueueSteer('stale-pause-barrier',{@@ -1296,6 +1323,7 @@ describe('InMemoryJobStore — approval expiry cleanup', () => {await expect(store.getJob('stale-pause-barrier')).resolves.toMatchObject({status: 'error',error: PAUSE_PERSISTENCE_TIMEOUT_ERROR,+ terminalHostActionPending: true,});const failedJob = await store.getJob('stale-pause-barrier');expect(failedJob?.pendingAction).toBeUndefined();diff --git a/packages/api/src/stream/__tests__/predecessorFence.spec.ts b/packages/api/src/stream/__tests__/predecessorFence.spec.tsindex 50e20695d0..892f07a1bc 100644--- a/packages/api/src/stream/__tests__/predecessorFence.spec.ts+++ b/packages/api/src/stream/__tests__/predecessorFence.spec.ts@@ -83,6 +83,18 @@ describe('generation predecessor create fence', () => {currentJob: { active: true, status: 'aborted' },});await store.updateJob(streamId, { terminalPersistencePending: false }, parent.createdAt);+ await store.updateJob(streamId, { terminalHostActionPending: true }, parent.createdAt);+ await expect(createWakeupJob(store, streamId)).rejects.toMatchObject({+ currentJob: { active: true, status: 'aborted' },+ });+ await expect(+ store.createJob(streamId, 'owner-1', streamId, undefined, {+ generationProtocolVersion: 2,+ }),+ ).rejects.toMatchObject({+ currentJob: { active: true, status: 'aborted' },+ });+ await store.updateJob(streamId, { terminalHostActionPending: false }, parent.createdAt);const wakeup = await createWakeupJob(store, streamId);expect(wakeup.createdAt).toBeGreaterThanOrEqual(parent.createdAt);} finally {diff --git a/packages/api/src/stream/__tests__/predecessorFence.stream_integration.spec.ts b/packages/api/src/stream/__tests__/predecessorFence.stream_integration.spec.tsindex d2854bfdd5..74c64a2fdb 100644--- a/packages/api/src/stream/__tests__/predecessorFence.stream_integration.spec.ts+++ b/packages/api/src/stream/__tests__/predecessorFence.stream_integration.spec.ts@@ -123,6 +123,20 @@ describe('Redis generation predecessor create fence', () => {currentJob: { active: true, status: 'aborted' },});await store.updateJob(streamId, { terminalPersistencePending: false }, parent.createdAt);+ await store.updateJob(streamId, { terminalHostActionPending: true }, parent.createdAt);+ await expect(+ createWakeupJob(store, streamId, 'redis-wakeup-terminal-host-action'),+ ).rejects.toMatchObject({+ currentJob: { active: true, status: 'aborted' },+ });+ await expect(+ store.createJob(streamId, 'owner-1', streamId, undefined, {+ generationProtocolVersion: 2,+ }),+ ).rejects.toMatchObject({+ currentJob: { active: true, status: 'aborted' },+ });+ await store.updateJob(streamId, { terminalHostActionPending: false }, parent.createdAt);const wakeup = await createWakeupJob(store, streamId, 'redis-wakeup-settled');expect(wakeup.createdAt).toBeGreaterThanOrEqual(parent.createdAt);});diff --git a/packages/api/src/stream/__tests__/startup.spec.ts b/packages/api/src/stream/__tests__/startup.spec.tsindex 6709419b49..21e9c06f71 100644--- a/packages/api/src/stream/__tests__/startup.spec.ts+++ b/packages/api/src/stream/__tests__/startup.spec.ts@@ -248,6 +248,41 @@ describe('GenerationJobManager startup telemetry', () => {await manager.destroy();});+ it('waits for provider-owner drain before settling an event-driven abort', async () => {+ const manager = createManager();+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const streamId = 'stream-event-provider-drain-wait';+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: { agentEventDeliveryKey: 'delivery-1' },+ });+ await expect(+ manager.beginProviderExecution(streamId, job.createdAt, job.metadata.providerExecutionId!),+ ).resolves.toBe(true);++ let abortSettled = false;+ const aborting = manager.abortJob(streamId).finally(() => (abortSettled = true));++ await new Promise<void>((resolve) => setImmediate(resolve));+ expect(job.abortController.signal.aborted).toBe(true);+ expect(abortSettled).toBe(false);+ expect(handler).not.toHaveBeenCalled();++ await manager.markProviderExecutionDrained(+ streamId,+ job.createdAt,+ job.metadata.providerExecutionId!,+ );+ await expect(aborting).resolves.toMatchObject({ success: true });+ expect(handler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({ agentEventDeliveryKey: 'delivery-1', status: 'aborted' }),+ [],+ expect.any(Array),+ );+ await manager.destroy();+ });+it('finishes a destructive abort immediately when the provider never started', async () => {const manager = createManager();const streamId = 'stream-provider-never-started';diff --git a/packages/api/src/stream/__tests__/terminalHostAction.spec.ts b/packages/api/src/stream/__tests__/terminalHostAction.spec.tsnew file mode 100644index 0000000000..2bf81b5953--- /dev/null+++ b/packages/api/src/stream/__tests__/terminalHostAction.spec.ts@@ -0,0 +1,514 @@+import { InMemoryEventTransport } from '~/stream/implementations/InMemoryEventTransport';+import { buildPendingAction, buildToolApprovalPayload } from '~/agents/hitl/policy';+import { InMemoryJobStore } from '~/stream/implementations/InMemoryJobStore';+import { GenerationJobManagerClass } from '~/stream/GenerationJobManager';++describe('GenerationJobManager terminal host actions', () => {+ let manager: GenerationJobManagerClass;+ let store: InMemoryJobStore;++ beforeEach(() => {+ manager = new GenerationJobManagerClass();+ store = new InMemoryJobStore({ ttlAfterComplete: 60_000 });+ manager.configure({+ jobStore: store,+ eventTransport: new InMemoryEventTransport(),+ cleanupOnComplete: false,+ });+ manager.initialize();+ });++ afterEach(async () => {+ await manager.destroy();+ });++ it('runs and acknowledges a generation-fenced host action before terminal cleanup', async () => {+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const job = await manager.createJob('conversation-1', 'user-1', 'conversation-1', {+ initialMetadata: {+ agentEventDeliveryKey: 'trigger_1',+ agentEventExpectedAction: { toolName: 'submit_move' },+ },+ });+ await manager.emitChunk('conversation-1', {+ event: 'on_run_step',+ data: {+ id: 'step-1',+ index: 0,+ type: 'tool_calls',+ status: 'in_progress',+ stepDetails: {+ type: 'tool_calls',+ tool_calls: [{ id: 'call-1', name: 'submit_move', args: { gameId: 'game-1' } }],+ },+ },+ });+ await manager.emitChunk('conversation-1', {+ event: 'on_run_step_completed',+ data: {+ result: {+ id: 'step-1',+ index: 0,+ type: 'tool_call',+ tool_call: { id: 'call-1', name: 'submit_move', output: 'ok' },+ },+ },+ });++ await expect(manager.completeJob('conversation-1', undefined, job.createdAt)).resolves.toBe(+ true,+ );++ expect(handler).toHaveBeenCalledWith(+ 'conversation-1',+ expect.objectContaining({+ createdAt: job.createdAt,+ agentEventDeliveryKey: 'trigger_1',+ status: 'complete',+ }),+ [+ expect.objectContaining({+ id: 'step-1',+ status: 'completed',+ stepDetails: expect.objectContaining({+ tool_calls: [+ expect.objectContaining({+ id: 'call-1',+ args: { gameId: 'game-1' },+ output: 'ok',+ }),+ ],+ }),+ }),+ ],+ expect.any(Array),+ );+ await expect(store.getJob('conversation-1')).resolves.not.toHaveProperty(+ 'terminalHostActionPending',+ );+ });++ it('waits for the provider evidence fence and retains a failed tool close', async () => {+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const streamId = 'conversation-provider-evidence';+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: {+ agentEventDeliveryKey: 'trigger_provider_evidence',+ },+ });+ const providerExecutionId = job.metadata.providerExecutionId!;+ await expect(+ manager.beginProviderExecution(streamId, job.createdAt, providerExecutionId),+ ).resolves.toBe(true);+ await manager.emitChunk(streamId, {+ event: 'on_run_step',+ data: {+ id: 'step-failed',+ index: 0,+ type: 'tool_calls',+ status: 'in_progress',+ stepDetails: {+ type: 'tool_calls',+ tool_calls: [{ id: 'call-failed', name: 'submit_move', args: {} }],+ },+ },+ });+ await manager.emitChunk(streamId, {+ event: 'on_run_step_completed',+ data: {+ result: {+ id: 'step-failed',+ index: 0,+ type: 'tool_call',+ tool_call: {+ id: 'call-failed',+ name: 'submit_move',+ output: 'Error: [submit_move] tool call failed: unavailable',+ },+ },+ },+ });+ await manager.emitChunk(streamId, {+ event: 'on_run_step_closed',+ data: {+ id: 'step-failed',+ index: 0,+ type: 'tool_calls',+ status: 'failed',+ closed_at: Date.now(),+ },+ });++ await expect(manager.completeJob(streamId, undefined, job.createdAt)).resolves.toBe(true);+ expect(handler).not.toHaveBeenCalled();+ await expect(store.getJob(streamId)).resolves.toMatchObject({+ providerDrained: false,+ terminalHostActionPending: true,+ });++ await expect(+ manager.markProviderExecutionDrained(streamId, job.createdAt, providerExecutionId),+ ).resolves.toBe(true);+ expect(handler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({ providerDrained: true }),+ [+ expect.objectContaining({+ id: 'step-failed',+ status: 'failed',+ stepDetails: expect.objectContaining({+ tool_calls: [expect.objectContaining({ executionStatus: 'error' })],+ }),+ }),+ ],+ expect.any(Array),+ );+ await expect(store.getJob(streamId)).resolves.not.toHaveProperty('terminalHostActionPending');+ });++ it('routes a post-drain approval expiry through the schedule-specific hook', async () => {+ const approvalHandler = jest.fn().mockResolvedValue(undefined);+ const genericHandler = jest.fn().mockResolvedValue(undefined);+ manager.setApprovalExpiredHandler(approvalHandler);+ manager.setTerminalHostActionHandler(genericHandler);+ const streamId = 'conversation-expired-approval-drain';+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: { scheduleId: 'schedule-1' },+ });+ const providerExecutionId = job.metadata.providerExecutionId!;+ await manager.beginProviderExecution(streamId, job.createdAt, providerExecutionId);+ await expect(+ store.transitionStatus(streamId, {+ from: 'running',+ to: 'aborted',+ expectCreatedAt: job.createdAt,+ patch: {+ completedAt: Date.now(),+ error: 'Approval expired before a decision was made',+ terminalHostActionPending: true,+ },+ }),+ ).resolves.toBe(true);++ await manager.markProviderExecutionDrained(streamId, job.createdAt, providerExecutionId);++ expect(approvalHandler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({ scheduleId: 'schedule-1', providerDrained: true }),+ );+ expect(genericHandler).not.toHaveBeenCalled();+ await expect(store.getJob(streamId)).resolves.not.toHaveProperty('terminalHostActionPending');+ });++ it('recovers a terminal host action after its provider owner is lost', async () => {+ const now = jest.spyOn(Date, 'now').mockReturnValue(10_000);+ try {+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const streamId = 'conversation-lost-provider-owner';+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: { agentEventDeliveryKey: 'trigger_lost_provider' },+ });+ await manager.beginProviderExecution(+ streamId,+ job.createdAt,+ job.metadata.providerExecutionId!,+ );+ await manager.completeJob(streamId, undefined, job.createdAt);+ expect(handler).not.toHaveBeenCalled();++ now.mockReturnValue(40_001);+ await (+ manager as unknown as { expireStaleApprovals: () => Promise<void> }+ ).expireStaleApprovals();++ expect(handler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({ providerDrained: true }),+ expect.any(Array),+ expect.any(Array),+ );+ await expect(store.getJob(streamId)).resolves.not.toHaveProperty('terminalHostActionPending');+ } finally {+ now.mockRestore();+ }+ });++ it('serializes run-step snapshots so an earlier write cannot erase completion evidence', async () => {+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const streamId = 'conversation-run-step-order';+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: { agentEventDeliveryKey: 'trigger_run_step_order' },+ });+ let releaseFirstWrite: (() => void) | undefined;+ const firstWrite = new Promise<void>((resolve) => {+ releaseFirstWrite = resolve;+ });+ const writes: Array<Array<{ status?: string }>> = [];+ Object.assign(store, { saveRunSteps: jest.fn() });+ jest+ .spyOn(store as InMemoryJobStore & { saveRunSteps: jest.Mock }, 'saveRunSteps')+ .mockImplementation(async (_id, steps) => {+ writes.push(structuredClone(steps));+ if (writes.length === 1) {+ await firstWrite;+ }+ });++ await manager.emitChunk(streamId, {+ event: 'on_run_step',+ data: {+ id: 'step-ordered',+ index: 0,+ type: 'tool_calls',+ status: 'in_progress',+ stepDetails: {+ type: 'tool_calls',+ tool_calls: [{ id: 'call-ordered', name: 'submit_move', args: {} }],+ },+ },+ });+ await manager.emitChunk(streamId, {+ event: 'on_run_step_completed',+ data: {+ result: {+ id: 'step-ordered',+ index: 0,+ type: 'tool_call',+ tool_call: { id: 'call-ordered', name: 'submit_move', output: 'ok' },+ },+ },+ });++ const completing = manager.completeJob(streamId, undefined, job.createdAt);+ await new Promise<void>((resolve) => setImmediate(resolve));+ expect(handler).not.toHaveBeenCalled();+ releaseFirstWrite?.();+ await expect(completing).resolves.toBe(true);++ expect(writes[writes.length - 1]).toEqual([expect.objectContaining({ status: 'completed' })]);+ expect(handler).toHaveBeenCalledTimes(1);+ });++ it('retains a failed host action and lets a later cleanup owner retry it', async () => {+ manager.setTerminalHostActionHandler(jest.fn().mockRejectedValue(new Error('mongo down')));+ const job = await manager.createJob('conversation-2', 'user-1', 'conversation-2', {+ initialMetadata: {+ agentEventDeliveryKey: 'trigger_2',+ agentEventExpectedAction: { toolName: 'submit_move' },+ },+ });+ await manager.emitChunk('conversation-2', {+ event: 'on_run_step_completed',+ data: {+ result: {+ id: 'step-2',+ index: 0,+ type: 'tool_calls',+ status: 'completed',+ stepDetails: {+ type: 'tool_calls',+ tool_calls: [{ id: 'call-2', name: 'submit_move', output: 'ok' }],+ },+ },+ },+ });+ await manager.completeJob('conversation-2', undefined, job.createdAt);+ await expect(store.getJob('conversation-2')).resolves.toMatchObject({+ terminalHostActionPending: true,+ });++ const recovered = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(recovered);+ await (+ manager as unknown as {+ cleanup: () => Promise<void>;+ }+ ).cleanup();++ expect(recovered).toHaveBeenCalledWith(+ 'conversation-2',+ expect.objectContaining({ createdAt: job.createdAt, agentEventDeliveryKey: 'trigger_2' }),+ [expect.objectContaining({ id: 'step-2', status: 'completed' })],+ expect.any(Array),+ );+ await expect(store.getJob('conversation-2')).resolves.not.toHaveProperty(+ 'terminalHostActionPending',+ );+ });++ it('retains completed run-step evidence through repeated host-action failures', async () => {+ const handler = jest+ .fn()+ .mockRejectedValueOnce(new Error('mongo down'))+ .mockRejectedValueOnce(new Error('mongo still down'))+ .mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const job = await manager.createJob('conversation-retry', 'user-1', 'conversation-retry', {+ initialMetadata: { agentEventDeliveryKey: 'trigger_retry' },+ });+ await manager.emitChunk('conversation-retry', {+ event: 'on_run_step_completed',+ data: {+ result: {+ id: 'step-retry',+ index: 0,+ type: 'tool_calls',+ status: 'completed',+ stepDetails: {+ type: 'tool_calls',+ tool_calls: [{ id: 'call-retry', name: 'submit_move', output: 'ok' }],+ },+ },+ },+ });+ await manager.completeJob('conversation-retry', undefined, job.createdAt);++ const cleanup = () =>+ (+ manager as unknown as {+ cleanup: () => Promise<void>;+ }+ ).cleanup();+ await cleanup();+ await expect(store.getJob('conversation-retry')).resolves.toMatchObject({+ terminalHostActionPending: true,+ });++ await cleanup();++ expect(handler).toHaveBeenLastCalledWith(+ 'conversation-retry',+ expect.objectContaining({ agentEventDeliveryKey: 'trigger_retry' }),+ [expect.objectContaining({ id: 'step-retry', status: 'completed' })],+ expect.any(Array),+ );+ await expect(store.getJob('conversation-retry')).resolves.not.toHaveProperty(+ 'terminalHostActionPending',+ );+ });++ it('does not let a stale buffered step replace completed durable evidence', async () => {+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const streamId = 'conversation-completed-evidence';+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: { agentEventDeliveryKey: 'trigger_completed' },+ });+ const completedStep = {+ id: 'step-shared',+ index: 0,+ type: 'tool_calls',+ status: 'completed',+ stepDetails: {+ type: 'tool_calls',+ tool_calls: [{ id: 'call-shared', name: 'submit_move', output: 'ok' }],+ },+ };+ store.setGraph(streamId, { contentData: [completedStep] } as never, job.createdAt);+ await manager.emitChunk(streamId, {+ event: 'on_run_step',+ data: { ...completedStep, status: 'in_progress' },+ });++ await manager.completeJob(streamId, undefined, job.createdAt);++ expect(handler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({ agentEventDeliveryKey: 'trigger_completed' }),+ [expect.objectContaining({ id: 'step-shared', status: 'completed' })],+ expect.any(Array),+ );+ });++ it('marks an aborted event generation for terminal handling', async () => {+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const job = await manager.createJob('conversation-3', 'user-1', 'conversation-3', {+ initialMetadata: { agentEventDeliveryKey: 'trigger_3' },+ });++ await expect(+ manager.abortJob('conversation-3', { expectedCreatedAt: job.createdAt }),+ ).resolves.toMatchObject({ success: true });++ expect(handler).toHaveBeenCalledWith(+ 'conversation-3',+ expect.objectContaining({+ createdAt: job.createdAt,+ agentEventDeliveryKey: 'trigger_3',+ status: 'aborted',+ }),+ [],+ expect.any(Array),+ );+ await expect(store.getJob('conversation-3')).resolves.not.toHaveProperty(+ 'terminalHostActionPending',+ );+ });++ it('settles a bound generation when graceful shutdown terminalizes it', async () => {+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const streamId = 'conversation-shutdown';+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: { agentEventDeliveryKey: 'trigger_shutdown' },+ });++ await (+ manager as unknown as { finalizeOwnedJobsForShutdown: () => Promise<void> }+ ).finalizeOwnedJobsForShutdown();++ expect(handler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({+ createdAt: job.createdAt,+ agentEventDeliveryKey: 'trigger_shutdown',+ status: 'error',+ }),+ [],+ expect.any(Array),+ );+ await expect(store.getJob(streamId)).resolves.not.toHaveProperty('terminalHostActionPending');+ });++ it('settles a bound generation when pause persistence times out', async () => {+ const now = jest.spyOn(Date, 'now').mockReturnValue(1_000);+ const handler = jest.fn().mockResolvedValue(undefined);+ manager.setTerminalHostActionHandler(handler);+ const streamId = 'conversation-pause-timeout';+ const job = await manager.createJob(streamId, 'user-1', streamId, {+ initialMetadata: { agentEventDeliveryKey: 'trigger_pause_timeout' },+ });+ const action = buildPendingAction(+ buildToolApprovalPayload([+ { name: 'submit_move', arguments: { gameId: 'game-1' }, tool_call_id: 'call-1' },+ ]),+ { streamId, conversationId: streamId },+ );+ await expect(+ manager.approvals.pause(streamId, action, {+ expectedCreatedAt: job.createdAt,+ persistencePending: true,+ }),+ ).resolves.toBe(true);++ now.mockReturnValue(31_001);+ await (manager as unknown as { cleanup: () => Promise<void> }).cleanup();++ expect(handler).toHaveBeenCalledWith(+ streamId,+ expect.objectContaining({+ agentEventDeliveryKey: 'trigger_pause_timeout',+ status: 'error',+ }),+ [],+ expect.any(Array),+ );+ await expect(store.getJob(streamId)).resolves.not.toHaveProperty('terminalHostActionPending');+ now.mockRestore();+ });+});diff --git a/packages/api/src/stream/implementations/InMemoryJobStore.ts b/packages/api/src/stream/implementations/InMemoryJobStore.tsindex 120048f613..2f186e8e7d 100644--- a/packages/api/src/stream/implementations/InMemoryJobStore.ts+++ b/packages/api/src/stream/implementations/InMemoryJobStore.ts@@ -30,6 +30,7 @@ import {STEER_QUEUE_MAX_DEPTH,PAUSE_PERSISTENCE_TIMEOUT_ERROR,PAUSE_PERSISTENCE_TIMEOUT_MS,+ PROVIDER_DRAIN_TIMEOUT_MS,isPendingActionStale,toWireRunSteps,} from '~/stream/interfaces/IJobStore';@@ -351,10 +352,11 @@ export class InMemoryJobStore implements IJobStoreV2 {const current = this.jobs.get(streamId);const currentCreatedAt = current?.createdAt ?? this.getRetainedGenerationEpoch(streamId);if (- rejectActivePredecessor === true &&- (current?.status === 'running' ||- current?.status === 'requires_action' ||- current?.terminalPersistencePending === true)+ current?.terminalHostActionPending === true ||+ (rejectActivePredecessor === true &&+ (current?.status === 'running' ||+ current?.status === 'requires_action' ||+ current?.terminalPersistencePending === true))) {throw new JobPredecessorMismatchError({createdAt: current.createdAt,@@ -1045,7 +1047,21 @@ export class InMemoryJobStore implements IJobStoreV2 {const pending: SerializableJobData[] = [];const now = Date.now();for (const job of this.jobs.values()) {- if (job.terminalHostActionPending === true) {+ if (job.terminalHostActionPending !== true) {+ continue;+ }+ if (+ job.providerDrained === false &&+ job.completedAt != null &&+ now - job.completedAt >= PROVIDER_DRAIN_TIMEOUT_MS+ ) {+ // No owner can renew this terminal segment. The pre-CAS snapshot is+ // already retained; force the same bounded recovery used by callers+ // waiting for a provider drain so one crashed process cannot hold the+ // conversation lane forever.+ job.providerDrained = true;+ }+ if (job.providerDrained !== false) {// Enumerating IS the retry attempt: refresh retention so evidence outlives a host// dependency that is unreachable for longer than the retention window.job.terminalHostActionRefreshedAt = now;@@ -1113,6 +1129,7 @@ export class InMemoryJobStore implements IJobStoreV2 {patch: {completedAt: now,error: PAUSE_PERSISTENCE_TIMEOUT_ERROR,+ ...(job.agentEventDeliveryKey != null && { terminalHostActionPending: true }),},clear: ['pendingAction',diff --git a/packages/api/src/stream/implementations/RedisJobStore.ts b/packages/api/src/stream/implementations/RedisJobStore.tsindex 78956eb747..8a165595fc 100644--- a/packages/api/src/stream/implementations/RedisJobStore.ts+++ b/packages/api/src/stream/implementations/RedisJobStore.ts@@ -33,6 +33,7 @@ import {STEER_QUEUE_MAX_DEPTH,PAUSE_PERSISTENCE_TIMEOUT_ERROR,PAUSE_PERSISTENCE_TIMEOUT_MS,+ PROVIDER_DRAIN_TIMEOUT_MS,isPendingActionStale,toWireRunSteps,} from '~/stream/interfaces/IJobStore';@@ -357,6 +358,7 @@ const JOB_CREATE_LUA ='local replacedProviderExecutionId = redis.call("HGET", KEYS[1], "providerExecutionId") ' +'local replacedProviderDrained = redis.call("HGET", KEYS[1], "providerDrained") ' +'local replacedTerminalPersistencePending = redis.call("HGET", KEYS[1], "terminalPersistencePending") ' ++ 'local replacedTerminalHostActionPending = redis.call("HGET", KEYS[1], "terminalHostActionPending") ' +'local replacedProtocol = redis.call("HGET", KEYS[1], "generationProtocolVersion") ' +'local MAX_SAFE_EPOCH = 9007199254740991 ' +'local function isSafeEpoch(value) return type(value) == "number" and value >= 0 ' +@@ -379,10 +381,13 @@ const JOB_CREATE_LUA ='local observedConversationId = replacedConversationId ' +'local observedActive = previousJobExists == 1 and ' +'(replacedStatus == "running" or replacedStatus == "requires_action" ' +- 'or replacedTerminalPersistencePending == "1") ' ++ 'or replacedTerminalPersistencePending == "1" or replacedTerminalHostActionPending == "1") ' +'if retainedEpoch and (not previousCreatedAt or retainedEpoch > previousCreatedAt) then ' +'previousCreatedAt = retainedEpoch observedCreatedAt = retainedEpochRaw ' +'observedStatus = nil observedConversationId = nil observedActive = false end ' ++ 'if observedActive and replacedTerminalHostActionPending == "1" then ' ++ 'return { previousUserId or "", previousTenantId or "", "0", "predecessor_mismatch", ' ++ 'observedCreatedAt, observedStatus or "", observedConversationId or "", "1", "1" } end ' +'if ARGV[13] == "1" and observedActive then ' +'return { previousUserId or "", previousTenantId or "", "0", "predecessor_mismatch", ' +'observedCreatedAt, observedStatus or "", observedConversationId or "", "1", "1" } end ' +@@ -608,6 +613,17 @@ const PROVIDER_DRAIN_LUA ='if redis.call("HGET", KEYS[1], "providerExecutionId") ~= ARGV[2] then return 0 end ' +'redis.call("HSET", KEYS[1], "providerDrained", "1") return 1';+/** Recover a terminal host action whose provider-owning process disappeared+ * after the terminal CAS. `completedAt` is immutable for this generation, so+ * the deadline cannot be extended by retry enumeration. */+const RECOVER_TERMINAL_PROVIDER_DRAIN_LUA =+ 'if redis.call("HGET", KEYS[1], "createdAt") ~= ARGV[1] then return 0 end ' ++ 'if redis.call("HGET", KEYS[1], "terminalHostActionPending") ~= "1" then return 0 end ' ++ 'if redis.call("HGET", KEYS[1], "providerDrained") ~= "0" then return 0 end ' ++ 'local completedAt = tonumber(redis.call("HGET", KEYS[1], "completedAt") or "") ' ++ 'if not completedAt or completedAt > tonumber(ARGV[2]) then return 0 end ' ++ 'redis.call("HSET", KEYS[1], "providerDrained", "1") return 1';+/** Exact initial provider-start fence. The controller rechecks account* deletion before this CAS; an abort/replacement that wins next prevents the* provider from starting after destructive cleanup has begun. */@@ -872,8 +888,9 @@ const CHUNK_APPEND_BATCH_LUA =* still-live approval after that window loses the tool/run-step timeline even though the* approval remains resumable. Reads the paused window from the job key (which* `transitionStatus` set); a normally-running job keeps the short running TTL. The write- * also requires an active status so a late provider event cannot recreate run steps after- * a same-epoch terminal transition deleted or retained the final timeline.+ * also requires either an active status or the exact terminal host-action marker. That+ * narrow terminal window lets a draining provider owner commit its final evidence before+ * the host callback acknowledges; after acknowledgement, late writes are fenced out.** KEYS: [runSteps, job]* ARGV: [runStepsJson, runningTtl, expectCreatedAt | ""]@@ -883,11 +900,12 @@ const RUNSTEPS_SAVE_LUA ='if not currentCreatedAt then return 0 end ' +'if ARGV[3] ~= "" and currentCreatedAt ~= ARGV[3] then return 0 end ' +'local currentStatus = redis.call("HGET", KEYS[2], "status") ' +- 'if currentStatus ~= "running" and currentStatus ~= "requires_action" then return 0 end ' ++ 'local terminalHostActionPending = redis.call("HGET", KEYS[2], "terminalHostActionPending") ' ++ 'if currentStatus ~= "running" and currentStatus ~= "requires_action" and terminalHostActionPending ~= "1" then return 0 end ' +'redis.call("SET", KEYS[1], ARGV[1]) ' +'local run = tonumber(ARGV[2]) ' +'local target = run ' +- 'if redis.call("HGET", KEYS[2], "status") == "requires_action" then ' ++ 'if currentStatus == "requires_action" or terminalHostActionPending == "1" then ' +'local jt = redis.call("TTL", KEYS[2]) ' +'if jt > target then target = jt end ' +'end ' +@@ -1570,6 +1588,38 @@ const KEYS = {idempotency: (key: string) => `stream:idem:${key}`,};+interface TerminalHostActionMember {+ streamId: string;+ createdAt?: number;+}++/** The retry index is global (and therefore outside the per-stream Redis+ * Cluster slot), so its members must carry generation identity. Otherwise an+ * acknowledgement for generation A can remove generation B's pre-armed hint. */+function terminalHostActionMember(streamId: string, createdAt: number): string {+ return JSON.stringify([streamId, createdAt]);+}++function parseTerminalHostActionMember(member: string): TerminalHostActionMember {+ try {+ const parsed = JSON.parse(member) as unknown;+ if (+ Array.isArray(parsed) &&+ parsed.length === 2 &&+ typeof parsed[0] === 'string' &&+ typeof parsed[1] === 'number' &&+ Number.isSafeInteger(parsed[1]) &&+ parsed[1] >= 0+ ) {+ return { streamId: parsed[0], createdAt: parsed[1] };+ }+ } catch {+ // Older releases indexed the bare stream id. Enumeration migrates any+ // still-live legacy member to the generation-scoped representation.+ }+ return { streamId: member };+}+/*** Default TTL values in seconds.* Can be overridden via constructor options.@@ -2260,6 +2310,7 @@ export class RedisJobStore implements IJobStoreV2 {job && (statusKey != null || job.providerDrained === false)? KEYS.userJobs(job.userId, job.tenantId): null;+ const terminalMember = job == null ? null : terminalHostActionMember(streamId, job.createdAt);if (this.isCluster) {const operations: Promise<unknown>[] = [@@ -2269,12 +2320,16 @@ export class RedisJobStore implements IJobStoreV2 {statusKey === KEYS.requiresActionJobs? this.redis.sadd(KEYS.requiresActionJobs, streamId): this.redis.srem(KEYS.requiresActionJobs, streamId),- // Terminal host-action membership follows the durable hash field, not status, so- // an aborted approval-expiry job stays enumerable for hook retry until acked.- job?.terminalHostActionPending === true- ? this.redis.sadd(KEYS.terminalHostActionJobs, streamId)- : this.redis.srem(KEYS.terminalHostActionJobs, streamId),];+ if (job?.terminalHostActionPending === true) {+ operations.push(this.redis.sadd(KEYS.terminalHostActionJobs, terminalMember!));+ } else if (terminalMember == null) {+ operations.push(this.redis.srem(KEYS.terminalHostActionJobs, streamId));+ } else {+ operations.push(this.redis.srem(KEYS.terminalHostActionJobs, streamId, terminalMember));+ }+ // Terminal host-action membership follows the durable hash field, not status, so+ // an aborted approval-expiry job stays enumerable for hook retry until acked.for (const userJobsKey of observedUserKeys) {if (userJobsKey !== activeUserKey) {operations.push(this.redis.srem(userJobsKey, streamId));@@ -2306,7 +2361,10 @@ export class RedisJobStore implements IJobStoreV2 {pipeline.srem(KEYS.requiresActionJobs, streamId);}if (job?.terminalHostActionPending === true) {- pipeline.sadd(KEYS.terminalHostActionJobs, streamId);+ pipeline.sadd(KEYS.terminalHostActionJobs, terminalMember!);+ pipeline.srem(KEYS.terminalHostActionJobs, streamId);+ } else if (terminalMember != null) {+ pipeline.srem(KEYS.terminalHostActionJobs, streamId, terminalMember);} else {pipeline.srem(KEYS.terminalHostActionJobs, streamId);}@@ -2484,6 +2542,17 @@ export class RedisJobStore implements IJobStoreV2 {ttl = this.pauseTtlSeconds(patch?.pendingAction);}const terminalJob = terminal ? await this.getJob(streamId) : null;+ // Redis Cluster cannot atomically update the same-slot job hash and this+ // global retry index. Arm a generation-scoped retry hint before the+ // terminal CAS. A predecessor acknowledgement can remove only its own+ // member, regardless of how a successor's SADD and CAS interleave.+ const terminalMemberCreatedAt = expectCreatedAt ?? terminalJob?.createdAt;+ if (terminal && patch?.terminalHostActionPending === true && terminalMemberCreatedAt != null) {+ await this.redis.sadd(+ KEYS.terminalHostActionJobs,+ terminalHostActionMember(streamId, terminalMemberCreatedAt),+ );+ }// 1) Single-winner decision: an atomic CAS on the single-slot job hash.// Works identically on cluster and single-node, so two concurrent@@ -2506,8 +2575,14 @@ export class RedisJobStore implements IJobStoreV2 {expectCreatedAt != null ? String(expectCreatedAt) : '',String(ttl),terminal ? '1' : '0',- String(this.ttl.chunksAfterComplete),- String(this.ttl.runStepsAfterComplete),+ String(+ terminal && patch?.terminalHostActionPending === true ? ttl : this.ttl.chunksAfterComplete,+ ),+ String(+ terminal && patch?.terminalHostActionPending === true+ ? ttl+ : this.ttl.runStepsAfterComplete,+ ),String(this.parkedRecoveryTtlSeconds()),String(GENERATION_EPOCH_GRACE_TTL_S),String(args.steerReceiptTtlSeconds ?? 0),@@ -2732,21 +2807,71 @@ export class RedisJobStore implements IJobStoreV2 {}async getTerminalHostActionJobs(): Promise<SerializableJobData[]> {- const streamIds = await this.redis.smembers(KEYS.terminalHostActionJobs);- if (streamIds.length === 0) {+ const members = await this.redis.smembers(KEYS.terminalHostActionJobs);+ if (members.length === 0) {return [];}- const jobs = await Promise.all(streamIds.map((streamId) => this.getJob(streamId)));+ const indexed = members.map((member) => ({ member, ...parseTerminalHostActionMember(member) }));+ const jobs = await Promise.all(indexed.map(({ streamId }) => this.getJob(streamId)));// The durable hash field is the source of truth; a stale set entry (job reaped, or the- // marker already cleared) is filtered out and self-heals via reconcileJobMembership.+ // marker/generation already replaced) is filtered out. Bare legacy members+ // are migrated while they are still live.const stale: string[] = [];- const pending: SerializableJobData[] = [];- for (let i = 0; i < streamIds.length; i++) {- const job = jobs[i];- if (job != null && job.terminalHostActionPending === true) {- pending.push(job);+ const legacy: string[] = [];+ const migrate: string[] = [];+ const heldByGeneration = new Map<string, SerializableJobData>();+ const readyByGeneration = new Map<string, SerializableJobData>();+ const providerLossCutoff = Date.now() - PROVIDER_DRAIN_TIMEOUT_MS;+ for (let i = 0; i < indexed.length; i++) {+ const indexedMember = indexed[i];+ let job = jobs[i];+ if (+ job != null &&+ job.terminalHostActionPending === true &&+ (indexedMember.createdAt == null || indexedMember.createdAt === job.createdAt)+ ) {+ if (indexedMember.createdAt == null) {+ legacy.push(indexedMember.member);+ migrate.push(terminalHostActionMember(job.streamId, job.createdAt));+ }+ if (+ job.providerDrained === false &&+ job.completedAt != null &&+ job.completedAt <= providerLossCutoff+ ) {+ const recovered =+ Number(+ await this.redis.eval(+ RECOVER_TERMINAL_PROVIDER_DRAIN_LUA,+ 1,+ KEYS.job(job.streamId),+ String(job.createdAt),+ String(providerLossCutoff),+ ),+ ) === 1;+ if (recovered) {+ job = { ...job, providerDrained: true };+ }+ }+ const generationKey = terminalHostActionMember(job.streamId, job.createdAt);+ heldByGeneration.set(generationKey, job);+ // A terminal provider can still be committing its last tool result.+ // The provider owner persists the complete run-step snapshot before it+ // flips this fence, so another replica must not settle earlier.+ if (job.providerDrained !== false) {+ readyByGeneration.set(generationKey, job);+ }} else {- stale.push(streamIds[i]);+ stale.push(indexedMember.member);+ }+ }+ if (migrate.length > 0) {+ try {+ await this.redis.sadd(KEYS.terminalHostActionJobs, ...migrate);+ await this.redis.srem(KEYS.terminalHostActionJobs, ...legacy);+ } catch {+ // Preserve the legacy hint if migration cannot prove the replacement+ // member was written. A duplicate hint is safer than lost discovery.}}if (stale.length > 0) {@@ -2755,32 +2880,44 @@ export class RedisJobStore implements IJobStoreV2 {// Enumerating IS the retry attempt: extend each pending job's TTL so unacknowledged// host-action evidence outlives a host dependency (e.g. Mongo) that stays unreachable// longer than the retention window. A deployment that stops sweeping lets it age out.- if (pending.length > 0 && this.ttl.requiresAction > 0) {+ const held = [...heldByGeneration.values()];+ if (held.length > 0 && this.ttl.requiresAction > 0) {await Promise.all(- pending.map((job) =>- this.redis.expire(KEYS.job(job.streamId), this.ttl.requiresAction).catch(() => undefined),+ held.flatMap((job) =>+ [KEYS.job(job.streamId), KEYS.chunks(job.streamId), KEYS.runSteps(job.streamId)].map(+ (key) => this.redis.expire(key, this.ttl.requiresAction).catch(() => undefined),+ ),),);}- return pending;+ return [...readyByGeneration.values()];}async clearTerminalHostAction(streamId: string, expectedCreatedAt?: number): Promise<void> {// Identity-fenced: only clear when the hash still holds this exact generation, so a// replacement at the same streamId is never cleared through its predecessor. The HDEL- // and the completed-TTL reset happen atomically; membership is then reconciled to SREM.+ // and configured evidence-TTL reset happen atomically. The global retry+ // member includes this generation, so removing it cannot affect a successor.const cleared = (await this.redis.eval('if redis.call("HGET", KEYS[1], "createdAt") ~= ARGV[1] then return 0 end ' +'redis.call("HDEL", KEYS[1], "terminalHostActionPending") ' +- 'if tonumber(ARGV[2]) > 0 then redis.call("EXPIRE", KEYS[1], ARGV[2]) end ' ++ 'if tonumber(ARGV[2]) > 0 then redis.call("EXPIRE", KEYS[1], ARGV[2]) else redis.call("DEL", KEYS[1]) end ' ++ 'if tonumber(ARGV[3]) > 0 then redis.call("EXPIRE", KEYS[2], ARGV[3]) else redis.call("DEL", KEYS[2]) end ' ++ 'if tonumber(ARGV[4]) > 0 then redis.call("EXPIRE", KEYS[3], ARGV[4]) else redis.call("DEL", KEYS[3]) end ' +'return 1',- 1,+ 3,KEYS.job(streamId),+ KEYS.chunks(streamId),+ KEYS.runSteps(streamId),expectedCreatedAt != null ? String(expectedCreatedAt) : '',String(this.ttl.completed),+ String(this.ttl.chunksAfterComplete),+ String(this.ttl.runStepsAfterComplete),)) as number;- if (cleared === 1) {- await this.redis.srem(KEYS.terminalHostActionJobs, streamId).catch(() => undefined);+ if (cleared === 1 && expectedCreatedAt != null) {+ await this.redis+ .srem(KEYS.terminalHostActionJobs, terminalHostActionMember(streamId, expectedCreatedAt))+ .catch(() => undefined);}}@@ -2919,6 +3056,7 @@ export class RedisJobStore implements IJobStoreV2 {patch: {completedAt: now,error: PAUSE_PERSISTENCE_TIMEOUT_ERROR,+ ...(job.agentEventDeliveryKey != null && { terminalHostActionPending: true }),},clear: ['pendingAction',@@ -4590,6 +4728,10 @@ export class RedisJobStore implements IJobStoreV2 {promptTokens: data.promptTokens ? parseInt(data.promptTokens, 10) : undefined,agent_id: data.agent_id || undefined,isTemporary: data.isTemporary != null ? data.isTemporary === '1' : undefined,+ agentEventDeliveryKey: data.agentEventDeliveryKey || undefined,+ agentEventExpectedAction: data.agentEventExpectedAction+ ? JSON.parse(data.agentEventExpectedAction)+ : undefined,scheduleId: data.scheduleId || undefined,scheduledFor: data.scheduledFor || undefined,scheduleConfigRevision: data.scheduleConfigRevisiondiff --git a/packages/api/src/stream/interfaces/IJobStore.ts b/packages/api/src/stream/interfaces/IJobStore.tsindex 59090aa887..7970274508 100644--- a/packages/api/src/stream/interfaces/IJobStore.ts+++ b/packages/api/src/stream/interfaces/IJobStore.ts@@ -69,6 +69,11 @@ export function toWireRunSteps(steps: readonly RunStep[]): Agents.RunStep[] {*/export const PAUSE_PERSISTENCE_TIMEOUT_MS = 30_000;export const PAUSE_PERSISTENCE_TIMEOUT_ERROR = 'Paused response persistence timed out';+/** Maximum time a terminal provider owner may remain undrained before its+ * process is treated as lost. Terminal host settlement retains the last+ * durable evidence through this grace period, then releases the lane from a+ * crashed owner instead of refreshing its fence forever. */+export const PROVIDER_DRAIN_TIMEOUT_MS = 30_000;/*** Job status enum.@@ -273,6 +278,8 @@ export interface SerializableJobData {* resume request can't be trusted to re-send the flag.*/isTemporary?: boolean;+ agentEventDeliveryKey?: string;+ agentEventExpectedAction?: import('~/agents/triggers/types').AgentTriggerExpectedAction;/*** Set when status is `requires_action`. Describes the human review the@@ -390,6 +397,8 @@ export type JobMetadataPatch = Partial<| 'model'| 'agent_id'| 'isTemporary'+ | 'agentEventDeliveryKey'+ | 'agentEventExpectedAction'| 'scheduleId'| 'scheduledFor'| 'scheduleConfigRevision'diff --git a/packages/api/src/stream/metadata.ts b/packages/api/src/stream/metadata.tsindex 259af3b7ce..61a34cbfe1 100644--- a/packages/api/src/stream/metadata.ts+++ b/packages/api/src/stream/metadata.ts@@ -36,6 +36,12 @@ export function sanitizeJobMetadata(metadata: Partial<GenerationJobMetadata>): Jif (metadata.isTemporary !== undefined) {patch.isTemporary = metadata.isTemporary;}+ if (metadata.agentEventDeliveryKey) {+ patch.agentEventDeliveryKey = metadata.agentEventDeliveryKey;+ }+ if (metadata.agentEventExpectedAction) {+ patch.agentEventExpectedAction = metadata.agentEventExpectedAction;+ }if (metadata.scheduleId) {patch.scheduleId = metadata.scheduleId;}diff --git a/packages/api/src/types/stream.ts b/packages/api/src/types/stream.tsindex 8f5423ce9d..3d488da5a6 100644--- a/packages/api/src/types/stream.ts+++ b/packages/api/src/types/stream.ts@@ -1,6 +1,7 @@import type { Agents, UserSubmittedMessageFieldPath } from 'librechat-data-provider';import type { EventEmitter } from 'events';import type { ActivityPhaseSnapshot } from '~/agents/activityPhases/runtime';+import type { AgentTriggerExpectedAction } from '../agents/triggers/types';import type { ResolvedAskUserQuestion } from '../agents/hitl/resume';import type { MCPRuntimeRequestBody } from '../mcp/types';import type { ServerSentEvent } from './events';@@ -41,6 +42,10 @@ export interface GenerationJobMetadata {agent_id?: string;/** Whether the originating turn was a temporary chat; a HITL resume keeps it so. */isTemporary?: boolean;+ /** Exact durable delivery whose accepted continuation created this generation. */+ agentEventDeliveryKey?: string;+ /** Optional action evidence contract declared by the authenticated event source. */+ agentEventExpectedAction?: AgentTriggerExpectedAction;/** Trusted scheduled-occurrence identity. These fields are accepted only from a* verified agent-trigger request and let pause/resume/reconciliation keep the* occurrence attached to the exact generation that owns it. */diff --git a/packages/data-schemas/src/methods/triggerDelivery.spec.ts b/packages/data-schemas/src/methods/triggerDelivery.spec.tsindex 8389f5050b..2203430d4e 100644--- a/packages/data-schemas/src/methods/triggerDelivery.spec.ts+++ b/packages/data-schemas/src/methods/triggerDelivery.spec.ts@@ -228,6 +228,8 @@ describe('agent trigger delivery methods', () => {availableAt: coalesceUntil,envelopeBytes: 128,envelope: {+ mode: 'continue',+ target: { bindingId: 'binding-1' },event: {id: `event-${index}`,source: { id: 'source-key', type: 'remote_api_key' },@@ -272,7 +274,13 @@ describe('agent trigger delivery methods', () => {claimToken: claim!.claimToken,now: coalesceUntil,});- const result = { mode: 'continue', status: 'started', conversationId: 'child-thread' };+ const result = {+ mode: 'continue',+ status: 'started',+ conversationId: 'child-thread',+ streamId: 'child-thread',+ generationCreatedAt: 1_787_000_000_000,+ };await expect(methods.completeAgentTriggerDelivery({id: claim!.id,@@ -281,6 +289,13 @@ describe('agent trigger delivery methods', () => {attempt: attempt!,result,settledAt: coalesceUntil,+ handling: {+ status: 'started',+ conversationId: 'child-thread',+ streamId: 'child-thread',+ generationCreatedAt: 1_787_000_000_000,+ startedAt: coalesceUntil,+ },}),).resolves.toBe(true);@@ -296,6 +311,34 @@ describe('agent trigger delivery methods', () => {);expect(receipts).toHaveLength(4);expect(receipts.every((receipt) => receipt?.status === 'succeeded')).toBe(true);+ expect(receipts.every((receipt) => receipt?.handling?.status === 'started')).toBe(true);++ await expect(+ methods.settleAgentTriggerHandlingOutcome({+ deliveryKey: root!.deliveryKey,+ conversationId: 'child-thread',+ generationCreatedAt: 1_787_000_000_000,+ status: 'completed_no_action',+ settledAt: new Date(coalesceUntil.getTime() + 1_000),+ }),+ ).resolves.toBe(true);++ let terminal = await Delivery.find({ orderingKey: 'commentary-lane' }).lean();+ expect(terminal.every((row) => row.handling?.status === 'completed_no_action')).toBe(true);++ const recoveringMember = terminal.find((row) => String(row._id) !== String(root!._id));+ await Delivery.updateOne({ _id: recoveringMember!._id }, { $unset: { handling: 1 } });+ await expect(+ methods.settleAgentTriggerHandlingOutcome({+ deliveryKey: root!.deliveryKey,+ conversationId: 'child-thread',+ generationCreatedAt: 1_787_000_000_000,+ status: 'completed_no_action',+ settledAt: new Date(coalesceUntil.getTime() + 1_000),+ }),+ ).resolves.toBe(true);+ terminal = await Delivery.find({ orderingKey: 'commentary-lane' }).lean();+ expect(terminal.every((row) => row.handling?.status === 'completed_no_action')).toBe(true);});it('recovers batch receipts and lane cleanup after root settlement was interrupted', async () => {@@ -1179,6 +1222,158 @@ describe('agent trigger delivery methods', () => {);});+ it('projects a bound continuation as started after generation admission', async () => {+ const user = new mongoose.Types.ObjectId();+ const queued = await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ user,+ envelope: {+ mode: 'continue',+ target: { bindingId: 'binding-1' },+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },+ },+ }),+ );+ const claimed = await methods.claimNextAgentTriggerDelivery({+ workerId: 'worker-1',+ claimToken: 'claim-1',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ });+ const attempt = await methods.beginAgentTriggerDeliveryAttempt({+ id: claimed!.id,+ workerId: 'worker-1',+ claimToken: 'claim-1',+ now: START,+ });++ await methods.completeAgentTriggerDelivery({+ id: claimed!.id,+ workerId: 'worker-1',+ claimToken: 'claim-1',+ attempt: attempt!,+ result: {+ mode: 'continue',+ status: 'started',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ },+ settledAt: new Date(START.getTime() + 1_000),+ handling: {+ status: 'started',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ startedAt: new Date(START.getTime() + 1_000),+ },+ });++ await expect(+ methods.getAgentTriggerDeliveryStatus(+ queued.delivery.deliveryKey,+ user,+ 'source-key-1',+ 'tenant-1',+ ),+ ).resolves.toMatchObject({+ status: 'succeeded',+ handling: {+ status: 'started',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ startedAt: new Date(START.getTime() + 1_000),+ },+ });+ });++ it('settles only the exact started generation and rejects stale terminal callbacks', async () => {+ const queued = await methods.enqueueAgentTriggerDelivery(+ enqueueInput({+ envelope: {+ mode: 'continue',+ target: { bindingId: 'binding-1' },+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },+ },+ }),+ );+ const claimed = await methods.claimNextAgentTriggerDelivery({+ workerId: 'worker-1',+ claimToken: 'claim-1',+ now: START,+ leaseUntil: new Date(START.getTime() + 60_000),+ });+ const attempt = await methods.beginAgentTriggerDeliveryAttempt({+ id: claimed!.id,+ workerId: 'worker-1',+ claimToken: 'claim-1',+ now: START,+ });+ await methods.completeAgentTriggerDelivery({+ id: claimed!.id,+ workerId: 'worker-1',+ claimToken: 'claim-1',+ attempt: attempt!,+ result: {+ mode: 'continue',+ status: 'started',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ },+ settledAt: new Date(START.getTime() + 1_000),+ handling: {+ status: 'started',+ conversationId: 'conversation-1',+ streamId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ startedAt: new Date(START.getTime() + 1_000),+ },+ });++ await expect(+ methods.settleAgentTriggerHandlingOutcome({+ deliveryKey: queued.delivery.deliveryKey,+ conversationId: 'conversation-1',+ generationCreatedAt: 1_786_999_999_999,+ status: 'failed',+ settledAt: new Date(START.getTime() + 2_000),+ error: 'stale failure',+ }),+ ).resolves.toBe(false);+ await expect(+ methods.settleAgentTriggerHandlingOutcome({+ deliveryKey: queued.delivery.deliveryKey,+ conversationId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ status: 'failed',+ settledAt: new Date(START.getTime() + 3_000),+ error: 'provider rejected the model',+ }),+ ).resolves.toBe(true);+ await expect(+ methods.settleAgentTriggerHandlingOutcome({+ deliveryKey: queued.delivery.deliveryKey,+ conversationId: 'conversation-1',+ generationCreatedAt: 1_787_000_000_000,+ status: 'applied',+ settledAt: new Date(START.getTime() + 4_000),+ action: { toolName: 'submit_move' },+ }),+ ).resolves.toBe(false);++ await expect(+ methods.getAgentTriggerDelivery(queued.delivery.deliveryKey),+ ).resolves.toMatchObject({+ handling: {+ status: 'failed',+ settledAt: new Date(START.getTime() + 3_000),+ error: 'provider rejected the model',+ },+ });+ });+it('reclaims an inactive high-cardinality lane after its tail succeeds', async () => {const orderingKey = 'one-shot-resource-lane';await methods.enqueueAgentTriggerDelivery(enqueueInput({ orderingKey }));diff --git a/packages/data-schemas/src/methods/triggerDelivery.ts b/packages/data-schemas/src/methods/triggerDelivery.tsindex db6b7e5d09..58cb662567 100644--- a/packages/data-schemas/src/methods/triggerDelivery.ts+++ b/packages/data-schemas/src/methods/triggerDelivery.ts@@ -2,6 +2,7 @@ import type { Model, Types } from 'mongoose';import type {AgentTriggerDeliveryClaim,AgentTriggerDeliveryFailure,+ AgentTriggerHandlingState,AgentTriggerDeliveryRecord,AgentTriggerDeliveryStatusRecord,AgentTriggerOrderingBlock,@@ -56,6 +57,16 @@ export interface AgentTriggerDeliveryFence {claimToken: string;}+export interface SettleAgentTriggerHandlingOutcomeInput {+ deliveryKey: string;+ conversationId: string;+ generationCreatedAt: number;+ status: Exclude<AgentTriggerHandlingState['status'], 'started'>;+ settledAt: Date;+ error?: string;+ action?: AgentTriggerHandlingState['action'];+}+export interface AgentTriggerDeliveryMethods {ensureAgentTriggerDeliveryIndexes: () => Promise<void>;enqueueAgentTriggerDelivery: (@@ -87,8 +98,12 @@ export interface AgentTriggerDeliveryMethods {attempt: number;result: unknown;settledAt: Date;+ handling?: AgentTriggerHandlingState;},) => Promise<boolean>;+ settleAgentTriggerHandlingOutcome: (+ input: SettleAgentTriggerHandlingOutcomeInput,+ ) => Promise<boolean>;retryAgentTriggerDelivery: (input: AgentTriggerDeliveryFence & {attempt: number;@@ -181,6 +196,7 @@ function toRecord(delivery: IAgentTriggerDelivery): AgentTriggerDeliveryRecord {...(delivery.batchMembersSettledAt != null && {batchMembersSettledAt: delivery.batchMembersSettledAt,}),+ ...(delivery.handling != null && { handling: delivery.handling }),};}@@ -796,10 +812,38 @@ export function createAgentTriggerDeliveryMethods(});}+ async function propagateBatchHandling(+ root: Pick<IAgentTriggerDelivery, '_id' | 'orderingKey' | 'batchMemberIds' | 'handling'>,+ ): Promise<void> {+ if (root._id == null || root.handling == null || !root.batchMemberIds?.length) {+ return;+ }+ const status = root.handling.status;+ await Delivery().updateMany(+ {+ _id: { $in: root.batchMemberIds },+ orderingKey: root.orderingKey,+ $or:+ status === 'started'+ ? [{ 'handling.status': 'started' }, { 'handling.status': { $exists: false } }]+ : [+ { 'handling.status': { $in: ['started', status] } },+ { 'handling.status': { $exists: false } },+ ],+ },+ { $set: { handling: root.handling } },+ );+ }+async function settleBatchMembers(root: Pick<IAgentTriggerDelivery,- '_id' | 'orderingKey' | 'batchMemberIds' | 'batchMembersSettledAt' | 'requeueCount'+ | '_id'+ | 'orderingKey'+ | 'batchMemberIds'+ | 'batchMembersSettledAt'+ | 'requeueCount'+ | 'handling'>,input: {attempt: number;@@ -826,6 +870,7 @@ export function createAgentTriggerDeliveryMethods(settledAt: input.settledAt,expiresAt: new Date(input.settledAt.getTime() + SUCCESS_RETENTION_MS),batchRootId: root._id,+ ...(root.handling != null && { handling: root.handling }),}: {status: input.status,@@ -877,6 +922,17 @@ export function createAgentTriggerDeliveryMethods(if (settledCount !== memberIds.length) {throw new Error('Not every agent trigger batch receipt could be settled');}+ if (input.status === 'succeeded' && root.handling != null) {+ const authoritative = await Delivery()+ .findById(root._id)+ .select('_id orderingKey batchMemberIds handling')+ .lean<+ Pick<IAgentTriggerDelivery, '_id' | 'orderingKey' | 'batchMemberIds' | 'handling'>+ >();+ if (authoritative != null) {+ await propagateBatchHandling(authoritative);+ }+ }}await Delivery().updateOne({ _id: root._id, status: input.status, batchMembersSettledAt: { $exists: false } },@@ -976,6 +1032,7 @@ export function createAgentTriggerDeliveryMethods(attempt: number;result: unknown;settledAt: Date;+ handling?: AgentTriggerHandlingState;},): Promise<boolean> {const completed = await Delivery()@@ -988,6 +1045,7 @@ export function createAgentTriggerDeliveryMethods(settledAt: input.settledAt,expiresAt: new Date(input.settledAt.getTime() + SUCCESS_RETENTION_MS),laneCleanupPendingAt: input.settledAt,+ ...(input.handling != null && { handling: input.handling }),},$unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1, lastError: 1 },$push: {@@ -1007,7 +1065,7 @@ export function createAgentTriggerDeliveryMethods({ new: true },).select(- '_id orderingKey laneCleanupPendingAt batchMemberIds batchMembersSettledAt requeueCount',+ '_id orderingKey laneCleanupPendingAt batchMemberIds batchMembersSettledAt requeueCount handling',).lean<Pick<@@ -1018,6 +1076,7 @@ export function createAgentTriggerDeliveryMethods(| 'batchMemberIds'| 'batchMembersSettledAt'| 'requeueCount'+ | 'handling'>>();if (completed?._id == null) {@@ -1044,6 +1103,61 @@ export function createAgentTriggerDeliveryMethods(return true;}+ async function settleAgentTriggerHandlingOutcome(+ input: SettleAgentTriggerHandlingOutcomeInput,+ ): Promise<boolean> {+ const error = input.error?.slice(0, MAX_ERROR_MESSAGE_LENGTH);+ const terminalHandling = {+ 'handling.status': input.status,+ 'handling.settledAt': input.settledAt,+ ...(error != null && { 'handling.error': error }),+ ...(input.action != null && { 'handling.action': input.action }),+ };+ const terminal = await Delivery()+ .findOneAndUpdate(+ {+ deliveryKey: input.deliveryKey,+ 'handling.status': 'started',+ 'handling.conversationId': input.conversationId,+ 'handling.generationCreatedAt': input.generationCreatedAt,+ },+ {+ $set: terminalHandling,+ $unset: {+ ...(error == null && { 'handling.error': 1 }),+ ...(input.action == null && { 'handling.action': 1 }),+ },+ },+ { new: true },+ )+ .select('_id orderingKey batchMemberIds handling')+ .lean<Pick<IAgentTriggerDelivery, '_id' | 'orderingKey' | 'batchMemberIds' | 'handling'>>();++ let authoritative = terminal;+ if (authoritative == null) {+ const existing = await Delivery()+ .findOne({+ deliveryKey: input.deliveryKey,+ 'handling.conversationId': input.conversationId,+ 'handling.generationCreatedAt': input.generationCreatedAt,+ })+ .select('_id orderingKey batchMemberIds handling')+ .lean<Pick<IAgentTriggerDelivery, '_id' | 'orderingKey' | 'batchMemberIds' | 'handling'>>();+ const replayed =+ existing?.handling?.status === input.status &&+ existing.handling.error === error &&+ existing.handling.action?.toolName === input.action?.toolName &&+ existing.handling.action?.toolCallId === input.action?.toolCallId;+ if (!replayed) {+ return false;+ }+ authoritative = existing;+ }++ await propagateBatchHandling(authoritative);+ return true;+ }+async function retryAgentTriggerDelivery(input: AgentTriggerDeliveryFence & {attempt: number;@@ -1154,7 +1268,9 @@ export function createAgentTriggerDeliveryMethods('envelope.event.source.type': 'remote_api_key',...tenantScope,})- .select('-_id deliveryKey status attempts availableAt createdAt settledAt result lastError')+ .select(+ '-_id deliveryKey status attempts availableAt createdAt settledAt result lastError handling',+ ).lean<AgentTriggerDeliveryStatusRecord>();if (delivery != null && delivery.status === 'batched') {const member = await Delivery()@@ -1172,11 +1288,17 @@ export function createAgentTriggerDeliveryMethods('envelope.event.source.type': 'remote_api_key',...tenantScope,})- .select('-_id status attempts availableAt settledAt result lastError')+ .select('-_id status attempts availableAt settledAt result lastError handling').lean<Pick<AgentTriggerDeliveryStatusRecord,- 'status' | 'attempts' | 'availableAt' | 'settledAt' | 'result' | 'lastError'+ | 'status'+ | 'attempts'+ | 'availableAt'+ | 'settledAt'+ | 'result'+ | 'lastError'+ | 'handling'>>();if (root == null) {@@ -1408,6 +1530,7 @@ export function createAgentTriggerDeliveryMethods(beginAgentTriggerDeliveryAttempt,deferAgentTriggerDeliveryAttempt,completeAgentTriggerDelivery,+ settleAgentTriggerHandlingOutcome,retryAgentTriggerDelivery,deadLetterAgentTriggerDelivery,getAgentTriggerDelivery,diff --git a/packages/data-schemas/src/schema/triggerDelivery.ts b/packages/data-schemas/src/schema/triggerDelivery.tsindex 3b6b567bac..7b1b1cdf9b 100644--- a/packages/data-schemas/src/schema/triggerDelivery.ts+++ b/packages/data-schemas/src/schema/triggerDelivery.ts@@ -24,6 +24,33 @@ const historySchema = new Schema({ _id: false },);+const handlingSchema = new Schema(+ {+ status: {+ type: String,+ enum: ['started', 'applied', 'completed_no_action', 'failed', 'cancelled'],+ required: true,+ },+ conversationId: { type: String, required: true, maxlength: 256 },+ streamId: { type: String, required: true, maxlength: 256 },+ generationCreatedAt: { type: Number, required: true, min: 0 },+ startedAt: { type: Date, required: true },+ settledAt: { type: Date },+ error: { type: String, maxlength: 2048 },+ action: {+ type: new Schema(+ {+ toolName: { type: String, required: true, maxlength: 256 },+ toolCallId: { type: String, maxlength: 256 },+ },+ { _id: false },+ ),+ required: false,+ },+ },+ { _id: false },+);+const triggerDeliverySchema: Schema<IAgentTriggerDeliveryDocument> = new Schema({deliveryKey: { type: String, required: true, maxlength: 128 },@@ -55,6 +82,7 @@ const triggerDeliverySchema: Schema<IAgentTriggerDeliveryDocument> = new Schema(batchRootId: { type: Schema.Types.ObjectId, ref: 'AgentTriggerDelivery' },batchRootRequeueCount: { type: Number, min: 0 },batchMembersSettledAt: { type: Date },+ handling: { type: handlingSchema },leaseBy: { type: String },leaseUntil: { type: Date },claimToken: { type: String },diff --git a/packages/data-schemas/src/types/triggerDelivery.ts b/packages/data-schemas/src/types/triggerDelivery.tsindex 11d46862c2..fb6b3c378c 100644--- a/packages/data-schemas/src/types/triggerDelivery.ts+++ b/packages/data-schemas/src/types/triggerDelivery.ts@@ -9,6 +9,20 @@ export type AgentTriggerDeliveryStatus =| 'dead';export type AgentTriggerDeliveryOutcome = 'succeeded' | 'retry' | 'dead';+export interface AgentTriggerHandlingState {+ status: 'started' | 'applied' | 'completed_no_action' | 'failed' | 'cancelled';+ conversationId: string;+ streamId: string;+ generationCreatedAt: number;+ startedAt: Date;+ settledAt?: Date;+ error?: string;+ action?: {+ toolName: string;+ toolCallId?: string;+ };+}+export interface AgentTriggerDeliveryFailure {code: string;message: string;@@ -49,6 +63,7 @@ export interface IAgentTriggerDelivery {batchRootId?: Types.ObjectId;batchRootRequeueCount?: number;batchMembersSettledAt?: Date;+ handling?: AgentTriggerHandlingState;leaseBy?: string;leaseUntil?: Date;claimToken?: string;@@ -87,6 +102,7 @@ export type AgentTriggerDeliveryStatusRecord = Pick<| 'settledAt'| 'result'| 'lastError'+ | 'handling'>;export interface IAgentTriggerLaneSequence {