Skip to content

Enhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active

🧾 feat: Store Durable Event Actor Receipts (#15265)

Danny Avila44edcbe01461
diff --git a/CONTEXT.md b/CONTEXT.md
index 78d4cd7ccf..ce2d0afb09 100644
--- a/CONTEXT.md
+++ b/CONTEXT.md
@@ -12,5 +12,6 @@
- **Agent event expected action**: an optional source-declared tool name and bounded argument subset evaluated against host-observed completed run steps. It is evidence policy, not authorization and not a model-authored success claim.
- **Event actor head**: the private, durable pointer on an event-bound child conversation to its latest committed LangGraph checkpoint, plus one previous checkpoint for safe cleanup. Only a qualifying applied action advances it through compare-and-swap; failed, cancelled, or no-action invocations leave it unchanged. A legacy-path event marks the head for a cold rebuild from durable message history before fork mode can resume. Every applied commit conflict, unverified commit, or post-commit persistence failure is retained in a private reconciliation journal that blocks later actor turns instead of continuing from stale state; an exact marker can be cleared only after its checkpoint is verified authoritative, its history is repaired, or its external action is explicitly compensated.
- **Event actor invocation fork**: a delivery-owned checkpoint namespace copied from the event actor head. A warm invocation receives only the new trusted event, then commits its terminal checkpoint when the expected action is observed or deletes the fork otherwise. Pause-capable actors remain on the existing resumable path until forked HITL has an explicit contract.
+- **Event actor receipt**: the private, terminal proof stored on the authoritative `AgentTriggerDelivery` row for one bound actor invocation. Its unique delivery identity, terminal resolution, exact checkpoint, and bounded action identity provide replay and recovery for the retention window without storing prompts, events, tool arguments, tool output, or conversation history. It does not own the actor checkpoint; the conversation keeps only the actor head and any active unresolved reconciliation until this receipt is durable.
- **Agent event actor mailbox**: the durable delivery-ordering lane for one authenticated source binding. When enabled after a fleet-wide rollout, it keeps later deliveries queued after transport admission until the current child turn records an authoritative terminal handling outcome. It serializes existing coalesced batches and individual events without becoming a second execution controller or actor checkpoint store.
- **Theme definition**: a versioned, data-only description of LibreChat semantic colors and shared appearance roles, optionally specialized by light or dark mode. The theme module validates and resolves partial definitions against bundled defaults before adapters apply them. A theme definition does not contain arbitrary CSS, application behavior, or alternate feature layouts.
diff --git a/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js b/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js
index 6de1902b44..4c9b309263 100644
--- a/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js
+++ b/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js
@@ -97,6 +97,11 @@ const mockBeginAgentEventActorLegacyTurn = jest.fn();
const mockCompleteAgentEventActorLegacyTurn = jest.fn();
const mockRecordAgentEventActorReconciliation = jest.fn();
const mockResolveAgentEventActorReconciliation = jest.fn();
+const mockClearAgentEventActorReconciliation = jest.fn();
+const mockAdmitAgentEventActorAction = jest.fn();
+const mockReleaseAgentEventActorAction = jest.fn();
+const mockHasAgentEventActorActionAdmission = jest.fn();
+const mockGetAgentEventActorReceipt = jest.fn();
const mockStartupTelemetry = {
mark: jest.fn(),
setStreamId: jest.fn(),
@@ -164,6 +169,7 @@ jest.mock('@librechat/data-schemas', () => ({
jest.mock('@librechat/api', () => ({
sendEvent: jest.fn(),
+ isEnabled: (value) => value === true || value === 'true' || value === '1',
isScheduleFireRequest: (...args) => mockIsScheduleFireRequest(...args),
exemptFromConcurrencyLimiter: (...args) => mockExemptFromConcurrencyLimiter(...args),
toPendingSteer: jest.fn((item) => item),
@@ -267,6 +273,11 @@ jest.mock('~/models', () => ({
mockRecordAgentEventActorReconciliation(...args),
resolveAgentEventActorReconciliation: (...args) =>
mockResolveAgentEventActorReconciliation(...args),
+ clearAgentEventActorReconciliation: (...args) => mockClearAgentEventActorReconciliation(...args),
+ admitAgentEventActorAction: (...args) => mockAdmitAgentEventActorAction(...args),
+ releaseAgentEventActorAction: (...args) => mockReleaseAgentEventActorAction(...args),
+ hasAgentEventActorActionAdmission: (...args) => mockHasAgentEventActorActionAdmission(...args),
+ getAgentEventActorReceipt: (...args) => mockGetAgentEventActorReceipt(...args),
isAgentTriggerPrincipalActive: (...args) => mockIsAgentTriggerPrincipalActive(...args),
isSubagentOwnerAdmissible: (...args) => mockIsSubagentOwnerAdmissible(...args),
}));
@@ -310,6 +321,7 @@ function nextTick() {
describe('ResumableAgentController resume metadata', () => {
beforeEach(() => {
jest.clearAllMocks();
+ process.env.ENABLE_AGENT_EVENT_DURABLE_RECEIPTS = 'true';
mockMCPContexts = new WeakMap();
mockCheckAndIncrementPendingRequest.mockResolvedValue({ allowed: true });
mockDecrementPendingRequest.mockResolvedValue(undefined);
@@ -388,6 +400,11 @@ describe('ResumableAgentController resume metadata', () => {
mockCompleteAgentEventActorLegacyTurn.mockResolvedValue(true);
mockRecordAgentEventActorReconciliation.mockResolvedValue(true);
mockResolveAgentEventActorReconciliation.mockResolvedValue(true);
+ mockClearAgentEventActorReconciliation.mockResolvedValue(true);
+ mockAdmitAgentEventActorAction.mockResolvedValue(true);
+ mockReleaseAgentEventActorAction.mockResolvedValue(true);
+ mockHasAgentEventActorActionAdmission.mockResolvedValue(false);
+ mockGetAgentEventActorReceipt.mockResolvedValue(null);
});
it.each([
@@ -3937,6 +3954,7 @@ describe('ResumableAgentController resume metadata', () => {
clientRequestId: 'req-event',
agentEventDelivery: {
deliveryKey: 'req-event',
+ target: { bindingId: 'binding-1' },
expectedAction: {
toolName: 'submit_move',
argumentSubset: { gameId: 'game-1', expectedPly: 7 },
@@ -4017,12 +4035,13 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-fork',
+ target: { bindingId: 'binding-1' },
event,
expectedAction: { toolName: 'submit_move', argumentSubset: { expectedPly: 8 } },
},
},
config: {
- endpoints: { agents: { eventDriven: { checkpointForks: true } } },
+ endpoints: { agents: { eventDriven: { checkpointForks: true, durableReceipts: true } } },
},
_isAgentTrigger: true,
_agentEventBindingParentConversationId: 'parent-conversation',
@@ -4052,6 +4071,11 @@ describe('ResumableAgentController resume metadata', () => {
commitState: expect.any(Function),
recordReconciliation: expect.any(Function),
resolveReconciliation: expect.any(Function),
+ clearReconciliation: expect.any(Function),
+ admitAction: expect.any(Function),
+ releaseAction: expect.any(Function),
+ hasActionAdmission: expect.any(Function),
+ getReceipt: expect.any(Function),
},
);
expect(mockExecuteAgentEventActor.mock.calls[0][0]).not.toHaveProperty('tenantId');
@@ -4093,6 +4117,7 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-receipt',
+ target: { bindingId: 'binding-1' },
event: {
id: 'game-1:ply-9',
type: 'chess.turn',
@@ -4103,7 +4128,7 @@ describe('ResumableAgentController resume metadata', () => {
},
},
config: {
- endpoints: { agents: { eventDriven: { checkpointForks: true } } },
+ endpoints: { agents: { eventDriven: { checkpointForks: true, durableReceipts: true } } },
},
_isAgentTrigger: true,
_agentEventBindingParentConversationId: 'parent-conversation',
@@ -4181,6 +4206,7 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-persistence',
+ target: { bindingId: 'binding-1' },
event: {
id: 'event-persistence',
type: 'test.event',
@@ -4191,7 +4217,9 @@ describe('ResumableAgentController resume metadata', () => {
expectedAction: { toolName: 'submit_move' },
},
},
- config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
+ config: {
+ endpoints: { agents: { eventDriven: { checkpointForks: true, durableReceipts: true } } },
+ },
_isAgentTrigger: true,
_agentEventBindingParentConversationId: 'parent-conversation',
_agentEventBindingParentAgentId: 'parent-agent',
@@ -4290,6 +4318,7 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-success',
+ target: { bindingId: 'binding-1' },
event: {
id: 'event-success',
type: 'test.event',
@@ -4300,7 +4329,9 @@ describe('ResumableAgentController resume metadata', () => {
expectedAction: { toolName: 'submit_move' },
},
},
- config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
+ config: {
+ endpoints: { agents: { eventDriven: { checkpointForks: true, durableReceipts: true } } },
+ },
_isAgentTrigger: true,
_agentEventBindingParentConversationId: 'parent-conversation',
_agentEventBindingParentAgentId: 'parent-agent',
@@ -4342,6 +4373,7 @@ describe('ResumableAgentController resume metadata', () => {
status: 'history_persisted',
checkpoint,
action: { toolName: 'submit_move', toolCallId: 'call-move' },
+ actionAdmitted: true,
observedAt: expect.any(Date),
},
});
@@ -4352,65 +4384,80 @@ describe('ResumableAgentController resume metadata', () => {
});
it.each([
+ [
+ 'durable-receipt rollout disabled',
+ { toolDefinitions: [] },
+ { eventDriven: { checkpointForks: true } },
+ undefined,
+ undefined,
+ false,
+ ],
[
'tool approval',
{ toolDefinitions: [] },
- { toolApproval: { enabled: true }, eventDriven: { checkpointForks: true } },
+ {
+ toolApproval: { enabled: true },
+ eventDriven: { checkpointForks: true, durableReceipts: true },
+ },
undefined,
undefined,
],
[
'primary ask_user_question',
{ toolDefinitions: [{ name: 'ask_user_question' }] },
- { eventDriven: { checkpointForks: true } },
+ { eventDriven: { checkpointForks: true, durableReceipts: true } },
undefined,
undefined,
],
[
'added-agent ask_user_question',
{ toolDefinitions: [] },
- { eventDriven: { checkpointForks: true } },
+ { eventDriven: { checkpointForks: true, durableReceipts: true } },
new Map([['added-agent', { toolDefinitions: [{ name: 'ask_user_question' }] }]]),
undefined,
],
[
'memory-checkpointer',
{ toolDefinitions: [] },
- { eventDriven: { checkpointForks: true }, checkpointer: { type: 'memory' } },
+ {
+ eventDriven: { checkpointForks: true, durableReceipts: true },
+ checkpointer: { type: 'memory' },
+ },
undefined,
undefined,
],
[
'always-apply skill prime',
{ toolDefinitions: [], alwaysApplySkillPrimes: [{ name: 'playbook', body: '# playbook' }] },
- { eventDriven: { checkpointForks: true } },
+ { eventDriven: { checkpointForks: true, durableReceipts: true } },
undefined,
undefined,
],
[
'manual skill prime',
{ toolDefinitions: [], manualSkillPrimes: [{ name: 'playbook', body: '# playbook' }] },
- { eventDriven: { checkpointForks: true } },
+ { eventDriven: { checkpointForks: true, durableReceipts: true } },
undefined,
undefined,
],
[
'skills-capable (history-derived re-priming)',
{ toolDefinitions: [] },
- { eventDriven: { checkpointForks: true } },
+ { eventDriven: { checkpointForks: true, durableReceipts: true } },
undefined,
{ primeInvokedSkills: async () => undefined },
],
[
'background-capable expected action',
{ toolDefinitions: [], backgroundToolNames: ['submit_move_mcp_chess'] },
- { eventDriven: { checkpointForks: true } },
+ { eventDriven: { checkpointForks: true, durableReceipts: true } },
undefined,
undefined,
],
])(
'keeps %s event actors on the existing resumable path',
- async (_label, agent, config, agentConfigs, clientOptions) => {
+ async (_label, agent, config, agentConfigs, clientOptions, durableReceiptsEnabled = true) => {
+ process.env.ENABLE_AGENT_EVENT_DURABLE_RECEIPTS = String(durableReceiptsEnabled);
mockGenerationJobManager.claimGeneration.mockResolvedValue(
wonGenerationClaim({
streamId: 'child-conversation',
@@ -4438,6 +4485,7 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-hitl',
+ target: { bindingId: 'binding-1' },
expectedAction: { toolName: 'submit_move' },
event: {
id: 'event-hitl',
@@ -4529,6 +4577,7 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-metadata-failure',
+ target: { bindingId: 'binding-1' },
expectedAction: { toolName: 'submit_move' },
event: {
id: 'event-metadata-failure',
@@ -4539,7 +4588,9 @@ describe('ResumableAgentController resume metadata', () => {
},
},
},
- config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
+ config: {
+ endpoints: { agents: { eventDriven: { checkpointForks: true, durableReceipts: true } } },
+ },
_isAgentTrigger: true,
_agentEventBindingParentConversationId: 'parent-conversation',
_agentEventBindingParentAgentId: 'parent-agent',
@@ -4600,6 +4651,7 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-fence-lost',
+ target: { bindingId: 'binding-1' },
expectedAction: { toolName: 'submit_move' },
event: {
id: 'event-fence-lost',
@@ -4610,7 +4662,9 @@ describe('ResumableAgentController resume metadata', () => {
},
},
},
- config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
+ config: {
+ endpoints: { agents: { eventDriven: { checkpointForks: true, durableReceipts: true } } },
+ },
_isAgentTrigger: true,
_agentEventBindingParentConversationId: 'parent-conversation',
_agentEventBindingParentAgentId: 'parent-agent',
@@ -4667,6 +4721,7 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-error-persistence',
+ target: { bindingId: 'binding-1' },
expectedAction: { toolName: 'submit_move' },
event: {
id: 'event-error-persistence',
@@ -4677,7 +4732,9 @@ describe('ResumableAgentController resume metadata', () => {
},
},
},
- config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
+ config: {
+ endpoints: { agents: { eventDriven: { checkpointForks: true, durableReceipts: true } } },
+ },
_isAgentTrigger: true,
_agentEventBindingParentConversationId: 'parent-conversation',
_agentEventBindingParentAgentId: 'parent-agent',
@@ -4745,6 +4802,7 @@ describe('ResumableAgentController resume metadata', () => {
endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
agentEventDelivery: {
deliveryKey: 'req-event-stale-legacy',
+ target: { bindingId: 'binding-1' },
expectedAction: { toolName: 'submit_move' },
event: {
id: 'event-stale-legacy',
@@ -4755,7 +4813,9 @@ describe('ResumableAgentController resume metadata', () => {
},
},
},
- config: { endpoints: { agents: { eventDriven: { checkpointForks: true } } } },
+ config: {
+ endpoints: { agents: { eventDriven: { checkpointForks: true, durableReceipts: true } } },
+ },
_isAgentTrigger: true,
_agentEventBindingParentConversationId: 'parent-conversation',
_agentEventBindingParentAgentId: 'parent-agent',
diff --git a/api/server/controllers/agents/request.js b/api/server/controllers/agents/request.js
index e0945a5bf2..e20087d6ea 100644
--- a/api/server/controllers/agents/request.js
+++ b/api/server/controllers/agents/request.js
@@ -33,6 +33,7 @@ const {
executeAgentEventActor,
findAgentEventAppliedAction,
createAgentEventActionRecorder,
+ isEnabled,
isHITLEnabled,
agentRequestsAskUserQuestion,
} = require('@librechat/api');
@@ -54,6 +55,11 @@ const {
completeAgentEventActorLegacyTurn,
recordAgentEventActorReconciliation,
resolveAgentEventActorReconciliation,
+ clearAgentEventActorReconciliation,
+ admitAgentEventActorAction,
+ releaseAgentEventActorAction,
+ hasAgentEventActorActionAdmission,
+ getAgentEventActorReceipt,
isAgentTriggerPrincipalActive,
isSubagentOwnerAdmissible,
} = require('~/models');
@@ -1312,6 +1318,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
isTemporary: req._agentEventBindingRetention?.isTemporary ?? req.body?.isTemporary,
...(agentEventDelivery != null && {
agentEventDeliveryKey: agentEventDelivery.deliveryKey,
+ agentEventBindingId: agentEventDelivery.target.bindingId,
...(agentEventDelivery.expectedAction != null && {
agentEventExpectedAction: agentEventDelivery.expectedAction,
}),
@@ -1836,6 +1843,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
...(eventActorTenantId == null ? {} : { tenantId: eventActorTenantId }),
reconciliation: {
invocationId: appliedEventActor.invocationId,
+ ...(appliedEventActor.actionAdmitted === true && { actionAdmitted: true }),
status: 'persistence_failed',
checkpoint: appliedEventActor.checkpoint,
action: appliedEventActor.action,
@@ -1945,6 +1953,13 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
const checkpointForksEnabled =
req.config?.endpoints?.[EModelEndpoint.agents]?.eventDriven?.checkpointForks === true;
+ /** This is a deployment barrier, not a principal-scoped feature. It is
+ * populated from the base config before the listener starts; reading
+ * merged req.config here would let a user/role/tenant override bypass
+ * the mixed-version rollout fence. */
+ const durableActorReceiptsEnabled = isEnabled(
+ process.env.ENABLE_AGENT_EVENT_DURABLE_RECEIPTS,
+ );
const agentsConfig = req.config?.endpoints?.[EModelEndpoint.agents];
const eventActorAgents = [
client?.options?.agent,
@@ -1985,6 +2000,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
);
const canUseEventActorFork =
checkpointForksEnabled &&
+ durableActorReceiptsEnabled &&
agentsConfig?.checkpointer?.type !== 'memory' &&
!eventActorMayPause &&
!eventActorHasSkillPrimes &&
@@ -2011,6 +2027,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
user: userId,
...(eventActorTenantId == null ? {} : { tenantId: eventActorTenantId }),
conversationId,
+ bindingId: agentEventDelivery.target.bindingId,
invocationId: eventTaskId,
event: agentEventDelivery.event,
expectedAction: agentEventDelivery.expectedAction,
@@ -2036,11 +2053,17 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
commitState: commitAgentEventActorState,
recordReconciliation: recordAgentEventActorReconciliation,
resolveReconciliation: resolveAgentEventActorReconciliation,
+ admitAction: admitAgentEventActorAction,
+ releaseAction: releaseAgentEventActorAction,
+ hasActionAdmission: hasAgentEventActorActionAdmission,
+ getReceipt: getAgentEventActorReceipt,
+ clearReconciliation: clearAgentEventActorReconciliation,
},
).then(({ value, execution }) => {
if (execution.status === 'applied') {
appliedEventActor = {
invocationId: eventTaskId,
+ actionAdmitted: typeof admitAgentEventActorAction === 'function',
checkpoint: execution.head.checkpoint,
action: execution.result.action,
};
@@ -2420,6 +2443,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
...(eventActorTenantId == null ? {} : { tenantId: eventActorTenantId }),
reconciliation: {
invocationId: appliedEventActor.invocationId,
+ ...(appliedEventActor.actionAdmitted === true && { actionAdmitted: true }),
status: 'history_persisted',
checkpoint: appliedEventActor.checkpoint,
action: appliedEventActor.action,
diff --git a/api/server/experimental.js b/api/server/experimental.js
index 78d63b97f6..9264df4544 100644
--- a/api/server/experimental.js
+++ b/api/server/experimental.js
@@ -34,6 +34,7 @@ const {
setupGracefulShutdown,
configureMessageFilterRegexValidator,
configureFileConfigRegexEngine,
+ configureAgentEventRuntime,
GenerationJobManager,
createAgentEventTerminalHandler,
waitForKeyvRedisClient,
@@ -400,6 +401,7 @@ if (cluster.isMaster) {
// principal) still merges DB `__base__` overrides, which must not drive which hook
// modules load in every worker (matches api/server/index.js's baseOnly usage).
const baseAppConfig = await getAppConfig({ baseOnly: true });
+ configureAgentEventRuntime(baseAppConfig?.endpoints?.agents?.eventDriven);
const toolApproval = baseAppConfig?.endpoints?.agents?.toolApproval;
await loadToolApprovalHooks(toolApproval?.enabled ? toolApproval.hooks : undefined, {
basePath: path.resolve(__dirname, '../..'),
diff --git a/api/server/experimental.spec.js b/api/server/experimental.spec.js
index 58349d3091..365171c409 100644
--- a/api/server/experimental.spec.js
+++ b/api/server/experimental.spec.js
@@ -68,6 +68,20 @@ describe('Experimental server configuration', () => {
expect(listenIndex).toBeGreaterThan(routingIndex);
});
+ it('projects base-only event rollout barriers before accepting requests', () => {
+ const baseConfigIndex = source.indexOf(
+ 'const baseAppConfig = await getAppConfig({ baseOnly: true });',
+ );
+ const eventRuntimeIndex = source.indexOf(
+ 'configureAgentEventRuntime(baseAppConfig?.endpoints?.agents?.eventDriven);',
+ );
+ const listenIndex = source.indexOf('const server = app.listen');
+
+ expect(baseConfigIndex).toBeGreaterThan(-1);
+ expect(eventRuntimeIndex).toBeGreaterThan(baseConfigIndex);
+ expect(listenIndex).toBeGreaterThan(eventRuntimeIndex);
+ });
+
it('matches the standard server pre-authentication tenant routes', () => {
expect(source).toContain("app.use('/oauth', preAuthTenantMiddleware, routes.oauth);");
expect(source).toContain("app.use('/api/auth', preAuthTenantMiddleware, routes.auth);");
diff --git a/api/server/index.js b/api/server/index.js
index bfa7545fe3..05594e2661 100644
--- a/api/server/index.js
+++ b/api/server/index.js
@@ -148,7 +148,21 @@ const configureGenerationStreams = () => {
const startServer = async () => {
await waitForKeyvRedisClient();
await configureSubagentTaskRouting();
- const { metricsMiddleware, metricsRouter } = createMetrics();
+ const { metricsMiddleware, metricsRouter } = createMetrics({
+ collectAgentEventActorStorageMetrics: () =>
+ runAsSystem(async () => {
+ const now = new Date();
+ const [receiptMetrics, reconciliationMetrics] = await Promise.all([
+ agentEventMethods.getAgentEventActorReceiptStorageMetrics(now),
+ agentEventMethods.getAgentEventActorReconciliationStorageMetrics(now),
+ ]);
+ return {
+ ...receiptMetrics,
+ pendingReconciliations: reconciliationMetrics.pending,
+ oldestPendingAgeSeconds: reconciliationMetrics.oldestPendingAgeSeconds,
+ };
+ }),
+ });
if (!process.env.METRICS_SECRET) {
logger.warn('[metrics] METRICS_SECRET is not set - /metrics will return 401 for all requests');
}
diff --git a/packages/api/src/agents/triggers/README.md b/packages/api/src/agents/triggers/README.md
index 732c9eed5f..eb83d825f4 100644
--- a/packages/api/src/agents/triggers/README.md
+++ b/packages/api/src/agents/triggers/README.md
@@ -183,6 +183,16 @@ in parallel. Existing coalesced batches occupy one mailbox position and retain e
individual receipt. An active mailbox record does not receive its normal success TTL; the 90-day
retention window begins only after terminal handling is recorded.
+Set `endpoints.agents.eventDriven.durableReceipts: true` only after every API replica runs the
+token-fenced delivery receipt implementation and all pre-upgrade bound-actor deliveries have
+drained. It defaults to false so a rolling deployment cannot overlap token-unaware admission
+release with a newer token-owned action. Keep it false during the binary rollout, then enable it
+deployment-wide together with `checkpointForks` after the old workers and their deliveries are
+gone. LibreChat projects this base-only setting into
+`ENABLE_AGENT_EVENT_DURABLE_RECEIPTS` before accepting requests; the controller deliberately does
+not read the merged per-principal config, so tenant, role, and user overrides cannot bypass the
+barrier.
+
### Coalescing observational child events
Sources that can prove several bound `continue` events are interchangeable observations may opt
diff --git a/packages/api/src/agents/triggers/actor.spec.ts b/packages/api/src/agents/triggers/actor.spec.ts
index b8bd8b1fdb..f73c6a02e0 100644
--- a/packages/api/src/agents/triggers/actor.spec.ts
+++ b/packages/api/src/agents/triggers/actor.spec.ts
@@ -47,6 +47,7 @@ describe('event actor host adapter', () => {
checkpointNs,
checkpointId: `checkpoint-${nextCheckpoint++}`,
}));
+ mockedDelete.mockReset();
mockedDelete.mockResolvedValue();
});
@@ -79,6 +80,9 @@ describe('event actor host adapter', () => {
}),
recordReconciliation: jest.fn(async () => true),
resolveReconciliation: jest.fn(async () => true),
+ admitAction: jest.fn(async () => true),
+ releaseAction: jest.fn(async () => true),
+ hasActionAdmission: jest.fn(async () => false),
});
it('cold-starts once, then forks and warm-continues only the next event', async () => {
@@ -753,4 +757,319 @@ describe('event actor host adapter', () => {
).rejects.toThrow('Event actor binding is no longer active');
expect(mockedGetCheckpointer).not.toHaveBeenCalled();
});
+
+ it('replays a delivery receipt and cleans its stranded active marker without executing', async () => {
+ const checkpoint = {
+ threadId: conversationId,
+ checkpointId: 'checkpoint-terminal',
+ checkpointNs: 'event-actor/event-replay',
+ };
+ const invoke = jest.fn(async () => 'must not run');
+ const clearReconciliation = jest.fn(async () => true);
+ const dependencies = {
+ ...deps(),
+ getSnapshot: jest.fn(async () => ({
+ state: { generation: 1, checkpoint },
+ reconciliations: [
+ {
+ invocationId: 'event-replay',
+ status: 'history_persisted' as const,
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(),
+ },
+ ],
+ legacyTurn: null,
+ epoch: 0,
+ })),
+ getReceipt: jest.fn(async () => ({
+ bindingId: 'binding-1',
+ resolution: 'checkpoint_verified' as const,
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ settledAt: new Date(),
+ })),
+ clearReconciliation,
+ };
+
+ await expect(
+ executeAgentEventActor(
+ {
+ user: 'user-1',
+ conversationId,
+ bindingId: 'binding-1',
+ invocationId: 'event-replay',
+ event: { id: 'event-replay' },
+ signal: new AbortController().signal,
+ invoke,
+ readAppliedAction: () => undefined,
+ },
+ dependencies,
+ ),
+ ).rejects.toThrow('already has a terminal receipt');
+
+ expect(clearReconciliation).toHaveBeenCalledWith({
+ user: 'user-1',
+ conversationId,
+ invocationId: 'event-replay',
+ checkpoint,
+ resolution: 'checkpoint_verified',
+ });
+ expect(invoke).not.toHaveBeenCalled();
+ expect(dependencies.recordReconciliation).not.toHaveBeenCalled();
+ });
+
+ it('abandons the lifecycle when delivery-owned action admission already settled', async () => {
+ const invoke = jest.fn(async () => 'must not run');
+ const resolveReconciliation = jest.fn(async () => true);
+ const dependencies = {
+ ...deps(),
+ getReceipt: jest.fn().mockResolvedValue(null),
+ admitAction: jest.fn().mockResolvedValue(false),
+ resolveReconciliation,
+ };
+
+ await expect(
+ executeAgentEventActor(
+ {
+ user: 'user-1',
+ conversationId,
+ bindingId: 'binding-1',
+ invocationId: 'event-race',
+ event: { id: 'event-race' },
+ signal: new AbortController().signal,
+ invoke,
+ readAppliedAction: () => undefined,
+ },
+ dependencies,
+ ),
+ ).rejects.toThrow('action admission was already consumed or settled');
+
+ expect(dependencies.recordReconciliation).toHaveBeenCalledTimes(1);
+ expect(dependencies.admitAction).toHaveBeenCalledWith({
+ deliveryKey: 'event-race',
+ user: 'user-1',
+ bindingId: 'binding-1',
+ conversationId,
+ admittedAt: expect.any(Date),
+ admissionId: expect.any(String),
+ });
+ expect(resolveReconciliation).toHaveBeenNthCalledWith(1, {
+ user: 'user-1',
+ conversationId,
+ invocationId: 'event-race',
+ checkpoint: expect.objectContaining({ threadId: conversationId }),
+ expectedActionAdmitted: false,
+ resolution: 'invocation_abandoned',
+ });
+ expect(invoke).not.toHaveBeenCalled();
+ });
+
+ it('releases delivery admission when the actor completes without an external action', async () => {
+ const dependencies = {
+ ...deps(),
+ getReceipt: jest.fn().mockResolvedValue(null),
+ };
+
+ await expect(
+ executeAgentEventActor(
+ {
+ user: 'user-1',
+ conversationId,
+ bindingId: 'binding-1',
+ invocationId: 'event-no-action',
+ event: { id: 'event-no-action' },
+ signal: new AbortController().signal,
+ invoke: async () => 'no action',
+ readAppliedAction: () => undefined,
+ },
+ dependencies,
+ ),
+ ).resolves.toMatchObject({ execution: { status: 'completed_no_action' } });
+
+ expect(dependencies.releaseAction).toHaveBeenCalledWith({
+ deliveryKey: 'event-no-action',
+ user: 'user-1',
+ bindingId: 'binding-1',
+ conversationId,
+ admissionId: expect.any(String),
+ });
+ expect(dependencies.releaseAction.mock.invocationCallOrder[0]).toBeLessThan(
+ dependencies.resolveReconciliation.mock.invocationCallOrder[0],
+ );
+ });
+
+ it('recovers a no-action lifecycle after admission was released before owner exit', async () => {
+ const invoke = jest.fn(async () => 'retried without action');
+ const checkpoint = {
+ threadId: conversationId,
+ checkpointNs: 'event-actor/orphaned-no-action',
+ };
+ const dependencies = {
+ ...deps(),
+ getReceipt: jest.fn().mockResolvedValue(null),
+ };
+ dependencies.getSnapshot.mockResolvedValue({
+ state: null,
+ reconciliations: [
+ {
+ invocationId: 'event-orphaned-no-action',
+ actionAdmitted: true,
+ status: 'invocation_pending',
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(),
+ },
+ ],
+ legacyTurn: null,
+ epoch: 0,
+ });
+
+ await expect(
+ executeAgentEventActor(
+ {
+ user: 'user-1',
+ conversationId,
+ bindingId: 'binding-1',
+ invocationId: 'event-orphaned-no-action',
+ event: { id: 'event-orphaned-no-action' },
+ signal: new AbortController().signal,
+ invoke,
+ readAppliedAction: () => undefined,
+ },
+ dependencies,
+ ),
+ ).resolves.toMatchObject({ execution: { status: 'completed_no_action' } });
+
+ expect(dependencies.hasActionAdmission).toHaveBeenCalledWith({
+ deliveryKey: 'event-orphaned-no-action',
+ user: 'user-1',
+ bindingId: 'binding-1',
+ conversationId,
+ admissionId: expect.any(String),
+ });
+ expect(dependencies.resolveReconciliation).toHaveBeenNthCalledWith(1, {
+ user: 'user-1',
+ conversationId,
+ invocationId: 'event-orphaned-no-action',
+ checkpoint,
+ expectedActionAdmitted: true,
+ resolution: 'invocation_abandoned',
+ });
+ expect(invoke).toHaveBeenCalledTimes(1);
+ });
+
+ it('never invokes after a retry supersedes its pre-admission lifecycle', async () => {
+ const invoke = jest.fn(async () => 'must not run');
+ const dependencies = {
+ ...deps(),
+ recordReconciliation: jest.fn().mockResolvedValueOnce(true).mockResolvedValueOnce(false),
+ admitAction: jest.fn().mockResolvedValue(true),
+ releaseAction: jest.fn().mockResolvedValue(true),
+ getReceipt: jest.fn().mockResolvedValue(null),
+ };
+
+ await expect(
+ executeAgentEventActor(
+ {
+ user: 'user-1',
+ conversationId,
+ bindingId: 'binding-1',
+ invocationId: 'event-pre-admission-race',
+ event: { id: 'event-pre-admission-race' },
+ signal: new AbortController().signal,
+ invoke,
+ readAppliedAction: () => undefined,
+ },
+ dependencies,
+ ),
+ ).rejects.toThrow('admission lifecycle was superseded before invoke');
+
+ expect(dependencies.admitAction).toHaveBeenCalledTimes(1);
+ expect(dependencies.releaseAction).toHaveBeenCalledTimes(1);
+ expect(invoke).not.toHaveBeenCalled();
+ });
+
+ it('does not clear an admission it failed to acquire', async () => {
+ const invoke = jest.fn(async () => 'must not run');
+ const dependencies = {
+ ...deps(),
+ recordReconciliation: jest.fn().mockResolvedValue(true),
+ admitAction: jest.fn().mockResolvedValue(false),
+ getReceipt: jest.fn().mockResolvedValue(null),
+ };
+
+ await expect(
+ executeAgentEventActor(
+ {
+ user: 'user-1',
+ conversationId,
+ bindingId: 'binding-1',
+ invocationId: 'event-orphaned-admission',
+ event: { id: 'event-orphaned-admission' },
+ signal: new AbortController().signal,
+ invoke,
+ readAppliedAction: () => undefined,
+ },
+ dependencies,
+ ),
+ ).rejects.toThrow('action admission was already consumed or settled');
+
+ expect(dependencies.resolveReconciliation).toHaveBeenCalledWith(
+ expect.objectContaining({
+ invocationId: 'event-orphaned-admission',
+ expectedActionAdmitted: false,
+ resolution: 'invocation_abandoned',
+ }),
+ );
+ expect(dependencies.releaseAction).not.toHaveBeenCalled();
+ expect(invoke).not.toHaveBeenCalled();
+ });
+
+ it('does not release or replay while legacy terminal proof awaits migration', async () => {
+ const invoke = jest.fn(async () => 'must not run');
+ const dependencies = {
+ ...deps(),
+ getReceipt: jest.fn().mockResolvedValue(null),
+ };
+ dependencies.getSnapshot.mockResolvedValue({
+ state: null,
+ reconciliations: [
+ {
+ invocationId: 'event-legacy-terminal',
+ status: 'settled',
+ resolution: 'checkpoint_verified',
+ checkpoint: {
+ threadId: conversationId,
+ checkpointId: 'checkpoint-terminal',
+ checkpointNs: 'event-actor/terminal',
+ },
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(),
+ },
+ ],
+ legacyTurn: null,
+ epoch: 0,
+ });
+
+ await expect(
+ executeAgentEventActor(
+ {
+ user: 'user-1',
+ conversationId,
+ bindingId: 'binding-1',
+ invocationId: 'event-legacy-terminal',
+ event: { id: 'event-legacy-terminal' },
+ signal: new AbortController().signal,
+ invoke,
+ readAppliedAction: () => undefined,
+ },
+ dependencies,
+ ),
+ ).rejects.toThrow('legacy terminal proof awaiting migration');
+
+ expect(dependencies.admitAction).not.toHaveBeenCalled();
+ expect(dependencies.releaseAction).not.toHaveBeenCalled();
+ expect(invoke).not.toHaveBeenCalled();
+ });
});
diff --git a/packages/api/src/agents/triggers/actor.ts b/packages/api/src/agents/triggers/actor.ts
index 2b2325520e..25444f4f4f 100644
--- a/packages/api/src/agents/triggers/actor.ts
+++ b/packages/api/src/agents/triggers/actor.ts
@@ -1,3 +1,4 @@
+import { createHash } from 'node:crypto';
import { logger } from '@librechat/data-schemas';
import {
createEventActorExecutor,
@@ -6,7 +7,11 @@ import {
type EventActorHead,
type EventActorHostAdapter,
} from '@librechat/agents';
-import type { ConversationMethods, IAgentEventActorState } from '@librechat/data-schemas';
+import type {
+ AgentTriggerDeliveryMethods,
+ ConversationMethods,
+ IAgentEventActorState,
+} from '@librechat/data-schemas';
import type { TCheckpointerConfig } from 'librechat-data-provider';
import type { AgentTriggerExpectedAction } from './envelope';
import type { AgentEventAppliedAction } from './outcome';
@@ -35,6 +40,8 @@ export interface ExecuteAgentEventActorInput<T> {
user: string;
tenantId?: string;
conversationId: string;
+ /** Authenticated binding that owns the delivery receipt. */
+ bindingId?: string;
invocationId: string;
event: EventActorEvent;
expectedAction?: AgentTriggerExpectedAction;
@@ -56,6 +63,11 @@ export interface AgentEventActorDependencies {
commitState: ConversationMethods['commitAgentEventActorState'];
recordReconciliation: ConversationMethods['recordAgentEventActorReconciliation'];
resolveReconciliation: ConversationMethods['resolveAgentEventActorReconciliation'];
+ admitAction?: AgentTriggerDeliveryMethods['admitAgentEventActorAction'];
+ releaseAction?: AgentTriggerDeliveryMethods['releaseAgentEventActorAction'];
+ hasActionAdmission?: AgentTriggerDeliveryMethods['hasAgentEventActorActionAdmission'];
+ getReceipt?: AgentTriggerDeliveryMethods['getAgentEventActorReceipt'];
+ clearReconciliation?: ConversationMethods['clearAgentEventActorReconciliation'];
}
function toHead(actorThreadId: string, state: IAgentEventActorState | null): EventActorHead {
@@ -80,6 +92,21 @@ function checkpointMatches(
);
}
+function actionAdmissionId(
+ invocationId: string,
+ checkpoint: { threadId: string; checkpointId?: string; checkpointNs: string },
+): string {
+ return createHash('sha256')
+ .update(invocationId)
+ .update('\0')
+ .update(checkpoint.threadId)
+ .update('\0')
+ .update(checkpoint.checkpointNs)
+ .update('\0')
+ .update(checkpoint.checkpointId ?? '')
+ .digest('hex');
+}
+
/**
* Executes one authenticated bound-child event through the SDK's checkpoint-fork lifecycle.
* The request controller still owns generation admission and terminal receipts; this adapter owns
@@ -91,6 +118,7 @@ export async function executeAgentEventActor<T>(
): Promise<ExecuteAgentEventActorResult<T>> {
let value: T | undefined;
let invocationError: unknown;
+ let ownedActionAdmissionId: string | undefined;
let observedState: IAgentEventActorState | null | undefined;
let observedEpoch: number | undefined;
const adapter: EventActorHostAdapter<EventActorEvent, EventActorResult> = {
@@ -106,7 +134,96 @@ export async function executeAgentEventActor<T>(
if (snapshot === undefined) {
throw new Error('Event actor binding is no longer active');
}
- const unresolved = snapshot.reconciliations.filter((item) => item.status !== 'settled');
+ if (input.bindingId != null && deps.getReceipt != null) {
+ const receipt = await deps.getReceipt({
+ deliveryKey: input.invocationId,
+ user: input.user,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ bindingId: input.bindingId,
+ conversationId: input.conversationId,
+ });
+ if (receipt != null) {
+ const marker = snapshot.reconciliations.find(
+ (item) => item.invocationId === input.invocationId,
+ );
+ if (marker != null && deps.clearReconciliation != null) {
+ const cleared = await deps.clearReconciliation({
+ user: input.user,
+ conversationId: input.conversationId,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ invocationId: input.invocationId,
+ checkpoint: receipt.checkpoint,
+ resolution: receipt.resolution,
+ });
+ if (!cleared) {
+ throw new Error('Event actor terminal marker could not be recovered');
+ }
+ }
+ throw new Error('Event actor invocation already has a terminal receipt');
+ }
+ }
+ if (
+ snapshot.reconciliations.some(
+ (item) => item.invocationId === input.invocationId && item.status === 'settled',
+ )
+ ) {
+ /** Mixed-version proof must be migrated by the terminal handler before
+ * the same delivery identity can execute again. Delivery admission
+ * alone cannot distinguish that proof from a pre-invoke orphan. */
+ throw new Error('Event actor invocation has legacy terminal proof awaiting migration');
+ }
+ let recoveredInvocationId: string | undefined;
+ const pendingInvocation = snapshot.reconciliations.find(
+ (item) => item.invocationId === input.invocationId && item.status === 'invocation_pending',
+ );
+ if (pendingInvocation != null && input.bindingId != null && deps.hasActionAdmission != null) {
+ const pendingAdmissionId = actionAdmissionId(
+ input.invocationId,
+ pendingInvocation.checkpoint,
+ );
+ const actionAdmitted = await deps.hasActionAdmission({
+ deliveryKey: input.invocationId,
+ user: input.user,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ bindingId: input.bindingId,
+ conversationId: input.conversationId,
+ admissionId: pendingAdmissionId,
+ });
+ if (pendingInvocation.actionAdmitted !== true || !actionAdmitted) {
+ /** Abandoning the conversation marker fences a paused pre-admission
+ * owner: it must confirm this exact marker after winning delivery
+ * admission and therefore cannot invoke after takeover. */
+ const abandoned = await deps.resolveReconciliation({
+ user: input.user,
+ conversationId: input.conversationId,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ invocationId: input.invocationId,
+ checkpoint: pendingInvocation.checkpoint,
+ expectedActionAdmitted: pendingInvocation.actionAdmitted === true,
+ resolution: 'invocation_abandoned',
+ });
+ if (!abandoned) {
+ throw new Error('Event actor orphaned no-action lifecycle could not be recovered');
+ }
+ if (actionAdmitted) {
+ const released = await deps.releaseAction?.({
+ deliveryKey: input.invocationId,
+ user: input.user,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ bindingId: input.bindingId,
+ conversationId: input.conversationId,
+ admissionId: pendingAdmissionId,
+ });
+ if (!released) {
+ throw new Error('Event actor orphaned action admission could not be released');
+ }
+ }
+ recoveredInvocationId = input.invocationId;
+ }
+ }
+ const unresolved = snapshot.reconciliations.filter(
+ (item) => item.status !== 'settled' && item.invocationId !== recoveredInvocationId,
+ );
if (unresolved.length > 0) {
throw new Error(
`Event actor is blocked on ${unresolved.map((item) => item.status).join(', ')} reconciliation`,
@@ -188,6 +305,78 @@ export async function executeAgentEventActor<T>(
if (!fenced) {
throw new Error('Event actor invocation could not acquire its durable lifecycle fence');
}
+ /** The delivery row is the serialization point between action admission
+ * and terminal settlement. A plain receipt read cannot close the final
+ * read-before-invoke race across two Mongo documents. */
+ if (input.bindingId != null && deps.admitAction != null) {
+ const admissionId = actionAdmissionId(input.invocationId, invocation.fork);
+ const admitted = await deps.admitAction({
+ deliveryKey: input.invocationId,
+ user: input.user,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ bindingId: input.bindingId,
+ conversationId: input.conversationId,
+ admittedAt: new Date(),
+ admissionId,
+ });
+ if (!admitted) {
+ const abandoned = await deps.resolveReconciliation({
+ user: input.user,
+ conversationId: input.conversationId,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ invocationId: invocation.invocationId,
+ checkpoint: {
+ threadId: invocation.fork.threadId,
+ checkpointNs: invocation.fork.checkpointNs,
+ ...(invocation.fork.checkpointId == null
+ ? {}
+ : { checkpointId: invocation.fork.checkpointId }),
+ },
+ expectedActionAdmitted: false,
+ resolution: 'invocation_abandoned',
+ });
+ if (!abandoned) {
+ throw new Error('Event actor duplicate lifecycle fence could not be abandoned');
+ }
+ throw new Error('Event actor action admission was already consumed or settled');
+ }
+ const confirmed = await deps.recordReconciliation({
+ user: input.user,
+ conversationId: input.conversationId,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ reconciliation: {
+ invocationId: invocation.invocationId,
+ actionAdmitted: true,
+ status: 'invocation_pending',
+ checkpoint: {
+ threadId: invocation.fork.threadId,
+ checkpointNs: invocation.fork.checkpointNs,
+ ...(invocation.fork.checkpointId == null
+ ? {}
+ : { checkpointId: invocation.fork.checkpointId }),
+ },
+ action: { toolName: input.expectedAction?.toolName ?? 'expected_action' },
+ observedAt: new Date(),
+ },
+ });
+ if (!confirmed) {
+ const released = await deps.releaseAction?.({
+ deliveryKey: input.invocationId,
+ user: input.user,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ bindingId: input.bindingId,
+ conversationId: input.conversationId,
+ admissionId,
+ });
+ if (!released) {
+ throw new Error(
+ 'Event actor lost admission lifecycle and could not release its action',
+ );
+ }
+ throw new Error('Event actor action admission lifecycle was superseded before invoke');
+ }
+ ownedActionAdmissionId = admissionId;
+ }
try {
value = await input.invoke({
checkpointNamespace: invocation.fork.checkpointNs,
@@ -318,6 +507,33 @@ export async function executeAgentEventActor<T>(
return { status: 'committed', head: toHead(input.conversationId, committed.state) };
},
async discard(request) {
+ /** Release the delivery-owned action admission before deleting either
+ * the fork or its conversation-side lifecycle evidence. A crash after
+ * this release is retryable; the inverse order can orphan admission
+ * forever with no durable marker left to recover it from. */
+ if (ownedActionAdmissionId != null && input.bindingId != null && deps.releaseAction != null) {
+ const releasedAction = await deps.releaseAction({
+ deliveryKey: input.invocationId,
+ user: input.user,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ bindingId: input.bindingId,
+ conversationId: input.conversationId,
+ admissionId: ownedActionAdmissionId,
+ });
+ if (!releasedAction) {
+ const receipt = await deps.getReceipt?.({
+ deliveryKey: input.invocationId,
+ user: input.user,
+ ...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
+ bindingId: input.bindingId,
+ conversationId: input.conversationId,
+ });
+ if (receipt == null) {
+ throw new Error('Event actor action admission could not be released');
+ }
+ }
+ ownedActionAdmissionId = undefined;
+ }
await deleteAgentCheckpoint(request.invocation.fork.threadId, input.checkpointer, undefined, {
throwOnError: true,
checkpointNamespace: request.invocation.fork.checkpointNs,
@@ -334,6 +550,9 @@ export async function executeAgentEventActor<T>(
? {}
: { checkpointId: request.invocation.fork.checkpointId }),
},
+ ...(input.bindingId != null && deps.admitAction != null
+ ? { expectedActionAdmitted: true }
+ : {}),
resolution: 'invocation_abandoned',
});
if (!released) {
@@ -419,6 +638,7 @@ export async function executeAgentEventActor<T>(
...(input.tenantId == null ? {} : { tenantId: input.tenantId }),
reconciliation: {
invocationId: input.invocationId,
+ ...(input.bindingId != null && deps.admitAction != null ? { actionAdmitted: true } : {}),
status: execution.status,
checkpoint: {
threadId: execution.checkpoint.threadId,
diff --git a/packages/api/src/agents/triggers/outcome.spec.ts b/packages/api/src/agents/triggers/outcome.spec.ts
index f4df31ec4a..ef0f2999fb 100644
--- a/packages/api/src/agents/triggers/outcome.spec.ts
+++ b/packages/api/src/agents/triggers/outcome.spec.ts
@@ -17,6 +17,10 @@ const createAgentEventTerminalHandler = (
getAgentEventActorSnapshot: jest.fn().mockResolvedValue(undefined),
getMessage: jest.fn().mockResolvedValue(null),
resolveAgentEventActorReconciliation: jest.fn().mockResolvedValue(true),
+ clearAgentEventActorReconciliation: jest.fn().mockResolvedValue(true),
+ settleAgentEventActorReceipt: jest.fn().mockResolvedValue(true),
+ getAgentEventActorReceipt: jest.fn().mockResolvedValue(null),
+ backfillAgentEventActorReceipt: jest.fn().mockResolvedValue(true),
completeAgentEventActorLegacyTurn: jest.fn().mockResolvedValue(true),
...methods,
});
@@ -571,6 +575,7 @@ describe('agent event terminal outcomes', () => {
reconciliations: [
{
invocationId: 'trigger_1',
+ actionAdmitted: true,
status: 'history_persisted',
checkpoint,
action: { toolName: 'submit_move' },
@@ -621,6 +626,154 @@ describe('agent event terminal outcomes', () => {
);
});
+ it('settles the actor receipt before clearing the conversation lifecycle', async () => {
+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
+ const settleAgentEventActorReceipt = jest.fn().mockResolvedValue(true);
+ const clearAgentEventActorReconciliation = jest.fn().mockResolvedValue(true);
+ const checkpoint = {
+ threadId: 'conversation-1',
+ checkpointId: 'checkpoint-1',
+ checkpointNs: 'event-actor/trigger_1',
+ };
+ const handler = createAgentEventTerminalHandler({
+ settleAgentTriggerHandlingOutcome,
+ settleAgentEventActorReceipt,
+ clearAgentEventActorReconciliation,
+ getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
+ state: { generation: 1, checkpoint },
+ reconciliations: [
+ {
+ invocationId: 'trigger_1',
+ actionAdmitted: true,
+ status: 'history_persisted',
+ checkpoint,
+ action: { toolName: 'submit_move', toolCallId: 'call-1' },
+ observedAt: new Date(),
+ },
+ ],
+ }),
+ getMessage: jest.fn(async ({ messageId }) => {
+ const isUser = messageId.endsWith(':user');
+ return {
+ messageId,
+ conversationId: 'conversation-1',
+ isCreatedByUser: isUser,
+ parentMessageId: isUser ? 'parent-message' : 'trigger_1:user',
+ } as never;
+ }),
+ });
+
+ await handler(
+ 'conversation-1',
+ job({ agentEventBindingId: 'binding-1', status: 'error', error: 'late provider error' }),
+ [],
+ );
+
+ expect(settleAgentEventActorReceipt).toHaveBeenCalledWith({
+ deliveryKey: 'trigger_1',
+ user: 'user-1',
+ bindingId: 'binding-1',
+ conversationId: 'conversation-1',
+ generationCreatedAt: 1_787_000_000_000,
+ status: 'applied',
+ settledAt: new Date(1_787_000_001_000),
+ requiresActionAdmission: true,
+ receipt: {
+ resolution: 'checkpoint_verified',
+ checkpoint,
+ action: { toolName: 'submit_move', toolCallId: 'call-1' },
+ },
+ });
+ expect(clearAgentEventActorReconciliation).toHaveBeenCalledWith({
+ user: 'user-1',
+ conversationId: 'conversation-1',
+ invocationId: 'trigger_1',
+ checkpoint,
+ resolution: 'checkpoint_verified',
+ });
+ expect(settleAgentEventActorReceipt.mock.invocationCallOrder[0]).toBeLessThan(
+ clearAgentEventActorReconciliation.mock.invocationCallOrder[0],
+ );
+ expect(settleAgentTriggerHandlingOutcome).not.toHaveBeenCalled();
+ });
+
+ it('converges after a crash between actor receipt settlement and marker cleanup', async () => {
+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
+ const settleAgentEventActorReceipt = jest.fn().mockResolvedValue(true);
+ const checkpoint = {
+ threadId: 'conversation-1',
+ checkpointId: 'checkpoint-1',
+ checkpointNs: 'event-actor/trigger_1',
+ };
+ const receipt = {
+ bindingId: 'binding-1',
+ resolution: 'checkpoint_verified' as const,
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ settledAt: new Date(1_787_000_001_000),
+ };
+ const clearAgentEventActorReconciliation = jest
+ .fn()
+ .mockResolvedValueOnce(false)
+ .mockResolvedValueOnce(true);
+ const getAgentEventActorReceipt = jest
+ .fn()
+ .mockResolvedValueOnce(null)
+ .mockResolvedValue(receipt);
+ const handler = createAgentEventTerminalHandler({
+ settleAgentTriggerHandlingOutcome,
+ settleAgentEventActorReceipt,
+ clearAgentEventActorReconciliation,
+ getAgentEventActorReceipt,
+ getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
+ state: { generation: 1, checkpoint },
+ reconciliations: [
+ {
+ invocationId: 'trigger_1',
+ actionAdmitted: true,
+ status: 'history_persisted',
+ checkpoint,
+ action: receipt.action,
+ observedAt: new Date(),
+ },
+ ],
+ }),
+ getMessage: jest.fn(async ({ messageId }) => {
+ const isUser = messageId.endsWith(':user');
+ return {
+ messageId,
+ conversationId: 'conversation-1',
+ isCreatedByUser: isUser,
+ parentMessageId: isUser ? 'parent-message' : 'trigger_1:user',
+ } as never;
+ }),
+ });
+ const actorJob = job({ agentEventBindingId: 'binding-1' });
+
+ await expect(handler('conversation-1', actorJob, [])).rejects.toThrow(
+ 'terminal marker could not be cleared',
+ );
+ await handler('conversation-1', actorJob, []);
+
+ expect(settleAgentEventActorReceipt).toHaveBeenCalledTimes(2);
+ expect(settleAgentEventActorReceipt).toHaveBeenLastCalledWith({
+ deliveryKey: 'trigger_1',
+ user: 'user-1',
+ bindingId: 'binding-1',
+ conversationId: 'conversation-1',
+ generationCreatedAt: 1_787_000_000_000,
+ status: 'applied',
+ settledAt: receipt.settledAt,
+ receipt: {
+ resolution: 'checkpoint_verified',
+ checkpoint,
+ action: receipt.action,
+ },
+ });
+ expect(clearAgentEventActorReconciliation).toHaveBeenCalledTimes(2);
+ expect(settleAgentTriggerHandlingOutcome).not.toHaveBeenCalled();
+ });
+
it('retains authoritative action proof until a failed settlement retry succeeds', async () => {
const settleAgentTriggerHandlingOutcome = jest
.fn()
@@ -802,6 +955,103 @@ describe('agent event terminal outcomes', () => {
expect(resolveAgentEventActorReconciliation).not.toHaveBeenCalled();
});
+ it('lazily migrates a legacy settled receipt before deleting its embedded copy', async () => {
+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
+ const backfillAgentEventActorReceipt = jest.fn().mockResolvedValue(true);
+ const clearAgentEventActorReconciliation = jest.fn().mockResolvedValue(true);
+ const checkpoint = {
+ threadId: 'conversation-1',
+ checkpointId: 'checkpoint-1',
+ checkpointNs: 'event-actor/trigger_1',
+ };
+ const observedAt = new Date('2026-08-25T00:00:00.000Z');
+ const handler = createAgentEventTerminalHandler({
+ settleAgentTriggerHandlingOutcome,
+ backfillAgentEventActorReceipt,
+ clearAgentEventActorReconciliation,
+ getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
+ state: { generation: 1, checkpoint },
+ reconciliations: [
+ {
+ invocationId: 'trigger_1',
+ status: 'settled',
+ resolution: 'checkpoint_verified',
+ checkpoint,
+ action: { toolName: 'submit_move', toolCallId: 'call-1' },
+ observedAt,
+ },
+ ],
+ }),
+ });
+
+ await handler('conversation-1', job({ agentEventBindingId: 'binding-1' }), []);
+
+ expect(settleAgentTriggerHandlingOutcome).toHaveBeenCalledWith({
+ deliveryKey: 'trigger_1',
+ conversationId: 'conversation-1',
+ generationCreatedAt: job().createdAt,
+ status: 'applied',
+ settledAt: observedAt,
+ action: { toolName: 'submit_move', toolCallId: 'call-1' },
+ });
+ expect(backfillAgentEventActorReceipt).toHaveBeenCalledWith(
+ expect.objectContaining({
+ deliveryKey: 'trigger_1',
+ bindingId: 'binding-1',
+ conversationId: 'conversation-1',
+ status: 'applied',
+ settledAt: observedAt,
+ receipt: {
+ resolution: 'checkpoint_verified',
+ checkpoint,
+ action: { toolName: 'submit_move', toolCallId: 'call-1' },
+ },
+ }),
+ );
+ expect(backfillAgentEventActorReceipt.mock.invocationCallOrder[0]).toBeLessThan(
+ clearAgentEventActorReconciliation.mock.invocationCallOrder[0],
+ );
+ expect(settleAgentTriggerHandlingOutcome.mock.invocationCallOrder[0]).toBeLessThan(
+ backfillAgentEventActorReceipt.mock.invocationCallOrder[0],
+ );
+ });
+
+ it('keeps legacy proof intact when its public outcome cannot be recovered', async () => {
+ const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(false);
+ const backfillAgentEventActorReceipt = jest.fn();
+ const clearAgentEventActorReconciliation = jest.fn();
+ const checkpoint = {
+ threadId: 'conversation-1',
+ checkpointId: 'checkpoint-1',
+ checkpointNs: 'event-actor/trigger_1',
+ };
+ const handler = createAgentEventTerminalHandler({
+ settleAgentTriggerHandlingOutcome,
+ backfillAgentEventActorReceipt,
+ clearAgentEventActorReconciliation,
+ getAgentEventActorSnapshot: jest.fn().mockResolvedValue({
+ state: { generation: 1, checkpoint },
+ reconciliations: [
+ {
+ invocationId: 'trigger_1',
+ status: 'settled',
+ resolution: 'checkpoint_verified',
+ checkpoint,
+ action: { toolName: 'submit_move', toolCallId: 'call-1' },
+ observedAt: new Date('2026-08-25T00:00:00.000Z'),
+ },
+ ],
+ }),
+ });
+
+ await expect(
+ handler('conversation-1', job({ agentEventBindingId: 'binding-1' }), []),
+ ).rejects.toThrow('legacy public outcome could not be recovered');
+
+ expect(backfillAgentEventActorReceipt).not.toHaveBeenCalled();
+ expect(clearAgentEventActorReconciliation).not.toHaveBeenCalled();
+ });
+
it('never replays a compensated receipt as applied even over fresh run evidence', async () => {
const settleAgentTriggerHandlingOutcome = jest.fn().mockResolvedValue(true);
const resolveAgentEventActorReconciliation = jest.fn().mockResolvedValue(true);
diff --git a/packages/api/src/agents/triggers/outcome.ts b/packages/api/src/agents/triggers/outcome.ts
index 2e019831e3..2aa953c133 100644
--- a/packages/api/src/agents/triggers/outcome.ts
+++ b/packages/api/src/agents/triggers/outcome.ts
@@ -1,4 +1,8 @@
-import type { ConversationMethods, MessageMethods } from '@librechat/data-schemas';
+import type {
+ AgentTriggerDeliveryMethods,
+ ConversationMethods,
+ MessageMethods,
+} from '@librechat/data-schemas';
import type { Agents } from 'librechat-data-provider';
import type { AgentTriggerExpectedAction } from './envelope';
import type { SerializableJobData } from '~/stream';
@@ -331,6 +335,10 @@ export function createAgentEventTerminalHandler(methods: {
) => Promise<boolean>;
getAgentEventActorSnapshot: ConversationMethods['getAgentEventActorSnapshot'];
resolveAgentEventActorReconciliation: ConversationMethods['resolveAgentEventActorReconciliation'];
+ clearAgentEventActorReconciliation: ConversationMethods['clearAgentEventActorReconciliation'];
+ settleAgentEventActorReceipt: AgentTriggerDeliveryMethods['settleAgentEventActorReceipt'];
+ getAgentEventActorReceipt: AgentTriggerDeliveryMethods['getAgentEventActorReceipt'];
+ backfillAgentEventActorReceipt: AgentTriggerDeliveryMethods['backfillAgentEventActorReceipt'];
completeAgentEventActorLegacyTurn: ConversationMethods['completeAgentEventActorLegacyTurn'];
getMessage: MessageMethods['getMessage'];
}): (
@@ -350,8 +358,10 @@ export function createAgentEventTerminalHandler(methods: {
}
const conversationId = job.conversationId ?? streamId;
const outcome = classifyAgentEventRunOutcome(job, runSteps, content);
+ const settledAt = new Date(job.completedAt ?? Date.now());
let committedAction: AgentEventAppliedAction | undefined;
let compensated = false;
+ let actorReceiptSettled = false;
const snapshot = await methods.getAgentEventActorSnapshot({
user: job.userId,
conversationId,
@@ -360,8 +370,130 @@ export function createAgentEventTerminalHandler(methods: {
const lifecycle = snapshot?.reconciliations.find(
(item) => item.invocationId === job.agentEventDeliveryKey,
);
- if (lifecycle != null) {
+ let durableReceipt =
+ job.agentEventBindingId == null
+ ? null
+ : await methods.getAgentEventActorReceipt({
+ deliveryKey: job.agentEventDeliveryKey,
+ user: job.userId,
+ ...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
+ bindingId: job.agentEventBindingId,
+ conversationId,
+ });
+ if (durableReceipt != null) {
+ actorReceiptSettled = true;
+ /** Settlement owns public batch propagation and lane cleanup as well as
+ * the private receipt. Replaying the exact settlement repairs a crash
+ * after the atomic root write but before either idempotent side effect. */
+ const finalized = await methods.settleAgentEventActorReceipt({
+ deliveryKey: job.agentEventDeliveryKey,
+ user: job.userId,
+ ...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
+ bindingId: durableReceipt.bindingId,
+ conversationId,
+ generationCreatedAt: job.createdAt,
+ status: durableReceipt.resolution === 'action_compensated' ? 'failed' : 'applied',
+ settledAt: durableReceipt.settledAt,
+ ...(durableReceipt.resolution === 'action_compensated' && {
+ error: 'Applied event actor action was explicitly compensated',
+ }),
+ receipt: {
+ resolution: durableReceipt.resolution,
+ checkpoint: durableReceipt.checkpoint,
+ action: durableReceipt.action,
+ },
+ });
+ if (!finalized) {
+ throw new Error(
+ `Agent event actor ${job.agentEventDeliveryKey} terminal receipt could not be finalized`,
+ );
+ }
+ if (durableReceipt.resolution === 'action_compensated') {
+ compensated = true;
+ } else {
+ committedAction = durableReceipt.action;
+ }
+ if (lifecycle != null) {
+ const cleared = await methods.clearAgentEventActorReconciliation({
+ user: job.userId,
+ conversationId,
+ ...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
+ invocationId: job.agentEventDeliveryKey,
+ checkpoint: durableReceipt.checkpoint,
+ resolution: durableReceipt.resolution,
+ });
+ if (!cleared) {
+ throw new Error(
+ `Agent event actor ${job.agentEventDeliveryKey} terminal marker could not be cleared`,
+ );
+ }
+ }
+ } else if (lifecycle != null) {
if (lifecycle.status === 'settled') {
+ /** Compatibility for receipts written by the pre-delivery-ledger build:
+ * copy the exact terminal proof only when the already-public handling
+ * outcome agrees, then remove the embedded representation. */
+ if (job.agentEventBindingId != null && lifecycle.resolution != null) {
+ const legacyCompensated = lifecycle.resolution === 'action_compensated';
+ /** A pre-ledger owner can settle the embedded lifecycle and crash
+ * before publishing the matching delivery outcome. Terminalize that
+ * exact started generation first; replay is idempotent when an older
+ * owner already completed the public write. The resulting terminal
+ * handling is then the delivery-side proof required by migration. */
+ const legacyStatus = legacyCompensated ? 'failed' : 'applied';
+ const legacyError = legacyCompensated
+ ? 'Applied event actor action was explicitly compensated'
+ : undefined;
+ const terminalized = await methods.settleAgentTriggerHandlingOutcome({
+ deliveryKey: job.agentEventDeliveryKey,
+ conversationId,
+ generationCreatedAt: job.createdAt,
+ status: legacyStatus,
+ settledAt: lifecycle.observedAt,
+ ...(legacyError == null ? {} : { error: legacyError }),
+ ...(!legacyCompensated && { action: lifecycle.action }),
+ });
+ if (!terminalized) {
+ throw new Error(
+ `Agent event actor ${job.agentEventDeliveryKey} legacy public outcome could not be recovered`,
+ );
+ }
+ const migrated = await methods.backfillAgentEventActorReceipt({
+ deliveryKey: job.agentEventDeliveryKey,
+ user: job.userId,
+ ...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
+ bindingId: job.agentEventBindingId,
+ conversationId,
+ generationCreatedAt: job.createdAt,
+ status: legacyStatus,
+ settledAt: lifecycle.observedAt,
+ ...(legacyError == null ? {} : { error: legacyError }),
+ receipt: {
+ resolution: lifecycle.resolution,
+ checkpoint: lifecycle.checkpoint,
+ action: lifecycle.action,
+ },
+ });
+ if (!migrated) {
+ throw new Error(
+ `Agent event actor ${job.agentEventDeliveryKey} legacy receipt could not be migrated`,
+ );
+ }
+ const cleared = await methods.clearAgentEventActorReconciliation({
+ user: job.userId,
+ conversationId,
+ ...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
+ invocationId: job.agentEventDeliveryKey,
+ checkpoint: lifecycle.checkpoint,
+ resolution: lifecycle.resolution,
+ });
+ if (!cleared) {
+ throw new Error(
+ `Agent event actor ${job.agentEventDeliveryKey} migrated marker could not be cleared`,
+ );
+ }
+ actorReceiptSettled = true;
+ }
/** A compensated receipt still tombstones its invocation id, but its
* external effect was explicitly undone — replaying it as applied
* would tell the source the operation stands and suppress the
@@ -375,7 +507,78 @@ export function createAgentEventTerminalHandler(methods: {
throw new Error(
`Agent event actor ${job.agentEventDeliveryKey} requires ${lifecycle.status} reconciliation`,
);
+ } else if (job.agentEventBindingId != null) {
+ const historyIsDurable = await hasDurableAgentEventHistory({
+ getMessage: methods.getMessage,
+ user: job.userId,
+ conversationId,
+ deliveryKey: job.agentEventDeliveryKey,
+ });
+ if (!historyIsDurable) {
+ throw new Error(
+ `Agent event actor ${job.agentEventDeliveryKey} has invalid durable message history`,
+ );
+ }
+ const stored = await methods.settleAgentEventActorReceipt({
+ deliveryKey: job.agentEventDeliveryKey,
+ user: job.userId,
+ ...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
+ bindingId: job.agentEventBindingId,
+ conversationId,
+ generationCreatedAt: job.createdAt,
+ status: 'applied',
+ settledAt,
+ ...(lifecycle.actionAdmitted === true && { requiresActionAdmission: true }),
+ receipt: {
+ resolution: 'checkpoint_verified',
+ checkpoint: lifecycle.checkpoint,
+ action: lifecycle.action,
+ },
+ });
+ if (!stored) {
+ durableReceipt = await methods.getAgentEventActorReceipt({
+ deliveryKey: job.agentEventDeliveryKey,
+ user: job.userId,
+ ...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
+ bindingId: job.agentEventBindingId,
+ conversationId,
+ });
+ if (durableReceipt == null) {
+ throw new Error(
+ `Agent event actor ${job.agentEventDeliveryKey} terminal receipt was not retained`,
+ );
+ }
+ }
+ durableReceipt ??= {
+ bindingId: job.agentEventBindingId,
+ resolution: 'checkpoint_verified',
+ checkpoint: lifecycle.checkpoint,
+ action: lifecycle.action,
+ settledAt,
+ };
+ actorReceiptSettled = true;
+ if (durableReceipt.resolution === 'action_compensated') {
+ compensated = true;
+ } else {
+ committedAction = durableReceipt.action;
+ }
+ const cleared = await methods.clearAgentEventActorReconciliation({
+ user: job.userId,
+ conversationId,
+ ...(job.tenantId == null ? {} : { tenantId: job.tenantId }),
+ invocationId: job.agentEventDeliveryKey,
+ checkpoint: durableReceipt.checkpoint,
+ resolution: durableReceipt.resolution,
+ });
+ if (!cleared) {
+ throw new Error(
+ `Agent event actor ${job.agentEventDeliveryKey} terminal marker could not be cleared`,
+ );
+ }
} else {
+ /** A generation created before binding identity was added must finish
+ * through the old receipt path; inventing binding scope here would be
+ * less safe than retaining the already-deployed mixed-version logic. */
const historyIsDurable = await hasDurableAgentEventHistory({
getMessage: methods.getMessage,
user: job.userId,
@@ -387,15 +590,6 @@ export function createAgentEventTerminalHandler(methods: {
`Agent event actor ${job.agentEventDeliveryKey} has invalid durable message history`,
);
}
- /** The persistence lifecycle was created by the same CAS that advanced
- * the actor head, and history_persisted is written only after the
- * controller's post-commit message barrier. Resolving BEFORE settlement
- * makes the receipt's status CAS the serialization point against a
- * concurrent compensation: whichever transition wins determines the
- * public outcome, and a crash between this resolve and the settle
- * below converges through the retained receipt's replay. The receipt
- * keeps its full action proof either way, so nothing is lost if the
- * settle write never lands. */
const resolved = await methods.resolveAgentEventActorReconciliation({
user: job.userId,
conversationId,
@@ -404,9 +598,7 @@ export function createAgentEventTerminalHandler(methods: {
checkpoint: lifecycle.checkpoint,
resolution: 'checkpoint_verified',
});
- if (resolved) {
- committedAction = lifecycle.action;
- } else {
+ if (!resolved) {
const reread = await methods.getAgentEventActorSnapshot({
user: job.userId,
conversationId,
@@ -422,6 +614,8 @@ export function createAgentEventTerminalHandler(methods: {
`Agent event actor ${job.agentEventDeliveryKey} settlement receipt was not retained`,
);
}
+ } else {
+ committedAction = lifecycle.action;
}
}
}
@@ -458,7 +652,9 @@ export function createAgentEventTerminalHandler(methods: {
throw new Error(`Failed to seal legacy event actor turn ${job.agentEventLegacyTurnToken}`);
}
}
- const settledAt = new Date(job.completedAt ?? Date.now());
+ if (actorReceiptSettled) {
+ return;
+ }
const settled = await methods.settleAgentTriggerHandlingOutcome({
deliveryKey: job.agentEventDeliveryKey,
conversationId,
diff --git a/packages/api/src/agents/triggers/service.delivery.spec.ts b/packages/api/src/agents/triggers/service.delivery.spec.ts
index 3eec41de66..ff1ecd8e78 100644
--- a/packages/api/src/agents/triggers/service.delivery.spec.ts
+++ b/packages/api/src/agents/triggers/service.delivery.spec.ts
@@ -367,6 +367,22 @@ describe('durable agent trigger service', () => {
await service.stop();
});
+ it('expires dormant legacy actor receipts during durable maintenance', async () => {
+ const expireLegacyAgentEventActorReceipts = jest.fn().mockResolvedValue(1);
+ const service = createAgentTriggerService({
+ methods: deliveryMethods({ expireLegacyAgentEventActorReceipts }),
+ purgeRecoveryIntervalMs: 60_000,
+ purgeRecoveryLimit: 17,
+ deliveryOptions: { concurrency: 1, tickMs: 60_000 },
+ });
+ await service.initialize({
+ address: { address: '127.0.0.1', family: 'IPv4', port: 3080 },
+ });
+
+ expect(expireLegacyAgentEventActorReceipts).toHaveBeenCalledWith(expect.any(Date), 17);
+ await service.stop();
+ });
+
it('settles interrupted batch receipts before reclaiming their lane', async () => {
let finishBatchRecovery: ((count: number) => void) | undefined;
const recoverAgentTriggerBatchReceipts = jest.fn(
diff --git a/packages/api/src/agents/triggers/service.ts b/packages/api/src/agents/triggers/service.ts
index c891fec73f..fccc635d2a 100644
--- a/packages/api/src/agents/triggers/service.ts
+++ b/packages/api/src/agents/triggers/service.ts
@@ -121,6 +121,7 @@ export interface AgentTriggerDeliveryPersistence {
) => Promise<void>;
cancelAgentTriggerUserPurge: (userId: string, fenceStartedAt: Date) => Promise<boolean>;
recoverAgentTriggerUserPurges: (limit?: number) => Promise<number>;
+ expireLegacyAgentEventActorReceipts?: (now: Date, limit?: number) => Promise<number>;
deleteAgentTriggerDeliveriesByUser: (userId: string) => Promise<void>;
}
@@ -302,21 +303,31 @@ export function createAgentTriggerService(deps: AgentTriggerServiceDeps = {}): A
}
const methods = deps.methods;
const current = runAsSystem(async () => {
- const [purgedUsers, publishedLanes, recoveredBatches] = await Promise.all([
- methods.recoverAgentTriggerUserPurges(purgeRecoveryLimit),
- methods.recoverAgentTriggerLanePublications(purgeRecoveryLimit),
- methods.recoverAgentTriggerBatchReceipts(purgeRecoveryLimit),
- ]);
+ const [purgedUsers, publishedLanes, recoveredBatches, expiredLegacyActorReceipts] =
+ await Promise.all([
+ methods.recoverAgentTriggerUserPurges(purgeRecoveryLimit),
+ methods.recoverAgentTriggerLanePublications(purgeRecoveryLimit),
+ methods.recoverAgentTriggerBatchReceipts(purgeRecoveryLimit),
+ methods.expireLegacyAgentEventActorReceipts?.(new Date(), purgeRecoveryLimit) ??
+ Promise.resolve(0),
+ ]);
const reclaimedLanes = await methods.reclaimInactiveAgentTriggerLanes(purgeRecoveryLimit);
if (publishedLanes > 0) {
deliveryEngine?.wake();
}
- if (purgedUsers > 0 || publishedLanes > 0 || recoveredBatches > 0 || reclaimedLanes > 0) {
+ if (
+ purgedUsers > 0 ||
+ publishedLanes > 0 ||
+ recoveredBatches > 0 ||
+ reclaimedLanes > 0 ||
+ expiredLegacyActorReceipts > 0
+ ) {
logger.info('[agent-triggers] recovered durable delivery maintenance', {
purgedUsers,
publishedLanes,
recoveredBatches,
reclaimedLanes,
+ expiredLegacyActorReceipts,
});
}
})
diff --git a/packages/api/src/app/agents.spec.ts b/packages/api/src/app/agents.spec.ts
index c15d695afb..0733ee6b96 100644
--- a/packages/api/src/app/agents.spec.ts
+++ b/packages/api/src/app/agents.spec.ts
@@ -9,6 +9,7 @@ describe('configureAgentEventRuntime', () => {
delete process.env.ENABLE_SUBAGENT_COMPLETION_WAKEUPS;
delete process.env.ENABLE_AGENT_EVENT_COALESCING;
delete process.env.ENABLE_AGENT_EVENT_ACTOR_MAILBOX;
+ delete process.env.ENABLE_AGENT_EVENT_DURABLE_RECEIPTS;
delete process.env.AGENT_TRIGGERS_SELF_URL;
});
@@ -22,6 +23,7 @@ describe('configureAgentEventRuntime', () => {
completionWakeups: false,
coalescing: true,
actorMailbox: true,
+ durableReceipts: true,
selfUrl: 'https://triggers.internal',
});
@@ -29,6 +31,7 @@ describe('configureAgentEventRuntime', () => {
expect(process.env.ENABLE_SUBAGENT_COMPLETION_WAKEUPS).toBe('false');
expect(process.env.ENABLE_AGENT_EVENT_COALESCING).toBe('true');
expect(process.env.ENABLE_AGENT_EVENT_ACTOR_MAILBOX).toBe('true');
+ expect(process.env.ENABLE_AGENT_EVENT_DURABLE_RECEIPTS).toBe('true');
expect(process.env.AGENT_TRIGGERS_SELF_URL).toBe('https://triggers.internal');
});
@@ -37,6 +40,7 @@ describe('configureAgentEventRuntime', () => {
process.env.ENABLE_SUBAGENT_COMPLETION_WAKEUPS = 'true';
process.env.ENABLE_AGENT_EVENT_COALESCING = 'true';
process.env.ENABLE_AGENT_EVENT_ACTOR_MAILBOX = 'true';
+ process.env.ENABLE_AGENT_EVENT_DURABLE_RECEIPTS = 'true';
process.env.AGENT_TRIGGERS_SELF_URL = 'https://legacy.internal';
configureAgentEventRuntime(undefined);
@@ -45,6 +49,7 @@ describe('configureAgentEventRuntime', () => {
expect(process.env.ENABLE_SUBAGENT_COMPLETION_WAKEUPS).toBe('true');
expect(process.env.ENABLE_AGENT_EVENT_COALESCING).toBe('true');
expect(process.env.ENABLE_AGENT_EVENT_ACTOR_MAILBOX).toBe('true');
+ expect(process.env.ENABLE_AGENT_EVENT_DURABLE_RECEIPTS).toBe('true');
expect(process.env.AGENT_TRIGGERS_SELF_URL).toBe('https://legacy.internal');
});
});
diff --git a/packages/api/src/app/agents.ts b/packages/api/src/app/agents.ts
index c14fdfa4f2..93b3474b97 100644
--- a/packages/api/src/app/agents.ts
+++ b/packages/api/src/app/agents.ts
@@ -14,6 +14,7 @@ export const configureAgentEventRuntime = (config?: AgentEventRuntimeConfig): vo
setBooleanEnvironmentFallback('ENABLE_SUBAGENT_COMPLETION_WAKEUPS', config?.completionWakeups);
setBooleanEnvironmentFallback('ENABLE_AGENT_EVENT_COALESCING', config?.coalescing);
setBooleanEnvironmentFallback('ENABLE_AGENT_EVENT_ACTOR_MAILBOX', config?.actorMailbox);
+ setBooleanEnvironmentFallback('ENABLE_AGENT_EVENT_DURABLE_RECEIPTS', config?.durableReceipts);
if (config?.selfUrl != null) {
process.env.AGENT_TRIGGERS_SELF_URL = config.selfUrl;
}
diff --git a/packages/api/src/app/metrics.spec.ts b/packages/api/src/app/metrics.spec.ts
index 7422ccaa37..6d6c1b87e6 100644
--- a/packages/api/src/app/metrics.spec.ts
+++ b/packages/api/src/app/metrics.spec.ts
@@ -2,6 +2,7 @@
import express from 'express';
import request from 'supertest';
import { EventEmitter } from 'events';
+import { recordAgentEventActorReceiptMetric } from '@librechat/data-schemas';
import type { Request, Response } from 'express';
import {
createMetrics,
@@ -132,6 +133,89 @@ describe('createMetrics', () => {
);
});
+ it('exposes bounded event actor receipt settlement, replay, conflict, and migration metrics', async () => {
+ const app = express();
+ process.env.METRICS_SECRET = 'test-secret';
+ const { metricsRouter } = createMetrics();
+ app.use('/metrics', metricsRouter);
+
+ recordAgentEventActorReceiptMetric({
+ operation: 'settle',
+ outcome: 'success',
+ resolution: 'checkpoint_verified',
+ });
+ recordAgentEventActorReceiptMetric({
+ operation: 'read',
+ outcome: 'hit',
+ resolution: 'checkpoint_verified',
+ });
+ recordAgentEventActorReceiptMetric({
+ operation: 'settle',
+ outcome: 'conflict',
+ resolution: 'action_compensated',
+ });
+ recordAgentEventActorReceiptMetric({
+ operation: 'backfill',
+ outcome: 'success',
+ resolution: 'history_repaired',
+ });
+
+ const response = await request(app)
+ .get('/metrics')
+ .set('Authorization', 'Bearer test-secret')
+ .expect(200);
+
+ expect(response.text).toContain(
+ 'agent_event_actor_receipt_operations_total{operation="settle",outcome="success",resolution="checkpoint_verified"} 1',
+ );
+ expect(response.text).toContain(
+ 'agent_event_actor_receipt_operations_total{operation="read",outcome="hit",resolution="checkpoint_verified"} 1',
+ );
+ expect(response.text).toContain(
+ 'agent_event_actor_receipt_operations_total{operation="settle",outcome="conflict",resolution="action_compensated"} 1',
+ );
+ expect(response.text).toContain(
+ 'agent_event_actor_receipt_operations_total{operation="backfill",outcome="success",resolution="history_repaired"} 1',
+ );
+ });
+
+ it('collects truthful event actor receipt, reconciliation, retry, and TTL gauges', async () => {
+ const app = express();
+ process.env.METRICS_SECRET = 'test-secret';
+ const collectAgentEventActorStorageMetrics = jest.fn(async () => ({
+ retainedByResolution: {
+ checkpoint_verified: 7,
+ action_compensated: 2,
+ history_repaired: 1,
+ },
+ expiryEligible: 3,
+ retryDeliveries: 4,
+ deadDeliveries: 5,
+ pendingReconciliations: 6,
+ oldestPendingAgeSeconds: 91,
+ }));
+ const { metricsRouter } = createMetrics({
+ collectAgentEventActorStorageMetrics,
+ });
+ app.use('/metrics', metricsRouter);
+
+ const response = await request(app)
+ .get('/metrics')
+ .set('Authorization', 'Bearer test-secret')
+ .expect(200);
+
+ expect(response.text).toContain(
+ 'agent_event_actor_receipts_retained{resolution="checkpoint_verified"} 7',
+ );
+ expect(response.text).toContain('agent_event_actor_receipts_expiry_eligible 3');
+ expect(response.text).toContain('agent_event_actor_reconciliations_pending 6');
+ expect(response.text).toContain('agent_event_actor_oldest_reconciliation_age_seconds 91');
+ expect(response.text).toContain('agent_event_actor_deliveries{state="retry"} 4');
+ expect(response.text).toContain('agent_event_actor_deliveries{state="dead"} 5');
+ await request(app).get('/metrics').set('Authorization', 'Bearer test-secret').expect(200);
+ expect(collectAgentEventActorStorageMetrics).toHaveBeenCalledTimes(1);
+ });
+
it('tracks SSE stream counts, active gauges, and stream duration', async () => {
const app = express();
process.env.METRICS_SECRET = 'test-secret';
diff --git a/packages/api/src/app/metrics.ts b/packages/api/src/app/metrics.ts
index 95edac6f12..0d31a7c2ed 100644
--- a/packages/api/src/app/metrics.ts
+++ b/packages/api/src/app/metrics.ts
@@ -1,7 +1,7 @@
import { Router } from 'express';
import { timingSafeEqual } from 'crypto';
-import { logger } from '@librechat/data-schemas';
import { Registry, collectDefaultMetrics, Counter, Gauge, Histogram } from 'prom-client';
+import { logger, setAgentEventActorReceiptMetricObserver } from '@librechat/data-schemas';
import type { Request, Response, NextFunction, RequestHandler } from 'express';
import type { Mongoose } from 'mongoose';
import type { AgentStartupMilestone, AgentStartupResult } from '~/agents/phases';
@@ -129,6 +129,24 @@ export interface PrometheusMetrics {
metricsRouter: Router;
}
+export interface AgentEventActorStorageMetricsSnapshot {
+ retainedByResolution: Record<
+ 'checkpoint_verified' | 'action_compensated' | 'history_repaired',
+ number
+ >;
+ expiryEligible: number;
+ retryDeliveries: number;
+ deadDeliveries: number;
+ pendingReconciliations: number;
+ oldestPendingAgeSeconds: number;
+}
+
+export interface MetricsOptions {
+ collectAgentEventActorStorageMetrics?: () => Promise<AgentEventActorStorageMetricsSnapshot>;
+}
+
+const AGENT_EVENT_ACTOR_STORAGE_METRICS_CACHE_MS = 60_000;
+
export type OpenIDUserLookupResult = 'found' | 'not_found' | 'migration' | 'auth_failed' | 'error';
export type GenerationJobStore = 'memory' | 'redis';
export type GenerationJobResult = 'created' | 'completed' | 'error' | 'aborted' | 'abort_failed';
@@ -255,6 +273,7 @@ const resetMetricRecorders = (): void => {
redisOperationMetrics = {
recordOperation: () => undefined,
};
+ setAgentEventActorReceiptMetricObserver();
};
export function recordGenerationJob(store: GenerationJobStore, result: GenerationJobResult): void {
@@ -429,7 +448,7 @@ export function instrumentMongooseQueryMetrics(mongoose: Mongoose): void {
queryPrototype[instrumented] = true;
}
-export function createMetrics(): PrometheusMetrics {
+export function createMetrics(options: MetricsOptions = {}): PrometheusMetrics {
if (!isMetricsConfigured()) {
resetMetricRecorders();
return {
@@ -639,6 +658,78 @@ export function createMetrics(): PrometheusMetrics {
registers: [registry],
});
+ const agentEventActorReceiptOperations = new Counter({
+ name: 'agent_event_actor_receipt_operations_total',
+ help: 'Event actor receipt storage operations by bounded outcome and resolution',
+ labelNames: ['operation', 'outcome', 'resolution'] as const,
+ registers: [registry],
+ });
+
+ const agentEventActorReceiptsRetained = new Gauge({
+ name: 'agent_event_actor_receipts_retained',
+ help: 'Delivery-owned event actor receipts currently retained for replay',
+ labelNames: ['resolution'] as const,
+ registers: [registry],
+ });
+ const agentEventActorReceiptsExpiryEligible = new Gauge({
+ name: 'agent_event_actor_receipts_expiry_eligible',
+ help: 'Retained event actor receipts whose Mongo TTL deadline has elapsed',
+ registers: [registry],
+ });
+ const agentEventActorReconciliationsPending = new Gauge({
+ name: 'agent_event_actor_reconciliations_pending',
+ help: 'Active event actor reconciliation markers awaiting a terminal delivery receipt',
+ registers: [registry],
+ });
+ const agentEventActorOldestReconciliationAge = new Gauge({
+ name: 'agent_event_actor_oldest_reconciliation_age_seconds',
+ help: 'Age in seconds of the oldest active event actor reconciliation marker',
+ registers: [registry],
+ });
+ const agentEventActorDeliveries = new Gauge({
+ name: 'agent_event_actor_deliveries',
+ help: 'Current retrying and dead delivery rows visible to the receipt ledger',
+ labelNames: ['state'] as const,
+ registers: [registry],
+ });
+
+ setAgentEventActorReceiptMetricObserver(({ operation, outcome, resolution }) => {
+ agentEventActorReceiptOperations.inc({
+ operation,
+ outcome,
+ resolution: resolution ?? 'none',
+ });
+ });
+
+ let actorStorageMetricsCache:
+ | { snapshot: AgentEventActorStorageMetricsSnapshot; expiresAt: number }
+ | undefined;
+ let actorStorageMetricsCollection: Promise<
+ AgentEventActorStorageMetricsSnapshot | undefined
+ > | null = null;
+ const collectActorStorageMetrics = async () => {
+ const now = Date.now();
+ if (actorStorageMetricsCache != null && actorStorageMetricsCache.expiresAt > now) {
+ return actorStorageMetricsCache.snapshot;
+ }
+ actorStorageMetricsCollection ??= Promise.resolve(
+ options.collectAgentEventActorStorageMetrics?.(),
+ )
+ .then((snapshot) => {
+ if (snapshot != null) {
+ actorStorageMetricsCache = {
+ snapshot,
+ expiresAt: Date.now() + AGENT_EVENT_ACTOR_STORAGE_METRICS_CACHE_MS,
+ };
+ }
+ return snapshot;
+ })
+ .finally(() => {
+ actorStorageMetricsCollection = null;
+ });
+ return actorStorageMetricsCollection;
+ };
+
generationJobMetrics = {
recordJob: (store, result) => generationJobs.inc({ store, result }),
setJobsInFlight: (store, count) => generationJobsInFlight.set({ store }, count),
@@ -777,8 +868,22 @@ export function createMetrics(): PrometheusMetrics {
return;
}
- void registry
- .metrics()
+ void Promise.resolve()
+ .then(async () => {
+ const snapshot = await collectActorStorageMetrics();
+ if (snapshot == null) {
+ return;
+ }
+ for (const [resolution, count] of Object.entries(snapshot.retainedByResolution)) {
+ agentEventActorReceiptsRetained.set({ resolution }, count);
+ }
+ agentEventActorReceiptsExpiryEligible.set(snapshot.expiryEligible);
+ agentEventActorReconciliationsPending.set(snapshot.pendingReconciliations);
+ agentEventActorOldestReconciliationAge.set(snapshot.oldestPendingAgeSeconds);
+ agentEventActorDeliveries.set({ state: 'retry' }, snapshot.retryDeliveries);
+ agentEventActorDeliveries.set({ state: 'dead' }, snapshot.deadDeliveries);
+ })
+ .then(() => registry.metrics())
.then((metrics) => {
res.set('Content-Type', registry.contentType);
res.end(metrics);
diff --git a/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts b/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts
index 014885d3d8..204136ca5b 100644
--- a/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts
+++ b/packages/api/src/stream/__tests__/RedisJobStore.stream_integration.spec.ts
@@ -1299,6 +1299,7 @@ describe('RedisJobStore Integration Tests', () => {
agent_id: 'saved-agent-1',
isTemporary: true,
agentEventDeliveryKey: 'trigger_1',
+ agentEventBindingId: 'binding-1',
agentEventExpectedAction: {
toolName: 'submit_move',
argumentSubset: { gameId: 'game-1', expectedPly: 7 },
@@ -1312,6 +1313,7 @@ describe('RedisJobStore Integration Tests', () => {
expect(turn1?.agent_id).toBe('saved-agent-1');
expect(turn1?.isTemporary).toBe(true);
expect(turn1?.agentEventDeliveryKey).toBe('trigger_1');
+ expect(turn1?.agentEventBindingId).toBe('binding-1');
expect(turn1?.agentEventExpectedAction).toEqual({
toolName: 'submit_move',
argumentSubset: { gameId: 'game-1', expectedPly: 7 },
@@ -1333,6 +1335,7 @@ describe('RedisJobStore Integration Tests', () => {
expect(turn2?.agent_id).toBeUndefined();
expect(turn2?.isTemporary).toBeUndefined();
expect(turn2?.agentEventDeliveryKey).toBeUndefined();
+ expect(turn2?.agentEventBindingId).toBeUndefined();
expect(turn2?.agentEventExpectedAction).toBeUndefined();
expect(turn2?.agentEventLegacyTurnToken).toBeUndefined();
expect(turn2?.discoveredTools).toBeUndefined();
diff --git a/packages/api/src/stream/implementations/RedisJobStore.ts b/packages/api/src/stream/implementations/RedisJobStore.ts
index d351963126..3e430e6868 100644
--- a/packages/api/src/stream/implementations/RedisJobStore.ts
+++ b/packages/api/src/stream/implementations/RedisJobStore.ts
@@ -4729,6 +4729,7 @@ export class RedisJobStore implements IJobStoreV2 {
agent_id: data.agent_id || undefined,
isTemporary: data.isTemporary != null ? data.isTemporary === '1' : undefined,
agentEventDeliveryKey: data.agentEventDeliveryKey || undefined,
+ agentEventBindingId: data.agentEventBindingId || undefined,
agentEventExpectedAction: data.agentEventExpectedAction
? JSON.parse(data.agentEventExpectedAction)
: undefined,
diff --git a/packages/api/src/stream/interfaces/IJobStore.ts b/packages/api/src/stream/interfaces/IJobStore.ts
index 5c74d080d7..0f64820664 100644
--- a/packages/api/src/stream/interfaces/IJobStore.ts
+++ b/packages/api/src/stream/interfaces/IJobStore.ts
@@ -279,6 +279,8 @@ export interface SerializableJobData {
*/
isTemporary?: boolean;
agentEventDeliveryKey?: string;
+ /** Trusted actor binding copied from the authenticated delivery envelope. */
+ agentEventBindingId?: string;
agentEventExpectedAction?: import('~/agents/triggers/types').AgentTriggerExpectedAction;
/** Exact durable legacy-turn fence carried across a HITL pause/resume. */
agentEventLegacyTurnToken?: string;
@@ -400,6 +402,7 @@ export type JobMetadataPatch = Partial<
| 'agent_id'
| 'isTemporary'
| 'agentEventDeliveryKey'
+ | 'agentEventBindingId'
| 'agentEventExpectedAction'
| 'agentEventLegacyTurnToken'
| 'scheduleId'
diff --git a/packages/api/src/stream/metadata.ts b/packages/api/src/stream/metadata.ts
index 52acfd3a70..745b9c898a 100644
--- a/packages/api/src/stream/metadata.ts
+++ b/packages/api/src/stream/metadata.ts
@@ -39,6 +39,9 @@ export function sanitizeJobMetadata(metadata: Partial<GenerationJobMetadata>): J
if (metadata.agentEventDeliveryKey) {
patch.agentEventDeliveryKey = metadata.agentEventDeliveryKey;
}
+ if (metadata.agentEventBindingId) {
+ patch.agentEventBindingId = metadata.agentEventBindingId;
+ }
if (metadata.agentEventExpectedAction) {
patch.agentEventExpectedAction = metadata.agentEventExpectedAction;
}
diff --git a/packages/api/src/types/stream.ts b/packages/api/src/types/stream.ts
index c8d63e3114..d52711640f 100644
--- a/packages/api/src/types/stream.ts
+++ b/packages/api/src/types/stream.ts
@@ -44,6 +44,8 @@ export interface GenerationJobMetadata {
isTemporary?: boolean;
/** Exact durable delivery whose accepted continuation created this generation. */
agentEventDeliveryKey?: string;
+ /** Trusted actor binding copied from the authenticated delivery envelope. */
+ agentEventBindingId?: string;
/** Optional action evidence contract declared by the authenticated event source. */
agentEventExpectedAction?: AgentTriggerExpectedAction;
/** Exact durable legacy-turn fence carried across a HITL pause/resume. */
diff --git a/packages/data-provider/src/config.spec.ts b/packages/data-provider/src/config.spec.ts
index 7fc011b59b..a508935916 100644
--- a/packages/data-provider/src/config.spec.ts
+++ b/packages/data-provider/src/config.spec.ts
@@ -78,6 +78,7 @@ describe('agent event runtime config', () => {
coalescing: true,
actorMailbox: true,
checkpointForks: true,
+ durableReceipts: true,
selfUrl: 'https://triggers.internal',
},
},
@@ -97,6 +98,7 @@ describe('agent event runtime config', () => {
coalescing: true,
actorMailbox: true,
checkpointForks: true,
+ durableReceipts: true,
selfUrl: 'https://triggers.internal',
});
expect(result.data.rateLimits?.agentEvents).toEqual({
diff --git a/packages/data-provider/src/config.ts b/packages/data-provider/src/config.ts
index cfa6927027..f7c6d81154 100644
--- a/packages/data-provider/src/config.ts
+++ b/packages/data-provider/src/config.ts
@@ -1058,6 +1058,11 @@ export const agentsEndpointSchema = baseEndpointSchema
* per-invocation forks. Keep off until every API worker runs an SDK
* and host adapter that understand the fork lifecycle. */
checkpointForks: z.boolean().optional(),
+ /** Admit checkpoint-forked external actions through the durable
+ * delivery receipt protocol. Enable only after every API replica
+ * runs the token-fenced receipt implementation and pre-upgrade
+ * actor deliveries have drained. */
+ durableReceipts: z.boolean().optional(),
/** Optional trusted origin for in-process trigger delivery. The bound
* listener remains the default and is safer for most deployments. */
selfUrl: z.string().url().optional(),
diff --git a/packages/data-schemas/src/index.ts b/packages/data-schemas/src/index.ts
index b63bc6a646..4cd74f5f45 100644
--- a/packages/data-schemas/src/index.ts
+++ b/packages/data-schemas/src/index.ts
@@ -35,6 +35,8 @@ export {
MAX_AUDIT_VERIFY_ROWS,
MAX_TOOL_FAVORITES,
AgentTriggerDeliveryConflictError,
+ recordAgentEventActorReceiptMetric,
+ setAgentEventActorReceiptMetricObserver,
MCPAuthorityProofError,
MAX_MCP_AUTHORITY_TARGETS,
createMCPAuthorityBootRevision,
diff --git a/packages/data-schemas/src/methods/conversation.spec.ts b/packages/data-schemas/src/methods/conversation.spec.ts
index d8a519c205..5e9e12e914 100644
--- a/packages/data-schemas/src/methods/conversation.spec.ts
+++ b/packages/data-schemas/src/methods/conversation.spec.ts
@@ -10,11 +10,7 @@ import type {
UpdateResult,
} from 'mongodb';
import type { IAgentEventActorReconciliation, IChatProject, IConversation } from '../types';
-import {
- AGENT_EVENT_ACTOR_RECEIPT_LIMIT,
- ConversationMethods,
- createConversationMethods,
-} from './conversation';
+import { ConversationMethods, createConversationMethods } from './conversation';
import { tenantStorage, runAsSystem } from '~/config/tenantContext';
import { createModels } from '../models';
@@ -4238,7 +4234,7 @@ describe('Conversation Operations', () => {
);
});
- it('transitions one actor lifecycle and retains exact settled receipts', async () => {
+ it('retains exact legacy settled receipts until delivery-ledger migration', async () => {
const conversationId = uuidv4();
await Conversation.create({
conversationId,
@@ -4548,7 +4544,258 @@ describe('Conversation Operations', () => {
expect(visible).not.toHaveProperty('agentEventActorReconciliations');
});
- it('prunes only settled receipts older than the retention window on admission', async () => {
+ it('confirms action admission only on the exact pending lifecycle', async () => {
+ const conversationId = uuidv4();
+ await Conversation.create({
+ conversationId,
+ user: 'actor-admission-user',
+ endpoint: EModelEndpoint.agents,
+ agentEventBinding: {
+ bindingId: `evtbind_${'a'.repeat(48)}`,
+ sourceKeyId: 'key-a',
+ actorId: 'player-a',
+ },
+ subagentThread: {
+ rootConversationId: 'parent',
+ parentConversationId: 'parent',
+ parentMessageId: 'parent-message',
+ parentToolCallId: 'event-binding',
+ parentAgentId: 'agent-director',
+ subagentType: 'agent-player',
+ subagentKind: 'agent',
+ depth: 1,
+ },
+ });
+ const pending = {
+ invocationId: 'event-admission',
+ status: 'invocation_pending' as const,
+ checkpoint: {
+ threadId: conversationId,
+ checkpointNs: 'event-actor/admission',
+ },
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(),
+ };
+
+ await expect(
+ methods.recordAgentEventActorReconciliation({
+ user: 'actor-admission-user',
+ conversationId,
+ reconciliation: pending,
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.recordAgentEventActorReconciliation({
+ user: 'actor-admission-user',
+ conversationId,
+ reconciliation: { ...pending, actionAdmitted: true },
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.recordAgentEventActorReconciliation({
+ user: 'actor-admission-user',
+ conversationId,
+ reconciliation: { ...pending, actionAdmitted: true },
+ }),
+ ).resolves.toBe(false);
+ await expect(
+ methods.getAgentEventActorSnapshot({
+ user: 'actor-admission-user',
+ conversationId,
+ }),
+ ).resolves.toMatchObject({
+ reconciliations: [
+ expect.objectContaining({
+ invocationId: 'event-admission',
+ status: 'invocation_pending',
+ actionAdmitted: true,
+ }),
+ ],
+ });
+ await expect(
+ methods.resolveAgentEventActorReconciliation({
+ user: 'actor-admission-user',
+ conversationId,
+ invocationId: pending.invocationId,
+ checkpoint: pending.checkpoint,
+ expectedActionAdmitted: false,
+ resolution: 'invocation_abandoned',
+ }),
+ ).resolves.toBe(false);
+ await expect(
+ methods.resolveAgentEventActorReconciliation({
+ user: 'actor-admission-user',
+ conversationId,
+ invocationId: pending.invocationId,
+ checkpoint: pending.checkpoint,
+ expectedActionAdmitted: true,
+ resolution: 'invocation_abandoned',
+ }),
+ ).resolves.toBe(true);
+ });
+
+ it('does not confirm a stale marker after a checkpoint-distinct successor wins', async () => {
+ const conversationId = uuidv4();
+ const invocationId = 'event-stale-confirmation';
+ const original = {
+ invocationId,
+ status: 'invocation_pending' as const,
+ checkpoint: {
+ threadId: conversationId,
+ checkpointId: 'checkpoint-original',
+ checkpointNs: 'event-actor/original',
+ },
+ action: { toolName: 'submit_move', toolCallId: 'call-original' },
+ observedAt: new Date(),
+ };
+ const successor = {
+ ...original,
+ checkpoint: {
+ threadId: conversationId,
+ checkpointId: 'checkpoint-successor',
+ checkpointNs: 'event-actor/successor',
+ },
+ action: { toolName: 'submit_move', toolCallId: 'call-successor' },
+ };
+ await Conversation.create({
+ conversationId,
+ user: 'actor-stale-confirmation-user',
+ endpoint: EModelEndpoint.agents,
+ agentEventBinding: {
+ bindingId: `evtbind_${'b'.repeat(48)}`,
+ sourceKeyId: 'key-a',
+ actorId: 'player-a',
+ },
+ subagentThread: {
+ rootConversationId: 'parent',
+ parentConversationId: 'parent',
+ parentMessageId: 'parent-message',
+ parentToolCallId: 'event-binding',
+ parentAgentId: 'agent-director',
+ subagentType: 'agent-player',
+ subagentKind: 'agent',
+ depth: 1,
+ },
+ agentEventActorReconciliations: [successor],
+ });
+ const staleRead = jest.spyOn(Conversation, 'findOne').mockImplementationOnce(
+ () =>
+ ({
+ select: () => ({
+ lean: () => Promise.resolve({ agentEventActorReconciliations: [original] }),
+ }),
+ }) as never,
+ );
+
+ await expect(
+ methods.recordAgentEventActorReconciliation({
+ user: 'actor-stale-confirmation-user',
+ conversationId,
+ reconciliation: { ...original, actionAdmitted: true },
+ }),
+ ).resolves.toBe(false);
+ staleRead.mockRestore();
+ await expect(
+ methods.getAgentEventActorSnapshot({
+ user: 'actor-stale-confirmation-user',
+ conversationId,
+ }),
+ ).resolves.toMatchObject({
+ reconciliations: [
+ expect.objectContaining({
+ checkpoint: expect.objectContaining({ checkpointId: 'checkpoint-successor' }),
+ action: expect.objectContaining({ toolCallId: 'call-successor' }),
+ }),
+ ],
+ });
+ });
+
+ it('clears an exact reconciled invocation without retaining a conversation receipt', async () => {
+ const conversationId = uuidv4();
+ const observedAt = new Date('2026-08-26T12:00:00.000Z');
+ const checkpoint = {
+ threadId: conversationId,
+ checkpointId: 'checkpoint-one',
+ checkpointNs: 'event-actor/event-one',
+ };
+ await Conversation.create({
+ conversationId,
+ user: 'actor-receipt-user',
+ endpoint: EModelEndpoint.agents,
+ agent_id: 'agent-player',
+ agentEventBinding: {
+ bindingId: `evtbind_${'f'.repeat(48)}`,
+ sourceKeyId: 'key-a',
+ actorId: 'player-a',
+ },
+ subagentThread: {
+ rootConversationId: 'parent',
+ parentConversationId: 'parent',
+ parentMessageId: 'parent-message',
+ parentToolCallId: 'event-binding',
+ parentAgentId: 'agent-director',
+ subagentType: 'agent-player',
+ subagentKind: 'agent',
+ depth: 1,
+ },
+ });
+ await methods.recordAgentEventActorReconciliation({
+ user: 'actor-receipt-user',
+ conversationId,
+ reconciliation: {
+ invocationId: 'event-one',
+ status: 'invocation_pending',
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ observedAt,
+ },
+ });
+ await methods.commitAgentEventActorState({
+ user: 'actor-receipt-user',
+ conversationId,
+ invocationId: 'event-one',
+ expectedEpoch: 0,
+ action: { toolName: 'submit_move' },
+ checkpoint,
+ });
+ await methods.recordAgentEventActorReconciliation({
+ user: 'actor-receipt-user',
+ conversationId,
+ reconciliation: {
+ invocationId: 'event-one',
+ status: 'history_persisted',
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ observedAt,
+ },
+ });
+
+ await expect(
+ methods.getAgentEventActorReconciliationStorageMetrics(
+ new Date(observedAt.getTime() + 12_000),
+ ),
+ ).resolves.toEqual({ pending: 1, oldestPendingAgeSeconds: 12 });
+
+ await expect(
+ methods.clearAgentEventActorReconciliation({
+ user: 'actor-receipt-user',
+ conversationId,
+ invocationId: 'event-one',
+ checkpoint,
+ resolution: 'checkpoint_verified',
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.getAgentEventActorSnapshot({ user: 'actor-receipt-user', conversationId }),
+ ).resolves.toMatchObject({ reconciliations: [] });
+ await expect(
+ methods.getAgentEventActorReconciliationStorageMetrics(
+ new Date(observedAt.getTime() + 12_000),
+ ),
+ ).resolves.toEqual({ pending: 0, oldestPendingAgeSeconds: 0 });
+ });
+
+ it('does not prune legacy settled receipts during new invocation admission', async () => {
const conversationId = uuidv4();
await Conversation.create({
conversationId,
@@ -4637,9 +4884,9 @@ describe('Conversation Operations', () => {
},
}),
).resolves.toBe(true);
- /** The expired tombstone is evicted by age; the recent receipt and the
- * newly admitted fence survive, and the expired id becomes admissible
- * again only because its whole stale-owner window has passed. */
+ /** Conversation admission no longer owns receipt retention. Lazy
+ * delivery-ledger migration removes either legacy receipt only after its
+ * authoritative row is durable. */
await expect(
methods.getAgentEventActorSnapshot({
user: 'actor-retention-user',
@@ -4650,12 +4897,365 @@ describe('Conversation Operations', () => {
epoch: 0,
legacyTurn: null,
reconciliations: [
+ expect.objectContaining({ invocationId: 'event-old', status: 'settled' }),
expect.objectContaining({ invocationId: 'event-recent', status: 'settled' }),
expect.objectContaining({ invocationId: 'event-next', status: 'invocation_pending' }),
],
});
});
+ it('expires dormant legacy settled receipts independently of matching delivery replay', async () => {
+ const oldConversationId = uuidv4();
+ const recentConversationId = uuidv4();
+ const settled = (invocationId: string, observedAt: Date) => ({
+ invocationId,
+ status: 'settled' as const,
+ resolution: 'action_compensated' as const,
+ checkpoint: {
+ threadId: invocationId === 'old' ? oldConversationId : recentConversationId,
+ checkpointNs: `event-actor/${invocationId}`,
+ },
+ action: { toolName: 'submit_move' },
+ observedAt,
+ });
+ const now = new Date();
+ await Conversation.create([
+ {
+ conversationId: oldConversationId,
+ user: 'actor-legacy-expiry-user',
+ endpoint: EModelEndpoint.agents,
+ agentEventActorReconciliations: [
+ settled('old', new Date(now.getTime() - 91 * 24 * 60 * 60_000)),
+ ],
+ },
+ {
+ conversationId: recentConversationId,
+ user: 'actor-legacy-expiry-user',
+ endpoint: EModelEndpoint.agents,
+ agentEventActorReconciliations: [
+ settled('recent', new Date(now.getTime() - 89 * 24 * 60 * 60_000)),
+ ],
+ },
+ ]);
+
+ await expect(methods.expireLegacyAgentEventActorReceipts(now, 1)).resolves.toBe(1);
+ await expect(
+ Conversation.findOne({ conversationId: oldConversationId })
+ .select('+agentEventActorReconciliations')
+ .lean(),
+ ).resolves.toMatchObject({ agentEventActorReconciliations: [] });
+ await expect(
+ Conversation.findOne({ conversationId: recentConversationId })
+ .select('+agentEventActorReconciliations')
+ .lean(),
+ ).resolves.toMatchObject({
+ agentEventActorReconciliations: [expect.objectContaining({ invocationId: 'recent' })],
+ });
+ });
+
+ it('retains an expired legacy receipt while its delivery handling is nonterminal', async () => {
+ const conversationId = uuidv4();
+ const userId = new mongoose.Types.ObjectId();
+ const now = new Date();
+ await Conversation.create({
+ conversationId,
+ user: userId.toString(),
+ endpoint: EModelEndpoint.agents,
+ agentEventActorReconciliations: [
+ {
+ invocationId: 'protected-old',
+ status: 'settled',
+ resolution: 'checkpoint_verified',
+ checkpoint: {
+ threadId: conversationId,
+ checkpointId: 'checkpoint-old',
+ checkpointNs: 'event-actor/protected-old',
+ },
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(now.getTime() - 91 * 24 * 60 * 60_000),
+ },
+ ],
+ });
+ const Delivery = mongoose.models.AgentTriggerDelivery;
+ await Delivery.create({
+ deliveryKey: 'protected-old',
+ fingerprint: 'protected-old-fingerprint',
+ orderingKey: 'protected-old-lane',
+ laneSequence: 1,
+ envelope: {},
+ user: userId,
+ status: 'succeeded',
+ attempts: 1,
+ availableAt: now,
+ awaitTerminalHandling: true,
+ handling: {
+ status: 'started',
+ conversationId,
+ streamId: conversationId,
+ generationCreatedAt: now.getTime(),
+ startedAt: now,
+ },
+ });
+
+ await expect(methods.expireLegacyAgentEventActorReceipts(now)).resolves.toBe(0);
+ await expect(
+ Conversation.findOne({ conversationId }).select('+agentEventActorReconciliations').lean(),
+ ).resolves.toMatchObject({
+ agentEventActorReconciliations: [
+ expect.objectContaining({ invocationId: 'protected-old' }),
+ ],
+ });
+
+ await Delivery.updateOne(
+ { deliveryKey: 'protected-old' },
+ { $set: { 'handling.status': 'applied', 'handling.settledAt': now } },
+ );
+ const updatedAtBeforeExpiry = (
+ await Conversation.findOne({ conversationId }).select('updatedAt').lean()
+ )?.updatedAt;
+ /** The first call reaches the end of the protected page and resets the
+ * cursor; the next bounded sweep revisits it after handling is terminal. */
+ await expect(methods.expireLegacyAgentEventActorReceipts(now)).resolves.toBe(0);
+ await expect(methods.expireLegacyAgentEventActorReceipts(now)).resolves.toBe(1);
+ await expect(
+ Conversation.findOne({ conversationId }).select('updatedAt').lean(),
+ ).resolves.toMatchObject({ updatedAt: updatedAtBeforeExpiry });
+ });
+
+ it('pages raw conversations before filtering and advances past protected receipts', async () => {
+ const freshConversationId = uuidv4();
+ const protectedConversationId = uuidv4();
+ const removableConversationId = uuidv4();
+ const userId = new mongoose.Types.ObjectId();
+ const now = new Date();
+ const oldReceipt = (invocationId: string, threadId: string) => ({
+ invocationId,
+ status: 'settled' as const,
+ resolution: 'action_compensated' as const,
+ checkpoint: { threadId, checkpointNs: `event-actor/${invocationId}` },
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(now.getTime() - 91 * 24 * 60 * 60_000),
+ });
+ await Conversation.create([
+ {
+ conversationId: freshConversationId,
+ user: userId.toString(),
+ endpoint: EModelEndpoint.agents,
+ agentEventActorReconciliations: [
+ { ...oldReceipt('fresh-first', freshConversationId), observedAt: now },
+ ],
+ },
+ {
+ conversationId: protectedConversationId,
+ user: userId.toString(),
+ endpoint: EModelEndpoint.agents,
+ agentEventActorReconciliations: [oldReceipt('protected-first', protectedConversationId)],
+ },
+ {
+ conversationId: removableConversationId,
+ user: userId.toString(),
+ endpoint: EModelEndpoint.agents,
+ agentEventActorReconciliations: [oldReceipt('removable-second', removableConversationId)],
+ },
+ ]);
+ const Delivery = mongoose.models.AgentTriggerDelivery;
+ await Delivery.create({
+ deliveryKey: 'protected-first',
+ fingerprint: 'protected-first-fingerprint',
+ orderingKey: 'protected-first-lane',
+ laneSequence: 1,
+ envelope: {},
+ user: userId,
+ status: 'succeeded',
+ attempts: 1,
+ availableAt: now,
+ awaitTerminalHandling: true,
+ });
+
+ /** The first raw page contains no expired receipt. The second contains
+ * a protected receipt, and only the third reaches removable work. */
+ await expect(methods.expireLegacyAgentEventActorReceipts(now, 1)).resolves.toBe(0);
+ await expect(methods.expireLegacyAgentEventActorReceipts(now, 1)).resolves.toBe(0);
+ await expect(methods.expireLegacyAgentEventActorReceipts(now, 1)).resolves.toBe(1);
+ await expect(
+ Conversation.findOne({ conversationId: protectedConversationId })
+ .select('+agentEventActorReconciliations')
+ .lean(),
+ ).resolves.toMatchObject({
+ agentEventActorReconciliations: [
+ expect.objectContaining({ invocationId: 'protected-first' }),
+ ],
+ });
+ await expect(
+ Conversation.findOne({ conversationId: removableConversationId })
+ .select('+agentEventActorReconciliations')
+ .lean(),
+ ).resolves.toMatchObject({ agentEventActorReconciliations: [] });
+ });
+
+ it('clears a compensated headless lifecycle without creating a partial actor head', async () => {
+ const conversationId = uuidv4();
+ const checkpoint = {
+ threadId: conversationId,
+ checkpointNs: 'event-actor/headless-compensation',
+ };
+ await Conversation.create({
+ conversationId,
+ user: 'actor-headless-clear-user',
+ endpoint: EModelEndpoint.agents,
+ agent_id: 'agent-player',
+ agentEventBinding: {
+ bindingId: `evtbind_${'d'.repeat(48)}`,
+ sourceKeyId: 'key-a',
+ actorId: 'player-a',
+ },
+ subagentThread: {
+ rootConversationId: 'parent',
+ parentConversationId: 'parent',
+ parentMessageId: 'parent-message',
+ parentToolCallId: 'event-binding',
+ parentAgentId: 'agent-director',
+ subagentType: 'agent-player',
+ subagentKind: 'agent',
+ depth: 1,
+ },
+ agentEventActorReconciliations: [
+ {
+ invocationId: 'headless-compensation',
+ status: 'commit_conflict',
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(),
+ },
+ ],
+ });
+
+ await expect(
+ methods.clearAgentEventActorReconciliation({
+ user: 'actor-headless-clear-user',
+ conversationId,
+ invocationId: 'headless-compensation',
+ checkpoint,
+ resolution: 'action_compensated',
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ Conversation.findOne({ conversationId })
+ .select('+agentEventActor +agentEventActorReconciliations')
+ .lean(),
+ ).resolves.toMatchObject({ agentEventActorReconciliations: [] });
+ const stored = await Conversation.findOne({ conversationId })
+ .select('+agentEventActor')
+ .lean();
+ expect(stored?.agentEventActor).toBeUndefined();
+ });
+
+ it('clears an exact settled legacy receipt after delivery-ledger migration', async () => {
+ const conversationId = uuidv4();
+ const checkpoint = {
+ threadId: conversationId,
+ checkpointId: 'checkpoint-migrated',
+ checkpointNs: 'event-actor/migrated',
+ };
+ await Conversation.create({
+ conversationId,
+ user: 'actor-migrated-clear-user',
+ endpoint: EModelEndpoint.agents,
+ agent_id: 'agent-player',
+ agentEventBinding: {
+ bindingId: `evtbind_${'c'.repeat(48)}`,
+ sourceKeyId: 'key-a',
+ actorId: 'player-a',
+ },
+ subagentThread: {
+ rootConversationId: 'parent',
+ parentConversationId: 'parent',
+ parentMessageId: 'parent-message',
+ parentToolCallId: 'event-binding',
+ parentAgentId: 'agent-director',
+ subagentType: 'agent-player',
+ subagentKind: 'agent',
+ depth: 1,
+ },
+ agentEventActor: { generation: 1, checkpoint },
+ agentEventActorReconciliations: [
+ {
+ invocationId: 'migrated',
+ status: 'settled',
+ resolution: 'checkpoint_verified',
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(),
+ },
+ ],
+ });
+
+ await expect(
+ methods.clearAgentEventActorReconciliation({
+ user: 'actor-migrated-clear-user',
+ conversationId,
+ invocationId: 'migrated',
+ checkpoint,
+ resolution: 'checkpoint_verified',
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.getAgentEventActorSnapshot({
+ user: 'actor-migrated-clear-user',
+ conversationId,
+ }),
+ ).resolves.toMatchObject({ reconciliations: [] });
+ });
+
+ it('does not compensate an action that has not passed invocation admission', async () => {
+ const conversationId = uuidv4();
+ const checkpoint = {
+ threadId: conversationId,
+ checkpointNs: 'event-actor/not-admitted',
+ };
+ await Conversation.create({
+ conversationId,
+ user: 'actor-unadmitted-user',
+ endpoint: EModelEndpoint.agents,
+ agent_id: 'agent-player',
+ agentEventBinding: {
+ bindingId: `evtbind_${'b'.repeat(48)}`,
+ sourceKeyId: 'key-a',
+ actorId: 'player-a',
+ },
+ subagentThread: {
+ rootConversationId: 'parent',
+ parentConversationId: 'parent',
+ parentMessageId: 'parent-message',
+ parentToolCallId: 'event-binding',
+ parentAgentId: 'agent-director',
+ subagentType: 'agent-player',
+ subagentKind: 'agent',
+ depth: 1,
+ },
+ agentEventActorReconciliations: [
+ {
+ invocationId: 'not-admitted',
+ actionAdmitted: true,
+ status: 'invocation_pending',
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ observedAt: new Date(),
+ },
+ ],
+ });
+
+ await expect(
+ methods.resolveAgentEventActorReconciliation({
+ user: 'actor-unadmitted-user',
+ conversationId,
+ invocationId: 'not-admitted',
+ checkpoint,
+ resolution: 'action_compensated',
+ }),
+ ).resolves.toBe(false);
+ });
+
it('versions every legacy invalidation so a stale prepared fork cannot commit past it', async () => {
const conversationId = uuidv4();
await Conversation.create({
@@ -4972,7 +5572,7 @@ describe('Conversation Operations', () => {
}
});
- it('refuses new invocations rather than evicting unexpired receipts at the cap', async () => {
+ it('does not let a legacy settled journal cap block a healthy actor', async () => {
const conversationId = uuidv4();
await Conversation.create({
conversationId,
@@ -4996,7 +5596,7 @@ describe('Conversation Operations', () => {
},
});
const bulkReceipts = (observedAt: Date) =>
- Array.from({ length: AGENT_EVENT_ACTOR_RECEIPT_LIMIT }, (_, index) => ({
+ Array.from({ length: 1025 }, (_, index) => ({
invocationId: `bulk-${index}`,
status: 'settled' as const,
resolution: 'checkpoint_verified' as const,
@@ -5029,9 +5629,7 @@ describe('Conversation Operations', () => {
observedAt: new Date(),
},
});
- /** A journal full of unexpired receipts fails closed: no receipt inside
- * its stale-owner window may ever be evicted to make room. */
- await expect(admit()).resolves.toBe(false);
+ await expect(admit()).resolves.toBe(true);
await expect(
methods.getAgentEventActorSnapshot({
user: 'actor-cap-user',
@@ -5041,35 +5639,10 @@ describe('Conversation Operations', () => {
expect.objectContaining({
reconciliations: expect.arrayContaining([
expect.objectContaining({ invocationId: 'bulk-0' }),
+ expect.objectContaining({ invocationId: 'event-at-cap' }),
]),
}),
);
- await Conversation.updateOne(
- { conversationId },
- {
- $set: {
- agentEventActorReconciliations: bulkReceipts(
- new Date(Date.now() - 8 * 24 * 60 * 60 * 1000),
- ),
- },
- },
- { timestamps: false },
- );
- /** Once the window passes, expired receipts prune and admission resumes. */
- await expect(admit()).resolves.toBe(true);
- await expect(
- methods.getAgentEventActorSnapshot({
- user: 'actor-cap-user',
- conversationId,
- }),
- ).resolves.toEqual({
- state: null,
- epoch: 0,
- legacyTurn: null,
- reconciliations: [
- expect.objectContaining({ invocationId: 'event-at-cap', status: 'invocation_pending' }),
- ],
- });
});
it('admits one cross-replica owner and fences renewal and release by token', async () => {
diff --git a/packages/data-schemas/src/methods/conversation.ts b/packages/data-schemas/src/methods/conversation.ts
index b19188c073..973f005e3c 100644
--- a/packages/data-schemas/src/methods/conversation.ts
+++ b/packages/data-schemas/src/methods/conversation.ts
@@ -1,5 +1,5 @@
import { RetentionMode } from 'librechat-data-provider';
-import type { FilterQuery, Model, SortOrder, Types } from 'mongoose';
+import type { AnyBulkWriteOperation, FilterQuery, Model, SortOrder, Types } from 'mongoose';
import type { DeleteResult } from 'mongoose';
import type {
IAgentEventActorCheckpoint,
@@ -7,6 +7,7 @@ import type {
IAgentEventActorSnapshot,
IAgentEventActorState,
IAgentEventBindingRecord,
+ IAgentTriggerDeliveryDocument,
AppConfig,
IChatProjectDocument,
IActiveSubagentThreadLease,
@@ -30,6 +31,8 @@ import { isValidObjectIdString } from '~/utils/objectId';
import { decrementTagCounts } from './conversationTag';
import logger from '~/config/winston';
+const AGENT_EVENT_ACTOR_RECEIPT_RETENTION_MS = 90 * 24 * 60 * 60_000;
+
type ConversationUpdateResult = {
value:
| (IConversation & {
@@ -66,25 +69,10 @@ export type AgentEventActorCommitResult =
}
| { status: 'stale'; state?: IAgentEventActorState };
-/**
- * How long a settled receipt keeps tombstoning its invocation id. A stale
- * same-id owner is bounded by time, not by how many newer invocations settle,
- * so eviction is primarily age-based: a receipt becomes prunable only once no
- * delayed duplicate of its invocation could still be admitted — far beyond any
- * generation, job, or delivery-retry lifetime.
- */
-const AGENT_EVENT_ACTOR_RECEIPT_RETENTION_MS = 7 * 24 * 60 * 60 * 1000;
-
-/**
- * Absolute size bound for the private invocation journal so a pathological
- * actor cannot grow its conversation document toward Mongo's document limit.
- * Never an eviction quota: a receipt inside its retention window is never
- * discarded. When the journal holds this many unexpired receipts, admission
- * of NEW invocations is refused (fail closed) until receipts age out, so
- * document integrity and same-id duplicate protection are both invariants
- * rather than a trade.
- */
-export const AGENT_EVENT_ACTOR_RECEIPT_LIMIT = 1024;
+export interface AgentEventActorReconciliationStorageMetrics {
+ pending: number;
+ oldestPendingAgeSeconds: number;
+}
const ARCHIVE_CONVERSATION_BATCH_SIZE = 500;
const PROJECT_STATS_REFRESH_CONCURRENCY = 10;
@@ -282,12 +270,25 @@ export interface ConversationMethods {
tenantId?: string;
invocationId: string;
checkpoint: IAgentEventActorReconciliation['checkpoint'];
+ expectedActionAdmitted?: boolean;
resolution:
| 'checkpoint_verified'
| 'action_compensated'
| 'history_repaired'
| 'invocation_abandoned';
}): Promise<boolean>;
+ clearAgentEventActorReconciliation(input: {
+ user: string;
+ conversationId: string;
+ tenantId?: string;
+ invocationId: string;
+ checkpoint: IAgentEventActorReconciliation['checkpoint'];
+ resolution: 'checkpoint_verified' | 'action_compensated' | 'history_repaired';
+ }): Promise<boolean>;
+ getAgentEventActorReconciliationStorageMetrics(
+ now: Date,
+ ): Promise<AgentEventActorReconciliationStorageMetrics>;
+ expireLegacyAgentEventActorReceipts(now: Date, limit?: number): Promise<number>;
reserveSubagentThread(input: {
user: string;
conversationId: string;
@@ -349,6 +350,8 @@ export function createConversationMethods(
mongoose: typeof import('mongoose'),
messageMethods?: Pick<MessageMethods, 'getMessages' | 'deleteMessages'>,
): ConversationMethods {
+ let legacyReceiptExpiryCursor: Types.ObjectId | undefined;
+
function getMessageMethods() {
if (!messageMethods) {
throw new Error('Message methods not injected into conversation methods');
@@ -717,7 +720,18 @@ export function createConversationMethods(
const journal = existing?.agentEventActorReconciliations ?? [];
const current = journal.find((item) => item.invocationId === input.reconciliation.invocationId);
if (current != null) {
+ const confirmsActionAdmission =
+ current.status === 'invocation_pending' &&
+ current.actionAdmitted !== true &&
+ input.reconciliation.status === 'invocation_pending' &&
+ input.reconciliation.actionAdmitted === true &&
+ current.checkpoint.threadId === input.reconciliation.checkpoint.threadId &&
+ current.checkpoint.checkpointId === input.reconciliation.checkpoint.checkpointId &&
+ current.checkpoint.checkpointNs === input.reconciliation.checkpoint.checkpointNs &&
+ current.action.toolName === input.reconciliation.action.toolName &&
+ current.action.toolCallId === input.reconciliation.action.toolCallId;
const canTransition =
+ confirmsActionAdmission ||
(current?.status === 'invocation_pending' &&
input.reconciliation.status !== 'invocation_pending') ||
(current?.status === 'persistence_pending' &&
@@ -727,6 +741,7 @@ export function createConversationMethods(
input.reconciliation.status === 'history_persisted');
const sameIdentity =
current?.status === input.reconciliation.status &&
+ current.actionAdmitted === input.reconciliation.actionAdmitted &&
current.checkpoint.threadId === input.reconciliation.checkpoint.threadId &&
current.checkpoint.checkpointId === input.reconciliation.checkpoint.checkpointId &&
current.checkpoint.checkpointNs === input.reconciliation.checkpoint.checkpointNs &&
@@ -734,7 +749,7 @@ export function createConversationMethods(
current.action.toolCallId === input.reconciliation.action.toolCallId;
/** A pending record is an exclusive ownership fence, not an idempotent
* receipt. A second executor must not inherit the first owner's claim. */
- if (input.reconciliation.status === 'invocation_pending') {
+ if (input.reconciliation.status === 'invocation_pending' && !confirmsActionAdmission) {
return false;
}
if (!canTransition) {
@@ -747,6 +762,18 @@ export function createConversationMethods(
$elemMatch: {
invocationId: input.reconciliation.invocationId,
status: current!.status,
+ ...(confirmsActionAdmission && {
+ actionAdmitted: { $ne: true },
+ 'checkpoint.threadId': current.checkpoint.threadId,
+ 'checkpoint.checkpointNs': current.checkpoint.checkpointNs,
+ ...(current.checkpoint.checkpointId == null
+ ? { 'checkpoint.checkpointId': { $exists: false } }
+ : { 'checkpoint.checkpointId': current.checkpoint.checkpointId }),
+ 'action.toolName': current.action.toolName,
+ ...(current.action.toolCallId == null
+ ? { 'action.toolCallId': { $exists: false } }
+ : { 'action.toolCallId': current.action.toolCallId }),
+ }),
},
},
},
@@ -762,41 +789,6 @@ export function createConversationMethods(
if (input.reconciliation.status !== 'invocation_pending') {
return false;
}
- const receiptCutoff = new Date(Date.now() - AGENT_EVENT_ACTOR_RECEIPT_RETENTION_MS);
- if (
- journal.some(
- (item) => item.status === 'settled' && item.observedAt.getTime() < receiptCutoff.getTime(),
- )
- ) {
- await Conversation.updateOne(
- ownership,
- {
- $pull: {
- agentEventActorReconciliations: {
- status: 'settled',
- observedAt: { $lt: receiptCutoff },
- },
- },
- },
- { timestamps: false },
- );
- }
- /** A receipt inside its retention window may NEVER be evicted — a stale
- * same-id owner is bounded by time, and count-based eviction would reopen
- * duplicate execution at high event rates. When the journal is full of
- * unexpired receipts, refuse admission (fail closed) instead of trading
- * away tombstone protection; the actor resumes once receipts age out.
- * Admission is serialized by the no-active-row filter, so the journal can
- * exceed the cap by at most the single row admitted after this check. */
- const retainedReceipts = journal.filter(
- (item) => item.status === 'settled' && item.observedAt.getTime() >= receiptCutoff.getTime(),
- );
- if (retainedReceipts.length >= AGENT_EVENT_ACTOR_RECEIPT_LIMIT) {
- logger.error(
- `[conversation] Event actor receipt journal for ${input.conversationId} is full of unexpired receipts; refusing new invocation ${input.reconciliation.invocationId} until receipts age out`,
- );
- return false;
- }
const recorded = await Conversation.findOneAndUpdate(
{
...ownership,
@@ -829,6 +821,7 @@ export function createConversationMethods(
tenantId?: string;
invocationId: string;
checkpoint: IAgentEventActorReconciliation['checkpoint'];
+ expectedActionAdmitted?: boolean;
resolution:
| 'checkpoint_verified'
| 'action_compensated'
@@ -902,11 +895,30 @@ export function createConversationMethods(
});
return replay != null;
}
+ let expectedAdmissionFilter: Record<string, unknown> = {};
+ if (input.expectedActionAdmitted != null) {
+ expectedAdmissionFilter = input.expectedActionAdmitted
+ ? { actionAdmitted: true }
+ : { actionAdmitted: { $ne: true } };
+ }
const checkpointFilter: Record<string, unknown> = {
...exactCheckpoint,
...(input.resolution === 'invocation_abandoned'
- ? { status: 'invocation_pending' }
- : { status: { $ne: 'settled' } }),
+ ? {
+ status: 'invocation_pending',
+ ...expectedAdmissionFilter,
+ }
+ : {
+ status: {
+ $in: [
+ 'persistence_pending',
+ 'history_persisted',
+ 'commit_conflict',
+ 'commit_indeterminate',
+ 'persistence_failed',
+ ],
+ },
+ }),
};
const ownership = {
...owner,
@@ -967,6 +979,227 @@ export function createConversationMethods(
return replayed != null;
}
+ /** Removes an active lifecycle only after its terminal proof is durable elsewhere. */
+ async function clearAgentEventActorReconciliation(input: {
+ user: string;
+ conversationId: string;
+ tenantId?: string;
+ invocationId: string;
+ checkpoint: IAgentEventActorReconciliation['checkpoint'];
+ resolution: 'checkpoint_verified' | 'action_compensated' | 'history_repaired';
+ }): Promise<boolean> {
+ if (input.checkpoint.threadId !== input.conversationId) {
+ throw new Error('Event actor reconciliation changed its logical thread');
+ }
+ if (
+ input.resolution === 'checkpoint_verified' &&
+ (typeof input.checkpoint.checkpointId !== 'string' ||
+ input.checkpoint.checkpointId.length === 0)
+ ) {
+ return false;
+ }
+ const Conversation = mongoose.models.Conversation as Model<IConversation>;
+ const exactCheckpoint = {
+ invocationId: input.invocationId,
+ 'checkpoint.threadId': input.checkpoint.threadId,
+ 'checkpoint.checkpointNs': input.checkpoint.checkpointNs,
+ ...(input.checkpoint.checkpointId == null
+ ? { 'checkpoint.checkpointId': { $exists: false } }
+ : { 'checkpoint.checkpointId': input.checkpoint.checkpointId }),
+ };
+ const owner = {
+ user: input.user,
+ conversationId: input.conversationId,
+ subagentThread: { $exists: true },
+ agentEventBinding: { $exists: true },
+ ...subagentLeaseTenantFilter(input.tenantId),
+ ...activeExpirationFilter<IConversation>(),
+ };
+ const activeStatuses =
+ input.resolution === 'checkpoint_verified'
+ ? ['history_persisted']
+ : [
+ 'persistence_pending',
+ 'history_persisted',
+ 'commit_conflict',
+ 'commit_indeterminate',
+ 'persistence_failed',
+ ];
+ const lifecycle = {
+ ...exactCheckpoint,
+ $or: [
+ { status: { $in: activeStatuses } },
+ { status: 'settled', resolution: input.resolution },
+ ],
+ };
+ const clearedWithHead = await Conversation.findOneAndUpdate(
+ {
+ ...owner,
+ agentEventActor: { $exists: true },
+ ...(input.resolution === 'checkpoint_verified' && {
+ 'agentEventActor.checkpoint.threadId': input.checkpoint.threadId,
+ 'agentEventActor.checkpoint.checkpointId': input.checkpoint.checkpointId,
+ 'agentEventActor.checkpoint.checkpointNs': input.checkpoint.checkpointNs,
+ }),
+ agentEventActorReconciliations: { $elemMatch: lifecycle },
+ },
+ {
+ ...(input.resolution === 'checkpoint_verified'
+ ? {}
+ : { $set: { 'agentEventActor.requiresColdStart': true } }),
+ $pull: { agentEventActorReconciliations: lifecycle },
+ },
+ { new: true, timestamps: false },
+ )
+ .select('+agentEventActorReconciliations')
+ .lean<IConversation>();
+ if (clearedWithHead != null) {
+ return true;
+ }
+ if (input.resolution !== 'checkpoint_verified') {
+ const clearedWithoutHead = await Conversation.findOneAndUpdate(
+ {
+ ...owner,
+ agentEventActor: { $exists: false },
+ agentEventActorReconciliations: { $elemMatch: lifecycle },
+ },
+ { $pull: { agentEventActorReconciliations: lifecycle } },
+ { new: true, timestamps: false },
+ )
+ .select('+agentEventActorReconciliations')
+ .lean<IConversation>();
+ if (clearedWithoutHead != null) {
+ return true;
+ }
+ }
+ const conflict = await Conversation.exists({
+ ...owner,
+ agentEventActorReconciliations: { $elemMatch: { invocationId: input.invocationId } },
+ });
+ if (conflict != null) {
+ return false;
+ }
+ return (await Conversation.exists(owner)) != null;
+ }
+
+ async function getAgentEventActorReconciliationStorageMetrics(
+ now: Date,
+ ): Promise<AgentEventActorReconciliationStorageMetrics> {
+ const Conversation = mongoose.models.Conversation as Model<IConversation>;
+ const [row] = await Conversation.aggregate<{ pending: number; oldestObservedAt?: Date }>([
+ {
+ $match: {
+ agentEventActorReconciliations: {
+ $elemMatch: { status: { $ne: 'settled' } },
+ },
+ },
+ },
+ { $unwind: '$agentEventActorReconciliations' },
+ { $match: { 'agentEventActorReconciliations.status': { $ne: 'settled' } } },
+ {
+ $group: {
+ _id: null,
+ pending: { $sum: 1 },
+ oldestObservedAt: { $min: '$agentEventActorReconciliations.observedAt' },
+ },
+ },
+ ]);
+ const oldestObservedAt = row?.oldestObservedAt;
+ return {
+ pending: row?.pending ?? 0,
+ oldestPendingAgeSeconds:
+ oldestObservedAt == null
+ ? 0
+ : Math.max(0, (now.getTime() - oldestObservedAt.getTime()) / 1000),
+ };
+ }
+
+ /** Bounded mixed-version cleanup for terminal receipts embedded by older
+ * builds. New receipts expire through the delivery collection TTL index,
+ * but dormant legacy conversations need an independent retirement path. */
+ async function expireLegacyAgentEventActorReceipts(now: Date, limit = 100): Promise<number> {
+ if (Number.isNaN(now.getTime())) {
+ throw new TypeError('now must be a valid date');
+ }
+ const boundedLimit = Math.max(1, Math.min(Math.trunc(limit), 1_000));
+ const cutoff = new Date(now.getTime() - AGENT_EVENT_ACTOR_RECEIPT_RETENTION_MS);
+ const Conversation = mongoose.models.Conversation as Model<IConversation>;
+ const Delivery = mongoose.models.AgentTriggerDelivery as
+ | Model<IAgentTriggerDeliveryDocument>
+ | undefined;
+ if (Delivery == null) {
+ return 0;
+ }
+ /** Advance a bounded candidate page before querying delivery protection.
+ * Reaching the end resets the cursor so newly expiry-eligible older rows
+ * enter the next sweep without making any one pass collection-wide. */
+ const candidates = await Conversation.find({
+ ...(legacyReceiptExpiryCursor == null ? {} : { _id: { $gt: legacyReceiptExpiryCursor } }),
+ })
+ .select('_id +agentEventActorReconciliations')
+ .sort({ _id: 1 })
+ .limit(boundedLimit)
+ .lean<
+ Array<Pick<IConversation, 'agentEventActorReconciliations'> & { _id: Types.ObjectId }>
+ >();
+ if (candidates.length === 0) {
+ legacyReceiptExpiryCursor = undefined;
+ return 0;
+ }
+ legacyReceiptExpiryCursor = candidates[candidates.length - 1]._id;
+ const expiredInvocationIds = [
+ ...new Set(
+ candidates.flatMap((candidate) =>
+ (candidate.agentEventActorReconciliations ?? [])
+ .filter((receipt) => receipt.status === 'settled' && receipt.observedAt <= cutoff)
+ .map((receipt) => receipt.invocationId),
+ ),
+ ),
+ ];
+ const protectedInvocationIds = new Set(
+ await Delivery.find({
+ deliveryKey: { $in: expiredInvocationIds },
+ $or: [{ handling: { $exists: false } }, { 'handling.status': 'started' }],
+ }).distinct<string>('deliveryKey'),
+ );
+ const operations = candidates.flatMap((candidate) => {
+ const removable = (candidate.agentEventActorReconciliations ?? [])
+ .filter(
+ (receipt) =>
+ receipt.status === 'settled' &&
+ receipt.observedAt <= cutoff &&
+ !protectedInvocationIds.has(receipt.invocationId),
+ )
+ .map((receipt) => receipt.invocationId);
+ return removable.length === 0
+ ? []
+ : [
+ {
+ updateOne: {
+ filter: { _id: candidate._id },
+ update: {
+ $pull: {
+ agentEventActorReconciliations: {
+ invocationId: { $in: removable },
+ status: 'settled',
+ observedAt: { $lte: cutoff },
+ },
+ },
+ },
+ },
+ },
+ ];
+ }) as unknown as AnyBulkWriteOperation[];
+ if (operations.length === 0) {
+ return 0;
+ }
+ const result = await tenantSafeBulkWrite(Conversation, operations, {
+ ordered: false,
+ timestamps: false,
+ });
+ return result.modifiedCount;
+ }
+
/** Creates immutable child lineage exactly once without overwriting a concurrent winner. */
async function reserveSubagentThread(input: {
user: string;
@@ -2345,6 +2578,9 @@ export function createConversationMethods(
completeAgentEventActorLegacyTurn,
recordAgentEventActorReconciliation,
resolveAgentEventActorReconciliation,
+ clearAgentEventActorReconciliation,
+ getAgentEventActorReconciliationStorageMetrics,
+ expireLegacyAgentEventActorReceipts,
reserveSubagentThread,
acquireSubagentThreadLease,
renewSubagentThreadLease,
diff --git a/packages/data-schemas/src/methods/index.ts b/packages/data-schemas/src/methods/index.ts
index cae8430995..eaa6a58cd5 100644
--- a/packages/data-schemas/src/methods/index.ts
+++ b/packages/data-schemas/src/methods/index.ts
@@ -60,6 +60,7 @@ import {
} from './message';
import {
createConversationMethods,
+ type AgentEventActorReconciliationStorageMetrics,
type ConversationMethods,
type ParentSubagentThreadRecord,
} from './conversation';
@@ -114,7 +115,11 @@ import { createScheduleMethods, type ScheduleMethods } from './schedule';
import {
createAgentTriggerDeliveryMethods,
AgentTriggerDeliveryConflictError,
+ recordAgentEventActorReceiptMetric,
+ setAgentEventActorReceiptMetricObserver,
type AgentTriggerDeliveryMethods,
+ type AgentEventActorReceiptMetric,
+ type AgentEventActorReceiptStorageMetrics,
} from './triggerDelivery';
import { createSkillSyncMethods, type SkillSyncMethods } from './skillSync';
import type {
@@ -406,6 +411,7 @@ export type {
SubagentThreadViewMessageRecord,
SubagentTaskResultClaim,
ConversationMethods,
+ AgentEventActorReconciliationStorageMetrics,
ChatProjectMethods,
TxMethods,
TransactionMethods,
@@ -426,6 +432,8 @@ export type {
UpsertSkillSyncCredentialInput,
SkillSyncMethods,
AgentTriggerDeliveryMethods,
+ AgentEventActorReceiptMetric,
+ AgentEventActorReceiptStorageMetrics,
ScheduleMethods,
AgentMethods,
ConfigMethods,
@@ -435,3 +443,5 @@ export type {
MCPAuthorityCredentialSourceDocument,
InsightsMethods,
};
+
+export { recordAgentEventActorReceiptMetric, setAgentEventActorReceiptMetricObserver };
diff --git a/packages/data-schemas/src/methods/triggerDelivery.spec.ts b/packages/data-schemas/src/methods/triggerDelivery.spec.ts
index 68772251a6..5ddb7b2f79 100644
--- a/packages/data-schemas/src/methods/triggerDelivery.spec.ts
+++ b/packages/data-schemas/src/methods/triggerDelivery.spec.ts
@@ -11,6 +11,8 @@ import type { IUser } from '~/types/user';
import {
AgentTriggerDeliveryConflictError,
createAgentTriggerDeliveryMethods,
+ recordAgentEventActorReceiptMetric,
+ setAgentEventActorReceiptMetricObserver,
type AgentTriggerDeliveryMethods,
} from './triggerDelivery';
import { createAgentTriggerLaneSequenceModel } from '../models/triggerLaneSequence';
@@ -52,6 +54,7 @@ afterAll(async () => {
}, DB_SETUP_TIMEOUT_MS);
beforeEach(async () => {
+ setAgentEventActorReceiptMetricObserver();
await Promise.all([
Delivery.deleteMany({}),
LaneSequence.deleteMany({}),
@@ -60,6 +63,10 @@ beforeEach(async () => {
]);
});
+afterAll(() => {
+ setAgentEventActorReceiptMetricObserver();
+});
+
function enqueueInput(
overrides: Partial<Parameters<typeof methods.enqueueAgentTriggerDelivery>[0]> = {},
) {
@@ -89,6 +96,19 @@ function transientFailure(overrides: Partial<AgentTriggerDeliveryFailure> = {})
}
describe('agent trigger delivery methods', () => {
+ it('never lets receipt metric failures affect storage correctness', () => {
+ setAgentEventActorReceiptMetricObserver(() => {
+ throw new Error('metrics unavailable');
+ });
+
+ expect(() =>
+ recordAgentEventActorReceiptMetric({
+ operation: 'settle',
+ outcome: 'success',
+ resolution: 'checkpoint_verified',
+ }),
+ ).not.toThrow();
+ });
it('enqueues idempotently and rejects key reuse with different content', async () => {
const input = enqueueInput();
const first = await methods.enqueueAgentTriggerDelivery(input);
@@ -771,8 +791,8 @@ describe('agent trigger delivery methods', () => {
status: 'succeeded',
batchRootRequeueCount: 1,
});
- await expect(methods.recoverAgentTriggerBatchReceipts()).rejects.toThrow(
- 'Not every agent trigger batch receipt could be settled',
+ await expect(methods.recoverAgentTriggerLanePublications()).rejects.toThrow(
+ 'Not every agent trigger batch receipt could be prepared for requeue',
);
await expect(Delivery.findById(memberId).lean()).resolves.toMatchObject({
status: 'succeeded',
@@ -781,7 +801,7 @@ describe('agent trigger delivery methods', () => {
expect((await Delivery.findById(root!._id).lean())?.batchMembersSettledAt).toBeUndefined();
});
- it('retries safely when batch requeue stops after preparing its members', async () => {
+ it('recovers batch requeue after claiming the root before member preparation', async () => {
const user = new mongoose.Types.ObjectId();
const coalesceUntil = new Date(Date.now() + 60_000);
const shared = {
@@ -816,24 +836,26 @@ describe('agent trigger delivery methods', () => {
error: transientFailure({ retryable: false }),
settledAt: coalesceUntil,
});
- const transition = jest.spyOn(Delivery, 'findOneAndUpdate').mockImplementationOnce(() => {
- throw new Error('process interrupted before root transition');
+ const transition = jest.spyOn(Delivery, 'updateMany').mockImplementationOnce(() => {
+ throw new Error('process interrupted before member preparation');
});
await expect(methods.requeueAgentTriggerDelivery(claim!.id, coalesceUntil)).rejects.toThrow(
- 'process interrupted before root transition',
+ 'process interrupted before member preparation',
);
transition.mockRestore();
await expect(Delivery.findById(claim!.id).lean()).resolves.toMatchObject({
- status: 'dead',
- requeueCount: 0,
+ status: 'staging',
+ requeueCount: 1,
});
const preparedMember = await Delivery.findOne({ batchRootId: claim!.id }).lean();
- expect(preparedMember).toMatchObject({ status: 'batched', batchRootRequeueCount: 1 });
+ expect(preparedMember).toMatchObject({ status: 'dead' });
- await expect(
- methods.requeueAgentTriggerDelivery(claim!.id, coalesceUntil),
- ).resolves.toMatchObject({ status: 'pending', requeueCount: 1 });
+ await expect(methods.recoverAgentTriggerLanePublications()).resolves.toBeGreaterThan(0);
+ await expect(Delivery.findById(claim!.id).lean()).resolves.toMatchObject({
+ status: 'pending',
+ requeueCount: 1,
+ });
await expect(Delivery.findById(preparedMember!._id).lean()).resolves.toMatchObject({
status: 'batched',
batchRootRequeueCount: 1,
@@ -1605,6 +1627,680 @@ describe('agent trigger delivery methods', () => {
});
});
+ it('settles the public outcome and exact actor receipt in one delivery row', async () => {
+ const queued = await methods.enqueueAgentTriggerDelivery(
+ enqueueInput({
+ awaitTerminalHandling: true,
+ envelope: {
+ mode: 'continue',
+ target: { bindingId: 'binding-1' },
+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },
+ },
+ }),
+ );
+ const claimed = await methods.claimNextAgentTriggerDelivery({
+ workerId: 'worker-1',
+ claimToken: 'claim-1',
+ now: START,
+ leaseUntil: new Date(START.getTime() + 60_000),
+ });
+ const attempt = await methods.beginAgentTriggerDeliveryAttempt({
+ id: claimed!.id,
+ workerId: 'worker-1',
+ claimToken: 'claim-1',
+ now: START,
+ });
+ const generationCreatedAt = START.getTime() + 1_000;
+ const actionAdmission = {
+ deliveryKey: queued.delivery.deliveryKey,
+ user: queued.delivery.user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-1',
+ conversationId: 'conversation-1',
+ admittedAt: new Date(generationCreatedAt + 500),
+ admissionId: 'admission-1',
+ };
+ /** The loopback child starts after the HTTP `started` response but before
+ * the delivery engine commits that result, so admission must work while
+ * the exact transport attempt is still leased. */
+ await expect(methods.admitAgentEventActorAction(actionAdmission)).resolves.toBe(true);
+ await expect(methods.hasAgentEventActorActionAdmission(actionAdmission)).resolves.toBe(true);
+ await expect(methods.admitAgentEventActorAction(actionAdmission)).resolves.toBe(false);
+ const successorAdmission = { ...actionAdmission, admissionId: 'admission-2' };
+ await expect(methods.admitAgentEventActorAction(successorAdmission)).resolves.toBe(false);
+ await expect(methods.releaseAgentEventActorAction(actionAdmission)).resolves.toBe(true);
+ await expect(methods.admitAgentEventActorAction(successorAdmission)).resolves.toBe(true);
+ /** The predecessor's delayed release cannot clear its successor. */
+ await expect(methods.releaseAgentEventActorAction(actionAdmission)).resolves.toBe(false);
+ await expect(methods.hasAgentEventActorActionAdmission(successorAdmission)).resolves.toBe(true);
+ await expect(methods.releaseAgentEventActorAction(successorAdmission)).resolves.toBe(true);
+ await expect(methods.hasAgentEventActorActionAdmission(actionAdmission)).resolves.toBe(false);
+ /** A pre-token worker's live admission must remain opaque but protected
+ * throughout a rolling upgrade. New workers may observe the fence, but
+ * cannot replace or release it without its (unavailable) owner token. */
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ { $set: { actorActionAdmittedAt: actionAdmission.admittedAt } },
+ );
+ await expect(methods.hasAgentEventActorActionAdmission(actionAdmission)).resolves.toBe(true);
+ await expect(methods.admitAgentEventActorAction(successorAdmission)).resolves.toBe(false);
+ await expect(methods.releaseAgentEventActorAction(actionAdmission)).resolves.toBe(false);
+ /** An old worker can also leave a newer worker's stale token behind, then
+ * write a fresh token-unaware timestamp. Timestamp ownership remains the
+ * admission fence even when that retained token does not identify it. */
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ { $set: { actorActionAdmissionId: 'stale-new-worker-token' } },
+ );
+ await expect(methods.hasAgentEventActorActionAdmission(actionAdmission)).resolves.toBe(true);
+ await expect(methods.admitAgentEventActorAction(successorAdmission)).resolves.toBe(false);
+ await expect(methods.releaseAgentEventActorAction(actionAdmission)).resolves.toBe(false);
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ { $unset: { actorActionAdmittedAt: 1, actorActionAdmissionId: 1 } },
+ );
+ await methods.completeAgentTriggerDelivery({
+ id: claimed!.id,
+ workerId: 'worker-1',
+ claimToken: 'claim-1',
+ attempt: attempt!,
+ result: { status: 'started' },
+ settledAt: new Date(generationCreatedAt),
+ awaitTerminalHandling: true,
+ handling: {
+ status: 'started',
+ conversationId: 'conversation-1',
+ streamId: 'conversation-1',
+ generationCreatedAt,
+ startedAt: new Date(generationCreatedAt),
+ },
+ });
+ const activeDelivery = await methods.getAgentTriggerDelivery(queued.delivery.deliveryKey);
+ expect(activeDelivery).toMatchObject({ handling: { status: 'started' } });
+ expect(activeDelivery).not.toHaveProperty('expiresAt');
+ const actorReceipt = {
+ bindingId: 'binding-1',
+ resolution: 'checkpoint_verified' as const,
+ checkpoint: {
+ threadId: 'conversation-1',
+ checkpointId: 'checkpoint-1',
+ checkpointNs: 'event:1',
+ },
+ action: { toolName: 'submit_move', toolCallId: 'call-1' },
+ };
+ const settlement = {
+ deliveryKey: queued.delivery.deliveryKey,
+ user: queued.delivery.user,
+ tenantId: 'tenant-1',
+ bindingId: actorReceipt.bindingId,
+ conversationId: 'conversation-1',
+ generationCreatedAt,
+ status: 'applied' as const,
+ settledAt: new Date(generationCreatedAt + 1_000),
+ requiresActionAdmission: true as const,
+ receipt: {
+ resolution: actorReceipt.resolution,
+ checkpoint: actorReceipt.checkpoint,
+ action: actorReceipt.action,
+ },
+ };
+
+ await expect(methods.settleAgentEventActorReceipt(settlement)).resolves.toBe(false);
+ await expect(methods.admitAgentEventActorAction(actionAdmission)).resolves.toBe(true);
+ await expect(methods.hasAgentEventActorActionAdmission(actionAdmission)).resolves.toBe(true);
+ await expect(methods.admitAgentEventActorAction(actionAdmission)).resolves.toBe(false);
+ await expect(methods.releaseAgentEventActorAction(actionAdmission)).resolves.toBe(true);
+ await expect(methods.hasAgentEventActorActionAdmission(actionAdmission)).resolves.toBe(false);
+ await expect(methods.admitAgentEventActorAction(actionAdmission)).resolves.toBe(true);
+ await expect(methods.settleAgentEventActorReceipt(settlement)).resolves.toBe(true);
+ await expect(methods.releaseAgentEventActorAction(actionAdmission)).resolves.toBe(false);
+ /** Simulate a receipt written by the pre-normalization build after
+ * transport dead-lettered the root. Exact replay must retire that dead
+ * state instead of leaving a non-requeueable dead letter. */
+ await Delivery.updateOne(
+ { deliveryKey: settlement.deliveryKey },
+ { $set: { status: 'dead', lastError: transientFailure() } },
+ );
+ await expect(methods.settleAgentEventActorReceipt(settlement)).resolves.toBe(true);
+ await expect(
+ Delivery.findOne({ deliveryKey: settlement.deliveryKey }).lean(),
+ ).resolves.toMatchObject({ status: 'succeeded' });
+ for (const conflict of [
+ { ...settlement, user: new mongoose.Types.ObjectId() },
+ { ...settlement, tenantId: 'tenant-2' },
+ { ...settlement, bindingId: 'binding-2' },
+ { ...settlement, conversationId: 'conversation-2' },
+ {
+ ...settlement,
+ receipt: {
+ ...settlement.receipt,
+ checkpoint: { ...settlement.receipt.checkpoint, checkpointId: 'checkpoint-2' },
+ },
+ },
+ {
+ ...settlement,
+ receipt: { ...settlement.receipt, action: { toolName: 'resign_game' } },
+ },
+ {
+ ...settlement,
+ status: 'failed' as const,
+ receipt: { ...settlement.receipt, resolution: 'action_compensated' as const },
+ },
+ ]) {
+ await expect(methods.settleAgentEventActorReceipt(conflict)).resolves.toBe(false);
+ }
+ await expect(
+ methods.getAgentTriggerDelivery(queued.delivery.deliveryKey),
+ ).resolves.toMatchObject({
+ handling: { status: 'applied', action: actorReceipt.action },
+ expiresAt: new Date(generationCreatedAt + 1_000 + 90 * 24 * 60 * 60_000),
+ actorReceipt: {
+ ...actorReceipt,
+ settledAt: new Date(generationCreatedAt + 1_000),
+ },
+ });
+ await expect(
+ methods.getAgentEventActorReceipt({
+ deliveryKey: queued.delivery.deliveryKey,
+ user: queued.delivery.user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-1',
+ conversationId: 'conversation-1',
+ }),
+ ).resolves.toEqual({
+ ...actorReceipt,
+ settledAt: new Date(generationCreatedAt + 1_000),
+ });
+ const publicStatus = await methods.getAgentTriggerDeliveryStatus(
+ queued.delivery.deliveryKey,
+ queued.delivery.user,
+ 'source-key-1',
+ 'tenant-1',
+ );
+ expect(publicStatus).not.toHaveProperty('actorReceipt');
+ await expect(
+ methods.getAgentEventActorReceipt({
+ deliveryKey: queued.delivery.deliveryKey,
+ user: new mongoose.Types.ObjectId(),
+ tenantId: 'tenant-1',
+ bindingId: 'binding-1',
+ conversationId: 'conversation-1',
+ }),
+ ).resolves.toBeNull();
+ });
+
+ it('serializes no-action settlement against renewed action admission', async () => {
+ const queued = await methods.enqueueAgentTriggerDelivery(
+ enqueueInput({
+ awaitTerminalHandling: true,
+ envelope: {
+ mode: 'continue',
+ target: { bindingId: 'binding-1' },
+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },
+ },
+ }),
+ );
+ const claimed = await methods.claimNextAgentTriggerDelivery({
+ workerId: 'worker-1',
+ claimToken: 'claim-1',
+ now: START,
+ leaseUntil: new Date(START.getTime() + 60_000),
+ });
+ const attempt = await methods.beginAgentTriggerDeliveryAttempt({
+ id: claimed!.id,
+ workerId: 'worker-1',
+ claimToken: 'claim-1',
+ now: START,
+ });
+ const generationCreatedAt = START.getTime() + 1_000;
+ await methods.completeAgentTriggerDelivery({
+ id: claimed!.id,
+ workerId: 'worker-1',
+ claimToken: 'claim-1',
+ attempt: attempt!,
+ result: { status: 'started' },
+ settledAt: new Date(generationCreatedAt),
+ awaitTerminalHandling: true,
+ handling: {
+ status: 'started',
+ conversationId: 'conversation-1',
+ streamId: 'conversation-1',
+ generationCreatedAt,
+ startedAt: new Date(generationCreatedAt),
+ },
+ });
+ const admission = {
+ deliveryKey: queued.delivery.deliveryKey,
+ user: queued.delivery.user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-1',
+ conversationId: 'conversation-1',
+ admittedAt: new Date(generationCreatedAt + 500),
+ admissionId: 'admission-1',
+ };
+ const noAction = {
+ deliveryKey: queued.delivery.deliveryKey,
+ conversationId: 'conversation-1',
+ generationCreatedAt,
+ status: 'completed_no_action' as const,
+ settledAt: new Date(generationCreatedAt + 1_000),
+ };
+
+ await expect(methods.admitAgentEventActorAction(admission)).resolves.toBe(true);
+ await expect(methods.settleAgentTriggerHandlingOutcome(noAction)).resolves.toBe(false);
+ await expect(methods.releaseAgentEventActorAction(admission)).resolves.toBe(true);
+ await expect(methods.settleAgentTriggerHandlingOutcome(noAction)).resolves.toBe(true);
+ await expect(methods.admitAgentEventActorAction(admission)).resolves.toBe(false);
+ });
+
+ it('lazily backfills one exact legacy receipt and rejects conflicting identities', async () => {
+ const queued = await methods.enqueueAgentTriggerDelivery(
+ enqueueInput({
+ envelope: {
+ mode: 'continue',
+ target: { bindingId: 'binding-legacy' },
+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },
+ },
+ }),
+ );
+ const generationCreatedAt = START.getTime() + 2_000;
+ const action = { toolName: 'submit_move', toolCallId: 'call-legacy' };
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ {
+ $set: {
+ status: 'dead',
+ handling: {
+ status: 'applied',
+ conversationId: 'conversation-legacy',
+ streamId: 'conversation-legacy',
+ generationCreatedAt,
+ startedAt: START,
+ settledAt: START,
+ action,
+ },
+ settledAt: START,
+ expiresAt: new Date(START.getTime() + 1_000),
+ },
+ },
+ );
+ const input = {
+ deliveryKey: queued.delivery.deliveryKey,
+ user: queued.delivery.user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-legacy',
+ conversationId: 'conversation-legacy',
+ generationCreatedAt,
+ status: 'applied' as const,
+ settledAt: new Date(START.getTime() + 3_000),
+ receipt: {
+ resolution: 'checkpoint_verified' as const,
+ checkpoint: {
+ threadId: 'conversation-legacy',
+ checkpointId: 'checkpoint-legacy',
+ checkpointNs: 'event-actor/legacy',
+ },
+ action,
+ },
+ };
+
+ await expect(
+ Promise.all([
+ methods.backfillAgentEventActorReceipt(input),
+ methods.backfillAgentEventActorReceipt(input),
+ ]),
+ ).resolves.toEqual([true, true]);
+ await expect(
+ methods.backfillAgentEventActorReceipt({
+ ...input,
+ receipt: {
+ ...input.receipt,
+ checkpoint: { ...input.receipt.checkpoint, checkpointId: 'different' },
+ },
+ }),
+ ).resolves.toBe(false);
+ await expect(
+ methods.getAgentEventActorReceipt({
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ tenantId: input.tenantId,
+ bindingId: input.bindingId,
+ conversationId: input.conversationId,
+ }),
+ ).resolves.toEqual({
+ bindingId: input.bindingId,
+ ...input.receipt,
+ settledAt: input.settledAt,
+ });
+ const stored = await Delivery.findOne({ deliveryKey: input.deliveryKey }).lean();
+ expect(stored?.status).toBe('succeeded');
+ expect(stored?.expiresAt?.getTime()).toBeGreaterThan(START.getTime() + 1_000);
+ });
+
+ it('does not let more than 1,024 unexpired receipts block or evict another actor receipt', async () => {
+ const user = new mongoose.Types.ObjectId();
+ const settledAt = new Date(START.getTime() + 10_000);
+ const expiresAt = new Date(settledAt.getTime() + 90 * 24 * 60 * 60_000);
+ await Delivery.insertMany(
+ Array.from({ length: 1_025 }, (_, index) => ({
+ deliveryKey: `retained-receipt-${index}`,
+ fingerprint: `retained-fingerprint-${index}`,
+ orderingKey: `retained-lane-${index}`,
+ laneSequence: 1,
+ envelope: { mode: 'continue', target: { bindingId: `binding-${index}` } },
+ user,
+ tenantId: 'tenant-1',
+ status: 'succeeded',
+ attempts: 1,
+ availableAt: START,
+ handling: {
+ status: 'applied',
+ conversationId: `conversation-${index}`,
+ streamId: `conversation-${index}`,
+ generationCreatedAt: START.getTime() + index,
+ startedAt: START,
+ settledAt,
+ action: { toolName: 'submit_move', toolCallId: `call-${index}` },
+ },
+ actorReceipt: {
+ bindingId: `binding-${index}`,
+ resolution: 'checkpoint_verified',
+ checkpoint: {
+ threadId: `conversation-${index}`,
+ checkpointId: `checkpoint-${index}`,
+ checkpointNs: `event-actor/${index}`,
+ },
+ action: { toolName: 'submit_move', toolCallId: `call-${index}` },
+ settledAt,
+ },
+ settledAt,
+ expiresAt,
+ })),
+ );
+
+ const queued = await methods.enqueueAgentTriggerDelivery(
+ enqueueInput({
+ user,
+ orderingKey: 'fresh-receipt-lane',
+ awaitTerminalHandling: true,
+ envelope: {
+ mode: 'continue',
+ target: { bindingId: 'binding-fresh' },
+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },
+ },
+ }),
+ );
+ const generationCreatedAt = settledAt.getTime() + 1_000;
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ {
+ $set: {
+ status: 'succeeded',
+ handling: {
+ status: 'started',
+ conversationId: 'conversation-fresh',
+ streamId: 'conversation-fresh',
+ generationCreatedAt,
+ startedAt: settledAt,
+ },
+ },
+ },
+ );
+
+ await expect(
+ methods.settleAgentEventActorReceipt({
+ deliveryKey: queued.delivery.deliveryKey,
+ user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-fresh',
+ conversationId: 'conversation-fresh',
+ generationCreatedAt,
+ status: 'applied',
+ settledAt: new Date(generationCreatedAt + 1_000),
+ receipt: {
+ resolution: 'checkpoint_verified',
+ checkpoint: {
+ threadId: 'conversation-fresh',
+ checkpointId: 'checkpoint-fresh',
+ checkpointNs: 'event-actor/fresh',
+ },
+ action: { toolName: 'submit_move', toolCallId: 'call-fresh' },
+ },
+ }),
+ ).resolves.toBe(true);
+
+ expect(await Delivery.countDocuments({ actorReceipt: { $exists: true } })).toBe(1_026);
+ await expect(
+ methods.getAgentEventActorReceipt({
+ deliveryKey: 'retained-receipt-0',
+ user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-0',
+ conversationId: 'conversation-0',
+ }),
+ ).resolves.toMatchObject({
+ resolution: 'checkpoint_verified',
+ action: { toolCallId: 'call-0' },
+ });
+ await expect(
+ methods.settleAgentEventActorReceipt({
+ deliveryKey: 'retained-receipt-0',
+ user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-0',
+ conversationId: 'conversation-0',
+ generationCreatedAt: START.getTime(),
+ status: 'applied',
+ settledAt,
+ receipt: {
+ resolution: 'checkpoint_verified',
+ checkpoint: {
+ threadId: 'conversation-0',
+ checkpointId: 'stale-owner-checkpoint',
+ checkpointNs: 'event-actor/0',
+ },
+ action: { toolName: 'submit_move', toolCallId: 'call-0' },
+ },
+ }),
+ ).resolves.toBe(false);
+ await Delivery.insertMany([
+ {
+ deliveryKey: 'actor-retry-without-mailbox-flag',
+ fingerprint: 'actor-retry-without-mailbox-flag',
+ orderingKey: 'actor-retry-lane',
+ laneSequence: 1,
+ envelope: { mode: 'continue', target: { bindingId: 'binding-retry' } },
+ user,
+ tenantId: 'tenant-1',
+ status: 'pending',
+ attempts: 2,
+ availableAt: START,
+ handling: {
+ status: 'started',
+ conversationId: 'conversation-retry',
+ streamId: 'conversation-retry',
+ generationCreatedAt: START.getTime(),
+ startedAt: START,
+ },
+ },
+ {
+ deliveryKey: 'actor-dead-without-mailbox-flag',
+ fingerprint: 'actor-dead-without-mailbox-flag',
+ orderingKey: 'actor-dead-lane',
+ laneSequence: 1,
+ envelope: { mode: 'continue', target: { bindingId: 'binding-dead' } },
+ user,
+ tenantId: 'tenant-1',
+ status: 'dead',
+ attempts: 3,
+ availableAt: START,
+ handling: {
+ status: 'started',
+ conversationId: 'conversation-dead',
+ streamId: 'conversation-dead',
+ generationCreatedAt: START.getTime(),
+ startedAt: START,
+ },
+ },
+ ]);
+ await expect(
+ methods.getAgentEventActorReceiptStorageMetrics(new Date(START.getTime() + 20_000)),
+ ).resolves.toMatchObject({
+ retainedByResolution: {
+ checkpoint_verified: 1_026,
+ action_compensated: 0,
+ history_repaired: 0,
+ },
+ expiryEligible: 0,
+ retryDeliveries: 1,
+ deadDeliveries: 1,
+ });
+ });
+
+ it('never migrates a compensated legacy receipt as an applied outcome', async () => {
+ const queued = await methods.enqueueAgentTriggerDelivery(
+ enqueueInput({
+ envelope: {
+ mode: 'continue',
+ target: { bindingId: 'binding-compensated' },
+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },
+ },
+ }),
+ );
+ const generationCreatedAt = START.getTime() + 6_000;
+ const checkpoint = {
+ threadId: 'conversation-compensated',
+ checkpointId: 'checkpoint-compensated',
+ checkpointNs: 'event-actor/compensated',
+ };
+ const action = { toolName: 'submit_move', toolCallId: 'call-compensated' };
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ {
+ $set: {
+ status: 'succeeded',
+ handling: {
+ status: 'failed',
+ conversationId: 'conversation-compensated',
+ streamId: 'conversation-compensated',
+ generationCreatedAt,
+ startedAt: START,
+ settledAt: START,
+ error: 'Applied action was compensated',
+ },
+ settledAt: START,
+ expiresAt: new Date(START.getTime() + 90 * 24 * 60 * 60_000),
+ },
+ },
+ );
+ const common = {
+ deliveryKey: queued.delivery.deliveryKey,
+ user: queued.delivery.user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-compensated',
+ conversationId: 'conversation-compensated',
+ generationCreatedAt,
+ settledAt: new Date(START.getTime() + 7_000),
+ receipt: { checkpoint, action },
+ };
+
+ await expect(
+ methods.backfillAgentEventActorReceipt({
+ ...common,
+ status: 'failed',
+ receipt: { ...common.receipt, resolution: 'action_compensated' },
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.backfillAgentEventActorReceipt({
+ ...common,
+ status: 'applied',
+ receipt: { ...common.receipt, resolution: 'checkpoint_verified' },
+ }),
+ ).resolves.toBe(false);
+ await expect(
+ methods.getAgentEventActorReceipt({
+ deliveryKey: common.deliveryKey,
+ user: common.user,
+ tenantId: common.tenantId,
+ bindingId: common.bindingId,
+ conversationId: common.conversationId,
+ }),
+ ).resolves.toMatchObject({ resolution: 'action_compensated' });
+ });
+
+ it('allows exactly one winner when verification races compensation', async () => {
+ const queued = await methods.enqueueAgentTriggerDelivery(
+ enqueueInput({
+ awaitTerminalHandling: true,
+ envelope: {
+ mode: 'continue',
+ target: { bindingId: 'binding-race' },
+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },
+ },
+ }),
+ );
+ const generationCreatedAt = START.getTime() + 4_000;
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ {
+ $set: {
+ status: 'succeeded',
+ awaitTerminalHandling: true,
+ handling: {
+ status: 'started',
+ conversationId: 'conversation-race',
+ streamId: 'conversation-race',
+ generationCreatedAt,
+ startedAt: START,
+ },
+ },
+ },
+ );
+ const checkpoint = {
+ threadId: 'conversation-race',
+ checkpointId: 'checkpoint-race',
+ checkpointNs: 'event-actor/race',
+ };
+ const common = {
+ deliveryKey: queued.delivery.deliveryKey,
+ user: queued.delivery.user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-race',
+ conversationId: 'conversation-race',
+ generationCreatedAt,
+ settledAt: new Date(START.getTime() + 5_000),
+ };
+ const [verified, compensated] = await Promise.all([
+ methods.settleAgentEventActorReceipt({
+ ...common,
+ status: 'applied',
+ receipt: {
+ resolution: 'checkpoint_verified',
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ },
+ }),
+ methods.settleAgentEventActorReceipt({
+ ...common,
+ status: 'failed',
+ error: 'operator compensated action',
+ receipt: {
+ resolution: 'action_compensated',
+ checkpoint,
+ action: { toolName: 'submit_move' },
+ },
+ }),
+ ]);
+
+ expect(Number(verified) + Number(compensated)).toBe(1);
+ const stored = await methods.getAgentTriggerDelivery(queued.delivery.deliveryKey);
+ expect(stored?.handling?.status).toBe(verified ? 'applied' : 'failed');
+ expect(stored?.actorReceipt?.resolution).toBe(
+ verified ? 'checkpoint_verified' : 'action_compensated',
+ );
+ });
+
it('reclaims an inactive high-cardinality lane after its tail succeeds', async () => {
const orderingKey = 'one-shot-resource-lane';
await methods.enqueueAgentTriggerDelivery(enqueueInput({ orderingKey }));
@@ -1792,6 +2488,163 @@ describe('agent trigger delivery methods', () => {
);
});
+ it('retires a dead letter when its admitted actor action later settles', async () => {
+ const user = new mongoose.Types.ObjectId();
+ const orderingKey = 'late-terminal-batch';
+ const queued = await methods.enqueueAgentTriggerDelivery(
+ enqueueInput({
+ user,
+ orderingKey,
+ envelope: {
+ mode: 'continue',
+ target: { bindingId: 'binding-late-terminal' },
+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },
+ },
+ }),
+ );
+ const generationCreatedAt = START.getTime() + 1_000;
+ const member = await Delivery.create({
+ ...enqueueInput({
+ user,
+ orderingKey,
+ envelope: {
+ mode: 'continue',
+ target: { bindingId: 'binding-late-terminal' },
+ event: { source: { id: 'source-key-1', type: 'remote_api_key' } },
+ },
+ }),
+ laneSequence: 2,
+ status: 'dead',
+ attempts: 3,
+ requeueCount: 0,
+ batchRootId: queued.delivery.id,
+ settledAt: START,
+ lastError: transientFailure(),
+ handling: {
+ status: 'started',
+ conversationId: 'conversation-late-terminal',
+ streamId: 'conversation-late-terminal',
+ generationCreatedAt,
+ startedAt: START,
+ },
+ });
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ {
+ $set: {
+ status: 'dead',
+ attempts: 3,
+ batchMemberIds: [member._id],
+ batchMembersSettledAt: START,
+ actorActionAdmittedAt: START,
+ handling: {
+ status: 'started',
+ conversationId: 'conversation-late-terminal',
+ streamId: 'conversation-late-terminal',
+ generationCreatedAt,
+ startedAt: START,
+ },
+ },
+ },
+ );
+
+ await expect(
+ methods.settleAgentEventActorReceipt({
+ deliveryKey: queued.delivery.deliveryKey,
+ user,
+ tenantId: 'tenant-1',
+ bindingId: 'binding-late-terminal',
+ conversationId: 'conversation-late-terminal',
+ generationCreatedAt,
+ status: 'applied',
+ settledAt: new Date(generationCreatedAt + 1_000),
+ requiresActionAdmission: true,
+ receipt: {
+ resolution: 'checkpoint_verified',
+ checkpoint: {
+ threadId: 'conversation-late-terminal',
+ checkpointId: 'checkpoint-late-terminal',
+ checkpointNs: 'event-actor/late-terminal',
+ },
+ action: { toolName: 'submit_move' },
+ },
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.getAgentTriggerDelivery(queued.delivery.deliveryKey),
+ ).resolves.toMatchObject({ status: 'succeeded', handling: { status: 'applied' } });
+ expect(
+ (await methods.getAgentTriggerDelivery(queued.delivery.deliveryKey))?.lastError,
+ ).toBeUndefined();
+ await expect(Delivery.findById(member._id).lean()).resolves.toMatchObject({
+ status: 'succeeded',
+ handling: { status: 'applied' },
+ });
+ await expect(
+ methods.getAgentTriggerDeliveryStatus(member.deliveryKey, user, 'source-key-1', 'tenant-1'),
+ ).resolves.toMatchObject({ status: 'succeeded', handling: { status: 'applied' } });
+ await expect(LaneSequence.findById(orderingKey).lean()).resolves.toBeNull();
+ await expect(
+ methods.requeueAgentTriggerDelivery(queued.delivery.id, START),
+ ).resolves.toBeNull();
+ });
+
+ it('does not requeue a legacy terminal outcome before its receipt is migrated', async () => {
+ const queued = await methods.enqueueAgentTriggerDelivery(enqueueInput());
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ {
+ $set: {
+ status: 'dead',
+ handling: {
+ status: 'applied',
+ conversationId: 'legacy-terminal-conversation',
+ streamId: 'legacy-terminal-conversation',
+ generationCreatedAt: START.getTime(),
+ startedAt: START,
+ settledAt: START,
+ action: { toolName: 'submit_move' },
+ },
+ },
+ },
+ );
+
+ await expect(
+ methods.requeueAgentTriggerDelivery(queued.delivery.id, START),
+ ).resolves.toBeNull();
+ await expect(Delivery.findById(queued.delivery.id).lean()).resolves.toMatchObject({
+ status: 'dead',
+ handling: { status: 'applied' },
+ });
+ });
+
+ it('does not requeue ambiguous legacy started handling', async () => {
+ const queued = await methods.enqueueAgentTriggerDelivery(enqueueInput());
+ await Delivery.updateOne(
+ { deliveryKey: queued.delivery.deliveryKey },
+ {
+ $set: {
+ status: 'dead',
+ handling: {
+ status: 'started',
+ conversationId: 'legacy-started-conversation',
+ streamId: 'legacy-started-conversation',
+ generationCreatedAt: START.getTime(),
+ startedAt: START,
+ },
+ },
+ },
+ );
+
+ await expect(
+ methods.requeueAgentTriggerDelivery(queued.delivery.id, START),
+ ).resolves.toBeNull();
+ await expect(Delivery.findById(queued.delivery.id).lean()).resolves.toMatchObject({
+ status: 'dead',
+ handling: { status: 'started' },
+ });
+ });
+
it('counts only live leases while an account deletion drains', async () => {
const user = new mongoose.Types.ObjectId();
await methods.enqueueAgentTriggerDelivery(enqueueInput({ user }));
diff --git a/packages/data-schemas/src/methods/triggerDelivery.ts b/packages/data-schemas/src/methods/triggerDelivery.ts
index 4161f24876..e9454271ec 100644
--- a/packages/data-schemas/src/methods/triggerDelivery.ts
+++ b/packages/data-schemas/src/methods/triggerDelivery.ts
@@ -1,6 +1,7 @@
import type { Model, Types } from 'mongoose';
import type {
AgentTriggerDeliveryClaim,
+ AgentEventActorReceipt,
AgentTriggerDeliveryFailure,
AgentTriggerHandlingState,
AgentTriggerDeliveryRecord,
@@ -30,6 +31,30 @@ const MAX_BATCH_BYTES = 512 * 1024;
type DuplicateKeyError = { code?: number };
+export type AgentEventActorReceiptMetric = {
+ operation: 'read' | 'settle' | 'backfill';
+ outcome: 'hit' | 'miss' | 'success' | 'replay' | 'conflict';
+ resolution?: AgentEventActorReceipt['resolution'];
+};
+
+let observeAgentEventActorReceipt = (_metric: AgentEventActorReceiptMetric): void => undefined;
+
+/** Installs one process-local, low-cardinality observer at the storage boundary. */
+export function setAgentEventActorReceiptMetricObserver(
+ observer?: (metric: AgentEventActorReceiptMetric) => void,
+): void {
+ observeAgentEventActorReceipt = observer ?? (() => undefined);
+}
+
+/** Emits one already-bounded receipt metric through the configured observer. */
+export function recordAgentEventActorReceiptMetric(metric: AgentEventActorReceiptMetric): void {
+ try {
+ observeAgentEventActorReceipt(metric);
+ } catch (error) {
+ logger.warn('[trigger-delivery] Event actor receipt metric observer failed', error);
+ }
+}
+
export class AgentTriggerDeliveryConflictError extends Error {
constructor(deliveryKey: string) {
super(`Agent trigger delivery key ${deliveryKey} was reused with different content`);
@@ -68,6 +93,48 @@ export interface SettleAgentTriggerHandlingOutcomeInput {
action?: AgentTriggerHandlingState['action'];
}
+export interface SettleAgentEventActorReceiptInput {
+ deliveryKey: string;
+ user: string | Types.ObjectId;
+ tenantId?: string;
+ bindingId: string;
+ conversationId: string;
+ generationCreatedAt: number;
+ status: 'applied' | 'failed';
+ settledAt: Date;
+ error?: string;
+ /** Present for executions that acquired the delivery-owned action CAS.
+ * Absent only for mixed-version terminal rows created before this field. */
+ requiresActionAdmission?: true;
+ receipt: Omit<AgentEventActorReceipt, 'bindingId' | 'settledAt'>;
+}
+
+export interface AdmitAgentEventActorActionInput extends GetAgentEventActorReceiptInput {
+ admittedAt: Date;
+ admissionId: string;
+}
+
+export interface AgentEventActorActionAdmissionInput extends GetAgentEventActorReceiptInput {
+ admissionId: string;
+}
+
+export interface GetAgentEventActorReceiptInput {
+ deliveryKey: string;
+ user: string | Types.ObjectId;
+ tenantId?: string;
+ bindingId: string;
+ conversationId: string;
+}
+
+export type BackfillAgentEventActorReceiptInput = SettleAgentEventActorReceiptInput;
+
+export interface AgentEventActorReceiptStorageMetrics {
+ retainedByResolution: Record<AgentEventActorReceipt['resolution'], number>;
+ expiryEligible: number;
+ retryDeliveries: number;
+ deadDeliveries: number;
+}
+
export interface AgentTriggerDeliveryMethods {
ensureAgentTriggerDeliveryIndexes: () => Promise<void>;
enqueueAgentTriggerDelivery: (
@@ -106,6 +173,19 @@ export interface AgentTriggerDeliveryMethods {
settleAgentTriggerHandlingOutcome: (
input: SettleAgentTriggerHandlingOutcomeInput,
) => Promise<boolean>;
+ admitAgentEventActorAction: (input: AdmitAgentEventActorActionInput) => Promise<boolean>;
+ releaseAgentEventActorAction: (input: AgentEventActorActionAdmissionInput) => Promise<boolean>;
+ hasAgentEventActorActionAdmission: (
+ input: AgentEventActorActionAdmissionInput,
+ ) => Promise<boolean>;
+ settleAgentEventActorReceipt: (input: SettleAgentEventActorReceiptInput) => Promise<boolean>;
+ getAgentEventActorReceipt: (
+ input: GetAgentEventActorReceiptInput,
+ ) => Promise<AgentEventActorReceipt | null>;
+ backfillAgentEventActorReceipt: (input: BackfillAgentEventActorReceiptInput) => Promise<boolean>;
+ getAgentEventActorReceiptStorageMetrics: (
+ now: Date,
+ ) => Promise<AgentEventActorReceiptStorageMetrics>;
retryAgentTriggerDelivery: (
input: AgentTriggerDeliveryFence & {
attempt: number;
@@ -202,6 +282,7 @@ function toRecord(delivery: IAgentTriggerDelivery): AgentTriggerDeliveryRecord {
awaitTerminalHandling: delivery.awaitTerminalHandling,
}),
...(delivery.handling != null && { handling: delivery.handling }),
+ ...(delivery.actorReceipt != null && { actorReceipt: delivery.actorReceipt }),
};
}
@@ -424,6 +505,56 @@ export function createAgentTriggerDeliveryMethods(
return abandonLanePublisher(lane);
}
+ /** A dead batch root owns requeue before any constituent is reset. Staging
+ * recovery replays this preparation, so a crash cannot leave a dead root
+ * beside members that were silently advanced to a new generation. */
+ async function prepareBatchMembersForRequeue(root: IAgentTriggerDelivery): Promise<void> {
+ const memberIds = root.batchMemberIds ?? [];
+ const requeueCount = root.requeueCount ?? 0;
+ if (memberIds.length === 0 || requeueCount <= 0 || root._id == null) {
+ return;
+ }
+ const previousRequeueCount = requeueCount - 1;
+ await Delivery().updateMany(
+ {
+ _id: { $in: memberIds },
+ orderingKey: root.orderingKey,
+ batchRootId: root._id,
+ status: { $in: ['staging', 'batched', 'succeeded', 'dead'] },
+ ...(previousRequeueCount === 0
+ ? {
+ $or: [{ batchRootRequeueCount: 0 }, { batchRootRequeueCount: { $exists: false } }],
+ }
+ : { batchRootRequeueCount: previousRequeueCount }),
+ },
+ {
+ $set: {
+ status: 'batched',
+ attempts: 0,
+ batchRootId: root._id,
+ batchRootRequeueCount: requeueCount,
+ },
+ $unset: {
+ lastError: 1,
+ result: 1,
+ settledAt: 1,
+ expiresAt: 1,
+ handling: 1,
+ },
+ },
+ );
+ const preparedCount = await Delivery().countDocuments({
+ _id: { $in: memberIds },
+ orderingKey: root.orderingKey,
+ status: 'batched',
+ batchRootId: root._id,
+ batchRootRequeueCount: requeueCount,
+ });
+ if (preparedCount !== memberIds.length) {
+ throw new Error('Not every agent trigger batch receipt could be prepared for requeue');
+ }
+ }
+
async function publishStagedDelivery(
delivery: IAgentTriggerDelivery,
): Promise<IAgentTriggerDelivery> {
@@ -444,6 +575,7 @@ export function createAgentTriggerDeliveryMethods(
if ((await UserPurge().exists({ _id: current.user })) != null) {
return current;
}
+ await prepareBatchMembersForRequeue(current);
const lane = await LaneSequence().findById(orderingKey).lean<IAgentTriggerLaneSequence>();
if (lane?.publisherDeliveryId != null) {
@@ -853,16 +985,25 @@ export function createAgentTriggerDeliveryMethods(
}
async function propagateBatchHandling(
- root: Pick<IAgentTriggerDelivery, '_id' | 'orderingKey' | 'batchMemberIds' | 'handling'>,
+ root: Pick<
+ IAgentTriggerDelivery,
+ '_id' | 'orderingKey' | 'batchMemberIds' | 'status' | 'settledAt' | 'handling'
+ >,
): Promise<void> {
if (root._id == null || root.handling == null || !root.batchMemberIds?.length) {
return;
}
const status = root.handling.status;
+ const retiresDeadMembers =
+ root.status === 'succeeded' && status !== 'started' && root.handling.settledAt != null;
await Delivery().updateMany(
{
_id: { $in: root.batchMemberIds },
orderingKey: root.orderingKey,
+ batchRootId: root._id,
+ ...(retiresDeadMembers && {
+ status: { $in: ['staging', 'batched', 'succeeded', 'dead'] },
+ }),
$or:
status === 'started'
? [{ 'handling.status': 'started' }, { 'handling.status': { $exists: false } }]
@@ -874,15 +1015,38 @@ export function createAgentTriggerDeliveryMethods(
{
$set: {
handling: root.handling,
+ ...(retiresDeadMembers && {
+ status: 'succeeded',
+ settledAt: root.settledAt ?? root.handling.settledAt,
+ }),
...(status !== 'started' &&
root.handling.settledAt != null && {
expiresAt: new Date(root.handling.settledAt.getTime() + SUCCESS_RETENTION_MS),
}),
},
+ ...(retiresDeadMembers && { $unset: { lastError: 1 } }),
},
);
}
+ async function finalizeAgentEventActorDelivery(
+ root: Pick<
+ IAgentTriggerDelivery,
+ '_id' | 'orderingKey' | 'batchMemberIds' | 'status' | 'settledAt' | 'handling'
+ >,
+ settledAt: Date,
+ ): Promise<void> {
+ await propagateBatchHandling(root);
+ /** Actor receipt finalization is terminal regardless of the mailbox
+ * rollout flag. Publishing cleanup unconditionally is safe because the
+ * lane reclaimer independently refuses any still-active lane. */
+ await LaneSequence().updateOne(
+ { _id: root.orderingKey },
+ { $set: { cleanupRequestedAt: settledAt } },
+ );
+ await reclaimLaneIfInactive(root.orderingKey);
+ }
+
async function settleBatchMembers(
root: Pick<
IAgentTriggerDelivery,
@@ -980,9 +1144,12 @@ export function createAgentTriggerDeliveryMethods(
if (input.status === 'succeeded' && root.handling != null) {
const authoritative = await Delivery()
.findById(root._id)
- .select('_id orderingKey batchMemberIds handling')
+ .select('_id orderingKey batchMemberIds status settledAt handling')
.lean<
- Pick<IAgentTriggerDelivery, '_id' | 'orderingKey' | 'batchMemberIds' | 'handling'>
+ Pick<
+ IAgentTriggerDelivery,
+ '_id' | 'orderingKey' | 'batchMemberIds' | 'status' | 'settledAt' | 'handling'
+ >
>();
if (authoritative != null) {
await propagateBatchHandling(authoritative);
@@ -1193,6 +1360,7 @@ export function createAgentTriggerDeliveryMethods(
'handling.status': 'started',
'handling.conversationId': input.conversationId,
'handling.generationCreatedAt': input.generationCreatedAt,
+ actorActionAdmittedAt: { $exists: false },
},
{
$set: terminalHandling,
@@ -1203,11 +1371,17 @@ export function createAgentTriggerDeliveryMethods(
},
{ new: true },
)
- .select('_id orderingKey batchMemberIds awaitTerminalHandling handling')
+ .select('_id orderingKey batchMemberIds status settledAt awaitTerminalHandling handling')
.lean<
Pick<
IAgentTriggerDelivery,
- '_id' | 'orderingKey' | 'batchMemberIds' | 'awaitTerminalHandling' | 'handling'
+ | '_id'
+ | 'orderingKey'
+ | 'batchMemberIds'
+ | 'status'
+ | 'settledAt'
+ | 'awaitTerminalHandling'
+ | 'handling'
>
>();
@@ -1219,11 +1393,17 @@ export function createAgentTriggerDeliveryMethods(
'handling.conversationId': input.conversationId,
'handling.generationCreatedAt': input.generationCreatedAt,
})
- .select('_id orderingKey batchMemberIds awaitTerminalHandling handling')
+ .select('_id orderingKey batchMemberIds status settledAt awaitTerminalHandling handling')
.lean<
Pick<
IAgentTriggerDelivery,
- '_id' | 'orderingKey' | 'batchMemberIds' | 'awaitTerminalHandling' | 'handling'
+ | '_id'
+ | 'orderingKey'
+ | 'batchMemberIds'
+ | 'status'
+ | 'settledAt'
+ | 'awaitTerminalHandling'
+ | 'handling'
>
>();
const replayed =
@@ -1237,17 +1417,437 @@ export function createAgentTriggerDeliveryMethods(
authoritative = existing;
}
- await propagateBatchHandling(authoritative);
- if (authoritative.awaitTerminalHandling === true) {
- await LaneSequence().updateOne(
- { _id: authoritative.orderingKey },
- { $set: { cleanupRequestedAt: input.settledAt } },
- );
- await reclaimLaneIfInactive(authoritative.orderingKey);
+ await finalizeAgentEventActorDelivery(authoritative, input.settledAt);
+ return true;
+ }
+
+ async function settleAgentEventActorReceipt(
+ input: SettleAgentEventActorReceiptInput,
+ ): Promise<boolean> {
+ const applied =
+ input.receipt.resolution === 'checkpoint_verified' ||
+ input.receipt.resolution === 'history_repaired';
+ if ((applied && input.status !== 'applied') || (!applied && input.status !== 'failed')) {
+ recordAgentEventActorReceiptMetric({
+ operation: 'settle',
+ outcome: 'conflict',
+ resolution: input.receipt.resolution,
+ });
+ return false;
+ }
+ if (
+ input.receipt.resolution === 'checkpoint_verified' &&
+ (typeof input.receipt.checkpoint.checkpointId !== 'string' ||
+ input.receipt.checkpoint.checkpointId.length === 0)
+ ) {
+ recordAgentEventActorReceiptMetric({
+ operation: 'settle',
+ outcome: 'conflict',
+ resolution: input.receipt.resolution,
+ });
+ return false;
+ }
+ const tenantScope =
+ input.tenantId == null ? { tenantId: { $exists: false } } : { tenantId: input.tenantId };
+ const actorReceipt: AgentEventActorReceipt = {
+ bindingId: input.bindingId,
+ ...input.receipt,
+ settledAt: input.settledAt,
+ };
+ const terminal = await Delivery()
+ .findOneAndUpdate(
+ {
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ ...tenantScope,
+ status: { $in: ['succeeded', 'dead'] },
+ 'envelope.target.bindingId': input.bindingId,
+ 'handling.status': 'started',
+ 'handling.conversationId': input.conversationId,
+ 'handling.generationCreatedAt': input.generationCreatedAt,
+ actorReceipt: { $exists: false },
+ ...(input.requiresActionAdmission === true
+ ? { actorActionAdmittedAt: { $exists: true } }
+ : { actorActionAdmittedAt: { $exists: false } }),
+ },
+ {
+ $set: {
+ status: 'succeeded',
+ settledAt: input.settledAt,
+ 'handling.status': input.status,
+ 'handling.settledAt': input.settledAt,
+ ...(applied && { 'handling.action': input.receipt.action }),
+ ...(!applied && {
+ 'handling.error': (input.error ?? 'Agent event actor action was compensated').slice(
+ 0,
+ MAX_ERROR_MESSAGE_LENGTH,
+ ),
+ }),
+ actorReceipt,
+ expiresAt: new Date(input.settledAt.getTime() + SUCCESS_RETENTION_MS),
+ },
+ $unset: {
+ lastError: 1,
+ ...(applied && { 'handling.error': 1 }),
+ ...(!applied && { 'handling.action': 1 }),
+ },
+ },
+ { new: true },
+ )
+ .select(
+ '_id orderingKey batchMemberIds status settledAt awaitTerminalHandling handling +actorReceipt',
+ )
+ .lean<
+ Pick<
+ IAgentTriggerDelivery,
+ | '_id'
+ | 'orderingKey'
+ | 'batchMemberIds'
+ | 'status'
+ | 'settledAt'
+ | 'awaitTerminalHandling'
+ | 'handling'
+ | 'actorReceipt'
+ >
+ >();
+ let authoritative = terminal;
+ let replayed = false;
+ if (authoritative == null) {
+ authoritative = await Delivery()
+ .findOneAndUpdate(
+ {
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ ...tenantScope,
+ status: { $in: ['succeeded', 'dead'] },
+ 'envelope.target.bindingId': input.bindingId,
+ 'handling.conversationId': input.conversationId,
+ 'handling.generationCreatedAt': input.generationCreatedAt,
+ 'handling.status': input.status,
+ actorReceipt,
+ },
+ {
+ $set: { status: 'succeeded', settledAt: actorReceipt.settledAt },
+ $max: {
+ expiresAt: new Date(actorReceipt.settledAt.getTime() + SUCCESS_RETENTION_MS),
+ },
+ $unset: { lastError: 1 },
+ },
+ { new: true },
+ )
+ .select(
+ '_id orderingKey batchMemberIds status settledAt awaitTerminalHandling handling +actorReceipt',
+ )
+ .lean<
+ Pick<
+ IAgentTriggerDelivery,
+ | '_id'
+ | 'orderingKey'
+ | 'batchMemberIds'
+ | 'status'
+ | 'settledAt'
+ | 'awaitTerminalHandling'
+ | 'handling'
+ | 'actorReceipt'
+ >
+ >();
+ if (authoritative == null) {
+ recordAgentEventActorReceiptMetric({
+ operation: 'settle',
+ outcome: 'conflict',
+ resolution: input.receipt.resolution,
+ });
+ return false;
+ }
+ replayed = true;
}
+ await finalizeAgentEventActorDelivery(authoritative, input.settledAt);
+ recordAgentEventActorReceiptMetric({
+ operation: 'settle',
+ outcome: replayed ? 'replay' : 'success',
+ resolution: input.receipt.resolution,
+ });
return true;
}
+ async function admitAgentEventActorAction(
+ input: AdmitAgentEventActorActionInput,
+ ): Promise<boolean> {
+ if (Number.isNaN(input.admittedAt.getTime())) {
+ throw new TypeError('admittedAt must be a valid date');
+ }
+ if (input.admissionId.length === 0 || input.admissionId.length > 64) {
+ throw new TypeError('admissionId must contain at most 64 characters');
+ }
+ const tenantScope =
+ input.tenantId == null ? { tenantId: { $exists: false } } : { tenantId: input.tenantId };
+ const admitted = await Delivery().updateOne(
+ {
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ ...tenantScope,
+ /** Loopback delivery returns `started` before the engine can persist
+ * transport completion, so the child may reach action admission while
+ * this exact attempt is still leased. The delivery row remains the
+ * serialization point: its identity fences plus the single admission
+ * marker allow only one overlapping attempt to proceed. */
+ status: { $in: ['leased', 'succeeded', 'dead'] },
+ 'envelope.target.bindingId': input.bindingId,
+ actorReceipt: { $exists: false },
+ actorActionAdmittedAt: { $exists: false },
+ $or: [
+ { handling: { $exists: false } },
+ {
+ 'handling.status': 'started',
+ 'handling.conversationId': input.conversationId,
+ },
+ ],
+ },
+ {
+ $set: {
+ actorActionAdmittedAt: input.admittedAt,
+ actorActionAdmissionId: input.admissionId,
+ },
+ },
+ );
+ return admitted.modifiedCount === 1;
+ }
+
+ async function releaseAgentEventActorAction(
+ input: AgentEventActorActionAdmissionInput,
+ ): Promise<boolean> {
+ const tenantScope =
+ input.tenantId == null ? { tenantId: { $exists: false } } : { tenantId: input.tenantId };
+ const released = await Delivery().updateOne(
+ {
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ ...tenantScope,
+ 'envelope.target.bindingId': input.bindingId,
+ $or: [
+ { handling: { $exists: false } },
+ { 'handling.conversationId': input.conversationId },
+ ],
+ actorReceipt: { $exists: false },
+ actorActionAdmittedAt: { $exists: true },
+ actorActionAdmissionId: input.admissionId,
+ },
+ { $unset: { actorActionAdmittedAt: 1, actorActionAdmissionId: 1 } },
+ );
+ return released.modifiedCount === 1;
+ }
+
+ async function hasAgentEventActorActionAdmission(
+ input: AgentEventActorActionAdmissionInput,
+ ): Promise<boolean> {
+ const tenantScope =
+ input.tenantId == null ? { tenantId: { $exists: false } } : { tenantId: input.tenantId };
+ return (
+ (await Delivery().exists({
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ ...tenantScope,
+ 'envelope.target.bindingId': input.bindingId,
+ $or: [
+ { handling: { $exists: false } },
+ { 'handling.conversationId': input.conversationId },
+ ],
+ actorReceipt: { $exists: false },
+ actorActionAdmittedAt: { $exists: true },
+ })) != null
+ );
+ }
+
+ async function getAgentEventActorReceipt(
+ input: GetAgentEventActorReceiptInput,
+ ): Promise<AgentEventActorReceipt | null> {
+ const tenantScope =
+ input.tenantId == null ? { tenantId: { $exists: false } } : { tenantId: input.tenantId };
+ const delivery = await Delivery()
+ .findOne({
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ ...tenantScope,
+ 'envelope.target.bindingId': input.bindingId,
+ 'handling.conversationId': input.conversationId,
+ actorReceipt: { $exists: true },
+ })
+ .select('+actorReceipt')
+ .lean<Pick<IAgentTriggerDelivery, 'actorReceipt'>>();
+ recordAgentEventActorReceiptMetric({
+ operation: 'read',
+ outcome: delivery?.actorReceipt == null ? 'miss' : 'hit',
+ ...(delivery?.actorReceipt?.resolution == null
+ ? {}
+ : { resolution: delivery.actorReceipt.resolution }),
+ });
+ return delivery?.actorReceipt ?? null;
+ }
+
+ /** Lazily moves one pre-ledger conversation receipt onto its already-terminal
+ * delivery. The public outcome must already agree, so migration can never
+ * rewrite history or manufacture a second winner. */
+ async function backfillAgentEventActorReceipt(
+ input: BackfillAgentEventActorReceiptInput,
+ ): Promise<boolean> {
+ const applied =
+ input.receipt.resolution === 'checkpoint_verified' ||
+ input.receipt.resolution === 'history_repaired';
+ if ((applied && input.status !== 'applied') || (!applied && input.status !== 'failed')) {
+ recordAgentEventActorReceiptMetric({
+ operation: 'backfill',
+ outcome: 'conflict',
+ resolution: input.receipt.resolution,
+ });
+ return false;
+ }
+ if (
+ input.receipt.resolution === 'checkpoint_verified' &&
+ (typeof input.receipt.checkpoint.checkpointId !== 'string' ||
+ input.receipt.checkpoint.checkpointId.length === 0)
+ ) {
+ recordAgentEventActorReceiptMetric({
+ operation: 'backfill',
+ outcome: 'conflict',
+ resolution: input.receipt.resolution,
+ });
+ return false;
+ }
+ const tenantScope =
+ input.tenantId == null ? { tenantId: { $exists: false } } : { tenantId: input.tenantId };
+ const actorReceipt: AgentEventActorReceipt = {
+ bindingId: input.bindingId,
+ ...input.receipt,
+ settledAt: input.settledAt,
+ };
+ const migrated = await Delivery()
+ .findOneAndUpdate(
+ {
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ ...tenantScope,
+ status: { $in: ['succeeded', 'dead'] },
+ 'envelope.target.bindingId': input.bindingId,
+ 'handling.conversationId': input.conversationId,
+ 'handling.generationCreatedAt': input.generationCreatedAt,
+ 'handling.status': input.status,
+ ...(applied && { 'handling.action': input.receipt.action }),
+ actorReceipt: { $exists: false },
+ },
+ {
+ $set: { status: 'succeeded', settledAt: input.settledAt, actorReceipt },
+ $max: { expiresAt: new Date(input.settledAt.getTime() + SUCCESS_RETENTION_MS) },
+ $unset: { lastError: 1 },
+ },
+ { new: true },
+ )
+ .select(
+ '_id orderingKey batchMemberIds status settledAt awaitTerminalHandling handling +actorReceipt',
+ )
+ .lean<
+ Pick<
+ IAgentTriggerDelivery,
+ | '_id'
+ | 'orderingKey'
+ | 'batchMemberIds'
+ | 'status'
+ | 'settledAt'
+ | 'awaitTerminalHandling'
+ | 'handling'
+ | 'actorReceipt'
+ >
+ >();
+ if (migrated != null) {
+ await finalizeAgentEventActorDelivery(migrated, input.settledAt);
+ recordAgentEventActorReceiptMetric({
+ operation: 'backfill',
+ outcome: 'success',
+ resolution: input.receipt.resolution,
+ });
+ return true;
+ }
+ const replayed = await Delivery()
+ .findOneAndUpdate(
+ {
+ deliveryKey: input.deliveryKey,
+ user: input.user,
+ ...tenantScope,
+ status: { $in: ['succeeded', 'dead'] },
+ 'envelope.target.bindingId': input.bindingId,
+ 'handling.conversationId': input.conversationId,
+ 'handling.generationCreatedAt': input.generationCreatedAt,
+ 'handling.status': input.status,
+ actorReceipt,
+ },
+ {
+ $set: { status: 'succeeded', settledAt: actorReceipt.settledAt },
+ $max: {
+ expiresAt: new Date(actorReceipt.settledAt.getTime() + SUCCESS_RETENTION_MS),
+ },
+ $unset: { lastError: 1 },
+ },
+ { new: true },
+ )
+ .select('_id orderingKey batchMemberIds status settledAt awaitTerminalHandling handling')
+ .lean<
+ Pick<
+ IAgentTriggerDelivery,
+ | '_id'
+ | 'orderingKey'
+ | 'batchMemberIds'
+ | 'status'
+ | 'settledAt'
+ | 'awaitTerminalHandling'
+ | 'handling'
+ >
+ >();
+ if (replayed != null) {
+ await finalizeAgentEventActorDelivery(replayed, actorReceipt.settledAt);
+ }
+ recordAgentEventActorReceiptMetric({
+ operation: 'backfill',
+ outcome: replayed != null ? 'replay' : 'conflict',
+ resolution: input.receipt.resolution,
+ });
+ return replayed != null;
+ }
+
+ async function getAgentEventActorReceiptStorageMetrics(
+ now: Date,
+ ): Promise<AgentEventActorReceiptStorageMetrics> {
+ const [byResolution, expiryEligible, retryDeliveries, deadDeliveries] = await Promise.all([
+ Delivery().aggregate<{ _id: AgentEventActorReceipt['resolution']; count: number }>([
+ { $match: { actorReceipt: { $exists: true } } },
+ { $group: { _id: '$actorReceipt.resolution', count: { $sum: 1 } } },
+ ]),
+ Delivery().countDocuments({ actorReceipt: { $exists: true }, expiresAt: { $lte: now } }),
+ Delivery().countDocuments({
+ status: 'pending',
+ attempts: { $gt: 0 },
+ 'envelope.target.bindingId': { $exists: true },
+ 'handling.status': 'started',
+ actorReceipt: { $exists: false },
+ }),
+ Delivery().countDocuments({
+ status: 'dead',
+ 'envelope.target.bindingId': { $exists: true },
+ 'handling.status': 'started',
+ actorReceipt: { $exists: false },
+ }),
+ ]);
+ const retainedByResolution: AgentEventActorReceiptStorageMetrics['retainedByResolution'] = {
+ checkpoint_verified: 0,
+ action_compensated: 0,
+ history_repaired: 0,
+ };
+ for (const row of byResolution) {
+ if (row._id in retainedByResolution) {
+ retainedByResolution[row._id] = row.count;
+ }
+ }
+ return { retainedByResolution, expiryEligible, retryDeliveries, deadDeliveries };
+ }
+
async function retryAgentTriggerDelivery(
input: AgentTriggerDeliveryFence & {
attempt: number;
@@ -1338,7 +1938,10 @@ export function createAgentTriggerDeliveryMethods(
async function getAgentTriggerDelivery(
deliveryKey: string,
): Promise<AgentTriggerDeliveryRecord | null> {
- const delivery = await Delivery().findOne({ deliveryKey }).lean<IAgentTriggerDelivery>();
+ const delivery = await Delivery()
+ .findOne({ deliveryKey })
+ .select('+actorReceipt')
+ .lean<IAgentTriggerDelivery>();
return delivery == null ? null : toRecord(delivery);
}
@@ -1417,60 +2020,31 @@ export function createAgentTriggerDeliveryMethods(
availableAt: Date,
): Promise<AgentTriggerDeliveryRecord | null> {
const candidate = await Delivery()
- .findOne({ _id: id, status: 'dead', batchRootId: { $exists: false } })
+ .findOne({
+ _id: id,
+ status: 'dead',
+ batchRootId: { $exists: false },
+ actorReceipt: { $exists: false },
+ actorActionAdmittedAt: { $exists: false },
+ handling: { $exists: false },
+ })
.lean<IAgentTriggerDelivery>();
if (candidate?._id == null) {
return null;
}
const previousRequeueCount = candidate.requeueCount ?? 0;
- const requeueCount = previousRequeueCount + 1;
- const memberIds = candidate.batchMemberIds ?? [];
- if (memberIds.length > 0) {
- await Delivery().updateMany(
- {
- _id: { $in: memberIds },
- orderingKey: candidate.orderingKey,
- batchRootId: candidate._id,
- status: { $in: ['staging', 'batched', 'succeeded', 'dead'] },
- ...(previousRequeueCount === 0
- ? {
- $or: [{ batchRootRequeueCount: 0 }, { batchRootRequeueCount: { $exists: false } }],
- }
- : { batchRootRequeueCount: previousRequeueCount }),
- },
- {
- $set: {
- status: 'batched',
- attempts: 0,
- batchRootId: candidate._id,
- batchRootRequeueCount: requeueCount,
- },
- $unset: {
- lastError: 1,
- result: 1,
- settledAt: 1,
- expiresAt: 1,
- },
- },
- );
- const preparedCount = await Delivery().countDocuments({
- _id: { $in: memberIds },
- orderingKey: candidate.orderingKey,
- status: 'batched',
- batchRootId: candidate._id,
- batchRootRequeueCount: requeueCount,
- });
- if (preparedCount !== memberIds.length) {
- throw new Error('Not every agent trigger batch receipt could be prepared for requeue');
- }
- }
-
+ /** Claim the root before touching members. A terminal receipt and requeue
+ * now serialize on this CAS; staging recovery finishes member preparation
+ * if the process exits before publication. */
const staged = await Delivery()
.findOneAndUpdate(
{
_id: candidate._id,
status: 'dead',
batchRootId: { $exists: false },
+ actorReceipt: { $exists: false },
+ actorActionAdmittedAt: { $exists: false },
+ handling: { $exists: false },
requeueCount: previousRequeueCount,
},
{
@@ -1491,6 +2065,7 @@ export function createAgentTriggerDeliveryMethods(
expiresAt: 1,
laneCleanupPendingAt: 1,
batchMembersSettledAt: 1,
+ handling: 1,
},
$inc: { requeueCount: 1 },
},
@@ -1501,8 +2076,8 @@ export function createAgentTriggerDeliveryMethods(
return null;
}
- // Requeue is a new lane admission. Retaining the dead letter's original
- // sequence could let it overlap a newer delivery that is already running.
+ // Requeue is a new lane admission. Publication first idempotently prepares
+ // its members, then allocates a new sequence on the original lane.
return toRecord(await publishStagedDelivery(staged));
}
@@ -1621,6 +2196,13 @@ export function createAgentTriggerDeliveryMethods(
deferAgentTriggerDeliveryAttempt,
completeAgentTriggerDelivery,
settleAgentTriggerHandlingOutcome,
+ admitAgentEventActorAction,
+ releaseAgentEventActorAction,
+ hasAgentEventActorActionAdmission,
+ settleAgentEventActorReceipt,
+ getAgentEventActorReceipt,
+ backfillAgentEventActorReceipt,
+ getAgentEventActorReceiptStorageMetrics,
retryAgentTriggerDelivery,
deadLetterAgentTriggerDelivery,
getAgentTriggerDelivery,
diff --git a/packages/data-schemas/src/schema/convo.ts b/packages/data-schemas/src/schema/convo.ts
index a4c8846e56..ac91a2d4e3 100644
--- a/packages/data-schemas/src/schema/convo.ts
+++ b/packages/data-schemas/src/schema/convo.ts
@@ -102,6 +102,7 @@ const convoSchema: Schema<IConversation> = new Schema(
type: [
{
invocationId: { type: String, required: true },
+ actionAdmitted: { type: Boolean, default: undefined },
status: {
type: String,
enum: [
@@ -219,6 +220,13 @@ convoSchema.index(
{ 'agentEventBinding.bindingId': 1 },
{ unique: true, sparse: true, name: 'agent_event_binding_unique' },
);
+convoSchema.index(
+ {
+ 'agentEventActorReconciliations.status': 1,
+ 'agentEventActorReconciliations.observedAt': 1,
+ },
+ { sparse: true, name: 'agent_event_actor_reconciliation_metrics' },
+);
// index for MeiliSearch sync operations
convoSchema.index({ _meiliIndex: 1, isTemporary: 1, expiredAt: 1 });
diff --git a/packages/data-schemas/src/schema/triggerDelivery.ts b/packages/data-schemas/src/schema/triggerDelivery.ts
index a7bbd27bea..7d23845fb7 100644
--- a/packages/data-schemas/src/schema/triggerDelivery.ts
+++ b/packages/data-schemas/src/schema/triggerDelivery.ts
@@ -51,6 +51,40 @@ const handlingSchema = new Schema(
{ _id: false },
);
+const actorReceiptSchema = new Schema(
+ {
+ bindingId: { type: String, required: true, maxlength: 256 },
+ resolution: {
+ type: String,
+ enum: ['checkpoint_verified', 'action_compensated', 'history_repaired'],
+ required: true,
+ },
+ checkpoint: {
+ type: new Schema(
+ {
+ threadId: { type: String, required: true, maxlength: 256 },
+ checkpointId: { type: String, maxlength: 512 },
+ checkpointNs: { type: String, required: true, maxlength: 512 },
+ },
+ { _id: false },
+ ),
+ required: true,
+ },
+ action: {
+ type: new Schema(
+ {
+ toolName: { type: String, required: true, maxlength: 256 },
+ toolCallId: { type: String, maxlength: 256 },
+ },
+ { _id: false },
+ ),
+ required: true,
+ },
+ settledAt: { type: Date, required: true },
+ },
+ { _id: false },
+);
+
const triggerDeliverySchema: Schema<IAgentTriggerDeliveryDocument> = new Schema(
{
deliveryKey: { type: String, required: true, maxlength: 128 },
@@ -84,6 +118,9 @@ const triggerDeliverySchema: Schema<IAgentTriggerDeliveryDocument> = new Schema(
batchMembersSettledAt: { type: Date },
awaitTerminalHandling: { type: Boolean },
handling: { type: handlingSchema },
+ actorReceipt: { type: actorReceiptSchema, select: false },
+ actorActionAdmittedAt: { type: Date, select: false },
+ actorActionAdmissionId: { type: String, maxlength: 64, select: false },
leaseBy: { type: String },
leaseUntil: { type: Date },
claimToken: { type: String },
@@ -117,6 +154,7 @@ triggerDeliverySchema.index(
{ sparse: true },
);
triggerDeliverySchema.index({ status: 1, updatedAt: -1 });
+triggerDeliverySchema.index({ 'actorReceipt.resolution': 1 }, { sparse: true });
triggerDeliverySchema.index({ stagingRecoveryAt: 1 }, { sparse: true });
triggerDeliverySchema.index({ laneCleanupPendingAt: 1 }, { sparse: true });
// Only successful rows receive expiresAt. Dead letters remain available until
diff --git a/packages/data-schemas/src/types/convo.ts b/packages/data-schemas/src/types/convo.ts
index a99d2ffdf4..9e8cb3ca7a 100644
--- a/packages/data-schemas/src/types/convo.ts
+++ b/packages/data-schemas/src/types/convo.ts
@@ -31,6 +31,9 @@ export interface IAgentEventActorState {
export interface IAgentEventActorReconciliation {
invocationId: string;
+ /** New-protocol executions acquire the delivery-owned action admission CAS
+ * before the external action may run. Absent only on mixed-version rows. */
+ actionAdmitted?: boolean;
status:
| 'invocation_pending'
| 'persistence_pending'
diff --git a/packages/data-schemas/src/types/triggerDelivery.ts b/packages/data-schemas/src/types/triggerDelivery.ts
index 79a3f6501b..450582629b 100644
--- a/packages/data-schemas/src/types/triggerDelivery.ts
+++ b/packages/data-schemas/src/types/triggerDelivery.ts
@@ -1,4 +1,5 @@
import type { Document, Types } from 'mongoose';
+import type { IAgentEventActorReconciliation } from './convo';
export type AgentTriggerDeliveryStatus =
| 'staging'
@@ -23,6 +24,18 @@ export interface AgentTriggerHandlingState {
};
}
+/** Private terminal proof for one delivery-owned event-actor invocation. */
+export interface AgentEventActorReceipt {
+ bindingId: string;
+ resolution: 'checkpoint_verified' | 'action_compensated' | 'history_repaired';
+ checkpoint: IAgentEventActorReconciliation['checkpoint'];
+ action: {
+ toolName: string;
+ toolCallId?: string;
+ };
+ settledAt: Date;
+}
+
export interface AgentTriggerDeliveryFailure {
code: string;
message: string;
@@ -67,6 +80,12 @@ export interface IAgentTriggerDelivery {
* an authoritative terminal handling outcome. */
awaitTerminalHandling?: boolean;
handling?: AgentTriggerHandlingState;
+ actorReceipt?: AgentEventActorReceipt;
+ /** Delivery-owned serialization point acquired exactly once before an event
+ * actor may invoke an external action. */
+ actorActionAdmittedAt?: Date;
+ /** Attempt identity that fences admission takeover and release. */
+ actorActionAdmissionId?: string;
leaseBy?: string;
leaseUntil?: Date;
claimToken?: string;