Skip to content

Enhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active

📬 feat: Move Agent Queued Turns Server-Side (#15370)

Danny Avilac7f0a8f24751
diff --git a/CONTEXT.md b/CONTEXT.md
index 30f79289b6..3880719db0 100644
--- a/CONTEXT.md
+++ b/CONTEXT.md
@@ -13,6 +13,7 @@
- **Subagent completion wakeup**: a durable internal `continue` trigger pre-registered before detached child execution so a process crash cannot lose the wakeup. Delivery defers until the child's terminal transcript is persisted, targets the initiating agent and exact parent response branch, carries task metadata rather than child output, waits for the parent generation to settle, and starts the parent turn that collects the result through the existing task store.
- **Agent continuation preparation**: the single source-dispatch seam that resolves a durable `continue` delivery immediately before admission. Bound Event Actor work selects its binding adapter; internal completion work selects an adapter by stable source identity. Preparation may resolve authoritative input and branch state or settle already-consumed work, but it does not own source result truth, delivery ordering, or generation execution.
- **Warm terminal steer continuation**: a queued steer accepted before a generation's terminal boundary may continue the same SDK `Run` without creating a replacement generation. After parallel Stop hooks fold, the serialized StopFinalize phase tells the job store whether another continuation is already planned or terminal progress is forbidden. The store atomically chooses among claiming the current protocol-v2 FIFO batch, keeping empty admission open for an already-planned segment, and sealing admission so every racing or later message becomes an ordinary follow-up. Claimed steer receipts remain the crash-recovery authority; protocol-v1 generations always seal because they cannot recover an ambiguous terminal claim. Tool-batch, preemption, and terminal boundaries share one durable apply-and-inject adapter, while the SDK owns the bounded Stop-continuation loop.
+- **Agent queued turn**: a server-owned ordinary follow-up accepted while an Agent generation is active. Its Mongo row is the sole FIFO, payload, and lifecycle authority; the trigger delivery is only a replayable wakeup and Agent execution enrollment is only an execution adapter. The lifecycle reserves the deterministic delivery identity before publication, admits only after the captured branch has a clean durable predecessor outcome, and commits the source-owned generation receipt only after provider invocation has been enrolled. An accepted or deduplicated loopback response requires that exact receipt; a process death before it remains explicit admission-indeterminate evidence instead of silently consuming the text. Admission reconciliation is leased, backoff-scheduled work: exact generation evidence may repair legacy ambiguity, while current source-owned ambiguity never infers provider outcome from transient job state. Aborted or failed predecessors and exhausted admission attempts remain visible terminal rows. Transport-ambiguous enqueue outcomes stay non-resendable until exact request-identity reconciliation succeeds; an elapsed client observation window may hide the warning locally but never converts ambiguity into permission to submit again. Conversation deletion cancels the rows, retires their deliveries, and removes their payloads before deleting the conversation wave.
- **Subagent activity stream**: an observational, task-scoped live projection of bounded child progress for the currently open private panel. It may cross API replicas through Redis, never carries hidden reasoning text, and never controls or settles execution. The durable child thread remains canonical and its existing polling view is the fallback for missed or unavailable live events.
- **Agent event handling outcome**: the durable, generation-fenced result of a previously accepted event delivery. `started` proves generation admission; terminal states distinguish verified tool application, clean completion without action, failure, and cancellation. Transport success remains separate so an accepted event cannot masquerade as completed work.
- **Agent event expected action**: an optional source-declared tool name and bounded argument subset evaluated against host-observed completed run steps. It is evidence policy, not authorization and not a model-authored success claim.
diff --git a/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js b/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js
index 693aa6cb56..09d41af482 100644
--- a/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js
+++ b/api/server/controllers/agents/__tests__/request.resumeMetadata.spec.js
@@ -76,6 +76,8 @@ const mockExemptFromConcurrencyLimiter = jest.fn();
const mockRecordScheduleOutcome = jest.fn();
const mockIsScheduleLive = jest.fn();
const mockDeleteAgentCheckpoint = jest.fn();
+const mockSettleAgentQueuedTurnExecutionAdmission = jest.fn();
+const mockVerifyAgentQueuedTurnExecutionAdmission = jest.fn();
const mockExecuteAgentEventActor = jest.fn();
const mockResumeAgentEventActor = jest.fn();
const mockCreateAgentEventActorTurn = jest.fn((input, dependencies) => {
@@ -379,6 +381,13 @@ jest.mock('~/server/services/Schedules', () => ({
isScheduleLive: (...args) => mockIsScheduleLive(...args),
}));
+jest.mock('~/server/services/Agents/triggers', () => ({
+ settleAgentQueuedTurnExecutionAdmission: (...args) =>
+ mockSettleAgentQueuedTurnExecutionAdmission(...args),
+ verifyAgentQueuedTurnExecutionAdmission: (...args) =>
+ mockVerifyAgentQueuedTurnExecutionAdmission(...args),
+}));
+
const AgentController = require('../request');
const { ErrorTypes } = require('librechat-data-provider');
const { disposeClient: mockDisposeClient } = require('~/server/cleanup');
@@ -485,6 +494,8 @@ describe('ResumableAgentController resume metadata', () => {
mockSaveMessage.mockResolvedValue({});
mockSaveConvo.mockResolvedValue({});
mockDeleteAgentCheckpoint.mockResolvedValue(undefined);
+ mockSettleAgentQueuedTurnExecutionAdmission.mockResolvedValue(true);
+ mockVerifyAgentQueuedTurnExecutionAdmission.mockResolvedValue(true);
mockGetAgentEventActorSnapshot.mockResolvedValue({ state: null, reconciliations: [] });
mockCommitAgentEventActorState.mockResolvedValue({ status: 'stale' });
mockBeginAgentEventActorLegacyTurn.mockResolvedValue(true);
@@ -1135,6 +1146,131 @@ describe('ResumableAgentController resume metadata', () => {
);
});
+ it('keeps a queued-turn source nonterminal when provider initialization fails', async () => {
+ mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
+ const admissionSource = {
+ source: 'agent-queued-turn',
+ sourceId: 'queued-turn-1',
+ claimId: 'queued-delivery-1',
+ claimBy: 'queued-worker-1',
+ };
+ const req = {
+ _isAgentTrigger: true,
+ user: { id: 'user-123', tenantId: 'tenant-1' },
+ body: {
+ text: 'Run the queued turn.',
+ messageId: 'queued-user-message',
+ parentMessageId: 'persisted-response_',
+ conversationId: 'conversation-123',
+ clientRequestId: 'queued-delivery-1',
+ agentContinuationAdmission: admissionSource,
+ endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
+ },
+ config: {},
+ };
+ const res = createResumableResponse();
+ const initializeClient = jest.fn().mockRejectedValue(new Error('stop before provider'));
+
+ await AgentController(req, res, jest.fn(), initializeClient, null);
+
+ expect(initializeClient).toHaveBeenCalledTimes(1);
+ expect(mockSettleAgentQueuedTurnExecutionAdmission).not.toHaveBeenCalled();
+ expect(res.status).toHaveBeenCalledWith(500);
+ });
+
+ it('commits a queued-turn receipt after provider invocation and before accepting HTTP', async () => {
+ mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
+ const admissionSource = {
+ source: 'agent-queued-turn',
+ sourceId: 'queued-turn-1',
+ claimId: 'queued-delivery-1',
+ claimBy: 'queued-worker-1',
+ };
+ const req = {
+ _isAgentTrigger: true,
+ user: { id: 'user-123', tenantId: 'tenant-1' },
+ body: {
+ text: 'Run the queued turn.',
+ messageId: 'queued-user-message',
+ parentMessageId: 'persisted-response_',
+ conversationId: 'conversation-123',
+ clientRequestId: 'queued-delivery-1',
+ agentContinuationAdmission: admissionSource,
+ endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
+ },
+ config: {},
+ };
+ const sendMessage = jest.fn(() => new Promise(() => {}));
+ const initializeClient = jest.fn().mockResolvedValue({ client: { options: {}, sendMessage } });
+ const res = createResumableResponse();
+
+ await AgentController(req, res, jest.fn(), initializeClient, null);
+ await nextTick();
+
+ expect(mockSettleAgentQueuedTurnExecutionAdmission).toHaveBeenCalledWith(admissionSource, {
+ userId: 'user-123',
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-123',
+ clientRequestId: 'queued-delivery-1',
+ generationId: 'conversation-123',
+ generationCreatedAt: 1000,
+ });
+ expect(initializeClient.mock.invocationCallOrder[0]).toBeLessThan(
+ sendMessage.mock.invocationCallOrder[0],
+ );
+ expect(sendMessage.mock.invocationCallOrder[0]).toBeLessThan(
+ mockSettleAgentQueuedTurnExecutionAdmission.mock.invocationCallOrder[0],
+ );
+ expect(mockSettleAgentQueuedTurnExecutionAdmission.mock.invocationCallOrder[0]).toBeLessThan(
+ res.json.mock.invocationCallOrder[0],
+ );
+ });
+
+ it('fails closed after provider invocation when the queued-turn receipt is unavailable', async () => {
+ mockGetMessages.mockResolvedValue([{ _id: 'persisted-parent' }]);
+ mockSettleAgentQueuedTurnExecutionAdmission.mockRejectedValue(new Error('mongo unavailable'));
+ const req = {
+ _isAgentTrigger: true,
+ user: { id: 'user-123' },
+ body: {
+ text: 'Run the queued turn.',
+ messageId: 'queued-user-message',
+ parentMessageId: 'persisted-response_',
+ conversationId: 'conversation-123',
+ clientRequestId: 'queued-delivery-1',
+ agentContinuationAdmission: {
+ source: 'agent-queued-turn',
+ sourceId: 'queued-turn-1',
+ claimId: 'queued-delivery-1',
+ claimBy: 'queued-worker-1',
+ },
+ endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
+ },
+ config: {},
+ };
+ const sendMessage = jest.fn(() => new Promise(() => {}));
+ const initializeClient = jest.fn().mockResolvedValue({ client: { options: {}, sendMessage } });
+ const res = createResumableResponse();
+
+ await AgentController(req, res, jest.fn(), initializeClient, null);
+ await nextTick();
+
+ expect(mockGenerationJobManager.beginProviderExecution).toHaveBeenCalledTimes(1);
+ expect(sendMessage).toHaveBeenCalledTimes(1);
+ expect(res.status).toHaveBeenCalledWith(500);
+ expect(mockGenerationJobManager.completeJob).toHaveBeenCalledWith(
+ 'conversation-123',
+ 'mongo unavailable',
+ 1000,
+ expect.objectContaining({ beforeErrorPublication: expect.any(Function) }),
+ );
+ expect(mockGenerationJobManager.markProviderExecutionDrained).toHaveBeenCalledWith(
+ 'conversation-123',
+ 1000,
+ 'provider-segment-1',
+ );
+ });
+
it('prefetches conversation state before admission and joins it with job metadata', async () => {
let resolveConversation;
let signalMetadataStarted;
@@ -2042,6 +2178,61 @@ describe('ResumableAgentController resume metadata', () => {
expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
});
+ it('fails closed when a deduplicated queued generation lacks its source receipt', async () => {
+ const reacquired = wonGenerationClaim({
+ streamId: 'conversation-123',
+ conversationId: 'conversation-123',
+ claimToken: 'reacquired-token',
+ });
+ mockGenerationJobManager.claimGeneration.mockResolvedValue(reacquired);
+ mockGenerationJobManager.resumeClaimedGeneration.mockResolvedValue({
+ ...reacquired.existing,
+ startedAt: 42,
+ });
+ mockVerifyAgentQueuedTurnExecutionAdmission.mockRejectedValue(
+ new Error('source receipt missing'),
+ );
+ const admissionSource = {
+ source: 'agent-queued-turn',
+ sourceId: 'queued-turn-1',
+ claimId: 'queued-delivery-1',
+ claimBy: 'queued-worker-1',
+ };
+ const req = {
+ _isAgentTrigger: true,
+ user: { id: 'user-123', tenantId: 'tenant-1' },
+ body: {
+ text: 'Retry a queued generation.',
+ messageId: 'user-msg',
+ parentMessageId: 'assistant-1',
+ clientRequestId: 'queued-delivery-1',
+ conversationId: 'conversation-123',
+ agentContinuationAdmission: admissionSource,
+ endpointOption: { endpoint: 'agents', modelOptions: { model: 'gpt-4.1' } },
+ },
+ config: {},
+ };
+ const res = createResumableResponse();
+
+ await AgentController(req, res, jest.fn(), jest.fn(), null);
+
+ expect(mockVerifyAgentQueuedTurnExecutionAdmission).toHaveBeenCalledWith(admissionSource, {
+ userId: 'user-123',
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-123',
+ clientRequestId: 'queued-delivery-1',
+ generationId: 'conversation-123',
+ generationCreatedAt: 42,
+ });
+ expect(res.status).toHaveBeenCalledWith(503);
+ expect(res.json).toHaveBeenCalledWith(
+ expect.objectContaining({
+ code: 'SERVER_NOT_READY',
+ }),
+ );
+ expect(mockGenerationJobManager.createJob).not.toHaveBeenCalled();
+ });
+
it('caps a reacquired v2 lease to the immutable v1 live-job protocol', async () => {
const reacquired = wonGenerationClaim({
streamId: 'conversation-123',
@@ -2090,7 +2281,7 @@ describe('ResumableAgentController resume metadata', () => {
conversationId: 'orig-stream',
claimedAt: Date.now() - 60000,
claimToken: 'existing-token',
- startedAt: Date.now() - 59000,
+ startedAt: 1000,
},
});
mockGenerationJobManager.getJob.mockResolvedValue(undefined);
@@ -2112,6 +2303,7 @@ describe('ResumableAgentController resume metadata', () => {
expect(res.json).toHaveBeenCalledWith({
streamId: 'orig-stream',
conversationId: 'orig-stream',
+ generationCreatedAt: 1000,
status: 'resumed',
generationProtocolVersion: 1,
});
@@ -2127,7 +2319,7 @@ describe('ResumableAgentController resume metadata', () => {
conversationId: 'orig-stream',
claimedAt: Date.now() - 60_000,
claimToken: 'existing-token',
- startedAt: Date.now() - 59_000,
+ startedAt: 2000,
generationProtocolVersion: 2,
},
});
@@ -2150,6 +2342,7 @@ describe('ResumableAgentController resume metadata', () => {
expect(res.json).toHaveBeenCalledWith({
conversationId: 'orig-stream',
+ generationCreatedAt: 2000,
status: 'settled',
generationProtocolVersion: 2,
});
diff --git a/api/server/controllers/agents/queuedTurns.js b/api/server/controllers/agents/queuedTurns.js
new file mode 100644
index 0000000000..3e66c9d3dd
--- /dev/null
+++ b/api/server/controllers/agents/queuedTurns.js
@@ -0,0 +1,80 @@
+const {
+ handleAgentQueuedTurnEnqueue,
+ handleAgentQueuedTurnList,
+ handleAgentQueuedTurnCancel,
+} = require('@librechat/api');
+const { logger } = require('@librechat/data-schemas');
+const db = require('~/models');
+const { createAgentAccessCheck } = require('./steer');
+const dependencies = (req) => {
+ /** Keep trigger-service composition lazy so importing the route for unrelated
+ * middleware probes does not initialize or require the worker graph. */
+ const {
+ scheduleAgentQueuedTurn,
+ cancelAgentQueuedTurn,
+ } = require('~/server/services/Agents/triggers');
+ return {
+ methods: db,
+ lifecycle: {
+ schedule: scheduleAgentQueuedTurn,
+ cancel: cancelAgentQueuedTurn,
+ },
+ getFiles: db.getFiles,
+ updateFilesUsage: db.updateFilesUsage,
+ checkAgentAccess: createAgentAccessCheck(req),
+ isPrincipalActive: db.isAgentTriggerPrincipalActive,
+ };
+};
+
+const send = (res, result) => res.status(result.status).json(result.body);
+
+const AgentQueuedTurnEnqueueController = async (req, res) => {
+ try {
+ return send(
+ res,
+ await handleAgentQueuedTurnEnqueue(req.user ?? {}, req.body ?? {}, dependencies(req)),
+ );
+ } catch (error) {
+ logger.error('[AgentQueuedTurns] Failed to enqueue turn', error);
+ return res.status(500).json({ code: 'QUEUED_TURN_FAILED' });
+ }
+};
+
+const AgentQueuedTurnListController = async (req, res) => {
+ try {
+ return send(
+ res,
+ await handleAgentQueuedTurnList(
+ req.user ?? {},
+ req.query?.conversationId,
+ dependencies(req),
+ req.query?.clientRequestIds,
+ ),
+ );
+ } catch (error) {
+ logger.error('[AgentQueuedTurns] Failed to list turns', error);
+ return res.status(500).json({ code: 'QUEUED_TURN_LIST_FAILED' });
+ }
+};
+
+const AgentQueuedTurnCancelController = async (req, res) => {
+ try {
+ return send(
+ res,
+ await handleAgentQueuedTurnCancel(
+ req.user ?? {},
+ req.params?.queuedTurnId,
+ dependencies(req),
+ ),
+ );
+ } catch (error) {
+ logger.error('[AgentQueuedTurns] Failed to cancel turn', error);
+ return res.status(500).json({ code: 'QUEUED_TURN_CANCEL_FAILED' });
+ }
+};
+
+module.exports = {
+ AgentQueuedTurnEnqueueController,
+ AgentQueuedTurnListController,
+ AgentQueuedTurnCancelController,
+};
diff --git a/api/server/controllers/agents/request.js b/api/server/controllers/agents/request.js
index 1941594534..363012e674 100644
--- a/api/server/controllers/agents/request.js
+++ b/api/server/controllers/agents/request.js
@@ -523,20 +523,30 @@ function sendSettledGeneration(
conversationId,
startupTelemetry,
generationProtocolVersion,
+ generationCreatedAt,
) {
startupTelemetry?.end('deduplicated');
if (generationProtocolVersion < GENERATION_PROTOCOL_V2) {
return sendGenerationJson(
res,
200,
- { streamId, conversationId, status: 'resumed' },
+ {
+ streamId,
+ conversationId,
+ ...(generationCreatedAt != null && { generationCreatedAt }),
+ status: 'resumed',
+ },
generationProtocolVersion,
);
}
return sendGenerationJson(
res,
200,
- { conversationId, status: 'settled' },
+ {
+ conversationId,
+ ...(generationCreatedAt != null && { generationCreatedAt }),
+ status: 'settled',
+ },
generationProtocolVersion,
);
}
@@ -753,6 +763,55 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
req._isAgentTrigger === true &&
!isNewConvo &&
(parentMessageId !== Constants.NO_PARENT || isBoundEventContinuation);
+ const queuedTurnAdmissionSource = isTriggerContinuation
+ ? req.body?.agentContinuationAdmission
+ : undefined;
+ const hasQueuedTurnAdmissionSource = queuedTurnAdmissionSource != null;
+ const verifyQueuedTurnAdmission = async (generationId, generationCreatedAt) => {
+ if (!hasQueuedTurnAdmissionSource) {
+ return true;
+ }
+ if (
+ typeof clientRequestId !== 'string' ||
+ !Number.isSafeInteger(generationCreatedAt) ||
+ generationCreatedAt < 0
+ ) {
+ return false;
+ }
+ try {
+ const {
+ verifyAgentQueuedTurnExecutionAdmission,
+ } = require('~/server/services/Agents/triggers');
+ const confirmed = await verifyAgentQueuedTurnExecutionAdmission(queuedTurnAdmissionSource, {
+ userId,
+ ...(tenantId != null && { tenantId }),
+ conversationId,
+ clientRequestId,
+ generationId,
+ generationCreatedAt,
+ });
+ return confirmed === true;
+ } catch (error) {
+ logger.warn(
+ '[ResumableAgentController] Deduplicated queued-turn admission is not confirmed',
+ error,
+ );
+ return false;
+ }
+ };
+ const rejectUnconfirmedQueuedTurnAdmission = () => {
+ res.set('Retry-After', '1');
+ startupTelemetry?.end('deduplicated');
+ return sendGenerationJson(
+ res,
+ 503,
+ {
+ code: 'SERVER_NOT_READY',
+ error: 'Queued turn execution is still being confirmed. Please retry shortly.',
+ },
+ generationProtocolVersion,
+ );
+ };
if (
await isUnpersistedPreliminaryParent({
@@ -942,6 +1001,14 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
? GENERATION_PROTOCOL_V2
: 1,
);
+ if (
+ !(await verifyQueuedTurnAdmission(
+ existingLiveGeneration.streamId,
+ existingLiveGeneration.startedAt,
+ ))
+ ) {
+ return rejectUnconfirmedQueuedTurnAdmission();
+ }
startupTelemetry?.end('deduplicated');
return sendGenerationJson(
res,
@@ -1006,12 +1073,16 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
// generation (usually fast completion + cleanup), never an abandoned
// pre-create lease that may be taken over and billed again. There is no
// attachable stream; the settled response refetches persisted history.
+ if (!(await verifyQueuedTurnAdmission(existingStreamId, claim.existing.startedAt))) {
+ return rejectUnconfirmedQueuedTurnAdmission();
+ }
return sendSettledGeneration(
res,
existingStreamId,
claim.existing.conversationId,
startupTelemetry,
generationProtocolVersion,
+ claim.existing.startedAt,
);
}
if (!liveJob && isLegacyTokenlessClaim && claimAgeMs >= IDEMPOTENCY_STARTUP_GRACE_MS) {
@@ -1020,6 +1091,9 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
* attach/refetch path: this covers fast completion without starting a
* second billed generation, while an abandoned pre-create claim ages
* out under the old server's bounded TTL. */
+ if (!(await verifyQueuedTurnAdmission(existingStreamId, claim.existing.startedAt))) {
+ return rejectUnconfirmedQueuedTurnAdmission();
+ }
return sendSettledGeneration(
res,
existingStreamId,
@@ -1108,6 +1182,9 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
(startedAt != null && liveJob.createdAt !== startedAt) ||
(liveClientRequestId != null && liveClientRequestId !== clientRequestId);
if (replacedGeneration) {
+ if (!(await verifyQueuedTurnAdmission(existingStreamId, startedAt))) {
+ return rejectUnconfirmedQueuedTurnAdmission();
+ }
// streamId === conversationId, so a later turn reuses the same route.
// Never pair this stale POST's optimistic submission with that newer
// job's SSE snapshot. If the replacement is still active, distinguish
@@ -1141,6 +1218,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
claim.existing.conversationId,
startupTelemetry,
generationProtocolVersion,
+ claim.existing.startedAt,
);
}
if (liveClientRequestId == null && !isLegacyTokenlessClaim) {
@@ -1164,6 +1242,9 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
clientRequestId,
streamId: existingStreamId,
});
+ if (!(await verifyQueuedTurnAdmission(existingStreamId, liveJob.createdAt))) {
+ return rejectUnconfirmedQueuedTurnAdmission();
+ }
startupTelemetry?.end('deduplicated');
return sendGenerationJson(
res,
@@ -1518,7 +1599,6 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
status: 409,
});
}
-
acceptAgentStartupTelemetry(req, streamId);
startupTelemetry?.mark('metadata_persisted');
req._resumableStreamId = streamId;
@@ -1543,14 +1623,29 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
recoveredSteerCommitted = true;
};
- // Send JSON response IMMEDIATELY so client can connect to SSE stream
- // This is critical: tool loading (MCP OAuth) may emit events that the client needs to receive
- sendGenerationJson(
- res,
- 200,
- { streamId, conversationId, generationCreatedAt: jobCreatedAt, status: 'started' },
- generationProtocolVersion,
- );
+ // Ordinary clients receive the stream id immediately so they can attach
+ // before tool loading emits events. Source-owned loopback work delays only
+ // until its provider invocation and Mongo receipt exist.
+ let generationStartResponseSent = false;
+ const sendGenerationStarted = () => {
+ if (generationStartResponseSent || res.headersSent) {
+ return;
+ }
+ generationStartResponseSent = true;
+ sendGenerationJson(
+ res,
+ 200,
+ { streamId, conversationId, generationCreatedAt: jobCreatedAt, status: 'started' },
+ generationProtocolVersion,
+ );
+ };
+ /** Ordinary clients need the stream id before tool discovery. A queued
+ * source instead keeps its local loopback response open until the provider
+ * invocation exists, so an accepted HTTP result can never retire text that
+ * died between job creation and provider startup. */
+ if (!hasQueuedTurnAdmissionSource) {
+ sendGenerationStarted();
+ }
await attachConversationCreatedAt(req, conversationId, conversationAnchorPromise).then(() =>
startupTelemetry?.mark('conversation_resolved'),
@@ -2298,6 +2393,21 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
})
: client.sendMessage(text, messageOptions);
+ if (hasQueuedTurnAdmissionSource) {
+ const {
+ settleAgentQueuedTurnExecutionAdmission,
+ } = require('~/server/services/Agents/triggers');
+ await settleAgentQueuedTurnExecutionAdmission(queuedTurnAdmissionSource, {
+ userId,
+ ...(tenantId != null && { tenantId }),
+ conversationId,
+ clientRequestId,
+ generationId: streamId,
+ generationCreatedAt: jobCreatedAt,
+ });
+ sendGenerationStarted();
+ }
+
if (titleEligible && titleTiming === 'immediate') {
immediateTitlePromise = addTitle(req, {
text: text || getAttachmentTitleText(req.body.files),
@@ -2780,6 +2890,14 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
job.abortController.signal.removeEventListener('abort', abortTitleOnJobAbort);
acceptsTitleEvents = false;
resolveConvoReady();
+ if (!res.headersSent) {
+ sendGenerationJson(
+ res,
+ 500,
+ { error: error.message || 'Failed to start generation' },
+ generationProtocolVersion,
+ );
+ }
try {
await recordEventActorPersistenceFailure(error);
} catch (reconciliationError) {
@@ -2899,6 +3017,14 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
`[ResumableAgentController] Unhandled error in background generation: ${err.message}`,
);
startupTelemetry?.end('error', err);
+ if (!res.headersSent) {
+ sendGenerationJson(
+ res,
+ 500,
+ { error: err.message || 'Failed to start generation' },
+ generationProtocolVersion,
+ );
+ }
let errorFinalized = false;
if (!pausePersistenceFailed) {
errorFinalized =
diff --git a/api/server/controllers/agents/steer.js b/api/server/controllers/agents/steer.js
index e6c9a0a19a..ecce5b8067 100644
--- a/api/server/controllers/agents/steer.js
+++ b/api/server/controllers/agents/steer.js
@@ -203,3 +203,4 @@ module.exports = SteerController;
module.exports.SteerDeliveryController = SteerDeliveryController;
module.exports.SteerCancelController = SteerCancelController;
module.exports.SteerArmController = SteerArmController;
+module.exports.createAgentAccessCheck = createAgentAccessCheck;
diff --git a/api/server/routes/agents/__tests__/idempotencyLimiter.spec.js b/api/server/routes/agents/__tests__/idempotencyLimiter.spec.js
index e7281ee2ed..373266a6c0 100644
--- a/api/server/routes/agents/__tests__/idempotencyLimiter.spec.js
+++ b/api/server/routes/agents/__tests__/idempotencyLimiter.spec.js
@@ -67,6 +67,11 @@ jest.mock('~/server/controllers/agents/steer', () => {
controller.SteerArmController = (_req, _res, next) => next();
return controller;
});
+jest.mock('~/server/controllers/agents/queuedTurns', () => ({
+ AgentQueuedTurnEnqueueController: (_req, res) => res.status(202).json({ queued: true }),
+ AgentQueuedTurnListController: (_req, res) => res.status(200).json({ queuedTurns: [] }),
+ AgentQueuedTurnCancelController: (_req, res) => res.status(200).json({ cancelled: true }),
+}));
jest.mock('~/models', () => ({}));
jest.mock('~/server/services/Schedules', () => ({}));
@@ -150,4 +155,29 @@ describe('start-generation idempotency before message limiters', () => {
expect(mockIpLimiter).not.toHaveBeenCalled();
expect(mockUserLimiter).not.toHaveBeenCalled();
});
+
+ it('keeps read-only queued-turn polling outside message admission limiters', async () => {
+ const responses = await Promise.all(
+ Array.from({ length: 3 }, () =>
+ request(app).get('/agents/chat/queued-turns').query({ conversationId: 'conversation-1' }),
+ ),
+ );
+
+ expect(responses.map((response) => response.status)).toEqual([200, 200, 200]);
+ expect(responses[0].body).toEqual({ queuedTurns: [] });
+ expect(mockIpLimiter).not.toHaveBeenCalled();
+ expect(mockUserLimiter).not.toHaveBeenCalled();
+ });
+
+ it.each([
+ ['enqueue', () => request(app).post('/agents/chat/queued-turns').send({ text: 'next' })],
+ ['cancel', () => request(app).delete('/agents/chat/queued-turns/queued-turn-1')],
+ ])('keeps queued-turn %s mutations behind message admission limiters', async (_label, send) => {
+ const response = await send();
+
+ expect(response.status).toBe(429);
+ expect(response.body).toEqual({ limited: 'ip' });
+ expect(mockIpLimiter).toHaveBeenCalledTimes(1);
+ expect(mockUserLimiter).not.toHaveBeenCalled();
+ });
});
diff --git a/api/server/routes/agents/index.js b/api/server/routes/agents/index.js
index b6c237708a..1805d9e135 100644
--- a/api/server/routes/agents/index.js
+++ b/api/server/routes/agents/index.js
@@ -35,6 +35,11 @@ const {
messageUserLimiter,
} = require('~/server/middleware');
const SteerController = require('~/server/controllers/agents/steer');
+const {
+ AgentQueuedTurnEnqueueController,
+ AgentQueuedTurnListController,
+ AgentQueuedTurnCancelController,
+} = require('~/server/controllers/agents/queuedTurns');
const {
GENERATION_PROTOCOL_HEADER,
GENERATION_PROTOCOL_V2,
@@ -228,7 +233,12 @@ router.get('/chat/stream/:streamId', async (req, res) => {
logger.warn(`[AgentStream] Refusing stream with invalid generation identity: ${streamId}`);
return sendGenerationJson(res, 403, { error: 'Unauthorized' }, requestProtocolVersion);
}
- const streamTelemetry = createSseStreamTelemetry({ req, res, streamId, isResume });
+ const streamTelemetry = createSseStreamTelemetry({
+ req,
+ res,
+ streamId,
+ isResume,
+ });
res.setHeader('Content-Encoding', 'identity');
res.setHeader('Content-Type', 'text/event-stream');
@@ -383,7 +393,9 @@ router.get('/chat/stream/:streamId', async (req, res) => {
final: true,
reconcile: true,
reconcileReason: generationReplaced ? 'generation_replaced' : 'terminal_payload_missing',
- ...(expectedGenerationTerminal && { terminalStatus: currentJob.status }),
+ ...(expectedGenerationTerminal && {
+ terminalStatus: currentJob.status,
+ }),
generationCreatedAt: authorizedGenerationCreatedAt,
conversation: {
conversationId: currentJob?.conversationId ?? job.conversationId ?? streamId,
@@ -975,7 +987,9 @@ router.post('/chat/abort', configMiddleware, async (req, res, next) => {
// Steers that never reached an injection boundary — restored client-side
// as queued chips so the user's words aren't dropped with the abort.
...(!abortResult.persistenceFailed &&
- abortResult.pendingSteers?.length > 0 && { pendingSteers: abortResult.pendingSteers }),
+ abortResult.pendingSteers?.length > 0 && {
+ pendingSteers: abortResult.pendingSteers,
+ }),
});
}
@@ -1076,6 +1090,28 @@ router.post(
SteerController.SteerArmController,
);
+router.post(
+ '/chat/queued-turns',
+ configMiddleware,
+ ...steerLimiters,
+ createMessageFilterPii({
+ getConfig: (req) => req.config?.messageFilter?.pii,
+ getFilters: (req) => req.config?.filters,
+ getFiles,
+ }),
+ moderateText,
+ AgentQueuedTurnEnqueueController,
+);
+/** Synchronizing durable queue state is read-only and polled while work is
+ * pending. It must not consume the model-submission admission budget. */
+router.get('/chat/queued-turns', configMiddleware, AgentQueuedTurnListController);
+router.delete(
+ '/chat/queued-turns/:queuedTurnId',
+ configMiddleware,
+ ...steerLimiters,
+ AgentQueuedTurnCancelController,
+);
+
router.use('/', v1);
const chatRouter = express.Router();
diff --git a/api/server/services/Agents/triggers.js b/api/server/services/Agents/triggers.js
index 0d1eea95af..1548511269 100644
--- a/api/server/services/Agents/triggers.js
+++ b/api/server/services/Agents/triggers.js
@@ -6,10 +6,20 @@ const {
SUBAGENT_COMPLETION_SOURCE,
createBackgroundToolCompletionWakeupResolver,
BACKGROUND_TOOL_COMPLETION_SOURCE,
+ createAgentQueuedTurnLifecycle,
+ AGENT_QUEUED_TURN_SOURCE,
GenerationJobManager,
} = require('@librechat/api');
const methods = require('~/models');
+const getGenerationAdmissionEvidence = (userId, clientRequestId, streamId, conversationId) =>
+ GenerationJobManager.getGenerationAdmissionEvidence(
+ userId,
+ clientRequestId,
+ streamId,
+ conversationId,
+ );
+
const subagentCompletionAdapter = createSubagentCompletionWakeupResolver({
methods,
getGenerationJob: (conversationId) => GenerationJobManager.getJob(conversationId),
@@ -22,23 +32,48 @@ const eventActorAdapter = createAgentEventContinueResolver({
methods,
getGenerationJob: (conversationId) => GenerationJobManager.getJob(conversationId),
});
+let service;
+const queuedTurnLifecycle = createAgentQueuedTurnLifecycle({
+ methods,
+ getGenerationJob: (conversationId) => GenerationJobManager.getJob(conversationId),
+ getGenerationAdmissionEvidence,
+ enqueue: (...args) => service.enqueue(...args),
+ retireDelivery: (...args) => service.retire(...args),
+ getDelivery: (...args) => service.getDelivery(...args),
+});
-const service = createAgentTriggerService({
+service = createAgentTriggerService({
methods,
isPrincipalActive: methods.isAgentTriggerPrincipalActive,
supportsDetachedActionCompletion: () => GenerationJobManager.supportsDetachedAgentEventActions,
+ settleSourceBeforeDeadLetter: queuedTurnLifecycle.settleBeforeDeadLetter,
prepareContinue: createAgentContinuationResolver({
eventActor: eventActorAdapter,
internalSources: new Map([
[SUBAGENT_COMPLETION_SOURCE, subagentCompletionAdapter],
[BACKGROUND_TOOL_COMPLETION_SOURCE, backgroundToolCompletionAdapter],
+ [AGENT_QUEUED_TURN_SOURCE, queuedTurnLifecycle.prepareContinue],
]),
}),
});
+const initializeAgentTriggerService = async (options) => {
+ await service.initialize(options);
+ await queuedTurnLifecycle.initialize();
+};
+
+const stopAgentTriggerService = async () => {
+ await queuedTurnLifecycle.stop();
+ await service.stop();
+};
+
+const purgeAgentTriggerDeliveriesForUser = async (userId) => {
+ await service.purgeUser(userId);
+};
+
module.exports = {
- initializeAgentTriggerService: service.initialize,
- stopAgentTriggerService: service.stop,
+ initializeAgentTriggerService,
+ stopAgentTriggerService,
dispatchAgentTrigger: service.dispatch,
enqueueAgentTrigger: service.enqueue,
getAgentTriggerDelivery: service.getDelivery,
@@ -50,5 +85,9 @@ module.exports = {
drainAgentTriggerDeliveriesForUser: service.drainUser,
prepareAgentTriggerUserPurge: service.prepareUserPurge,
cancelAgentTriggerUserPurge: service.cancelUserPurge,
- purgeAgentTriggerDeliveriesForUser: service.purgeUser,
+ purgeAgentTriggerDeliveriesForUser,
+ scheduleAgentQueuedTurn: queuedTurnLifecycle.schedule,
+ cancelAgentQueuedTurn: queuedTurnLifecycle.cancel,
+ settleAgentQueuedTurnExecutionAdmission: queuedTurnLifecycle.recordExecutionAdmission,
+ verifyAgentQueuedTurnExecutionAdmission: queuedTurnLifecycle.verifyExecutionAdmission,
};
diff --git a/api/server/services/Agents/triggers.spec.js b/api/server/services/Agents/triggers.spec.js
index 48491d402b..230ae910cf 100644
--- a/api/server/services/Agents/triggers.spec.js
+++ b/api/server/services/Agents/triggers.spec.js
@@ -2,6 +2,17 @@ const mockCreateAgentTriggerService = jest.fn();
const mockGenerationJobManager = {
supportsDetachedAgentEventActions: true,
getJob: jest.fn(),
+ getGenerationAdmissionEvidence: jest.fn(),
+};
+const mockQueuedTurnLifecycle = {
+ prepareContinue: jest.fn(),
+ settleBeforeDeadLetter: jest.fn(),
+ recordExecutionAdmission: jest.fn(),
+ verifyExecutionAdmission: jest.fn(),
+ initialize: jest.fn(),
+ stop: jest.fn(),
+ schedule: jest.fn(),
+ cancel: jest.fn(),
};
jest.mock('@librechat/api', () => ({
@@ -10,8 +21,10 @@ jest.mock('@librechat/api', () => ({
createAgentEventContinueResolver: jest.fn(() => jest.fn()),
createBackgroundToolCompletionWakeupResolver: jest.fn(() => jest.fn()),
createSubagentCompletionWakeupResolver: jest.fn(() => jest.fn()),
+ createAgentQueuedTurnLifecycle: jest.fn(() => mockQueuedTurnLifecycle),
BACKGROUND_TOOL_COMPLETION_SOURCE: 'background-tool-completion',
SUBAGENT_COMPLETION_SOURCE: 'subagent-completion',
+ AGENT_QUEUED_TURN_SOURCE: 'agent-queued-turn',
GenerationJobManager: mockGenerationJobManager,
}));
diff --git a/client/src/components/Chat/Input/PendingSteerChips.tsx b/client/src/components/Chat/Input/PendingSteerChips.tsx
index a2d423886b..eef0566152 100644
--- a/client/src/components/Chat/Input/PendingSteerChips.tsx
+++ b/client/src/components/Chat/Input/PendingSteerChips.tsx
@@ -2,7 +2,18 @@ import { memo, useMemo, useRef, useState, useCallback } from 'react';
import { useAtomValue } from 'jotai';
import { useRecoilValue } from 'recoil';
import { useToastContext } from '@librechat/client';
-import { X, Zap, Send, Clock, Pencil, Trash2, Paperclip, RotateCcw, TextQuote } from 'lucide-react';
+import {
+ X,
+ Zap,
+ Send,
+ Clock,
+ Pencil,
+ Trash2,
+ Paperclip,
+ RotateCcw,
+ TextQuote,
+ TriangleAlert,
+} from 'lucide-react';
import type { TMessage } from 'librechat-data-provider';
import type { SteeringControls, QueuedMessageContext } from '~/hooks/Chat/useSteering';
import type { PendingSteer, QueuedMessage } from '~/store/families';
@@ -91,6 +102,14 @@ function QueuedRow({
const fileCount = message.files?.length ?? 0;
const quoteCount = message.quotes?.length ?? 0;
const isRecovered = message.recoverySteerId != null;
+ const isRejected = message.server?.status === 'rejected';
+ const isUnconfirmed =
+ message.server?.status === 'uncertain' && message.server.reconciliationExpired === true;
+ const requiresDiscard = isRecovered || message.server?.id != null;
+ const serverActionable =
+ message.server == null ||
+ message.server.status === 'rejected' ||
+ (message.server.id != null && message.server.status === 'queued');
const actionPendingRef = useRef(false);
const [actionPending, setActionPending] = useState(false);
/** A recovered item has a replayable parked source. Edit/remove must first
@@ -128,7 +147,8 @@ function QueuedRow({
// generation. Re-steering it would leave or duplicate the parked source;
// Edit/remove are safe because `afterDiscard` tombstones that source first.
const canSteerNow = steering.duringRunActive && steering.canSteer && !isRecovered;
- const showPrimary = canSteerNow || (!steering.duringRunActive && steering.canSendQueuedNow);
+ const showPrimary =
+ serverActionable && (canSteerNow || (!steering.duringRunActive && steering.canSendQueuedNow));
/** `canSteer` is defined as false while paused on approval, but the
* escalation control must stay visible-and-disabled there — hiding it
* during the pause is exactly the discoverability gap this button fixes. */
@@ -140,10 +160,13 @@ function QueuedRow({
key: 'edit',
label: localize('com_ui_edit_message'),
icon: <Pencil className="h-4 w-4" aria-hidden="true" />,
- disabled: actionPending,
+ disabled: actionPending || !serverActionable,
onClick: () => {
- const context = { quotes: message.quotes, manualSkills: message.manualSkills };
- if (!isRecovered) {
+ const context = {
+ quotes: message.quotes,
+ manualSkills: message.manualSkills,
+ };
+ if (!requiresDiscard) {
steering.removeQueued(message.id);
onEditToComposer(message.text, message.files, {
quotes: message.quotes,
@@ -159,7 +182,10 @@ function QueuedRow({
conversationId,
);
if (!restored) {
- showToast({ message: localize('com_ui_steer_edit_queued'), status: 'info' });
+ showToast({
+ message: localize('com_ui_steer_edit_queued'),
+ status: 'info',
+ });
return false;
}
steering.removeQueued(message.id);
@@ -172,7 +198,11 @@ function QueuedRow({
return (
<div role="listitem" className={ROW_CLASS} data-testid="queued-message-row">
- <Clock className="h-4 w-4 shrink-0 text-cyan-500" aria-hidden="true" />
+ {isRejected || isUnconfirmed ? (
+ <TriangleAlert className="h-4 w-4 shrink-0 text-text-warning" aria-hidden="true" />
+ ) : (
+ <Clock className="h-4 w-4 shrink-0 text-cyan-500" aria-hidden="true" />
+ )}
<span className="min-w-0 flex-1 truncate" title={message.text}>
{message.text}
</span>
@@ -182,13 +212,22 @@ function QueuedRow({
/>
<AttachmentCount
count={fileCount}
- label={localize('com_ui_queued_attachment_count', { 0: String(fileCount) })}
+ label={localize('com_ui_queued_attachment_count', {
+ 0: String(fileCount),
+ })}
/>
+ {(isRejected || isUnconfirmed) && (
+ <span className="shrink-0 text-xs text-text-warning">
+ {localize(
+ isUnconfirmed ? 'com_ui_steer_delivery_unconfirmed' : 'com_ui_queued_turn_failed',
+ )}
+ </span>
+ )}
{showPrimary && (
<button
type="button"
className={PRIMARY_BTN_CLASS}
- disabled={actionPending}
+ disabled={actionPending || !serverActionable}
onClick={() => steering.sendQueuedNow(message)}
>
{canSteerNow ? (
@@ -208,15 +247,23 @@ function QueuedRow({
<EscalateNowButton
surface="queued"
messageText={message.text}
- disabled={steering.pausedOnApproval || interruptPending || actionPending}
+ disabled={
+ steering.pausedOnApproval || interruptPending || actionPending || !serverActionable
+ }
onClick={() => steering.sendQueuedNow(message, { preempt: true })}
/>
)}
<button
type="button"
- aria-label={localize('com_ui_remove_queued')}
- disabled={actionPending}
+ aria-label={localize(
+ isUnconfirmed ? 'com_ui_dismiss_unconfirmed_delivery' : 'com_ui_remove_queued',
+ )}
+ disabled={actionPending || (!serverActionable && !isUnconfirmed)}
onClick={() => {
+ if (isUnconfirmed) {
+ steering.removeQueued(message.id);
+ return;
+ }
const remove = () => {
/* Same safety net as the in-flight cancel: once removal is safely
* settled, return the words to the composer when it is free (the
@@ -230,7 +277,7 @@ function QueuedRow({
steering.removeQueued(message.id);
return true;
};
- if (!isRecovered) {
+ if (!requiresDiscard) {
remove();
return;
}
@@ -310,7 +357,9 @@ function FailedSteerRow({
</span>
<QuoteCount
count={steer.quotes?.length ?? 0}
- label={localize('com_ui_queued_quote_count', { 0: String(steer.quotes?.length ?? 0) })}
+ label={localize('com_ui_queued_quote_count', {
+ 0: String(steer.quotes?.length ?? 0),
+ })}
/>
<span className="shrink-0 text-xs text-red-500">
{localize(
diff --git a/client/src/components/Chat/Input/__tests__/PendingSteerChips.test.tsx b/client/src/components/Chat/Input/__tests__/PendingSteerChips.test.tsx
index 7765152de0..09b972c012 100644
--- a/client/src/components/Chat/Input/__tests__/PendingSteerChips.test.tsx
+++ b/client/src/components/Chat/Input/__tests__/PendingSteerChips.test.tsx
@@ -163,6 +163,48 @@ describe('PendingSteerChips — queued primary availability', () => {
});
});
+describe('PendingSteerChips — terminal server state', () => {
+ beforeEach(() => {
+ jest.clearAllMocks();
+ });
+
+ it('shows a durable rejection as failed work', () => {
+ renderChips([
+ {
+ id: 'rejected-turn',
+ text: 'could not be admitted',
+ createdAt: 1,
+ server: { status: 'rejected', errorCode: 'ADMISSION_FAILED' },
+ },
+ ]);
+
+ expect(screen.getByText('com_ui_queued_turn_failed')).toBeInTheDocument();
+ });
+
+ it('dismisses an expired ambiguous receipt without restoring or resending it', () => {
+ renderChips([
+ {
+ id: 'uncertain-turn',
+ text: 'delivery outcome unknown',
+ createdAt: 1,
+ server: {
+ status: 'uncertain',
+ uncertainSince: 1,
+ reconciliationExpired: true,
+ },
+ },
+ ]);
+
+ expect(screen.getByText('com_ui_steer_delivery_unconfirmed')).toBeInTheDocument();
+ fireEvent.click(screen.getByLabelText('com_ui_dismiss_unconfirmed_delivery'));
+
+ expect(mockRemoveQueued).toHaveBeenCalledWith('uncertain-turn');
+ expect(mockDiscardQueued).not.toHaveBeenCalled();
+ expect(mockRestoreToComposer).not.toHaveBeenCalled();
+ expect(mockSendQueuedNow).not.toHaveBeenCalled();
+ });
+});
+
describe('PendingSteerChips — queued trash', () => {
beforeEach(() => {
jest.clearAllMocks();
@@ -240,6 +282,32 @@ describe('PendingSteerChips — queued trash', () => {
expect(mockRemoveQueued).toHaveBeenCalledWith('q-recovered');
});
+ it('discards a dead server row before restoring and removing it', async () => {
+ const dead = {
+ id: 'q-dead',
+ text: 'recover dead work',
+ createdAt: 1,
+ clientRequestId: 'client-dead',
+ server: {
+ id: 'server-dead',
+ status: 'rejected' as const,
+ errorCode: 'ADMISSION_FAILED',
+ },
+ };
+ renderChips([dead]);
+
+ fireEvent.click(screen.getByLabelText('com_ui_remove_queued'));
+
+ await waitFor(() => expect(mockDiscardQueued).toHaveBeenCalledWith(dead));
+ expect(mockRestoreToComposer).toHaveBeenCalledWith(
+ 'recover dead work',
+ undefined,
+ { quotes: undefined, manualSkills: undefined },
+ CONVO_ID,
+ );
+ expect(mockRemoveQueued).toHaveBeenCalledWith('q-dead');
+ });
+
it('offers Edit for a recovered row and leaves it untouched when discard is refused', async () => {
const user = userEvent.setup();
mockDiscardQueued.mockResolvedValueOnce(false);
diff --git a/client/src/data-provider/SSE/__tests__/queuedTurns.test.ts b/client/src/data-provider/SSE/__tests__/queuedTurns.test.ts
new file mode 100644
index 0000000000..1ea5de10fe
--- /dev/null
+++ b/client/src/data-provider/SSE/__tests__/queuedTurns.test.ts
@@ -0,0 +1,119 @@
+const mockListAgentQueuedTurns = jest.fn();
+const mockEnqueueAgentQueuedTurn = jest.fn();
+const mockCancelAgentQueuedTurn = jest.fn();
+
+jest.mock('librechat-data-provider', () => ({
+ QueryKeys: { agentQueuedTurns: 'agentQueuedTurns' },
+ MutationKeys: {
+ enqueueAgentQueuedTurn: 'enqueueAgentQueuedTurn',
+ cancelAgentQueuedTurn: 'cancelAgentQueuedTurn',
+ },
+ dataService: {
+ listAgentQueuedTurns: (...args: unknown[]) => mockListAgentQueuedTurns(...args),
+ enqueueAgentQueuedTurn: (...args: unknown[]) => mockEnqueueAgentQueuedTurn(...args),
+ cancelAgentQueuedTurn: (...args: unknown[]) => mockCancelAgentQueuedTurn(...args),
+ },
+}));
+
+import {
+ cancelAgentQueuedTurn,
+ enqueueAgentQueuedTurn,
+ fetchAgentQueuedTurns,
+ isDefiniteQueuedTurnRejection,
+ isDefiniteQueuedTurnsUnsupported,
+} from '../queuedTurns';
+
+describe('Agent queued-turn data adapter', () => {
+ beforeEach(() => {
+ jest.clearAllMocks();
+ });
+
+ it('posts the stable request identity and exact visible parent', async () => {
+ const input = {
+ conversationId: 'conversation/one',
+ clientRequestId: 'client-1',
+ parentMessageId: 'assistant-1',
+ text: 'next turn',
+ };
+ mockEnqueueAgentQueuedTurn.mockResolvedValueOnce({
+ receipt: { queuedTurnId: 'queued-1' },
+ });
+
+ await expect(enqueueAgentQueuedTurn(input)).resolves.toEqual({
+ queuedTurnId: 'queued-1',
+ });
+
+ expect(mockEnqueueAgentQueuedTurn).toHaveBeenCalledWith(input);
+ });
+
+ it('unwraps the shared list and cancel responses', async () => {
+ mockListAgentQueuedTurns.mockResolvedValueOnce({ queuedTurns: [] });
+ mockCancelAgentQueuedTurn.mockResolvedValueOnce({
+ receipt: { queuedTurnId: 'queued/two', status: 'cancelled' },
+ });
+
+ await expect(
+ fetchAgentQueuedTurns('conversation/one', ['request/one', 'request,two']),
+ ).resolves.toEqual([]);
+ await expect(
+ cancelAgentQueuedTurn({
+ conversationId: 'conversation/one',
+ queuedTurnId: 'queued/two',
+ }),
+ ).resolves.toEqual({ queuedTurnId: 'queued/two', status: 'cancelled' });
+
+ expect(mockListAgentQueuedTurns).toHaveBeenCalledWith('conversation/one', [
+ 'request/one',
+ 'request,two',
+ ]);
+ expect(mockCancelAgentQueuedTurn).toHaveBeenCalledWith('queued/two');
+ });
+
+ it('distinguishes an old route from an application-level 404', () => {
+ expect(isDefiniteQueuedTurnsUnsupported({ response: { status: 404 } })).toBe(true);
+ expect(
+ isDefiniteQueuedTurnsUnsupported({
+ response: { status: 404, data: { code: 'CONVERSATION_NOT_FOUND' } },
+ }),
+ ).toBe(false);
+ });
+
+ it('keeps the explicit priority fallback on the legacy local path', () => {
+ expect(
+ isDefiniteQueuedTurnsUnsupported({
+ response: { status: 501, data: { code: 'QUEUED_TURN_PRIORITY_UNSUPPORTED' } },
+ }),
+ ).toBe(true);
+ });
+
+ it.each([400, 409, 500, undefined])(
+ 'keeps status %s server-owned because it is not an unsupported route',
+ (status) => {
+ const error = status == null ? new Error('network') : { response: { status } };
+ expect(isDefiniteQueuedTurnsUnsupported(error)).toBe(false);
+ },
+ );
+
+ it.each([400, 403, 409, 413, 429])(
+ 'classifies %s as a definite non-commit rejection',
+ (status) => {
+ expect(isDefiniteQueuedTurnRejection({ response: { status } })).toBe(true);
+ },
+ );
+
+ it('classifies a coded conversation 404 as a definite rejection', () => {
+ expect(
+ isDefiniteQueuedTurnRejection({
+ response: { status: 404, data: { code: 'CONVERSATION_NOT_FOUND' } },
+ }),
+ ).toBe(true);
+ });
+
+ it.each([408, 425, 500, 503, undefined])(
+ 'keeps %s outcome-ambiguous for request-id reconciliation',
+ (status) => {
+ const error = status == null ? new Error('network') : { response: { status } };
+ expect(isDefiniteQueuedTurnRejection(error)).toBe(false);
+ },
+ );
+});
diff --git a/client/src/data-provider/SSE/index.ts b/client/src/data-provider/SSE/index.ts
index 5074ce40b0..d27c0d4a0d 100644
--- a/client/src/data-provider/SSE/index.ts
+++ b/client/src/data-provider/SSE/index.ts
@@ -1,3 +1,4 @@
export * from './queries';
export * from './mutations';
export * from './protocol';
+export * from './queuedTurns';
diff --git a/client/src/data-provider/SSE/queuedTurns.ts b/client/src/data-provider/SSE/queuedTurns.ts
new file mode 100644
index 0000000000..fb781c98f8
--- /dev/null
+++ b/client/src/data-provider/SSE/queuedTurns.ts
@@ -0,0 +1,126 @@
+import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
+import { dataService, MutationKeys, QueryKeys } from 'librechat-data-provider';
+import type {
+ TAgentQueuedTurnReceipt,
+ TEnqueueAgentQueuedTurnRequest,
+} from 'librechat-data-provider';
+
+export type AgentQueuedTurnReceipt = TAgentQueuedTurnReceipt;
+export type EnqueueAgentQueuedTurnRequest = TEnqueueAgentQueuedTurnRequest;
+
+export const agentQueuedTurnsQueryKey = (
+ conversationId: string,
+ clientRequestIds: readonly string[] = [],
+) => [QueryKeys.agentQueuedTurns, conversationId, ...clientRequestIds] as const;
+
+function queuedTurnErrorResponse(error: unknown): { status?: number; code?: string } {
+ const response = (
+ error as { response?: { status?: unknown; data?: { code?: unknown } } } | undefined
+ )?.response;
+ return {
+ ...(typeof response?.status === 'number' && { status: response.status }),
+ ...(typeof response?.data?.code === 'string' && { code: response.data.code }),
+ };
+}
+
+export function isDefiniteQueuedTurnsUnsupported(error: unknown): boolean {
+ const { status, code } = queuedTurnErrorResponse(error);
+ return (
+ (status === 404 && code == null) ||
+ (status === 501 &&
+ (code == null ||
+ code === 'QUEUED_TURNS_UNSUPPORTED' ||
+ code === 'QUEUED_TURN_PRIORITY_UNSUPPORTED'))
+ );
+}
+
+/** A bounded origin 4xx proves the queued row was not committed. Timeouts and
+ * early-data responses can be generated while the origin request continues,
+ * so those remain outcome-ambiguous and must reconcile by request identity. */
+export function isDefiniteQueuedTurnRejection(error: unknown): boolean {
+ const { status } = queuedTurnErrorResponse(error);
+ return (
+ !isDefiniteQueuedTurnsUnsupported(error) &&
+ typeof status === 'number' &&
+ status >= 400 &&
+ status < 500 &&
+ status !== 408 &&
+ status !== 425
+ );
+}
+
+export async function fetchAgentQueuedTurns(
+ conversationId: string,
+ clientRequestIds?: string[],
+): Promise<AgentQueuedTurnReceipt[]> {
+ const response = await dataService.listAgentQueuedTurns(conversationId, clientRequestIds);
+ return response.queuedTurns;
+}
+
+export async function enqueueAgentQueuedTurn(
+ input: EnqueueAgentQueuedTurnRequest,
+): Promise<AgentQueuedTurnReceipt> {
+ const response = await dataService.enqueueAgentQueuedTurn(input);
+ return response.receipt;
+}
+
+export async function cancelAgentQueuedTurn(input: {
+ conversationId: string;
+ queuedTurnId: string;
+}): Promise<AgentQueuedTurnReceipt> {
+ const response = await dataService.cancelAgentQueuedTurn(input.queuedTurnId);
+ return response.receipt;
+}
+
+export function useAgentQueuedTurns(
+ conversationId: string,
+ enabled: boolean,
+ clientRequestIds: string[] = [],
+ reconcileUntil?: number,
+) {
+ const knownIds = [...new Set(clientRequestIds)].sort();
+ return useQuery({
+ queryKey: agentQueuedTurnsQueryKey(conversationId, knownIds),
+ queryFn: () => fetchAgentQueuedTurns(conversationId, knownIds),
+ enabled: enabled && conversationId.length > 0,
+ staleTime: 1_000,
+ refetchOnMount: true,
+ refetchOnWindowFocus: true,
+ refetchInterval: (receipts) => {
+ return (reconcileUntil != null && Date.now() < reconcileUntil) ||
+ (Array.isArray(receipts) &&
+ receipts.some((item) => item.status === 'queued' || item.status === 'claimed'))
+ ? 2_000
+ : false;
+ },
+ retry: false,
+ });
+}
+
+export function useEnqueueAgentQueuedTurnMutation() {
+ const queryClient = useQueryClient();
+ return useMutation({
+ mutationKey: [MutationKeys.enqueueAgentQueuedTurn],
+ mutationFn: enqueueAgentQueuedTurn,
+ /** A retry can encounter admission middleware while the first request is
+ * still committing. Reconcile ambiguous outcomes through the read-only
+ * known-id projection instead of issuing a second mutating POST. */
+ retry: false,
+ onSettled: (_receipt, _error, input) =>
+ queryClient.invalidateQueries({
+ queryKey: [QueryKeys.agentQueuedTurns, input.conversationId],
+ }),
+ });
+}
+
+export function useCancelAgentQueuedTurnMutation() {
+ const queryClient = useQueryClient();
+ return useMutation({
+ mutationKey: [MutationKeys.cancelAgentQueuedTurn],
+ mutationFn: cancelAgentQueuedTurn,
+ onSuccess: (_receipt, input) =>
+ queryClient.invalidateQueries({
+ queryKey: [QueryKeys.agentQueuedTurns, input.conversationId],
+ }),
+ });
+}
diff --git a/client/src/hooks/Chat/__tests__/useQueueDrain.spec.tsx b/client/src/hooks/Chat/__tests__/useQueueDrain.spec.tsx
index 7a3f68d099..762a5f692e 100644
--- a/client/src/hooks/Chat/__tests__/useQueueDrain.spec.tsx
+++ b/client/src/hooks/Chat/__tests__/useQueueDrain.spec.tsx
@@ -24,10 +24,10 @@ function setup(
const setters: {
setRunEnd?: (value: RunEnd | null) => void;
setIsSubmitting?: (value: boolean) => void;
- setQueue?: (value: { id: string; text: string; createdAt: number }[]) => void;
- setNewConvoQueue?: (value: { id: string; text: string; createdAt: number }[]) => void;
+ setQueue?: (value: QueuedMessage[]) => void;
+ setNewConvoQueue?: (value: QueuedMessage[]) => void;
setInterruptFlag?: (value: DrainAfterAbort | false) => void;
- queueRef?: { current: { id: string; text: string; createdAt: number }[] };
+ queueRef?: { current: QueuedMessage[] };
} = {};
function Harness() {
@@ -66,7 +66,11 @@ const emptyOverrides = expect.objectContaining({
overrideQueuedMessageOrigin: expect.any(Object),
});
-const queuedMessage = (id: string, text: string) => ({ id, text, createdAt: Date.now() });
+const queuedMessage = (id: string, text: string) => ({
+ id,
+ text,
+ createdAt: Date.now(),
+});
const runEnd = (overrides: Partial<RunEnd> = {}): RunEnd => ({
conversationId: CONVO_ID,
@@ -97,6 +101,55 @@ describe('useQueueDrain', () => {
expect(ask).toHaveBeenCalledWith({ text: 'first follow-up' }, emptyOverrides);
});
+ it('does not locally drain or renew server-owned Agent rows', async () => {
+ const { ask, setters } = setup(({ set }) => {
+ set(store.queuedMessagesByConvoId(CONVO_ID), [
+ {
+ ...queuedMessage('q-server', 'the server starts this turn'),
+ files: [{ file_id: 'server-held-file' }],
+ clientRequestId: 'client-request-1',
+ server: { id: 'server-queue-1', status: 'queued', revision: 1 },
+ },
+ ]);
+ });
+
+ act(() => {
+ setters.setRunEnd!(runEnd());
+ });
+
+ await new Promise((resolve) => setTimeout(resolve, 25));
+ expect(ask).not.toHaveBeenCalled();
+ expect(mockMarkFilesUsage).not.toHaveBeenCalledWith({
+ file_ids: ['server-held-file'],
+ });
+ });
+
+ it('retains the terminal boundary until server authority releases a local successor', async () => {
+ const localSuccessor = queuedMessage('q-local', 'legacy successor');
+ const queue: QueuedMessage[] = [
+ {
+ ...queuedMessage('q-server', 'server-owned predecessor'),
+ server: { id: 'server-queue-1', status: 'queued', revision: 1 },
+ },
+ localSuccessor,
+ ];
+ const { ask, setters } = setup(({ set }) => {
+ set(store.queuedMessagesByConvoId(CONVO_ID), queue);
+ });
+
+ act(() => {
+ setters.setRunEnd!(runEnd());
+ });
+ await new Promise((resolve) => setTimeout(resolve, 25));
+ expect(ask).not.toHaveBeenCalled();
+
+ act(() => {
+ setters.setQueue!([localSuccessor]);
+ });
+ await waitFor(() => expect(ask).toHaveBeenCalledTimes(1));
+ expect(ask).toHaveBeenCalledWith({ text: 'legacy successor' }, emptyOverrides);
+ });
+
it('parks a mismatched signal instead of draining into the wrong conversation', async () => {
const { ask, setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [queuedMessage('q1', 'stay put')]);
@@ -152,12 +205,18 @@ describe('useQueueDrain', () => {
it('renews the TTL hold on attachments that stay queued', async () => {
const { setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
- { ...queuedMessage('q1', 'first'), files: [{ file_id: 'sent', type: 'image/png' }] },
+ {
+ ...queuedMessage('q1', 'first'),
+ files: [{ file_id: 'sent', type: 'image/png' }],
+ },
{
...queuedMessage('q2', 'second'),
files: [{ file_id: 'still-queued', type: 'image/png' }],
},
- { ...queuedMessage('q3', 'third'), files: [{ file_id: 'also-queued', type: 'image/png' }] },
+ {
+ ...queuedMessage('q3', 'third'),
+ files: [{ file_id: 'also-queued', type: 'image/png' }],
+ },
]);
});
@@ -176,7 +235,10 @@ describe('useQueueDrain', () => {
it('does not renew when nothing with attachments stays queued', async () => {
const { setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
- { ...queuedMessage('q1', 'only one'), files: [{ file_id: 'sent', type: 'image/png' }] },
+ {
+ ...queuedMessage('q1', 'only one'),
+ files: [{ file_id: 'sent', type: 'image/png' }],
+ },
]);
});
@@ -193,7 +255,10 @@ describe('useQueueDrain', () => {
* enqueue-time hold. */
it('renews every queued attachment across multiple capped batches', async () => {
const manyFiles = (prefix: string, n: number) =>
- Array.from({ length: n }, (_, i) => ({ file_id: `${prefix}-${i}`, type: 'image/png' }));
+ Array.from({ length: n }, (_, i) => ({
+ file_id: `${prefix}-${i}`,
+ type: 'image/png',
+ }));
const { setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
{ ...queuedMessage('q1', 'first'), files: manyFiles('sent', 2) },
@@ -222,7 +287,10 @@ describe('useQueueDrain', () => {
it('renews a restored item when the send is refused', async () => {
const { ask, setters } = setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
- { ...queuedMessage('q1', 'refused'), files: [{ file_id: 'restored', type: 'image/png' }] },
+ {
+ ...queuedMessage('q1', 'refused'),
+ files: [{ file_id: 'restored', type: 'image/png' }],
+ },
]);
});
ask.mockReturnValue(false);
@@ -243,7 +311,10 @@ describe('useQueueDrain', () => {
try {
setup(({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [
- { ...queuedMessage('q1', 'waiting'), files: [{ file_id: 'held', type: 'image/png' }] },
+ {
+ ...queuedMessage('q1', 'waiting'),
+ files: [{ file_id: 'held', type: 'image/png' }],
+ },
]);
});
@@ -287,10 +358,16 @@ describe('useQueueDrain', () => {
it('renews the pre-migration NEW_CONVO queue alongside the active one', async () => {
setup(({ set }) => {
set(store.queuedMessagesByConvoId(Constants.NEW_CONVO), [
- { ...queuedMessage('n1', 'queued pre-migration'), files: [{ file_id: 'pending-migrate' }] },
+ {
+ ...queuedMessage('n1', 'queued pre-migration'),
+ files: [{ file_id: 'pending-migrate' }],
+ },
]);
set(store.queuedMessagesByConvoId(CONVO_ID), [
- { ...queuedMessage('q1', 'queued after'), files: [{ file_id: 'already-migrated' }] },
+ {
+ ...queuedMessage('q1', 'queued after'),
+ files: [{ file_id: 'already-migrated' }],
+ },
]);
});
@@ -303,7 +380,10 @@ describe('useQueueDrain', () => {
it('does not double-count the queue before migration', async () => {
setup(({ set }) => {
set(store.queuedMessagesByConvoId(Constants.NEW_CONVO), [
- { ...queuedMessage('n1', 'new convo'), files: [{ file_id: 'only-once' }] },
+ {
+ ...queuedMessage('n1', 'new convo'),
+ files: [{ file_id: 'only-once' }],
+ },
]);
}, Constants.NEW_CONVO as string);
@@ -558,7 +638,12 @@ describe('useQueueDrain', () => {
{ id: 'ordinary-before-url', text: 'ordinary follow-up', createdAt: 1 },
]);
set(store.queuedMessagesByConvoId(CONVO_ID), [
- { id: 'interrupt-after-url', text: 'interrupt next', createdAt: 2, priority: true },
+ {
+ id: 'interrupt-after-url',
+ text: 'interrupt next',
+ createdAt: 2,
+ priority: true,
+ },
]);
set(store.drainAfterAbortByIndex(INDEX), {
conversationId: CONVO_ID,
diff --git a/client/src/hooks/Chat/__tests__/useSteering.spec.tsx b/client/src/hooks/Chat/__tests__/useSteering.spec.tsx
index 8892b5f465..0fa405366d 100644
--- a/client/src/hooks/Chat/__tests__/useSteering.spec.tsx
+++ b/client/src/hooks/Chat/__tests__/useSteering.spec.tsx
@@ -1,5 +1,5 @@
import React from 'react';
-import { act, renderHook } from '@testing-library/react';
+import { act, renderHook, waitFor } from '@testing-library/react';
import { RecoilRoot, useRecoilValue, useSetRecoilState, type MutableSnapshot } from 'recoil';
import { Constants, ContentTypes, EModelEndpoint, LocalStorageKeys } from 'librechat-data-provider';
import type { TConversation, TMessage } from 'librechat-data-provider';
@@ -12,16 +12,43 @@ const UUID_V4_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}
const mockMutate = jest.fn();
const mockCancelSteer = jest.fn();
+const mockEnqueueQueuedTurn = jest.fn();
+const mockCancelQueuedTurn = jest.fn();
const mockShowToast = jest.fn();
const mockMarkUsage = jest.fn();
let mockMessages: TMessage[] | undefined;
+let mockLatestMessage: TMessage | null | undefined;
+let mockServerQueuedTurns: unknown[] | undefined;
+const mockUseAgentQueuedTurns = jest.fn((..._args: unknown[]) => ({ data: mockServerQueuedTurns }));
jest.mock('~/data-provider', () => ({
useCancelSteerMutation: () => ({ mutateAsync: mockCancelSteer }),
useSteerMessageMutation: () => ({ mutate: mockMutate }),
- useGetMessagesByConvoId: (_id: string, config?: { select?: (messages: unknown) => unknown }) => ({
- data: config?.select ? config.select(mockMessages) : mockMessages,
+ useAgentQueuedTurns: (...args: unknown[]) => mockUseAgentQueuedTurns(...args),
+ useEnqueueAgentQueuedTurnMutation: () => ({ mutate: mockEnqueueQueuedTurn }),
+ useCancelAgentQueuedTurnMutation: () => ({
+ mutateAsync: mockCancelQueuedTurn,
}),
+ isDefiniteQueuedTurnRejection: (error: unknown) => {
+ const response = (
+ error as { response?: { status?: unknown; data?: { code?: unknown } } } | undefined
+ )?.response;
+ const unsupported =
+ (response?.status === 404 && response.data?.code == null) ||
+ (response?.status === 501 && response.data?.code === 'QUEUED_TURN_PRIORITY_UNSUPPORTED');
+ return (
+ !unsupported &&
+ typeof response?.status === 'number' &&
+ response.status >= 400 &&
+ response.status < 500 &&
+ response.status !== 408 &&
+ response.status !== 425
+ );
+ },
+ isDefiniteQueuedTurnsUnsupported: (error: unknown) => {
+ const status = (error as { response?: { status?: unknown } } | undefined)?.response?.status;
+ return status === 404 || status === 501;
+ },
useMarkFilesUsageMutation: () => ({ mutate: mockMarkUsage }),
supportsGenerationProtocolV2: (value: unknown) =>
value != null &&
@@ -29,6 +56,13 @@ jest.mock('~/data-provider', () => ({
(value as { generationProtocolVersion?: unknown }).generationProtocolVersion === 2,
}));
+jest.mock('~/hooks/Messages', () => ({
+ useLatestMessage: () =>
+ mockLatestMessage === undefined
+ ? (mockMessages?.[mockMessages.length - 1] ?? null)
+ : mockLatestMessage,
+}));
+
jest.mock('@librechat/client', () => ({
useToastContext: () => ({ showToast: mockShowToast }),
}));
@@ -96,6 +130,8 @@ describe('useSteering', () => {
beforeEach(() => {
jest.clearAllMocks();
mockMessages = undefined;
+ mockLatestMessage = undefined;
+ mockServerQueuedTurns = undefined;
});
describe('effectiveAction', () => {
@@ -116,7 +152,9 @@ describe('useSteering', () => {
});
it('degrades to queue without a real conversation id', () => {
- const { result } = setup({ conversationId: Constants.NEW_CONVO as string });
+ const { result } = setup({
+ conversationId: Constants.NEW_CONVO as string,
+ });
expect(result.current.effectiveAction).toBe('queue');
});
@@ -128,7 +166,12 @@ describe('useSteering', () => {
content: [
{
type: ContentTypes.TOOL_CALL,
- tool_call: { id: 'call-1', name: 'shell', approval: { actionId: 'a1' }, output: '' },
+ tool_call: {
+ id: 'call-1',
+ name: 'shell',
+ approval: { actionId: 'a1' },
+ output: '',
+ },
},
],
} as unknown as TMessage,
@@ -174,7 +217,10 @@ describe('useSteering', () => {
result.current.interruptAndSend('now stoppable');
});
expect(mockMutate).toHaveBeenCalledWith(
- expect.objectContaining({ generationCreatedAt: 42, text: 'now fenced' }),
+ expect.objectContaining({
+ generationCreatedAt: 42,
+ text: 'now fenced',
+ }),
expect.anything(),
);
expect(stopGenerating).toHaveBeenCalledTimes(1);
@@ -182,8 +228,11 @@ describe('useSteering', () => {
it('is inactive but exposes the paused state for ask-user answer mode', () => {
expect(
- setup({ conversation: { endpoint: EModelEndpoint.assistants } as TConversation }).result
- .current.duringRunActive,
+ setup({
+ conversation: {
+ endpoint: EModelEndpoint.assistants,
+ } as TConversation,
+ }).result.current.duringRunActive,
).toBe(false);
expect(setup({ index: 1 }).result.current.duringRunActive).toBe(false);
const answerMode = setup({ answerModeActive: true }).result.current;
@@ -193,6 +242,367 @@ describe('useSteering', () => {
});
});
+ describe('server-owned Agent queued turns', () => {
+ function setupServerQueue(initializer = withActiveGeneration()) {
+ const sendNow = jest.fn();
+ const wrapper = ({ children }: { children: React.ReactNode }) => (
+ <RecoilRoot initializeState={initializer}>{children}</RecoilRoot>
+ );
+ const rendered = renderHook(
+ () => ({
+ steering: useSteering({
+ index: 0,
+ conversationId: CONVO_ID,
+ conversation: agentsConversation,
+ isSubmitting: true,
+ answerModeActive: false,
+ sendNow,
+ stopGenerating: jest.fn(),
+ }),
+ queue: useQueue(CONVO_ID),
+ }),
+ { wrapper },
+ );
+ return { ...rendered, sendNow };
+ }
+
+ beforeEach(() => {
+ mockMessages = [
+ {
+ messageId: 'visible-assistant-tail',
+ isCreatedByUser: false,
+ content: [],
+ } as unknown as TMessage,
+ ];
+ });
+
+ it('keeps startup turns local until the server generation epoch exists', async () => {
+ const { result } = setupServerQueue(
+ withActiveGeneration(({ set }) => {
+ set(store.activeGenerationCreatedAtByConvoId(CONVO_ID), null);
+ }),
+ );
+
+ await act(async () => {
+ expect(result.current.steering.queueFromComposer('wait for the epoch')).toBe(true);
+ await Promise.resolve();
+ });
+
+ expect(mockEnqueueQueuedTurn).not.toHaveBeenCalled();
+ expect(result.current.queue).toEqual([
+ expect.objectContaining({ text: 'wait for the epoch' }),
+ ]);
+ expect(result.current.queue[0].server).toBeUndefined();
+ });
+
+ it('enrolls one optimistic row with a stable request and visible branch identity', async () => {
+ mockEnqueueQueuedTurn.mockImplementation((input, options) => {
+ options.onSuccess({
+ ...input,
+ queuedTurnId: 'server-queued-1',
+ status: 'queued',
+ revision: 1,
+ createdAt: new Date(100).toISOString(),
+ updatedAt: new Date(100).toISOString(),
+ });
+ });
+ const { result } = setupServerQueue();
+
+ await act(async () => {
+ expect(result.current.steering.queueFromComposer('server owns this')).toBe(true);
+ await Promise.resolve();
+ });
+
+ expect(mockEnqueueQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ conversationId: CONVO_ID,
+ parentMessageId: 'visible-assistant-tail',
+ text: 'server owns this',
+ expectedPredecessorCreatedAt: 41,
+ clientRequestId: expect.stringMatching(UUID_V4_PATTERN),
+ }),
+ expect.any(Object),
+ );
+ await waitFor(() =>
+ expect(result.current.queue).toEqual([
+ expect.objectContaining({
+ text: 'server owns this',
+ parentMessageId: 'visible-assistant-tail',
+ clientRequestId: expect.stringMatching(UUID_V4_PATTERN),
+ server: { id: 'server-queued-1', status: 'queued', revision: 1 },
+ }),
+ ]),
+ );
+ });
+
+ it('anchors enqueue to the selected branch tail instead of a later hidden sibling', async () => {
+ mockMessages = [
+ {
+ messageId: 'visible-assistant-tail',
+ isCreatedByUser: false,
+ content: [],
+ } as unknown as TMessage,
+ {
+ messageId: 'hidden-later-sibling',
+ isCreatedByUser: false,
+ content: [],
+ } as unknown as TMessage,
+ ];
+ mockLatestMessage = mockMessages[0];
+ const { result } = setupServerQueue();
+
+ await act(async () => {
+ result.current.steering.queueFromComposer('stay on the visible branch');
+ await Promise.resolve();
+ });
+
+ expect(mockEnqueueQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({ parentMessageId: 'visible-assistant-tail' }),
+ expect.any(Object),
+ );
+ });
+
+ it.each([404, 501])('falls back to the legacy local queue on definite %s', async (status) => {
+ mockEnqueueQueuedTurn.mockImplementation((_input, options) => {
+ options.onError({ response: { status } });
+ });
+ const { result } = setupServerQueue();
+
+ await act(async () => {
+ result.current.steering.queueFromComposer('legacy fallback');
+ await Promise.resolve();
+ });
+
+ await waitFor(() => {
+ expect(result.current.queue).toEqual([
+ expect.objectContaining({ text: 'legacy fallback' }),
+ ]);
+ expect(result.current.queue[0].server).toBeUndefined();
+ expect(result.current.queue[0].clientRequestId).toBeUndefined();
+ });
+ });
+
+ it('never converts an ambiguous enqueue failure into a client-drained row', async () => {
+ mockEnqueueQueuedTurn.mockImplementation((_input, options) => {
+ options.onError(Object.assign(new Error('connection reset'), { code: 'ERR_NETWORK' }));
+ });
+ const { result } = setupServerQueue();
+
+ await act(async () => {
+ result.current.steering.queueFromComposer('outcome unknown');
+ await Promise.resolve();
+ });
+
+ await waitFor(() => {
+ expect(result.current.queue[0]).toMatchObject({
+ text: 'outcome unknown',
+ server: { status: 'uncertain', uncertainSince: expect.any(Number) },
+ });
+ expect(result.current.queue[0].clientRequestId).toEqual(
+ expect.stringMatching(UUID_V4_PATTERN),
+ );
+ });
+ const clientRequestId = result.current.queue[0].clientRequestId;
+ const uncertainSince = result.current.queue[0].server?.uncertainSince;
+ expect(mockUseAgentQueuedTurns).toHaveBeenLastCalledWith(
+ CONVO_ID,
+ true,
+ [clientRequestId],
+ uncertainSince! + 60_000,
+ );
+ });
+
+ it('expires reconciliation as ambiguous instead of making the row resendable', async () => {
+ jest.useFakeTimers({ now: 1_000 });
+ try {
+ mockEnqueueQueuedTurn.mockImplementation((_input, options) => {
+ options.onError(Object.assign(new Error('connection reset'), { code: 'ERR_NETWORK' }));
+ });
+ const { result } = setupServerQueue();
+
+ act(() => {
+ result.current.steering.queueFromComposer('outcome remains unknown');
+ });
+ await act(async () => {
+ jest.runAllTicks();
+ await Promise.resolve();
+ });
+ expect(result.current.queue[0]).toMatchObject({ server: { status: 'uncertain' } });
+ expect(result.current.queue[0].server).not.toHaveProperty('reconciliationExpired');
+
+ act(() => {
+ jest.advanceTimersByTime(60_000);
+ });
+ expect(result.current.queue[0]).toMatchObject({
+ server: { status: 'uncertain', reconciliationExpired: true },
+ });
+ } finally {
+ jest.useRealTimers();
+ }
+ });
+
+ it('keeps a definitely rejected enqueue non-drainable but explicitly actionable', async () => {
+ mockEnqueueQueuedTurn.mockImplementation((_input, options) => {
+ options.onError({ response: { status: 413, data: { code: 'TEXT_TOO_LARGE' } } });
+ });
+ const { result } = setupServerQueue();
+
+ await act(async () => {
+ result.current.steering.queueFromComposer('definitely rejected');
+ await Promise.resolve();
+ });
+
+ await waitFor(() => {
+ expect(result.current.queue[0]).toMatchObject({
+ text: 'definitely rejected',
+ server: { status: 'rejected', errorCode: 'TEXT_TOO_LARGE' },
+ });
+ });
+
+ await act(async () => {
+ await expect(result.current.steering.discardQueued(result.current.queue[0])).resolves.toBe(
+ true,
+ );
+ });
+ expect(mockCancelQueuedTurn).not.toHaveBeenCalled();
+ expect(result.current.queue[0].server).toBeUndefined();
+ });
+
+ it('projects the authoritative server snapshot into Recoil', async () => {
+ mockServerQueuedTurns = [
+ {
+ queuedTurnId: 'server-snapshot-1',
+ clientRequestId: 'client-snapshot-1',
+ conversationId: CONVO_ID,
+ parentMessageId: 'visible-assistant-tail',
+ text: 'restored after reload',
+ status: 'queued',
+ revision: 3,
+ createdAt: new Date(200).toISOString(),
+ updatedAt: new Date(200).toISOString(),
+ },
+ ];
+ const { result } = setupServerQueue();
+
+ await waitFor(() => expect(result.current.queue).toHaveLength(1));
+ expect(result.current.queue[0]).toMatchObject({
+ id: 'client-snapshot-1',
+ text: 'restored after reload',
+ server: { id: 'server-snapshot-1', status: 'queued', revision: 3 },
+ });
+ });
+
+ it('uses server sequence instead of Mongo timestamps for projected queue order', async () => {
+ mockServerQueuedTurns = [
+ {
+ queuedTurnId: 'server-second',
+ clientRequestId: 'client-second',
+ conversationId: CONVO_ID,
+ parentMessageId: 'visible-assistant-tail',
+ text: 'second',
+ status: 'queued',
+ position: 2,
+ revision: 20,
+ createdAt: new Date(100).toISOString(),
+ updatedAt: new Date(100).toISOString(),
+ },
+ {
+ queuedTurnId: 'server-first',
+ clientRequestId: 'client-first',
+ conversationId: CONVO_ID,
+ parentMessageId: 'visible-assistant-tail',
+ text: 'first',
+ status: 'queued',
+ position: 1,
+ revision: 10,
+ createdAt: new Date(300).toISOString(),
+ updatedAt: new Date(300).toISOString(),
+ },
+ ];
+ const { result } = setupServerQueue();
+
+ await waitFor(() => expect(result.current.queue).toHaveLength(2));
+ expect(result.current.queue.map((item) => item.text)).toEqual(['first', 'second']);
+ expect(result.current.queue.map((item) => item.server?.position)).toEqual([1, 2]);
+ });
+
+ it('keeps dead work recoverable and dismisses its durable record before local control', async () => {
+ mockServerQueuedTurns = [
+ {
+ queuedTurnId: 'server-dead-1',
+ clientRequestId: 'client-dead-1',
+ conversationId: CONVO_ID,
+ parentMessageId: 'visible-assistant-tail',
+ text: 'recover this failed turn',
+ status: 'dead',
+ revision: 4,
+ failure: { code: 'ADMISSION_FAILED', message: 'The run could not be admitted' },
+ createdAt: new Date(250).toISOString(),
+ updatedAt: new Date(300).toISOString(),
+ },
+ ];
+ mockCancelQueuedTurn.mockResolvedValueOnce({ status: 'cancelled' });
+ const { result } = setupServerQueue();
+
+ await waitFor(() => expect(result.current.queue).toHaveLength(1));
+ expect(result.current.queue[0]).toMatchObject({
+ text: 'recover this failed turn',
+ server: {
+ id: 'server-dead-1',
+ status: 'rejected',
+ errorCode: 'ADMISSION_FAILED',
+ errorMessage: 'The run could not be admitted',
+ },
+ });
+
+ await act(async () => {
+ await expect(result.current.steering.discardQueued(result.current.queue[0])).resolves.toBe(
+ true,
+ );
+ });
+ expect(mockCancelQueuedTurn).toHaveBeenCalledWith({
+ conversationId: CONVO_ID,
+ queuedTurnId: 'server-dead-1',
+ });
+ expect(result.current.queue[0].server).toBeUndefined();
+ });
+
+ it('cancels the durable copy before downgrading a row to local control', async () => {
+ mockServerQueuedTurns = [
+ {
+ queuedTurnId: 'server-cancel-1',
+ clientRequestId: 'client-cancel-1',
+ conversationId: CONVO_ID,
+ parentMessageId: 'visible-assistant-tail',
+ text: 'edit after cancel',
+ status: 'queued',
+ revision: 1,
+ createdAt: new Date(300).toISOString(),
+ updatedAt: new Date(300).toISOString(),
+ },
+ ];
+ mockCancelQueuedTurn.mockResolvedValueOnce({ status: 'cancelled' });
+ const { result } = setupServerQueue();
+
+ await waitFor(() => expect(result.current.queue).toHaveLength(1));
+ let discarded = false;
+ await act(async () => {
+ discarded = await result.current.steering.discardQueued(result.current.queue[0]);
+ });
+
+ expect(discarded).toBe(true);
+ expect(mockCancelQueuedTurn).toHaveBeenCalledWith({
+ conversationId: CONVO_ID,
+ queuedTurnId: 'server-cancel-1',
+ });
+ expect(result.current.queue[0]).toMatchObject({
+ text: 'edit after cancel',
+ clientRequestId: 'client-cancel-1',
+ });
+ expect(result.current.queue[0].server).toBeUndefined();
+ });
+ });
+
describe('queueReclaimedSteer', () => {
const reclaimed = {
steerId: 's-reclaimed',
@@ -311,7 +721,9 @@ describe('useSteering', () => {
act(() => {
result.current.steering.queueReclaimedSteer(reclaimed);
});
- expect(result.current.parkedRunEnd).toMatchObject({ outcome: 'completed' });
+ expect(result.current.parkedRunEnd).toMatchObject({
+ outcome: 'completed',
+ });
expect(result.current.queue.map((item) => item.id)).toEqual(['s-reclaimed']);
});
@@ -379,7 +791,9 @@ describe('useSteering', () => {
result.current.steering.queueReclaimedSteer(reclaimed);
});
// Untouched: the already-armed signal drains it.
- expect(result.current.parkedRunEnd).toMatchObject({ outcome: 'completed' });
+ expect(result.current.parkedRunEnd).toMatchObject({
+ outcome: 'completed',
+ });
expect(result.current.queue.map((item) => item.id)).toEqual(['s-reclaimed']);
});
@@ -471,7 +885,9 @@ describe('useSteering', () => {
rerender({ convoId: 'convo-elsewhere', isSubmitting: false });
});
act(() => {
- rejectSteer?.({ response: { status: 409, data: { code: 'RUN_PAUSED' } } });
+ rejectSteer?.({
+ response: { status: 409, data: { code: 'RUN_PAUSED' } },
+ });
});
expect(result.current.queueHere).toEqual([
@@ -488,7 +904,10 @@ describe('useSteering', () => {
// The run-end is keyed by conversation, so the new chat's end can never
// be mistaken for this one's. Parking it here would give `drainNext` a
// foreign `end.conversationId` and drain the wrong queue into this chat.
- const { result, rerender } = setupNavigable({ convoId: CONVO_ID, isSubmitting: true });
+ const { result, rerender } = setupNavigable({
+ convoId: CONVO_ID,
+ isSubmitting: true,
+ });
const queueReclaimed = result.current.steering.queueReclaimedSteer;
act(() => {
// The user leaves for a chat whose own run then completes.
@@ -674,14 +1093,20 @@ describe('useSteering', () => {
onError({ response: { data: { code: 'NO_ACTIVE_RUN' } } });
});
const queuedOrigin = {
- item: { id: 'queued-original', text: 'late queued turn', createdAt: 10 },
+ item: {
+ id: 'queued-original',
+ text: 'late queued turn',
+ createdAt: 10,
+ },
beforeIds: ['queued-before'],
afterIds: ['queued-after'],
};
const { result, sendNow } = setup({ isSubmitting: false });
act(() => {
- result.current.submitSteer('late queued turn', undefined, undefined, { queuedOrigin });
+ result.current.submitSteer('late queued turn', undefined, undefined, {
+ queuedOrigin,
+ });
});
expect(sendNow).toHaveBeenCalledWith('late queued turn', [], {
@@ -704,7 +1129,9 @@ describe('useSteering', () => {
it('never normal-sends into a newer live run on RUN_REPLACED', () => {
mockMutate.mockImplementation((_params, { onError }) => {
onError({
- response: { data: { code: 'RUN_REPLACED', generationProtocolVersion: 2 } },
+ response: {
+ data: { code: 'RUN_REPLACED', generationProtocolVersion: 2 },
+ },
});
});
const { result, sendNow } = setup({ isSubmitting: false });
@@ -771,7 +1198,11 @@ describe('useSteering', () => {
content: [
{
type: ContentTypes.TOOL_CALL,
- [ContentTypes.TOOL_CALL]: { id: 'call_1', name: 't', approval: 'pending' },
+ [ContentTypes.TOOL_CALL]: {
+ id: 'call_1',
+ name: 't',
+ approval: 'pending',
+ },
},
],
} as unknown as TMessage,
@@ -808,7 +1239,9 @@ describe('useSteering', () => {
it('retry resubmits a failed interrupt-steer AS an interrupt', () => {
const { result } = setup();
act(() => {
- result.current.retrySteer('chip-1', 'retry me', undefined, undefined, { preempt: true });
+ result.current.retrySteer('chip-1', 'retry me', undefined, undefined, {
+ preempt: true,
+ });
});
expect(mockMutate).toHaveBeenCalledWith(
expect.objectContaining({ text: 'retry me', preempt: true }),
@@ -884,7 +1317,11 @@ describe('useSteering', () => {
expect(mockMutate).toHaveBeenCalledTimes(1);
expect(mockMutate).toHaveBeenCalledWith(
- expect.objectContaining({ text: 'first', preempt: true, generationCreatedAt: 41 }),
+ expect.objectContaining({
+ text: 'first',
+ preempt: true,
+ generationCreatedAt: 41,
+ }),
expect.anything(),
);
@@ -895,7 +1332,11 @@ describe('useSteering', () => {
expect(mockMutate).toHaveBeenCalledTimes(2);
expect(mockMutate).toHaveBeenNthCalledWith(
2,
- expect.objectContaining({ text: 'second', preempt: true, generationCreatedAt: 41 }),
+ expect.objectContaining({
+ text: 'second',
+ preempt: true,
+ generationCreatedAt: 41,
+ }),
expect.anything(),
);
@@ -906,7 +1347,11 @@ describe('useSteering', () => {
expect(mockMutate).toHaveBeenCalledTimes(3);
expect(mockMutate).toHaveBeenNthCalledWith(
3,
- expect.objectContaining({ text: 'third', preempt: true, generationCreatedAt: 41 }),
+ expect.objectContaining({
+ text: 'third',
+ preempt: true,
+ generationCreatedAt: 41,
+ }),
expect.anything(),
);
});
@@ -943,7 +1388,10 @@ describe('useSteering', () => {
useSteering({
index: 0,
conversationId,
- conversation: { ...agentsConversation, conversationId } as TConversation,
+ conversation: {
+ ...agentsConversation,
+ conversationId,
+ } as TConversation,
isSubmitting: true,
answerModeActive: false,
sendNow,
@@ -963,7 +1411,10 @@ describe('useSteering', () => {
expect(mockMutate).toHaveBeenCalledTimes(2);
expect(mockMutate).toHaveBeenNthCalledWith(
1,
- expect.objectContaining({ conversationId: CONVO_ID, text: 'first conversation' }),
+ expect.objectContaining({
+ conversationId: CONVO_ID,
+ text: 'first conversation',
+ }),
expect.anything(),
);
expect(mockMutate).toHaveBeenNthCalledWith(
@@ -1100,7 +1551,12 @@ describe('useSteering', () => {
it('converts the ACK chip to pending when the steer is not yet applied', () => {
mockMutate.mockImplementation((_params, { onSuccess }) => {
- onSuccess({ steerId: 'srv-2', status: 'queued', position: 1, conversationId: CONVO_ID });
+ onSuccess({
+ steerId: 'srv-2',
+ status: 'queued',
+ position: 1,
+ conversationId: CONVO_ID,
+ });
});
const { result } = setupWithState();
act(() => {
@@ -1215,14 +1671,23 @@ describe('useSteering', () => {
const now = jest.spyOn(Date, 'now').mockReturnValueOnce(1_000).mockReturnValue(9_000);
try {
mockMutate.mockImplementation((_params, { onSuccess }) => {
- onSuccess({ steerId: 'srv-t', status: 'queued', position: 1, conversationId: CONVO_ID });
+ onSuccess({
+ steerId: 'srv-t',
+ status: 'queued',
+ position: 1,
+ conversationId: CONVO_ID,
+ });
});
const { result } = setupWithState();
act(() => {
result.current.steering.submitSteer('submitted first');
});
expect(result.current.chips).toEqual([
- expect.objectContaining({ steerId: 'srv-t', status: 'pending', createdAt: 1_000 }),
+ expect.objectContaining({
+ steerId: 'srv-t',
+ status: 'pending',
+ createdAt: 1_000,
+ }),
]);
} finally {
now.mockRestore();
@@ -1279,13 +1744,23 @@ describe('useSteering', () => {
it('does not duplicate a chip already reseeded under the server id (SSE reconnect)', () => {
mockMutate.mockImplementation((_params, { onSuccess }) => {
- onSuccess({ steerId: 'srv-3', status: 'queued', position: 1, conversationId: CONVO_ID });
+ onSuccess({
+ steerId: 'srv-3',
+ status: 'queued',
+ position: 1,
+ conversationId: CONVO_ID,
+ });
});
const { result } = setupWithState({}, ({ set }) => {
// seedSteerChips already re-minted this chip from resumeState before
// the 202 ACK landed — the ACK must upsert, not append.
set(store.pendingSteersByConvoId(CONVO_ID), [
- { steerId: 'srv-3', text: 'reseeded', status: 'pending', createdAt: 1 },
+ {
+ steerId: 'srv-3',
+ text: 'reseeded',
+ status: 'pending',
+ createdAt: 1,
+ },
]);
});
act(() => {
@@ -1296,7 +1771,10 @@ describe('useSteering', () => {
it('restores the queued item (same id, original slot) when sendNow refuses', () => {
const refusingSendNow = jest.fn().mockReturnValue(false);
- const { result } = setupWithState({ isSubmitting: false, sendNow: refusingSendNow });
+ const { result } = setupWithState({
+ isSubmitting: false,
+ sendNow: refusingSendNow,
+ });
act(() => {
result.current.steering.enqueue('refused send');
result.current.steering.enqueue('still behind');
@@ -1324,13 +1802,20 @@ describe('useSteering', () => {
it('queues the steer text+files when a settled NO_ACTIVE_RUN send is refused', () => {
const steerFiles = [
- { file_id: 'file-refused', filepath: '/uploads/file-refused.png', type: 'image/png' },
+ {
+ file_id: 'file-refused',
+ filepath: '/uploads/file-refused.png',
+ type: 'image/png',
+ },
];
mockMutate.mockImplementationOnce((_params, { onError }) => {
onError({ response: { data: { code: 'NO_ACTIVE_RUN' } } });
});
const refusingSendNow = jest.fn().mockReturnValue(false);
- const { result } = setupWithState({ isSubmitting: false, sendNow: refusingSendNow });
+ const { result } = setupWithState({
+ isSubmitting: false,
+ sendNow: refusingSendNow,
+ });
act(() => {
result.current.steering.submitSteer('refused words', steerFiles);
});
@@ -1340,7 +1825,10 @@ describe('useSteering', () => {
expect.objectContaining({
expectedPredecessorCreatedAt: 41,
queuedMessageOrigin: expect.objectContaining({
- item: expect.objectContaining({ text: 'refused words', files: steerFiles }),
+ item: expect.objectContaining({
+ text: 'refused words',
+ files: steerFiles,
+ }),
}),
}),
);
@@ -1353,13 +1841,20 @@ describe('useSteering', () => {
it('queues the steer text+files when a settled-run rejection send is refused', () => {
const steerFiles = [
- { file_id: 'file-paused', filepath: '/uploads/file-paused.png', type: 'image/png' },
+ {
+ file_id: 'file-paused',
+ filepath: '/uploads/file-paused.png',
+ type: 'image/png',
+ },
];
mockMutate.mockImplementationOnce((_params, { onError }) => {
onError({ response: { data: { code: 'STEER_UNSUPPORTED' } } });
});
const refusingSendNow = jest.fn().mockReturnValue(false);
- const { result } = setupWithState({ isSubmitting: false, sendNow: refusingSendNow });
+ const { result } = setupWithState({
+ isSubmitting: false,
+ sendNow: refusingSendNow,
+ });
act(() => {
result.current.steering.submitSteer('unsupported words', steerFiles);
});
@@ -1369,19 +1864,28 @@ describe('useSteering', () => {
expect.objectContaining({
expectedPredecessorCreatedAt: 41,
queuedMessageOrigin: expect.objectContaining({
- item: expect.objectContaining({ text: 'unsupported words', files: steerFiles }),
+ item: expect.objectContaining({
+ text: 'unsupported words',
+ files: steerFiles,
+ }),
}),
}),
);
expect(result.current.chips).toEqual([]);
expect(result.current.queue).toEqual([
- expect.objectContaining({ text: 'unsupported words', files: steerFiles }),
+ expect.objectContaining({
+ text: 'unsupported words',
+ files: steerFiles,
+ }),
]);
});
it('does not restore the queued item when sendNow accepts', () => {
const acceptingSendNow = jest.fn().mockReturnValue(undefined);
- const { result } = setupWithState({ isSubmitting: false, sendNow: acceptingSendNow });
+ const { result } = setupWithState({
+ isSubmitting: false,
+ sendNow: acceptingSendNow,
+ });
act(() => {
result.current.steering.enqueue('accepted send');
});
@@ -1423,7 +1927,10 @@ describe('useSteering', () => {
it('does not lose a second queued item clicked before submitting state renders', () => {
const acceptingSendNow = jest.fn().mockReturnValue(undefined);
- const { result } = setupWithState({ isSubmitting: false, sendNow: acceptingSendNow });
+ const { result } = setupWithState({
+ isSubmitting: false,
+ sendNow: acceptingSendNow,
+ });
act(() => {
result.current.steering.enqueue('first send');
result.current.steering.enqueue('second send');
@@ -1510,8 +2017,15 @@ describe('useSteering', () => {
});
it('atomically discards a recovered source and downgrades its row in place', async () => {
- mockCancelSteer.mockResolvedValueOnce({ removed: true, generationProtocolVersion: 2 });
- const before: QueuedMessage = { id: 'queued-before', text: 'before', createdAt: 0 };
+ mockCancelSteer.mockResolvedValueOnce({
+ removed: true,
+ generationProtocolVersion: 2,
+ });
+ const before: QueuedMessage = {
+ id: 'queued-before',
+ text: 'before',
+ createdAt: 0,
+ };
const recovered: QueuedMessage = {
id: 'queued-leftover',
text: 'edit this next',
@@ -1520,12 +2034,22 @@ describe('useSteering', () => {
recoverySteerId: 'server-leftover',
recoveryClientSteerId: 'client-leftover',
expectedPredecessorCreatedAt: 41,
- files: [{ file_id: 'file-1', filepath: '/uploads/file-1.txt', type: 'text/plain' }],
+ files: [
+ {
+ file_id: 'file-1',
+ filepath: '/uploads/file-1.txt',
+ type: 'text/plain',
+ },
+ ],
quotes: ['keep quote'],
manualSkills: ['keep skill'],
priority: true,
};
- const after: QueuedMessage = { id: 'queued-after', text: 'after', createdAt: 2 };
+ const after: QueuedMessage = {
+ id: 'queued-after',
+ text: 'after',
+ createdAt: 2,
+ };
const { result } = setupWithState({}, ({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [before, recovered, after]);
});
@@ -1548,7 +2072,13 @@ describe('useSteering', () => {
text: 'edit this next',
createdAt: 1,
expectedPredecessorCreatedAt: 41,
- files: [{ file_id: 'file-1', filepath: '/uploads/file-1.txt', type: 'text/plain' }],
+ files: [
+ {
+ file_id: 'file-1',
+ filepath: '/uploads/file-1.txt',
+ type: 'text/plain',
+ },
+ ],
quotes: ['keep quote'],
manualSkills: ['keep skill'],
priority: true,
@@ -1558,7 +2088,10 @@ describe('useSteering', () => {
});
it('keeps a recovered queue row when its parked source cannot be discarded', async () => {
- mockCancelSteer.mockResolvedValueOnce({ removed: false, generationProtocolVersion: 2 });
+ mockCancelSteer.mockResolvedValueOnce({
+ removed: false,
+ generationProtocolVersion: 2,
+ });
const recovered = {
id: 'queued-leftover',
text: 'do not duplicate me',
@@ -1680,10 +2213,26 @@ describe('useSteering', () => {
mockMutate.mockImplementationOnce((_params, { onError }) => {
rejectSteer = onError;
});
- const before = { id: 'q-drained', text: 'drained meanwhile', createdAt: 10 };
- const selected = { id: 'q-racing', text: 'restore before successor', createdAt: 20 };
- const after = { id: 'q-survives', text: 'surviving successor', createdAt: 30 };
- const addedLater = { id: 'q-later', text: 'queued during request', createdAt: 40 };
+ const before = {
+ id: 'q-drained',
+ text: 'drained meanwhile',
+ createdAt: 10,
+ };
+ const selected = {
+ id: 'q-racing',
+ text: 'restore before successor',
+ createdAt: 20,
+ };
+ const after = {
+ id: 'q-survives',
+ text: 'surviving successor',
+ createdAt: 30,
+ };
+ const addedLater = {
+ id: 'q-later',
+ text: 'queued during request',
+ createdAt: 40,
+ };
const { result } = setupWithState({}, ({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [before, selected, after]);
});
@@ -1707,8 +2256,16 @@ describe('useSteering', () => {
mockMutate.mockImplementation((_params, { onError }) => {
rejectSteer.push(onError);
});
- const first = { id: 'q-concurrent-first', text: 'first instruction', createdAt: 10 };
- const second = { id: 'q-concurrent-second', text: 'second instruction', createdAt: 20 };
+ const first = {
+ id: 'q-concurrent-first',
+ text: 'first instruction',
+ createdAt: 10,
+ };
+ const second = {
+ id: 'q-concurrent-second',
+ text: 'second instruction',
+ createdAt: 20,
+ };
const { result } = setupWithState({}, ({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [first, second]);
});
@@ -1768,8 +2325,16 @@ describe('useSteering', () => {
}),
);
});
- const first = { id: 'q-receipt-first', text: 'first accepted', createdAt: 10 };
- const second = { id: 'q-receipt-second', text: 'second accepted', createdAt: 20 };
+ const first = {
+ id: 'q-receipt-first',
+ text: 'first accepted',
+ createdAt: 10,
+ };
+ const second = {
+ id: 'q-receipt-second',
+ text: 'second accepted',
+ createdAt: 20,
+ };
const { result } = setupWithState({}, ({ set }) => {
set(store.queuedMessagesByConvoId(CONVO_ID), [first, second]);
});
@@ -2002,7 +2567,12 @@ describe('useSteering', () => {
content: [
{
type: ContentTypes.TOOL_CALL,
- tool_call: { id: 'call-1', name: 'shell', approval: { actionId: 'a1' }, output: '' },
+ tool_call: {
+ id: 'call-1',
+ name: 'shell',
+ approval: { actionId: 'a1' },
+ output: '',
+ },
},
],
} as unknown as TMessage,
@@ -2107,13 +2677,18 @@ describe('useSteering', () => {
result.current.steering.steerFromComposer('steer text');
});
expect(mockMutate).toHaveBeenCalledWith(
- expect.objectContaining({ text: 'steer text', quotes: ['quoted excerpt'] }),
+ expect.objectContaining({
+ text: 'steer text',
+ quotes: ['quoted excerpt'],
+ }),
expect.anything(),
);
expect(mockMutate.mock.calls[0][0]).not.toHaveProperty('manualSkills');
// Consumed like a normal send's quotes; the excerpts now ride the steer.
expect(result.current.pendingQuotes).toEqual([]);
- expect(result.current.chips[0]).toMatchObject({ quotes: ['quoted excerpt'] });
+ expect(result.current.chips[0]).toMatchObject({
+ quotes: ['quoted excerpt'],
+ });
// A skill pick configures a NEW turn's run — it keeps waiting for one.
expect(result.current.pendingSkills).toEqual(['skill-1']);
});
@@ -2145,7 +2720,10 @@ describe('useSteering', () => {
result.current.steering.sendQueuedNow(item);
});
expect(mockMutate).toHaveBeenCalledWith(
- expect.objectContaining({ text: 'queued with quotes', quotes: ['queued excerpt'] }),
+ expect.objectContaining({
+ text: 'queued with quotes',
+ quotes: ['queued excerpt'],
+ }),
expect.anything(),
);
});
@@ -2281,7 +2859,12 @@ describe('useSteering', () => {
// Re-staging at the ACK would let that leftover auto-send bare text
// while the excerpts glue onto an unrelated composer draft.
mockMutate.mockImplementationOnce((_params, { onSuccess }) => {
- onSuccess({ steerId: 'srv-old', status: 'queued', position: 1, conversationId: CONVO_ID });
+ onSuccess({
+ steerId: 'srv-old',
+ status: 'queued',
+ position: 1,
+ conversationId: CONVO_ID,
+ });
});
const { result } = setupWithContext({}, stageContext);
act(() => {
@@ -2374,7 +2957,9 @@ describe('useSteering', () => {
quotes: ['quoted excerpt'],
});
});
- expect(result.current.queue[0]).toMatchObject({ quotes: ['quoted excerpt'] });
+ expect(result.current.queue[0]).toMatchObject({
+ quotes: ['quoted excerpt'],
+ });
act(() => {
deferredOnSuccess?.({
steerId: 'srv-converted',
@@ -2415,7 +3000,12 @@ describe('useSteering', () => {
// The run ended before the 202 landed: the ACK's queued conversion is
// the only surviving copy of the steer, so it must keep quotes + skills.
mockMutate.mockImplementationOnce((_params, { onSuccess }) => {
- onSuccess({ steerId: 'srv-late', status: 'queued', position: 1, conversationId: CONVO_ID });
+ onSuccess({
+ steerId: 'srv-late',
+ status: 'queued',
+ position: 1,
+ conversationId: CONVO_ID,
+ });
});
const { result } = setupWithContext({ isSubmitting: false });
act(() => {
@@ -2543,7 +3133,10 @@ describe('useSteering', () => {
// The quotes were consumed into the steer, so its queued fallback must
// carry them — dropping them here would lose the user's references.
expect(result.current.queue).toEqual([
- expect.objectContaining({ text: 'degraded steer', quotes: ['quoted excerpt'] }),
+ expect.objectContaining({
+ text: 'degraded steer',
+ quotes: ['quoted excerpt'],
+ }),
]);
expect(result.current.queue[0].manualSkills).toBeUndefined();
expect(result.current.pendingQuotes).toEqual([]);
diff --git a/client/src/hooks/Chat/useQueueDrain.ts b/client/src/hooks/Chat/useQueueDrain.ts
index b38cc13997..e636c113a2 100644
--- a/client/src/hooks/Chat/useQueueDrain.ts
+++ b/client/src/hooks/Chat/useQueueDrain.ts
@@ -16,6 +16,9 @@ const QUEUE_USAGE_RENEW_INTERVAL_MS = 30 * 60 * 1000;
const collectQueuedFileIds = (items: QueuedMessage[]): string[] => {
const fileIds: string[] = [];
for (const item of items) {
+ if (item.server != null) {
+ continue;
+ }
for (const file of item.files ?? []) {
if (typeof file.file_id === 'string' && file.file_id.length > 0) {
fileIds.push(file.file_id);
@@ -86,6 +89,7 @@ export default function useQueueDrain(
* during the first turn stay keyed here until that run ends. Renewing only
* the active id would skip them for the whole of that run. */
const newConvoQueue = useRecoilValue(store.queuedMessagesByConvoId(Constants.NEW_CONVO));
+ const hasServerOwnedQueue = [...ownQueue, ...newConvoQueue].some((item) => item.server != null);
/* Deduped because the two subscriptions are the same atom before migration.
* Keyed by id list so the effect re-runs when the held set changes, not
@@ -146,7 +150,9 @@ export default function useQueueDrain(
}
set(store.pendingRunEndByConvoId(end.conversationId), {
...end,
- ...((end.interruptArmed === true || interruptArmed) && { interruptArmed: true }),
+ ...((end.interruptArmed === true || interruptArmed) && {
+ interruptArmed: true,
+ }),
});
set(store.runEndByIndex(index), null);
return true;
@@ -188,7 +194,9 @@ export default function useQueueDrain(
}
set(store.pendingRunEndByConvoId(end.conversationId), {
...end,
- ...((end.interruptArmed === true || interruptArmed) && { interruptArmed: true }),
+ ...((end.interruptArmed === true || interruptArmed) && {
+ interruptArmed: true,
+ }),
});
set(store.runEndByIndex(index), null);
return null;
@@ -205,19 +213,8 @@ export default function useQueueDrain(
if (end == null) {
return null;
}
- // Consume the signal first — a hard double-fire guard even if the
- // effect re-runs before Recoil propagates.
- if (fromParked && activeConversationId) {
- set(store.pendingRunEndByConvoId(activeConversationId), null);
- } else {
- set(store.runEndByIndex(index), null);
- }
-
const indexArmed = snapshot.getLoadable(store.drainAfterAbortByIndex(index)).getValue();
const matchingIndexArm = matchesInterruptArm(indexArmed, end);
- if (matchingIndexArm) {
- set(store.drainAfterAbortByIndex(index), false);
- }
const interruptArmed = matchingIndexArm || end.interruptArmed === true;
const conversationId = end.conversationId;
@@ -241,6 +238,30 @@ export default function useQueueDrain(
: ownQueue;
const shouldDrain = end.outcome === 'completed' || interruptArmed;
+ /** A server-owned Agent row means the backend owns the next fresh-turn
+ * admission. Do not let a legacy/recovered local row race or overtake
+ * it. Keep this terminal boundary available until the authoritative
+ * queue snapshot proves whether a server-started successor now owns it. */
+ const serverOwnsBoundary = merged.some((item) => item.server != null);
+ if (serverOwnsBoundary) {
+ /** Keep the one-shot terminal signal until the authoritative snapshot
+ * removes the server row. If it was admitted, its own later terminal
+ * signal orders the remaining local queue; if it was cancelled/dead,
+ * this signal still lets the legacy successor make progress. */
+ return null;
+ }
+
+ // Consume only after server authority has yielded the boundary — a
+ // hard double-fire guard even if the effect re-runs before propagation.
+ if (fromParked && activeConversationId) {
+ set(store.pendingRunEndByConvoId(activeConversationId), null);
+ } else {
+ set(store.runEndByIndex(index), null);
+ }
+ if (matchingIndexArm) {
+ set(store.drainAfterAbortByIndex(index), false);
+ }
+
const next = shouldDrain ? (merged[0] ?? null) : null;
const remainder = next ? merged.slice(1) : merged;
@@ -340,6 +361,7 @@ export default function useQueueDrain(
drainNext,
restoreQueued,
markFilesUsage,
+ hasServerOwnedQueue,
ask,
]);
}
diff --git a/client/src/hooks/Chat/useSteering.ts b/client/src/hooks/Chat/useSteering.ts
index 6eb4ca7225..626a5c8114 100644
--- a/client/src/hooks/Chat/useSteering.ts
+++ b/client/src/hooks/Chat/useSteering.ts
@@ -2,16 +2,30 @@ import { useCallback, useEffect, useMemo, useRef } from 'react';
import { v4 } from 'uuid';
import { useToastContext } from '@librechat/client';
import { useRecoilValue, useSetRecoilState, useRecoilCallback } from 'recoil';
-import { Constants, ContentTypes, isAssistantsEndpoint } from 'librechat-data-provider';
-import type { TMessage, TConversation, TMessageContentParts } from 'librechat-data-provider';
+import {
+ Constants,
+ ContentTypes,
+ isAgentsEndpoint,
+ isAssistantsEndpoint,
+} from 'librechat-data-provider';
+import type {
+ TAgentQueuedTurnFileRef,
+ TMessage,
+ TConversation,
+ TMessageContentParts,
+} from 'librechat-data-provider';
import type { RunEnd, PendingSteer, QueuedMessage, QueuedMessageOrigin } from '~/store/families';
-import type { GenerationProtocolVersion } from '~/data-provider';
+import type { AgentQueuedTurnReceipt, GenerationProtocolVersion } from '~/data-provider';
import type { ExtendedFile, FileSetter } from '~/common';
import {
- useGetMessagesByConvoId,
useCancelSteerMutation,
useSteerMessageMutation,
useMarkFilesUsageMutation,
+ useAgentQueuedTurns,
+ useCancelAgentQueuedTurnMutation,
+ useEnqueueAgentQueuedTurnMutation,
+ isDefiniteQueuedTurnRejection,
+ isDefiniteQueuedTurnsUnsupported,
supportsGenerationProtocolV2,
} from '~/data-provider';
import {
@@ -23,6 +37,7 @@ import {
mergeRestagedQuotes,
} from '~/utils';
import useSteerConvert from '~/hooks/Chat/useSteerConvert';
+import { useLatestMessage } from '~/hooks/Messages';
import { useSetFilesToDelete } from '~/hooks/Files';
import useLocalize from '~/hooks/useLocalize';
import store from '~/store';
@@ -42,6 +57,10 @@ export interface QueuedMessageContext {
/** Server-side cap on a usage touch (mirrors `FILES_USAGE_MAX_IDS`). */
const QUEUE_USAGE_MAX_FILES = 10;
+/** Bound transport-outcome reconciliation while still guaranteeing several
+ * list reads after the enqueue promise settles. Focus/remount remains a later
+ * reconciliation path for exceptionally slow intermediaries. */
+const QUEUED_TURN_RECONCILIATION_MS = 60_000;
type SteerErrorCode =
| 'NO_ACTIVE_RUN'
@@ -97,30 +116,139 @@ function isSameRunEpoch(a: RunEnd | null, b: RunEnd): boolean {
/** True when the latest assistant message carries an unresolved tool approval —
* the run is (or is about to be) paused, so a steer POST would 409. */
-function hasLiveToolApproval(messages: TMessage[] | undefined): boolean {
- if (!messages || messages.length === 0) {
+function hasLiveToolApproval(message: TMessage | null): boolean {
+ if (message?.isCreatedByUser !== false) {
return false;
}
- for (let i = messages.length - 1; i >= 0; i--) {
- const message = messages[i];
- if (message.isCreatedByUser !== false) {
- continue;
- }
- const content = message.content;
- if (!Array.isArray(content)) {
+ const content = message.content;
+ if (!Array.isArray(content)) {
+ return false;
+ }
+ return content.some((part: TMessageContentParts | undefined) => {
+ if (part?.type !== ContentTypes.TOOL_CALL) {
return false;
}
- return content.some((part: TMessageContentParts | undefined) => {
- if (part?.type !== ContentTypes.TOOL_CALL) {
- return false;
- }
- const toolCall = part[ContentTypes.TOOL_CALL] as
- | { approval?: unknown; output?: string | null }
- | undefined;
- return toolCall?.approval != null && (toolCall.output?.length ?? 0) === 0;
- });
+ const toolCall = part[ContentTypes.TOOL_CALL] as
+ | { approval?: unknown; output?: string | null }
+ | undefined;
+ return toolCall?.approval != null && (toolCall.output?.length ?? 0) === 0;
+ });
+}
+
+interface LiveMessageState {
+ approval: boolean;
+ parentMessageId?: string;
+}
+
+function selectLiveMessageState(message: TMessage | null): LiveMessageState {
+ const parentMessageId =
+ message?.isCreatedByUser === false && typeof message.messageId === 'string'
+ ? message.messageId
+ : undefined;
+ return { approval: hasLiveToolApproval(message), parentMessageId };
+}
+
+function compareQueuedMessages(a: QueuedMessage, b: QueuedMessage): number {
+ const priority = Number(b.priority ?? false) - Number(a.priority ?? false);
+ if (priority !== 0) {
+ return priority;
+ }
+ const aServerOrder = a.server?.revision;
+ const bServerOrder = b.server?.revision;
+ if (aServerOrder != null && bServerOrder != null && aServerOrder !== bServerOrder) {
+ return aServerOrder - bServerOrder;
+ }
+ if (a.server != null && b.server == null) {
+ return -1;
+ }
+ if (a.server == null && b.server != null) {
+ return 1;
}
- return false;
+ return a.createdAt - b.createdAt;
+}
+
+function queuedTurnCreatedAt(receipt: AgentQueuedTurnReceipt): number {
+ const createdAt = Date.parse(receipt.createdAt);
+ return Number.isFinite(createdAt) ? createdAt : 0;
+}
+
+function toQueuedTurnFileRefs(files: TMessage['files']): TAgentQueuedTurnFileRef[] | undefined {
+ const refs = (files ?? []).flatMap((file): TAgentQueuedTurnFileRef[] => {
+ if (typeof file.file_id !== 'string' || file.file_id.length === 0) {
+ return [];
+ }
+ return [
+ {
+ file_id: file.file_id,
+ ...(file.type != null && { type: file.type }),
+ ...(file.filepath != null && { filepath: file.filepath }),
+ ...(file.filename != null && { filename: file.filename }),
+ ...(file.height != null && { height: file.height }),
+ ...(file.width != null && { width: file.width }),
+ ...(file.bytes != null && { bytes: file.bytes }),
+ },
+ ];
+ });
+ return refs.length > 0 ? refs : undefined;
+}
+
+function reconcileServerQueuedTurns(
+ previous: QueuedMessage[],
+ receipts: AgentQueuedTurnReceipt[],
+): QueuedMessage[] {
+ const previousByClientRequestId = new Map(
+ previous.flatMap((item) =>
+ item.clientRequestId != null ? [[item.clientRequestId, item] as const] : [],
+ ),
+ );
+ const observedClientRequestIds = new Set(receipts.map((receipt) => receipt.clientRequestId));
+ const projected = receipts.flatMap((receipt): QueuedMessage[] => {
+ if (receipt.status === 'admitted' || receipt.status === 'cancelled') {
+ return [];
+ }
+ const optimistic = previousByClientRequestId.get(receipt.clientRequestId);
+ return [
+ {
+ id: optimistic?.id ?? receipt.clientRequestId,
+ text: receipt.text,
+ createdAt: queuedTurnCreatedAt(receipt),
+ clientRequestId: receipt.clientRequestId,
+ parentMessageId: receipt.parentMessageId,
+ ...(receipt.expectedPredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: receipt.expectedPredecessorCreatedAt,
+ }),
+ ...(receipt.files != null && receipt.files.length > 0 && { files: receipt.files }),
+ ...(receipt.quotes != null && receipt.quotes.length > 0 && { quotes: receipt.quotes }),
+ ...(receipt.manualSkills != null &&
+ receipt.manualSkills.length > 0 && {
+ manualSkills: receipt.manualSkills,
+ }),
+ ...(receipt.priority === true && { priority: true }),
+ server: {
+ id: receipt.queuedTurnId,
+ status: receipt.status === 'dead' ? 'rejected' : receipt.status,
+ revision: receipt.revision,
+ ...(receipt.position != null && { position: receipt.position }),
+ ...(receipt.failure?.code != null && { errorCode: receipt.failure.code }),
+ ...(receipt.failure?.message != null && { errorMessage: receipt.failure.message }),
+ },
+ },
+ ];
+ });
+ const retained = previous.filter((item) => {
+ if (item.server == null) {
+ return true;
+ }
+ if (item.clientRequestId == null || observedClientRequestIds.has(item.clientRequestId)) {
+ return false;
+ }
+ return (
+ item.server.status === 'sending' ||
+ item.server.status === 'uncertain' ||
+ item.server.status === 'rejected'
+ );
+ });
+ return [...retained, ...projected].sort(compareQueuedMessages);
}
export interface UseSteeringParams {
@@ -177,6 +305,8 @@ export default function useSteering({
const { mutate: steerMessage } = useSteerMessageMutation();
const { mutateAsync: cancelSteer } = useCancelSteerMutation();
const { mutate: markFilesUsage } = useMarkFilesUsageMutation();
+ const { mutate: enqueueAgentQueuedTurn } = useEnqueueAgentQueuedTurnMutation();
+ const { mutateAsync: cancelAgentQueuedTurn } = useCancelAgentQueuedTurnMutation();
const defaultAction = useRecoilValue<DuringRunAction>(store.duringRunDefaultAction);
const setDefaultAction = useSetRecoilState(store.duringRunDefaultAction);
const steerInterruptsByDefault = useRecoilValue(store.steerInterruptsByDefault);
@@ -185,15 +315,87 @@ export default function useSteering({
const steerable = !isAssistantsEndpoint(endpoint);
const hasRealConvoId =
conversationId != null && conversationId !== '' && conversationId !== Constants.NEW_CONVO;
+ const serverQueueEnabled = isAgentsEndpoint(endpoint) && hasRealConvoId;
/** v1 gates the during-run UI to the primary composer, like the HITL popover. */
const enabled = steerable && index === 0;
const queueKey = hasRealConvoId ? conversationId : Constants.NEW_CONVO;
+ const queuedMessages = useRecoilValue(store.queuedMessagesByConvoId(queueKey));
+ const setQueuedMessages = useSetRecoilState(store.queuedMessagesByConvoId(queueKey));
+ const knownClientRequestIds = useMemo(
+ () =>
+ Array.from(
+ new Set(
+ queuedMessages.flatMap((item) =>
+ item.server != null && item.clientRequestId != null ? [item.clientRequestId] : [],
+ ),
+ ),
+ ).slice(0, 100),
+ [queuedMessages],
+ );
+ const reconciliationUntil = queuedMessages.reduce<number | undefined>((latest, item) => {
+ if (item.server?.status !== 'uncertain' || item.server.uncertainSince == null) {
+ return latest;
+ }
+ const until = item.server.uncertainSince + QUEUED_TURN_RECONCILIATION_MS;
+ return latest == null ? until : Math.max(latest, until);
+ }, undefined);
+ const nextReconciliationExpiry = queuedMessages.reduce<number | undefined>((earliest, item) => {
+ if (
+ item.server?.status !== 'uncertain' ||
+ item.server.uncertainSince == null ||
+ item.server.reconciliationExpired === true
+ ) {
+ return earliest;
+ }
+ const expiry = item.server.uncertainSince + QUEUED_TURN_RECONCILIATION_MS;
+ return earliest == null ? expiry : Math.min(earliest, expiry);
+ }, undefined);
+ const { data: serverQueuedTurns } = useAgentQueuedTurns(
+ conversationId,
+ serverQueueEnabled,
+ knownClientRequestIds,
+ reconciliationUntil,
+ );
const activeGenerationCreatedAt = useRecoilValue(
store.activeGenerationCreatedAtByConvoId(queueKey),
);
const activeGenerationProtocolVersion = useRecoilValue(
store.activeGenerationProtocolVersionByConvoId(queueKey),
);
+
+ useEffect(() => {
+ if (!serverQueueEnabled || serverQueuedTurns == null) {
+ return;
+ }
+ setQueuedMessages((previous) => reconcileServerQueuedTurns(previous, serverQueuedTurns));
+ }, [serverQueueEnabled, serverQueuedTurns, setQueuedMessages]);
+ useEffect(() => {
+ if (!serverQueueEnabled || nextReconciliationExpiry == null) {
+ return;
+ }
+ const expire = () => {
+ const observedAt = Date.now();
+ setQueuedMessages((previous) =>
+ previous.map((item) => {
+ if (
+ item.server?.status !== 'uncertain' ||
+ item.server.uncertainSince == null ||
+ item.server.reconciliationExpired === true ||
+ item.server.uncertainSince + QUEUED_TURN_RECONCILIATION_MS > observedAt
+ ) {
+ return item;
+ }
+ return {
+ ...item,
+ server: { ...item.server, reconciliationExpired: true },
+ };
+ }),
+ );
+ };
+ const delay = Math.max(0, nextReconciliationExpiry - Date.now());
+ const timer = window.setTimeout(expire, delay);
+ return () => window.clearTimeout(timer);
+ }, [nextReconciliationExpiry, serverQueueEnabled, setQueuedMessages]);
/** The start POST installs this epoch before any live mutation may be sent.
* `isSubmitting` flips earlier so the user can keep typing; during that
* bounded interval submits degrade to the local queue and Stop/steer refuse. */
@@ -299,22 +501,17 @@ export default function useSteering({
}
}, [pendingSteers, queueKey]);
- /** Boolean `select` so streaming deltas don't notify this subscription:
- * structural sharing only re-renders the composer when the flag flips. */
- const { data: liveToolApproval } = useGetMessagesByConvoId<boolean>(
- hasRealConvoId ? conversationId : '',
- {
- enabled: hasRealConvoId,
- select: hasLiveToolApproval,
- },
- );
+ /** The exact visible assistant tail is captured into a server queued turn,
+ * while the approval bit keeps the steering controls honest. */
+ const latestMessage = useLatestMessage(index, hasRealConvoId ? conversationId : null);
+ const liveMessageState = useMemo(() => selectLiveMessageState(latestMessage), [latestMessage]);
/** Both approval cards and `ask_user_question` suspend the current
* generation while keeping its submission slot occupied. Answer mode hides
* the ordinary during-run composer, but waiting-message controls still need
* to know a live generation exists so they remain discoverable and disabled
* instead of looking as though the action disappeared. */
const pausedOnApproval =
- enabled && isSubmitting ? answerModeActive || (liveToolApproval ?? false) : false;
+ enabled && isSubmitting ? answerModeActive || (liveMessageState?.approval ?? false) : false;
/** Whether a steer can reach the live run right now — independent of the
* user's default action, so the per-send menu can always override to
@@ -523,6 +720,22 @@ export default function useSteering({
[markFilesUsage],
);
+ const updateQueuedMessage = useRecoilCallback(
+ ({ set }) =>
+ (id: string, update: (item: QueuedMessage) => QueuedMessage | null) => {
+ set(store.queuedMessagesByConvoId(queueKey), (previous) =>
+ previous.flatMap((item) => {
+ if (item.id !== id) {
+ return [item];
+ }
+ const next = update(item);
+ return next == null ? [] : [next];
+ }),
+ );
+ },
+ [queueKey],
+ );
+
const enqueue = useRecoilCallback(
({ set }) =>
(
@@ -545,11 +758,23 @@ export default function useSteering({
if (trimmed.length === 0) {
return;
}
- const item = {
+ const parentMessageId = liveMessageState?.parentMessageId;
+ /** The active epoch is the authority-transfer fence. During startup an
+ * existing branch tail may already be visible while no generation owns
+ * it yet, so keep those turns local until the epoch is concrete. */
+ const serverOwned =
+ serverQueueEnabled && parentMessageId != null && activeGenerationCreatedAt != null;
+ const generatedClientRequestId = options?.clientRequestId == null;
+ const clientRequestId = options?.clientRequestId ?? (serverOwned ? v4() : undefined);
+ const item: QueuedMessage = {
id: options?.id ?? v4(),
text: trimmed,
createdAt: options?.createdAt ?? Date.now(),
- ...(options?.clientRequestId && { clientRequestId: options.clientRequestId }),
+ ...(clientRequestId != null && { clientRequestId }),
+ ...(serverOwned && {
+ parentMessageId,
+ server: { status: 'sending' },
+ }),
...((options?.expectedPredecessorCreatedAt ?? activeGenerationCreatedAt) != null && {
expectedPredecessorCreatedAt:
options?.expectedPredecessorCreatedAt ?? activeGenerationCreatedAt ?? undefined,
@@ -557,21 +782,117 @@ export default function useSteering({
...(options?.files && options.files.length > 0 && { files: options.files }),
...(options?.quotes && options.quotes.length > 0 && { quotes: options.quotes }),
...(options?.manualSkills &&
- options.manualSkills.length > 0 && { manualSkills: options.manualSkills }),
+ options.manualSkills.length > 0 && {
+ manualSkills: options.manualSkills,
+ }),
...(options?.front && { priority: true }),
};
set(store.queuedMessagesByConvoId(queueKey), (prev) =>
- [...prev, item].sort(
- (a, b) =>
- Number(b.priority ?? false) - Number(a.priority ?? false) ||
- a.createdAt - b.createdAt,
- ),
+ [...prev, item].sort(compareQueuedMessages),
);
if (options?.skipUsageMark !== true) {
markQueuedFilesUsage(options?.files);
}
+ if (!serverOwned || clientRequestId == null) {
+ return;
+ }
+ const serverFiles = toQueuedTurnFileRefs(item.files);
+ /** Commit the optimistic Recoil row before mutation callbacks can
+ * reconcile it. This also makes synchronous test/adaptor completions
+ * obey the same ordering as a real network response. */
+ queueMicrotask(() =>
+ enqueueAgentQueuedTurn(
+ {
+ conversationId,
+ clientRequestId,
+ parentMessageId,
+ text: item.text,
+ ...(serverFiles != null && { files: serverFiles }),
+ ...(item.quotes != null && item.quotes.length > 0 && { quotes: item.quotes }),
+ ...(item.manualSkills != null &&
+ item.manualSkills.length > 0 && {
+ manualSkills: item.manualSkills,
+ }),
+ ...(item.priority === true && { priority: true }),
+ ...(item.expectedPredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: item.expectedPredecessorCreatedAt,
+ }),
+ },
+ {
+ onSuccess: (receipt) => {
+ updateQueuedMessage(item.id, (current) => {
+ if (receipt.status === 'admitted' || receipt.status === 'cancelled') {
+ return null;
+ }
+ return {
+ ...current,
+ text: receipt.text,
+ createdAt: queuedTurnCreatedAt(receipt),
+ parentMessageId: receipt.parentMessageId,
+ server: {
+ id: receipt.queuedTurnId,
+ status: receipt.status === 'dead' ? 'rejected' : receipt.status,
+ revision: receipt.revision,
+ ...(receipt.position != null && { position: receipt.position }),
+ ...(receipt.failure?.code != null && { errorCode: receipt.failure.code }),
+ ...(receipt.failure?.message != null && {
+ errorMessage: receipt.failure.message,
+ }),
+ },
+ };
+ });
+ },
+ onError: (error) => {
+ updateQueuedMessage(item.id, (current) => {
+ if (isDefiniteQueuedTurnsUnsupported(error)) {
+ const {
+ server: _server,
+ parentMessageId: _parentMessageId,
+ clientRequestId: fallbackClientRequestId,
+ ...legacy
+ } = current;
+ return {
+ ...legacy,
+ ...(!generatedClientRequestId && fallbackClientRequestId != null
+ ? { clientRequestId: fallbackClientRequestId }
+ : {}),
+ };
+ }
+ if (!isDefiniteQueuedTurnRejection(error)) {
+ return {
+ ...current,
+ server: {
+ ...current.server,
+ status: 'uncertain',
+ uncertainSince: current.server?.uncertainSince ?? Date.now(),
+ },
+ };
+ }
+ const code = getSteerErrorCode(error);
+ return {
+ ...current,
+ server: {
+ ...current.server,
+ status: 'rejected',
+ ...(code != null && { errorCode: code }),
+ },
+ };
+ });
+ },
+ },
+ ),
+ );
},
- [queueKey, markQueuedFilesUsage, activeGenerationCreatedAt],
+ [
+ queueKey,
+ conversationId,
+ serverQueueEnabled,
+ liveMessageState?.parentMessageId,
+ markQueuedFilesUsage,
+ activeGenerationCreatedAt,
+ enqueueAgentQueuedTurn,
+ updateQueuedMessage,
+ ],
);
/** Consumes completed composer attachments into message file refs (clearing
@@ -678,7 +999,10 @@ export default function useSteering({
return rest;
}
const { quotes: _originQuotes, ...originItem } = rest.queuedOrigin.item;
- return { ...rest, queuedOrigin: { ...rest.queuedOrigin, item: originItem } };
+ return {
+ ...rest,
+ queuedOrigin: { ...rest.queuedOrigin, item: originItem },
+ };
});
return changed ? next : prev;
});
@@ -709,6 +1033,27 @@ export default function useSteering({
[queueKey],
);
+ const downgradeServerQueuedTurn = useRecoilCallback(
+ ({ snapshot, set }) =>
+ (id: string): boolean => {
+ const queue = snapshot.getLoadable(store.queuedMessagesByConvoId(queueKey)).getValue();
+ let found = false;
+ const next = queue.map((item) => {
+ if (item.id !== id) {
+ return item;
+ }
+ found = true;
+ const { server: _server, parentMessageId: _parentMessageId, ...local } = item;
+ return local;
+ });
+ if (found) {
+ set(store.queuedMessagesByConvoId(queueKey), next);
+ }
+ return found;
+ },
+ [queueKey],
+ );
+
/** Once a parked source is discarded it must never be retried as a recovery
* attempt. Downgrade the row in place so a guarded Edit that finds a newer
* draft can leave the same words, context, identity, and queue position as
@@ -747,11 +1092,46 @@ export default function useSteering({
* terminal source generation, not whichever run now occupies the chat. */
const discardQueued = useCallback(
async (item: QueuedMessage): Promise<boolean> => {
+ if (item.server != null) {
+ if (item.server.status === 'rejected' && item.server.id == null) {
+ return downgradeServerQueuedTurn(item.id);
+ }
+ if (item.server.id == null || !hasRealConvoId) {
+ showToast({
+ message: localize('com_ui_steer_cancel_failed'),
+ status: 'error',
+ });
+ return false;
+ }
+ try {
+ const receipt = await cancelAgentQueuedTurn({
+ conversationId,
+ queuedTurnId: item.server.id,
+ });
+ if (receipt.status !== 'cancelled') {
+ showToast({
+ message: localize('com_ui_steer_cancel_failed'),
+ status: 'error',
+ });
+ return false;
+ }
+ return downgradeServerQueuedTurn(item.id);
+ } catch {
+ showToast({
+ message: localize('com_ui_steer_cancel_failed'),
+ status: 'error',
+ });
+ return false;
+ }
+ }
if (item.recoverySteerId == null) {
return true;
}
if (item.recoveryClientSteerId == null || !hasRealConvoId) {
- showToast({ message: localize('com_ui_steer_cancel_failed'), status: 'error' });
+ showToast({
+ message: localize('com_ui_steer_cancel_failed'),
+ status: 'error',
+ });
return false;
}
try {
@@ -761,16 +1141,31 @@ export default function useSteering({
clientSteerId: item.recoveryClientSteerId,
});
if (removed !== true) {
- showToast({ message: localize('com_ui_steer_cancel_failed'), status: 'error' });
+ showToast({
+ message: localize('com_ui_steer_cancel_failed'),
+ status: 'error',
+ });
return false;
}
return downgradeQueuedRecovery(item.id);
} catch {
- showToast({ message: localize('com_ui_steer_cancel_failed'), status: 'error' });
+ showToast({
+ message: localize('com_ui_steer_cancel_failed'),
+ status: 'error',
+ });
return false;
}
},
- [cancelSteer, conversationId, downgradeQueuedRecovery, hasRealConvoId, localize, showToast],
+ [
+ cancelSteer,
+ cancelAgentQueuedTurn,
+ conversationId,
+ downgradeQueuedRecovery,
+ downgradeServerQueuedTurn,
+ hasRealConvoId,
+ localize,
+ showToast,
+ ],
);
/** Capture-then-remove, including the item's neighbours, so any refused send
@@ -967,8 +1362,12 @@ export default function useSteering({
expectedPredecessorCreatedAt: targetGenerationCreatedAt,
...(files && { files }),
...carried,
- ...(context?.clientRequestId && { clientRequestId: context.clientRequestId }),
- ...(context?.recoverySteerId && { recoverySteerId: context.recoverySteerId }),
+ ...(context?.clientRequestId && {
+ clientRequestId: context.clientRequestId,
+ }),
+ ...(context?.recoverySteerId && {
+ recoverySteerId: context.recoverySteerId,
+ }),
},
beforeIds: [],
afterIds: [],
@@ -1050,7 +1449,9 @@ export default function useSteering({
generationCreatedAt: targetGenerationCreatedAt,
}),
generationProtocolVersion: targetGenerationProtocolVersion,
- ...(opts?.queuedOrigin && { queuedOrigin: opts.queuedOrigin }),
+ ...(opts?.queuedOrigin && {
+ queuedOrigin: opts.queuedOrigin,
+ }),
...carried,
});
} else {
@@ -1080,7 +1481,9 @@ export default function useSteering({
generationCreatedAt: targetGenerationCreatedAt,
}),
generationProtocolVersion: targetGenerationProtocolVersion,
- ...(opts?.queuedOrigin && { queuedOrigin: opts.queuedOrigin }),
+ ...(opts?.queuedOrigin && {
+ queuedOrigin: opts.queuedOrigin,
+ }),
...carried,
} satisfies PendingSteer;
if (acknowledgeSteer(conversationId, localId, acknowledged)) {
@@ -1137,11 +1540,18 @@ export default function useSteering({
return;
}
restoreRejectedSteer();
- showToast({ message: localize('com_ui_steer_paused_queued'), status: 'info' });
+ showToast({
+ message: localize('com_ui_steer_paused_queued'),
+ status: 'info',
+ });
return;
}
const errorData = (
- error as { response?: { data?: { generationProtocolVersion?: unknown } } }
+ error as {
+ response?: {
+ data?: { generationProtocolVersion?: unknown };
+ };
+ }
)?.response?.data;
if (
code === 'RUN_REPLACED' &&
@@ -1154,7 +1564,10 @@ export default function useSteering({
* drain regardless of this tab's local submission state. */
replaceSteerChip(conversationId, localId, null);
restoreRejectedSteer();
- showToast({ message: localize('com_ui_steer_paused_queued'), status: 'info' });
+ showToast({
+ message: localize('com_ui_steer_paused_queued'),
+ status: 'info',
+ });
return;
}
if (isDefiniteSteerRejection(error)) {
@@ -1173,7 +1586,9 @@ export default function useSteering({
generationCreatedAt: targetGenerationCreatedAt,
}),
generationProtocolVersion: targetGenerationProtocolVersion,
- ...(opts?.queuedOrigin && { queuedOrigin: opts.queuedOrigin }),
+ ...(opts?.queuedOrigin && {
+ queuedOrigin: opts.queuedOrigin,
+ }),
...carried,
});
return;
@@ -1190,7 +1605,9 @@ export default function useSteering({
generationCreatedAt: targetGenerationCreatedAt,
}),
generationProtocolVersion: targetGenerationProtocolVersion,
- ...(opts?.queuedOrigin && { queuedOrigin: opts.queuedOrigin }),
+ ...(opts?.queuedOrigin && {
+ queuedOrigin: opts.queuedOrigin,
+ }),
...carried,
});
} finally {
@@ -1256,7 +1673,10 @@ export default function useSteering({
if (trimmed.length === 0 || filesLoading) {
return false;
}
- enqueue(trimmed, { files: takeComposerFiles(), ...takeComposerContext() });
+ enqueue(trimmed, {
+ files: takeComposerFiles(),
+ ...takeComposerContext(),
+ });
takeComposerDraft();
return true;
},
@@ -1274,7 +1694,10 @@ export default function useSteering({
) => {
/** A failed interrupt-steer must retry AS an interrupt — resubmitting it
* as an ordinary steer would silently let generation run on. */
- submitSteer(text, steerFiles, context, { ...opts, clientSteerId: steerId });
+ submitSteer(text, steerFiles, context, {
+ ...opts,
+ clientSteerId: steerId,
+ });
},
[submitSteer],
);
@@ -1356,7 +1779,7 @@ export default function useSteering({
* restore context so a degraded steer requeues/sends with them intact.
* `preempt` escalates the steer to interrupt at the next safe token
* boundary; it only means something on the live-run path. */
- const sendQueuedNow = useCallback(
+ const sendLocalQueuedNow = useCallback(
(item: QueuedMessage, opts?: { preempt?: boolean }) => {
/** In answer mode (and any other submission-owned non-steerable state)
* there is no immediate path. Refuse before touching queue state so a
@@ -1366,7 +1789,11 @@ export default function useSteering({
}
/** UI callers always find the item; a stale/direct caller has no original
* neighbours, so restoration falls back to the queue's priority split. */
- const origin = takeQueued(item.id) ?? { item, beforeIds: [], afterIds: [] };
+ const origin = takeQueued(item.id) ?? {
+ item,
+ beforeIds: [],
+ afterIds: [],
+ };
const taken = origin.item;
if (duringRunActive && canSteer) {
const consumed = submitSteer(
@@ -1432,6 +1859,30 @@ export default function useSteering({
],
);
+ const queuedActionClaimsRef = useRef(new Set<string>());
+ const sendQueuedNow = useCallback(
+ (item: QueuedMessage, opts?: { preempt?: boolean }) => {
+ if (item.server == null) {
+ sendLocalQueuedNow(item, opts);
+ return;
+ }
+ if (queuedActionClaimsRef.current.has(item.id)) {
+ return;
+ }
+ queuedActionClaimsRef.current.add(item.id);
+ void discardQueued(item)
+ .then((discarded) => {
+ if (discarded) {
+ sendLocalQueuedNow(item, opts);
+ }
+ })
+ .finally(() => {
+ queuedActionClaimsRef.current.delete(item.id);
+ });
+ },
+ [discardQueued, sendLocalQueuedNow],
+ );
+
/** Abort the current run and auto-send this text once the abort settles. */
const interruptAndSend = useCallback(
(text: string): boolean => {
@@ -1439,7 +1890,11 @@ export default function useSteering({
if (trimmed.length === 0 || filesLoading || !canControlGeneration) {
return false;
}
- enqueue(trimmed, { front: true, files: takeComposerFiles(), ...takeComposerContext() });
+ enqueue(trimmed, {
+ front: true,
+ files: takeComposerFiles(),
+ ...takeComposerContext(),
+ });
takeComposerDraft();
armDrainAfterAbort();
stopGenerating();
diff --git a/client/src/locales/en/translation.json b/client/src/locales/en/translation.json
index acef6e5991..83c279259b 100644
--- a/client/src/locales/en/translation.json
+++ b/client/src/locales/en/translation.json
@@ -1790,6 +1790,7 @@
"com_ui_queued_attachment_count": "{{0}} attachments queued with this message",
"com_ui_queued_messages": "Queued messages",
"com_ui_queued_quote_count": "{{0}} quoted excerpts included with this message",
+ "com_ui_queued_turn_failed": "Queued message failed",
"com_ui_quote_selections": "{{0}} selections",
"com_ui_quotes_queued": "Quotes added for your next message",
"com_ui_ran_n_agents": "Ran {{0}} agents",
@@ -1829,6 +1830,7 @@
"com_ui_remove_all_quotes": "Remove all selections",
"com_ui_remove_from_project": "Remove from project",
"com_ui_remove_queued": "Remove message",
+ "com_ui_dismiss_unconfirmed_delivery": "Dismiss unconfirmed delivery",
"com_ui_remove_quote": "Remove quote",
"com_ui_remove_skill_var": "Remove {{0}}",
"com_ui_remove_user": "Remove {{0}}",
diff --git a/client/src/store/families.ts b/client/src/store/families.ts
index 371d73971a..3ae2cb9294 100644
--- a/client/src/store/families.ts
+++ b/client/src/store/families.ts
@@ -74,7 +74,11 @@ const conversationByIndex = atomFamily<TConversation | null, string | number>({
({ onSet, node }) => {
onSet(async (newValue, oldValue) => {
const index = Number(node.key.split('__')[1]);
- logger.log('conversation', 'Setting conversation:', { index, newValue, oldValue });
+ logger.log('conversation', 'Setting conversation:', {
+ index,
+ newValue,
+ oldValue,
+ });
if (newValue?.assistant_id != null && newValue.assistant_id) {
localStorage.setItem(
`${LocalStorageKeys.ASST_ID_PREFIX}${index}${newValue.endpoint}`,
@@ -392,9 +396,33 @@ export type QueuedMessage = {
id: string;
text: string;
createdAt: number;
- /** Stable only for this queued recovery attempt and its transport retries.
- * A failed generation re-converts the durable source with a fresh key. */
+ /** Server authority for an Agent queued turn. Absence means the row remains
+ * on the legacy mounted-client drain path (including a definite old-server
+ * fallback). `uncertain` is deliberately still server-owned: falling back
+ * after an ambiguous POST could submit the same words twice. */
+ server?: {
+ id?: string;
+ status: 'sending' | 'uncertain' | 'rejected' | 'queued' | 'claimed';
+ errorCode?: string;
+ errorMessage?: string;
+ /** Observation time for a transport-ambiguous enqueue. The logical item
+ * may be much older than the request that just became uncertain. */
+ uncertainSince?: number;
+ /** The bounded reconciliation window elapsed without authoritative
+ * evidence. The outcome remains ambiguous and must never become resendable. */
+ reconciliationExpired?: boolean;
+ /** Current one-based projection; server sequence remains the stable
+ * fallback when predecessors settle and positions close up. */
+ position?: number;
+ revision?: number;
+ };
+ /** Stable identity for server enqueue/retry. Recovered steer rows also use
+ * it to dismiss their parked source; a later recovery attempt gets a fresh
+ * identity. */
clientRequestId?: string;
+ /** Exact visible branch leaf captured when this turn entered the server
+ * queue. The server revalidates it before admitting the fresh successor. */
+ parentMessageId?: string;
/** Correlation used only to durably dismiss/reclaim the parked source. */
recoveryClientSteerId?: string;
recoverySteerId?: string;
diff --git a/packages/api/src/agents/index.ts b/packages/api/src/agents/index.ts
index f33d702b4b..5cf3a71986 100644
--- a/packages/api/src/agents/index.ts
+++ b/packages/api/src/agents/index.ts
@@ -67,3 +67,5 @@ export * from './view';
export * from './reasoningLabels';
export * from './toolValidation';
export * from './remote';
+export * from './queuedTurns';
+export * from './queuedTurnHttp';
diff --git a/packages/api/src/agents/queuedTurnHttp.spec.ts b/packages/api/src/agents/queuedTurnHttp.spec.ts
new file mode 100644
index 0000000000..83dc7892d8
--- /dev/null
+++ b/packages/api/src/agents/queuedTurnHttp.spec.ts
@@ -0,0 +1,204 @@
+import { Types } from 'mongoose';
+import { AgentQueuedTurnLaneRetiredError } from '@librechat/data-schemas';
+import type { AgentQueuedTurnMethods, AgentQueuedTurnRecord } from '@librechat/data-schemas';
+import type { AgentQueuedTurnHttpDeps } from './queuedTurnHttp';
+import {
+ handleAgentQueuedTurnCancel,
+ handleAgentQueuedTurnEnqueue,
+ handleAgentQueuedTurnList,
+} from './queuedTurnHttp';
+
+const USER_ID = '507f191e810c19729de860ea';
+
+function turn(status: AgentQueuedTurnRecord['status']): AgentQueuedTurnRecord {
+ return {
+ queuedTurnId: 'queued-turn-1',
+ user: new Types.ObjectId(USER_ID),
+ conversationId: 'conversation-1',
+ agentId: 'agent_1',
+ parentMessageId: 'assistant-1',
+ clientRequestId: 'client-request-1',
+ fingerprint: 'fingerprint-1',
+ sequence: 1,
+ status,
+ priority: false,
+ text: 'follow up',
+ attempts: status === 'admitted' ? 1 : 0,
+ availableAt: new Date('2026-08-30T12:00:00Z'),
+ createdAt: new Date('2026-08-30T12:00:00Z'),
+ };
+}
+
+function requestBody() {
+ return {
+ conversationId: 'conversation-1',
+ parentMessageId: 'assistant-1',
+ clientRequestId: 'client-request-1',
+ text: 'follow up',
+ };
+}
+
+describe('Agent queued-turn HTTP admission receipts', () => {
+ it('rejects an enqueue after conversation deletion closes its lane', async () => {
+ const methods = {
+ getConvo: jest.fn(async () => ({ agent_id: 'agent_1', endpoint: 'agents' })),
+ getAgentQueuedTurnByClientRequestId: jest.fn(async () => null),
+ enqueueAgentQueuedTurn: jest.fn(async () => {
+ throw new AgentQueuedTurnLaneRetiredError();
+ }),
+ };
+ const deps = {
+ methods: methods as unknown as AgentQueuedTurnMethods & {
+ getConvo: typeof methods.getConvo;
+ },
+ lifecycle: { schedule: jest.fn(), cancel: jest.fn() },
+ checkAgentAccess: jest.fn(async () => true),
+ } satisfies AgentQueuedTurnHttpDeps;
+
+ await expect(
+ handleAgentQueuedTurnEnqueue({ id: USER_ID }, requestBody(), deps),
+ ).resolves.toEqual({
+ status: 409,
+ body: { code: 'QUEUED_TURN_CONVERSATION_DELETING' },
+ });
+ });
+
+ it('resolves a scheduling-pending response through an exact same-body terminal replay', async () => {
+ const enqueueAgentQueuedTurn = jest
+ .fn()
+ .mockResolvedValueOnce({ turn: turn('queued'), replayed: false });
+ const getAgentQueuedTurnByClientRequestId = jest
+ .fn()
+ .mockResolvedValueOnce(null)
+ .mockResolvedValueOnce(turn('admitted'));
+ const schedule = jest.fn().mockRejectedValueOnce(new Error('scheduler unavailable'));
+ const methods = {
+ getConvo: jest.fn(async () => ({
+ agent_id: 'agent_1',
+ endpoint: 'agents',
+ })),
+ enqueueAgentQueuedTurn,
+ getAgentQueuedTurnByClientRequestId,
+ listActiveAgentQueuedTurns: jest.fn(async () => []),
+ };
+ const deps = {
+ methods: methods as unknown as AgentQueuedTurnMethods & {
+ getConvo: typeof methods.getConvo;
+ },
+ lifecycle: { schedule, cancel: jest.fn() },
+ checkAgentAccess: jest.fn(async () => true),
+ } satisfies AgentQueuedTurnHttpDeps;
+
+ await expect(
+ handleAgentQueuedTurnEnqueue({ id: USER_ID }, requestBody(), deps),
+ ).resolves.toMatchObject({
+ status: 503,
+ body: { code: 'QUEUED_TURN_SCHEDULING_PENDING' },
+ });
+ await expect(
+ handleAgentQueuedTurnEnqueue({ id: USER_ID }, requestBody(), deps),
+ ).resolves.toMatchObject({
+ status: 200,
+ body: {
+ receipt: {
+ queuedTurnId: 'queued-turn-1',
+ clientRequestId: 'client-request-1',
+ status: 'admitted',
+ },
+ },
+ });
+
+ expect(getAgentQueuedTurnByClientRequestId).toHaveBeenCalledTimes(2);
+ expect(getAgentQueuedTurnByClientRequestId.mock.calls[1][0]).toEqual(
+ getAgentQueuedTurnByClientRequestId.mock.calls[0][0],
+ );
+ expect(enqueueAgentQueuedTurn).toHaveBeenCalledTimes(1);
+ expect(methods.getConvo).toHaveBeenCalledTimes(1);
+ expect(schedule).toHaveBeenCalledTimes(1);
+ });
+
+ it('surfaces a dead receipt and lets the user dismiss its delivery', async () => {
+ const dead = {
+ ...turn('dead'),
+ deliveryKey: 'delivery-1',
+ settledAt: new Date('2026-08-30T12:01:00Z'),
+ terminalReceipt: {
+ outcome: 'dead' as const,
+ settledAt: new Date('2026-08-30T12:01:00Z'),
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'could not admit turn' },
+ },
+ };
+ const cancelled = {
+ ...dead,
+ status: 'cancelled' as const,
+ terminalReceipt: {
+ outcome: 'cancelled' as const,
+ settledAt: new Date('2026-08-30T12:02:00Z'),
+ },
+ };
+ const methods = {
+ getConvo: jest.fn(async () => ({ agent_id: 'agent_1', endpoint: 'agents' })),
+ listAgentQueuedTurnReceipts: jest.fn(async () => [dead]),
+ };
+ const cancel = jest.fn(async () => ({ outcome: 'cancelled' as const, turn: cancelled }));
+ const deps = {
+ methods: methods as unknown as AgentQueuedTurnMethods & {
+ getConvo: typeof methods.getConvo;
+ },
+ lifecycle: { schedule: jest.fn(), cancel },
+ checkAgentAccess: jest.fn(async () => true),
+ } satisfies AgentQueuedTurnHttpDeps;
+
+ await expect(
+ handleAgentQueuedTurnList({ id: USER_ID }, 'conversation-1', deps),
+ ).resolves.toMatchObject({
+ status: 200,
+ body: {
+ queuedTurns: [
+ {
+ queuedTurnId: 'queued-turn-1',
+ status: 'dead',
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'could not admit turn' },
+ },
+ ],
+ },
+ });
+ await expect(
+ handleAgentQueuedTurnCancel({ id: USER_ID }, 'queued-turn-1', deps),
+ ).resolves.toMatchObject({
+ status: 200,
+ body: { receipt: { queuedTurnId: 'queued-turn-1', status: 'cancelled' } },
+ });
+ expect(cancel).toHaveBeenCalledWith(expect.objectContaining({ queuedTurnId: 'queued-turn-1' }));
+ });
+
+ it('retires a cancelled source after its published delivery receipt expires', async () => {
+ const cancelled = {
+ ...turn('cancelled'),
+ deliveryKey: 'delivery-expired',
+ deliveryState: 'published' as const,
+ terminalReceipt: {
+ outcome: 'cancelled' as const,
+ settledAt: new Date('2026-08-30T12:02:00Z'),
+ },
+ };
+ const methods = {
+ getConvo: jest.fn(async () => ({ agent_id: 'agent_1', endpoint: 'agents' })),
+ };
+ const cancel = jest.fn(async () => ({
+ outcome: 'already_cancelled' as const,
+ turn: cancelled,
+ }));
+ const deps = {
+ methods: methods as unknown as AgentQueuedTurnMethods & {
+ getConvo: typeof methods.getConvo;
+ },
+ lifecycle: { schedule: jest.fn(), cancel },
+ } satisfies AgentQueuedTurnHttpDeps;
+
+ await expect(
+ handleAgentQueuedTurnCancel({ id: USER_ID }, 'queued-turn-1', deps),
+ ).resolves.toMatchObject({ status: 200 });
+ expect(cancel).toHaveBeenCalledTimes(1);
+ });
+});
diff --git a/packages/api/src/agents/queuedTurnHttp.ts b/packages/api/src/agents/queuedTurnHttp.ts
new file mode 100644
index 0000000000..765a2b835a
--- /dev/null
+++ b/packages/api/src/agents/queuedTurnHttp.ts
@@ -0,0 +1,452 @@
+import { Types } from 'mongoose';
+import {
+ enqueueAgentQueuedTurnSchema,
+ isAgentsEndpoint,
+ isEphemeralAgentId,
+} from 'librechat-data-provider';
+import {
+ AgentQueuedTurnCapacityError,
+ AgentQueuedTurnConflictError,
+ AgentQueuedTurnLaneRetiredError,
+} from '@librechat/data-schemas';
+import type {
+ AgentQueuedTurnActiveRecord,
+ AgentQueuedTurnMethods,
+ AgentQueuedTurnRecord,
+ IMongoFile,
+} from '@librechat/data-schemas';
+import type {
+ TAgentQueuedTurnFileRef,
+ TAgentQueuedTurnReceipt,
+ TFile,
+} from 'librechat-data-provider';
+import type { AgentQueuedTurnLifecycle } from './queuedTurns';
+import type { SteerFileFetcher } from './steering/request';
+import type { SteerRequestUser } from './steering/refs';
+import { buildOwnerFilter, collectFileIds, toSteerFileRef } from './steering/refs';
+import { getReferencedQuotes } from '~/utils';
+
+const MAX_QUEUED_TURN_LENGTH = 16_000;
+const MAX_QUEUED_TURN_FILES = 10;
+const CAPABILITY = { supported: true, durability: 'durable' } as const;
+
+interface QueuedTurnConversation {
+ agent_id?: string;
+ endpoint?: string;
+ tenantId?: string;
+}
+
+interface QueuedTurnHttpMethods extends AgentQueuedTurnMethods {
+ getConvo: (userId: string, conversationId: string) => Promise<QueuedTurnConversation | null>;
+}
+
+export interface AgentQueuedTurnHttpDeps {
+ methods: QueuedTurnHttpMethods;
+ lifecycle: Pick<AgentQueuedTurnLifecycle, 'schedule' | 'cancel'>;
+ getFiles?: SteerFileFetcher;
+ updateFilesUsage?: (
+ files: Array<{ file_id: string }>,
+ fileIds?: string[],
+ options?: { user?: string; tenantId?: string | null },
+ ) => Promise<unknown[]>;
+ checkAgentAccess?: (run: { agentId?: string; endpoint?: string }) => Promise<boolean>;
+ isPrincipalActive?: (userId: string) => boolean | Promise<boolean>;
+}
+
+export interface AgentQueuedTurnHttpResult {
+ status: number;
+ body: Record<string, unknown>;
+}
+
+function owner(user: SteerRequestUser): { user: Types.ObjectId; tenantId?: string } | null {
+ if (typeof user.id !== 'string' || !Types.ObjectId.isValid(user.id)) {
+ return null;
+ }
+ return {
+ user: new Types.ObjectId(user.id),
+ ...(user.tenantId != null && { tenantId: user.tenantId }),
+ };
+}
+
+function sameTenant(actual: unknown, expected: string | undefined): boolean {
+ return expected == null ? actual == null : actual === expected;
+}
+
+function receipt(
+ turn: AgentQueuedTurnRecord | AgentQueuedTurnActiveRecord,
+ position?: number,
+): TAgentQueuedTurnReceipt {
+ const updatedAt = turn.updatedAt ?? turn.createdAt;
+ return {
+ queuedTurnId: turn.queuedTurnId,
+ conversationId: turn.conversationId,
+ parentMessageId: turn.parentMessageId,
+ clientRequestId: turn.clientRequestId,
+ text: turn.text,
+ ...(turn.files != null && { files: turn.files }),
+ ...(turn.quotes != null && { quotes: turn.quotes }),
+ ...(turn.manualSkills != null && { manualSkills: turn.manualSkills }),
+ priority: turn.priority,
+ ...(turn.expectedPredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: turn.expectedPredecessorCreatedAt,
+ }),
+ status: turn.status,
+ ...(position != null && { position }),
+ revision: turn.sequence,
+ createdAt: turn.createdAt.toISOString(),
+ updatedAt: updatedAt.toISOString(),
+ ...(turn.terminalReceipt?.failure != null && {
+ failure: {
+ code: turn.terminalReceipt.failure.code,
+ message: turn.terminalReceipt.failure.message,
+ },
+ }),
+ };
+}
+
+function parseClientRequestIds(raw: unknown): string[] | null {
+ if (raw == null) {
+ return [];
+ }
+ const values = Array.isArray(raw) ? raw : [raw];
+ if (values.length > 100 || values.some((value) => typeof value !== 'string')) {
+ return null;
+ }
+ const normalized = [...new Set((values as string[]).map((value) => value.trim()))];
+ return normalized.some((value) => value.length === 0 || value.length > 128) ? null : normalized;
+}
+
+function sameStrings(left: readonly string[] | undefined, right: readonly string[] | undefined) {
+ const a = left ?? [];
+ const b = right ?? [];
+ return a.length === b.length && a.every((value, index) => value === b[index]);
+}
+
+function uniqueStrings(values: readonly string[] | undefined): string[] | undefined {
+ return values == null ? undefined : [...new Set(values)];
+}
+
+function matchesReplayIntent(
+ turn: AgentQueuedTurnRecord,
+ input: {
+ parentMessageId: string;
+ clientRequestId: string;
+ files?: readonly TAgentQueuedTurnFileRef[];
+ manualSkills?: readonly string[];
+ expectedPredecessorCreatedAt?: number;
+ },
+ text: string,
+ quotes: readonly string[] | undefined,
+): boolean {
+ return (
+ turn.parentMessageId === input.parentMessageId &&
+ turn.clientRequestId === input.clientRequestId &&
+ turn.text === text &&
+ sameStrings(
+ turn.files?.map((file) => file.file_id),
+ uniqueStrings(input.files?.map((file) => file.file_id)),
+ ) &&
+ sameStrings(turn.quotes, quotes) &&
+ sameStrings(turn.manualSkills, uniqueStrings(input.manualSkills)) &&
+ turn.expectedPredecessorCreatedAt === input.expectedPredecessorCreatedAt
+ );
+}
+
+async function authorizeConversation(
+ user: SteerRequestUser,
+ conversationId: string,
+ deps: AgentQueuedTurnHttpDeps,
+): Promise<
+ | { status: 200; conversation: QueuedTurnConversation & { agent_id: string } }
+ | AgentQueuedTurnHttpResult
+> {
+ if (user.id == null) {
+ return { status: 401, body: { code: 'UNAUTHORIZED' } };
+ }
+ const conversation = await deps.methods.getConvo(user.id, conversationId);
+ if (conversation == null || !sameTenant(conversation.tenantId, user.tenantId)) {
+ return { status: 404, body: { code: 'CONVERSATION_NOT_FOUND' } };
+ }
+ const agentId = conversation.agent_id;
+ if (
+ !isAgentsEndpoint(conversation.endpoint) ||
+ typeof agentId !== 'string' ||
+ isEphemeralAgentId(agentId)
+ ) {
+ return { status: 501, body: { code: 'QUEUED_TURNS_UNSUPPORTED' } };
+ }
+ if (
+ deps.checkAgentAccess != null &&
+ !(await deps.checkAgentAccess({ agentId, endpoint: conversation.endpoint }))
+ ) {
+ return { status: 403, body: { code: 'FORBIDDEN' } };
+ }
+ return { status: 200, conversation: { ...conversation, agent_id: agentId } };
+}
+
+async function resolveFiles(
+ raw: unknown,
+ user: SteerRequestUser,
+ deps: AgentQueuedTurnHttpDeps,
+): Promise<{ files?: TAgentQueuedTurnFileRef[]; error?: AgentQueuedTurnHttpResult }> {
+ if (raw == null) {
+ return {};
+ }
+ if (!Array.isArray(raw) || raw.length > MAX_QUEUED_TURN_FILES) {
+ return { error: { status: 400, body: { code: 'INVALID_FILES' } } };
+ }
+ const requested = raw.map(toSteerFileRef);
+ if (requested.some((file) => file == null)) {
+ return { error: { status: 400, body: { code: 'INVALID_FILES' } } };
+ }
+ const files = requested as Partial<TFile>[];
+ if (files.length === 0) {
+ return {};
+ }
+ if (deps.getFiles == null || deps.updateFilesUsage == null) {
+ return {
+ error: { status: 503, body: { code: 'FILE_STORAGE_UNAVAILABLE' } },
+ };
+ }
+ const ids = collectFileIds(files);
+ const filter = buildOwnerFilter(ids, user);
+ if (filter == null) {
+ return { error: { status: 400, body: { code: 'INVALID_FILES' } } };
+ }
+ const docs = (await deps.getFiles(filter, {}, {})) ?? [];
+ const byId = new Map(docs.map((doc: IMongoFile) => [doc.file_id, doc]));
+ const resolved = ids.flatMap((id): TAgentQueuedTurnFileRef[] => {
+ const ref = toSteerFileRef(byId.get(id));
+ return ref == null ? [] : [{ ...ref, file_id: id }];
+ });
+ if (resolved.length !== ids.length) {
+ return { error: { status: 400, body: { code: 'INVALID_FILES' } } };
+ }
+ const retained = await deps.updateFilesUsage(
+ ids.map((file_id) => ({ file_id })),
+ undefined,
+ { user: user.id, tenantId: user.tenantId },
+ );
+ const retainedIds = new Set<string>();
+ for (const file of retained) {
+ if (
+ file != null &&
+ typeof file === 'object' &&
+ 'file_id' in file &&
+ typeof file.file_id === 'string'
+ ) {
+ retainedIds.add(file.file_id);
+ }
+ }
+ if (!ids.every((id) => retainedIds.has(id))) {
+ return { error: { status: 503, body: { code: 'FILE_RETENTION_FAILED' } } };
+ }
+ return { files: resolved };
+}
+
+export async function handleAgentQueuedTurnEnqueue(
+ user: SteerRequestUser,
+ body: unknown,
+ deps: AgentQueuedTurnHttpDeps,
+): Promise<AgentQueuedTurnHttpResult> {
+ const parsed = enqueueAgentQueuedTurnSchema.safeParse(body);
+ if (!parsed.success) {
+ return { status: 400, body: { code: 'INVALID_QUEUED_TURN' } };
+ }
+ const input = parsed.data;
+ const text = input.text.replace(/\0/g, '').trim();
+ if (text.length === 0) {
+ return { status: 400, body: { code: 'EMPTY_TEXT' } };
+ }
+ if (text.length > MAX_QUEUED_TURN_LENGTH) {
+ return {
+ status: 413,
+ body: { code: 'QUEUED_TURN_TOO_LONG', maxLength: MAX_QUEUED_TURN_LENGTH },
+ };
+ }
+ /** Interrupt-and-send remains the existing warm steer path; a future lane
+ * arbiter can add durable front insertion without weakening FIFO here. */
+ if (input.priority === true) {
+ return { status: 501, body: { code: 'QUEUED_TURN_PRIORITY_UNSUPPORTED' } };
+ }
+ const scope = owner(user);
+ if (scope == null) {
+ return { status: 401, body: { code: 'UNAUTHORIZED' } };
+ }
+ if (deps.isPrincipalActive != null && !(await deps.isPrincipalActive(user.id!))) {
+ return { status: 409, body: { code: 'USER_DELETION_IN_PROGRESS' } };
+ }
+ const quotes = getReferencedQuotes(input.quotes) ?? undefined;
+ /** Request identity is the durable receipt address. Resolve it before
+ * mutable conversation/agent/file preconditions so an accepted turn remains
+ * observable even if those resources change before a lost-response replay. */
+ const existing = await deps.methods.getAgentQueuedTurnByClientRequestId({
+ ...scope,
+ conversationId: input.conversationId,
+ clientRequestId: input.clientRequestId,
+ });
+ if (existing != null) {
+ if (!matchesReplayIntent(existing, input, text, quotes)) {
+ return { status: 409, body: { code: 'QUEUED_TURN_IDEMPOTENCY_CONFLICT' } };
+ }
+ if (existing.status !== 'queued' && existing.status !== 'claimed') {
+ return {
+ status: 200,
+ body: { receipt: receipt(existing), capability: CAPABILITY },
+ };
+ }
+ try {
+ await deps.lifecycle.schedule(existing);
+ } catch {
+ return { status: 503, body: { code: 'QUEUED_TURN_SCHEDULING_PENDING' } };
+ }
+ const active = await deps.methods.listActiveAgentQueuedTurns({
+ ...scope,
+ conversationId: input.conversationId,
+ });
+ const position = active.findIndex((turn) => turn.queuedTurnId === existing.queuedTurnId);
+ return {
+ status: 202,
+ body: {
+ receipt: receipt(existing, position >= 0 ? position + 1 : undefined),
+ capability: CAPABILITY,
+ },
+ };
+ }
+ const authorized = await authorizeConversation(user, input.conversationId, deps);
+ if ('body' in authorized) {
+ return authorized;
+ }
+ const resolvedFiles = await resolveFiles(input.files, user, deps);
+ if (resolvedFiles.error != null) {
+ return resolvedFiles.error;
+ }
+ try {
+ const queued = await deps.methods.enqueueAgentQueuedTurn({
+ ...scope,
+ conversationId: input.conversationId,
+ agentId: authorized.conversation.agent_id,
+ parentMessageId: input.parentMessageId,
+ clientRequestId: input.clientRequestId,
+ text,
+ ...(resolvedFiles.files != null && { files: resolvedFiles.files }),
+ ...(quotes != null && { quotes }),
+ ...(input.manualSkills != null && { manualSkills: input.manualSkills }),
+ priority: false,
+ ...(input.expectedPredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: input.expectedPredecessorCreatedAt,
+ }),
+ });
+ /** A same-body replay is the transport-independent receipt lookup. It can
+ * arrive after the original row already settled, in which case no new
+ * scheduling side effect is valid or necessary. */
+ if (queued.replayed && queued.turn.status !== 'queued' && queued.turn.status !== 'claimed') {
+ return {
+ status: 200,
+ body: { receipt: receipt(queued.turn), capability: CAPABILITY },
+ };
+ }
+ try {
+ await deps.lifecycle.schedule(queued.turn);
+ } catch {
+ /** The row is the outbox source of truth; periodic recovery repairs the
+ * record-to-delivery seam. A retry with the same clientRequestId replays. */
+ return { status: 503, body: { code: 'QUEUED_TURN_SCHEDULING_PENDING' } };
+ }
+ const active = await deps.methods.listActiveAgentQueuedTurns({
+ ...scope,
+ conversationId: input.conversationId,
+ });
+ const position = active.findIndex((turn) => turn.queuedTurnId === queued.turn.queuedTurnId);
+ return {
+ status: 202,
+ body: {
+ receipt: receipt(queued.turn, position >= 0 ? position + 1 : undefined),
+ capability: CAPABILITY,
+ },
+ };
+ } catch (error) {
+ if (error instanceof AgentQueuedTurnCapacityError) {
+ return { status: 429, body: { code: 'QUEUED_TURN_QUEUE_FULL' } };
+ }
+ if (error instanceof AgentQueuedTurnConflictError) {
+ return {
+ status: 409,
+ body: { code: 'QUEUED_TURN_IDEMPOTENCY_CONFLICT' },
+ };
+ }
+ if (error instanceof AgentQueuedTurnLaneRetiredError) {
+ return { status: 409, body: { code: 'QUEUED_TURN_CONVERSATION_DELETING' } };
+ }
+ throw error;
+ }
+}
+
+export async function handleAgentQueuedTurnList(
+ user: SteerRequestUser,
+ conversationId: unknown,
+ deps: AgentQueuedTurnHttpDeps,
+ rawClientRequestIds?: unknown,
+): Promise<AgentQueuedTurnHttpResult> {
+ if (typeof conversationId !== 'string' || conversationId.length === 0) {
+ return { status: 400, body: { code: 'INVALID_CONVERSATION' } };
+ }
+ const scope = owner(user);
+ if (scope == null) {
+ return { status: 401, body: { code: 'UNAUTHORIZED' } };
+ }
+ const clientRequestIds = parseClientRequestIds(rawClientRequestIds);
+ if (clientRequestIds == null) {
+ return { status: 400, body: { code: 'INVALID_CLIENT_REQUEST_IDS' } };
+ }
+ const authorized = await authorizeConversation(user, conversationId, deps);
+ if ('body' in authorized) {
+ return authorized;
+ }
+ const turns = await deps.methods.listAgentQueuedTurnReceipts({
+ ...scope,
+ conversationId,
+ clientRequestIds,
+ });
+ let activePosition = 0;
+ return {
+ status: 200,
+ body: {
+ queuedTurns: turns.map((turn) => {
+ if (turn.status !== 'queued' && turn.status !== 'claimed') {
+ return receipt(turn);
+ }
+ activePosition += 1;
+ return receipt(turn, activePosition);
+ }),
+ capability: CAPABILITY,
+ revision: turns.reduce((latest, turn) => Math.max(latest, turn.sequence), 0),
+ },
+ };
+}
+
+export async function handleAgentQueuedTurnCancel(
+ user: SteerRequestUser,
+ queuedTurnId: unknown,
+ deps: AgentQueuedTurnHttpDeps,
+): Promise<AgentQueuedTurnHttpResult> {
+ if (typeof queuedTurnId !== 'string' || queuedTurnId.length === 0) {
+ return { status: 400, body: { code: 'INVALID_QUEUED_TURN' } };
+ }
+ const scope = owner(user);
+ if (scope == null) {
+ return { status: 401, body: { code: 'UNAUTHORIZED' } };
+ }
+ const cancelled = await deps.lifecycle.cancel({
+ ...scope,
+ queuedTurnId,
+ settledAt: new Date(),
+ });
+ if (cancelled.outcome === 'not_found') {
+ return { status: 404, body: { code: 'QUEUED_TURN_NOT_FOUND' } };
+ }
+ if (cancelled.outcome === 'not_cancellable') {
+ return { status: 409, body: { code: 'QUEUED_TURN_ALREADY_ADMITTING' } };
+ }
+ return { status: 200, body: { receipt: receipt(cancelled.turn) } };
+}
diff --git a/packages/api/src/agents/queuedTurns.spec.ts b/packages/api/src/agents/queuedTurns.spec.ts
new file mode 100644
index 0000000000..3757982b30
--- /dev/null
+++ b/packages/api/src/agents/queuedTurns.spec.ts
@@ -0,0 +1,942 @@
+import { Types } from 'mongoose';
+import type {
+ AgentQueuedTurnClaim,
+ AgentQueuedTurnMethods,
+ AgentQueuedTurnRecord,
+ ConversationMethods,
+ IMessage,
+ MessageMethods,
+} from '@librechat/data-schemas';
+import type { AgentQueuedTurnResolverDeps, AgentQueuedTurnSchedulerDeps } from './queuedTurns';
+import type { AgentContinueTriggerEnvelope } from './triggers/envelope';
+import { AGENT_QUEUED_TURN_SOURCE, createAgentQueuedTurnLifecycle } from './queuedTurns';
+import { getAgentTriggerIdempotencyKey } from './triggers/envelope';
+import { AgentTriggerExecutionError } from './triggers/host';
+
+const NOW = Date.parse('2026-08-30T12:00:00Z');
+const USER_ID = '507f191e810c19729de860ea';
+
+function envelope(): AgentContinueTriggerEnvelope {
+ return {
+ version: 1,
+ mode: 'continue',
+ requestId: 'request-1',
+ deliveryId: 'queued-turn-1',
+ receivedAt: NOW,
+ principal: { userId: USER_ID, tenantId: 'tenant-1' },
+ event: {
+ id: 'queued-turn-1',
+ type: 'agent.queued-turn',
+ occurredAt: NOW,
+ source: { id: AGENT_QUEUED_TURN_SOURCE, type: 'internal' },
+ payload: { queuedTurnId: 'queued-turn-1' },
+ },
+ target: {
+ agentId: 'agent-1',
+ conversationId: 'conversation-1',
+ parentMessageId: 'assistant-1',
+ },
+ input: 'queued words',
+ };
+}
+
+function claim(): AgentQueuedTurnClaim {
+ return {
+ queuedTurnId: 'queued-turn-1',
+ user: new Types.ObjectId(USER_ID),
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ agentId: 'agent-1',
+ parentMessageId: 'assistant-1',
+ clientRequestId: 'client-1',
+ fingerprint: 'fingerprint-1',
+ sequence: 1,
+ status: 'claimed',
+ priority: false,
+ text: 'queued words',
+ files: [{ file_id: 'file-1' }],
+ quotes: ['quote'],
+ manualSkills: ['skill-1'],
+ attempts: 1,
+ availableAt: new Date(NOW),
+ claimId: 'trigger-1',
+ claimBy: 'worker-1',
+ claimUntil: new Date(NOW + 60_000),
+ createdAt: new Date(NOW - 1_000),
+ };
+}
+
+function persistedMessage(
+ value: Pick<IMessage, 'messageId' | 'parentMessageId' | 'isCreatedByUser'> &
+ Partial<Pick<IMessage, 'createdAt' | 'unfinished' | 'error'>>,
+): IMessage {
+ return value as unknown as IMessage;
+}
+
+function resolverMethods() {
+ const turn = claim();
+ const methods = {
+ getConvo: jest.fn(async () => ({ tenantId: 'tenant-1', agent_id: 'agent-1' })),
+ getMessages: jest.fn(
+ async (..._args: Parameters<MessageMethods['getMessages']>): Promise<IMessage[]> => [
+ persistedMessage({
+ messageId: 'assistant-1',
+ parentMessageId: 'user-1',
+ isCreatedByUser: false,
+ createdAt: new Date(NOW - 500),
+ }),
+ persistedMessage({
+ messageId: 'assistant-2',
+ parentMessageId: 'assistant-1',
+ isCreatedByUser: false,
+ createdAt: new Date(NOW),
+ }),
+ ],
+ ),
+ claimNextAgentQueuedTurn: jest.fn(
+ async (..._args: Parameters<AgentQueuedTurnMethods['claimNextAgentQueuedTurn']>) => ({
+ outcome: 'acquired' as const,
+ claim: turn,
+ }),
+ ),
+ releaseAgentQueuedTurn: jest.fn(async () => ({ outcome: 'released' as const, turn })),
+ beginAgentQueuedTurnAdmission: jest.fn(async () => ({
+ outcome: 'started' as const,
+ turn,
+ })),
+ getEffectiveAgentQueuedTurnPredecessor: jest.fn(
+ async (
+ ..._args: Parameters<AgentQueuedTurnMethods['getEffectiveAgentQueuedTurnPredecessor']>
+ ): Promise<number | undefined> => undefined,
+ ),
+ markAgentQueuedTurnAdmitted: jest.fn(async () => ({
+ outcome: 'admitted' as const,
+ turn: { ...turn, status: 'admitted' as const },
+ })),
+ hasAgentQueuedTurnAdmissionReceipt: jest.fn(async () => true),
+ };
+ return {
+ methods: methods as unknown as AgentQueuedTurnMethods &
+ Pick<ConversationMethods, 'getConvo'> &
+ Pick<MessageMethods, 'getMessages'>,
+ spies: methods,
+ };
+}
+
+function createAgentQueuedTurnResolver(deps: AgentQueuedTurnResolverDeps) {
+ return createAgentQueuedTurnLifecycle({
+ ...deps,
+ getGenerationAdmissionEvidence: async () => null,
+ enqueue: async () => ({ deliveryKey: 'unused-delivery' }),
+ }).prepareContinue;
+}
+
+function createAgentQueuedTurnDeadLetterSettlement({
+ methods,
+ getGenerationAdmissionEvidence = async () => null,
+ now,
+}: {
+ methods: Pick<AgentQueuedTurnMethods, 'deadLetterAgentQueuedTurn'>;
+ getGenerationAdmissionEvidence?: AgentQueuedTurnSchedulerDeps['getGenerationAdmissionEvidence'];
+ now?: () => number;
+}) {
+ return createAgentQueuedTurnLifecycle({
+ methods: methods as AgentQueuedTurnSchedulerDeps['methods'] &
+ AgentQueuedTurnResolverDeps['methods'],
+ getGenerationJob: async () => null,
+ getGenerationAdmissionEvidence,
+ enqueue: async () => ({ deliveryKey: 'unused-delivery' }),
+ ...(now != null && { now }),
+ }).settleBeforeDeadLetter;
+}
+
+function createAgentQueuedTurnScheduler(deps: AgentQueuedTurnSchedulerDeps) {
+ return createAgentQueuedTurnLifecycle({
+ ...deps,
+ methods: deps.methods as AgentQueuedTurnSchedulerDeps['methods'] &
+ AgentQueuedTurnResolverDeps['methods'],
+ getGenerationJob: async () => null,
+ });
+}
+
+describe('Agent queued-turn continuation', () => {
+ it('dead-letters a delivery while preserving an admission-indeterminate source', async () => {
+ const deadLetterAgentQueuedTurn = jest.fn(async () => ({
+ outcome: 'admission_indeterminate' as const,
+ turn: claim(),
+ }));
+ const settle = createAgentQueuedTurnDeadLetterSettlement({
+ methods: { deadLetterAgentQueuedTurn },
+ now: () => NOW,
+ });
+
+ await expect(
+ settle(envelope(), {
+ code: 'ATTEMPTS_EXHAUSTED',
+ message: 'admission receipt unavailable',
+ certainty: 'ambiguous',
+ retryable: true,
+ attemptedAt: new Date(NOW),
+ }),
+ ).resolves.toBeUndefined();
+ expect(deadLetterAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-1',
+ deliveryKey: getAgentTriggerIdempotencyKey(envelope()),
+ }),
+ );
+ });
+
+ it('passes exact live-generation evidence into dead-letter reconciliation', async () => {
+ const deliveryKey = getAgentTriggerIdempotencyKey(envelope());
+ const deadLetterAgentQueuedTurn = jest.fn(async () => ({
+ outcome: 'admission_reconciled' as const,
+ turn: { ...claim(), status: 'admitted' as const },
+ }));
+ const settle = createAgentQueuedTurnDeadLetterSettlement({
+ methods: { deadLetterAgentQueuedTurn },
+ getGenerationAdmissionEvidence: async () => ({
+ generationId: 'generation-1',
+ generationCreatedAt: NOW + 1,
+ }),
+ now: () => NOW,
+ });
+
+ await settle(envelope(), {
+ code: 'ATTEMPTS_EXHAUSTED',
+ message: 'admission receipt unavailable',
+ certainty: 'ambiguous',
+ retryable: true,
+ attemptedAt: new Date(NOW),
+ });
+ expect(deadLetterAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ deliveryKey,
+ admissionEvidence: {
+ generationId: 'generation-1',
+ generationCreatedAt: NOW + 1,
+ },
+ }),
+ );
+ });
+
+ it('defers without claiming while the predecessor generation remains active', async () => {
+ const { methods, spies } = resolverMethods();
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => ({ status: 'running' }),
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).rejects.toMatchObject({
+ code: 'PARENT_NOT_READY',
+ retryable: true,
+ deferWithoutAttempt: true,
+ });
+ expect(spies.claimNextAgentQueuedTurn).not.toHaveBeenCalled();
+ });
+
+ it('preserves queued context and settles only after fresh-turn admission', async () => {
+ const { methods, spies } = resolverMethods();
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ const prepared = await resolve(envelope(), { idempotencyKey: 'trigger-1' });
+ expect(prepared).toMatchObject({
+ status: 'ready',
+ input: 'queued words',
+ parentMessageId: 'assistant-2',
+ files: [{ file_id: 'file-1' }],
+ quotes: ['quote'],
+ manualSkills: ['skill-1'],
+ admissionSource: {
+ source: AGENT_QUEUED_TURN_SOURCE,
+ sourceId: 'queued-turn-1',
+ claimId: 'trigger-1',
+ claimBy: 'worker-1',
+ },
+ });
+ if (prepared?.status !== 'ready') {
+ throw new Error('Expected a ready queued turn');
+ }
+ expect(spies.markAgentQueuedTurnAdmitted).not.toHaveBeenCalled();
+ expect(spies.beginAgentQueuedTurnAdmission).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-1',
+ admissionId: 'trigger-1',
+ admissionProtocolVersion: 2,
+ }),
+ );
+ await prepared.settleOnAdmission?.({
+ mode: 'continue',
+ status: 'started',
+ conversationId: 'conversation-1',
+ streamId: 'stream-1',
+ generationCreatedAt: NOW + 1,
+ });
+ expect(spies.markAgentQueuedTurnAdmitted).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-1',
+ admissionId: 'trigger-1',
+ admissionMode: 'ordinary',
+ generationId: 'stream-1',
+ generationCreatedAt: NOW + 1,
+ }),
+ );
+ });
+
+ it('records execution admission through the source-owned lifecycle seam', async () => {
+ const { methods, spies } = resolverMethods();
+ const lifecycle = createAgentQueuedTurnLifecycle({
+ methods,
+ getGenerationJob: async () => null,
+ getGenerationAdmissionEvidence: async () => null,
+ enqueue: async () => ({ deliveryKey: 'queued-delivery-1' }),
+ });
+
+ await expect(
+ lifecycle.recordExecutionAdmission(
+ {
+ source: AGENT_QUEUED_TURN_SOURCE,
+ sourceId: 'queued-turn-1',
+ claimId: 'trigger-1',
+ claimBy: 'worker-1',
+ },
+ {
+ userId: USER_ID,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ clientRequestId: 'trigger-1',
+ generationId: 'conversation-1',
+ generationCreatedAt: NOW + 1,
+ },
+ ),
+ ).resolves.toBe(true);
+ expect(spies.markAgentQueuedTurnAdmitted).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-1',
+ claimId: 'trigger-1',
+ claimBy: 'worker-1',
+ admissionId: 'trigger-1',
+ generationId: 'conversation-1',
+ generationCreatedAt: NOW + 1,
+ }),
+ );
+ });
+
+ it('requires the exact durable source receipt before deduplicated success', async () => {
+ const { methods, spies } = resolverMethods();
+ const lifecycle = createAgentQueuedTurnLifecycle({
+ methods,
+ getGenerationJob: async () => null,
+ getGenerationAdmissionEvidence: async () => null,
+ enqueue: async () => ({ deliveryKey: 'queued-delivery-1' }),
+ });
+ const source = {
+ source: AGENT_QUEUED_TURN_SOURCE,
+ sourceId: 'queued-turn-1',
+ claimId: 'trigger-1',
+ claimBy: 'worker-1',
+ };
+ const admission = {
+ userId: USER_ID,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ clientRequestId: 'trigger-1',
+ generationId: 'conversation-1',
+ generationCreatedAt: NOW + 1,
+ };
+
+ await expect(lifecycle.verifyExecutionAdmission(source, admission)).resolves.toBe(true);
+ expect(spies.hasAgentQueuedTurnAdmissionReceipt).toHaveBeenCalledWith({
+ user: new Types.ObjectId(USER_ID),
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: 'queued-turn-1',
+ admissionId: 'trigger-1',
+ generationId: 'conversation-1',
+ generationCreatedAt: NOW + 1,
+ });
+
+ spies.hasAgentQueuedTurnAdmissionReceipt.mockResolvedValueOnce(false);
+ await expect(lifecycle.verifyExecutionAdmission(source, admission)).rejects.toThrow(
+ 'not yet confirmed',
+ );
+ });
+
+ it('retires cancellation through the same lifecycle interface', async () => {
+ const { methods: resolver } = resolverMethods();
+ const cancelled = {
+ ...claim(),
+ status: 'cancelled' as const,
+ deliveryKey: 'delivery-cancelled',
+ terminalReceipt: {
+ outcome: 'cancelled' as const,
+ settledAt: new Date(NOW),
+ },
+ };
+ const cancelAgentQueuedTurn = jest.fn(async () => ({
+ outcome: 'cancelled' as const,
+ turn: cancelled,
+ }));
+ const markAgentQueuedTurnDeliveryRetired = jest.fn(async () => true);
+ const retireDelivery = jest.fn().mockResolvedValueOnce(false).mockResolvedValueOnce(true);
+ const lifecycle = createAgentQueuedTurnLifecycle({
+ methods: {
+ ...resolver,
+ cancelAgentQueuedTurn,
+ markAgentQueuedTurnDeliveryRetired,
+ },
+ getGenerationJob: async () => null,
+ getGenerationAdmissionEvidence: async () => null,
+ enqueue: async () => ({ deliveryKey: 'unused-delivery' }),
+ retireDelivery,
+ });
+
+ await expect(
+ lifecycle.cancel({
+ user: new Types.ObjectId(USER_ID),
+ tenantId: 'tenant-1',
+ queuedTurnId: 'queued-turn-1',
+ }),
+ ).resolves.toMatchObject({ outcome: 'cancelled' });
+ expect(retireDelivery).toHaveBeenNthCalledWith(
+ 1,
+ 'delivery-cancelled',
+ AGENT_QUEUED_TURN_SOURCE,
+ 'queued_turn_cancelled',
+ );
+ expect(retireDelivery).toHaveBeenNthCalledWith(
+ 2,
+ 'delivery-cancelled',
+ AGENT_QUEUED_TURN_SOURCE,
+ 'queued_turn_cancelled',
+ { onlyIfDead: true },
+ );
+ expect(markAgentQueuedTurnDeliveryRetired).toHaveBeenCalledWith({
+ deliveryKey: 'delivery-cancelled',
+ });
+ });
+
+ it('settles an obsolete-lane delivery without crossing ordinary admission', async () => {
+ const { methods, spies } = resolverMethods();
+ (
+ spies.beginAgentQueuedTurnAdmission as unknown as jest.MockedFunction<
+ AgentQueuedTurnMethods['beginAgentQueuedTurnAdmission']
+ >
+ ).mockResolvedValueOnce({
+ outcome: 'retired',
+ turn: { ...claim(), status: 'cancelled' },
+ });
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toEqual({
+ status: 'settled',
+ });
+ expect(spies.markAgentQueuedTurnAdmitted).not.toHaveBeenCalled();
+ });
+
+ it('uses the latest admitted queued generation as the effective predecessor epoch', async () => {
+ const { methods, spies } = resolverMethods();
+ const claimed = claim();
+ claimed.expectedPredecessorCreatedAt = NOW;
+ spies.claimNextAgentQueuedTurn.mockResolvedValueOnce({ outcome: 'acquired', claim: claimed });
+ spies.getEffectiveAgentQueuedTurnPredecessor.mockResolvedValueOnce(NOW + 250);
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toMatchObject({
+ status: 'ready',
+ expectedPredecessorCreatedAt: NOW + 250,
+ });
+ expect(spies.getEffectiveAgentQueuedTurnPredecessor).toHaveBeenCalledWith({
+ user: new Types.ObjectId(USER_ID),
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ sequence: 1,
+ expectedPredecessorCreatedAt: NOW,
+ });
+ });
+
+ it.each([
+ ['aborted', 'PREDECESSOR_ABORTED'],
+ ['error', 'PREDECESSOR_FAILED'],
+ ] as const)('dead-letters a queued turn whose predecessor is %s', async (status, code) => {
+ const { methods, spies } = resolverMethods();
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => ({ status }),
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toEqual({
+ status: 'settled',
+ });
+ expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ disposition: 'dead',
+ failure: expect.objectContaining({ code }),
+ }),
+ );
+ expect(spies.getMessages).toHaveBeenCalledTimes(1);
+ });
+
+ it.each([
+ [{ unfinished: true }, 'PREDECESSOR_ABORTED'],
+ [{ error: true }, 'PREDECESSOR_FAILED'],
+ ] as const)(
+ 'uses durable response history after the transient predecessor job is cleaned up',
+ async (terminal, code) => {
+ const { methods, spies } = resolverMethods();
+ spies.getMessages.mockResolvedValueOnce([
+ persistedMessage({
+ messageId: 'assistant-1',
+ parentMessageId: 'user-1',
+ isCreatedByUser: false,
+ createdAt: new Date(NOW),
+ ...terminal,
+ }),
+ ]);
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toEqual({
+ status: 'settled',
+ });
+ expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ disposition: 'dead',
+ failure: expect.objectContaining({ code }),
+ }),
+ );
+ },
+ );
+
+ it('allows an explicitly prioritized interrupt successor after its predecessor is unfinished', async () => {
+ const { methods, spies } = resolverMethods();
+ const prioritized = claim();
+ prioritized.priority = true;
+ spies.claimNextAgentQueuedTurn.mockResolvedValueOnce({
+ outcome: 'acquired',
+ claim: prioritized,
+ });
+ spies.getMessages.mockResolvedValueOnce([
+ persistedMessage({
+ messageId: 'assistant-1',
+ parentMessageId: 'user-1',
+ isCreatedByUser: false,
+ createdAt: new Date(NOW),
+ unfinished: true,
+ }),
+ ]);
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).resolves.toMatchObject({
+ status: 'ready',
+ parentMessageId: 'assistant-1',
+ });
+ });
+
+ it('dead-letters the queue row when admission is definitely rejected', async () => {
+ const { methods, spies } = resolverMethods();
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+ const prepared = await resolve(envelope(), { idempotencyKey: 'trigger-1' });
+ if (prepared?.status !== 'ready') {
+ throw new Error('Expected a ready queued turn');
+ }
+ await prepared.releaseOnDefiniteFailure?.(
+ new AgentTriggerExecutionError('forbidden', {
+ mode: 'continue',
+ certainty: 'definite',
+ retryable: false,
+ status: 403,
+ code: 'FORBIDDEN',
+ }),
+ );
+ expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ disposition: 'dead',
+ failure: { code: 'FORBIDDEN', message: 'forbidden' },
+ }),
+ );
+ });
+
+ it('releases a claim when admission defers with PARENT_NOT_READY', async () => {
+ const { methods, spies } = resolverMethods();
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+ const prepared = await resolve(envelope(), { idempotencyKey: 'trigger-1' });
+ if (prepared?.status !== 'ready') {
+ throw new Error('Expected a ready queued turn');
+ }
+ await prepared.releaseOnDefiniteFailure?.(
+ new AgentTriggerExecutionError('parent raced', {
+ mode: 'continue',
+ certainty: 'definite',
+ retryable: true,
+ deferWithoutAttempt: true,
+ status: 409,
+ code: 'PARENT_NOT_READY',
+ }),
+ );
+ expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-1',
+ claimId: 'trigger-1',
+ claimBy: 'worker-1',
+ disposition: 'retry',
+ }),
+ );
+ });
+
+ it('dead-letters a retryable admission rejection on the final delivery attempt', async () => {
+ const { methods, spies } = resolverMethods();
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+ const prepared = await resolve(envelope(), {
+ idempotencyKey: 'trigger-1',
+ attempt: 3,
+ maxAttempts: 3,
+ });
+ if (prepared?.status !== 'ready') {
+ throw new Error('Expected a ready queued turn');
+ }
+
+ await prepared.releaseOnDefiniteFailure?.(
+ new AgentTriggerExecutionError('still busy', {
+ mode: 'continue',
+ certainty: 'definite',
+ retryable: true,
+ status: 503,
+ code: 'ADMISSION_BUSY',
+ }),
+ );
+ expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ disposition: 'dead',
+ failure: { code: 'ADMISSION_BUSY', message: 'still busy' },
+ }),
+ );
+ });
+
+ it('releases an acquired claim when branch preparation is temporarily unavailable', async () => {
+ const { methods, spies } = resolverMethods();
+ spies.getMessages.mockRejectedValueOnce(new Error('read unavailable'));
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).rejects.toMatchObject({
+ code: 'QUEUED_TURN_PREPARATION_UNAVAILABLE',
+ retryable: true,
+ deferWithoutAttempt: true,
+ });
+ expect(spies.releaseAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-1',
+ claimId: 'trigger-1',
+ claimBy: 'worker-1',
+ disposition: 'retry',
+ }),
+ );
+ });
+
+ it('keeps a failed preparation release retryable under the same claim fence', async () => {
+ const { methods, spies } = resolverMethods();
+ spies.getMessages.mockRejectedValueOnce(new Error('read unavailable'));
+ spies.releaseAgentQueuedTurn.mockRejectedValueOnce(new Error('write unavailable'));
+ const resolve = createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ claimBy: 'worker-1',
+ });
+
+ await expect(resolve(envelope(), { idempotencyKey: 'trigger-1' })).rejects.toMatchObject({
+ code: 'QUEUED_TURN_PREPARATION_RELEASE_FAILED',
+ retryable: true,
+ deferWithoutAttempt: true,
+ });
+ });
+
+ it('uses a replica-unique default claim owner that remains stable for the process', async () => {
+ const first = resolverMethods();
+ const second = resolverMethods();
+ const firstResolve = createAgentQueuedTurnResolver({
+ methods: first.methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ });
+ const secondResolve = createAgentQueuedTurnResolver({
+ methods: second.methods,
+ getGenerationJob: async () => null,
+ now: () => NOW,
+ });
+
+ await firstResolve(envelope(), { idempotencyKey: 'trigger-1' });
+ await secondResolve(envelope(), { idempotencyKey: 'trigger-1' });
+ const firstOwner = first.spies.claimNextAgentQueuedTurn.mock.calls[0][0].claimBy;
+ const secondOwner = second.spies.claimNextAgentQueuedTurn.mock.calls[0][0].claimBy;
+ expect(firstOwner).toMatch(/^agent-queued-turn:\d+:[0-9a-f-]{36}$/);
+ expect(secondOwner).toBe(firstOwner);
+ });
+});
+
+describe('Agent queued-turn delivery scheduling', () => {
+ function queuedTurn(id: string, sequence: number): AgentQueuedTurnRecord {
+ const { claimId: _claimId, claimBy: _claimBy, claimUntil: _claimUntil, ...record } = claim();
+ return {
+ ...record,
+ queuedTurnId: id,
+ sequence,
+ status: 'queued',
+ };
+ }
+
+ it('uses independent delivery lanes so publication order cannot invert queue order', async () => {
+ const enqueue = jest.fn(async (value: unknown) => ({
+ deliveryKey: getAgentTriggerIdempotencyKey(value as AgentContinueTriggerEnvelope),
+ }));
+ const reserveAgentQueuedTurnDelivery = jest.fn(async (input) => ({
+ outcome: 'reserved' as const,
+ turn: { ...queuedTurn(input.queuedTurnId, 1), deliveryKey: input.deliveryKey },
+ }));
+ const markQueuedTurnScheduled = jest.fn(async (input) => ({
+ outcome: 'scheduled' as const,
+ turn: queuedTurn(input.queuedTurnId, 1),
+ }));
+ const scheduler = createAgentQueuedTurnScheduler({
+ methods: {
+ reserveAgentQueuedTurnDelivery,
+ markQueuedTurnScheduled,
+ } as unknown as AgentQueuedTurnMethods,
+ enqueue,
+ getGenerationAdmissionEvidence: async () => null,
+ });
+
+ await scheduler.schedule(queuedTurn('queued-turn-2', 2));
+ await scheduler.schedule(queuedTurn('queued-turn-1', 1));
+
+ expect(enqueue).toHaveBeenNthCalledWith(
+ 1,
+ expect.anything(),
+ expect.objectContaining({ orderingKey: 'agent-queued-turn-delivery:queued-turn-2' }),
+ );
+ expect(enqueue).toHaveBeenNthCalledWith(
+ 2,
+ expect.anything(),
+ expect.objectContaining({ orderingKey: 'agent-queued-turn-delivery:queued-turn-1' }),
+ );
+ expect(reserveAgentQueuedTurnDelivery).toHaveBeenCalledTimes(2);
+ expect(reserveAgentQueuedTurnDelivery.mock.invocationCallOrder[0]).toBeLessThan(
+ enqueue.mock.invocationCallOrder[0],
+ );
+ expect(reserveAgentQueuedTurnDelivery.mock.calls[0]?.[0].deliveryKey).toBe(
+ getAgentTriggerIdempotencyKey(enqueue.mock.calls[0]?.[0] as AgentContinueTriggerEnvelope),
+ );
+ });
+
+ it('reconciles quarantined admission receipts during durable recovery', async () => {
+ const turn = {
+ ...queuedTurn('queued-turn-indeterminate', 1),
+ status: 'dead' as const,
+ deliveryKey: 'delivery-indeterminate',
+ deliveryState: 'published' as const,
+ admissionId: 'delivery-indeterminate',
+ admissionStartedAt: new Date(NOW - 1_000),
+ reconciliationClaimId: 'reconciliation-claim',
+ reconciliationClaimBy: 'reconciler-1',
+ terminalReceipt: {
+ outcome: 'dead' as const,
+ settledAt: new Date(NOW),
+ failure: {
+ code: 'ADMISSION_INDETERMINATE',
+ message: 'The queued turn may have been admitted',
+ },
+ },
+ };
+ const deadLetterAgentQueuedTurn = jest.fn(async () => ({
+ outcome: 'admission_reconciled' as const,
+ turn: { ...turn, status: 'admitted' as const },
+ }));
+ const getGenerationAdmissionEvidence = jest.fn(async () => ({
+ generationId: 'conversation-1',
+ generationCreatedAt: NOW + 1,
+ }));
+ const scheduler = createAgentQueuedTurnScheduler({
+ methods: {
+ findQueuedTurnsNeedingDelivery: jest.fn(async () => []),
+ claimQueuedTurnsForAdmissionReconciliation: jest.fn(async (input) => [
+ {
+ ...turn,
+ reconciliationClaimId: input.claimId,
+ reconciliationClaimBy: input.claimBy,
+ },
+ ]),
+ deadLetterAgentQueuedTurn,
+ } as unknown as AgentQueuedTurnMethods,
+ enqueue: jest.fn(),
+ getGenerationAdmissionEvidence,
+ });
+
+ await expect(scheduler.recover()).resolves.toBe(1);
+ expect(getGenerationAdmissionEvidence).toHaveBeenCalledWith(
+ USER_ID,
+ 'delivery-indeterminate',
+ 'conversation-1',
+ 'conversation-1',
+ );
+ expect(deadLetterAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-indeterminate',
+ deliveryKey: 'delivery-indeterminate',
+ admissionEvidence: {
+ generationId: 'conversation-1',
+ generationCreatedAt: NOW + 1,
+ },
+ }),
+ );
+ });
+
+ it('rotates source-fenced ambiguity without trusting transient job-store evidence', async () => {
+ const turn = {
+ ...queuedTurn('queued-turn-source-fenced', 1),
+ status: 'dead' as const,
+ deliveryKey: 'delivery-source-fenced',
+ deliveryState: 'published' as const,
+ admissionId: 'delivery-source-fenced',
+ admissionStartedAt: new Date(NOW - 1_000),
+ admissionProtocolVersion: 2 as const,
+ terminalReceipt: {
+ outcome: 'dead' as const,
+ settledAt: new Date(NOW),
+ failure: {
+ code: 'ADMISSION_INDETERMINATE',
+ message: 'The queued turn may have been admitted',
+ },
+ },
+ };
+ const deferAgentQueuedTurnAdmissionReconciliation = jest.fn(async () => true);
+ const getGenerationAdmissionEvidence = jest.fn(async () => ({
+ generationId: 'stale-generation',
+ generationCreatedAt: NOW,
+ }));
+ const scheduler = createAgentQueuedTurnScheduler({
+ methods: {
+ findQueuedTurnsNeedingDelivery: jest.fn(async () => []),
+ claimQueuedTurnsForAdmissionReconciliation: jest.fn(async (input) => [
+ {
+ ...turn,
+ reconciliationClaimId: input.claimId,
+ reconciliationClaimBy: input.claimBy,
+ },
+ ]),
+ deferAgentQueuedTurnAdmissionReconciliation,
+ } as unknown as AgentQueuedTurnMethods,
+ enqueue: jest.fn(),
+ getGenerationAdmissionEvidence,
+ });
+
+ await expect(scheduler.recover()).resolves.toBe(0);
+ expect(getGenerationAdmissionEvidence).not.toHaveBeenCalled();
+ expect(deferAgentQueuedTurnAdmissionReconciliation).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-source-fenced',
+ deliveryKey: 'delivery-source-fenced',
+ }),
+ );
+ });
+
+ it('quarantines an expired provider-admission owner before any replay', async () => {
+ const turn = {
+ ...queuedTurn('queued-turn-owner-lost', 1),
+ status: 'claimed' as const,
+ claimId: 'delivery-owner-lost',
+ claimBy: 'worker-gone',
+ claimUntil: new Date(NOW - 1),
+ deliveryKey: 'delivery-owner-lost',
+ deliveryState: 'published' as const,
+ admissionId: 'delivery-owner-lost',
+ admissionStartedAt: new Date(NOW - 1_000),
+ admissionProtocolVersion: 2 as const,
+ };
+ const deadLetterAgentQueuedTurn = jest.fn(async () => ({
+ outcome: 'admission_indeterminate' as const,
+ turn: {
+ ...turn,
+ status: 'dead' as const,
+ terminalReceipt: {
+ outcome: 'dead' as const,
+ settledAt: new Date(NOW),
+ failure: { code: 'ADMISSION_INDETERMINATE', message: 'owner disappeared' },
+ },
+ },
+ }));
+ const getGenerationAdmissionEvidence = jest.fn(async () => null);
+ const scheduler = createAgentQueuedTurnScheduler({
+ methods: {
+ findQueuedTurnsNeedingDelivery: jest.fn(async () => []),
+ claimQueuedTurnsForAdmissionReconciliation: jest.fn(async (input) => [
+ {
+ ...turn,
+ reconciliationClaimId: input.claimId,
+ reconciliationClaimBy: input.claimBy,
+ },
+ ]),
+ deadLetterAgentQueuedTurn,
+ } as unknown as AgentQueuedTurnMethods,
+ enqueue: jest.fn(),
+ getGenerationAdmissionEvidence,
+ });
+
+ await expect(scheduler.recover()).resolves.toBe(1);
+ expect(getGenerationAdmissionEvidence).not.toHaveBeenCalled();
+ expect(deadLetterAgentQueuedTurn).toHaveBeenCalledWith(
+ expect.objectContaining({
+ queuedTurnId: 'queued-turn-owner-lost',
+ deliveryKey: 'delivery-owner-lost',
+ failure: expect.objectContaining({ code: 'ADMISSION_INDETERMINATE' }),
+ }),
+ );
+ });
+});
diff --git a/packages/api/src/agents/queuedTurns.ts b/packages/api/src/agents/queuedTurns.ts
new file mode 100644
index 0000000000..4a9fafa26a
--- /dev/null
+++ b/packages/api/src/agents/queuedTurns.ts
@@ -0,0 +1,1008 @@
+import { Types } from 'mongoose';
+import { randomUUID } from 'node:crypto';
+import {
+ AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
+ logger,
+ runAsSystem,
+} from '@librechat/data-schemas';
+import type {
+ AgentQueuedTurnClaim,
+ AgentQueuedTurnMethods,
+ AgentQueuedTurnRecord,
+ ConversationMethods,
+ IMessage,
+ MessageMethods,
+} from '@librechat/data-schemas';
+import type {
+ AgentContinuationAdmissionSource,
+ AgentTriggerExecutionHostDeps,
+} from './triggers/host';
+import type { AgentContinueTriggerEnvelope, AgentTriggerEnvelope } from './triggers/envelope';
+import type { AgentTriggerEnqueueOptions } from './triggers/delivery';
+import type { AgentTriggerDeliveryFailure } from './triggers/engine';
+import { getAgentTriggerIdempotencyKey, parseAgentTriggerEnvelope } from './triggers/envelope';
+import { createAgentTriggerEnvelope } from './triggers/envelope';
+import { AgentTriggerExecutionError } from './triggers/host';
+
+export const AGENT_QUEUED_TURN_SOURCE = 'agent-queued-turn';
+const AGENT_QUEUED_TURN_EVENT = 'agent.queued-turn';
+const MESSAGE_SELECT = 'messageId parentMessageId isCreatedByUser createdAt unfinished error';
+const CLAIM_LEASE_MS = 2 * 60 * 1000;
+const RECONCILIATION_LEASE_MS = 2 * 60 * 1000;
+const RECONCILIATION_BACKOFF_BASE_MS = 5_000;
+const RECONCILIATION_BACKOFF_MAX_MS = 5 * 60 * 1000;
+const DEFAULT_RECOVERY_INTERVAL_MS = 30_000;
+const DEFAULT_RECOVERY_LIMIT = 100;
+const MAX_FAILURE_CODE_LENGTH = 128;
+const MAX_FAILURE_MESSAGE_LENGTH = 2048;
+/** PID is commonly identical across container replicas. Keep claim ownership
+ * stable within this process while fencing every other process instance. */
+const PROCESS_CLAIM_OWNER = `agent-queued-turn:${process.pid}:${randomUUID()}`;
+
+interface GenerationState {
+ streamId?: unknown;
+ status?: unknown;
+ createdAt?: unknown;
+ error?: unknown;
+ metadata?: {
+ idempotencyClientRequestId?: unknown;
+ terminalPersistencePending?: unknown;
+ };
+}
+
+interface GenerationAdmissionEvidence {
+ generationId: string;
+ generationCreatedAt: number;
+}
+
+type GetGenerationAdmissionEvidence = (
+ userId: string,
+ clientRequestId: string,
+ streamId: string,
+ conversationId?: string,
+) => Promise<GenerationAdmissionEvidence | null>;
+
+type QueuedTurnResolverMethods = AgentQueuedTurnMethods &
+ Pick<ConversationMethods, 'getConvo'> &
+ Pick<MessageMethods, 'getMessages'>;
+
+export interface AgentQueuedTurnResolverDeps {
+ methods: QueuedTurnResolverMethods;
+ getGenerationJob: (conversationId: string) => Promise<GenerationState | null>;
+ now?: () => number;
+ claimBy?: string;
+}
+
+export type EnqueueAgentQueuedTurnDelivery = (
+ envelope: unknown,
+ options?: AgentTriggerEnqueueOptions,
+) => Promise<{ deliveryKey: string }>;
+
+export interface AgentQueuedTurnSchedulerDeps {
+ methods: AgentQueuedTurnMethods;
+ enqueue: EnqueueAgentQueuedTurnDelivery;
+ getGenerationAdmissionEvidence: GetGenerationAdmissionEvidence;
+ recoveryIntervalMs?: number;
+ recoveryLimit?: number;
+}
+
+export interface AgentQueuedTurnScheduler {
+ initialize: () => Promise<void>;
+ stop: () => Promise<void>;
+ schedule: (turn: AgentQueuedTurnRecord) => Promise<string>;
+ recover: () => Promise<number>;
+}
+
+export interface AgentQueuedTurnLifecycle {
+ prepareContinue: NonNullable<AgentTriggerExecutionHostDeps['prepareContinue']>;
+ settleBeforeDeadLetter: (
+ rawEnvelope: unknown,
+ failure: AgentTriggerDeliveryFailure,
+ ) => Promise<void>;
+ recordExecutionAdmission: (
+ rawSource: unknown,
+ input: AgentQueuedTurnExecutionAdmission,
+ ) => Promise<boolean>;
+ verifyExecutionAdmission: (
+ rawSource: unknown,
+ input: AgentQueuedTurnExecutionAdmission,
+ ) => Promise<boolean>;
+ initialize: () => Promise<void>;
+ stop: () => Promise<void>;
+ schedule: (turn: AgentQueuedTurnRecord) => Promise<string>;
+ cancel: (
+ input: Parameters<AgentQueuedTurnMethods['cancelAgentQueuedTurn']>[0],
+ ) => ReturnType<AgentQueuedTurnMethods['cancelAgentQueuedTurn']>;
+ recover: () => Promise<number>;
+}
+
+export interface AgentQueuedTurnLifecycleDeps {
+ methods: QueuedTurnResolverMethods;
+ getGenerationJob: AgentQueuedTurnResolverDeps['getGenerationJob'];
+ getGenerationAdmissionEvidence: GetGenerationAdmissionEvidence;
+ enqueue: EnqueueAgentQueuedTurnDelivery;
+ retireDelivery?: (
+ deliveryKey: string,
+ sourceId: string,
+ reason: string,
+ options?: { onlyIfDead?: boolean },
+ ) => Promise<boolean>;
+ getDelivery?: (deliveryKey: string) => Promise<unknown | null>;
+ now?: () => number;
+ claimBy?: string;
+ recoveryIntervalMs?: number;
+ recoveryLimit?: number;
+}
+
+export interface AgentQueuedTurnExecutionAdmission {
+ userId: string;
+ tenantId?: string;
+ conversationId: string;
+ clientRequestId: string;
+ generationId: string;
+ generationCreatedAt: number;
+}
+
+function parseQueuedTurnAdmissionSource(raw: unknown): AgentContinuationAdmissionSource | null {
+ if (raw == null || typeof raw !== 'object' || Array.isArray(raw)) {
+ return null;
+ }
+ if (!('source' in raw) || raw.source !== AGENT_QUEUED_TURN_SOURCE) {
+ return null;
+ }
+ const sourceId = 'sourceId' in raw ? raw.sourceId : undefined;
+ const claimId = 'claimId' in raw ? raw.claimId : undefined;
+ const claimBy = 'claimBy' in raw ? raw.claimBy : undefined;
+ if (
+ typeof sourceId !== 'string' ||
+ sourceId.length === 0 ||
+ sourceId.length > 128 ||
+ typeof claimId !== 'string' ||
+ claimId.length === 0 ||
+ claimId.length > 128 ||
+ typeof claimBy !== 'string' ||
+ claimBy.length === 0 ||
+ claimBy.length > 256
+ ) {
+ throw new TypeError('Agent queued turn admission source is invalid');
+ }
+ return { source: AGENT_QUEUED_TURN_SOURCE, sourceId, claimId, claimBy };
+}
+
+/** Commits the source-owned admission receipt only after the controller has
+ * invoked the provider implementation. A process death before that boundary
+ * leaves the source nonterminal for fail-closed reconciliation. */
+async function settleAgentQueuedTurnExecutionAdmission(
+ rawSource: unknown,
+ input: AgentQueuedTurnExecutionAdmission,
+ methods: Pick<AgentQueuedTurnMethods, 'markAgentQueuedTurnAdmitted'>,
+): Promise<boolean> {
+ const source = parseQueuedTurnAdmissionSource(rawSource);
+ if (source == null) {
+ return false;
+ }
+ if (!Types.ObjectId.isValid(input.userId)) {
+ throw new TypeError('Agent queued turn admission principal is invalid');
+ }
+ const settled = await methods.markAgentQueuedTurnAdmitted({
+ user: new Types.ObjectId(input.userId),
+ ...(input.tenantId != null && { tenantId: input.tenantId }),
+ conversationId: input.conversationId,
+ queuedTurnId: source.sourceId,
+ claimId: source.claimId,
+ claimBy: source.claimBy,
+ admissionId: input.clientRequestId,
+ admissionMode: 'ordinary',
+ generationId: input.generationId,
+ generationCreatedAt: input.generationCreatedAt,
+ settledAt: new Date(),
+ });
+ if (settled.outcome === 'conflict') {
+ throw new Error('The queued turn execution admission could not be committed');
+ }
+ return true;
+}
+
+/** Deduplicated HTTP success is valid only when the original controller
+ * already crossed the provider-invocation boundary and committed its exact
+ * source receipt. Never manufacture that receipt from transient job state. */
+async function verifyAgentQueuedTurnExecutionAdmission(
+ rawSource: unknown,
+ input: AgentQueuedTurnExecutionAdmission,
+ methods: Pick<AgentQueuedTurnMethods, 'hasAgentQueuedTurnAdmissionReceipt'>,
+): Promise<boolean> {
+ const source = parseQueuedTurnAdmissionSource(rawSource);
+ if (source == null) {
+ return false;
+ }
+ if (!Types.ObjectId.isValid(input.userId)) {
+ throw new TypeError('Agent queued turn admission principal is invalid');
+ }
+ const confirmed = await methods.hasAgentQueuedTurnAdmissionReceipt({
+ user: new Types.ObjectId(input.userId),
+ ...(input.tenantId != null && { tenantId: input.tenantId }),
+ conversationId: input.conversationId,
+ queuedTurnId: source.sourceId,
+ admissionId: input.clientRequestId,
+ generationId: input.generationId,
+ generationCreatedAt: input.generationCreatedAt,
+ });
+ if (!confirmed) {
+ throw new Error('The queued turn execution admission is not yet confirmed');
+ }
+ return true;
+}
+
+function createAgentQueuedTurnDeadLetterSettlement({
+ methods,
+ getGenerationAdmissionEvidence,
+ now = Date.now,
+}: {
+ methods: Pick<AgentQueuedTurnMethods, 'deadLetterAgentQueuedTurn'>;
+ getGenerationAdmissionEvidence?: GetGenerationAdmissionEvidence;
+ now?: () => number;
+}) {
+ return async (rawEnvelope: unknown, failure: AgentTriggerDeliveryFailure): Promise<void> => {
+ let envelope: AgentTriggerEnvelope;
+ try {
+ envelope = parseAgentTriggerEnvelope(rawEnvelope);
+ } catch {
+ return;
+ }
+ if (envelope.mode !== 'continue') {
+ return;
+ }
+ const queuedTurnId = payloadQueuedTurnId(envelope);
+ if (queuedTurnId === undefined) {
+ return;
+ }
+ if (queuedTurnId === null || !Types.ObjectId.isValid(envelope.principal.userId)) {
+ return;
+ }
+ const deliveryKey = getAgentTriggerIdempotencyKey(envelope);
+ const admissionEvidence = await getGenerationAdmissionEvidence?.(
+ envelope.principal.userId,
+ deliveryKey,
+ envelope.target.conversationId,
+ envelope.target.conversationId,
+ );
+ const settled = await methods.deadLetterAgentQueuedTurn({
+ user: new Types.ObjectId(envelope.principal.userId),
+ ...(envelope.principal.tenantId != null && { tenantId: envelope.principal.tenantId }),
+ conversationId: envelope.target.conversationId,
+ queuedTurnId,
+ deliveryKey,
+ settledAt: new Date(now()),
+ failure: normalizeFailure(failure.code, failure.message),
+ ...(admissionEvidence != null && { admissionEvidence }),
+ });
+ if (settled.outcome === 'conflict') {
+ throw new Error('Queued turn delivery no longer owns its source row');
+ }
+ };
+}
+
+function executionError(
+ message: string,
+ options: {
+ code: string;
+ retryable: boolean;
+ deferWithoutAttempt?: boolean;
+ status?: number;
+ retryAfter?: string;
+ },
+): AgentTriggerExecutionError {
+ return new AgentTriggerExecutionError(message, {
+ mode: 'continue',
+ certainty: 'definite',
+ ...options,
+ });
+}
+
+function isGenerationActive(job: GenerationState | null): boolean {
+ return (
+ job?.status === 'running' ||
+ job?.status === 'requires_action' ||
+ job?.metadata?.terminalPersistencePending === true
+ );
+}
+
+function sameTenant(actual: unknown, expected: string | undefined): boolean {
+ return expected == null ? actual == null : actual === expected;
+}
+
+function timestamp(message: IMessage): number {
+ const value = message.createdAt;
+ if (value instanceof Date) {
+ return value.getTime();
+ }
+ const parsed = typeof value === 'string' || typeof value === 'number' ? new Date(value) : null;
+ return parsed != null && Number.isFinite(parsed.getTime()) ? parsed.getTime() : 0;
+}
+
+/** Finds the current assistant leaf on the exact visible branch captured when
+ * the turn was queued. Unrelated branch activity can never retarget the turn. */
+function latestAssistantDescendant(messages: IMessage[], anchorId: string): IMessage | undefined {
+ const byId = new Map(messages.map((message) => [message.messageId, message]));
+ if (!byId.has(anchorId)) {
+ return;
+ }
+ const memo = new Map<string, boolean>([[anchorId, true]]);
+ const reachesAnchor = (message: IMessage, visiting = new Set<string>()): boolean => {
+ const known = memo.get(message.messageId);
+ if (known != null) {
+ return known;
+ }
+ if (visiting.has(message.messageId)) {
+ memo.set(message.messageId, false);
+ return false;
+ }
+ visiting.add(message.messageId);
+ const parent =
+ typeof message.parentMessageId === 'string' ? byId.get(message.parentMessageId) : undefined;
+ const reachable = parent != null && reachesAnchor(parent, visiting);
+ visiting.delete(message.messageId);
+ memo.set(message.messageId, reachable);
+ return reachable;
+ };
+ const descendants = messages
+ .filter((message) => message.isCreatedByUser === false && reachesAnchor(message))
+ .sort((left, right) => {
+ const time = timestamp(left) - timestamp(right);
+ return time === 0 ? left.messageId.localeCompare(right.messageId) : time;
+ });
+ return descendants[descendants.length - 1];
+}
+
+function boundedFailureValue(value: string, fallback: string, maxLength: number): string {
+ const normalized = value.trim();
+ return (normalized.length === 0 ? fallback : normalized).slice(0, maxLength);
+}
+
+function normalizeFailure(code: string, message: string): { code: string; message: string } {
+ return {
+ code: boundedFailureValue(code, 'DELIVERY_FAILED', MAX_FAILURE_CODE_LENGTH),
+ message: boundedFailureValue(
+ message,
+ 'Queued turn delivery failed',
+ MAX_FAILURE_MESSAGE_LENGTH,
+ ),
+ };
+}
+
+function reconciliationBackoff(attempts: number | undefined): number {
+ const exponent = Math.max(0, Math.min((attempts ?? 1) - 1, 6));
+ return Math.min(RECONCILIATION_BACKOFF_BASE_MS * 2 ** exponent, RECONCILIATION_BACKOFF_MAX_MS);
+}
+
+function payloadQueuedTurnId(envelope: AgentContinueTriggerEnvelope): string | null | undefined {
+ if (
+ envelope.event.source.type !== 'internal' ||
+ envelope.event.source.id !== AGENT_QUEUED_TURN_SOURCE
+ ) {
+ return;
+ }
+ const payload = envelope.event.payload;
+ if (payload == null || typeof payload !== 'object' || Array.isArray(payload)) {
+ return null;
+ }
+ const value = (payload as Record<string, unknown>).queuedTurnId;
+ return typeof value === 'string' && value.length > 0 ? value : null;
+}
+
+function releaseClaim(
+ methods: AgentQueuedTurnMethods,
+ envelope: AgentContinueTriggerEnvelope,
+ claim: AgentQueuedTurnClaim,
+): Promise<unknown> {
+ return methods.releaseAgentQueuedTurn({
+ user: claim.user,
+ ...(claim.tenantId != null && { tenantId: claim.tenantId }),
+ conversationId: envelope.target.conversationId,
+ queuedTurnId: claim.queuedTurnId,
+ claimId: claim.claimId,
+ claimBy: claim.claimBy,
+ disposition: 'retry',
+ availableAt: new Date(),
+ });
+}
+
+function deadClaim(
+ methods: AgentQueuedTurnMethods,
+ envelope: AgentContinueTriggerEnvelope,
+ claim: AgentQueuedTurnClaim,
+ code: string,
+ message: string,
+): Promise<unknown> {
+ return methods.releaseAgentQueuedTurn({
+ user: claim.user,
+ ...(claim.tenantId != null && { tenantId: claim.tenantId }),
+ conversationId: envelope.target.conversationId,
+ queuedTurnId: claim.queuedTurnId,
+ claimId: claim.claimId,
+ claimBy: claim.claimBy,
+ disposition: 'dead',
+ settledAt: new Date(),
+ failure: normalizeFailure(code, message),
+ });
+}
+
+/** Resolves one queued message into a fresh ordinary Agent turn. The durable
+ * claim remains owned by the queue record until generation admission is known. */
+function createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob,
+ now = Date.now,
+ claimBy = PROCESS_CLAIM_OWNER,
+}: AgentQueuedTurnResolverDeps): NonNullable<AgentTriggerExecutionHostDeps['prepareContinue']> {
+ return async (envelope, context) => {
+ const queuedTurnId = payloadQueuedTurnId(envelope);
+ if (queuedTurnId === undefined) {
+ return;
+ }
+ if (queuedTurnId === null) {
+ throw executionError('The queued turn payload is invalid.', {
+ code: 'INVALID_QUEUED_TURN',
+ retryable: false,
+ });
+ }
+
+ let generation: GenerationState | null;
+ try {
+ generation = await getGenerationJob(envelope.target.conversationId);
+ } catch (error) {
+ throw executionError(
+ `Parent generation state is temporarily unavailable: ${
+ error instanceof Error ? error.message : String(error)
+ }`,
+ { code: 'PARENT_STATE_UNAVAILABLE', retryable: true, deferWithoutAttempt: true },
+ );
+ }
+ if (
+ isGenerationActive(generation) &&
+ generation?.metadata?.idempotencyClientRequestId !== context.idempotencyKey
+ ) {
+ throw executionError('The preceding generation has not settled yet.', {
+ code: 'PARENT_NOT_READY',
+ retryable: true,
+ status: 409,
+ retryAfter: '1',
+ deferWithoutAttempt: true,
+ });
+ }
+
+ const userId = envelope.principal.userId;
+ const conversation = await methods.getConvo(userId, envelope.target.conversationId);
+ if (
+ conversation == null ||
+ !sameTenant(conversation.tenantId, envelope.principal.tenantId) ||
+ conversation.agent_id !== envelope.target.agentId
+ ) {
+ throw executionError('The queued turn conversation is no longer available.', {
+ code: 'PARENT_NOT_FOUND',
+ retryable: false,
+ status: 404,
+ });
+ }
+
+ const claimNow = new Date(now());
+ const claimed = await methods.claimNextAgentQueuedTurn({
+ user: new Types.ObjectId(userId),
+ ...(envelope.principal.tenantId != null && {
+ tenantId: envelope.principal.tenantId,
+ }),
+ conversationId: envelope.target.conversationId,
+ queuedTurnId,
+ claimId: context.idempotencyKey,
+ claimBy,
+ now: claimNow,
+ leaseUntil: new Date(claimNow.getTime() + CLAIM_LEASE_MS),
+ });
+ if (claimed.outcome === 'missing') {
+ return { status: 'settled' };
+ }
+ if (claimed.outcome === 'blocked') {
+ throw executionError('An earlier queued turn has not settled yet.', {
+ code: 'QUEUED_TURN_BLOCKED',
+ retryable: true,
+ status: 409,
+ retryAfter: '1',
+ deferWithoutAttempt: true,
+ });
+ }
+
+ const claim = claimed.claim;
+ let messages: IMessage[];
+ try {
+ messages = await methods.getMessages(
+ { user: userId, conversationId: envelope.target.conversationId },
+ MESSAGE_SELECT,
+ { sort: { createdAt: 1, _id: 1 } },
+ );
+ } catch (error) {
+ try {
+ await releaseClaim(methods, envelope, claim);
+ } catch (releaseError) {
+ throw executionError(
+ `The queued turn branch read failed and its claim could not be released: ${
+ releaseError instanceof Error ? releaseError.message : String(releaseError)
+ }`,
+ {
+ code: 'QUEUED_TURN_PREPARATION_RELEASE_FAILED',
+ retryable: true,
+ deferWithoutAttempt: true,
+ },
+ );
+ }
+ throw executionError(
+ `The queued turn branch is temporarily unavailable: ${
+ error instanceof Error ? error.message : String(error)
+ }`,
+ {
+ code: 'QUEUED_TURN_PREPARATION_UNAVAILABLE',
+ retryable: true,
+ deferWithoutAttempt: true,
+ },
+ );
+ }
+ const parentMessage = latestAssistantDescendant(messages, claim.parentMessageId);
+ if (parentMessage == null) {
+ await deadClaim(
+ methods,
+ envelope,
+ claim,
+ 'PARENT_NOT_FOUND',
+ 'The queued turn branch is no longer available.',
+ );
+ throw executionError('The queued turn branch is no longer available.', {
+ code: 'PARENT_NOT_FOUND',
+ retryable: false,
+ status: 404,
+ });
+ }
+ const predecessorFailed = generation?.status === 'error' || parentMessage.error === true;
+ const predecessorAborted =
+ !predecessorFailed &&
+ (generation?.status === 'aborted' ||
+ (claim.priority !== true && parentMessage.unfinished === true));
+ if (predecessorAborted || predecessorFailed) {
+ await deadClaim(
+ methods,
+ envelope,
+ claim,
+ predecessorAborted ? 'PREDECESSOR_ABORTED' : 'PREDECESSOR_FAILED',
+ predecessorAborted
+ ? 'The preceding generation was aborted. Review this turn before sending it.'
+ : 'The preceding generation failed. Review this turn before sending it.',
+ );
+ return { status: 'settled' };
+ }
+
+ const effectivePredecessorCreatedAt =
+ (await methods.getEffectiveAgentQueuedTurnPredecessor({
+ user: claim.user,
+ ...(claim.tenantId != null && { tenantId: claim.tenantId }),
+ conversationId: claim.conversationId,
+ sequence: claim.sequence,
+ ...(claim.expectedPredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: claim.expectedPredecessorCreatedAt,
+ }),
+ })) ?? claim.expectedPredecessorCreatedAt;
+
+ const admission = await methods.beginAgentQueuedTurnAdmission({
+ user: claim.user,
+ ...(claim.tenantId != null && { tenantId: claim.tenantId }),
+ conversationId: claim.conversationId,
+ queuedTurnId: claim.queuedTurnId,
+ claimId: claim.claimId,
+ claimBy: claim.claimBy,
+ admissionId: context.idempotencyKey,
+ startedAt: new Date(now()),
+ admissionProtocolVersion: 2,
+ });
+ if (admission.outcome === 'conflict') {
+ throw executionError('The queued turn admission fence is no longer owned.', {
+ code: 'QUEUED_TURN_ADMISSION_FENCE_LOST',
+ retryable: true,
+ deferWithoutAttempt: true,
+ });
+ }
+ if (admission.outcome === 'retired') {
+ return { status: 'settled' };
+ }
+
+ return {
+ status: 'ready',
+ input: claim.text,
+ parentMessageId: parentMessage.messageId,
+ ...(effectivePredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: effectivePredecessorCreatedAt,
+ }),
+ ...(claim.files != null && { files: claim.files }),
+ ...(claim.quotes != null && { quotes: claim.quotes }),
+ ...(claim.manualSkills != null && { manualSkills: claim.manualSkills }),
+ admissionSource: {
+ source: AGENT_QUEUED_TURN_SOURCE,
+ sourceId: claim.queuedTurnId,
+ claimId: claim.claimId,
+ claimBy: claim.claimBy,
+ },
+ releaseOnDefiniteFailure: async (error) => {
+ if (
+ error?.retryable === false ||
+ (context.attempt != null &&
+ context.maxAttempts != null &&
+ context.attempt >= context.maxAttempts)
+ ) {
+ await deadClaim(
+ methods,
+ envelope,
+ claim,
+ error?.code ?? 'ADMISSION_REJECTED',
+ error?.message ?? 'Queued turn admission failed',
+ );
+ return;
+ }
+ await releaseClaim(methods, envelope, claim);
+ },
+ settleOnAdmission: async (result) => {
+ const settled = await methods.markAgentQueuedTurnAdmitted({
+ user: claim.user,
+ ...(claim.tenantId != null && { tenantId: claim.tenantId }),
+ conversationId: claim.conversationId,
+ queuedTurnId: claim.queuedTurnId,
+ claimId: claim.claimId,
+ claimBy: claim.claimBy,
+ admissionId: context.idempotencyKey,
+ admissionMode: 'ordinary',
+ ...(result.streamId != null && { generationId: result.streamId }),
+ ...(result.generationCreatedAt != null && {
+ generationCreatedAt: result.generationCreatedAt,
+ }),
+ settledAt: new Date(now()),
+ });
+ if (settled.outcome === 'conflict') {
+ throw new Error('The queued turn admission could not be committed');
+ }
+ },
+ };
+ };
+}
+
+function deliveryEnvelope(turn: AgentQueuedTurnRecord) {
+ const occurredAt = turn.createdAt.getTime();
+ return createAgentTriggerEnvelope({
+ mode: 'continue',
+ requestId: randomUUID(),
+ deliveryId: turn.queuedTurnId,
+ receivedAt: Date.now(),
+ principal: {
+ id: turn.user.toString(),
+ ...(turn.tenantId != null && { tenantId: turn.tenantId }),
+ },
+ event: {
+ id: turn.queuedTurnId,
+ type: AGENT_QUEUED_TURN_EVENT,
+ occurredAt,
+ source: { id: AGENT_QUEUED_TURN_SOURCE, type: 'internal' },
+ payload: { queuedTurnId: turn.queuedTurnId },
+ },
+ input: turn.text,
+ target: {
+ agentId: turn.agentId,
+ conversationId: turn.conversationId,
+ parentMessageId: turn.parentMessageId,
+ },
+ });
+}
+
+/** Repairs the intentional record-first outbox seam by replaying a stable
+ * delivery identity until the queue row records the scheduling receipt. */
+function createAgentQueuedTurnScheduler({
+ methods,
+ enqueue,
+ getGenerationAdmissionEvidence,
+ recoveryIntervalMs = DEFAULT_RECOVERY_INTERVAL_MS,
+ recoveryLimit = DEFAULT_RECOVERY_LIMIT,
+}: AgentQueuedTurnSchedulerDeps): AgentQueuedTurnScheduler {
+ let timer: NodeJS.Timeout | undefined;
+ let recovery: Promise<number> | undefined;
+
+ const schedule = async (turn: AgentQueuedTurnRecord): Promise<string> => {
+ const envelope = deliveryEnvelope(turn);
+ const deliveryKey = getAgentTriggerIdempotencyKey(envelope);
+ const reserved = await runAsSystem(() =>
+ methods.reserveAgentQueuedTurnDelivery({
+ user: turn.user,
+ ...(turn.tenantId != null && { tenantId: turn.tenantId }),
+ conversationId: turn.conversationId,
+ queuedTurnId: turn.queuedTurnId,
+ deliveryKey,
+ }),
+ );
+ if (reserved.outcome === 'conflict') {
+ throw new Error('The queued turn delivery identity could not be reserved');
+ }
+ const receipt = await enqueue(envelope, {
+ /** Queue sequence in Mongo is the sole conversation-ordering authority.
+ * A lane per durable row prevents a later published delivery from
+ * blocking recovery of an earlier record-first outbox row. */
+ orderingKey: `agent-queued-turn-delivery:${turn.queuedTurnId}`,
+ requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
+ });
+ if (receipt.deliveryKey !== deliveryKey) {
+ throw new Error('The queued turn delivery identity changed during publication');
+ }
+ const marked = await runAsSystem(() =>
+ methods.markQueuedTurnScheduled({
+ user: turn.user,
+ ...(turn.tenantId != null && { tenantId: turn.tenantId }),
+ conversationId: turn.conversationId,
+ queuedTurnId: turn.queuedTurnId,
+ deliveryKey: receipt.deliveryKey,
+ scheduledAt: new Date(),
+ }),
+ );
+ if (marked.outcome === 'conflict') {
+ throw new Error('The queued turn scheduling receipt could not be committed');
+ }
+ return receipt.deliveryKey;
+ };
+
+ const recover = (): Promise<number> => {
+ if (recovery != null) {
+ return recovery;
+ }
+ const task = (async () => {
+ const reconciliationNow = new Date();
+ const reconciliationClaimId = randomUUID();
+ const [turns, quarantined] = await Promise.all([
+ runAsSystem(() => methods.findQueuedTurnsNeedingDelivery(recoveryLimit)),
+ runAsSystem(() =>
+ methods.claimQueuedTurnsForAdmissionReconciliation({
+ claimId: reconciliationClaimId,
+ claimBy: PROCESS_CLAIM_OWNER,
+ now: reconciliationNow,
+ leaseUntil: new Date(reconciliationNow.getTime() + RECONCILIATION_LEASE_MS),
+ limit: recoveryLimit,
+ }),
+ ),
+ ]);
+ let repaired = 0;
+ for (const turn of quarantined) {
+ const deliveryKey = turn.deliveryKey;
+ if (
+ deliveryKey == null ||
+ turn.reconciliationClaimId !== reconciliationClaimId ||
+ turn.reconciliationClaimBy !== PROCESS_CLAIM_OWNER
+ ) {
+ continue;
+ }
+ const defer = () =>
+ runAsSystem(() =>
+ methods.deferAgentQueuedTurnAdmissionReconciliation({
+ user: turn.user,
+ ...(turn.tenantId != null && { tenantId: turn.tenantId }),
+ conversationId: turn.conversationId,
+ queuedTurnId: turn.queuedTurnId,
+ deliveryKey,
+ claimId: reconciliationClaimId,
+ claimBy: PROCESS_CLAIM_OWNER,
+ availableAt: new Date(
+ Date.now() + reconciliationBackoff(turn.reconciliationAttempts),
+ ),
+ }),
+ );
+ try {
+ if (turn.status === 'claimed') {
+ const result = await runAsSystem(() =>
+ methods.deadLetterAgentQueuedTurn({
+ user: turn.user,
+ ...(turn.tenantId != null && { tenantId: turn.tenantId }),
+ conversationId: turn.conversationId,
+ queuedTurnId: turn.queuedTurnId,
+ deliveryKey,
+ settledAt: new Date(),
+ failure: {
+ code: 'ADMISSION_INDETERMINATE',
+ message: 'The queued turn provider admission owner disappeared',
+ },
+ }),
+ );
+ if (result.outcome === 'admission_indeterminate') {
+ repaired += 1;
+ }
+ continue;
+ }
+ if (turn.admissionProtocolVersion === 2) {
+ /** The provider may have been invoked before its Mongo receipt
+ * became durable. Transient job evidence cannot decide that
+ * boundary, so retain explicit indeterminate evidence and rotate
+ * the work with bounded backoff until an exact late receipt or
+ * operator reconciliation arrives. */
+ await defer();
+ continue;
+ }
+ const admissionEvidence = await getGenerationAdmissionEvidence(
+ turn.user.toString(),
+ deliveryKey,
+ turn.conversationId,
+ turn.conversationId,
+ );
+ if (admissionEvidence == null) {
+ await defer();
+ continue;
+ }
+ const settled = await runAsSystem(() =>
+ methods.deadLetterAgentQueuedTurn({
+ user: turn.user,
+ ...(turn.tenantId != null && { tenantId: turn.tenantId }),
+ conversationId: turn.conversationId,
+ queuedTurnId: turn.queuedTurnId,
+ deliveryKey,
+ settledAt: new Date(),
+ failure: turn.terminalReceipt?.failure ?? {
+ code: 'ADMISSION_INDETERMINATE',
+ message: 'The queued turn admission requires reconciliation',
+ },
+ admissionEvidence,
+ reconciliationClaimId,
+ reconciliationClaimBy: PROCESS_CLAIM_OWNER,
+ }),
+ );
+ if (settled.outcome === 'admission_reconciled') {
+ repaired += 1;
+ }
+ } catch (error) {
+ await defer().catch(() => undefined);
+ logger.warn(
+ `[agentQueuedTurns] Failed to reconcile admission ${turn.queuedTurnId}: ${
+ error instanceof Error ? error.message : String(error)
+ }`,
+ );
+ }
+ }
+ for (const turn of turns) {
+ try {
+ await schedule(turn);
+ repaired += 1;
+ } catch (error) {
+ logger.warn(
+ `[agentQueuedTurns] Failed to repair delivery ${turn.queuedTurnId}: ${
+ error instanceof Error ? error.message : String(error)
+ }`,
+ );
+ }
+ }
+ return repaired;
+ })();
+ recovery = task;
+ void task.then(
+ () => {
+ if (recovery === task) {
+ recovery = undefined;
+ }
+ },
+ () => {
+ if (recovery === task) {
+ recovery = undefined;
+ }
+ },
+ );
+ return task;
+ };
+
+ return {
+ schedule,
+ recover,
+ initialize: async () => {
+ await runAsSystem(() => methods.ensureAgentQueuedTurnIndexes());
+ timer = setInterval(() => {
+ void recover().catch((error: unknown) => {
+ logger.warn('[agentQueuedTurns] Delivery recovery pass failed', error);
+ });
+ }, recoveryIntervalMs);
+ timer.unref?.();
+ await recover().catch((error: unknown) => {
+ logger.warn('[agentQueuedTurns] Initial delivery recovery pass failed', error);
+ });
+ },
+ stop: async () => {
+ if (timer != null) {
+ clearInterval(timer);
+ timer = undefined;
+ }
+ await recovery;
+ },
+ };
+}
+
+async function cancelAgentQueuedTurn(
+ input: Parameters<AgentQueuedTurnMethods['cancelAgentQueuedTurn']>[0],
+ deps: Pick<AgentQueuedTurnLifecycleDeps, 'methods' | 'retireDelivery' | 'getDelivery'>,
+): ReturnType<AgentQueuedTurnMethods['cancelAgentQueuedTurn']> {
+ const cancelled = await deps.methods.cancelAgentQueuedTurn(input);
+ if (
+ (cancelled.outcome !== 'cancelled' && cancelled.outcome !== 'already_cancelled') ||
+ cancelled.turn.deliveryKey == null ||
+ deps.retireDelivery == null
+ ) {
+ return cancelled;
+ }
+ const deliveryKey = cancelled.turn.deliveryKey;
+ let retired = await deps.retireDelivery(
+ deliveryKey,
+ AGENT_QUEUED_TURN_SOURCE,
+ 'queued_turn_cancelled',
+ );
+ if (!retired) {
+ retired = await deps.retireDelivery(
+ deliveryKey,
+ AGENT_QUEUED_TURN_SOURCE,
+ 'queued_turn_cancelled',
+ { onlyIfDead: true },
+ );
+ }
+ if (!retired && deps.getDelivery != null) {
+ const fenced = await deps.methods.beginAgentQueuedTurnMissingDeliveryRetirement({
+ deliveryKey,
+ });
+ if (fenced && (await deps.getDelivery(deliveryKey)) == null) {
+ retired = await deps.methods.markAgentQueuedTurnMissingDeliveryRetired({ deliveryKey });
+ }
+ }
+ if (retired) {
+ await deps.methods.markAgentQueuedTurnDeliveryRetired({ deliveryKey });
+ }
+ return cancelled;
+}
+
+/** Owns the complete backend Agent queued-turn lifecycle while retaining
+ * Mongo, trigger delivery, and generation execution as internal adapters. */
+export function createAgentQueuedTurnLifecycle({
+ methods,
+ getGenerationJob,
+ getGenerationAdmissionEvidence,
+ enqueue,
+ retireDelivery,
+ getDelivery,
+ now,
+ claimBy,
+ recoveryIntervalMs,
+ recoveryLimit,
+}: AgentQueuedTurnLifecycleDeps): AgentQueuedTurnLifecycle {
+ const scheduler = createAgentQueuedTurnScheduler({
+ methods,
+ enqueue,
+ getGenerationAdmissionEvidence,
+ ...(recoveryIntervalMs != null && { recoveryIntervalMs }),
+ ...(recoveryLimit != null && { recoveryLimit }),
+ });
+ return {
+ prepareContinue: createAgentQueuedTurnResolver({
+ methods,
+ getGenerationJob,
+ ...(now != null && { now }),
+ ...(claimBy != null && { claimBy }),
+ }),
+ settleBeforeDeadLetter: createAgentQueuedTurnDeadLetterSettlement({
+ methods,
+ getGenerationAdmissionEvidence,
+ ...(now != null && { now }),
+ }),
+ recordExecutionAdmission: (rawSource, input) =>
+ settleAgentQueuedTurnExecutionAdmission(rawSource, input, methods),
+ verifyExecutionAdmission: (rawSource, input) =>
+ verifyAgentQueuedTurnExecutionAdmission(rawSource, input, methods),
+ initialize: scheduler.initialize,
+ stop: scheduler.stop,
+ schedule: scheduler.schedule,
+ cancel: (input) =>
+ cancelAgentQueuedTurn(input, {
+ methods,
+ ...(retireDelivery != null && { retireDelivery }),
+ ...(getDelivery != null && { getDelivery }),
+ }),
+ recover: scheduler.recover,
+ };
+}
diff --git a/packages/api/src/agents/triggers/dispatch.ts b/packages/api/src/agents/triggers/dispatch.ts
index e8ceb196c9..09d7607eca 100644
--- a/packages/api/src/agents/triggers/dispatch.ts
+++ b/packages/api/src/agents/triggers/dispatch.ts
@@ -8,6 +8,9 @@ import { getAgentTriggerIdempotencyKey, parseAgentTriggerEnvelope } from './enve
export interface AgentTriggerDispatchContext {
idempotencyKey: string;
+ /** Durable delivery attempt metadata, when dispatched by the queue engine. */
+ attempt?: number;
+ maxAttempts?: number;
signal?: AbortSignal;
}
@@ -41,7 +44,7 @@ export interface AgentTriggerDispatchHandlers<FireResult, ContinueResult, SteerR
export function dispatchAgentTrigger<FireResult, ContinueResult, SteerResult>(
envelope: unknown,
handlers: AgentTriggerDispatchHandlers<FireResult, ContinueResult, SteerResult>,
- options?: { signal?: AbortSignal },
+ options?: { signal?: AbortSignal; attempt?: number; maxAttempts?: number },
): Promise<ContinueResult | FireResult | SteerResult> {
let normalized: AgentTriggerEnvelope;
try {
@@ -51,6 +54,8 @@ export function dispatchAgentTrigger<FireResult, ContinueResult, SteerResult>(
}
const context: AgentTriggerDispatchContext = {
idempotencyKey: getAgentTriggerIdempotencyKey(normalized),
+ ...(options?.attempt != null && { attempt: options.attempt }),
+ ...(options?.maxAttempts != null && { maxAttempts: options.maxAttempts }),
...(options?.signal != null && { signal: options.signal }),
};
if (normalized.mode === 'fire') {
diff --git a/packages/api/src/agents/triggers/engine.spec.ts b/packages/api/src/agents/triggers/engine.spec.ts
index e456fc692c..91f24aa8f5 100644
--- a/packages/api/src/agents/triggers/engine.spec.ts
+++ b/packages/api/src/agents/triggers/engine.spec.ts
@@ -1,4 +1,8 @@
-import type { AgentTriggerDeliveryRecord, AgentTriggerDeliveryStore } from './engine';
+import type {
+ AgentTriggerDeliveryFailure,
+ AgentTriggerDeliveryRecord,
+ AgentTriggerDeliveryStore,
+} from './engine';
import type { AgentTriggerExecutionResult } from './host';
import { AgentTriggerDeliveryDeferredError, createAgentTriggerDeliveryEngine } from './engine';
import { createAgentTriggerEnvelope } from './envelope';
@@ -67,7 +71,10 @@ describe('createAgentTriggerDeliveryEngine', () => {
await expect(engine.runTick()).resolves.toBe(1);
- expect(dispatch).toHaveBeenCalledWith({ version: 1 }, { signal: expect.any(AbortSignal) });
+ expect(dispatch).toHaveBeenCalledWith(
+ { version: 1 },
+ { signal: expect.any(AbortSignal), attempt: 1, maxAttempts: 8 },
+ );
expect(store.beginAttempt).toHaveBeenCalledWith({
id: 'delivery-row-1',
workerId: 'worker-1',
@@ -463,11 +470,21 @@ describe('createAgentTriggerDeliveryEngine', () => {
});
it('dead-letters invalid envelopes without retrying', async () => {
- const store = storeWith();
+ const transitions: string[] = [];
+ const store = storeWith({
+ dead: jest.fn(async () => {
+ transitions.push('delivery-dead');
+ return true;
+ }),
+ });
+ const settleSourceBeforeDeadLetter = jest.fn(async () => {
+ transitions.push('source-dead');
+ });
const engine = createAgentTriggerDeliveryEngine(
{
store,
dispatch: async () => Promise.reject(new AgentTriggerDispatchError('invalid envelope')),
+ settleSourceBeforeDeadLetter,
now: () => START,
},
{ concurrency: 1 },
@@ -485,9 +502,76 @@ describe('createAgentTriggerDeliveryEngine', () => {
}),
}),
);
+ expect(settleSourceBeforeDeadLetter).toHaveBeenCalledWith(
+ { version: 1 },
+ expect.objectContaining({ code: 'INVALID_ENVELOPE' }),
+ );
+ expect(transitions).toEqual(['source-dead', 'delivery-dead']);
expect(store.retry).not.toHaveBeenCalled();
});
+ it('bounds one terminal failure before source and delivery settlement', async () => {
+ const oversized = 'x'.repeat(3_000);
+ const store = storeWith();
+ const settleSourceBeforeDeadLetter = jest.fn(
+ async (_envelope: unknown, _failure: AgentTriggerDeliveryFailure) => undefined,
+ );
+ const engine = createAgentTriggerDeliveryEngine(
+ {
+ store,
+ dispatch: async () =>
+ Promise.reject(
+ new AgentTriggerExecutionError(oversized, {
+ mode: 'fire',
+ certainty: 'definite',
+ retryable: false,
+ code: oversized,
+ }),
+ ),
+ settleSourceBeforeDeadLetter,
+ now: () => START,
+ },
+ { concurrency: 1 },
+ );
+
+ await engine.runTick();
+
+ const sourceFailure = settleSourceBeforeDeadLetter.mock.calls[0]?.[1];
+ const deliveryFailure = (store.dead as jest.Mock).mock.calls[0]?.[0]?.error;
+ expect(sourceFailure).toMatchObject({
+ code: 'x'.repeat(128),
+ message: 'x'.repeat(2_048),
+ });
+ expect(deliveryFailure).toEqual(sourceFailure);
+ });
+
+ it('releases the delivery when source terminalization fails before dead-lettering', async () => {
+ const store = storeWith();
+ const settleSourceBeforeDeadLetter = jest.fn(async () => {
+ throw new Error('source write unavailable');
+ });
+ const engine = createAgentTriggerDeliveryEngine(
+ {
+ store,
+ dispatch: async () => Promise.reject(new AgentTriggerDispatchError('invalid envelope')),
+ settleSourceBeforeDeadLetter,
+ now: () => START,
+ workerId: 'worker-1',
+ },
+ { concurrency: 1 },
+ );
+
+ await engine.runTick();
+
+ expect(store.dead).not.toHaveBeenCalled();
+ expect(store.release).toHaveBeenCalledWith({
+ id: 'delivery-row-1',
+ workerId: 'worker-1',
+ claimToken: 'claim-1',
+ availableAt: START,
+ });
+ });
+
it('dead-letters an exhausted row without dispatching again', async () => {
const store = storeWith({
claimNext: jest.fn(async () => delivery({ attempts: 8 })),
@@ -505,6 +589,77 @@ describe('createAgentTriggerDeliveryEngine', () => {
expect(store.dead).toHaveBeenCalledWith(expect.objectContaining({ claimToken: 'claim-1' }));
});
+ it('bounds a persisted last failure before exhausting its source', async () => {
+ const oversized = 'x'.repeat(3_000);
+ const store = storeWith({
+ claimNext: jest.fn(async () =>
+ delivery({
+ attempts: 8,
+ lastError: {
+ code: oversized,
+ message: oversized,
+ certainty: 'ambiguous',
+ retryable: true,
+ attemptedAt: START,
+ },
+ }),
+ ),
+ });
+ const settleSourceBeforeDeadLetter = jest.fn(async () => undefined);
+ const engine = createAgentTriggerDeliveryEngine(
+ { store, dispatch: jest.fn(), settleSourceBeforeDeadLetter, now: () => START },
+ { concurrency: 1, maxAttempts: 8 },
+ );
+
+ await engine.runTick();
+
+ expect(settleSourceBeforeDeadLetter).toHaveBeenCalledWith(
+ { version: 1 },
+ expect.objectContaining({ code: 'x'.repeat(128), message: 'x'.repeat(2_048) }),
+ );
+ expect(store.dead).toHaveBeenCalledWith(
+ expect.objectContaining({
+ error: expect.objectContaining({
+ code: 'x'.repeat(128),
+ message: 'x'.repeat(2_048),
+ }),
+ }),
+ );
+ });
+
+ it('does not dead-letter an exhausted row until its source is terminal', async () => {
+ const store = storeWith({
+ claimNext: jest.fn(async () => delivery({ attempts: 8 })),
+ });
+ const settleSourceBeforeDeadLetter = jest.fn(async () => {
+ throw new Error('source write unavailable');
+ });
+ const engine = createAgentTriggerDeliveryEngine(
+ {
+ store,
+ dispatch: jest.fn(async () => successResult()),
+ settleSourceBeforeDeadLetter,
+ now: () => START,
+ workerId: 'worker-1',
+ },
+ { concurrency: 1, maxAttempts: 8 },
+ );
+
+ await engine.runTick();
+
+ expect(settleSourceBeforeDeadLetter).toHaveBeenCalledWith(
+ { version: 1 },
+ expect.objectContaining({ retryable: true }),
+ );
+ expect(store.dead).not.toHaveBeenCalled();
+ expect(store.release).toHaveBeenCalledWith({
+ id: 'delivery-row-1',
+ workerId: 'worker-1',
+ claimToken: 'claim-1',
+ availableAt: START,
+ });
+ });
+
it('rechecks a leased predecessor promptly instead of waiting for its full lease', async () => {
const store = storeWith({
findEarlierUnsettled: jest.fn(async () => ({
diff --git a/packages/api/src/agents/triggers/engine.ts b/packages/api/src/agents/triggers/engine.ts
index d6c3ad19ca..164854a0e7 100644
--- a/packages/api/src/agents/triggers/engine.ts
+++ b/packages/api/src/agents/triggers/engine.ts
@@ -16,6 +16,8 @@ const ORDERING_RECHECK_MS = 250;
const ACTIVE_HANDLING_RECHECK_MS = 5_000;
const DEFAULT_DEFER_MS = 5_000;
const MAX_RETRY_AFTER_MS = 24 * 60 * 60_000;
+const MAX_FAILURE_CODE_LENGTH = 128;
+const MAX_FAILURE_MESSAGE_LENGTH = 2048;
function startedHandling(
delivery: Pick<AgentTriggerDeliveryRecord, 'envelope'>,
@@ -200,8 +202,14 @@ export interface AgentTriggerDeliveryEngineDeps {
store: AgentTriggerDeliveryStore;
dispatch: (
envelope: unknown,
- options?: { signal?: AbortSignal },
+ options?: { signal?: AbortSignal; attempt?: number; maxAttempts?: number },
) => Promise<AgentTriggerExecutionResult>;
+ /** Source-owned terminalization must commit before its delivery can become
+ * dead, including recovery after a crash that exhausted the attempt budget. */
+ settleSourceBeforeDeadLetter?: (
+ envelope: unknown,
+ failure: AgentTriggerDeliveryFailure,
+ ) => Promise<void>;
now?: () => Date;
random?: () => number;
workerId?: string;
@@ -261,6 +269,19 @@ function failure(error: unknown, attemptedAt: Date): AgentTriggerDeliveryFailure
};
}
+function normalizeFailure(failure: AgentTriggerDeliveryFailure): AgentTriggerDeliveryFailure {
+ const code = failure.code.trim();
+ const message = failure.message.trim();
+ return {
+ ...failure,
+ code: (code.length === 0 ? 'DELIVERY_FAILED' : code).slice(0, MAX_FAILURE_CODE_LENGTH),
+ message: (message.length === 0 ? 'Agent trigger delivery failed' : message).slice(
+ 0,
+ MAX_FAILURE_MESSAGE_LENGTH,
+ ),
+ };
+}
+
function retryAt(
error: unknown,
attempt: number,
@@ -362,9 +383,25 @@ export function createAgentTriggerDeliveryEngine(
}
if (delivery.attempts >= maxAttempts) {
- const recorded =
+ const recorded = normalizeFailure(
delivery.lastError ??
- failure(new Error('Delivery attempt limit was already exhausted'), now());
+ failure(new Error('Delivery attempt limit was already exhausted'), now()),
+ );
+ try {
+ await deps.settleSourceBeforeDeadLetter?.(delivery.envelope, recorded);
+ } catch (error) {
+ logger.error('[agent-triggers] source terminalization failed before dead-lettering', {
+ deliveryKey: delivery.deliveryKey,
+ error: error instanceof Error ? error.message : String(error),
+ });
+ await deps.store.release({
+ id: delivery.id,
+ workerId,
+ claimToken: delivery.claimToken,
+ availableAt: now(),
+ });
+ return;
+ }
const deadLettered = await deps.store.dead({
id: delivery.id,
workerId,
@@ -426,7 +463,11 @@ export function createAgentTriggerDeliveryEngine(
members.length === 0
? delivery.envelope
: createAgentTriggerBatchEnvelope(delivery, members);
- result = await deps.dispatch(dispatchEnvelope, { signal: controller.signal });
+ result = await deps.dispatch(dispatchEnvelope, {
+ signal: controller.signal,
+ attempt,
+ maxAttempts,
+ });
} catch (error) {
const attemptedAt = now();
const deletionCancelled = controller.signal.aborted && cancelledUsers.has(userId);
@@ -464,8 +505,26 @@ export function createAgentTriggerDeliveryEngine(
}
return;
}
- const recorded = failure(error, attemptedAt);
+ const recorded = normalizeFailure(failure(error, attemptedAt));
if (!recorded.retryable || attempt >= maxAttempts) {
+ try {
+ await deps.settleSourceBeforeDeadLetter?.(delivery.envelope, recorded);
+ } catch (settlementError) {
+ logger.error('[agent-triggers] source terminalization failed before dead-lettering', {
+ deliveryKey: delivery.deliveryKey,
+ error:
+ settlementError instanceof Error
+ ? settlementError.message
+ : String(settlementError),
+ });
+ await deps.store.release({
+ id: delivery.id,
+ workerId,
+ claimToken: delivery.claimToken,
+ availableAt: attemptedAt,
+ });
+ return;
+ }
const deadLettered = await deps.store.dead({
id: delivery.id,
workerId,
diff --git a/packages/api/src/agents/triggers/host.spec.ts b/packages/api/src/agents/triggers/host.spec.ts
index 706fa0056c..b5dcf2ac4e 100644
--- a/packages/api/src/agents/triggers/host.spec.ts
+++ b/packages/api/src/agents/triggers/host.spec.ts
@@ -216,8 +216,12 @@ describe('createAgentTriggerExecutionHost fire adapter', () => {
const bodies = fetcher.mock.calls.map(([, init]) => JSON.parse(String(init?.body)) as unknown);
expect(bodies).toEqual([
- expect.objectContaining({ clientRequestId: getAgentTriggerIdempotencyKey(envelope) }),
- expect.objectContaining({ clientRequestId: getAgentTriggerIdempotencyKey(envelope) }),
+ expect.objectContaining({
+ clientRequestId: getAgentTriggerIdempotencyKey(envelope),
+ }),
+ expect.objectContaining({
+ clientRequestId: getAgentTriggerIdempotencyKey(envelope),
+ }),
]);
});
@@ -419,7 +423,10 @@ describe('createAgentTriggerExecutionHost fire adapter', () => {
resolveToken('signed-token');
resolveTimezone('UTC');
- await expect(pending).resolves.toMatchObject({ mode: 'fire', status: 'started' });
+ await expect(pending).resolves.toMatchObject({
+ mode: 'fire',
+ status: 'started',
+ });
});
it('observes caller cancellation while asynchronous setup is pending', async () => {
@@ -429,7 +436,9 @@ describe('createAgentTriggerExecutionHost fire adapter', () => {
deps(fetcher, { mintToken: () => new Promise<string>(() => undefined) }),
);
const controller = new AbortController();
- const pending = host.dispatch(createFireEnvelope(), { signal: controller.signal });
+ const pending = host.dispatch(createFireEnvelope(), {
+ signal: controller.signal,
+ });
controller.abort();
await pending.catch((error: unknown) => {
@@ -579,6 +588,97 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
});
});
+ it('carries a prepared queued-turn payload and settles it after admission', async () => {
+ const envelope = createContinueEnvelope();
+ const admitted = {
+ mode: 'continue' as const,
+ streamId: 'conversation-1',
+ conversationId: 'conversation-1',
+ generationCreatedAt: 50,
+ status: 'started' as const,
+ };
+ const settleOnAdmission = jest.fn(async () => undefined);
+ const getBaseUrl = jest.fn(() => 'http://127.0.0.1:3080');
+ const admissionSource = {
+ source: 'agent-queued-turn',
+ sourceId: 'queued-turn-1',
+ claimId: 'queued-delivery-1',
+ claimBy: 'queued-worker-1',
+ };
+ const prepareContinue = jest.fn(async () => ({
+ status: 'ready' as const,
+ input: 'queued user turn',
+ parentMessageId: 'latest-response',
+ expectedPredecessorCreatedAt: 49,
+ files: [{ file_id: 'file-1' }],
+ quotes: ['quoted context'],
+ manualSkills: ['research'],
+ admissionSource,
+ settleOnAdmission,
+ }));
+ const fetcher = fetchMock(async () => response(admitted));
+ const host = createAgentTriggerExecutionHost(
+ deps(fetcher, {
+ prepareContinue,
+ getBaseUrl,
+ }),
+ );
+
+ await expect(host.dispatch(envelope, { attempt: 3, maxAttempts: 3 })).resolves.toEqual(
+ admitted,
+ );
+ expect(JSON.parse(String(fetcher.mock.calls[0][1]?.body))).toMatchObject({
+ text: 'queued user turn',
+ parentMessageId: 'latest-response',
+ expectedPredecessorCreatedAt: 49,
+ files: [{ file_id: 'file-1' }],
+ quotes: ['quoted context'],
+ manualSkills: ['research'],
+ agentContinuationAdmission: admissionSource,
+ });
+ expect(getBaseUrl).toHaveBeenCalledWith({ localOnly: true });
+ expect(prepareContinue).toHaveBeenCalledWith(envelope, {
+ idempotencyKey: getAgentTriggerIdempotencyKey(envelope),
+ attempt: 3,
+ maxAttempts: 3,
+ });
+ expect(settleOnAdmission).toHaveBeenCalledWith(admitted);
+ });
+
+ it('keeps a prepared queued turn claimed when admission settlement is unavailable', async () => {
+ const releaseOnDefiniteFailure = jest.fn(async () => undefined);
+ const host = createAgentTriggerExecutionHost(
+ deps(
+ fetchMock(async () =>
+ response({
+ mode: 'continue',
+ streamId: 'conversation-1',
+ conversationId: 'conversation-1',
+ status: 'started',
+ }),
+ ),
+ {
+ prepareContinue: async () => ({
+ status: 'ready',
+ input: 'queued user turn',
+ parentMessageId: 'response-1',
+ releaseOnDefiniteFailure,
+ settleOnAdmission: async () => {
+ throw new Error('mongo unavailable');
+ },
+ }),
+ },
+ ),
+ );
+
+ await expect(host.dispatch(createContinueEnvelope())).rejects.toMatchObject({
+ certainty: 'ambiguous',
+ retryable: true,
+ code: 'PREPARATION_SETTLEMENT_FAILED',
+ });
+ expect(releaseOnDefiniteFailure).not.toHaveBeenCalled();
+ });
+
it('carries server-resolved binding identity only on bound child continuations', async () => {
const base = createContinueEnvelope();
if (base.mode !== 'continue') {
@@ -653,7 +753,10 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
id: completion.taskId,
type: EVENT_ACTOR_DETACHED_COMPLETION_TYPE,
occurredAt: completion.generationCreatedAt + 1,
- source: { id: EVENT_ACTOR_DETACHED_COMPLETION_SOURCE, type: 'internal' },
+ source: {
+ id: EVENT_ACTOR_DETACHED_COMPLETION_SOURCE,
+ type: 'internal',
+ },
payload: completion,
},
expectedAction: { toolName: 'submit_move' },
@@ -787,7 +890,10 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
deps(
fetchMock(async () =>
response(
- { code: 'PARENT_STATE_UNAVAILABLE', error: 'Parent state is unavailable.' },
+ {
+ code: 'PARENT_STATE_UNAVAILABLE',
+ error: 'Parent state is unavailable.',
+ },
{ status: 503, headers: { 'retry-after': '1' } },
),
),
@@ -841,7 +947,11 @@ describe('createAgentTriggerExecutionHost continue adapter', () => {
const host = createAgentTriggerExecutionHost(
deps(
fetchMock(async () =>
- response({ streamId: 'other', conversationId: 'other', status: 'started' }),
+ response({
+ streamId: 'other',
+ conversationId: 'other',
+ status: 'started',
+ }),
),
),
);
diff --git a/packages/api/src/agents/triggers/host.ts b/packages/api/src/agents/triggers/host.ts
index 341041f27e..d460f6ae7e 100644
--- a/packages/api/src/agents/triggers/host.ts
+++ b/packages/api/src/agents/triggers/host.ts
@@ -1,5 +1,6 @@
import { logger, tenantStorage } from '@librechat/data-schemas';
import { Constants, EModelEndpoint } from 'librechat-data-provider';
+import type { TFile } from 'librechat-data-provider';
import type {
AgentContinueTriggerEnvelope,
AgentFireTriggerEnvelope,
@@ -37,14 +38,33 @@ type FireStatus = 'started' | 'resumed' | 'replaced' | 'settled';
export type AgentTriggerFetch = (input: string | URL, init?: RequestInit) => Promise<Response>;
+export interface AgentContinuationAdmissionSource {
+ source: string;
+ sourceId: string;
+ claimId: string;
+ claimBy: string;
+}
+
export type AgentTriggerContinuePreparation =
| {
status: 'ready';
input: string;
parentMessageId: string;
+ expectedPredecessorCreatedAt?: number;
+ /** Complete user-turn context for internal continuations that must start
+ * a fresh ordinary Agent turn rather than inject into an existing run. */
+ files?: Partial<TFile>[];
+ quotes?: string[];
+ manualSkills?: string[];
+ /** Trusted source identity committed by execution enrollment before the
+ * provider-start fence opens. */
+ admissionSource?: AgentContinuationAdmissionSource;
/** Compensates a durable pre-admission claim only when the host knows
* that no generation was admitted. Ambiguous outcomes retain the claim. */
- releaseOnDefiniteFailure?: () => MaybePromise<void>;
+ releaseOnDefiniteFailure?: (error?: AgentTriggerExecutionError) => MaybePromise<void>;
+ /** Commits the source handoff after generation admission. Failure is
+ * outcome-ambiguous: the same delivery retries with the same request id. */
+ settleOnAdmission?: (result: AgentTriggerContinueResult) => MaybePromise<void>;
}
| { status: 'settled' };
@@ -146,7 +166,7 @@ export interface AgentTriggerExecutionHostDeps {
export interface AgentTriggerExecutionHost {
dispatch: (
envelope: unknown,
- options?: { signal?: AbortSignal },
+ options?: { signal?: AbortSignal; attempt?: number; maxAttempts?: number },
) => Promise<AgentTriggerExecutionResult>;
}
@@ -302,7 +322,11 @@ async function readResponseBody(response: Response): Promise<BoundedResponseBody
const remaining = MAX_RESPONSE_BODY_BYTES - size;
if (chunk.value.byteLength > remaining) {
if (remaining > 0) {
- parts.push(decoder.decode(chunk.value.subarray(0, remaining), { stream: true }));
+ parts.push(
+ decoder.decode(chunk.value.subarray(0, remaining), {
+ stream: true,
+ }),
+ );
}
parts.push(decoder.decode());
await reader.cancel().catch(() => undefined);
@@ -587,7 +611,12 @@ async function startRun(
context.signal,
),
setupValue(
- () => deps.getBaseUrl(detachedCompletion == null ? undefined : { localOnly: true }),
+ () =>
+ deps.getBaseUrl(
+ detachedCompletion == null && readyPreparation?.admissionSource == null
+ ? undefined
+ : { localOnly: true },
+ ),
mode,
scope,
context.signal,
@@ -625,13 +654,28 @@ async function startRun(
endpoint: EModelEndpoint.agents,
agent_id: envelope.target.agentId,
parentMessageId,
+ ...(readyPreparation?.expectedPredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: readyPreparation.expectedPredecessorCreatedAt,
+ }),
...(envelope.mode === 'continue' && {
conversationId: envelope.target.conversationId,
}),
+ ...(readyPreparation?.files != null && {
+ files: readyPreparation.files,
+ }),
+ ...(readyPreparation?.quotes != null && {
+ quotes: readyPreparation.quotes,
+ }),
+ ...(readyPreparation?.manualSkills != null && {
+ manualSkills: readyPreparation.manualSkills,
+ }),
isContinued: false,
isRegenerate: false,
clientRequestId: context.idempotencyKey,
generationProtocolVersion: 2,
+ ...(readyPreparation?.admissionSource != null && {
+ agentContinuationAdmission: readyPreparation.admissionSource,
+ }),
...(envelope.mode === 'continue' &&
envelope.target.bindingId != null && {
agentEventDelivery: {
@@ -670,7 +714,9 @@ async function startRun(
...(run?.conversationId != null && {
newConversationId: run.conversationId,
}),
- ...(run?.chatProjectId != null && { chatProjectId: run.chatProjectId }),
+ ...(run?.chatProjectId != null && {
+ chatProjectId: run.chatProjectId,
+ }),
...(run?.files != null && { files: run.files }),
...(typeof timezone === 'string' && timezone.trim().length > 0
? { timezone: timezone.trim() }
@@ -772,6 +818,28 @@ async function startRun(
status: response.status,
});
}
+ if (
+ mode === 'continue' &&
+ result.mode === 'continue' &&
+ readyPreparation?.settleOnAdmission != null
+ ) {
+ try {
+ await readyPreparation.settleOnAdmission(result);
+ } catch (error) {
+ throw executionError(
+ `Agent trigger continue admitted its generation but could not settle its prepared source: ${
+ error instanceof Error ? error.message : String(error)
+ }`,
+ {
+ mode,
+ certainty: 'ambiguous',
+ retryable: true,
+ code: 'PREPARATION_SETTLEMENT_FAILED',
+ status: response.status,
+ },
+ );
+ }
+ }
return result;
} catch (error) {
if (
@@ -781,7 +849,7 @@ async function startRun(
canReleasePreparedResult(error)
) {
try {
- await preparation.releaseOnDefiniteFailure();
+ await preparation.releaseOnDefiniteFailure(error);
} catch (releaseError) {
throw executionError(
`Agent trigger ${mode} could not release its rejected preparation: ${
@@ -792,6 +860,7 @@ async function startRun(
certainty: 'definite',
retryable: true,
code: 'PREPARATION_RELEASE_FAILED',
+ deferWithoutAttempt: true,
},
);
}
diff --git a/packages/api/src/agents/triggers/service.delivery.spec.ts b/packages/api/src/agents/triggers/service.delivery.spec.ts
index bdc827c2f4..7f8d13fb6f 100644
--- a/packages/api/src/agents/triggers/service.delivery.spec.ts
+++ b/packages/api/src/agents/triggers/service.delivery.spec.ts
@@ -1,6 +1,7 @@
import {
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
+ AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
getTenantId,
SYSTEM_TENANT_ID,
} from '@librechat/data-schemas';
@@ -180,6 +181,7 @@ describe('durable agent trigger service', () => {
expect.objectContaining({
workerCapabilities: [
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
+ AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
],
}),
@@ -215,7 +217,10 @@ describe('durable agent trigger service', () => {
expect(methods.claimNextAgentTriggerDelivery).toHaveBeenCalledWith(
expect.objectContaining({
- workerCapabilities: [AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1],
+ workerCapabilities: [
+ AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
+ AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
+ ],
}),
);
await service.stop();
diff --git a/packages/api/src/agents/triggers/service.ts b/packages/api/src/agents/triggers/service.ts
index b8f51f0274..786e54fab4 100644
--- a/packages/api/src/agents/triggers/service.ts
+++ b/packages/api/src/agents/triggers/service.ts
@@ -1,6 +1,7 @@
import {
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
+ AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
logger,
runAsSystem,
} from '@librechat/data-schemas';
@@ -11,6 +12,7 @@ import type {
import type {
AgentTriggerDeliveryFailure,
AgentTriggerDeliveryEngine,
+ AgentTriggerDeliveryEngineDeps,
AgentTriggerDeliveryEngineOptions,
AgentTriggerDeliveryRecord,
AgentTriggerDeliveryStore,
@@ -54,6 +56,7 @@ export interface AgentTriggerServiceDeps {
purgeRecoveryIntervalMs?: number;
purgeRecoveryLimit?: number;
supportsDetachedActionCompletion?: () => boolean;
+ settleSourceBeforeDeadLetter?: AgentTriggerDeliveryEngineDeps['settleSourceBeforeDeadLetter'];
}
export interface AgentTriggerDeliveryReceipt {
@@ -180,6 +183,7 @@ function createDeliveryStore(
...input,
workerCapabilities: [
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
+ AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
...(supportsDetachedActionCompletion()
? [AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1]
: []),
@@ -272,7 +276,9 @@ export function createAgentTriggerService(deps: AgentTriggerServiceDeps = {}): A
((principal) => generateAgentTriggerToken(principal.userId, AGENT_TRIGGER_TOKEN_TTL)),
...(deps.fetch != null && { fetch: deps.fetch }),
...(deps.getTimezone != null && { getTimezone: deps.getTimezone }),
- ...(deps.prepareContinue != null && { prepareContinue: deps.prepareContinue }),
+ ...(deps.prepareContinue != null && {
+ prepareContinue: deps.prepareContinue,
+ }),
...(deps.timeoutMs != null && { timeoutMs: deps.timeoutMs }),
});
@@ -304,7 +310,7 @@ export function createAgentTriggerService(deps: AgentTriggerServiceDeps = {}): A
const dispatchForActivePrincipal = async (
envelope: unknown,
- options?: { signal?: AbortSignal },
+ options?: { signal?: AbortSignal; attempt?: number; maxAttempts?: number },
): Promise<AgentTriggerExecutionResult> => {
const parsed = parseAgentTriggerEnvelope(envelope);
await requireActivePrincipal(parsed.principal.userId);
@@ -440,6 +446,9 @@ export function createAgentTriggerService(deps: AgentTriggerServiceDeps = {}): A
{
store: createDeliveryStore(methods, supportsDetachedActionCompletion),
dispatch: dispatchForActivePrincipal,
+ ...(deps.settleSourceBeforeDeadLetter != null && {
+ settleSourceBeforeDeadLetter: deps.settleSourceBeforeDeadLetter,
+ }),
},
deps.deliveryOptions,
);
diff --git a/packages/api/src/stream/GenerationJobManager.ts b/packages/api/src/stream/GenerationJobManager.ts
index bf1c4c0eca..6a5f3a8f9c 100644
--- a/packages/api/src/stream/GenerationJobManager.ts
+++ b/packages/api/src/stream/GenerationJobManager.ts
@@ -2939,6 +2939,51 @@ class GenerationJobManagerClass {
);
}
+ /** Returns immutable proof that one exact request crossed generation
+ * admission, even after the generation job itself has completed or been
+ * replaced. Built-in stores retain this started receipt for the bounded
+ * idempotency horizon; legacy custom stores fall back to the live job. */
+ async getGenerationAdmissionEvidence(
+ userId: string,
+ clientRequestId: string,
+ streamId: string,
+ conversationId: string = streamId,
+ ): Promise<{ generationId: string; generationCreatedAt: number } | null> {
+ if (!CLIENT_REQUEST_ID_PATTERN.test(clientRequestId)) {
+ return null;
+ }
+ const getClaim = this.jobStore.getIdempotencyClaim;
+ if (getClaim != null) {
+ const claim = await getClaim.call(
+ this.jobStore,
+ this.generationClaimKey(userId, clientRequestId, streamId),
+ );
+ if (claim == null) {
+ return null;
+ }
+ const normalized = normalizeTokenClaim(claim, 'admission evidence');
+ assertClaimMatchesRequest(normalized, streamId, conversationId);
+ if (normalized.startedAt == null) {
+ return null;
+ }
+ return {
+ generationId: normalized.streamId,
+ generationCreatedAt: normalized.startedAt,
+ };
+ }
+
+ const job = await this.jobStore.getJob(streamId);
+ if (
+ job == null ||
+ job.userId !== userId ||
+ (job.conversationId ?? streamId) !== conversationId ||
+ job.idempotencyClientRequestId !== clientRequestId
+ ) {
+ return null;
+ }
+ return { generationId: streamId, generationCreatedAt: job.createdAt };
+ }
+
/** Atomically prevents an unpublished automatic continuation from starting
* after its delivery has been retired for manual recovery. If recovery wins
* the unstarted idempotency claim (directly or through the existing takeover
diff --git a/packages/api/src/stream/__tests__/idempotencyClaim.spec.ts b/packages/api/src/stream/__tests__/idempotencyClaim.spec.ts
index 8427eb4585..f8c30b4b8d 100644
--- a/packages/api/src/stream/__tests__/idempotencyClaim.spec.ts
+++ b/packages/api/src/stream/__tests__/idempotencyClaim.spec.ts
@@ -64,6 +64,20 @@ describe('InMemoryJobStore.claimIdempotencyKey', () => {
await expect(store.hasIdempotencyKey('user:missing')).resolves.toBe(false);
});
+ it('reads an idempotency receipt without creating a missing claim', async () => {
+ const receipt = {
+ streamId: 's1',
+ conversationId: 'c1',
+ claimToken: 'token-1',
+ claimedAt: 1,
+ startedAt: 2,
+ };
+ await expect(store.getIdempotencyClaim('user:missing')).resolves.toBeNull();
+ await store.claimIdempotencyKey('user:existing', receipt, 1200);
+ await expect(store.getIdempotencyClaim('user:existing')).resolves.toEqual(receipt);
+ await expect(store.hasIdempotencyKey('user:missing')).resolves.toBe(false);
+ });
+
it('does not report an expired claim as existing', async () => {
jest.useFakeTimers();
try {
@@ -234,6 +248,27 @@ describe('GenerationJobManager start-generation claim', () => {
await expect(manager.hasGenerationClaim('user-2', 'req-1')).resolves.toBe(false);
});
+ it('reads durable admission evidence after the live job is gone', async () => {
+ await store.claimIdempotencyKey(
+ '{stream-a}:user-1:req-1',
+ {
+ streamId: 'stream-a',
+ conversationId: 'convo-a',
+ claimToken: 'claim-token',
+ claimedAt: 1,
+ startedAt: 42,
+ },
+ 1200,
+ );
+
+ await expect(
+ manager.getGenerationAdmissionEvidence('user-1', 'req-1', 'stream-a', 'convo-a'),
+ ).resolves.toEqual({ generationId: 'stream-a', generationCreatedAt: 42 });
+ await expect(
+ manager.getGenerationAdmissionEvidence('user-1', 'req-2', 'stream-a', 'convo-a'),
+ ).resolves.toBeNull();
+ });
+
it('claims the exact legacy key before the same-slot primary with staggered TTLs', async () => {
const claimSpy = jest.spyOn(store, 'claimIdempotencyKey');
const result = await manager.claimGeneration(
diff --git a/packages/api/src/stream/implementations/InMemoryJobStore.ts b/packages/api/src/stream/implementations/InMemoryJobStore.ts
index 605291bd40..2d059e47ff 100644
--- a/packages/api/src/stream/implementations/InMemoryJobStore.ts
+++ b/packages/api/src/stream/implementations/InMemoryJobStore.ts
@@ -906,6 +906,18 @@ export class InMemoryJobStore implements IJobStoreV2 {
return false;
}
+ async getIdempotencyClaim(key: string): Promise<IdempotencyClaimValue | null> {
+ const existing = this.idempotencyClaims.get(key);
+ if (existing == null) {
+ return null;
+ }
+ if (existing.expiresAt <= Date.now()) {
+ this.idempotencyClaims.delete(key);
+ return null;
+ }
+ return { ...existing.value };
+ }
+
async takeoverIdempotencyKey(
key: string,
expected: IdempotencyClaimValue,
diff --git a/packages/api/src/stream/implementations/RedisJobStore.ts b/packages/api/src/stream/implementations/RedisJobStore.ts
index eef54ff58a..8670ddb953 100644
--- a/packages/api/src/stream/implementations/RedisJobStore.ts
+++ b/packages/api/src/stream/implementations/RedisJobStore.ts
@@ -2755,6 +2755,18 @@ export class RedisJobStore implements IJobStoreV2 {
return (await this.redis.exists(KEYS.idempotency(key))) === 1;
}
+ async getIdempotencyClaim(key: string): Promise<IdempotencyClaimValue | null> {
+ const value = await this.redis.get(KEYS.idempotency(key));
+ if (value == null) {
+ return null;
+ }
+ try {
+ return JSON.parse(value) as IdempotencyClaimValue;
+ } catch {
+ throw new Error('Invalid generation idempotency claim');
+ }
+ }
+
async takeoverIdempotencyKey(
key: string,
expected: IdempotencyClaimValue,
diff --git a/packages/api/src/stream/interfaces/IJobStore.ts b/packages/api/src/stream/interfaces/IJobStore.ts
index 2013fcf9ff..50c9394efa 100644
--- a/packages/api/src/stream/interfaces/IJobStore.ts
+++ b/packages/api/src/stream/interfaces/IJobStore.ts
@@ -903,6 +903,10 @@ export interface IJobStore {
* where every request remains subject to the limiter. */
hasIdempotencyKey?(key: string): Promise<boolean>;
+ /** Read-only claim receipt used by durable source reconcilers. Optional
+ * stores fall back to inspecting the current generation only. */
+ getIdempotencyClaim?(key: string): Promise<IdempotencyClaimValue | null>;
+
deleteJob(streamId: string, expectedCreatedAt?: number): Promise<boolean>;
hasJob(streamId: string): Promise<boolean>;
getRunningJobs(): Promise<SerializableJobData[]>;
diff --git a/packages/data-provider/specs/api-endpoints.spec.ts b/packages/data-provider/specs/api-endpoints.spec.ts
index e00fc378a4..415206e09b 100644
--- a/packages/data-provider/specs/api-endpoints.spec.ts
+++ b/packages/data-provider/specs/api-endpoints.spec.ts
@@ -1,7 +1,12 @@
/**
* @jest-environment jsdom
*/
-import { buildLoginRedirectUrl, getSharedLinks } from '../src/api-endpoints';
+import {
+ agentQueuedTurn,
+ buildLoginRedirectUrl,
+ getSharedLinks,
+ agentQueuedTurnsByConversation,
+} from '../src/api-endpoints';
describe('buildLoginRedirectUrl', () => {
afterEach(() => {
@@ -88,3 +93,27 @@ describe('getSharedLinks', () => {
);
});
});
+
+describe('agent queued turns', () => {
+ it('encodes conversation identity exactly once', () => {
+ expect(agentQueuedTurnsByConversation('conversation/a b')).toBe(
+ '/api/agents/chat/queued-turns?conversationId=conversation%2Fa%20b',
+ );
+ });
+
+ it('encodes deduplicated known request identities as repeated bounded query values', () => {
+ expect(
+ agentQueuedTurnsByConversation('conversation/one', [
+ 'request/one',
+ 'request,two',
+ 'request/one',
+ ]),
+ ).toBe(
+ '/api/agents/chat/queued-turns?conversationId=conversation%2Fone&clientRequestIds=request%2Fone&clientRequestIds=request%2Ctwo',
+ );
+ });
+
+ it('encodes queued-turn identity as one path segment', () => {
+ expect(agentQueuedTurn('turn/a b')).toBe('/api/agents/chat/queued-turns/turn%2Fa%20b');
+ });
+});
diff --git a/packages/data-provider/src/api-endpoints.ts b/packages/data-provider/src/api-endpoints.ts
index 156b6403f2..c958c8d299 100644
--- a/packages/data-provider/src/api-endpoints.ts
+++ b/packages/data-provider/src/api-endpoints.ts
@@ -313,6 +313,22 @@ export const agents = ({ path = '', options }: { path?: string; options?: object
export const activeJobs = () => `${BASE_URL}/api/agents/chat/active`;
+const agentQueuedTurnsRoot = `${BASE_URL}/api/agents/chat/queued-turns`;
+export const agentQueuedTurns = () => agentQueuedTurnsRoot;
+export const agentQueuedTurnsByConversation = (
+ conversationId: string,
+ clientRequestIds: string[] = [],
+) => {
+ const uniqueIds = Array.from(new Set(clientRequestIds)).slice(0, 100);
+ const knownIds =
+ uniqueIds.length > 0
+ ? `&${uniqueIds.map((id) => `clientRequestIds=${encodeURIComponent(id)}`).join('&')}`
+ : '';
+ return `${agentQueuedTurnsRoot}?conversationId=${encodeURIComponent(conversationId)}${knownIds}`;
+};
+export const agentQueuedTurn = (queuedTurnId: string) =>
+ `${agentQueuedTurnsRoot}/${encodeURIComponent(queuedTurnId)}`;
+
export const mcp = {
tools: `${BASE_URL}/api/mcp/tools`,
servers: `${BASE_URL}/api/mcp/servers`,
diff --git a/packages/data-provider/src/data-service.ts b/packages/data-provider/src/data-service.ts
index 1621370b61..7b0e8194e8 100644
--- a/packages/data-provider/src/data-service.ts
+++ b/packages/data-provider/src/data-service.ts
@@ -6,6 +6,7 @@ import * as permissions from './accessPermissions';
import * as endpoints from './api-endpoints';
import { uploadEventStream } from './upload';
import * as mcp from './types/mcpServers';
+import * as qt from './types/queuedTurns';
import * as sch from './types/schedules';
import * as a from './types/assistants';
import * as m from './types/mutations';
@@ -72,7 +73,9 @@ export function getFavorites(): Promise<q.TUserFavorite[]> {
}
export function updateFavorites(favorites: q.TUserFavorite[]): Promise<q.TUserFavorite[]> {
- return request.post(`${endpoints.apiBaseUrl()}/api/user/settings/favorites`, { favorites });
+ return request.post(`${endpoints.apiBaseUrl()}/api/user/settings/favorites`, {
+ favorites,
+ });
}
/** Tool favorites — starred marketplace items (builtins, tools, MCP servers, skills). */
@@ -135,7 +138,10 @@ export function updateSharedLink(
targetMessageId?: string,
snapshotFiles?: boolean,
): Promise<t.TSharedLinkResponse> {
- return request.patch(endpoints.updateSharedLink(shareId), { targetMessageId, snapshotFiles });
+ return request.patch(endpoints.updateSharedLink(shareId), {
+ targetMessageId,
+ snapshotFiles,
+ });
}
export function deleteSharedLink(shareId: string): Promise<m.TDeleteSharedLinkResponse> {
@@ -882,7 +888,9 @@ export function forkSharedConversation(
}
export function deleteConversation(payload: t.TDeleteConversationRequest) {
- return request.deleteWithOptions(endpoints.deleteConversation(), { data: { arg: payload } });
+ return request.deleteWithOptions(endpoints.deleteConversation(), {
+ data: { arg: payload },
+ });
}
export function clearAllConversations(): Promise<unknown> {
@@ -944,7 +952,9 @@ export function assignConversationToProject(
payload: t.TAssignConversationToProjectRequest,
): Promise<t.TAssignConversationToProjectResponse> {
const { conversationId, projectId } = payload;
- return request.put(endpoints.projectConversation(conversationId), { projectId });
+ return request.put(endpoints.projectConversation(conversationId), {
+ projectId,
+ });
}
export function pinConversation(
@@ -967,7 +977,9 @@ export function updateMessage(payload: t.TUpdateMessageRequest): Promise<unknown
throw new Error('conversationId is required');
}
- return request.put(endpoints.messages({ conversationId, messageId }), { text });
+ return request.put(endpoints.messages({ conversationId, messageId }), {
+ text,
+ });
}
export function updateMessageContent(payload: t.TUpdateMessageContent): Promise<unknown> {
@@ -976,7 +988,10 @@ export function updateMessageContent(payload: t.TUpdateMessageContent): Promise<
throw new Error('conversationId is required');
}
- return request.put(endpoints.messages({ conversationId, messageId }), { text, index });
+ return request.put(endpoints.messages({ conversationId, messageId }), {
+ text,
+ index,
+ });
}
export const editArtifact = async ({
@@ -1108,6 +1123,25 @@ export function getSchedules(): Promise<sch.TSchedulesResponse> {
return request.get(endpoints.schedules());
}
+export function enqueueAgentQueuedTurn(
+ payload: qt.TEnqueueAgentQueuedTurnRequest,
+): Promise<qt.TEnqueueAgentQueuedTurnResponse> {
+ return request.post(endpoints.agentQueuedTurns(), payload);
+}
+
+export function listAgentQueuedTurns(
+ conversationId: string,
+ clientRequestIds?: string[],
+): Promise<qt.TListAgentQueuedTurnsResponse> {
+ return request.get(endpoints.agentQueuedTurnsByConversation(conversationId, clientRequestIds));
+}
+
+export function cancelAgentQueuedTurn(
+ queuedTurnId: string,
+): Promise<qt.TCancelAgentQueuedTurnResponse> {
+ return request.delete(endpoints.agentQueuedTurn(queuedTurnId));
+}
+
export function getSchedule(id: string): Promise<sch.TSchedule> {
return request.get(endpoints.schedule(id));
}
@@ -1448,7 +1482,10 @@ export const updateMemory = (
originalKey?: string,
agentId?: string,
): Promise<q.UpdateMemoryResponse> => {
- return request.patch(endpoints.memory(originalKey || key, agentId), { key, value });
+ return request.patch(endpoints.memory(originalKey || key, agentId), {
+ key,
+ value,
+ });
};
export const updateMemoryById = (
diff --git a/packages/data-provider/src/index.ts b/packages/data-provider/src/index.ts
index 42c5336c6d..7c3e1651bc 100644
--- a/packages/data-provider/src/index.ts
+++ b/packages/data-provider/src/index.ts
@@ -37,6 +37,7 @@ export * from './types/web';
export * from './types/graph';
export * from './types/insights';
export * from './types/subagents';
+export * from './types/queuedTurns';
/* access permissions */
export * from './accessPermissions';
/* query/mutation keys */
diff --git a/packages/data-provider/src/keys.ts b/packages/data-provider/src/keys.ts
index 65e26d69de..d58856d066 100644
--- a/packages/data-provider/src/keys.ts
+++ b/packages/data-provider/src/keys.ts
@@ -95,6 +95,7 @@ export enum QueryKeys {
schedule = 'schedule',
parentSubagents = 'parentSubagents',
subagentThread = 'subagentThread',
+ agentQueuedTurns = 'agentQueuedTurns',
}
// Dynamic query keys that require parameters
@@ -104,6 +105,8 @@ export const DynamicQueryKeys = {
export enum MutationKeys {
subagentControl = 'subagentControl',
+ enqueueAgentQueuedTurn = 'enqueueAgentQueuedTurn',
+ cancelAgentQueuedTurn = 'cancelAgentQueuedTurn',
updateLangfuseConnection = 'updateLangfuseConnection',
testLangfuseConnection = 'testLangfuseConnection',
createAgentApiKey = 'createAgentApiKey',
diff --git a/packages/data-provider/src/types/index.ts b/packages/data-provider/src/types/index.ts
index f8df4b0a1d..961463e2bd 100644
--- a/packages/data-provider/src/types/index.ts
+++ b/packages/data-provider/src/types/index.ts
@@ -1,3 +1,4 @@
export * from './queries';
export * from './mcpServers';
export * from './subagents';
+export * from './queuedTurns';
diff --git a/packages/data-provider/src/types/queuedTurns.spec.ts b/packages/data-provider/src/types/queuedTurns.spec.ts
new file mode 100644
index 0000000000..03c5a4fe69
--- /dev/null
+++ b/packages/data-provider/src/types/queuedTurns.spec.ts
@@ -0,0 +1,109 @@
+import {
+ agentQueuedTurnCapabilitySchema,
+ agentQueuedTurnReceiptSchema,
+ cancelAgentQueuedTurnSchema,
+ enqueueAgentQueuedTurnSchema,
+ listAgentQueuedTurnsSchema,
+} from './queuedTurns';
+
+const request = {
+ conversationId: 'conversation-1',
+ parentMessageId: 'message-1',
+ clientRequestId: 'request-1',
+ text: 'Continue with this context',
+ files: [{ file_id: 'file-1', filename: 'context.txt' }],
+ quotes: ['quoted context'],
+ manualSkills: ['research'],
+ expectedPredecessorCreatedAt: 42,
+};
+
+describe('agent queued turn schemas', () => {
+ it('accepts the complete enqueue contract', () => {
+ expect(enqueueAgentQueuedTurnSchema.parse(request)).toEqual(request);
+ });
+
+ it('allows attachment-only queued turns', () => {
+ expect(
+ enqueueAgentQueuedTurnSchema.parse({
+ conversationId: 'conversation-1',
+ parentMessageId: 'message-1',
+ clientRequestId: 'request-1',
+ text: '',
+ files: [{ file_id: 'file-1' }],
+ }),
+ ).toEqual({
+ conversationId: 'conversation-1',
+ parentMessageId: 'message-1',
+ clientRequestId: 'request-1',
+ text: '',
+ files: [{ file_id: 'file-1' }],
+ });
+ });
+
+ it('rejects an invalid predecessor epoch', () => {
+ expect(() =>
+ enqueueAgentQueuedTurnSchema.parse({
+ ...request,
+ expectedPredecessorCreatedAt: -1,
+ }),
+ ).toThrow();
+ });
+
+ it('validates list and cancel identities', () => {
+ expect(
+ listAgentQueuedTurnsSchema.parse({
+ conversationId: 'conversation-1',
+ clientRequestIds: ['request-1', 'request-1', 'request-2'],
+ }),
+ ).toEqual({
+ conversationId: 'conversation-1',
+ clientRequestIds: ['request-1', 'request-2'],
+ });
+ expect(cancelAgentQueuedTurnSchema.parse({ queuedTurnId: 'turn-1' })).toEqual({
+ queuedTurnId: 'turn-1',
+ });
+ expect(() => listAgentQueuedTurnsSchema.parse({ conversationId: '' })).toThrow();
+ expect(() => cancelAgentQueuedTurnSchema.parse({ queuedTurnId: '' })).toThrow();
+ });
+
+ it('parses a durable queued receipt', () => {
+ const receipt = {
+ ...request,
+ queuedTurnId: 'turn-1',
+ status: 'queued',
+ position: 0,
+ revision: 1,
+ createdAt: '2026-08-30T12:00:00.000Z',
+ updatedAt: '2026-08-30T12:00:00.000Z',
+ };
+
+ expect(agentQueuedTurnReceiptSchema.parse(receipt)).toEqual(receipt);
+ });
+
+ it('parses a bounded terminal failure projection', () => {
+ const receipt = {
+ ...request,
+ queuedTurnId: 'turn-1',
+ status: 'dead',
+ revision: 1,
+ failure: { code: 'PARENT_NOT_FOUND', message: 'The selected branch is unavailable' },
+ createdAt: '2026-08-30T12:00:00.000Z',
+ updatedAt: '2026-08-30T12:01:00.000Z',
+ };
+
+ expect(agentQueuedTurnReceiptSchema.parse(receipt)).toEqual(receipt);
+ });
+
+ it('requires durability only when the capability is supported', () => {
+ expect(agentQueuedTurnCapabilitySchema.parse({ supported: false })).toEqual({
+ supported: false,
+ });
+ expect(
+ agentQueuedTurnCapabilitySchema.parse({
+ supported: true,
+ durability: 'process_local',
+ }),
+ ).toEqual({ supported: true, durability: 'process_local' });
+ expect(() => agentQueuedTurnCapabilitySchema.parse({ supported: true })).toThrow();
+ });
+});
diff --git a/packages/data-provider/src/types/queuedTurns.ts b/packages/data-provider/src/types/queuedTurns.ts
new file mode 100644
index 0000000000..5e7ef7daa0
--- /dev/null
+++ b/packages/data-provider/src/types/queuedTurns.ts
@@ -0,0 +1,93 @@
+import { z } from 'zod';
+
+export const agentQueuedTurnStatuses = [
+ 'queued',
+ 'claimed',
+ 'admitted',
+ 'cancelled',
+ 'dead',
+] as const;
+
+export const agentQueuedTurnDurability = ['process_local', 'durable'] as const;
+
+export const agentQueuedTurnFileRefSchema = z.object({
+ file_id: z.string().trim().min(1),
+ type: z.string().optional(),
+ filepath: z.string().optional(),
+ filename: z.string().optional(),
+ height: z.number().optional(),
+ width: z.number().optional(),
+ bytes: z.number().nonnegative().optional(),
+});
+export type TAgentQueuedTurnFileRef = z.infer<typeof agentQueuedTurnFileRefSchema>;
+
+export const enqueueAgentQueuedTurnSchema = z.object({
+ conversationId: z.string().trim().min(1),
+ parentMessageId: z.string().trim().min(1),
+ clientRequestId: z.string().trim().min(1).max(128),
+ text: z.string(),
+ files: z.array(agentQueuedTurnFileRefSchema).optional(),
+ quotes: z.array(z.string()).optional(),
+ manualSkills: z.array(z.string().trim().min(1)).optional(),
+ priority: z.boolean().optional(),
+ expectedPredecessorCreatedAt: z.number().int().nonnegative().optional(),
+});
+export type TEnqueueAgentQueuedTurnRequest = z.infer<typeof enqueueAgentQueuedTurnSchema>;
+
+export const listAgentQueuedTurnsSchema = z.object({
+ conversationId: z.string().trim().min(1),
+ clientRequestIds: z
+ .array(z.string().trim().min(1).max(128))
+ .max(100)
+ .transform((ids) => Array.from(new Set(ids)))
+ .optional(),
+});
+export type TListAgentQueuedTurnsRequest = z.infer<typeof listAgentQueuedTurnsSchema>;
+
+export const cancelAgentQueuedTurnSchema = z.object({
+ queuedTurnId: z.string().trim().min(1),
+});
+export type TCancelAgentQueuedTurnRequest = z.infer<typeof cancelAgentQueuedTurnSchema>;
+
+export const agentQueuedTurnReceiptSchema = enqueueAgentQueuedTurnSchema.extend({
+ queuedTurnId: z.string().trim().min(1),
+ status: z.enum(agentQueuedTurnStatuses),
+ position: z.number().int().nonnegative().optional(),
+ revision: z.number().int().nonnegative(),
+ failure: z
+ .object({
+ code: z.string().trim().min(1).max(128),
+ message: z.string().max(2048).optional(),
+ })
+ .optional(),
+ createdAt: z.string(),
+ updatedAt: z.string(),
+});
+export type TAgentQueuedTurnReceipt = z.infer<typeof agentQueuedTurnReceiptSchema>;
+
+export const agentQueuedTurnCapabilitySchema = z.discriminatedUnion('supported', [
+ z.object({ supported: z.literal(false) }),
+ z.object({
+ supported: z.literal(true),
+ durability: z.enum(agentQueuedTurnDurability),
+ }),
+]);
+export type TAgentQueuedTurnCapability = z.infer<typeof agentQueuedTurnCapabilitySchema>;
+
+export const enqueueAgentQueuedTurnResponseSchema = z.object({
+ receipt: agentQueuedTurnReceiptSchema,
+ capability: agentQueuedTurnCapabilitySchema,
+});
+export type TEnqueueAgentQueuedTurnResponse = z.infer<typeof enqueueAgentQueuedTurnResponseSchema>;
+
+export const listAgentQueuedTurnsResponseSchema = z.object({
+ queuedTurns: z.array(agentQueuedTurnReceiptSchema),
+ capability: agentQueuedTurnCapabilitySchema,
+ revision: z.number().int().nonnegative(),
+});
+export type TListAgentQueuedTurnsResponse = z.infer<typeof listAgentQueuedTurnsResponseSchema>;
+
+export const cancelAgentQueuedTurnResponseSchema = z.object({
+ receipt: agentQueuedTurnReceiptSchema,
+});
+export type TCancelAgentQueuedTurnResponse = z.infer<typeof cancelAgentQueuedTurnResponseSchema>;
diff --git a/packages/data-schemas/src/index.ts b/packages/data-schemas/src/index.ts
index 2a581dce95..499a8e2aa2 100644
--- a/packages/data-schemas/src/index.ts
+++ b/packages/data-schemas/src/index.ts
@@ -35,6 +35,9 @@ export {
MAX_AUDIT_VERIFY_ROWS,
MAX_TOOL_FAVORITES,
AgentTriggerDeliveryConflictError,
+ AgentQueuedTurnCapacityError,
+ AgentQueuedTurnConflictError,
+ AgentQueuedTurnLaneRetiredError,
recordAgentEventActorReceiptMetric,
setAgentEventActorReceiptMetricObserver,
MCPAuthorityProofError,
@@ -64,6 +67,7 @@ export {
export {
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
+ AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
} from './types/triggerDelivery';
export type * from './types';
export type * from './methods';
diff --git a/packages/data-schemas/src/methods/conversation.spec.ts b/packages/data-schemas/src/methods/conversation.spec.ts
index a73a08c88b..c35dec7f3d 100644
--- a/packages/data-schemas/src/methods/conversation.spec.ts
+++ b/packages/data-schemas/src/methods/conversation.spec.ts
@@ -1767,6 +1767,59 @@ describe('Conversation Operations', () => {
});
describe('deleteConvos', () => {
+ it('retires queued-turn work before each conversation deletion wave', async () => {
+ const conversationId = uuidv4();
+ await Conversation.create({
+ conversationId,
+ user: 'user123',
+ tenantId: 'tenant-1',
+ endpoint: EModelEndpoint.agents,
+ });
+ const transitions: string[] = [];
+ const deleteAgentQueuedTurns = jest.fn(async () => {
+ transitions.push('queue-retired');
+ });
+ const scopedMethods = createConversationMethods(mongoose, {
+ getMessages,
+ deleteMessages,
+ deleteAgentQueuedTurns,
+ });
+
+ await scopedMethods.deleteConvos(
+ 'user123',
+ { conversationId },
+ {
+ beforeDelete: async () => {
+ transitions.push('generation-drained');
+ },
+ },
+ );
+
+ expect(deleteAgentQueuedTurns).toHaveBeenCalledWith('user123', [
+ { conversationId, tenantId: 'tenant-1' },
+ ]);
+ expect(transitions).toEqual(['queue-retired', 'generation-drained']);
+ });
+
+ it('fails closed before deleting a conversation when queued-turn retirement fails', async () => {
+ const conversationId = uuidv4();
+ await Conversation.create({
+ conversationId,
+ user: 'user123',
+ endpoint: EModelEndpoint.agents,
+ });
+ const scopedMethods = createConversationMethods(mongoose, {
+ getMessages,
+ deleteMessages,
+ deleteAgentQueuedTurns: async () => Promise.reject(new Error('retirement unavailable')),
+ });
+
+ await expect(scopedMethods.deleteConvos('user123', { conversationId })).rejects.toThrow(
+ 'retirement unavailable',
+ );
+ expect(await Conversation.findOne({ conversationId })).not.toBeNull();
+ });
+
it('should delete conversations and associated messages', async () => {
await Conversation.create({
conversationId: mockConversationData.conversationId,
diff --git a/packages/data-schemas/src/methods/conversation.ts b/packages/data-schemas/src/methods/conversation.ts
index 68e0375b04..feec42cfd4 100644
--- a/packages/data-schemas/src/methods/conversation.ts
+++ b/packages/data-schemas/src/methods/conversation.ts
@@ -466,17 +466,25 @@ export interface ConversationMethods {
archiveAllConvos(user: string): Promise<{ archivedCount: number }>;
}
+export interface ConversationMethodDeps
+ extends Pick<MessageMethods, 'getMessages' | 'deleteMessages'> {
+ deleteAgentQueuedTurns?: (
+ user: string,
+ conversations: Array<{ conversationId: string; tenantId?: string; allTenants?: true }>,
+ ) => Promise<void>;
+}
+
export function createConversationMethods(
mongoose: typeof import('mongoose'),
- messageMethods?: Pick<MessageMethods, 'getMessages' | 'deleteMessages'>,
+ deps?: ConversationMethodDeps,
): ConversationMethods {
let legacyReceiptExpiryCursor: Types.ObjectId | undefined;
function getMessageMethods() {
- if (!messageMethods) {
+ if (!deps) {
throw new Error('Message methods not injected into conversation methods');
}
- return messageMethods;
+ return deps;
}
function getVisibleConversationRetentionFilter(): FilterQuery<IConversation> {
@@ -2865,7 +2873,10 @@ export function createConversationMethods(
const Conversation = mongoose.models.Conversation as Model<IConversation>;
const { deleteMessages, getMessages } = getMessageMethods();
const userFilter = { ...filter, user };
- type DeletionConversation = Pick<IConversation, 'conversationId' | 'chatProjectId' | 'tags'>;
+ type DeletionConversation = Pick<
+ IConversation,
+ 'conversationId' | 'tenantId' | 'chatProjectId' | 'tags'
+ >;
const retryCascadeOperation = async <T>(operation: () => PromiseLike<T> | T): Promise<T> => {
let lastError: unknown;
for (let attempt = 1; attempt <= 3; attempt += 1) {
@@ -2881,7 +2892,7 @@ export function createConversationMethods(
throw lastError;
};
let conversations = await Conversation.find(userFilter)
- .select('conversationId chatProjectId tags')
+ .select('conversationId tenantId chatProjectId tags')
.lean<DeletionConversation[]>();
const recoveryConversationIds: string[] = [];
if (!conversations.length && typeof filter.conversationId === 'string') {
@@ -2895,7 +2906,7 @@ export function createConversationMethods(
user,
'subagentThread.rootConversationId': filter.conversationId,
})
- .select('conversationId chatProjectId tags')
+ .select('conversationId tenantId chatProjectId tags')
.lean<DeletionConversation[]>(),
),
getMessages({ user, conversationId: filter.conversationId }, '_id', { limit: 1 }),
@@ -2970,6 +2981,13 @@ export function createConversationMethods(
break;
}
const waveIds = wave.map((conversation) => conversation.conversationId);
+ await deps?.deleteAgentQueuedTurns?.(
+ user,
+ wave.map((conversation) => ({
+ conversationId: conversation.conversationId,
+ ...(conversation.tenantId != null && { tenantId: conversation.tenantId }),
+ })),
+ );
await options?.beforeDelete?.(waveIds);
const result = await Conversation.deleteMany({ user, conversationId: { $in: waveIds } });
acknowledged &&= result.acknowledged;
@@ -2984,7 +3002,7 @@ export function createConversationMethods(
user,
'subagentThread.parentConversationId': { $in: waveIds },
})
- .select('conversationId chatProjectId tags')
+ .select('conversationId tenantId chatProjectId tags')
.lean<DeletionConversation[]>(),
);
}
@@ -2994,6 +3012,16 @@ export function createConversationMethods(
...deletedConversations.map((conversation) => conversation.conversationId),
];
+ if (recoveryConversationIds.length > 0) {
+ await deps?.deleteAgentQueuedTurns?.(
+ user,
+ recoveryConversationIds.map((conversationId) => ({
+ conversationId,
+ allTenants: true,
+ })),
+ );
+ }
+
const deleteConvoResult: DeleteResult = { acknowledged, deletedCount };
/**
diff --git a/packages/data-schemas/src/methods/index.ts b/packages/data-schemas/src/methods/index.ts
index 8e959a7daa..b6cc67b312 100644
--- a/packages/data-schemas/src/methods/index.ts
+++ b/packages/data-schemas/src/methods/index.ts
@@ -123,6 +123,13 @@ import {
type ValidationIssue,
} from './skill';
import { createScheduleMethods, type ScheduleMethods } from './schedule';
+import {
+ createAgentQueuedTurnMethods,
+ AgentQueuedTurnCapacityError,
+ AgentQueuedTurnConflictError,
+ AgentQueuedTurnLaneRetiredError,
+ type AgentQueuedTurnMethods,
+} from './queuedTurn';
import {
createAgentTriggerDeliveryMethods,
AgentTriggerDeliveryConflictError,
@@ -191,6 +198,11 @@ export {
export { AUDIT_SCHEMA_VERSION, MAX_AUDIT_EXPORT_ROWS, MAX_AUDIT_LOG_LIMIT, MAX_AUDIT_VERIFY_ROWS };
export { MAX_TOOL_FAVORITES };
export { AgentTriggerDeliveryConflictError };
+export {
+ AgentQueuedTurnCapacityError,
+ AgentQueuedTurnConflictError,
+ AgentQueuedTurnLaneRetiredError,
+};
export type AllMethods = UserMethods &
SessionMethods &
@@ -230,6 +242,7 @@ export type AllMethods = UserMethods &
SkillMethods &
SkillSyncMethods &
AgentTriggerDeliveryMethods &
+ AgentQueuedTurnMethods &
ScheduleMethods &
AgentMethods &
ConfigMethods &
@@ -283,9 +296,77 @@ export function createMethods(
const messageMethods = createMessageMethods(mongoose);
+ const agentQueuedTurnMethods = createAgentQueuedTurnMethods(mongoose);
+ const agentTriggerDeliveryMethods = createAgentTriggerDeliveryMethods(mongoose, {
+ purgeQueuedTurnsForUser: (user) =>
+ agentQueuedTurnMethods.deleteAllAgentQueuedTurnsForUser({
+ user: typeof user === 'string' ? new mongoose.Types.ObjectId(user) : user,
+ }),
+ });
+
const conversationMethods = createConversationMethods(mongoose, {
getMessages: messageMethods.getMessages,
deleteMessages: messageMethods.deleteMessages,
+ deleteAgentQueuedTurns: async (user, conversations) => {
+ /** Queued-turn ownership is ObjectId-backed. Conversation methods also
+ * support synthetic/non-ObjectId owners in embedded integrations and
+ * tests; those owners cannot have queued-turn rows to retire. */
+ if (!mongoose.isObjectIdOrHexString(user)) {
+ return;
+ }
+ const owner = new mongoose.Types.ObjectId(user);
+ const settledAt = new Date();
+ const deliveryKeys = await agentQueuedTurnMethods.prepareAgentQueuedTurnConversationDeletion({
+ user: owner,
+ targets: conversations,
+ settledAt,
+ });
+ await Promise.all(
+ deliveryKeys.map(async (deliveryKey) => {
+ const retirement = {
+ deliveryKey,
+ sourceId: 'agent-queued-turn',
+ reason: 'queued_turn_conversation_deleted',
+ settledAt,
+ };
+ let retired = await agentTriggerDeliveryMethods.retireAgentTriggerDelivery(retirement);
+ if (!retired) {
+ /** A delivery can exhaust immediately before owner deletion wins.
+ * Convert that operator-requeueable dead row into the same terminal
+ * retirement receipt before removing its source record. */
+ retired = await agentTriggerDeliveryMethods.retireAgentTriggerDelivery({
+ ...retirement,
+ onlyIfDead: true,
+ });
+ }
+ if (!retired) {
+ /** Successful and dead delivery receipts have bounded retention.
+ * A source in `published` proves the delivery existed, so a missing
+ * receipt after both retirement attempts is terminal absence. */
+ const fenced =
+ await agentQueuedTurnMethods.beginAgentQueuedTurnMissingDeliveryRetirement({
+ deliveryKey,
+ });
+ if (fenced) {
+ const delivery =
+ await agentTriggerDeliveryMethods.getAgentTriggerDelivery(deliveryKey);
+ if (delivery == null) {
+ retired = await agentQueuedTurnMethods.markAgentQueuedTurnMissingDeliveryRetired({
+ deliveryKey,
+ });
+ }
+ }
+ }
+ if (retired) {
+ await agentQueuedTurnMethods.markAgentQueuedTurnDeliveryRetired({ deliveryKey });
+ }
+ }),
+ );
+ await agentQueuedTurnMethods.deletePreparedAgentQueuedTurnConversations({
+ user: owner,
+ targets: conversations,
+ });
+ },
});
// ACL entry methods (used internally for removeAllPermissions)
@@ -341,7 +422,6 @@ export function createMethods(
isExternalSkillId: deps.isExternalSkillId,
};
const agentMethods = createAgentMethods(mongoose, agentDeps);
-
return {
...createUserMethods(mongoose, { getCache: deps.getCache }),
...createSessionMethods(mongoose),
@@ -383,7 +463,8 @@ export function createMethods(
...promptMethods,
...skillMethods,
...createSkillSyncMethods(mongoose),
- ...createAgentTriggerDeliveryMethods(mongoose),
+ ...agentTriggerDeliveryMethods,
+ ...agentQueuedTurnMethods,
...createScheduleMethods(mongoose),
/* Tier 5 */
...agentMethods,
@@ -455,6 +536,7 @@ export type {
UpsertSkillSyncCredentialInput,
SkillSyncMethods,
AgentTriggerDeliveryMethods,
+ AgentQueuedTurnMethods,
AgentTriggerProducerLeaseStatus,
AgentEventActorReceiptMetric,
AgentEventActorReceiptStorageMetrics,
diff --git a/packages/data-schemas/src/methods/queuedTurn.spec.ts b/packages/data-schemas/src/methods/queuedTurn.spec.ts
new file mode 100644
index 0000000000..a30965aada
--- /dev/null
+++ b/packages/data-schemas/src/methods/queuedTurn.spec.ts
@@ -0,0 +1,1667 @@
+import mongoose from 'mongoose';
+import { MongoMemoryServer } from 'mongodb-memory-server';
+import type { Model } from 'mongoose';
+import type {
+ IAgentQueuedTurnDocument,
+ IAgentQueuedTurnSequenceDocument,
+} from '~/types/queuedTurn';
+import {
+ AgentQueuedTurnCapacityError,
+ AgentQueuedTurnConflictError,
+ AgentQueuedTurnLaneRetiredError,
+ createAgentQueuedTurnMethods,
+ type AgentQueuedTurnMethods,
+} from './queuedTurn';
+import {
+ createAgentQueuedTurnModel,
+ createAgentQueuedTurnSequenceModel,
+} from '../models/queuedTurn';
+
+const DB_SETUP_TIMEOUT_MS = 60_000;
+const START = new Date('2026-08-30T12:00:00.000Z');
+const LATER = new Date(START.getTime() + 60_000);
+let mongoServer: MongoMemoryServer;
+let Turn: Model<IAgentQueuedTurnDocument>;
+let Sequence: Model<IAgentQueuedTurnSequenceDocument>;
+let methods: AgentQueuedTurnMethods;
+let user: mongoose.Types.ObjectId;
+let counter = 0;
+
+beforeAll(async () => {
+ mongoServer = await MongoMemoryServer.create();
+ await mongoose.connect(mongoServer.getUri());
+ Turn = createAgentQueuedTurnModel(mongoose);
+ Sequence = createAgentQueuedTurnSequenceModel(mongoose);
+ await Promise.all([Turn.init(), Sequence.init()]);
+ methods = createAgentQueuedTurnMethods(mongoose);
+}, DB_SETUP_TIMEOUT_MS);
+
+afterAll(async () => {
+ await mongoose.disconnect();
+ await mongoServer.stop();
+}, DB_SETUP_TIMEOUT_MS);
+
+beforeEach(async () => {
+ await Promise.all([Turn.deleteMany({}), Sequence.deleteMany({})]);
+ user = new mongoose.Types.ObjectId();
+ counter += 1;
+});
+
+function enqueueInput(
+ overrides: Partial<Parameters<typeof methods.enqueueAgentQueuedTurn>[0]> = {},
+) {
+ return {
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ agentId: 'agent-1',
+ parentMessageId: 'parent-1',
+ clientRequestId: `request-${counter}`,
+ text: 'follow up',
+ availableAt: START,
+ ...overrides,
+ };
+}
+
+function claimInput(
+ queuedTurnId: string,
+ overrides: Partial<Parameters<typeof methods.claimNextAgentQueuedTurn>[0]> = {},
+) {
+ return {
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId,
+ claimId: 'claim-1',
+ claimBy: 'worker-1',
+ now: START,
+ leaseUntil: LATER,
+ ...overrides,
+ };
+}
+
+describe('agent queued turn methods', () => {
+ it('normalizes payloads, replays the same request, and conflicts on changed content', async () => {
+ const input = enqueueInput({
+ text: ' follow up ',
+ files: [
+ { file_id: ' file-1 ', filename: ' report.pdf ' },
+ { file_id: 'file-1', filename: 'ignored duplicate' },
+ ],
+ quotes: [' quote ', '', 'second'],
+ manualSkills: [' skill-a ', 'skill-a'],
+ expectedPredecessorCreatedAt: 42,
+ });
+
+ const first = await methods.enqueueAgentQueuedTurn(input);
+ const replay = await methods.enqueueAgentQueuedTurn({
+ ...input,
+ text: 'follow up',
+ files: [{ file_id: 'file-1', filename: 'report.pdf' }],
+ quotes: ['quote', 'second'],
+ manualSkills: ['skill-a'],
+ });
+
+ expect(first.replayed).toBe(false);
+ expect(first.turn).toMatchObject({
+ queuedTurnId: expect.any(String),
+ agentId: 'agent-1',
+ parentMessageId: 'parent-1',
+ sequence: 1,
+ status: 'queued',
+ text: 'follow up',
+ files: [{ file_id: 'file-1', filename: 'report.pdf' }],
+ quotes: ['quote', 'second'],
+ manualSkills: ['skill-a'],
+ expectedPredecessorCreatedAt: 42,
+ });
+ expect(replay).toMatchObject({
+ replayed: true,
+ turn: { queuedTurnId: first.turn.queuedTurnId },
+ });
+ await expect(
+ methods.enqueueAgentQueuedTurn({ ...input, agentId: 'agent-2' }),
+ ).rejects.toBeInstanceOf(AgentQueuedTurnConflictError);
+ await expect(
+ methods.enqueueAgentQueuedTurn({
+ ...input,
+ parentMessageId: 'different-parent',
+ }),
+ ).rejects.toBeInstanceOf(AgentQueuedTurnConflictError);
+ expect(await Turn.countDocuments()).toBe(1);
+ });
+
+ it('looks up terminal receipts by owner-scoped request identity', async () => {
+ const created = await methods.enqueueAgentQueuedTurn(enqueueInput());
+
+ await expect(
+ methods.getAgentQueuedTurnByClientRequestId({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ clientRequestId: created.turn.clientRequestId,
+ }),
+ ).resolves.toMatchObject({ queuedTurnId: created.turn.queuedTurnId });
+ await expect(
+ methods.getAgentQueuedTurnByClientRequestId({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'another-conversation',
+ clientRequestId: created.turn.clientRequestId,
+ }),
+ ).resolves.toBeNull();
+ });
+
+ it('allocates monotonic conversation sequences under concurrent enqueue', async () => {
+ const results = await Promise.all(
+ Array.from({ length: 8 }, (_, index) =>
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({
+ clientRequestId: `concurrent-${index}`,
+ text: `turn ${index}`,
+ }),
+ ),
+ ),
+ );
+
+ const sequences = results.map(({ turn }) => turn.sequence).sort((a, b) => a - b);
+ expect(new Set(sequences).size).toBe(8);
+ expect(sequences.every((value, index) => index === 0 || value > sequences[index - 1])).toBe(
+ true,
+ );
+ });
+
+ it('blocks claims behind a visible reservation and repairs it oldest-first', async () => {
+ const first = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'visible-first' }),
+ );
+ const lane = await Sequence.findOne({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ }).lean();
+ if (lane?.laneId == null) {
+ throw new Error('Expected a durable queued-turn lane');
+ }
+ await Turn.create({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ agentId: 'agent-1',
+ parentMessageId: 'parent-1',
+ clientRequestId: 'interrupted-reservation',
+ fingerprint: 'x'.repeat(43),
+ laneId: lane.laneId,
+ activeSlot: ((first.turn.activeSlot ?? 0) + 1) % 100,
+ status: 'reserving',
+ priority: false,
+ text: 'repair me',
+ attempts: 0,
+ availableAt: START,
+ deliveryState: 'pending',
+ });
+
+ await expect(
+ methods.claimNextAgentQueuedTurn(claimInput(first.turn.queuedTurnId)),
+ ).resolves.toMatchObject({ outcome: 'blocked' });
+ await methods.findQueuedTurnsNeedingDelivery();
+ await expect(
+ methods.claimNextAgentQueuedTurn(claimInput(first.turn.queuedTurnId)),
+ ).resolves.toMatchObject({ outcome: 'acquired' });
+ await expect(
+ Turn.findOne({ clientRequestId: 'interrupted-reservation' }).lean(),
+ ).resolves.toMatchObject({ status: 'queued', sequence: expect.any(Number) });
+ });
+
+ it('caps each conversation at 100 active turns while preserving exact replay', async () => {
+ const inputs = Array.from({ length: 101 }, (_, index) =>
+ enqueueInput({ clientRequestId: `capacity-${index}`, text: `turn ${index}` }),
+ );
+ const results = await Promise.allSettled(
+ inputs.map((input) => methods.enqueueAgentQueuedTurn(input)),
+ );
+
+ expect(results.filter((result) => result.status === 'fulfilled')).toHaveLength(100);
+ const rejected = results.filter(
+ (result): result is PromiseRejectedResult => result.status === 'rejected',
+ );
+ expect(rejected).toHaveLength(1);
+ expect(rejected[0].reason).toBeInstanceOf(AgentQueuedTurnCapacityError);
+
+ const first = results.find(
+ (
+ result,
+ ): result is PromiseFulfilledResult<
+ Awaited<ReturnType<typeof methods.enqueueAgentQueuedTurn>>
+ > => result.status === 'fulfilled',
+ );
+ if (first == null) {
+ throw new Error('Expected one successful enqueue');
+ }
+ const replayInput = inputs.find(
+ (input) => input.clientRequestId === first.value.turn.clientRequestId,
+ );
+ if (replayInput == null) {
+ throw new Error('Expected the matching replay input');
+ }
+ await expect(methods.enqueueAgentQueuedTurn(replayInput)).resolves.toMatchObject({
+ replayed: true,
+ turn: { queuedTurnId: first.value.turn.queuedTurnId },
+ });
+ expect(await methods.listActiveAgentQueuedTurns(enqueueInput())).toHaveLength(100);
+ });
+
+ it('reuses a terminalized active slot and isolates capacity by owner scope', async () => {
+ const queued = await Promise.all(
+ Array.from({ length: 100 }, (_, index) =>
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: `slot-${index}`, text: `turn ${index}` }),
+ ),
+ ),
+ );
+ await expect(
+ methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued[0].turn.queuedTurnId,
+ settledAt: START,
+ }),
+ ).resolves.toMatchObject({ outcome: 'cancelled' });
+ await expect(
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'reused-slot', text: 'replacement' }),
+ ),
+ ).resolves.toMatchObject({ replayed: false, turn: { status: 'queued' } });
+ await expect(
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'capacity-overflow', text: 'overflow' }),
+ ),
+ ).rejects.toBeInstanceOf(AgentQueuedTurnCapacityError);
+
+ await expect(
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({
+ conversationId: 'conversation-2',
+ clientRequestId: 'other-conversation',
+ }),
+ ),
+ ).resolves.toMatchObject({ replayed: false });
+ await expect(
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ tenantId: 'tenant-2', clientRequestId: 'other-tenant' }),
+ ),
+ ).resolves.toMatchObject({ replayed: false });
+ await expect(
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ user: new mongoose.Types.ObjectId(), clientRequestId: 'other-user' }),
+ ),
+ ).resolves.toMatchObject({ replayed: false });
+ });
+
+ it('keeps untenant records in the untenant owner scope', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ tenantId: undefined, clientRequestId: 'untenant' }),
+ );
+
+ expect(queued.turn.tenantId).toBeUndefined();
+ await expect(
+ methods.listActiveAgentQueuedTurns({
+ user,
+ conversationId: 'conversation-1',
+ }),
+ ).resolves.toMatchObject([{ queuedTurnId: queued.turn.queuedTurnId }]);
+ await expect(
+ methods.listActiveAgentQueuedTurns({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ }),
+ ).resolves.toEqual([]);
+ });
+
+ it('lists and claims priority then FIFO without skipping the expected head', async () => {
+ const first = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'first', text: 'first' }),
+ );
+ const second = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'second', text: 'second' }),
+ );
+ const priority = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({
+ clientRequestId: 'priority',
+ text: 'priority',
+ priority: true,
+ }),
+ );
+
+ const active = await methods.listActiveAgentQueuedTurns(enqueueInput());
+ expect(active.map((turn) => turn.clientRequestId)).toEqual(['priority', 'first', 'second']);
+ await expect(
+ methods.claimNextAgentQueuedTurn(claimInput(first.turn.queuedTurnId)),
+ ).resolves.toEqual({
+ outcome: 'blocked',
+ claim: null,
+ });
+
+ const acquired = await methods.claimNextAgentQueuedTurn(claimInput(priority.turn.queuedTurnId));
+ expect(acquired).toMatchObject({
+ outcome: 'acquired',
+ claim: { queuedTurnId: priority.turn.queuedTurnId, attempts: 1 },
+ });
+ const replayed = await methods.claimNextAgentQueuedTurn(claimInput(priority.turn.queuedTurnId));
+ expect(replayed).toMatchObject({
+ outcome: 'replayed',
+ claim: { queuedTurnId: priority.turn.queuedTurnId, attempts: 1 },
+ });
+ await expect(
+ methods.claimNextAgentQueuedTurn(
+ claimInput(second.turn.queuedTurnId, {
+ claimId: 'other',
+ claimBy: 'worker-2',
+ }),
+ ),
+ ).resolves.toEqual({ outcome: 'blocked', claim: null });
+ });
+
+ it('reclaims an expired exact lease but never replays a different claim identity', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(enqueueInput());
+ const first = await methods.claimNextAgentQueuedTurn(
+ claimInput(queued.turn.queuedTurnId, {
+ now: START,
+ leaseUntil: new Date(START.getTime() + 1),
+ }),
+ );
+ expect(first.outcome).toBe('acquired');
+
+ const reclaimed = await methods.claimNextAgentQueuedTurn(
+ claimInput(queued.turn.queuedTurnId, {
+ claimId: 'claim-2',
+ claimBy: 'worker-2',
+ now: new Date(START.getTime() + 2),
+ leaseUntil: LATER,
+ }),
+ );
+ expect(reclaimed).toMatchObject({
+ outcome: 'acquired',
+ claim: { attempts: 2 },
+ });
+ });
+
+ it('repairs delivery scheduling idempotently and leaves unscheduled replays discoverable', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(enqueueInput());
+ expect(await methods.findQueuedTurnsNeedingDelivery()).toMatchObject([
+ { queuedTurnId: queued.turn.queuedTurnId },
+ ]);
+ const replay = await methods.enqueueAgentQueuedTurn(enqueueInput());
+ expect(replay.replayed).toBe(true);
+ expect(replay.turn).not.toHaveProperty('scheduledAt');
+
+ await expect(
+ methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-1',
+ }),
+ ).resolves.toMatchObject({ outcome: 'reserved' });
+ const scheduled = await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-1',
+ scheduledAt: START,
+ });
+ expect(scheduled).toMatchObject({
+ outcome: 'scheduled',
+ turn: { deliveryKey: 'delivery-1', scheduledAt: START },
+ });
+ await expect(
+ methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-1',
+ }),
+ ).resolves.toMatchObject({ outcome: 'already_reserved' });
+ await expect(
+ methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-1',
+ }),
+ ).resolves.toMatchObject({ outcome: 'already_scheduled' });
+ await expect(
+ methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-2',
+ }),
+ ).resolves.toMatchObject({ outcome: 'conflict' });
+ await expect(
+ methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-2',
+ }),
+ ).resolves.toMatchObject({ outcome: 'conflict' });
+ expect(await methods.findQueuedTurnsNeedingDelivery()).toEqual([]);
+ });
+
+ it('acknowledges a reserved delivery after a worker claims the queue row', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(enqueueInput());
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-fast',
+ });
+ await methods.claimNextAgentQueuedTurn(claimInput(queued.turn.queuedTurnId));
+
+ await expect(
+ methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-fast',
+ scheduledAt: START,
+ }),
+ ).resolves.toMatchObject({
+ outcome: 'scheduled',
+ turn: { status: 'claimed', deliveryKey: 'delivery-fast', scheduledAt: START },
+ });
+ });
+
+ it('cancels claimed work until its admission fence starts', async () => {
+ const cancellable = await methods.enqueueAgentQueuedTurn(enqueueInput());
+ const cancelled = await methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: cancellable.turn.queuedTurnId,
+ settledAt: START,
+ });
+ expect(cancelled).toMatchObject({
+ outcome: 'cancelled',
+ turn: { status: 'cancelled' },
+ });
+ await expect(
+ methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: cancellable.turn.queuedTurnId,
+ }),
+ ).resolves.toMatchObject({ outcome: 'already_cancelled' });
+ await expect(
+ methods.claimNextAgentQueuedTurn(claimInput(cancellable.turn.queuedTurnId)),
+ ).resolves.toEqual({ outcome: 'missing', claim: null });
+
+ const claimed = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'claimed', text: 'claimed' }),
+ );
+ await methods.claimNextAgentQueuedTurn(claimInput(claimed.turn.queuedTurnId));
+ await expect(
+ methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: claimed.turn.queuedTurnId,
+ }),
+ ).resolves.toMatchObject({ outcome: 'cancelled', turn: { status: 'cancelled' } });
+
+ const admitting = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'admitting', text: 'admitting' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: admitting.turn.queuedTurnId,
+ deliveryKey: 'admission-cancel',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: admitting.turn.queuedTurnId,
+ deliveryKey: 'admission-cancel',
+ scheduledAt: START,
+ });
+ await methods.claimNextAgentQueuedTurn(claimInput(admitting.turn.queuedTurnId));
+ await methods.beginAgentQueuedTurnAdmission({
+ ...claimInput(admitting.turn.queuedTurnId),
+ admissionId: 'admission-cancel',
+ startedAt: START,
+ });
+ await expect(
+ methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: admitting.turn.queuedTurnId,
+ }),
+ ).resolves.toMatchObject({ outcome: 'not_cancellable' });
+ });
+
+ it('retries, dead-letters, and admits only through the exact claim fence', async () => {
+ const retry = await methods.enqueueAgentQueuedTurn(enqueueInput());
+ await methods.claimNextAgentQueuedTurn(claimInput(retry.turn.queuedTurnId));
+ await expect(
+ methods.releaseAgentQueuedTurn({
+ ...claimInput(retry.turn.queuedTurnId),
+ disposition: 'retry',
+ availableAt: LATER,
+ }),
+ ).resolves.toMatchObject({
+ outcome: 'released',
+ turn: { status: 'queued', availableAt: LATER },
+ });
+
+ const dead = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'dead', text: 'dead', priority: true }),
+ );
+ await methods.claimNextAgentQueuedTurn(claimInput(dead.turn.queuedTurnId));
+ await expect(
+ methods.releaseAgentQueuedTurn({
+ ...claimInput(dead.turn.queuedTurnId),
+ disposition: 'dead',
+ settledAt: START,
+ failure: { code: 'BROKEN', message: 'cannot continue' },
+ }),
+ ).resolves.toMatchObject({
+ outcome: 'dead',
+ turn: {
+ status: 'dead',
+ terminalReceipt: { outcome: 'dead', failure: { code: 'BROKEN' } },
+ },
+ });
+ await methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: dead.turn.queuedTurnId,
+ settledAt: START,
+ });
+
+ const admitted = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({
+ clientRequestId: 'admitted',
+ text: 'admitted',
+ priority: true,
+ }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: admitted.turn.queuedTurnId,
+ deliveryKey: 'admission-1',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: admitted.turn.queuedTurnId,
+ deliveryKey: 'admission-1',
+ scheduledAt: START,
+ });
+ await methods.claimNextAgentQueuedTurn(claimInput(admitted.turn.queuedTurnId));
+ const admissionInput = {
+ ...claimInput(admitted.turn.queuedTurnId),
+ admissionId: 'admission-1',
+ admissionMode: 'ordinary' as const,
+ generationId: 'generation-1',
+ generationCreatedAt: 84,
+ settledAt: START,
+ };
+ await expect(
+ methods.beginAgentQueuedTurnAdmission({
+ ...claimInput(admitted.turn.queuedTurnId),
+ admissionId: 'admission-1',
+ startedAt: START,
+ }),
+ ).resolves.toMatchObject({ outcome: 'started' });
+ await expect(methods.markAgentQueuedTurnAdmitted(admissionInput)).resolves.toMatchObject({
+ outcome: 'admitted',
+ turn: {
+ status: 'admitted',
+ deliveryState: 'published',
+ terminalReceipt: { admissionId: 'admission-1' },
+ },
+ });
+ await expect(
+ methods.hasAgentQueuedTurnAdmissionReceipt({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: admitted.turn.queuedTurnId,
+ admissionId: 'admission-1',
+ generationId: 'generation-1',
+ generationCreatedAt: 84,
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.hasAgentQueuedTurnAdmissionReceipt({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: admitted.turn.queuedTurnId,
+ admissionId: 'admission-1',
+ generationId: 'generation-1',
+ generationCreatedAt: 85,
+ }),
+ ).resolves.toBe(false);
+ await expect(methods.markAgentQueuedTurnAdmitted(admissionInput)).resolves.toMatchObject({
+ outcome: 'already_admitted',
+ });
+ await expect(
+ methods.markAgentQueuedTurnAdmitted({
+ ...admissionInput,
+ admissionId: 'different',
+ }),
+ ).resolves.toMatchObject({ outcome: 'conflict' });
+ });
+
+ it('lists a dead receipt until the user dismisses it', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(enqueueInput());
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-dead',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-dead',
+ scheduledAt: START,
+ });
+ await expect(
+ methods.deadLetterAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-dead',
+ settledAt: START,
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'could not admit turn' },
+ }),
+ ).resolves.toMatchObject({ outcome: 'dead' });
+
+ await expect(
+ methods.listAgentQueuedTurnReceipts({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ }),
+ ).resolves.toMatchObject([
+ {
+ queuedTurnId: queued.turn.queuedTurnId,
+ status: 'dead',
+ terminalReceipt: {
+ outcome: 'dead',
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'could not admit turn' },
+ },
+ },
+ ]);
+ await expect(
+ methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ settledAt: LATER,
+ }),
+ ).resolves.toMatchObject({ outcome: 'cancelled', turn: { status: 'cancelled' } });
+ await expect(
+ methods.listAgentQueuedTurnReceipts({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ }),
+ ).resolves.toEqual([]);
+ });
+
+ it('keeps an ambiguous admission fenced when its delivery exhausts', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'ambiguous-admission' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-ambiguous-admission',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-ambiguous-admission',
+ scheduledAt: START,
+ });
+ await methods.claimNextAgentQueuedTurn(
+ claimInput(queued.turn.queuedTurnId, { claimId: 'delivery-ambiguous-admission' }),
+ );
+ await methods.beginAgentQueuedTurnAdmission({
+ ...claimInput(queued.turn.queuedTurnId, { claimId: 'delivery-ambiguous-admission' }),
+ admissionId: 'delivery-ambiguous-admission',
+ startedAt: START,
+ });
+
+ await expect(
+ methods.claimNextAgentQueuedTurn(
+ claimInput(queued.turn.queuedTurnId, {
+ claimId: 'replacement-delivery',
+ claimBy: 'replacement-worker',
+ now: LATER,
+ leaseUntil: new Date(LATER.getTime() + 60_000),
+ }),
+ ),
+ ).resolves.toEqual({ outcome: 'blocked', claim: null });
+ await expect(
+ methods.claimQueuedTurnsForAdmissionReconciliation({
+ claimId: 'reconciliation-owner',
+ claimBy: 'reconciliation-worker',
+ now: LATER,
+ leaseUntil: new Date(LATER.getTime() + 60_000),
+ limit: 1,
+ }),
+ ).resolves.toMatchObject([
+ {
+ queuedTurnId: queued.turn.queuedTurnId,
+ status: 'claimed',
+ reconciliationClaimId: 'reconciliation-owner',
+ },
+ ]);
+
+ await expect(
+ methods.deadLetterAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-ambiguous-admission',
+ settledAt: LATER,
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'admission result unavailable' },
+ }),
+ ).resolves.toMatchObject({
+ outcome: 'admission_indeterminate',
+ turn: {
+ status: 'dead',
+ admissionId: 'delivery-ambiguous-admission',
+ deliveryState: 'published',
+ terminalReceipt: { failure: { code: 'ADMISSION_INDETERMINATE' } },
+ },
+ });
+ await expect(
+ methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ }),
+ ).resolves.toMatchObject({ outcome: 'not_cancellable' });
+ await expect(
+ methods.claimNextAgentQueuedTurn(
+ claimInput(queued.turn.queuedTurnId, {
+ claimId: 'delivery-ambiguous-admission',
+ now: LATER,
+ leaseUntil: new Date(LATER.getTime() + 60_000),
+ }),
+ ),
+ ).resolves.toMatchObject({ outcome: 'missing' });
+ await expect(
+ methods.prepareAgentQueuedTurnConversationDeletion({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ }),
+ ).rejects.toThrow('admission must settle');
+ });
+
+ it('reconciles an ambiguous admission from durable generation evidence', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'reconciled-admission' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-reconciled-admission',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-reconciled-admission',
+ scheduledAt: START,
+ });
+ await methods.claimNextAgentQueuedTurn(
+ claimInput(queued.turn.queuedTurnId, { claimId: 'delivery-reconciled-admission' }),
+ );
+ await methods.beginAgentQueuedTurnAdmission({
+ ...claimInput(queued.turn.queuedTurnId, { claimId: 'delivery-reconciled-admission' }),
+ admissionId: 'delivery-reconciled-admission',
+ startedAt: START,
+ });
+
+ await expect(
+ methods.deadLetterAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-reconciled-admission',
+ settledAt: LATER,
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'admission result unavailable' },
+ admissionEvidence: {
+ generationId: 'generation-reconciled',
+ generationCreatedAt: 42,
+ },
+ }),
+ ).resolves.toMatchObject({
+ outcome: 'admission_reconciled',
+ turn: {
+ status: 'admitted',
+ terminalReceipt: {
+ admissionId: 'delivery-reconciled-admission',
+ generationId: 'generation-reconciled',
+ generationCreatedAt: 42,
+ },
+ },
+ });
+ });
+
+ it('reconciles a quarantined admission after exact evidence arrives and unblocks its successor', async () => {
+ const first = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'late-reconciled-admission' }),
+ );
+ const successor = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'late-reconciled-successor' }),
+ );
+ const deliveryKey = 'delivery-late-reconciled-admission';
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: first.turn.queuedTurnId,
+ deliveryKey,
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: first.turn.queuedTurnId,
+ deliveryKey,
+ scheduledAt: START,
+ });
+ await methods.claimNextAgentQueuedTurn(
+ claimInput(first.turn.queuedTurnId, { claimId: deliveryKey }),
+ );
+ await methods.beginAgentQueuedTurnAdmission({
+ ...claimInput(first.turn.queuedTurnId, { claimId: deliveryKey }),
+ admissionId: deliveryKey,
+ startedAt: START,
+ admissionProtocolVersion: 2,
+ });
+ await methods.deadLetterAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: first.turn.queuedTurnId,
+ deliveryKey,
+ settledAt: LATER,
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'admission result unavailable' },
+ });
+
+ await expect(
+ methods.claimQueuedTurnsForAdmissionReconciliation({
+ claimId: 'reconciliation-1',
+ claimBy: 'reconciler-1',
+ now: new Date(LATER.getTime() + 1),
+ leaseUntil: new Date(LATER.getTime() + 60_001),
+ }),
+ ).resolves.toMatchObject([{ queuedTurnId: first.turn.queuedTurnId }]);
+ await expect(
+ methods.claimNextAgentQueuedTurn(
+ claimInput(successor.turn.queuedTurnId, { claimId: 'successor-delivery' }),
+ ),
+ ).resolves.toMatchObject({ outcome: 'blocked' });
+ await expect(
+ methods.deadLetterAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: first.turn.queuedTurnId,
+ deliveryKey,
+ settledAt: LATER,
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'admission result unavailable' },
+ admissionEvidence: {
+ generationId: 'generation-late-reconciled',
+ generationCreatedAt: 43,
+ },
+ }),
+ ).resolves.toMatchObject({ outcome: 'admission_reconciled', turn: { status: 'admitted' } });
+ await expect(
+ methods.claimQueuedTurnsForAdmissionReconciliation({
+ claimId: 'reconciliation-2',
+ claimBy: 'reconciler-1',
+ now: new Date(LATER.getTime() + 1),
+ leaseUntil: new Date(LATER.getTime() + 60_001),
+ }),
+ ).resolves.toEqual([]);
+ await expect(
+ methods.claimNextAgentQueuedTurn(
+ claimInput(successor.turn.queuedTurnId, { claimId: 'successor-delivery' }),
+ ),
+ ).resolves.toMatchObject({ outcome: 'acquired' });
+ });
+
+ it('leases reconciliation fairly while preserving a late exact receipt', async () => {
+ const quarantine = async (conversationId: string, clientRequestId: string) => {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ conversationId, clientRequestId }),
+ );
+ const deliveryKey = `delivery-${clientRequestId}`;
+ const scope = { user, tenantId: 'tenant-1', conversationId };
+ await methods.reserveAgentQueuedTurnDelivery({
+ ...scope,
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey,
+ });
+ await methods.markQueuedTurnScheduled({
+ ...scope,
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey,
+ scheduledAt: START,
+ });
+ const claim = {
+ ...scope,
+ queuedTurnId: queued.turn.queuedTurnId,
+ claimId: deliveryKey,
+ claimBy: 'worker-1',
+ now: START,
+ leaseUntil: LATER,
+ };
+ await methods.claimNextAgentQueuedTurn(claim);
+ await methods.beginAgentQueuedTurnAdmission({
+ ...claim,
+ admissionId: deliveryKey,
+ startedAt: START,
+ admissionProtocolVersion: 2,
+ });
+ await methods.deadLetterAgentQueuedTurn({
+ ...scope,
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey,
+ settledAt: LATER,
+ failure: { code: 'ATTEMPTS_EXHAUSTED', message: 'admission result unavailable' },
+ });
+ return { queued, deliveryKey, claim };
+ };
+ const first = await quarantine('conversation-1', 'reconciliation-fair-1');
+ const second = await quarantine('conversation-2', 'reconciliation-fair-2');
+ const claimNow = new Date(LATER.getTime() + 1);
+ const firstBatch = await methods.claimQueuedTurnsForAdmissionReconciliation({
+ claimId: 'reconciliation-fair-claim-1',
+ claimBy: 'reconciler-1',
+ now: claimNow,
+ leaseUntil: new Date(claimNow.getTime() + 60_000),
+ limit: 1,
+ });
+ expect(firstBatch).toMatchObject([{ queuedTurnId: first.queued.turn.queuedTurnId }]);
+ await methods.deferAgentQueuedTurnAdmissionReconciliation({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: first.queued.turn.queuedTurnId,
+ deliveryKey: first.deliveryKey,
+ claimId: 'reconciliation-fair-claim-1',
+ claimBy: 'reconciler-1',
+ availableAt: new Date(claimNow.getTime() + 60_000),
+ });
+
+ const secondBatch = await methods.claimQueuedTurnsForAdmissionReconciliation({
+ claimId: 'reconciliation-fair-claim-2',
+ claimBy: 'reconciler-1',
+ now: claimNow,
+ leaseUntil: new Date(claimNow.getTime() + 60_000),
+ limit: 1,
+ });
+ expect(secondBatch).toMatchObject([{ queuedTurnId: second.queued.turn.queuedTurnId }]);
+ await expect(
+ methods.deferAgentQueuedTurnAdmissionReconciliation({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-2',
+ queuedTurnId: second.queued.turn.queuedTurnId,
+ deliveryKey: second.deliveryKey,
+ claimId: 'reconciliation-fair-claim-2',
+ claimBy: 'reconciler-1',
+ availableAt: new Date(claimNow.getTime() + 60_000),
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.markAgentQueuedTurnAdmitted({
+ ...second.claim,
+ admissionId: second.deliveryKey,
+ admissionMode: 'ordinary',
+ generationId: 'generation-late-proof',
+ generationCreatedAt: 44,
+ settledAt: new Date(claimNow.getTime() + 1),
+ }),
+ ).resolves.toMatchObject({ outcome: 'admitted', turn: { status: 'admitted' } });
+ });
+
+ it('scopes effective predecessor epochs to one captured queue root', async () => {
+ const first = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'root-a-1', expectedPredecessorCreatedAt: 10 }),
+ );
+ const sameRoot = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'root-a-2', expectedPredecessorCreatedAt: 10 }),
+ );
+ const laterRoot = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'root-b-1', expectedPredecessorCreatedAt: 20 }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: first.turn.queuedTurnId,
+ deliveryKey: 'admission-root-a',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: first.turn.queuedTurnId,
+ deliveryKey: 'admission-root-a',
+ scheduledAt: START,
+ });
+ await methods.claimNextAgentQueuedTurn(claimInput(first.turn.queuedTurnId));
+ await methods.beginAgentQueuedTurnAdmission({
+ ...claimInput(first.turn.queuedTurnId),
+ admissionId: 'admission-root-a',
+ startedAt: START,
+ });
+ await methods.markAgentQueuedTurnAdmitted({
+ ...claimInput(first.turn.queuedTurnId),
+ admissionId: 'admission-root-a',
+ admissionMode: 'ordinary',
+ generationCreatedAt: 11,
+ settledAt: START,
+ });
+
+ await expect(
+ methods.getEffectiveAgentQueuedTurnPredecessor({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ sequence: sameRoot.turn.sequence,
+ expectedPredecessorCreatedAt: 10,
+ }),
+ ).resolves.toBe(11);
+ await expect(
+ methods.getEffectiveAgentQueuedTurnPredecessor({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ sequence: laterRoot.turn.sequence,
+ expectedPredecessorCreatedAt: 20,
+ }),
+ ).resolves.toBeUndefined();
+ });
+
+ it('projects the newest failures when more than 100 dead receipts await dismissal', async () => {
+ const deadIds: string[] = [];
+ for (let index = 0; index < 101; index++) {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: `dead-${index}`, text: `dead ${index}` }),
+ );
+ const deliveryKey = `delivery-${index}`;
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey,
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey,
+ scheduledAt: START,
+ });
+ await methods.deadLetterAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey,
+ settledAt: START,
+ failure: { code: 'FAILED', message: `failure ${index}` },
+ });
+ deadIds.push(queued.turn.queuedTurnId);
+ }
+
+ const receipts = await methods.listAgentQueuedTurnReceipts({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ });
+ expect(receipts).toHaveLength(100);
+ expect(receipts.map((turn) => turn.queuedTurnId)).toContain(deadIds[100]);
+ expect(receipts.map((turn) => turn.queuedTurnId)).not.toContain(deadIds[0]);
+ });
+
+ it('isolates owner, tenant, and conversation scopes and drains or deletes exact scopes', async () => {
+ const otherUser = new mongoose.Types.ObjectId();
+ await Promise.all([
+ methods.enqueueAgentQueuedTurn(enqueueInput({ clientRequestId: 'keep', text: 'keep' })),
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({
+ clientRequestId: 'other-conversation',
+ conversationId: 'conversation-2',
+ text: 'other conversation',
+ }),
+ ),
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({
+ clientRequestId: 'other-user',
+ user: otherUser,
+ text: 'other user',
+ }),
+ ),
+ ]);
+
+ expect(
+ await methods.drainAgentQueuedTurns({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ settledAt: START,
+ }),
+ ).toBe(1);
+ expect(
+ await methods.deleteAgentQueuedTurns({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ }),
+ ).toBe(1);
+ expect(await Turn.countDocuments()).toBe(2);
+ expect(await Sequence.countDocuments()).toBe(2);
+ });
+
+ it('purges a user across every tenant while preserving other users', async () => {
+ const otherUser = new mongoose.Types.ObjectId();
+ await Promise.all([
+ methods.enqueueAgentQueuedTurn(enqueueInput({ clientRequestId: 'tenant-one' })),
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ tenantId: 'tenant-2', clientRequestId: 'tenant-two' }),
+ ),
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ tenantId: undefined, clientRequestId: 'untenant' }),
+ ),
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ user: otherUser, clientRequestId: 'other-user' }),
+ ),
+ ]);
+
+ await expect(methods.deleteAllAgentQueuedTurnsForUser({ user })).resolves.toBe(3);
+ expect(await Turn.countDocuments({ user })).toBe(0);
+ expect(await Sequence.countDocuments({ user })).toBe(0);
+ expect(await Turn.countDocuments({ user: otherUser })).toBe(1);
+ expect(await Sequence.countDocuments({ user: otherUser })).toBe(1);
+ });
+
+ it('prepares conversation deletion for replay-safe delivery retirement before payload purge', async () => {
+ const tenantOne = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'delete-tenant-one' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: tenantOne.turn.queuedTurnId,
+ deliveryKey: 'delivery-delete-one',
+ });
+ await Sequence.updateOne(
+ { user, tenantId: 'tenant-1', conversationId: 'conversation-1' },
+ {
+ $set: {
+ writerId: 'enqueue-in-flight',
+ writerUntil: new Date(Date.now() + 25),
+ },
+ },
+ );
+ await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ tenantId: 'tenant-2', clientRequestId: 'delete-tenant-two' }),
+ );
+ await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ conversationId: 'conversation-2', clientRequestId: 'keep-conversation' }),
+ );
+
+ await expect(
+ methods.prepareAgentQueuedTurnConversationDeletion({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ settledAt: START,
+ }),
+ ).resolves.toEqual(['delivery-delete-one']);
+ await expect(
+ Sequence.findOne({ user, tenantId: 'tenant-1', conversationId: 'conversation-1' }).lean(),
+ ).resolves.toMatchObject({ retiredAt: START });
+ expect(
+ await Turn.findById(tenantOne.turn.queuedTurnId).select('status terminalReceipt').lean(),
+ ).toMatchObject({
+ status: 'cancelled',
+ terminalReceipt: { failure: { code: 'OWNER_DRAINED' } },
+ });
+ await expect(methods.findQueuedTurnsNeedingDelivery()).resolves.toEqual(
+ expect.arrayContaining([
+ expect.objectContaining({
+ queuedTurnId: tenantOne.turn.queuedTurnId,
+ deliveryState: 'publishing',
+ }),
+ ]),
+ );
+ await expect(
+ methods.deletePreparedAgentQueuedTurnConversations({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ }),
+ ).rejects.toThrow('deliveries must retire');
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: tenantOne.turn.queuedTurnId,
+ deliveryKey: 'delivery-delete-one',
+ scheduledAt: START,
+ });
+ await expect(
+ methods.markAgentQueuedTurnDeliveryRetired({ deliveryKey: 'delivery-delete-one' }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.deletePreparedAgentQueuedTurnConversations({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ }),
+ ).resolves.toBe(1);
+ expect(await Turn.countDocuments({ user, conversationId: 'conversation-1' })).toBe(1);
+ await expect(
+ methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'cannot-reopen-deleted-lane' }),
+ ),
+ ).rejects.toBeInstanceOf(AgentQueuedTurnLaneRetiredError);
+
+ await methods.prepareAgentQueuedTurnConversationDeletion({
+ user,
+ targets: [{ conversationId: 'conversation-1', allTenants: true }],
+ settledAt: START,
+ });
+ await methods.deletePreparedAgentQueuedTurnConversations({
+ user,
+ targets: [{ conversationId: 'conversation-1', allTenants: true }],
+ });
+ expect(await Turn.countDocuments({ user, conversationId: 'conversation-1' })).toBe(0);
+ expect(await Turn.countDocuments({ user, conversationId: 'conversation-2' })).toBe(1);
+ });
+
+ it('includes preexisting dead deliveries in conversation retirement', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'dead-before-delete' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-dead-before-delete',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-dead-before-delete',
+ scheduledAt: START,
+ });
+ const claim = await methods.claimNextAgentQueuedTurn(claimInput(queued.turn.queuedTurnId));
+ expect(claim.outcome).toBe('acquired');
+ if (claim.outcome !== 'acquired') {
+ throw new Error('Expected queued-turn claim');
+ }
+ await methods.releaseAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ claimId: claim.claim.claimId,
+ claimBy: claim.claim.claimBy,
+ disposition: 'dead',
+ settledAt: START,
+ failure: { code: 'TEST_DEAD', message: 'dead before conversation deletion' },
+ });
+
+ await expect(
+ methods.prepareAgentQueuedTurnConversationDeletion({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ }),
+ ).resolves.toEqual(['delivery-dead-before-delete']);
+ });
+
+ it('keeps an admission-crossing delivery and its conversation until admission settles', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'admission-during-delete' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-admission-during-delete',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-admission-during-delete',
+ scheduledAt: START,
+ });
+ await methods.claimNextAgentQueuedTurn(claimInput(queued.turn.queuedTurnId));
+ await expect(
+ methods.beginAgentQueuedTurnAdmission({
+ ...claimInput(queued.turn.queuedTurnId),
+ admissionId: 'delivery-admission-during-delete',
+ startedAt: START,
+ }),
+ ).resolves.toMatchObject({ outcome: 'started' });
+
+ await expect(
+ methods.prepareAgentQueuedTurnConversationDeletion({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ settledAt: START,
+ }),
+ ).rejects.toThrow('admission must settle');
+ await expect(Turn.findById(queued.turn.queuedTurnId).lean()).resolves.toMatchObject({
+ status: 'claimed',
+ admissionId: 'delivery-admission-during-delete',
+ });
+ await expect(
+ Sequence.findOne({ user, tenantId: 'tenant-1', conversationId: 'conversation-1' }).lean(),
+ ).resolves.not.toHaveProperty('retiredAt');
+
+ await methods.markAgentQueuedTurnAdmitted({
+ ...claimInput(queued.turn.queuedTurnId),
+ admissionId: 'delivery-admission-during-delete',
+ admissionMode: 'ordinary',
+ generationCreatedAt: 42,
+ settledAt: LATER,
+ });
+ await expect(
+ methods.prepareAgentQueuedTurnConversationDeletion({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ settledAt: LATER,
+ }),
+ ).resolves.toEqual(['delivery-admission-during-delete']);
+ await expect(
+ methods.deletePreparedAgentQueuedTurnConversations({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ }),
+ ).rejects.toThrow('deliveries must retire');
+ await expect(
+ methods.markAgentQueuedTurnDeliveryRetired({
+ deliveryKey: 'delivery-admission-during-delete',
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.deletePreparedAgentQueuedTurnConversations({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ }),
+ ).resolves.toBe(1);
+ });
+
+ it('fences a stale writer that resumes after conversation deletion', async () => {
+ const remoteMethods = createAgentQueuedTurnMethods(mongoose);
+ const originalDistinct = Turn.distinct.bind(Turn);
+ let releaseDistinct!: () => void;
+ let reportDistinct!: () => void;
+ const distinctReleased = new Promise<void>((resolve) => {
+ releaseDistinct = resolve;
+ });
+ const distinctReached = new Promise<void>((resolve) => {
+ reportDistinct = resolve;
+ });
+ const distinctSpy = jest.spyOn(Turn, 'distinct').mockImplementationOnce(
+ (...args: Parameters<typeof Turn.distinct>) =>
+ (async () => {
+ const result = await originalDistinct(...args);
+ reportDistinct();
+ await distinctReleased;
+ return result;
+ })() as ReturnType<typeof Turn.distinct>,
+ );
+ const enqueue = methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'stale-writer-after-delete' }),
+ );
+ await distinctReached;
+ await Sequence.updateOne(
+ { user, tenantId: 'tenant-1', conversationId: 'conversation-1' },
+ { $set: { writerUntil: new Date(0) } },
+ );
+ await remoteMethods.prepareAgentQueuedTurnConversationDeletion({
+ user,
+ targets: [{ conversationId: 'conversation-1', tenantId: 'tenant-1' }],
+ settledAt: START,
+ });
+ const retiredLane = await Sequence.findOne({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ }).lean();
+ await Sequence.deleteOne({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ retiredAt: START,
+ });
+ releaseDistinct();
+ await expect(enqueue).rejects.toThrow('lane writer lease was lost');
+ distinctSpy.mockRestore();
+
+ const recreated = await remoteMethods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'recreated-lane-generation' }),
+ );
+ expect(recreated.turn.laneId).not.toBe(retiredLane?.laneId);
+
+ await expect(remoteMethods.findQueuedTurnsNeedingDelivery()).resolves.toEqual([
+ expect.objectContaining({ queuedTurnId: recreated.turn.queuedTurnId, status: 'queued' }),
+ ]);
+ await expect(
+ Turn.findOne({ clientRequestId: 'stale-writer-after-delete' }).lean(),
+ ).resolves.toMatchObject({ status: 'cancelled', deliveryState: 'retired' });
+ });
+
+ it('terminalizes an obsolete claimed row before a recreated lane can continue', async () => {
+ const obsolete = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'obsolete-lane-claim' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: obsolete.turn.queuedTurnId,
+ deliveryKey: 'delivery-obsolete-lane',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: obsolete.turn.queuedTurnId,
+ deliveryKey: 'delivery-obsolete-lane',
+ scheduledAt: START,
+ });
+ await methods.claimNextAgentQueuedTurn(
+ claimInput(obsolete.turn.queuedTurnId, { claimId: 'delivery-obsolete-lane' }),
+ );
+ await Sequence.deleteOne({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ });
+ const successor = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'recreated-lane-successor' }),
+ );
+ expect(successor.turn.sequence).toBeGreaterThan(obsolete.turn.sequence);
+ expect(successor.turn.laneId).not.toBe(obsolete.turn.laneId);
+
+ await expect(
+ methods.beginAgentQueuedTurnAdmission({
+ ...claimInput(obsolete.turn.queuedTurnId, { claimId: 'delivery-obsolete-lane' }),
+ admissionId: 'delivery-obsolete-lane',
+ startedAt: LATER,
+ }),
+ ).resolves.toMatchObject({
+ outcome: 'retired',
+ turn: {
+ status: 'cancelled',
+ terminalReceipt: { failure: { code: 'LANE_RETIRED' } },
+ },
+ });
+ await expect(
+ methods.claimNextAgentQueuedTurn(
+ claimInput(successor.turn.queuedTurnId, {
+ claimId: 'delivery-successor',
+ now: LATER,
+ leaseUntil: new Date(LATER.getTime() + 60_000),
+ }),
+ ),
+ ).resolves.toMatchObject({ outcome: 'acquired' });
+ });
+
+ it('expires deletion fences after the bounded stale-writer safety window', async () => {
+ await methods.prepareAgentQueuedTurnConversationDeletion({
+ user,
+ targets: [{ conversationId: 'never-queued', tenantId: 'tenant-1' }],
+ settledAt: START,
+ });
+
+ await expect(
+ Sequence.findOne({ user, tenantId: 'tenant-1', conversationId: 'never-queued' }).lean(),
+ ).resolves.toMatchObject({
+ retiredAt: START,
+ expiresAt: new Date(START.getTime() + 24 * 60 * 60_000),
+ });
+ });
+
+ it('retires a terminal source when its published delivery receipt has expired', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'expired-delivery' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-expired',
+ });
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-expired',
+ scheduledAt: START,
+ });
+ await methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ settledAt: LATER,
+ });
+
+ await expect(
+ methods.beginAgentQueuedTurnMissingDeliveryRetirement({
+ deliveryKey: 'delivery-expired',
+ }),
+ ).resolves.toBe(true);
+ await expect(
+ methods.markAgentQueuedTurnMissingDeliveryRetired({ deliveryKey: 'delivery-expired' }),
+ ).resolves.toBe(true);
+ await expect(Turn.findById(queued.turn.queuedTurnId).lean()).resolves.toMatchObject({
+ status: 'cancelled',
+ deliveryState: 'retired',
+ });
+ });
+
+ it('does not infer delivery absence while publication can still cross the probe', async () => {
+ const queued = await methods.enqueueAgentQueuedTurn(
+ enqueueInput({ clientRequestId: 'publication-races-absence' }),
+ );
+ await methods.reserveAgentQueuedTurnDelivery({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-publication-race',
+ });
+ await methods.cancelAgentQueuedTurn({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ settledAt: START,
+ });
+
+ await expect(
+ methods.beginAgentQueuedTurnMissingDeliveryRetirement({
+ deliveryKey: 'delivery-publication-race',
+ }),
+ ).resolves.toBe(false);
+ await methods.markQueuedTurnScheduled({
+ user,
+ tenantId: 'tenant-1',
+ conversationId: 'conversation-1',
+ queuedTurnId: queued.turn.queuedTurnId,
+ deliveryKey: 'delivery-publication-race',
+ scheduledAt: LATER,
+ });
+ await expect(
+ methods.beginAgentQueuedTurnMissingDeliveryRetirement({
+ deliveryKey: 'delivery-publication-race',
+ }),
+ ).resolves.toBe(true);
+ await expect(Turn.findById(queued.turn.queuedTurnId).lean()).resolves.toMatchObject({
+ deliveryState: 'retiring',
+ });
+ });
+});
diff --git a/packages/data-schemas/src/methods/queuedTurn.ts b/packages/data-schemas/src/methods/queuedTurn.ts
new file mode 100644
index 0000000000..83af0cfd38
--- /dev/null
+++ b/packages/data-schemas/src/methods/queuedTurn.ts
@@ -0,0 +1,2318 @@
+import { createHash, randomUUID } from 'node:crypto';
+import type { FilterQuery, Model, Types } from 'mongoose';
+import type {
+ AgentQueuedTurnActiveRecord,
+ AgentQueuedTurnClaim,
+ AgentQueuedTurnFailure,
+ AgentQueuedTurnFileRef,
+ AgentQueuedTurnRecord,
+ IAgentQueuedTurn,
+ IAgentQueuedTurnDocument,
+ IAgentQueuedTurnSequence,
+ IAgentQueuedTurnSequenceDocument,
+} from '~/types/queuedTurn';
+import { createIndexesWithRetry } from '~/utils/retry';
+
+const DUPLICATE_KEY = 11000;
+const MAX_FILES = 20;
+const MAX_QUOTES = 10;
+const MAX_QUOTE_LENGTH = 1500;
+const MAX_MANUAL_SKILLS = 32;
+const MAX_TEXT_LENGTH = 32_768;
+export const MAX_ACTIVE_AGENT_QUEUED_TURNS = 100;
+const LANE_WRITER_LEASE_MS = 30_000;
+const LANE_WRITER_RETRY_MS = 5;
+const LANE_WRITER_RETRIES = LANE_WRITER_LEASE_MS / LANE_WRITER_RETRY_MS;
+const RETIRED_LANE_RETENTION_MS = 24 * 60 * 60_000;
+
+interface DuplicateKeyError {
+ code?: number;
+}
+
+export class AgentQueuedTurnConflictError extends Error {
+ constructor(clientRequestId: string) {
+ super(`Agent queued turn idempotency conflict: ${clientRequestId}`);
+ this.name = 'AgentQueuedTurnConflictError';
+ }
+}
+
+export class AgentQueuedTurnCapacityError extends Error {
+ constructor() {
+ super(`Agent queued turn capacity is limited to ${MAX_ACTIVE_AGENT_QUEUED_TURNS}`);
+ this.name = 'AgentQueuedTurnCapacityError';
+ }
+}
+
+export class AgentQueuedTurnLaneRetiredError extends Error {
+ constructor() {
+ super('Agent queued turn conversation is being deleted');
+ this.name = 'AgentQueuedTurnLaneRetiredError';
+ }
+}
+
+class AgentQueuedTurnLaneMissingError extends Error {
+ constructor() {
+ super('Agent queued turn lane is missing');
+ this.name = 'AgentQueuedTurnLaneMissingError';
+ }
+}
+
+class AgentQueuedTurnLaneGenerationError extends Error {
+ constructor() {
+ super('Agent queued turn belongs to a retired lane generation');
+ this.name = 'AgentQueuedTurnLaneGenerationError';
+ }
+}
+
+interface AgentQueuedTurnLaneWriter {
+ writerId: string;
+ laneId: string;
+}
+
+export interface AgentQueuedTurnOwnerScope {
+ user: Types.ObjectId;
+ tenantId?: string;
+}
+
+export interface AgentQueuedTurnConversationScope extends AgentQueuedTurnOwnerScope {
+ conversationId: string;
+}
+
+export interface AgentQueuedTurnDeletionTarget {
+ conversationId: string;
+ tenantId?: string;
+ /** Recovery for a conversation row that was already deleted cannot recover
+ * its tenant. User + conversation identity is still sufficient to purge it. */
+ allTenants?: true;
+}
+
+export interface EnqueueAgentQueuedTurnInput extends AgentQueuedTurnConversationScope {
+ agentId: string;
+ parentMessageId: string;
+ clientRequestId: string;
+ text: string;
+ files?: readonly AgentQueuedTurnFileRef[];
+ quotes?: readonly string[];
+ manualSkills?: readonly string[];
+ expectedPredecessorCreatedAt?: number;
+ priority?: boolean;
+ availableAt?: Date;
+}
+
+export interface AgentQueuedTurnClaimFence extends AgentQueuedTurnConversationScope {
+ queuedTurnId: string;
+ claimId: string;
+ claimBy: string;
+}
+
+export type CancelAgentQueuedTurnResult =
+ | { outcome: 'cancelled' | 'already_cancelled'; turn: AgentQueuedTurnRecord }
+ | { outcome: 'not_cancellable'; turn: AgentQueuedTurnRecord }
+ | { outcome: 'not_found'; turn: null };
+
+export type ReleaseAgentQueuedTurnResult =
+ | { outcome: 'released' | 'dead'; turn: AgentQueuedTurnRecord }
+ | { outcome: 'conflict'; turn: AgentQueuedTurnRecord | null };
+
+export type AdmitAgentQueuedTurnResult =
+ | { outcome: 'admitted' | 'already_admitted'; turn: AgentQueuedTurnRecord }
+ | { outcome: 'conflict'; turn: AgentQueuedTurnRecord | null };
+
+export type DeadLetterAgentQueuedTurnResult =
+ | {
+ outcome: 'dead' | 'already_terminal' | 'admission_reconciled';
+ turn: AgentQueuedTurnRecord;
+ }
+ | { outcome: 'admission_indeterminate'; turn: AgentQueuedTurnRecord }
+ | { outcome: 'missing'; turn: null }
+ | { outcome: 'conflict'; turn: AgentQueuedTurnRecord };
+
+export type ScheduleAgentQueuedTurnResult =
+ | { outcome: 'scheduled' | 'already_scheduled'; turn: AgentQueuedTurnRecord }
+ | { outcome: 'conflict'; turn: AgentQueuedTurnRecord | null };
+
+export type ReserveAgentQueuedTurnDeliveryResult =
+ | { outcome: 'reserved' | 'already_reserved'; turn: AgentQueuedTurnRecord }
+ | { outcome: 'conflict'; turn: AgentQueuedTurnRecord | null };
+
+export type ClaimAgentQueuedTurnResult =
+ | { outcome: 'acquired' | 'replayed'; claim: AgentQueuedTurnClaim }
+ | { outcome: 'blocked'; claim: null }
+ | { outcome: 'missing'; claim: null };
+
+export type BeginAgentQueuedTurnAdmissionResult =
+ | { outcome: 'started' | 'already_started' | 'retired'; turn: AgentQueuedTurnRecord }
+ | { outcome: 'conflict'; turn: AgentQueuedTurnRecord | null };
+
+export interface AgentQueuedTurnAdmissionEvidence {
+ generationId?: string;
+ generationCreatedAt: number;
+}
+
+export interface ClaimAgentQueuedTurnReconciliationInput {
+ claimId: string;
+ claimBy: string;
+ now: Date;
+ leaseUntil: Date;
+ limit?: number;
+}
+
+export interface AgentQueuedTurnMethods {
+ ensureAgentQueuedTurnIndexes: () => Promise<void>;
+ enqueueAgentQueuedTurn: (
+ input: EnqueueAgentQueuedTurnInput,
+ ) => Promise<{ turn: AgentQueuedTurnRecord; replayed: boolean }>;
+ getAgentQueuedTurnByClientRequestId: (
+ input: AgentQueuedTurnConversationScope & { clientRequestId: string },
+ ) => Promise<AgentQueuedTurnRecord | null>;
+ listActiveAgentQueuedTurns: (
+ input: AgentQueuedTurnConversationScope & { limit?: number },
+ ) => Promise<AgentQueuedTurnActiveRecord[]>;
+ listAgentQueuedTurnReceipts: (
+ input: AgentQueuedTurnConversationScope & { clientRequestIds?: readonly string[] },
+ ) => Promise<AgentQueuedTurnActiveRecord[]>;
+ findQueuedTurnsNeedingDelivery: (limit?: number) => Promise<AgentQueuedTurnRecord[]>;
+ claimQueuedTurnsForAdmissionReconciliation: (
+ input: ClaimAgentQueuedTurnReconciliationInput,
+ ) => Promise<AgentQueuedTurnRecord[]>;
+ deferAgentQueuedTurnAdmissionReconciliation: (
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ deliveryKey: string;
+ claimId: string;
+ claimBy: string;
+ availableAt: Date;
+ },
+ ) => Promise<boolean>;
+ reserveAgentQueuedTurnDelivery: (
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ deliveryKey: string;
+ },
+ ) => Promise<ReserveAgentQueuedTurnDeliveryResult>;
+ markQueuedTurnScheduled: (
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ deliveryKey: string;
+ scheduledAt?: Date;
+ },
+ ) => Promise<ScheduleAgentQueuedTurnResult>;
+ cancelAgentQueuedTurn: (
+ input: AgentQueuedTurnOwnerScope & {
+ queuedTurnId: string;
+ conversationId?: string;
+ settledAt?: Date;
+ },
+ ) => Promise<CancelAgentQueuedTurnResult>;
+ claimNextAgentQueuedTurn: (
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ claimId: string;
+ claimBy: string;
+ now: Date;
+ leaseUntil: Date;
+ },
+ ) => Promise<ClaimAgentQueuedTurnResult>;
+ releaseAgentQueuedTurn: (
+ input: AgentQueuedTurnClaimFence &
+ (
+ | { disposition: 'retry'; availableAt: Date }
+ | {
+ disposition: 'dead';
+ settledAt: Date;
+ failure: AgentQueuedTurnFailure;
+ }
+ ),
+ ) => Promise<ReleaseAgentQueuedTurnResult>;
+ beginAgentQueuedTurnAdmission: (
+ input: AgentQueuedTurnClaimFence & {
+ admissionId: string;
+ startedAt: Date;
+ admissionProtocolVersion?: 2;
+ },
+ ) => Promise<BeginAgentQueuedTurnAdmissionResult>;
+ markAgentQueuedTurnAdmitted: (
+ input: AgentQueuedTurnClaimFence & {
+ admissionId: string;
+ admissionMode: 'warm' | 'ordinary';
+ generationId?: string;
+ generationCreatedAt?: number;
+ settledAt: Date;
+ },
+ ) => Promise<AdmitAgentQueuedTurnResult>;
+ hasAgentQueuedTurnAdmissionReceipt: (
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ admissionId: string;
+ generationId: string;
+ generationCreatedAt: number;
+ },
+ ) => Promise<boolean>;
+ deadLetterAgentQueuedTurn: (
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ deliveryKey: string;
+ settledAt: Date;
+ failure: AgentQueuedTurnFailure;
+ admissionEvidence?: AgentQueuedTurnAdmissionEvidence;
+ reconciliationClaimId?: string;
+ reconciliationClaimBy?: string;
+ },
+ ) => Promise<DeadLetterAgentQueuedTurnResult>;
+ getEffectiveAgentQueuedTurnPredecessor: (
+ input: AgentQueuedTurnConversationScope & {
+ sequence: number;
+ expectedPredecessorCreatedAt?: number;
+ },
+ ) => Promise<number | undefined>;
+ drainAgentQueuedTurns: (
+ input: AgentQueuedTurnOwnerScope & {
+ conversationId?: string;
+ settledAt?: Date;
+ },
+ ) => Promise<number>;
+ deleteAgentQueuedTurns: (
+ input: AgentQueuedTurnOwnerScope & { conversationId?: string },
+ ) => Promise<number>;
+ prepareAgentQueuedTurnConversationDeletion: (input: {
+ user: Types.ObjectId;
+ targets: readonly AgentQueuedTurnDeletionTarget[];
+ settledAt?: Date;
+ }) => Promise<string[]>;
+ deletePreparedAgentQueuedTurnConversations: (input: {
+ user: Types.ObjectId;
+ targets: readonly AgentQueuedTurnDeletionTarget[];
+ }) => Promise<number>;
+ markAgentQueuedTurnDeliveryRetired: (input: { deliveryKey: string }) => Promise<boolean>;
+ beginAgentQueuedTurnMissingDeliveryRetirement: (input: {
+ deliveryKey: string;
+ }) => Promise<boolean>;
+ markAgentQueuedTurnMissingDeliveryRetired: (input: { deliveryKey: string }) => Promise<boolean>;
+ deleteAllAgentQueuedTurnsForUser: (input: { user: Types.ObjectId }) => Promise<number>;
+}
+
+function tenantScope(tenantId: string | undefined):
+ | { tenantId: string }
+ | {
+ tenantId: { $exists: false };
+ } {
+ return tenantId == null ? { tenantId: { $exists: false } } : { tenantId };
+}
+
+function ownerScope(input: AgentQueuedTurnOwnerScope) {
+ return { user: input.user, ...tenantScope(input.tenantId) };
+}
+
+function ownerFields(input: AgentQueuedTurnOwnerScope) {
+ return {
+ user: input.user,
+ ...(input.tenantId != null && { tenantId: input.tenantId }),
+ };
+}
+
+function conversationScope(input: AgentQueuedTurnConversationScope) {
+ return {
+ ...ownerScope(input),
+ conversationId: requireBoundedString(input.conversationId, 256),
+ };
+}
+
+function conversationFields(input: AgentQueuedTurnConversationScope) {
+ return {
+ ...ownerFields(input),
+ conversationId: requireBoundedString(input.conversationId, 256),
+ };
+}
+
+function deletionScope(input: {
+ user: Types.ObjectId;
+ targets: readonly AgentQueuedTurnDeletionTarget[];
+}) {
+ const targets = input.targets.map((target) => {
+ const conversationId = requireBoundedString(target.conversationId, 256);
+ if (target.allTenants === true) {
+ return { conversationId };
+ }
+ return { conversationId, ...tenantScope(target.tenantId) };
+ });
+ if (targets.length === 0) {
+ throw new TypeError('Agent queued turn deletion requires at least one conversation');
+ }
+ return { user: input.user, $or: targets };
+}
+
+function requireBoundedString(value: string, maxLength: number): string {
+ if (typeof value !== 'string') {
+ throw new TypeError('Agent queued turn value must be a string');
+ }
+ const normalized = value.trim();
+ if (normalized.length === 0 || normalized.length > maxLength) {
+ throw new TypeError('Agent queued turn value has an invalid length');
+ }
+ return normalized;
+}
+
+function normalizeOptionalString(value: string | undefined, maxLength: number): string | undefined {
+ if (value == null) {
+ return undefined;
+ }
+ return requireBoundedString(value, maxLength);
+}
+
+function normalizeFiles(files: readonly AgentQueuedTurnFileRef[] | undefined) {
+ if (files == null || files.length === 0) {
+ return undefined;
+ }
+ if (files.length > MAX_FILES) {
+ throw new TypeError('Agent queued turn has too many files');
+ }
+ const seen = new Set<string>();
+ const normalized: AgentQueuedTurnFileRef[] = [];
+ for (const file of files) {
+ const fileId = requireBoundedString(file.file_id, 256);
+ if (seen.has(fileId)) {
+ continue;
+ }
+ seen.add(fileId);
+ normalized.push({
+ file_id: fileId,
+ ...(normalizeOptionalString(file.type, 256) != null && {
+ type: normalizeOptionalString(file.type, 256),
+ }),
+ ...(normalizeOptionalString(file.filepath, 2048) != null && {
+ filepath: normalizeOptionalString(file.filepath, 2048),
+ }),
+ ...(normalizeOptionalString(file.filename, 1024) != null && {
+ filename: normalizeOptionalString(file.filename, 1024),
+ }),
+ ...(file.height != null && {
+ height: requireNonnegativeNumber(file.height),
+ }),
+ ...(file.width != null && {
+ width: requireNonnegativeNumber(file.width),
+ }),
+ ...(file.bytes != null && {
+ bytes: requireNonnegativeNumber(file.bytes),
+ }),
+ });
+ }
+ return normalized.length === 0 ? undefined : normalized;
+}
+
+function requireNonnegativeNumber(value: number): number {
+ if (!Number.isFinite(value) || value < 0) {
+ throw new TypeError('Agent queued turn file metadata must be nonnegative');
+ }
+ return value;
+}
+
+function normalizeQuotes(quotes: readonly string[] | undefined) {
+ if (quotes == null || quotes.length === 0) {
+ return undefined;
+ }
+ const normalized: string[] = [];
+ for (const quote of quotes) {
+ if (typeof quote !== 'string') {
+ throw new TypeError('Agent queued turn quote must be a string');
+ }
+ const trimmed = quote.trim();
+ if (trimmed.length === 0) {
+ continue;
+ }
+ normalized.push(trimmed.slice(0, MAX_QUOTE_LENGTH));
+ if (normalized.length >= MAX_QUOTES) {
+ break;
+ }
+ }
+ return normalized.length === 0 ? undefined : normalized;
+}
+
+function normalizeManualSkills(manualSkills: readonly string[] | undefined) {
+ if (manualSkills == null || manualSkills.length === 0) {
+ return undefined;
+ }
+ if (manualSkills.length > MAX_MANUAL_SKILLS) {
+ throw new TypeError('Agent queued turn has too many manual skills');
+ }
+ const seen = new Set<string>();
+ const normalized: string[] = [];
+ for (const skill of manualSkills) {
+ const name = requireBoundedString(skill, 256);
+ if (seen.has(name)) {
+ continue;
+ }
+ seen.add(name);
+ normalized.push(name);
+ }
+ return normalized.length === 0 ? undefined : normalized;
+}
+
+function normalizeText(text: string): string {
+ if (typeof text !== 'string') {
+ throw new TypeError('Agent queued turn text must be a string');
+ }
+ const normalized = text.trim();
+ if (normalized.length > MAX_TEXT_LENGTH) {
+ throw new TypeError('Agent queued turn text is too long');
+ }
+ return normalized;
+}
+
+function normalizePredecessor(value: number | undefined): number | undefined {
+ if (value == null) {
+ return undefined;
+ }
+ if (!Number.isSafeInteger(value) || value < 0) {
+ throw new TypeError('Agent queued turn predecessor epoch must be a nonnegative integer');
+ }
+ return value;
+}
+
+function normalizeEnqueue(input: EnqueueAgentQueuedTurnInput) {
+ const normalized = {
+ agentId: requireBoundedString(input.agentId, 256),
+ parentMessageId: requireBoundedString(input.parentMessageId, 256),
+ clientRequestId: requireBoundedString(input.clientRequestId, 128),
+ text: normalizeText(input.text),
+ files: normalizeFiles(input.files),
+ quotes: normalizeQuotes(input.quotes),
+ manualSkills: normalizeManualSkills(input.manualSkills),
+ expectedPredecessorCreatedAt: normalizePredecessor(input.expectedPredecessorCreatedAt),
+ priority: input.priority === true,
+ };
+ if (normalized.text.length === 0 && normalized.files == null && normalized.quotes == null) {
+ throw new TypeError('Agent queued turn must contain text, a file, or a quote');
+ }
+ return normalized;
+}
+
+function fingerprint(payload: ReturnType<typeof normalizeEnqueue>): string {
+ return createHash('sha256').update(JSON.stringify(payload)).digest('base64url');
+}
+
+function laneKey(input: AgentQueuedTurnConversationScope): string {
+ return createHash('sha256')
+ .update(
+ JSON.stringify([
+ input.tenantId ?? null,
+ input.user.toString(),
+ requireBoundedString(input.conversationId, 256),
+ ]),
+ )
+ .digest('base64url');
+}
+
+function effectiveDeliveryState(turn: IAgentQueuedTurn) {
+ if (turn.deliveryState != null) {
+ return turn.deliveryState;
+ }
+ if (turn.deliveryKey == null) {
+ return 'pending' as const;
+ }
+ if (turn.scheduledAt == null) {
+ return 'publishing' as const;
+ }
+ return 'published' as const;
+}
+
+function toRecord(turn: IAgentQueuedTurn): AgentQueuedTurnRecord {
+ if (
+ turn._id == null ||
+ turn.createdAt == null ||
+ turn.sequence == null ||
+ turn.status === 'reserving'
+ ) {
+ throw new Error('Agent queued turn is missing its durable identity');
+ }
+ return {
+ queuedTurnId: turn._id.toString(),
+ user: turn.user,
+ ...(turn.tenantId != null && { tenantId: turn.tenantId }),
+ conversationId: turn.conversationId,
+ agentId: turn.agentId,
+ parentMessageId: turn.parentMessageId,
+ clientRequestId: turn.clientRequestId,
+ fingerprint: turn.fingerprint,
+ ...(turn.laneId != null && { laneId: turn.laneId }),
+ sequence: turn.sequence,
+ ...(turn.activeSlot != null && { activeSlot: turn.activeSlot }),
+ status: turn.status,
+ priority: turn.priority,
+ text: turn.text,
+ ...(turn.files != null && { files: turn.files }),
+ ...(turn.quotes != null && { quotes: turn.quotes }),
+ ...(turn.manualSkills != null && { manualSkills: turn.manualSkills }),
+ ...(turn.expectedPredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: turn.expectedPredecessorCreatedAt,
+ }),
+ attempts: turn.attempts,
+ availableAt: turn.availableAt,
+ ...(turn.deliveryKey != null && { deliveryKey: turn.deliveryKey }),
+ deliveryState: effectiveDeliveryState(turn),
+ ...(turn.scheduledAt != null && { scheduledAt: turn.scheduledAt }),
+ ...(turn.claimId != null && { claimId: turn.claimId }),
+ ...(turn.claimBy != null && { claimBy: turn.claimBy }),
+ ...(turn.claimUntil != null && { claimUntil: turn.claimUntil }),
+ ...(turn.admissionId != null && { admissionId: turn.admissionId }),
+ ...(turn.admissionStartedAt != null && { admissionStartedAt: turn.admissionStartedAt }),
+ ...(turn.admissionProtocolVersion != null && {
+ admissionProtocolVersion: turn.admissionProtocolVersion,
+ }),
+ ...(turn.reconciliationAvailableAt != null && {
+ reconciliationAvailableAt: turn.reconciliationAvailableAt,
+ }),
+ ...(turn.reconciliationClaimId != null && {
+ reconciliationClaimId: turn.reconciliationClaimId,
+ }),
+ ...(turn.reconciliationClaimBy != null && {
+ reconciliationClaimBy: turn.reconciliationClaimBy,
+ }),
+ ...(turn.reconciliationClaimUntil != null && {
+ reconciliationClaimUntil: turn.reconciliationClaimUntil,
+ }),
+ ...(turn.reconciliationAttempts != null && {
+ reconciliationAttempts: turn.reconciliationAttempts,
+ }),
+ ...(turn.terminalReceipt != null && {
+ terminalReceipt: turn.terminalReceipt,
+ }),
+ createdAt: turn.createdAt,
+ ...(turn.updatedAt != null && { updatedAt: turn.updatedAt }),
+ };
+}
+
+function toActiveRecord(turn: IAgentQueuedTurn): AgentQueuedTurnActiveRecord {
+ const record = toRecord(turn);
+ return {
+ queuedTurnId: record.queuedTurnId,
+ conversationId: record.conversationId,
+ agentId: record.agentId,
+ parentMessageId: record.parentMessageId,
+ clientRequestId: record.clientRequestId,
+ sequence: record.sequence,
+ ...(record.activeSlot != null && { activeSlot: record.activeSlot }),
+ status: record.status,
+ priority: record.priority,
+ text: record.text,
+ ...(record.files != null && { files: record.files }),
+ ...(record.quotes != null && { quotes: record.quotes }),
+ ...(record.manualSkills != null && { manualSkills: record.manualSkills }),
+ ...(record.expectedPredecessorCreatedAt != null && {
+ expectedPredecessorCreatedAt: record.expectedPredecessorCreatedAt,
+ }),
+ attempts: record.attempts,
+ availableAt: record.availableAt,
+ ...(record.deliveryKey != null && { deliveryKey: record.deliveryKey }),
+ ...(record.deliveryState != null && { deliveryState: record.deliveryState }),
+ ...(record.scheduledAt != null && { scheduledAt: record.scheduledAt }),
+ createdAt: record.createdAt,
+ ...(record.updatedAt != null && { updatedAt: record.updatedAt }),
+ ...(record.terminalReceipt != null && { terminalReceipt: record.terminalReceipt }),
+ };
+}
+
+function requireClaim(turn: IAgentQueuedTurn | null): AgentQueuedTurnClaim | null {
+ if (
+ turn == null ||
+ turn.status !== 'claimed' ||
+ turn.claimId == null ||
+ turn.claimBy == null ||
+ turn.claimUntil == null
+ ) {
+ return null;
+ }
+ return toRecord(turn) as AgentQueuedTurnClaim;
+}
+
+export function createAgentQueuedTurnMethods(
+ mongoose: typeof import('mongoose'),
+): AgentQueuedTurnMethods {
+ const Turn = () => mongoose.models.AgentQueuedTurn as Model<IAgentQueuedTurnDocument>;
+ const Sequence = () =>
+ mongoose.models.AgentQueuedTurnSequence as Model<IAgentQueuedTurnSequenceDocument>;
+ const localLaneTails = new Map<string, Promise<void>>();
+
+ async function serializeLocalLane<T>(
+ input: AgentQueuedTurnConversationScope,
+ operation: () => Promise<T>,
+ ): Promise<T> {
+ const key = laneKey(input);
+ const previous = localLaneTails.get(key) ?? Promise.resolve();
+ let release!: () => void;
+ const current = new Promise<void>((resolve) => {
+ release = resolve;
+ });
+ const tail = previous.catch(() => undefined).then(() => current);
+ localLaneTails.set(key, tail);
+ await previous.catch(() => undefined);
+ try {
+ return await operation();
+ } finally {
+ release();
+ if (localLaneTails.get(key) === tail) {
+ localLaneTails.delete(key);
+ }
+ }
+ }
+
+ async function ensureAgentQueuedTurnIndexes(): Promise<void> {
+ await Promise.all([createIndexesWithRetry(Turn()), createIndexesWithRetry(Sequence())]);
+ }
+
+ async function acquireLaneWriter(
+ input: AgentQueuedTurnConversationScope,
+ createIfMissing = true,
+ ): Promise<AgentQueuedTurnLaneWriter> {
+ const _id = laneKey(input);
+ const scope = conversationScope(input);
+ const writerId = randomUUID();
+ const insertedLaneId = randomUUID();
+ for (let attempt = 0; attempt < LANE_WRITER_RETRIES; attempt++) {
+ const now = new Date();
+ const lane = await Sequence()
+ .findOneAndUpdate(
+ {
+ _id,
+ ...scope,
+ retiredAt: { $exists: false },
+ $or: [{ writerId: { $exists: false } }, { writerUntil: { $lte: now } }],
+ },
+ {
+ $set: {
+ writerId,
+ writerUntil: new Date(now.getTime() + LANE_WRITER_LEASE_MS),
+ },
+ $setOnInsert: { ...conversationFields(input), laneId: insertedLaneId, value: 0 },
+ },
+ { new: true, upsert: createIfMissing },
+ )
+ .lean<IAgentQueuedTurnSequence>()
+ .catch((error: unknown) => {
+ if ((error as DuplicateKeyError).code === DUPLICATE_KEY) {
+ return null;
+ }
+ throw error;
+ });
+ if (lane?.writerId === writerId) {
+ if (lane.laneId == null) {
+ throw new Error('Agent queued turn lane is missing its generation');
+ }
+ if (lane.laneId === insertedLaneId && lane.value === 0) {
+ const predecessor = await Turn()
+ .findOne({ ...scope, sequence: { $exists: true } })
+ .sort({ sequence: -1 })
+ .select('sequence')
+ .lean<Pick<IAgentQueuedTurn, 'sequence'>>();
+ if (predecessor?.sequence != null && predecessor.sequence > 0) {
+ await Sequence().updateOne(
+ { _id, ...scope, writerId, laneId: insertedLaneId },
+ { $max: { value: predecessor.sequence } },
+ );
+ }
+ }
+ return { writerId, laneId: lane.laneId };
+ }
+ const retired = await Sequence().exists({ _id, ...scope, retiredAt: { $exists: true } });
+ if (retired != null) {
+ throw new AgentQueuedTurnLaneRetiredError();
+ }
+ if (!createIfMissing && (await Sequence().exists({ _id, ...scope })) == null) {
+ throw new AgentQueuedTurnLaneMissingError();
+ }
+ await new Promise((resolve) => setTimeout(resolve, LANE_WRITER_RETRY_MS));
+ }
+ throw new Error('Agent queued turn lane writer lease did not become available');
+ }
+
+ async function releaseLaneWriter(
+ input: AgentQueuedTurnConversationScope,
+ writerId: string,
+ ): Promise<void> {
+ await Sequence().updateOne(
+ { _id: laneKey(input), ...conversationScope(input), writerId },
+ { $unset: { writerId: 1, writerUntil: 1 } },
+ );
+ }
+
+ async function repairLaneReservation(
+ input: AgentQueuedTurnConversationScope,
+ writer: AgentQueuedTurnLaneWriter,
+ ): Promise<void> {
+ const _id = laneKey(input);
+ const scope = conversationScope(input);
+ const liveWriter = {
+ _id,
+ ...scope,
+ writerId: writer.writerId,
+ writerUntil: { $gt: new Date() },
+ retiredAt: { $exists: false },
+ };
+ const lane = await Sequence().findOne(liveWriter).lean<IAgentQueuedTurnSequence>();
+ if (lane == null) {
+ const retired = await Sequence().exists({ _id, ...scope, retiredAt: { $exists: true } });
+ if (retired != null) {
+ throw new AgentQueuedTurnLaneRetiredError();
+ }
+ throw new Error('Agent queued turn lane writer lease was lost');
+ }
+ if (lane.laneId !== writer.laneId) {
+ throw new AgentQueuedTurnLaneGenerationError();
+ }
+ if (lane.reservationId != null) {
+ await Turn().updateOne(
+ {
+ ...scope,
+ _id: lane.reservationId,
+ laneId: writer.laneId,
+ status: 'reserving',
+ sequence: { $exists: false },
+ },
+ { $set: { reservationWriterId: writer.writerId } },
+ );
+ await Turn().updateOne(
+ {
+ ...scope,
+ _id: lane.reservationId,
+ laneId: writer.laneId,
+ status: 'reserving',
+ sequence: { $exists: false },
+ reservationWriterId: writer.writerId,
+ },
+ {
+ $set: { status: 'queued', sequence: lane.value },
+ $unset: { reservationWriterId: 1 },
+ },
+ );
+ await Sequence().updateOne(
+ { ...liveWriter, reservationId: lane.reservationId },
+ { $unset: { reservationId: 1 } },
+ );
+ return;
+ }
+ const head = await Turn()
+ .findOne({
+ ...scope,
+ laneId: writer.laneId,
+ status: 'reserving',
+ sequence: { $exists: false },
+ })
+ .sort({ createdAt: 1, _id: 1 })
+ .lean<IAgentQueuedTurn>();
+ if (head?._id == null) {
+ return;
+ }
+ const reservationId = head._id.toString();
+ const owned = await Turn().updateOne(
+ {
+ ...scope,
+ _id: reservationId,
+ laneId: writer.laneId,
+ status: 'reserving',
+ sequence: { $exists: false },
+ },
+ { $set: { reservationWriterId: writer.writerId } },
+ );
+ if (owned.matchedCount !== 1) {
+ return;
+ }
+ const claimed = await Sequence()
+ .findOneAndUpdate(
+ { ...liveWriter, reservationId: { $exists: false } },
+ { $inc: { value: 1 }, $set: { reservationId } },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurnSequence>();
+ if (claimed == null) {
+ return;
+ }
+ const assigned = await Turn().updateOne(
+ {
+ ...scope,
+ _id: reservationId,
+ laneId: writer.laneId,
+ status: 'reserving',
+ sequence: { $exists: false },
+ reservationWriterId: writer.writerId,
+ },
+ {
+ $set: { status: 'queued', sequence: claimed.value },
+ $unset: { reservationWriterId: 1 },
+ },
+ );
+ await Sequence().updateOne(
+ { ...liveWriter, reservationId, value: claimed.value },
+ assigned.modifiedCount === 1
+ ? { $unset: { reservationId: 1 } }
+ : { $inc: { value: -1 }, $unset: { reservationId: 1 } },
+ );
+ }
+
+ /** Finish reservations oldest-first. The lane record durably names the row
+ * that owns its next value, so another process can complete either half. */
+ async function finalizeVisibleReservation(
+ target: IAgentQueuedTurn,
+ writer: AgentQueuedTurnLaneWriter,
+ ): Promise<AgentQueuedTurnRecord> {
+ if (target._id == null) {
+ throw new Error('Agent queued turn reservation is missing its durable identity');
+ }
+ const scopeInput: AgentQueuedTurnConversationScope = {
+ user: target.user,
+ ...(target.tenantId != null && { tenantId: target.tenantId }),
+ conversationId: target.conversationId,
+ };
+ const scope = conversationScope(scopeInput);
+ const targetId = target._id.toString();
+ if (target.laneId !== writer.laneId) {
+ throw new AgentQueuedTurnLaneGenerationError();
+ }
+ for (let repair = 0; repair <= MAX_ACTIVE_AGENT_QUEUED_TURNS; repair++) {
+ await repairLaneReservation(scopeInput, writer);
+ const current = await Turn()
+ .findOne({ ...scope, _id: targetId })
+ .lean<IAgentQueuedTurn>();
+ if (current == null) {
+ throw new Error('Agent queued turn reservation disappeared');
+ }
+ if (current.status !== 'reserving') {
+ return toRecord(current);
+ }
+ }
+ throw new Error('Agent queued turn reservation repair exceeded lane capacity');
+ }
+
+ async function enqueueAgentQueuedTurn(
+ input: EnqueueAgentQueuedTurnInput,
+ ): Promise<{ turn: AgentQueuedTurnRecord; replayed: boolean }> {
+ const scope = conversationScope(input);
+ const normalized = normalizeEnqueue(input);
+ const requestFingerprint = fingerprint(normalized);
+ const existing = await Turn()
+ .findOne({ ...scope, clientRequestId: normalized.clientRequestId })
+ .lean<IAgentQueuedTurn>();
+ if (existing != null) {
+ if (existing.fingerprint !== requestFingerprint) {
+ throw new AgentQueuedTurnConflictError(normalized.clientRequestId);
+ }
+ if (existing.status !== 'reserving') {
+ return { turn: toRecord(existing), replayed: true };
+ }
+ }
+
+ return serializeLocalLane(input, async () => {
+ const writer = await acquireLaneWriter(input);
+ try {
+ const replay = await Turn()
+ .findOne({ ...scope, clientRequestId: normalized.clientRequestId })
+ .lean<IAgentQueuedTurn>();
+ if (replay != null) {
+ if (replay.fingerprint !== requestFingerprint) {
+ throw new AgentQueuedTurnConflictError(normalized.clientRequestId);
+ }
+ return {
+ turn:
+ replay.status === 'reserving'
+ ? await finalizeVisibleReservation(replay, writer)
+ : toRecord(replay),
+ replayed: true,
+ };
+ }
+
+ const firstSlot =
+ createHash('sha256').update(normalized.clientRequestId).digest().readUInt32BE(0) %
+ MAX_ACTIVE_AGENT_QUEUED_TURNS;
+ const usedSlots = new Set(
+ (await Turn().distinct('activeSlot', {
+ ...scope,
+ activeSlot: { $exists: true },
+ })) as number[],
+ );
+ for (let offset = 0; offset < MAX_ACTIVE_AGENT_QUEUED_TURNS; offset++) {
+ const activeSlot = (firstSlot + offset) % MAX_ACTIVE_AGENT_QUEUED_TURNS;
+ if (usedSlots.has(activeSlot)) {
+ continue;
+ }
+ try {
+ const created = await Turn().create({
+ ...conversationFields(input),
+ ...normalized,
+ fingerprint: requestFingerprint,
+ laneId: writer.laneId,
+ activeSlot,
+ status: 'reserving',
+ attempts: 0,
+ availableAt: input.availableAt ?? new Date(),
+ deliveryState: 'pending',
+ reservationWriterId: writer.writerId,
+ });
+ return {
+ turn: await finalizeVisibleReservation(created.toObject(), writer),
+ replayed: false,
+ };
+ } catch (error) {
+ if ((error as DuplicateKeyError).code !== DUPLICATE_KEY) {
+ throw error;
+ }
+ usedSlots.add(activeSlot);
+ const racedReplay = await Turn()
+ .findOne({ ...scope, clientRequestId: normalized.clientRequestId })
+ .lean<IAgentQueuedTurn>();
+ if (racedReplay != null) {
+ if (racedReplay.fingerprint !== requestFingerprint) {
+ throw new AgentQueuedTurnConflictError(normalized.clientRequestId);
+ }
+ return {
+ turn:
+ racedReplay.status === 'reserving'
+ ? await finalizeVisibleReservation(racedReplay, writer)
+ : toRecord(racedReplay),
+ replayed: true,
+ };
+ }
+ }
+ }
+ throw new AgentQueuedTurnCapacityError();
+ } finally {
+ await releaseLaneWriter(input, writer.writerId);
+ }
+ });
+ }
+
+ async function listActiveAgentQueuedTurns(
+ input: AgentQueuedTurnConversationScope & { limit?: number },
+ ): Promise<AgentQueuedTurnActiveRecord[]> {
+ const limit = input.limit ?? 100;
+ if (!Number.isSafeInteger(limit) || limit <= 0 || limit > 100) {
+ throw new TypeError('Agent queued turn list limit must be between 1 and 100');
+ }
+ const turns = await Turn()
+ .find({
+ ...conversationScope(input),
+ status: { $in: ['queued', 'claimed'] },
+ })
+ .sort({ priority: -1, sequence: 1 })
+ .limit(limit)
+ .lean<IAgentQueuedTurn[]>();
+ return turns.map(toActiveRecord);
+ }
+
+ async function listAgentQueuedTurnReceipts(
+ input: AgentQueuedTurnConversationScope & { clientRequestIds?: readonly string[] },
+ ): Promise<AgentQueuedTurnActiveRecord[]> {
+ const clientRequestIds = [
+ ...new Set((input.clientRequestIds ?? []).map((value) => requireBoundedString(value, 128))),
+ ];
+ if (clientRequestIds.length > MAX_ACTIVE_AGENT_QUEUED_TURNS) {
+ throw new TypeError(
+ `Agent queued turn receipt lookup is limited to ${MAX_ACTIVE_AGENT_QUEUED_TURNS} ids`,
+ );
+ }
+ const scope = conversationScope(input);
+ const [active, dead, known] = await Promise.all([
+ Turn()
+ .find({ ...scope, status: { $in: ['queued', 'claimed'] } })
+ .sort({ priority: -1, sequence: 1 })
+ .limit(MAX_ACTIVE_AGENT_QUEUED_TURNS)
+ .lean<IAgentQueuedTurn[]>(),
+ Turn()
+ .find({ ...scope, status: 'dead' })
+ .sort({ sequence: -1 })
+ .limit(MAX_ACTIVE_AGENT_QUEUED_TURNS)
+ .lean<IAgentQueuedTurn[]>(),
+ clientRequestIds.length === 0
+ ? Promise.resolve([])
+ : Turn()
+ .find({ ...scope, clientRequestId: { $in: clientRequestIds } })
+ .limit(clientRequestIds.length)
+ .lean<IAgentQueuedTurn[]>(),
+ ]);
+ const turns = new Map<string, IAgentQueuedTurn>();
+ for (const turn of [...active, ...dead, ...known]) {
+ if (turn._id != null) {
+ turns.set(turn._id.toString(), turn);
+ }
+ }
+ return [...turns.values()]
+ .filter((turn) => turn.sequence != null)
+ .sort(
+ (left, right) =>
+ Number(right.priority) - Number(left.priority) ||
+ (left.sequence ?? Number.MAX_SAFE_INTEGER) - (right.sequence ?? Number.MAX_SAFE_INTEGER),
+ )
+ .map(toActiveRecord);
+ }
+
+ async function getAgentQueuedTurnByClientRequestId(
+ input: AgentQueuedTurnConversationScope & { clientRequestId: string },
+ ): Promise<AgentQueuedTurnRecord | null> {
+ const turn = await Turn()
+ .findOne({
+ ...conversationScope(input),
+ clientRequestId: requireBoundedString(input.clientRequestId, 128),
+ })
+ .lean<IAgentQueuedTurn>();
+ return turn == null ? null : toRecord(turn);
+ }
+
+ async function cancelAgentQueuedTurn(
+ input: AgentQueuedTurnOwnerScope & {
+ queuedTurnId: string;
+ conversationId?: string;
+ settledAt?: Date;
+ },
+ ): Promise<CancelAgentQueuedTurnResult> {
+ const settledAt = input.settledAt ?? new Date();
+ const scope = {
+ ...ownerScope(input),
+ ...(input.conversationId != null && {
+ conversationId: requireBoundedString(input.conversationId, 256),
+ }),
+ };
+ const turn = await Turn()
+ .findOneAndUpdate(
+ {
+ ...scope,
+ _id: input.queuedTurnId,
+ status: { $in: ['queued', 'claimed', 'dead'] },
+ admissionStartedAt: { $exists: false },
+ },
+ {
+ $set: {
+ status: 'cancelled',
+ terminalReceipt: { outcome: 'cancelled', settledAt },
+ },
+ $unset: { activeSlot: 1, claimId: 1, claimBy: 1, claimUntil: 1 },
+ },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ if (turn != null) {
+ const deliveryState =
+ turn.deliveryKey == null ? 'retired' : (turn.deliveryState ?? 'published');
+ const state = await Turn()
+ .findOneAndUpdate(
+ {
+ _id: turn._id,
+ status: 'cancelled',
+ ...(turn.deliveryKey == null && { deliveryKey: { $exists: false } }),
+ },
+ { $set: { deliveryState } },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ return { outcome: 'cancelled', turn: toRecord(state ?? turn) };
+ }
+ const current = await Turn()
+ .findOne({ ...scope, _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ if (current == null) {
+ return { outcome: 'not_found', turn: null };
+ }
+ if (current.status === 'cancelled') {
+ return { outcome: 'already_cancelled', turn: toRecord(current) };
+ }
+ return { outcome: 'not_cancellable', turn: toRecord(current) };
+ }
+
+ async function findQueuedTurnsNeedingDelivery(limit = 100): Promise<AgentQueuedTurnRecord[]> {
+ if (!Number.isSafeInteger(limit) || limit <= 0 || limit > 1000) {
+ throw new TypeError('Agent queued turn recovery limit must be between 1 and 1000');
+ }
+ const reservations = await Turn()
+ .find({ status: 'reserving' })
+ .sort({ createdAt: 1, _id: 1 })
+ .limit(limit)
+ .lean<IAgentQueuedTurn[]>();
+ for (const reservation of reservations) {
+ const scopeInput: AgentQueuedTurnConversationScope = {
+ user: reservation.user,
+ ...(reservation.tenantId != null && { tenantId: reservation.tenantId }),
+ conversationId: reservation.conversationId,
+ };
+ try {
+ await serializeLocalLane(scopeInput, async () => {
+ const writer = await acquireLaneWriter(scopeInput, false);
+ try {
+ await finalizeVisibleReservation(reservation, writer);
+ } finally {
+ await releaseLaneWriter(scopeInput, writer.writerId);
+ }
+ });
+ } catch (error) {
+ if (
+ !(error instanceof AgentQueuedTurnLaneRetiredError) &&
+ !(error instanceof AgentQueuedTurnLaneMissingError) &&
+ !(error instanceof AgentQueuedTurnLaneGenerationError)
+ ) {
+ throw error;
+ }
+ await Turn().updateOne(
+ { _id: reservation._id, status: 'reserving' },
+ {
+ $set: {
+ status: 'cancelled',
+ deliveryState: 'retired',
+ terminalReceipt: {
+ outcome: 'cancelled',
+ settledAt: new Date(),
+ failure: {
+ code: 'OWNER_DRAINED',
+ message: 'Queued turn owner was drained',
+ },
+ },
+ },
+ $unset: { activeSlot: 1, reservationWriterId: 1 },
+ },
+ );
+ continue;
+ }
+ }
+ const turns = await Turn()
+ .find({
+ $or: [
+ {
+ status: 'queued',
+ $or: [
+ { deliveryState: { $in: ['pending', 'publishing'] } },
+ { deliveryState: { $exists: false }, scheduledAt: { $exists: false } },
+ ],
+ },
+ { status: { $in: ['cancelled', 'dead'] }, deliveryState: 'publishing' },
+ {
+ status: { $in: ['cancelled', 'dead'] },
+ deliveryKey: { $exists: true },
+ deliveryState: { $exists: false },
+ },
+ ],
+ })
+ .sort({ availableAt: 1, sequence: 1, _id: 1 })
+ .limit(limit)
+ .lean<IAgentQueuedTurn[]>();
+ return turns.map(toRecord);
+ }
+
+ async function claimQueuedTurnsForAdmissionReconciliation(
+ input: ClaimAgentQueuedTurnReconciliationInput,
+ ): Promise<AgentQueuedTurnRecord[]> {
+ const limit = input.limit ?? 100;
+ if (!Number.isSafeInteger(limit) || limit <= 0 || limit > 1000) {
+ throw new TypeError('Agent queued turn reconciliation limit must be between 1 and 1000');
+ }
+ const claimId = requireBoundedString(input.claimId, 128);
+ const claimBy = requireBoundedString(input.claimBy, 256);
+ if (
+ !Number.isFinite(input.now.getTime()) ||
+ !Number.isFinite(input.leaseUntil.getTime()) ||
+ input.leaseUntil <= input.now
+ ) {
+ throw new TypeError('Agent queued turn reconciliation lease is invalid');
+ }
+ const eligible: FilterQuery<IAgentQueuedTurnDocument> = {
+ admissionId: { $exists: true },
+ admissionStartedAt: { $exists: true },
+ deliveryKey: { $exists: true },
+ $and: [
+ {
+ $or: [
+ {
+ status: 'dead',
+ 'terminalReceipt.outcome': 'dead',
+ 'terminalReceipt.failure.code': 'ADMISSION_INDETERMINATE',
+ },
+ { status: 'claimed', claimUntil: { $lte: input.now } },
+ ],
+ },
+ {
+ $or: [
+ { reconciliationAvailableAt: { $lte: input.now } },
+ { reconciliationAvailableAt: { $exists: false } },
+ ],
+ },
+ {
+ $or: [
+ { reconciliationClaimUntil: { $lte: input.now } },
+ { reconciliationClaimUntil: { $exists: false } },
+ ],
+ },
+ ],
+ };
+ const candidates = await Turn()
+ .find({
+ ...eligible,
+ })
+ .sort({ reconciliationAvailableAt: 1, admissionStartedAt: 1, _id: 1 })
+ .limit(limit)
+ .select('_id')
+ .lean<IAgentQueuedTurn[]>();
+ if (candidates.length === 0) {
+ return [];
+ }
+ await Turn().updateMany(
+ {
+ ...eligible,
+ _id: { $in: candidates.flatMap((turn) => (turn._id == null ? [] : [turn._id])) },
+ },
+ {
+ $set: {
+ reconciliationClaimId: claimId,
+ reconciliationClaimBy: claimBy,
+ reconciliationClaimUntil: input.leaseUntil,
+ },
+ $inc: { reconciliationAttempts: 1 },
+ },
+ );
+ const turns = await Turn()
+ .find({ reconciliationClaimId: claimId, reconciliationClaimBy: claimBy })
+ .sort({ reconciliationAvailableAt: 1, admissionStartedAt: 1, _id: 1 })
+ .lean<IAgentQueuedTurn[]>();
+ return turns.map(toRecord);
+ }
+
+ async function deferAgentQueuedTurnAdmissionReconciliation(
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ deliveryKey: string;
+ claimId: string;
+ claimBy: string;
+ availableAt: Date;
+ },
+ ): Promise<boolean> {
+ if (!Number.isFinite(input.availableAt.getTime())) {
+ throw new TypeError('Agent queued turn reconciliation availability is invalid');
+ }
+ const result = await Turn().updateOne(
+ {
+ ...conversationScope(input),
+ _id: input.queuedTurnId,
+ deliveryKey: requireBoundedString(input.deliveryKey, 128),
+ status: 'dead',
+ 'terminalReceipt.failure.code': 'ADMISSION_INDETERMINATE',
+ reconciliationClaimId: requireBoundedString(input.claimId, 128),
+ reconciliationClaimBy: requireBoundedString(input.claimBy, 256),
+ },
+ {
+ $set: { reconciliationAvailableAt: input.availableAt },
+ $unset: {
+ reconciliationClaimId: 1,
+ reconciliationClaimBy: 1,
+ reconciliationClaimUntil: 1,
+ },
+ },
+ );
+ return result.modifiedCount === 1;
+ }
+
+ async function reserveAgentQueuedTurnDelivery(
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ deliveryKey: string;
+ },
+ ): Promise<ReserveAgentQueuedTurnDeliveryResult> {
+ const scope = conversationScope(input);
+ const deliveryKey = requireBoundedString(input.deliveryKey, 128);
+ const turn = await Turn()
+ .findOneAndUpdate(
+ {
+ ...scope,
+ _id: input.queuedTurnId,
+ status: { $in: ['queued', 'claimed'] },
+ $or: [{ deliveryState: 'pending' }, { deliveryState: { $exists: false } }],
+ deliveryKey: { $exists: false },
+ },
+ { $set: { deliveryKey, deliveryState: 'publishing' } },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ if (turn != null) {
+ return { outcome: 'reserved', turn: toRecord(turn) };
+ }
+ const current = await Turn()
+ .findOne({ ...scope, _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ if (
+ current?.deliveryKey === deliveryKey &&
+ (current.deliveryState == null ||
+ current.deliveryState === 'publishing' ||
+ current.deliveryState === 'published')
+ ) {
+ return { outcome: 'already_reserved', turn: toRecord(current) };
+ }
+ return { outcome: 'conflict', turn: current == null ? null : toRecord(current) };
+ }
+
+ async function markQueuedTurnScheduled(
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ deliveryKey: string;
+ scheduledAt?: Date;
+ },
+ ): Promise<ScheduleAgentQueuedTurnResult> {
+ const deliveryKey = requireBoundedString(input.deliveryKey, 128);
+ const turn = await Turn()
+ .findOneAndUpdate(
+ {
+ ...conversationScope(input),
+ _id: input.queuedTurnId,
+ deliveryKey,
+ $or: [{ deliveryState: 'publishing' }, { deliveryState: { $exists: false } }],
+ scheduledAt: { $exists: false },
+ },
+ {
+ $set: {
+ scheduledAt: input.scheduledAt ?? new Date(),
+ deliveryState: 'published',
+ },
+ },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ if (turn != null) {
+ return { outcome: 'scheduled', turn: toRecord(turn) };
+ }
+ const current = await Turn()
+ .findOne({ ...conversationScope(input), _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ if (
+ current?.scheduledAt != null &&
+ current.deliveryKey === deliveryKey &&
+ current.deliveryState === 'published'
+ ) {
+ return { outcome: 'already_scheduled', turn: toRecord(current) };
+ }
+ return {
+ outcome: 'conflict',
+ turn: current == null ? null : toRecord(current),
+ };
+ }
+
+ async function claimNextAgentQueuedTurn(
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ claimId: string;
+ claimBy: string;
+ now: Date;
+ leaseUntil: Date;
+ },
+ ): Promise<ClaimAgentQueuedTurnResult> {
+ const claimId = requireBoundedString(input.claimId, 128);
+ const claimBy = requireBoundedString(input.claimBy, 256);
+ if (input.leaseUntil.getTime() <= input.now.getTime()) {
+ throw new TypeError('Agent queued turn lease must end after claim time');
+ }
+ const scope = conversationScope(input);
+ const replay = await Turn()
+ .findOne({
+ ...scope,
+ _id: input.queuedTurnId,
+ status: 'claimed',
+ claimId,
+ claimBy,
+ claimUntil: { $gt: input.now },
+ })
+ .lean<IAgentQueuedTurn>();
+ const replayedClaim = requireClaim(replay);
+ if (replayedClaim != null) {
+ return { outcome: 'replayed', claim: replayedClaim };
+ }
+
+ const expected = await Turn()
+ .findOne({ ...scope, _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ if (expected == null || (expected.status !== 'queued' && expected.status !== 'claimed')) {
+ return { outcome: 'missing', claim: null };
+ }
+
+ const unfinishedReservation = await Turn().exists({ ...scope, status: 'reserving' });
+ if (unfinishedReservation != null) {
+ return { outcome: 'blocked', claim: null };
+ }
+
+ const head = await Turn()
+ .findOne({ ...scope, status: { $in: ['queued', 'claimed', 'dead'] } })
+ .sort({ priority: -1, sequence: 1 })
+ .lean<IAgentQueuedTurn>();
+ if (head == null) {
+ return { outcome: 'blocked', claim: null };
+ }
+ if (
+ head._id?.toString() !== input.queuedTurnId ||
+ head.availableAt.getTime() > input.now.getTime() ||
+ (head.status === 'claimed' &&
+ head.claimUntil != null &&
+ head.claimUntil.getTime() > input.now.getTime())
+ ) {
+ return { outcome: 'blocked', claim: null };
+ }
+
+ const turn = await Turn()
+ .findOneAndUpdate(
+ {
+ ...scope,
+ _id: input.queuedTurnId,
+ availableAt: { $lte: input.now },
+ $or: [
+ { status: 'queued' },
+ {
+ status: 'claimed',
+ claimUntil: { $lte: input.now },
+ admissionStartedAt: { $exists: false },
+ },
+ ],
+ },
+ {
+ $set: {
+ status: 'claimed',
+ claimId,
+ claimBy,
+ claimUntil: input.leaseUntil,
+ },
+ $inc: { attempts: 1 },
+ $unset: { terminalReceipt: 1 },
+ },
+ { new: true, sort: { priority: -1, sequence: 1 } },
+ )
+ .lean<IAgentQueuedTurn>();
+ const acquired = requireClaim(turn);
+ if (acquired != null) {
+ return { outcome: 'acquired', claim: acquired };
+ }
+ const racedReplay = await Turn()
+ .findOne({
+ ...scope,
+ _id: input.queuedTurnId,
+ status: 'claimed',
+ claimId,
+ claimBy,
+ claimUntil: { $gt: input.now },
+ })
+ .lean<IAgentQueuedTurn>();
+ const racedClaim = requireClaim(racedReplay);
+ return racedClaim == null
+ ? { outcome: 'blocked', claim: null }
+ : { outcome: 'replayed', claim: racedClaim };
+ }
+
+ async function releaseAgentQueuedTurn(
+ input: AgentQueuedTurnClaimFence &
+ (
+ | { disposition: 'retry'; availableAt: Date }
+ | {
+ disposition: 'dead';
+ settledAt: Date;
+ failure: AgentQueuedTurnFailure;
+ }
+ ),
+ ): Promise<ReleaseAgentQueuedTurnResult> {
+ const fence = {
+ ...conversationScope(input),
+ _id: input.queuedTurnId,
+ status: 'claimed',
+ claimId: requireBoundedString(input.claimId, 128),
+ claimBy: requireBoundedString(input.claimBy, 256),
+ };
+ const update =
+ input.disposition === 'retry'
+ ? {
+ $set: { status: 'queued', availableAt: input.availableAt },
+ $unset: {
+ claimId: 1,
+ claimBy: 1,
+ claimUntil: 1,
+ admissionId: 1,
+ admissionStartedAt: 1,
+ terminalReceipt: 1,
+ },
+ }
+ : {
+ $set: {
+ status: 'dead',
+ terminalReceipt: {
+ outcome: 'dead',
+ settledAt: input.settledAt,
+ failure: {
+ code: requireBoundedString(input.failure.code, 128),
+ message: requireBoundedString(input.failure.message, 2048),
+ },
+ },
+ },
+ $unset: {
+ activeSlot: 1,
+ claimId: 1,
+ claimBy: 1,
+ claimUntil: 1,
+ admissionId: 1,
+ admissionStartedAt: 1,
+ },
+ };
+ const turn = await Turn()
+ .findOneAndUpdate(fence, update, { new: true })
+ .lean<IAgentQueuedTurn>();
+ if (turn != null) {
+ return {
+ outcome: input.disposition === 'retry' ? 'released' : 'dead',
+ turn: toRecord(turn),
+ };
+ }
+ const current = await Turn()
+ .findOne({ ...conversationScope(input), _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ return {
+ outcome: 'conflict',
+ turn: current == null ? null : toRecord(current),
+ };
+ }
+
+ async function beginAgentQueuedTurnAdmission(
+ input: AgentQueuedTurnClaimFence & {
+ admissionId: string;
+ startedAt: Date;
+ admissionProtocolVersion?: 2;
+ },
+ ): Promise<BeginAgentQueuedTurnAdmissionResult> {
+ const admissionId = requireBoundedString(input.admissionId, 128);
+ if (!Number.isFinite(input.startedAt.getTime())) {
+ throw new TypeError('Agent queued turn admission start is invalid');
+ }
+ const retireObsoleteClaim = async (): Promise<AgentQueuedTurnRecord | null> => {
+ const retired = await Turn()
+ .findOneAndUpdate(
+ {
+ ...conversationScope(input),
+ _id: input.queuedTurnId,
+ status: 'claimed',
+ claimId: requireBoundedString(input.claimId, 128),
+ claimBy: requireBoundedString(input.claimBy, 256),
+ admissionStartedAt: { $exists: false },
+ },
+ {
+ $set: {
+ status: 'cancelled',
+ terminalReceipt: {
+ outcome: 'cancelled',
+ settledAt: input.startedAt,
+ failure: {
+ code: 'LANE_RETIRED',
+ message: 'The queued turn belongs to a retired conversation lane',
+ },
+ },
+ },
+ $unset: { activeSlot: 1, claimId: 1, claimBy: 1, claimUntil: 1 },
+ },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ return retired == null ? null : toRecord(retired);
+ };
+ return serializeLocalLane(input, async () => {
+ let writer: AgentQueuedTurnLaneWriter;
+ try {
+ writer = await acquireLaneWriter(input, false);
+ } catch (error) {
+ if (
+ error instanceof AgentQueuedTurnLaneRetiredError ||
+ error instanceof AgentQueuedTurnLaneMissingError
+ ) {
+ const retired = await retireObsoleteClaim();
+ if (retired != null) {
+ return { outcome: 'retired', turn: retired };
+ }
+ const current = await Turn()
+ .findOne({ ...conversationScope(input), _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ return {
+ outcome: 'conflict',
+ turn: current == null ? null : toRecord(current),
+ };
+ }
+ throw error;
+ }
+ try {
+ const turn = await Turn()
+ .findOneAndUpdate(
+ {
+ ...conversationScope(input),
+ _id: input.queuedTurnId,
+ status: 'claimed',
+ claimId: requireBoundedString(input.claimId, 128),
+ claimBy: requireBoundedString(input.claimBy, 256),
+ deliveryKey: admissionId,
+ deliveryState: 'published',
+ laneId: writer.laneId,
+ admissionId: { $exists: false },
+ },
+ {
+ $set: {
+ admissionId,
+ admissionStartedAt: input.startedAt,
+ ...(input.admissionProtocolVersion != null && {
+ admissionProtocolVersion: input.admissionProtocolVersion,
+ }),
+ },
+ },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ if (turn != null) {
+ return { outcome: 'started', turn: toRecord(turn) };
+ }
+ const current = await Turn()
+ .findOne({ ...conversationScope(input), _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ if (
+ current?.status === 'claimed' &&
+ current.claimId === input.claimId &&
+ current.claimBy === input.claimBy &&
+ current.admissionStartedAt == null &&
+ current.laneId !== writer.laneId
+ ) {
+ const retired = await retireObsoleteClaim();
+ if (retired != null) {
+ return { outcome: 'retired', turn: retired };
+ }
+ }
+ if (
+ current?.status === 'claimed' &&
+ current.claimId === input.claimId &&
+ current.claimBy === input.claimBy &&
+ current.deliveryKey === admissionId &&
+ current.deliveryState === 'published' &&
+ current.laneId === writer.laneId &&
+ current.admissionId === admissionId &&
+ current.admissionStartedAt != null
+ ) {
+ return { outcome: 'already_started', turn: toRecord(current) };
+ }
+ return {
+ outcome: 'conflict',
+ turn: current == null ? null : toRecord(current),
+ };
+ } finally {
+ await releaseLaneWriter(input, writer.writerId);
+ }
+ });
+ }
+
+ async function markAgentQueuedTurnAdmitted(
+ input: AgentQueuedTurnClaimFence & {
+ admissionId: string;
+ admissionMode: 'warm' | 'ordinary';
+ generationId?: string;
+ generationCreatedAt?: number;
+ settledAt: Date;
+ },
+ ): Promise<AdmitAgentQueuedTurnResult> {
+ const admissionId = requireBoundedString(input.admissionId, 128);
+ const generationId = normalizeOptionalString(input.generationId, 256);
+ const generationCreatedAt = normalizePredecessor(input.generationCreatedAt);
+ const turn = await Turn()
+ .findOneAndUpdate(
+ {
+ ...conversationScope(input),
+ _id: input.queuedTurnId,
+ deliveryKey: admissionId,
+ deliveryState: 'published',
+ admissionId,
+ admissionStartedAt: { $exists: true },
+ $or: [
+ {
+ status: 'claimed',
+ claimId: requireBoundedString(input.claimId, 128),
+ claimBy: requireBoundedString(input.claimBy, 256),
+ },
+ {
+ status: 'dead',
+ 'terminalReceipt.outcome': 'dead',
+ 'terminalReceipt.failure.code': 'ADMISSION_INDETERMINATE',
+ },
+ ],
+ },
+ {
+ $set: {
+ status: 'admitted',
+ terminalReceipt: {
+ outcome: 'admitted',
+ settledAt: input.settledAt,
+ admissionId,
+ admissionMode: input.admissionMode,
+ ...(generationId != null && { generationId }),
+ ...(generationCreatedAt != null && { generationCreatedAt }),
+ },
+ },
+ $unset: {
+ activeSlot: 1,
+ claimId: 1,
+ claimBy: 1,
+ claimUntil: 1,
+ admissionId: 1,
+ admissionStartedAt: 1,
+ admissionProtocolVersion: 1,
+ reconciliationAvailableAt: 1,
+ reconciliationClaimId: 1,
+ reconciliationClaimBy: 1,
+ reconciliationClaimUntil: 1,
+ },
+ },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ if (turn != null) {
+ return { outcome: 'admitted', turn: toRecord(turn) };
+ }
+ const current = await Turn()
+ .findOne({ ...conversationScope(input), _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ if (
+ current?.status === 'admitted' &&
+ current.terminalReceipt?.outcome === 'admitted' &&
+ current.terminalReceipt.admissionId === admissionId
+ ) {
+ return { outcome: 'already_admitted', turn: toRecord(current) };
+ }
+ return {
+ outcome: 'conflict',
+ turn: current == null ? null : toRecord(current),
+ };
+ }
+
+ async function hasAgentQueuedTurnAdmissionReceipt(
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ admissionId: string;
+ generationId: string;
+ generationCreatedAt: number;
+ },
+ ): Promise<boolean> {
+ const generationCreatedAt = normalizePredecessor(input.generationCreatedAt);
+ if (generationCreatedAt == null) {
+ throw new TypeError('Agent queued turn admission generation is invalid');
+ }
+ return (
+ (await Turn().exists({
+ ...conversationScope(input),
+ _id: input.queuedTurnId,
+ status: 'admitted',
+ 'terminalReceipt.outcome': 'admitted',
+ 'terminalReceipt.admissionId': requireBoundedString(input.admissionId, 128),
+ 'terminalReceipt.generationId': requireBoundedString(input.generationId, 256),
+ 'terminalReceipt.generationCreatedAt': generationCreatedAt,
+ })) != null
+ );
+ }
+
+ async function deadLetterAgentQueuedTurn(
+ input: AgentQueuedTurnConversationScope & {
+ queuedTurnId: string;
+ deliveryKey: string;
+ settledAt: Date;
+ failure: AgentQueuedTurnFailure;
+ admissionEvidence?: AgentQueuedTurnAdmissionEvidence;
+ reconciliationClaimId?: string;
+ reconciliationClaimBy?: string;
+ },
+ ): Promise<DeadLetterAgentQueuedTurnResult> {
+ const scope = conversationScope(input);
+ const deliveryKey = requireBoundedString(input.deliveryKey, 128);
+ if (input.admissionEvidence != null) {
+ const generationId = normalizeOptionalString(input.admissionEvidence.generationId, 256);
+ const generationCreatedAt = normalizePredecessor(input.admissionEvidence.generationCreatedAt);
+ if (generationCreatedAt == null) {
+ throw new TypeError('Agent queued turn admission evidence is invalid');
+ }
+ const reconciled = await Turn()
+ .findOneAndUpdate(
+ {
+ ...scope,
+ _id: input.queuedTurnId,
+ deliveryKey,
+ admissionId: deliveryKey,
+ admissionStartedAt: { $exists: true },
+ ...(input.reconciliationClaimId != null && {
+ reconciliationClaimId: requireBoundedString(input.reconciliationClaimId, 128),
+ }),
+ ...(input.reconciliationClaimBy != null && {
+ reconciliationClaimBy: requireBoundedString(input.reconciliationClaimBy, 256),
+ }),
+ $or: [
+ { status: 'claimed' },
+ {
+ status: 'dead',
+ 'terminalReceipt.outcome': 'dead',
+ 'terminalReceipt.failure.code': 'ADMISSION_INDETERMINATE',
+ },
+ ],
+ },
+ {
+ $set: {
+ status: 'admitted',
+ terminalReceipt: {
+ outcome: 'admitted',
+ settledAt: input.settledAt,
+ admissionId: deliveryKey,
+ admissionMode: 'ordinary',
+ ...(generationId != null && { generationId }),
+ generationCreatedAt,
+ },
+ },
+ $unset: {
+ activeSlot: 1,
+ claimId: 1,
+ claimBy: 1,
+ claimUntil: 1,
+ admissionId: 1,
+ admissionStartedAt: 1,
+ admissionProtocolVersion: 1,
+ reconciliationAvailableAt: 1,
+ reconciliationClaimId: 1,
+ reconciliationClaimBy: 1,
+ reconciliationClaimUntil: 1,
+ },
+ },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ if (reconciled != null) {
+ return { outcome: 'admission_reconciled', turn: toRecord(reconciled) };
+ }
+ }
+ const turn = await Turn()
+ .findOneAndUpdate(
+ {
+ ...scope,
+ _id: input.queuedTurnId,
+ deliveryKey,
+ status: { $in: ['queued', 'claimed'] },
+ admissionStartedAt: { $exists: false },
+ },
+ {
+ $set: {
+ status: 'dead',
+ terminalReceipt: {
+ outcome: 'dead',
+ settledAt: input.settledAt,
+ failure: {
+ code: requireBoundedString(input.failure.code, 128),
+ message: requireBoundedString(input.failure.message, 2048),
+ },
+ },
+ },
+ $unset: { activeSlot: 1, claimId: 1, claimBy: 1, claimUntil: 1 },
+ },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ if (turn != null) {
+ return { outcome: 'dead', turn: toRecord(turn) };
+ }
+ const current = await Turn()
+ .findOne({ ...scope, _id: input.queuedTurnId })
+ .lean<IAgentQueuedTurn>();
+ if (current == null) {
+ return { outcome: 'missing', turn: null };
+ }
+ if (
+ current.status === 'claimed' &&
+ current.deliveryKey === deliveryKey &&
+ current.admissionStartedAt != null
+ ) {
+ const quarantined = await Turn()
+ .findOneAndUpdate(
+ {
+ ...scope,
+ _id: input.queuedTurnId,
+ deliveryKey,
+ status: 'claimed',
+ admissionId: deliveryKey,
+ admissionStartedAt: { $exists: true },
+ },
+ {
+ $set: {
+ status: 'dead',
+ reconciliationAvailableAt: input.settledAt,
+ reconciliationAttempts: 0,
+ terminalReceipt: {
+ outcome: 'dead',
+ settledAt: input.settledAt,
+ failure: {
+ code: 'ADMISSION_INDETERMINATE',
+ message: 'The queued turn may have been admitted and requires reconciliation',
+ },
+ },
+ },
+ $unset: {
+ claimId: 1,
+ claimBy: 1,
+ claimUntil: 1,
+ reconciliationClaimId: 1,
+ reconciliationClaimBy: 1,
+ reconciliationClaimUntil: 1,
+ },
+ },
+ { new: true },
+ )
+ .lean<IAgentQueuedTurn>();
+ if (quarantined != null) {
+ return { outcome: 'admission_indeterminate', turn: toRecord(quarantined) };
+ }
+ }
+ if (['admitted', 'cancelled', 'dead'].includes(current.status)) {
+ return { outcome: 'already_terminal', turn: toRecord(current) };
+ }
+ return { outcome: 'conflict', turn: toRecord(current) };
+ }
+
+ async function getEffectiveAgentQueuedTurnPredecessor(
+ input: AgentQueuedTurnConversationScope & {
+ sequence: number;
+ expectedPredecessorCreatedAt?: number;
+ },
+ ): Promise<number | undefined> {
+ if (!Number.isSafeInteger(input.sequence) || input.sequence <= 0) {
+ throw new TypeError('Agent queued turn sequence must be a positive integer');
+ }
+ const rootEpoch = normalizePredecessor(input.expectedPredecessorCreatedAt);
+ if (rootEpoch == null) {
+ return undefined;
+ }
+ const predecessor = await Turn()
+ .findOne({
+ ...conversationScope(input),
+ sequence: { $lt: input.sequence },
+ expectedPredecessorCreatedAt: rootEpoch,
+ status: 'admitted',
+ 'terminalReceipt.outcome': 'admitted',
+ 'terminalReceipt.generationCreatedAt': { $exists: true },
+ })
+ .sort({ sequence: -1 })
+ .select('terminalReceipt.generationCreatedAt')
+ .lean<IAgentQueuedTurn>();
+ return predecessor?.terminalReceipt?.generationCreatedAt;
+ }
+
+ async function drainAgentQueuedTurns(
+ input: AgentQueuedTurnOwnerScope & {
+ conversationId?: string;
+ settledAt?: Date;
+ },
+ ): Promise<number> {
+ const settledAt = input.settledAt ?? new Date();
+ const result = await Turn().updateMany(
+ {
+ ...ownerScope(input),
+ ...(input.conversationId != null && {
+ conversationId: requireBoundedString(input.conversationId, 256),
+ }),
+ status: { $in: ['queued', 'claimed'] },
+ admissionStartedAt: { $exists: false },
+ },
+ {
+ $set: {
+ status: 'cancelled',
+ terminalReceipt: {
+ outcome: 'cancelled',
+ settledAt,
+ failure: {
+ code: 'OWNER_DRAINED',
+ message: 'Queued turn owner was drained',
+ },
+ },
+ },
+ $unset: { activeSlot: 1, claimId: 1, claimBy: 1, claimUntil: 1 },
+ },
+ );
+ await Turn().updateMany(
+ {
+ ...ownerScope(input),
+ ...(input.conversationId != null && {
+ conversationId: requireBoundedString(input.conversationId, 256),
+ }),
+ status: 'cancelled',
+ deliveryKey: { $exists: false },
+ },
+ { $set: { deliveryState: 'retired' } },
+ );
+ return result.modifiedCount;
+ }
+
+ async function deleteAgentQueuedTurns(
+ input: AgentQueuedTurnOwnerScope & { conversationId?: string },
+ ): Promise<number> {
+ const scope = {
+ ...ownerScope(input),
+ ...(input.conversationId != null && {
+ conversationId: requireBoundedString(input.conversationId, 256),
+ }),
+ };
+ const [turns] = await Promise.all([Turn().deleteMany(scope), Sequence().deleteMany(scope)]);
+ return turns.deletedCount;
+ }
+
+ async function retireConversationLane(
+ user: Types.ObjectId,
+ target: AgentQueuedTurnDeletionTarget,
+ retiredAt: Date,
+ ): Promise<void> {
+ if (target.allTenants === true) {
+ const lanes = await Sequence()
+ .find({ user, conversationId: requireBoundedString(target.conversationId, 256) })
+ .select('tenantId')
+ .lean<IAgentQueuedTurnSequence[]>();
+ for (const lane of lanes) {
+ await retireConversationLane(
+ user,
+ {
+ conversationId: target.conversationId,
+ ...(lane.tenantId != null && { tenantId: lane.tenantId }),
+ },
+ retiredAt,
+ );
+ }
+ return;
+ }
+ const input: AgentQueuedTurnConversationScope = {
+ user,
+ ...(target.tenantId != null && { tenantId: target.tenantId }),
+ conversationId: target.conversationId,
+ };
+ const _id = laneKey(input);
+ const scope = conversationScope(input);
+ return serializeLocalLane(input, async () => {
+ let writer: AgentQueuedTurnLaneWriter;
+ try {
+ writer = await acquireLaneWriter(input);
+ } catch (error) {
+ if (error instanceof AgentQueuedTurnLaneRetiredError) {
+ return;
+ }
+ throw error;
+ }
+ try {
+ const admissionInFlight = await Turn().exists({
+ ...scope,
+ admissionStartedAt: { $exists: true },
+ });
+ if (admissionInFlight != null) {
+ throw new Error('Agent queued turn admission must settle before conversation deletion');
+ }
+ const retired = await Sequence().updateOne(
+ { _id, ...scope, writerId: writer.writerId, retiredAt: { $exists: false } },
+ {
+ $set: {
+ retiredAt,
+ expiresAt: new Date(retiredAt.getTime() + RETIRED_LANE_RETENTION_MS),
+ },
+ $unset: { writerId: 1, writerUntil: 1 },
+ },
+ );
+ if (retired.modifiedCount !== 1) {
+ throw new Error('Agent queued turn lane retirement fence was lost');
+ }
+ } finally {
+ await releaseLaneWriter(input, writer.writerId);
+ }
+ });
+ }
+
+ async function prepareAgentQueuedTurnConversationDeletion(input: {
+ user: Types.ObjectId;
+ targets: readonly AgentQueuedTurnDeletionTarget[];
+ settledAt?: Date;
+ }): Promise<string[]> {
+ const scope = deletionScope(input);
+ const settledAt = input.settledAt ?? new Date();
+ for (const target of input.targets) {
+ await retireConversationLane(input.user, target, settledAt);
+ }
+ await Turn().updateMany(
+ {
+ ...scope,
+ $or: [
+ { status: { $in: ['reserving', 'queued'] } },
+ { status: 'claimed', admissionStartedAt: { $exists: false } },
+ ],
+ },
+ {
+ $set: {
+ status: 'cancelled',
+ terminalReceipt: {
+ outcome: 'cancelled',
+ settledAt,
+ failure: {
+ code: 'OWNER_DRAINED',
+ message: 'Queued turn owner was drained',
+ },
+ },
+ },
+ $unset: {
+ activeSlot: 1,
+ reservationWriterId: 1,
+ claimId: 1,
+ claimBy: 1,
+ claimUntil: 1,
+ admissionId: 1,
+ admissionStartedAt: 1,
+ },
+ },
+ );
+ await Promise.all([
+ Turn().updateMany(
+ { ...scope, status: 'cancelled', deliveryKey: { $exists: false } },
+ { $set: { deliveryState: 'retired' } },
+ ),
+ Turn().updateMany(
+ {
+ ...scope,
+ status: { $in: ['admitted', 'cancelled', 'dead'] },
+ deliveryKey: { $exists: true },
+ deliveryState: { $exists: false },
+ },
+ { $set: { deliveryState: 'published' } },
+ ),
+ ]);
+ const turns = await Turn()
+ .find({
+ ...scope,
+ status: { $in: ['admitted', 'cancelled', 'dead'] },
+ deliveryKey: { $exists: true },
+ deliveryState: { $ne: 'retired' },
+ })
+ .select('deliveryKey')
+ .lean<Array<Pick<IAgentQueuedTurn, 'deliveryKey'>>>();
+ return [
+ ...new Set(turns.flatMap((turn) => (turn.deliveryKey == null ? [] : [turn.deliveryKey]))),
+ ];
+ }
+
+ async function deletePreparedAgentQueuedTurnConversations(input: {
+ user: Types.ObjectId;
+ targets: readonly AgentQueuedTurnDeletionTarget[];
+ }): Promise<number> {
+ const scope = deletionScope(input);
+ const blocker = await Turn().exists({
+ ...scope,
+ $or: [
+ { deliveryKey: { $exists: true }, deliveryState: { $ne: 'retired' } },
+ { status: { $in: ['reserving', 'queued', 'claimed'] } },
+ { admissionStartedAt: { $exists: true } },
+ ],
+ });
+ if (blocker != null) {
+ throw new Error('Agent queued turn deliveries must retire before conversation deletion');
+ }
+ const turns = await Turn().deleteMany(scope);
+ return turns.deletedCount;
+ }
+
+ async function markAgentQueuedTurnDeliveryRetired(input: {
+ deliveryKey: string;
+ }): Promise<boolean> {
+ const deliveryKey = requireBoundedString(input.deliveryKey, 128);
+ const result = await Turn().updateOne(
+ {
+ deliveryKey,
+ status: { $in: ['admitted', 'cancelled', 'dead'] },
+ },
+ { $set: { deliveryState: 'retired' } },
+ );
+ if (result.modifiedCount === 1) {
+ return true;
+ }
+ const achieved = await Turn().exists({
+ deliveryKey,
+ deliveryState: 'retired',
+ });
+ return achieved != null;
+ }
+
+ /** Freezes a terminal source before probing another collection for an aged
+ * delivery receipt. A scheduler can only advance `publishing` to
+ * `published`, so it can never cross this fence after the absence read. */
+ async function beginAgentQueuedTurnMissingDeliveryRetirement(input: {
+ deliveryKey: string;
+ }): Promise<boolean> {
+ const deliveryKey = requireBoundedString(input.deliveryKey, 128);
+ const result = await Turn().updateOne(
+ {
+ deliveryKey,
+ status: { $in: ['admitted', 'cancelled', 'dead'] },
+ deliveryState: 'published',
+ },
+ { $set: { deliveryState: 'retiring' } },
+ );
+ if (result.modifiedCount === 1) {
+ return true;
+ }
+ return (
+ (await Turn().exists({
+ deliveryKey,
+ status: { $in: ['admitted', 'cancelled', 'dead'] },
+ deliveryState: 'retiring',
+ })) != null
+ );
+ }
+
+ /** A retirement-fenced source proves that its delivery was durably created
+ * before the absence read. Once that delivery has aged out, absence is
+ * terminal proof rather than permission to publish the payload again. */
+ async function markAgentQueuedTurnMissingDeliveryRetired(input: {
+ deliveryKey: string;
+ }): Promise<boolean> {
+ const deliveryKey = requireBoundedString(input.deliveryKey, 128);
+ const result = await Turn().updateOne(
+ {
+ deliveryKey,
+ status: { $in: ['admitted', 'cancelled', 'dead'] },
+ deliveryState: 'retiring',
+ },
+ { $set: { deliveryState: 'retired' } },
+ );
+ if (result.modifiedCount === 1) {
+ return true;
+ }
+ return (
+ (await Turn().exists({
+ deliveryKey,
+ status: { $in: ['admitted', 'cancelled', 'dead'] },
+ deliveryState: 'retired',
+ })) != null
+ );
+ }
+
+ async function deleteAllAgentQueuedTurnsForUser(input: {
+ user: Types.ObjectId;
+ }): Promise<number> {
+ const [turns] = await Promise.all([
+ Turn().deleteMany({ user: input.user }),
+ Sequence().deleteMany({ user: input.user }),
+ ]);
+ return turns.deletedCount;
+ }
+
+ return {
+ ensureAgentQueuedTurnIndexes,
+ enqueueAgentQueuedTurn,
+ getAgentQueuedTurnByClientRequestId,
+ listActiveAgentQueuedTurns,
+ listAgentQueuedTurnReceipts,
+ findQueuedTurnsNeedingDelivery,
+ claimQueuedTurnsForAdmissionReconciliation,
+ deferAgentQueuedTurnAdmissionReconciliation,
+ reserveAgentQueuedTurnDelivery,
+ markQueuedTurnScheduled,
+ cancelAgentQueuedTurn,
+ claimNextAgentQueuedTurn,
+ beginAgentQueuedTurnAdmission,
+ releaseAgentQueuedTurn,
+ markAgentQueuedTurnAdmitted,
+ hasAgentQueuedTurnAdmissionReceipt,
+ deadLetterAgentQueuedTurn,
+ getEffectiveAgentQueuedTurnPredecessor,
+ drainAgentQueuedTurns,
+ deleteAgentQueuedTurns,
+ prepareAgentQueuedTurnConversationDeletion,
+ deletePreparedAgentQueuedTurnConversations,
+ markAgentQueuedTurnDeliveryRetired,
+ beginAgentQueuedTurnMissingDeliveryRetirement,
+ markAgentQueuedTurnMissingDeliveryRetired,
+ deleteAllAgentQueuedTurnsForUser,
+ };
+}
diff --git a/packages/data-schemas/src/methods/triggerDelivery.spec.ts b/packages/data-schemas/src/methods/triggerDelivery.spec.ts
index b4a7ded00c..6ca5b42499 100644
--- a/packages/data-schemas/src/methods/triggerDelivery.spec.ts
+++ b/packages/data-schemas/src/methods/triggerDelivery.spec.ts
@@ -639,6 +639,50 @@ describe('agent trigger delivery methods', () => {
).resolves.toMatchObject({ id: successor.delivery.id });
});
+ it('treats an already successful internal delivery as terminal retirement evidence', async () => {
+ const source = { id: 'agent-queued-turn', type: 'internal' };
+ const queued = await methods.enqueueAgentTriggerDelivery(
+ enqueueInput({
+ deliveryKey: 'queued-turn-already-succeeded',
+ envelope: { event: { source } },
+ }),
+ );
+ const claim = await methods.claimNextAgentTriggerDelivery({
+ workerId: 'queued-turn-worker',
+ claimToken: 'queued-turn-claim',
+ now: START,
+ leaseUntil: new Date(START.getTime() + 60_000),
+ });
+ const attempt = await methods.beginAgentTriggerDeliveryAttempt({
+ id: claim!.id,
+ workerId: 'queued-turn-worker',
+ claimToken: 'queued-turn-claim',
+ now: START,
+ });
+ await methods.completeAgentTriggerDelivery({
+ id: claim!.id,
+ workerId: 'queued-turn-worker',
+ claimToken: 'queued-turn-claim',
+ attempt: attempt!,
+ result: { status: 'settled' },
+ settledAt: START,
+ });
+
+ await expect(
+ methods.retireAgentTriggerDelivery({
+ deliveryKey: queued.delivery.deliveryKey,
+ sourceId: source.id,
+ settledAt: new Date(START.getTime() + 1),
+ reason: 'source already terminalized',
+ }),
+ ).resolves.toBe(true);
+ await expect(Delivery.findById(queued.delivery.id).lean()).resolves.toMatchObject({
+ status: 'succeeded',
+ result: { status: 'settled' },
+ settledAt: START,
+ });
+ });
+
it('does not retire a completion already leased by a resolver for a manual poll', async () => {
const user = new mongoose.Types.ObjectId();
const source = { id: 'background-tool-completion', type: 'internal' };
@@ -4213,6 +4257,43 @@ describe('agent trigger delivery methods', () => {
expect(await UserPurge.countDocuments({ _id: user })).toBe(0);
});
+ it('retains trigger purge state when queued-turn cleanup fails', async () => {
+ const user = new mongoose.Types.ObjectId();
+ const fenceStartedAt = new Date(START);
+ let unavailable = true;
+ const purgeQueuedTurnsForUser = jest.fn(async () => {
+ if (unavailable) {
+ throw new Error('queued-turn store unavailable');
+ }
+ });
+ const purgeMethods = createAgentTriggerDeliveryMethods(mongoose, {
+ purgeQueuedTurnsForUser,
+ });
+ await User.create({
+ _id: user,
+ email: '[email protected]',
+ provider: 'local',
+ agentTriggerDeletionStartedAt: fenceStartedAt,
+ });
+ await purgeMethods.enqueueAgentTriggerDelivery(enqueueInput({ user }));
+ await purgeMethods.prepareAgentTriggerUserPurge(user, fenceStartedAt);
+ await User.deleteOne({ _id: user });
+
+ await expect(purgeMethods.deleteAgentTriggerDeliveriesByUser(user)).rejects.toThrow(
+ 'queued-turn store unavailable',
+ );
+ expect(await Delivery.countDocuments({ user })).toBe(1);
+ expect(await LaneSequence.countDocuments({ user })).toBe(1);
+ expect(await UserPurge.countDocuments({ _id: user })).toBe(1);
+
+ unavailable = false;
+ await expect(purgeMethods.deleteAgentTriggerDeliveriesByUser(user)).resolves.toBeUndefined();
+ expect(await Delivery.countDocuments({ user })).toBe(0);
+ expect(await LaneSequence.countDocuments({ user })).toBe(0);
+ expect(await UserPurge.countDocuments({ _id: user })).toBe(0);
+ expect(purgeQueuedTurnsForUser).toHaveBeenCalledTimes(2);
+ });
+
it('recovers an armed purge after the user deletion commits', async () => {
const user = new mongoose.Types.ObjectId();
const fenceStartedAt = new Date(START);
@@ -4236,6 +4317,42 @@ describe('agent trigger delivery methods', () => {
expect(await UserPurge.countDocuments({ _id: user })).toBe(0);
});
+ it('retains a recovery marker when queued-turn purge cannot be confirmed', async () => {
+ const user = new mongoose.Types.ObjectId();
+ const fenceStartedAt = new Date(START);
+ let unavailable = true;
+ const purgeQueuedTurnsForUser = jest.fn(async () => {
+ if (unavailable) {
+ throw new Error('queued-turn store unavailable');
+ }
+ });
+ const purgeMethods = createAgentTriggerDeliveryMethods(mongoose, {
+ purgeQueuedTurnsForUser,
+ });
+ await User.create({
+ _id: user,
+ email: '[email protected]',
+ provider: 'local',
+ agentTriggerDeletionStartedAt: fenceStartedAt,
+ });
+ await purgeMethods.enqueueAgentTriggerDelivery(enqueueInput({ user }));
+ await purgeMethods.prepareAgentTriggerUserPurge(user, fenceStartedAt);
+ await User.deleteOne({ _id: user });
+
+ await expect(purgeMethods.recoverAgentTriggerUserPurges()).rejects.toThrow(
+ 'queued-turn store unavailable',
+ );
+ expect(await Delivery.countDocuments({ user })).toBe(1);
+ expect(await LaneSequence.countDocuments({ user })).toBe(1);
+ expect(await UserPurge.countDocuments({ _id: user })).toBe(1);
+
+ unavailable = false;
+ await expect(purgeMethods.recoverAgentTriggerUserPurges()).resolves.toBe(1);
+ expect(await Delivery.countDocuments({ user })).toBe(0);
+ expect(await LaneSequence.countDocuments({ user })).toBe(0);
+ expect(await UserPurge.countDocuments({ _id: user })).toBe(0);
+ });
+
it('rotates active purge markers so a later committed purge is recovered', async () => {
const users = Array.from({ length: 3 }, () => new mongoose.Types.ObjectId());
for (const [index, user] of users.entries()) {
diff --git a/packages/data-schemas/src/methods/triggerDelivery.ts b/packages/data-schemas/src/methods/triggerDelivery.ts
index 033a74ff60..a4569db5fd 100644
--- a/packages/data-schemas/src/methods/triggerDelivery.ts
+++ b/packages/data-schemas/src/methods/triggerDelivery.ts
@@ -19,6 +19,7 @@ import type {
import {
AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1,
+ AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1,
} from '~/types/triggerDelivery';
import { createIndexesWithRetry } from '~/utils/retry';
import logger from '~/config/winston';
@@ -412,15 +413,25 @@ function toRecord(delivery: IAgentTriggerDelivery): AgentTriggerDeliveryRecord {
...(delivery.history != null && { history: delivery.history }),
...(delivery.settledAt != null && { settledAt: delivery.settledAt }),
...(delivery.expiresAt != null && { expiresAt: delivery.expiresAt }),
- ...(delivery.requeueCount != null && { requeueCount: delivery.requeueCount }),
+ ...(delivery.requeueCount != null && {
+ requeueCount: delivery.requeueCount,
+ }),
...(delivery.updatedAt != null && { updatedAt: delivery.updatedAt }),
- ...(delivery.envelopeBytes != null && { envelopeBytes: delivery.envelopeBytes }),
+ ...(delivery.envelopeBytes != null && {
+ envelopeBytes: delivery.envelopeBytes,
+ }),
...(delivery.coalesceKey != null && { coalesceKey: delivery.coalesceKey }),
- ...(delivery.coalesceFrom != null && { coalesceFrom: delivery.coalesceFrom }),
- ...(delivery.coalesceUntil != null && { coalesceUntil: delivery.coalesceUntil }),
+ ...(delivery.coalesceFrom != null && {
+ coalesceFrom: delivery.coalesceFrom,
+ }),
+ ...(delivery.coalesceUntil != null && {
+ coalesceUntil: delivery.coalesceUntil,
+ }),
...(delivery.batchSize != null && { batchSize: delivery.batchSize }),
...(delivery.batchBytes != null && { batchBytes: delivery.batchBytes }),
- ...(delivery.batchMemberIds != null && { batchMemberIds: delivery.batchMemberIds }),
+ ...(delivery.batchMemberIds != null && {
+ batchMemberIds: delivery.batchMemberIds,
+ }),
...(delivery.batchRootId != null && { batchRootId: delivery.batchRootId }),
...(delivery.batchMembersSettledAt != null && {
batchMembersSettledAt: delivery.batchMembersSettledAt,
@@ -429,7 +440,9 @@ function toRecord(delivery: IAgentTriggerDelivery): AgentTriggerDeliveryRecord {
awaitTerminalHandling: delivery.awaitTerminalHandling,
}),
...(delivery.handling != null && { handling: delivery.handling }),
- ...(delivery.actorReceipt != null && { actorReceipt: delivery.actorReceipt }),
+ ...(delivery.actorReceipt != null && {
+ actorReceipt: delivery.actorReceipt,
+ }),
};
}
@@ -451,6 +464,9 @@ function requireClaim(delivery: IAgentTriggerDelivery | null): AgentTriggerDeliv
export function createAgentTriggerDeliveryMethods(
mongoose: typeof import('mongoose'),
+ deps: {
+ purgeQueuedTurnsForUser?: (user: string | Types.ObjectId) => Promise<unknown>;
+ } = {},
): AgentTriggerDeliveryMethods {
const Delivery = () =>
mongoose.models.AgentTriggerDelivery as Model<IAgentTriggerDeliveryDocument>;
@@ -490,7 +506,11 @@ export function createAgentTriggerDeliveryMethods(
const released = await LaneSequence().updateOne(publisherFence, {
$inc: { value: -1 },
$set: { cleanupRequestedAt: new Date() },
- $unset: { publisherDeliveryId: 1, publisherRequeueCount: 1, publisherStartedAt: 1 },
+ $unset: {
+ publisherDeliveryId: 1,
+ publisherRequeueCount: 1,
+ publisherStartedAt: 1,
+ },
});
return released.modifiedCount === 1;
}
@@ -524,7 +544,10 @@ export function createAgentTriggerDeliveryMethods(
return false;
}
}
- const publicationLane = { ...lane, publisherRequeueCount: publicationGeneration };
+ const publicationLane = {
+ ...lane,
+ publisherRequeueCount: publicationGeneration,
+ };
let batchRoot: IAgentTriggerDelivery | null = null;
if (
staged?._id != null &&
@@ -713,7 +736,11 @@ export function createAgentTriggerDeliveryMethods(
if (publicationCommitted) {
const released = await LaneSequence().updateOne(publisherFence, {
$set: { tailDeliveryId: publisherDeliveryId },
- $unset: { publisherDeliveryId: 1, publisherRequeueCount: 1, publisherStartedAt: 1 },
+ $unset: {
+ publisherDeliveryId: 1,
+ publisherRequeueCount: 1,
+ publisherStartedAt: 1,
+ },
});
return published.modifiedCount === 1 || released.modifiedCount === 1;
}
@@ -933,7 +960,8 @@ export function createAgentTriggerDeliveryMethods(
if (
input.requiredWorkerCapability != null &&
input.requiredWorkerCapability !== AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1 &&
- input.requiredWorkerCapability !== AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1
+ input.requiredWorkerCapability !== AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1 &&
+ input.requiredWorkerCapability !== AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1
) {
throw new TypeError('Agent trigger delivery requires an unsupported worker capability');
}
@@ -1080,7 +1108,10 @@ export function createAgentTriggerDeliveryMethods(
.lean<IAgentTriggerDelivery[]>()
: [];
const staged = [
- ...legacyStaged.map((delivery) => ({ ...delivery, stagingRecoveryAt: legacyCursor })),
+ ...legacyStaged.map((delivery) => ({
+ ...delivery,
+ stagingRecoveryAt: legacyCursor,
+ })),
...indexedStaged,
];
const stagingRecoveryCursor = new Date();
@@ -1444,7 +1475,9 @@ export function createAgentTriggerDeliveryMethods(
: earlier.availableAt,
...(leaseUntil != null && { leaseUntil }),
...(earlier.status === 'succeeded' &&
- earlier.handling?.status === 'started' && { reason: 'active_handling' as const }),
+ earlier.handling?.status === 'started' && {
+ reason: 'active_handling' as const,
+ }),
};
}
@@ -1601,7 +1634,10 @@ export function createAgentTriggerDeliveryMethods(
leaseUntil: 1,
claimToken: 1,
...(input.status === 'succeeded'
- ? { lastError: 1, ...(awaitsTerminalHandling && { expiresAt: 1 }) }
+ ? {
+ lastError: 1,
+ ...(awaitsTerminalHandling && { expiresAt: 1 }),
+ }
: { result: 1, expiresAt: 1 }),
},
$push: {
@@ -1644,7 +1680,11 @@ export function createAgentTriggerDeliveryMethods(
}
}
await Delivery().updateOne(
- { _id: root._id, status: input.status, batchMembersSettledAt: { $exists: false } },
+ {
+ _id: root._id,
+ status: input.status,
+ batchMembersSettledAt: { $exists: false },
+ },
{ $set: { batchMembersSettledAt: input.settledAt } },
{ timestamps: false },
);
@@ -1800,11 +1840,18 @@ export function createAgentTriggerDeliveryMethods(
}
const update = {
$inc: { attempts: -1 },
- $set: { availableAt: input.availableAt, claimAvailableAt: input.availableAt },
+ $set: {
+ availableAt: input.availableAt,
+ claimAvailableAt: input.availableAt,
+ },
$unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1 },
};
const shieldResult = await Delivery().updateOne(
- { _id: input.id, ...shieldCapabilityFence(input), attempts: input.attempt },
+ {
+ _id: input.id,
+ ...shieldCapabilityFence(input),
+ attempts: input.attempt,
+ },
{
$inc: update.$inc,
$set: {
@@ -1827,7 +1874,11 @@ export function createAgentTriggerDeliveryMethods(
return true;
}
const capabilityResult = await Delivery().updateOne(
- { _id: input.id, ...legacyCapabilityFence(input), attempts: input.attempt },
+ {
+ _id: input.id,
+ ...legacyCapabilityFence(input),
+ attempts: input.attempt,
+ },
{
...update,
$set: { ...update.$set, status: 'capability_pending' },
@@ -2042,7 +2093,10 @@ export function createAgentTriggerDeliveryMethods(
'envelope.event.source.type': 'internal',
'envelope.event.source.id': input.sourceId,
status: 'succeeded',
- 'result.backgroundToolCompletionRetired': true,
+ $or: [
+ { handling: { $exists: false } },
+ { 'handling.status': { $in: ['applied', 'completed_no_action', 'failed', 'cancelled'] } },
+ ],
})) != null
);
}
@@ -2071,7 +2125,9 @@ export function createAgentTriggerDeliveryMethods(
requiredWorkerCapability: AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1,
'envelope.event.source.type': 'internal',
'envelope.event.source.id': input.sourceId,
- status: { $in: ['pending', 'leased', 'capability_pending', 'capability_leased'] },
+ status: {
+ $in: ['pending', 'leased', 'capability_pending', 'capability_leased'],
+ },
capabilityStatus: { $ne: 'dead' },
settledAt: { $exists: false },
},
@@ -2878,8 +2934,14 @@ export function createAgentTriggerDeliveryMethods(
actorReceipt: { $exists: false },
},
{
- $set: { status: 'succeeded', settledAt: input.settledAt, actorReceipt },
- $max: { expiresAt: new Date(input.settledAt.getTime() + SUCCESS_RETENTION_MS) },
+ $set: {
+ status: 'succeeded',
+ settledAt: input.settledAt,
+ actorReceipt,
+ },
+ $max: {
+ expiresAt: new Date(input.settledAt.getTime() + SUCCESS_RETENTION_MS),
+ },
$unset: { lastError: 1 },
},
{ new: true },
@@ -2959,11 +3021,17 @@ export function createAgentTriggerDeliveryMethods(
now: Date,
): Promise<AgentEventActorReceiptStorageMetrics> {
const [byResolution, expiryEligible, retryDeliveries, deadDeliveries] = await Promise.all([
- Delivery().aggregate<{ _id: AgentEventActorReceipt['resolution']; count: number }>([
+ Delivery().aggregate<{
+ _id: AgentEventActorReceipt['resolution'];
+ count: number;
+ }>([
{ $match: { actorReceipt: { $exists: true } } },
{ $group: { _id: '$actorReceipt.resolution', count: { $sum: 1 } } },
]),
- Delivery().countDocuments({ actorReceipt: { $exists: true }, expiresAt: { $lte: now } }),
+ Delivery().countDocuments({
+ actorReceipt: { $exists: true },
+ expiresAt: { $lte: now },
+ }),
Delivery().countDocuments({
$or: [
{ status: { $in: ['pending', 'capability_pending'] } },
@@ -2994,7 +3062,12 @@ export function createAgentTriggerDeliveryMethods(
retainedByResolution[row._id] = row.count;
}
}
- return { retainedByResolution, expiryEligible, retryDeliveries, deadDeliveries };
+ return {
+ retainedByResolution,
+ expiryEligible,
+ retryDeliveries,
+ deadDeliveries,
+ };
}
async function retryAgentTriggerDelivery(
@@ -3012,7 +3085,13 @@ export function createAgentTriggerDeliveryMethods(
claimAvailableAt: input.availableAt,
lastError: error,
},
- $unset: { leaseBy: 1, leaseUntil: 1, claimToken: 1, settledAt: 1, expiresAt: 1 },
+ $unset: {
+ leaseBy: 1,
+ leaseUntil: 1,
+ claimToken: 1,
+ settledAt: 1,
+ expiresAt: 1,
+ },
$push: {
history: {
$each: [
@@ -3454,12 +3533,19 @@ export function createAgentTriggerDeliveryMethods(
continue;
}
await Delivery().updateMany(
- { user: marker._id, actorActionAdmissionClosedAt: marker.fenceStartedAt },
+ {
+ user: marker._id,
+ actorActionAdmissionClosedAt: marker.fenceStartedAt,
+ },
{ $unset: { actorActionAdmissionClosedAt: 1 } },
);
- await UserPurge().deleteOne({ _id: marker._id, fenceStartedAt: marker.fenceStartedAt });
+ await UserPurge().deleteOne({
+ _id: marker._id,
+ fenceStartedAt: marker.fenceStartedAt,
+ });
continue;
}
+ await deps.purgeQueuedTurnsForUser?.(marker._id);
await Promise.all([
Delivery().deleteMany({ user: marker._id }),
LaneSequence().deleteMany({ user: marker._id }),
@@ -3474,6 +3560,7 @@ export function createAgentTriggerDeliveryMethods(
}
async function deleteAgentTriggerDeliveriesByUser(user: string | Types.ObjectId): Promise<void> {
+ await deps.purgeQueuedTurnsForUser?.(user);
await Promise.all([Delivery().deleteMany({ user }), LaneSequence().deleteMany({ user })]);
await UserPurge().deleteOne({ _id: user });
}
diff --git a/packages/data-schemas/src/models/index.ts b/packages/data-schemas/src/models/index.ts
index ecf2f610e3..d309b04101 100644
--- a/packages/data-schemas/src/models/index.ts
+++ b/packages/data-schemas/src/models/index.ts
@@ -1,3 +1,4 @@
+import { createAgentQueuedTurnModel, createAgentQueuedTurnSequenceModel } from './queuedTurn';
import { createAgentTriggerLaneSequenceModel } from './triggerLaneSequence';
import { createScheduleModel, createScheduleRunModel } from './schedule';
import { createSkillSyncCredentialModel } from './skillSyncCredential';
@@ -89,6 +90,8 @@ export function createModels(mongoose: typeof import('mongoose')): {
AgentTriggerDelivery: ReturnType<typeof createAgentTriggerDeliveryModel>;
AgentTriggerLaneSequence: ReturnType<typeof createAgentTriggerLaneSequenceModel>;
AgentTriggerUserPurge: ReturnType<typeof createAgentTriggerUserPurgeModel>;
+ AgentQueuedTurn: ReturnType<typeof createAgentQueuedTurnModel>;
+ AgentQueuedTurnSequence: ReturnType<typeof createAgentQueuedTurnSequenceModel>;
Schedule: ReturnType<typeof createScheduleModel>;
ScheduleRun: ReturnType<typeof createScheduleRunModel>;
RefreshTokenBridge: ReturnType<typeof createRefreshTokenBridgeModel>;
@@ -136,6 +139,8 @@ export function createModels(mongoose: typeof import('mongoose')): {
AgentTriggerDelivery: createAgentTriggerDeliveryModel(mongoose),
AgentTriggerLaneSequence: createAgentTriggerLaneSequenceModel(mongoose),
AgentTriggerUserPurge: createAgentTriggerUserPurgeModel(mongoose),
+ AgentQueuedTurn: createAgentQueuedTurnModel(mongoose),
+ AgentQueuedTurnSequence: createAgentQueuedTurnSequenceModel(mongoose),
Schedule: createScheduleModel(mongoose),
ScheduleRun: createScheduleRunModel(mongoose),
RefreshTokenBridge: createRefreshTokenBridgeModel(mongoose),
diff --git a/packages/data-schemas/src/models/queuedTurn.ts b/packages/data-schemas/src/models/queuedTurn.ts
new file mode 100644
index 0000000000..039eed2731
--- /dev/null
+++ b/packages/data-schemas/src/models/queuedTurn.ts
@@ -0,0 +1,31 @@
+import type { Model } from 'mongoose';
+import type {
+ IAgentQueuedTurnDocument,
+ IAgentQueuedTurnSequenceDocument,
+} from '~/types/queuedTurn';
+import { applyTenantIsolation } from '~/models/plugins/tenantIsolation';
+import queuedTurnSequenceSchema from '~/schema/queuedTurnSequence';
+import queuedTurnSchema from '~/schema/queuedTurn';
+
+export function createAgentQueuedTurnModel(
+ mongoose: typeof import('mongoose'),
+): Model<IAgentQueuedTurnDocument> {
+ applyTenantIsolation(queuedTurnSchema);
+ return (
+ mongoose.models.AgentQueuedTurn ||
+ mongoose.model<IAgentQueuedTurnDocument>('AgentQueuedTurn', queuedTurnSchema)
+ );
+}
+
+export function createAgentQueuedTurnSequenceModel(
+ mongoose: typeof import('mongoose'),
+): Model<IAgentQueuedTurnSequenceDocument> {
+ applyTenantIsolation(queuedTurnSequenceSchema);
+ return (
+ mongoose.models.AgentQueuedTurnSequence ||
+ mongoose.model<IAgentQueuedTurnSequenceDocument>(
+ 'AgentQueuedTurnSequence',
+ queuedTurnSequenceSchema,
+ )
+ );
+}
diff --git a/packages/data-schemas/src/schema/index.ts b/packages/data-schemas/src/schema/index.ts
index 966d49bf87..2b9f6255eb 100644
--- a/packages/data-schemas/src/schema/index.ts
+++ b/packages/data-schemas/src/schema/index.ts
@@ -38,5 +38,7 @@ export { default as configSchema } from './config';
export { default as triggerDeliverySchema } from './triggerDelivery';
export { default as triggerLaneSequenceSchema } from './triggerLaneSequence';
export { default as triggerUserPurgeSchema } from './triggerUserPurge';
+export { default as queuedTurnSchema } from './queuedTurn';
+export { default as queuedTurnSequenceSchema } from './queuedTurnSequence';
export { default as scheduleSchema } from './schedule';
export { default as scheduleRunSchema } from './scheduleRun';
diff --git a/packages/data-schemas/src/schema/queuedTurn.ts b/packages/data-schemas/src/schema/queuedTurn.ts
new file mode 100644
index 0000000000..22f9cc1a93
--- /dev/null
+++ b/packages/data-schemas/src/schema/queuedTurn.ts
@@ -0,0 +1,156 @@
+import { Schema } from 'mongoose';
+import type { IAgentQueuedTurnDocument } from '~/types/queuedTurn';
+
+const fileRefSchema = new Schema(
+ {
+ file_id: { type: String, required: true, maxlength: 256 },
+ type: { type: String, maxlength: 256 },
+ filepath: { type: String, maxlength: 2048 },
+ filename: { type: String, maxlength: 1024 },
+ height: { type: Number, min: 0 },
+ width: { type: Number, min: 0 },
+ bytes: { type: Number, min: 0 },
+ },
+ { _id: false },
+);
+
+const failureSchema = new Schema(
+ {
+ code: { type: String, required: true, maxlength: 128 },
+ message: { type: String, required: true, maxlength: 2048 },
+ },
+ { _id: false },
+);
+
+const terminalReceiptSchema = new Schema(
+ {
+ outcome: {
+ type: String,
+ enum: ['admitted', 'cancelled', 'dead'],
+ required: true,
+ },
+ settledAt: { type: Date, required: true },
+ admissionId: { type: String, maxlength: 128 },
+ admissionMode: { type: String, enum: ['warm', 'ordinary'] },
+ generationId: { type: String, maxlength: 256 },
+ generationCreatedAt: { type: Number, min: 0 },
+ failure: { type: failureSchema },
+ },
+ { _id: false },
+);
+
+const queuedTurnSchema: Schema<IAgentQueuedTurnDocument> = new Schema(
+ {
+ user: {
+ type: Schema.Types.ObjectId,
+ ref: 'User',
+ required: true,
+ index: true,
+ },
+ tenantId: { type: String, index: true },
+ conversationId: { type: String, required: true, maxlength: 256 },
+ agentId: { type: String, required: true, maxlength: 256 },
+ parentMessageId: { type: String, required: true, maxlength: 256 },
+ clientRequestId: { type: String, required: true, maxlength: 128 },
+ fingerprint: { type: String, required: true, minlength: 43, maxlength: 64 },
+ laneId: { type: String, required: true, maxlength: 128 },
+ sequence: { type: Number, min: 1 },
+ reservationWriterId: { type: String, maxlength: 128 },
+ activeSlot: { type: Number, min: 0, max: 99 },
+ status: {
+ type: String,
+ enum: ['reserving', 'queued', 'claimed', 'admitted', 'cancelled', 'dead'],
+ required: true,
+ default: 'reserving',
+ },
+ priority: { type: Boolean, required: true, default: false },
+ text: { type: String, required: true, maxlength: 32_768 },
+ files: { type: [fileRefSchema], default: undefined },
+ quotes: { type: [String], default: undefined },
+ manualSkills: { type: [String], default: undefined },
+ expectedPredecessorCreatedAt: { type: Number, min: 0 },
+ attempts: { type: Number, required: true, default: 0, min: 0 },
+ availableAt: { type: Date, required: true },
+ deliveryKey: { type: String, maxlength: 128 },
+ deliveryState: {
+ type: String,
+ enum: ['pending', 'publishing', 'published', 'retiring', 'retired'],
+ required: true,
+ default: 'pending',
+ },
+ scheduledAt: { type: Date },
+ claimId: { type: String, maxlength: 128 },
+ claimBy: { type: String, maxlength: 256 },
+ claimUntil: { type: Date },
+ admissionStartedAt: { type: Date },
+ admissionId: { type: String, maxlength: 128 },
+ admissionProtocolVersion: { type: Number, enum: [2] },
+ reconciliationAvailableAt: { type: Date },
+ reconciliationClaimId: { type: String, maxlength: 128 },
+ reconciliationClaimBy: { type: String, maxlength: 256 },
+ reconciliationClaimUntil: { type: Date },
+ reconciliationAttempts: { type: Number, min: 0 },
+ terminalReceipt: { type: terminalReceiptSchema },
+ },
+ { timestamps: true },
+);
+
+queuedTurnSchema.index(
+ { tenantId: 1, user: 1, conversationId: 1, clientRequestId: 1 },
+ { unique: true },
+);
+queuedTurnSchema.index(
+ { tenantId: 1, user: 1, conversationId: 1, sequence: 1 },
+ {
+ name: 'agent_queued_turn_sequence',
+ unique: true,
+ partialFilterExpression: { sequence: { $exists: true } },
+ },
+);
+queuedTurnSchema.index(
+ { tenantId: 1, user: 1, conversationId: 1, activeSlot: 1 },
+ {
+ name: 'agent_queued_turn_active_capacity',
+ unique: true,
+ partialFilterExpression: { activeSlot: { $exists: true } },
+ },
+);
+queuedTurnSchema.index(
+ {
+ tenantId: 1,
+ user: 1,
+ conversationId: 1,
+ status: 1,
+ priority: -1,
+ availableAt: 1,
+ sequence: 1,
+ },
+ { name: 'agent_queued_turn_claim' },
+);
+queuedTurnSchema.index(
+ {
+ tenantId: 1,
+ user: 1,
+ conversationId: 1,
+ status: 1,
+ priority: -1,
+ sequence: 1,
+ },
+ { name: 'agent_queued_turn_active' },
+);
+queuedTurnSchema.index({ tenantId: 1, user: 1, status: 1, claimUntil: 1 });
+queuedTurnSchema.index({ status: 1, createdAt: 1, _id: 1 });
+queuedTurnSchema.index({
+ status: 1,
+ reconciliationAvailableAt: 1,
+ reconciliationClaimUntil: 1,
+ _id: 1,
+});
+queuedTurnSchema.index({
+ status: 1,
+ scheduledAt: 1,
+ availableAt: 1,
+ sequence: 1,
+});
+
+export default queuedTurnSchema;
diff --git a/packages/data-schemas/src/schema/queuedTurnSequence.ts b/packages/data-schemas/src/schema/queuedTurnSequence.ts
new file mode 100644
index 0000000000..9b7e2a67b8
--- /dev/null
+++ b/packages/data-schemas/src/schema/queuedTurnSequence.ts
@@ -0,0 +1,28 @@
+import { Schema } from 'mongoose';
+import type { IAgentQueuedTurnSequenceDocument } from '~/types/queuedTurn';
+
+const queuedTurnSequenceSchema: Schema<IAgentQueuedTurnSequenceDocument> = new Schema(
+ {
+ _id: { type: String, required: true },
+ user: {
+ type: Schema.Types.ObjectId,
+ ref: 'User',
+ required: true,
+ index: true,
+ },
+ tenantId: { type: String, index: true },
+ conversationId: { type: String, required: true, maxlength: 256 },
+ laneId: { type: String, required: true, maxlength: 128 },
+ value: { type: Number, required: true, min: 0 },
+ reservationId: { type: String, maxlength: 24 },
+ writerId: { type: String, maxlength: 128 },
+ writerUntil: { type: Date },
+ retiredAt: { type: Date },
+ expiresAt: { type: Date },
+ },
+ { timestamps: true },
+);
+
+queuedTurnSequenceSchema.index({ expiresAt: 1 }, { expireAfterSeconds: 0 });
+
+export default queuedTurnSequenceSchema;
diff --git a/packages/data-schemas/src/types/index.ts b/packages/data-schemas/src/types/index.ts
index 6d5aa1fbcf..3dea872423 100644
--- a/packages/data-schemas/src/types/index.ts
+++ b/packages/data-schemas/src/types/index.ts
@@ -34,6 +34,7 @@ export * from './prompts';
export * from './skill';
export * from './skillSync';
export * from './triggerDelivery';
+export * from './queuedTurn';
export * from './schedule';
/* Access Control */
export * from './accessRole';
diff --git a/packages/data-schemas/src/types/queuedTurn.ts b/packages/data-schemas/src/types/queuedTurn.ts
new file mode 100644
index 0000000000..ba9f47c75f
--- /dev/null
+++ b/packages/data-schemas/src/types/queuedTurn.ts
@@ -0,0 +1,154 @@
+import type { Document, Types } from 'mongoose';
+
+export type AgentQueuedTurnStatus =
+ | 'reserving'
+ | 'queued'
+ | 'claimed'
+ | 'admitted'
+ | 'cancelled'
+ | 'dead';
+
+export type AgentQueuedTurnDeliveryState =
+ | 'pending'
+ | 'publishing'
+ | 'published'
+ | 'retiring'
+ | 'retired';
+
+export interface AgentQueuedTurnFileRef {
+ file_id: string;
+ type?: string;
+ filepath?: string;
+ filename?: string;
+ height?: number;
+ width?: number;
+ bytes?: number;
+}
+
+export interface AgentQueuedTurnFailure {
+ code: string;
+ message: string;
+}
+
+export interface AgentQueuedTurnTerminalReceipt {
+ outcome: 'admitted' | 'cancelled' | 'dead';
+ settledAt: Date;
+ admissionId?: string;
+ admissionMode?: 'warm' | 'ordinary';
+ generationId?: string;
+ generationCreatedAt?: number;
+ failure?: AgentQueuedTurnFailure;
+}
+
+export interface IAgentQueuedTurn {
+ _id?: Types.ObjectId;
+ user: Types.ObjectId;
+ tenantId?: string;
+ conversationId: string;
+ agentId: string;
+ parentMessageId: string;
+ clientRequestId: string;
+ fingerprint: string;
+ /** Immutable generation of the conversation lane that admitted this row. */
+ laneId?: string;
+ /** Assigned after the row is durably visible as `reserving`. */
+ sequence?: number;
+ /** Fences a reserving row to the lane writer allowed to assign its sequence. */
+ reservationWriterId?: string;
+ /** Bounded active-lane capacity token. Present only while queued/claimed. */
+ activeSlot?: number;
+ status: AgentQueuedTurnStatus;
+ priority: boolean;
+ text: string;
+ files?: AgentQueuedTurnFileRef[];
+ quotes?: string[];
+ manualSkills?: string[];
+ expectedPredecessorCreatedAt?: number;
+ attempts: number;
+ availableAt: Date;
+ deliveryKey?: string;
+ deliveryState?: AgentQueuedTurnDeliveryState;
+ scheduledAt?: Date;
+ claimId?: string;
+ claimBy?: string;
+ claimUntil?: Date;
+ /** Durable proof that ordinary admission may have crossed the HTTP boundary. */
+ admissionStartedAt?: Date;
+ admissionId?: string;
+ /** Version 2 requires accepted and deduplicated execution responses to prove
+ * the exact post-invocation source receipt. */
+ admissionProtocolVersion?: 2;
+ reconciliationAvailableAt?: Date;
+ reconciliationClaimId?: string;
+ reconciliationClaimBy?: string;
+ reconciliationClaimUntil?: Date;
+ reconciliationAttempts?: number;
+ terminalReceipt?: AgentQueuedTurnTerminalReceipt;
+ createdAt?: Date;
+ updatedAt?: Date;
+}
+
+export interface IAgentQueuedTurnDocument extends Omit<IAgentQueuedTurn, '_id'>, Document {}
+
+export interface IAgentQueuedTurnSequence {
+ _id: string;
+ user: Types.ObjectId;
+ tenantId?: string;
+ conversationId: string;
+ /** Changes whenever a fully retired lane is recreated. */
+ laneId: string;
+ value: number;
+ /** Visible reservation currently owning `value`; recovery completes it. */
+ reservationId?: string;
+ writerId?: string;
+ writerUntil?: Date;
+ retiredAt?: Date;
+ expiresAt?: Date;
+ createdAt?: Date;
+ updatedAt?: Date;
+}
+
+export interface IAgentQueuedTurnSequenceDocument
+ extends Omit<IAgentQueuedTurnSequence, '_id'>,
+ Document<string> {}
+
+export interface AgentQueuedTurnRecord
+ extends Omit<IAgentQueuedTurn, '_id' | 'createdAt' | 'sequence' | 'status'> {
+ queuedTurnId: string;
+ sequence: number;
+ status: Exclude<AgentQueuedTurnStatus, 'reserving'>;
+ createdAt: Date;
+}
+
+export type AgentQueuedTurnActiveRecord = Pick<
+ AgentQueuedTurnRecord,
+ | 'queuedTurnId'
+ | 'conversationId'
+ | 'agentId'
+ | 'parentMessageId'
+ | 'clientRequestId'
+ | 'sequence'
+ | 'activeSlot'
+ | 'status'
+ | 'priority'
+ | 'text'
+ | 'files'
+ | 'quotes'
+ | 'manualSkills'
+ | 'expectedPredecessorCreatedAt'
+ | 'attempts'
+ | 'availableAt'
+ | 'deliveryKey'
+ | 'deliveryState'
+ | 'scheduledAt'
+ | 'createdAt'
+ | 'updatedAt'
+ | 'terminalReceipt'
+>;
+
+export interface AgentQueuedTurnClaim extends AgentQueuedTurnRecord {
+ status: 'claimed';
+ claimId: string;
+ claimBy: string;
+ claimUntil: Date;
+}
diff --git a/packages/data-schemas/src/types/triggerDelivery.ts b/packages/data-schemas/src/types/triggerDelivery.ts
index b8c68a0ddf..7e91c341fc 100644
--- a/packages/data-schemas/src/types/triggerDelivery.ts
+++ b/packages/data-schemas/src/types/triggerDelivery.ts
@@ -15,6 +15,7 @@ export type AgentTriggerDeliveryStatus =
export const AGENT_TRIGGER_WORKER_CAPABILITY_DETACHED_ACTION_V1 = 'event_actor_detached_action_v1';
export const AGENT_TRIGGER_WORKER_CAPABILITY_BACKGROUND_COMPLETION_V1 =
'background_tool_completion_v1';
+export const AGENT_TRIGGER_WORKER_CAPABILITY_QUEUED_TURN_V1 = 'agent_queued_turn_v1';
export type AgentTriggerDeliveryOutcome = 'succeeded' | 'retry' | 'dead';
export interface AgentTriggerHandlingState {