Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(chat): long conversations no longer fail with a context-length error (#15923)

Hazem Adel3b227019c010
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.md
index ab6278b87cd..bc2b2300bfd 100644
--- a/brain/knowledge/flows-execution/chat.md
+++ b/brain/knowledge/flows-execution/chat.md
@@ -37,7 +37,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-
- **Write-check gate**: before a live `ap_test_flow`, `__flow_write_check` RPC flags write/destructive PIECE steps; read-only flows run ungated; gate fails open on RPC error.
- **Cloud rollout cap**: opens to non-embed users without `chatEnabled` until 200 distinct users have sent a message (`CLOUD_CHAT_ROLLOUT_CAP`); grandfathered after close. Embedded sessions never see chat.
- **Flow correctness is 100% prompt/guide-driven — nothing in code enforces it.** The "#1 silent bug" ("Class A"): the agent frames a *recurring* automation as a *one-time task* and omits any anti-reprocessing step, so run N+1 redoes run N's work (re-pays, re-sends). It's a design-time reasoning gap, not a testing gap — `ap_test_flow` runs ONCE, so a single test looks perfect; the bug only shows on the 2nd run. Fix lives in the prompt (`chat-system-prompt.md` `<decision_framework>` + `build_flow.md` "Recurring flows must not reprocess") + capability eval fixtures with a `recurring_avoids_reprocessing` judge dimension. The platform already has every primitive (Tables New-Record webhook, polling `DedupeStrategy`, `_dedupe_key`, Store, update/delete-record); the agent just wasn't reaching for them. Watch the `build_flow.md` "don't over-build" bias — it once actively discouraged the fix.
-- **The context budget ignores tool schemas and reserved `max_tokens`.** Anthropic/OpenRouter count both against the 200k window; `agent-compaction.ts` budgets neither. It trims history to `COMPACTION_THRESHOLD (0.7) × 200_000 = 140_000` and its fit check looks only at message chars, while `run-agent-turn.ts` sets `maxOutputTokens: tier.thinkingBudget + 32_000` → 52k reserved on premium, plus ~12k of tool schemas (62 tools, 41 via MCP). 140k + 12k + 52k = 204k, so a conversation that compacts to just under the threshold still 400s with "maximum context length is 200000 tokens" — and it gets retried ~6× (`streamText maxRetries: 3` × `MAX_STREAM_RETRIES`), burning ~20s per turn. `maxOutputTokens` is set at the `streamText` call level, so the full thinking budget stays reserved even on step one where `prepareStep` disables thinking and swaps in haiku-4.5 (real case: 148_628 text + 11_872 tool + 52_000 output = 212_500; dropping the unused 20k reservation alone would have fit). `ESTIMATED_TOKENS_PER_MESSAGE = 200` also sizes the recent window by message *count*, so a 12-message history holding ~235k tokens of uploaded documents summarized only 1 message. When budgeting, subtract the reserved output window and tool-schema size from `getMaxContextTokens`, and don't reserve `thinkingBudget` on a thinking-disabled step.
+- **The context budget subtracts what the request reserves.** Anthropic/OpenRouter count tool schemas and the reserved `max_tokens` against the 200k window, so `getAgentConfig` passes `reservedTokens` (the clamped output window from `agentAiUtils.affordableOutputTokens`, the same helper the worker uses for `maxOutputTokens`, plus `TOOL_SCHEMA_TOKEN_ESTIMATE = 12_000`) into `agentCompaction`. The threshold, the fit check and the recent window are all measured against `maxContext - reservedTokens`, and the recent window is sized by estimated tokens, not message count. The tool-schema figure is a constant because the tools are built in the worker and the API never sees their definitions. The worker sets `maxOutputTokens` per step in `prepareStep`, so a thinking-disabled step (round one, discovery phase) does not reserve `thinkingBudget`. The system prompt's size is added to the reserve for the window and fit check, and each message is capped at 20k chars in the summary request so the summarizer itself cannot overflow.
- **A write tool in `BUILD_ONLY_TOOL_NAMES` is only reachable if something flips the phase for it.** The denylist is the consistent home for anything that writes (`ap_create_flow`, `ap_create_table`, `ap_lock_and_publish` are all in it), but the only route out of `discovery` is `ap_set_phase`, whose description tells the model to switch when it starts *building an automation*. A tool for a subject with no build guide and no sibling build-only call, the agent-building tools being the case that found this, becomes invisible in any conversation that never builds a flow: `activeToolsForPhase` filters it out and the prompt names no tool, so the model cannot discover that it exists. Classifying by "does it write" is not enough; check what would actually flip the phase in a conversation about *that* subject. The four agent *write* tools (`ap_create_agent`, `ap_update_agent`, `ap_add_agent_tool`, `ap_remove_agent_tool`) were added to the set and then reverted for exactly this, with a test pinning the choice; `ap_list_agents` is a read and was never in it, so the group is five tools and only four were ever candidates.
- **Capability notes are built where `discoveryOnly` is not known, so a prompt can promise tools the worker has stripped.** `getAgentConfig` composes the system prompt in the api, while `discoveryOnly` rides on the job data; before Aug 2026 it never crossed that boundary. Meanwhile the worker strips image tools, email tools and the agent tools on such a run, so the notes claimed all three. Two of the three had been wrong since long before anyone noticed, because each note computed its own availability term. The flag now travels with the config request and the three notes read one shared `actingRun = !dryRun && !discoveryOnly`. Whenever a tool group is gated on a run mode in the worker, the note that advertises it has to be gated on the same term, in one place.
- **Tool output can't be trusted to waive a charge.** A user's own MCP server or `ap_run_code` controls its output's top-level keys, and they reach the persisted part unchanged. Chat billing only honours `billedAtCost` from `ap_web_search` and `ap_generate_image`; a new built-in tool that bills at cost must be added to `TOOLS_THAT_BILL_AT_COST` in `chatBilling`.
diff --git a/packages/server/api/src/app/ee/agent/agent-compaction.ts b/packages/server/api/src/app/ee/agent/agent-compaction.ts
index 5c22903a735..889d0dd5bfa 100644
--- a/packages/server/api/src/app/ee/agent/agent-compaction.ts
+++ b/packages/server/api/src/app/ee/agent/agent-compaction.ts
@@ -10,8 +10,8 @@ const COMPACTION_THRESHOLD = 0.7
const RECENT_WINDOW_RATIO = 0.3
const CHARS_PER_TOKEN_ESTIMATE = 4
const MIN_MESSAGES_BEFORE_COMPACTION = 6
-const ESTIMATED_TOKENS_PER_MESSAGE = 200
const MAX_TOOL_RESULT_CHARS_FOR_SUMMARY = 2_000
+const SUMMARY_OUTPUT_RESERVE_TOKENS = 4_000
const COMPACTION_SYSTEM_PROMPT = readFileSync(
path.resolve('packages/server/api/src/assets/prompts/chat-compaction-prompt.md'),
@@ -22,20 +22,33 @@ function estimateTokenCount({ messages, systemPromptLength }: {
messages: ModelMessage[]
systemPromptLength: number
}): number {
- const totalChars = JSON.stringify(messages).length + systemPromptLength
- return Math.ceil(totalChars / CHARS_PER_TOKEN_ESTIMATE)
+ return tokensIn(JSON.stringify(messages)) + Math.ceil(systemPromptLength / CHARS_PER_TOKEN_ESTIMATE)
}
-function shouldCompact({ estimatedTokens, provider, messageCount }: {
+function contextBudget({ provider, reservedTokens }: { provider: AIProviderName, reservedTokens: number }): number {
+ return Math.max(0, aiProviderUtils.getMaxContextTokens({ provider }) - reservedTokens)
+}
+
+function recentWindowSizeFor({ messages, targetTokens }: { messages: ModelMessage[], targetTokens: number }): number {
+ let tokens = 0
+ let size = 0
+ for (let i = messages.length - 1; i > 0 && tokens < targetTokens; i--) {
+ tokens += tokensIn(JSON.stringify(messages[i]))
+ size++
+ }
+ return Math.min(Math.max(2, size), messages.length - 1)
+}
+
+function shouldCompact({ estimatedTokens, provider, messageCount, reservedTokens }: {
estimatedTokens: number
provider: AIProviderName
messageCount: number
+ reservedTokens: number
}): boolean {
if (messageCount < MIN_MESSAGES_BEFORE_COMPACTION) {
return false
}
- const maxContext = aiProviderUtils.getMaxContextTokens({ provider })
- return estimatedTokens > maxContext * COMPACTION_THRESHOLD
+ return estimatedTokens > contextBudget({ provider, reservedTokens }) * COMPACTION_THRESHOLD
}
/**
@@ -55,20 +68,19 @@ function snapToSafeMessageBoundary({ messages, rawCutoff }: {
return idx
}
-async function compactMessages({ messages, existingSummary, summarizedUpToIndex, provider, model, log }: {
+async function compactMessages({ messages, existingSummary, summarizedUpToIndex, provider, reservedTokens, model, log }: {
messages: ModelMessage[]
existingSummary: string | null
summarizedUpToIndex: number | null
provider: AIProviderName
+ reservedTokens: number
model: LanguageModel
log: FastifyBaseLogger
}): Promise<{ summary: string, summarizedUpToIndex: number }> {
- const maxContext = aiProviderUtils.getMaxContextTokens({ provider })
- const targetRecentTokens = maxContext * RECENT_WINDOW_RATIO
- const recentWindowSize = Math.min(
- Math.max(2, Math.floor(targetRecentTokens / ESTIMATED_TOKENS_PER_MESSAGE)),
- messages.length - 1,
- )
+ const recentWindowSize = recentWindowSizeFor({
+ messages,
+ targetTokens: contextBudget({ provider, reservedTokens }) * RECENT_WINDOW_RATIO,
+ })
const rawCutoff = messages.length - recentWindowSize
const newCutoffIndex = snapToSafeMessageBoundary({ messages, rawCutoff })
@@ -83,12 +95,17 @@ async function compactMessages({ messages, existingSummary, summarizedUpToIndex,
contentToSummarize += `Previous conversation summary:\n${existingSummary}\n\nNew messages since last summary:\n`
}
- for (const msg of messagesToSummarize) {
- const content = extractTextContent(msg)
+ const texts = messagesToSummarize.map((msg) => extractTextContent(msg))
+ const summaryInputTokens = contextBudget({ provider, reservedTokens: SUMMARY_OUTPUT_RESERVE_TOKENS }) * COMPACTION_THRESHOLD
+ - tokensIn(COMPACTION_SYSTEM_PROMPT)
+ - tokensIn(contentToSummarize)
+ const perMessageTokens = fairShareCap({ lengths: texts.map(tokensIn), budget: summaryInputTokens })
+ messagesToSummarize.forEach((msg, i) => {
+ const content = truncateToTokens({ text: texts[i], maxTokens: perMessageTokens })
if (content) {
contentToSummarize += `[${msg.role}]: ${content}\n`
}
- }
+ })
log.info({
totalMessages: messages.length,
@@ -108,11 +125,12 @@ async function compactMessages({ messages, existingSummary, summarizedUpToIndex,
return { summary, summarizedUpToIndex: newCutoffIndex }
}
-function buildCompactedPayload({ messages, summary, summarizedUpToIndex, provider }: {
+function buildCompactedPayload({ messages, summary, summarizedUpToIndex, provider, reservedTokens }: {
messages: ModelMessage[]
summary: string | null
summarizedUpToIndex: number | null
provider: AIProviderName
+ reservedTokens: number
}): ModelMessage[] {
if (!summary || summarizedUpToIndex === null) {
return messages
@@ -121,19 +139,18 @@ function buildCompactedPayload({ messages, summary, summarizedUpToIndex, provide
const recentMessages = messages.slice(summarizedUpToIndex)
const summaryText = `[Previous conversation summary]\n${summary}\n[End of summary — conversation continues below]`
- const maxContext = aiProviderUtils.getMaxContextTokens({ provider })
- const threshold = maxContext * COMPACTION_THRESHOLD
- const summaryCharLen = JSON.stringify(summaryText).length
- const recentLengths = recentMessages.map((m) => JSON.stringify(m).length)
+ const budget = contextBudget({ provider, reservedTokens })
+ const threshold = budget * COMPACTION_THRESHOLD
+ const recentTokens = recentMessages.map((m) => tokensIn(JSON.stringify(m)))
- let runningCharLen = summaryCharLen + recentLengths.reduce((a, b) => a + b, 0)
+ let runningTokens = tokensIn(JSON.stringify(summaryText)) + recentTokens.reduce((a, b) => a + b, 0)
let startIdx = 0
while (
startIdx < recentMessages.length - 1
- && (Math.ceil(runningCharLen / CHARS_PER_TOKEN_ESTIMATE) > threshold || recentMessages[startIdx].role === 'tool')
+ && (runningTokens > threshold || recentMessages[startIdx].role === 'tool')
) {
- runningCharLen -= recentLengths[startIdx]
+ runningTokens -= recentTokens[startIdx]
startIdx++
}
@@ -152,9 +169,7 @@ function buildCompactedPayload({ messages, summary, summarizedUpToIndex, provide
...trimmedRecent.slice(1),
]
: [{ role: 'user', content: summaryText }, ...trimmedRecent]
- const finalEstimate = Math.ceil(runningCharLen / CHARS_PER_TOKEN_ESTIMATE)
-
- if (finalEstimate > maxContext) {
+ if (runningTokens > budget) {
throw new ActivepiecesError({
code: ErrorCode.CHAT_CONTEXT_LIMIT_EXCEEDED,
params: {},
@@ -164,10 +179,47 @@ function buildCompactedPayload({ messages, summary, summarizedUpToIndex, provide
return finalPayload
}
-function truncateForSummary(output: string): string {
+function fairShareCap({ lengths, budget }: { lengths: number[], budget: number }): number {
+ const ascending = [...lengths].sort((a, b) => a - b)
+ let remaining = budget
+ for (let i = 0; i < ascending.length; i++) {
+ const share = Math.floor(remaining / (ascending.length - i))
+ if (ascending[i] > share) {
+ return Math.max(0, share)
+ }
+ remaining -= ascending[i]
+ }
+ return Number.POSITIVE_INFINITY
+}
+
+function tokenCost(codePoint: string): number {
+ return codePoint.charCodeAt(0) < 128 ? 1 / CHARS_PER_TOKEN_ESTIMATE : 1
+}
+
+function tokensIn(text: string): number {
+ let tokens = 0
+ for (const codePoint of text) {
+ tokens += tokenCost(codePoint)
+ }
+ return Math.ceil(tokens)
+}
+
+function truncateToTokens({ text, maxTokens }: { text: string, maxTokens: number }): string {
+ if (tokensIn(text) <= maxTokens) return text
+ const codePoints = [...text]
+ let used = 0
+ let kept = 0
+ while (kept < codePoints.length && used + tokenCost(codePoints[kept]) <= maxTokens) {
+ used += tokenCost(codePoints[kept])
+ kept++
+ }
+ return `${codePoints.slice(0, kept).join('')}…[truncated ${codePoints.length - kept} chars]`
+}
+
+function truncateForSummary({ output, limit }: { output: string, limit: number }): string {
const codePoints = [...output]
- if (codePoints.length <= MAX_TOOL_RESULT_CHARS_FOR_SUMMARY) return output
- return `${codePoints.slice(0, MAX_TOOL_RESULT_CHARS_FOR_SUMMARY).join('')}…[truncated ${codePoints.length - MAX_TOOL_RESULT_CHARS_FOR_SUMMARY} chars]`
+ if (codePoints.length <= limit) return output
+ return `${codePoints.slice(0, limit).join('')}…[truncated ${codePoints.length - limit} chars]`
}
function extractTextContent(message: ModelMessage): string {
@@ -187,7 +239,7 @@ function extractTextContent(message: ModelMessage): string {
}
else if (part.type === 'tool-result' && 'output' in part) {
const output = typeof part.output === 'string' ? part.output : JSON.stringify(part.output)
- text += `[Tool result: ${truncateForSummary(output)}]`
+ text += `[Tool result: ${truncateForSummary({ output, limit: MAX_TOOL_RESULT_CHARS_FOR_SUMMARY })}]`
}
}
}
diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
index c2e80fc9fe3..26c56491db0 100644
--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
@@ -204,10 +204,13 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
})
await agentApprovalGate.clearCancel({ conversationId })
+ const reservedOutputTokens = await agentAiUtils.affordableOutputTokens({ provider: providerConfig.provider, modelIds: [resolvedModelId, fastModelId], thinkingBudget: tier.thinkingBudget })
+ const reservedTokens = reservedOutputTokens + TOOL_SCHEMA_TOKEN_ESTIMATE
+ const payloadReservedTokens = reservedTokens + agentAiUtils.estimateTokenCount({ messages: [], systemPromptLength: systemPromptText.length })
const estimatedTokens = agentCompaction.estimateTokenCount({ messages: llmHistory, systemPromptLength: systemPromptText.length })
let compactionState = { summary: conversation.summary ?? null, summarizedUpToIndex: conversation.summarizedUpToIndex ?? null }
- const willCompact = agentCompaction.shouldCompact({ estimatedTokens, provider: providerConfig.provider, messageCount: llmHistory.length })
+ const willCompact = agentCompaction.shouldCompact({ estimatedTokens, provider: providerConfig.provider, messageCount: llmHistory.length, reservedTokens })
log.debug({ estimatedTokens, willCompact, messageCount: llmHistory.length, systemPromptLength: systemPromptText.length }, '[agentRpc#getAgentConfig] Compaction decision')
if (willCompact) {
const model = aiUtils.createModel({
@@ -219,6 +222,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
existingSummary: compactionState.summary,
summarizedUpToIndex: compactionState.summarizedUpToIndex,
provider: providerConfig.provider,
+ reservedTokens: payloadReservedTokens,
model,
log,
})
@@ -234,6 +238,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
summary: compactionState.summary,
summarizedUpToIndex: compactionState.summarizedUpToIndex,
provider: providerConfig.provider,
+ reservedTokens: payloadReservedTokens,
})
log.info({
@@ -275,3 +280,5 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
},
})
+
+const TOOL_SCHEMA_TOKEN_ESTIMATE = 12_000
diff --git a/packages/server/api/test/unit/app/agent/agent-compaction.test.ts b/packages/server/api/test/unit/app/agent/agent-compaction.test.ts
index 98e13544913..38fd7293c23 100644
--- a/packages/server/api/test/unit/app/agent/agent-compaction.test.ts
+++ b/packages/server/api/test/unit/app/agent/agent-compaction.test.ts
@@ -1,8 +1,18 @@
import { AIProviderName, ErrorCode } from '@activepieces/core-utils'
-import { ModelMessage } from 'ai'
-import { describe, expect, it } from 'vitest'
+import { generateText, ModelMessage } from 'ai'
+import { MockLanguageModelV3 } from 'ai/test'
+import Fastify from 'fastify'
+import { describe, expect, it, vi } from 'vitest'
import { agentCompaction } from '../../../../src/app/ee/agent/agent-compaction'
+vi.mock('ai', async (importOriginal) => ({
+ ...(await importOriginal<Record<string, unknown>>()),
+ generateText: vi.fn().mockResolvedValue({ text: 'summary' }),
+}))
+
+const summaryModel = new MockLanguageModelV3()
+const silentLog = Fastify({ logger: false }).log
+
function makeMessages(count: number, charsPer = 100): ModelMessage[] {
return Array.from({ length: count }, (_, i) => ({
role: i % 2 === 0 ? 'user' as const : 'assistant' as const,
@@ -36,6 +46,7 @@ describe('agentCompaction.shouldCompact', () => {
const result = agentCompaction.shouldCompact({
estimatedTokens: 999_999,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
messageCount: 5,
})
expect(result).toBe(false)
@@ -46,6 +57,7 @@ describe('agentCompaction.shouldCompact', () => {
const result = agentCompaction.shouldCompact({
estimatedTokens: 100_000,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
messageCount: 20,
})
expect(result).toBe(false)
@@ -56,6 +68,7 @@ describe('agentCompaction.shouldCompact', () => {
const result = agentCompaction.shouldCompact({
estimatedTokens: 150_000,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
messageCount: 20,
})
expect(result).toBe(true)
@@ -66,6 +79,7 @@ describe('agentCompaction.shouldCompact', () => {
const result = agentCompaction.shouldCompact({
estimatedTokens: 150_000,
provider: AIProviderName.GOOGLE,
+ reservedTokens: 0,
messageCount: 20,
})
expect(result).toBe(false)
@@ -80,6 +94,7 @@ describe('agentCompaction.buildCompactedPayload', () => {
summary: null,
summarizedUpToIndex: null,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
})
expect(result).toBe(messages)
})
@@ -91,6 +106,7 @@ describe('agentCompaction.buildCompactedPayload', () => {
summary: 'User discussed flow creation.',
summarizedUpToIndex: 7,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
})
expect(result.length).toBe(4) // 1 summary + 3 recent (index 7,8,9)
@@ -115,6 +131,7 @@ describe('agentCompaction.buildCompactedPayload', () => {
summary: 'Short summary.',
summarizedUpToIndex: 5,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
})
// Should have trimmed some recent messages
@@ -134,6 +151,7 @@ describe('agentCompaction.buildCompactedPayload', () => {
summary: 'Summary',
summarizedUpToIndex: 0,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
})).toThrow(expect.objectContaining({
error: expect.objectContaining({
code: ErrorCode.CHAT_CONTEXT_LIMIT_EXCEEDED,
@@ -159,6 +177,7 @@ describe('agentCompaction.buildCompactedPayload', () => {
summary: 'Summary of earlier messages.',
summarizedUpToIndex: 4,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
})
// First message should be summary, second should NOT be a tool message
@@ -177,9 +196,119 @@ describe('agentCompaction.buildCompactedPayload', () => {
summary: 'Brief summary.',
summarizedUpToIndex: 15,
provider: AIProviderName.ANTHROPIC,
+ reservedTokens: 0,
})
// 1 summary + 5 recent messages (index 15-19)
expect(result.length).toBe(6)
})
})
+
+describe('agentCompaction with reserved tokens', () => {
+ const RESERVED_TOKENS = 11_872 + 52_000
+
+ it('compacts a history that fits the window only if output and tool schemas were free', () => {
+ expect(agentCompaction.shouldCompact({
+ estimatedTokens: 110_000,
+ provider: AIProviderName.ANTHROPIC,
+ messageCount: 20,
+ reservedTokens: RESERVED_TOKENS,
+ })).toBe(true)
+ })
+
+ it('keeps the payload plus reserved tokens inside the context window', () => {
+ const messages: ModelMessage[] = Array.from({ length: 12 }, (_, i) => ({
+ role: i % 2 === 0 ? 'user' as const : 'assistant' as const,
+ content: 'x'.repeat(78_000),
+ }))
+ const result = agentCompaction.buildCompactedPayload({
+ messages,
+ summary: 'summary',
+ summarizedUpToIndex: 0,
+ provider: AIProviderName.ANTHROPIC,
+ reservedTokens: RESERVED_TOKENS,
+ })
+ const payloadTokens = Math.ceil(JSON.stringify(result).length / 4)
+ expect(payloadTokens + RESERVED_TOKENS).toBeLessThanOrEqual(200_000)
+ })
+
+ it('sizes the recent window by tokens, so a few huge messages are summarized', async () => {
+ const messages: ModelMessage[] = Array.from({ length: 12 }, (_, i) => ({
+ role: i % 2 === 0 ? 'user' as const : 'assistant' as const,
+ content: 'x'.repeat(78_000),
+ }))
+ const result = await agentCompaction.compactMessages({
+ messages,
+ existingSummary: null,
+ summarizedUpToIndex: null,
+ provider: AIProviderName.ANTHROPIC,
+ reservedTokens: RESERVED_TOKENS,
+ model: summaryModel,
+ log: silentLog,
+ })
+ expect(result.summarizedUpToIndex).toBeGreaterThanOrEqual(8)
+ })
+
+ it('bounds the summary request so oversized documents cannot overflow it', async () => {
+ const messages: ModelMessage[] = Array.from({ length: 12 }, (_, i) => ({
+ role: i % 2 === 0 ? 'user' as const : 'assistant' as const,
+ content: 'x'.repeat(500_000),
+ }))
+ await agentCompaction.compactMessages({
+ messages,
+ existingSummary: null,
+ summarizedUpToIndex: null,
+ provider: AIProviderName.ANTHROPIC,
+ reservedTokens: RESERVED_TOKENS,
+ model: summaryModel,
+ log: silentLog,
+ })
+ const request = vi.mocked(generateText).mock.calls.at(-1)?.[0]
+ expect(String(request?.prompt).length / 4 + 4_000).toBeLessThan(200_000)
+ })
+
+ it('counts non-Latin text as a token per character, so a long Chinese document cannot overflow the summarizer', async () => {
+ const messages: ModelMessage[] = [
+ { role: 'user', content: '工作流'.repeat(60_000) },
+ ...Array.from({ length: 19 }, (_, i) => ({
+ role: i % 2 === 0 ? 'assistant' as const : 'user' as const,
+ content: i % 2 === 0 ? 'Noted.' : 'x'.repeat(29_000),
+ })),
+ ]
+ await agentCompaction.compactMessages({
+ messages,
+ existingSummary: null,
+ summarizedUpToIndex: null,
+ provider: AIProviderName.OPENROUTER,
+ reservedTokens: 66_000,
+ model: summaryModel,
+ log: silentLog,
+ })
+ const prompt = String(vi.mocked(generateText).mock.calls.at(-1)?.[0]?.prompt)
+ const chineseChars = [...prompt].filter((char) => char.charCodeAt(0) >= 128).length
+ const latinChars = [...prompt].length - chineseChars
+ expect(chineseChars + latinChars / 4).toBeLessThan(128_000 - 4_000)
+ })
+
+ it('keeps a long document whole in the summary request when the whole request fits', async () => {
+ const documentText = `START ${'d'.repeat(66_000)} THE-LAST-DETAIL`
+ const messages: ModelMessage[] = [
+ { role: 'user', content: documentText },
+ ...Array.from({ length: 19 }, (_, i) => ({
+ role: i % 2 === 0 ? 'assistant' as const : 'user' as const,
+ content: i % 2 === 0 ? 'Noted.' : 'x'.repeat(29_000),
+ })),
+ ]
+ await agentCompaction.compactMessages({
+ messages,
+ existingSummary: null,
+ summarizedUpToIndex: null,
+ provider: AIProviderName.OPENROUTER,
+ reservedTokens: 66_000,
+ model: summaryModel,
+ log: silentLog,
+ })
+ const request = vi.mocked(generateText).mock.calls.at(-1)?.[0]
+ expect(String(request?.prompt)).toContain('THE-LAST-DETAIL')
+ })
+})
diff --git a/packages/server/utils/src/agent-ai-utils.ts b/packages/server/utils/src/agent-ai-utils.ts
index 2f538ce405d..7cf901e05b2 100644
--- a/packages/server/utils/src/agent-ai-utils.ts
+++ b/packages/server/utils/src/agent-ai-utils.ts
@@ -3,9 +3,11 @@ import { agentPersistenceUtils, agentToolClassification, PersistedAgentPart, Per
import { agentProviderOptions } from './agent-provider-options'
import { ModelMessage, TelemetryOptions } from 'ai'
import { createEvlogIntegration } from 'evlog/ai'
+import { modelCatalog } from './model-catalog'
import { wideEvent } from './wide-event'
+const MAX_RESPONSE_OUTPUT_TOKENS = 32_000
const KEEP_RECENT_TOOL_RESULTS = 6
const COLLAPSE_OUTPUT_OVER_CHARS = 600
// Tool results that are the agent's working memory of an action's input schema — never collapsed,
@@ -385,7 +387,20 @@ function isPlainObject(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value)
}
+async function affordableOutputTokens({ provider, modelIds, thinkingBudget }: { provider: AIProviderName, modelIds: (string | undefined)[], thinkingBudget: number }): Promise<number> {
+ const catalog = await modelCatalog.load()
+ const ceilings = modelIds.map((modelId) => isNil(modelId) ? undefined : catalog.lookup({ provider, modelId })?.maxOutputTokens)
+ return clampOutputTokens({ thinkingBudget, ceilings })
+}
+
+function clampOutputTokens({ thinkingBudget, ceilings }: { thinkingBudget: number, ceilings: (number | undefined)[] }): number {
+ const known = ceilings.filter((ceiling) => !isNil(ceiling))
+ return Math.min(thinkingBudget + MAX_RESPONSE_OUTPUT_TOKENS, ...known)
+}
+
export const agentAiUtils = {
+ affordableOutputTokens,
+ clampOutputTokens,
stripThinkingBlocks,
sanitizeTruncatedAssistantTail,
collectStepMessages,
diff --git a/packages/server/utils/test/agent-ai-utils.test.ts b/packages/server/utils/test/agent-ai-utils.test.ts
index 1fdb69e4409..f7fd823538c 100644
--- a/packages/server/utils/test/agent-ai-utils.test.ts
+++ b/packages/server/utils/test/agent-ai-utils.test.ts
@@ -1,8 +1,9 @@
import { AIProviderName } from '@activepieces/core-utils'
import { ACTIVEPIECES_CHAT_TIERS, PersistedAgentPartType, PersistedToolCallStatus } from '@activepieces/shared'
import { ModelMessage } from 'ai'
-import { describe, expect, it } from 'vitest'
+import { describe, expect, it, vi } from 'vitest'
import { agentAiUtils } from '../src/agent-ai-utils'
+import { modelCatalog } from '../src/model-catalog'
const { sanitizeTruncatedAssistantTail } = agentAiUtils
@@ -498,3 +499,75 @@ describe('a message transform never hands the provider an empty history', () =>
expect(stripped[0].role).toBe('user')
})
})
+
+const catalogOf = (byId: Record<string, number>) => ({
+ lookup: ({ modelId }: { modelId: string }) => {
+ const stripped = modelId.replace(/^(us|eu|apac|global)\./, '')
+ const maxOutputTokens = byId[stripped]
+ return maxOutputTokens === undefined ? undefined : { maxOutputTokens }
+ },
+})
+
+describe('agentAiUtils.affordableOutputTokens', () => {
+ const SMART_TIER_THINKING = 10_000
+
+ it('reads through a region-prefixed Bedrock id, which is how the picker spells them', async () => {
+ vi.spyOn(modelCatalog, 'load').mockResolvedValue(catalogOf({ 'amazon.nova-pro-v1:0': 10_000 }))
+
+ await expect(agentAiUtils.affordableOutputTokens({
+ provider: AIProviderName.BEDROCK,
+ modelIds: ['eu.amazon.nova-pro-v1:0'],
+ thinkingBudget: SMART_TIER_THINKING,
+ })).resolves.toBe(10_000)
+ })
+
+ it('keeps the turn inside the smaller ceiling when the fast round runs a different model', async () => {
+ vi.spyOn(modelCatalog, 'load').mockResolvedValue(catalogOf({ 'amazon.nova-pro-v1:0': 10_000, 'amazon.nova-micro-v1:0': 5_000 }))
+
+ await expect(agentAiUtils.affordableOutputTokens({
+ provider: AIProviderName.BEDROCK,
+ modelIds: ['eu.amazon.nova-pro-v1:0', 'eu.amazon.nova-micro-v1:0'],
+ thinkingBudget: SMART_TIER_THINKING,
+ })).resolves.toBe(5_000)
+ })
+
+ it('leaves the budget alone for a model the catalog has never heard of', async () => {
+ vi.spyOn(modelCatalog, 'load').mockResolvedValue(catalogOf({}))
+
+ await expect(agentAiUtils.affordableOutputTokens({
+ provider: AIProviderName.BEDROCK,
+ modelIds: ['eu.some.brand-new-model-v9:0'],
+ thinkingBudget: SMART_TIER_THINKING,
+ })).resolves.toBe(42_000)
+ })
+
+ it('does not shrink a turn because the catalog was unreachable', async () => {
+ vi.spyOn(modelCatalog, 'load').mockResolvedValue({ lookup: () => undefined })
+
+ await expect(agentAiUtils.affordableOutputTokens({
+ provider: AIProviderName.BEDROCK,
+ modelIds: ['eu.amazon.nova-pro-v1:0', undefined],
+ thinkingBudget: SMART_TIER_THINKING,
+ })).resolves.toBe(42_000)
+ })
+})
+
+describe('agentAiUtils.clampOutputTokens', () => {
+ const SMART_TIER_THINKING = 10_000
+
+ it('asks for the full budget when no model declares a ceiling', () => {
+ expect(agentAiUtils.clampOutputTokens({ thinkingBudget: SMART_TIER_THINKING, ceilings: [undefined, undefined] })).toBe(42_000)
+ })
+
+ it('never asks a model for more than it accepts', () => {
+ expect(agentAiUtils.clampOutputTokens({ thinkingBudget: SMART_TIER_THINKING, ceilings: [10_000] })).toBe(10_000)
+ })
+
+ it('respects the smaller ceiling when the turn spans two models', () => {
+ expect(agentAiUtils.clampOutputTokens({ thinkingBudget: SMART_TIER_THINKING, ceilings: [64_000, 8_192] })).toBe(8_192)
+ })
+
+ it('leaves a generous ceiling alone rather than raising the ask to meet it', () => {
+ expect(agentAiUtils.clampOutputTokens({ thinkingBudget: SMART_TIER_THINKING, ceilings: [200_000] })).toBe(42_000)
+ })
+})
diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
index f3fc9209de3..0340a57f9a7 100644
--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
@@ -1,9 +1,8 @@
import { AIProviderName, ErrorCode, formatPieceError, isNil, isObject, isProviderBillingError, isTransientProviderError, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'
-import { agentAiUtils, ContentPartLike, modelCatalog } from '@activepieces/server-utils'
+import { agentAiUtils, ContentPartLike } from '@activepieces/server-utils'
import { AgentPhase, AgentRunSource, agentToolClassification, agentToolPhases, AI_PROVIDER_ENTITY_TYPES, aiProviderUtils, apErrorOf, CHAT_CREDITS_PER_TOOL_CALL, chatBilling, ChatToolCall, PersistedAgentPart } from '@activepieces/shared'
import { APICallError, generateText, isLoopFinished, isStepCount, LanguageModel, LanguageModelUsage, ModelMessage, NoSuchToolError, RetryError, StepResultPerformance, StopCondition, streamText, ToolExecutionOptions, ToolSet } from 'ai'
-const MAX_RESPONSE_OUTPUT_TOKENS = 32_000
const MAX_AUTO_CONTINUATIONS = 3
const MAX_EMPTY_CONTINUATIONS = 2
const MAX_STREAM_RETRIES = 1
@@ -91,7 +90,8 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
let lastFinishReason = ''
let budgetExceeded = false
- const maxOutputTokens = await affordableOutputTokens({ provider, modelIds: [modelId, fastModelId], thinkingBudget: tier.thinkingBudget })
+ const maxOutputTokens = await agentAiUtils.affordableOutputTokens({ provider, modelIds: [modelId, fastModelId], thinkingBudget: tier.thinkingBudget })
+ const maxOutputTokensWithoutThinking = agentAiUtils.clampOutputTokens({ thinkingBudget: 0, ceilings: [maxOutputTokens] })
const runStreamAttempt = (attemptMessages: ModelMessage[]): ReturnType<typeof streamText> => streamText({
model,
@@ -128,6 +128,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
return {
...(usesFastModel ? { model: fastModel } : {}),
activeTools: agentToolPhases.activeToolsForPhase({ phase: phaseState.phase, allToolNames }).filter((name) => paidToolsAffordable || !chatBilling.isPaidTool(name)),
+ maxOutputTokens: disableThinking ? maxOutputTokensWithoutThinking : maxOutputTokens,
providerOptions: agentAiUtils.buildProviderOptions({ provider, tier, modelId: lastStepModelId, disableThinking }),
...boundContextForStep({ baseMessages: attemptMessages, steps, systemPrompt, provider }),
}
@@ -412,17 +413,6 @@ export function looksEmptyResultText(text: string): boolean {
return /"found"\s*:\s*false|\bempty result\b|no results matched|"result"\s*:\s*\[\s*\]|"results"\s*:\s*\[\s*\]/i.test(text)
}
-export async function affordableOutputTokens({ provider, modelIds, thinkingBudget }: { provider: AIProviderName, modelIds: (string | undefined)[], thinkingBudget: number }): Promise<number> {
- const catalog = await modelCatalog.load()
- const ceilings = modelIds.map((modelId) => isNil(modelId) ? undefined : catalog.lookup({ provider, modelId })?.maxOutputTokens)
- return clampOutputTokens({ thinkingBudget, ceilings })
-}
-
-export function clampOutputTokens({ thinkingBudget, ceilings }: { thinkingBudget: number, ceilings: (number | undefined)[] }): number {
- const known = ceilings.filter((ceiling) => !isNil(ceiling))
- return Math.min(thinkingBudget + MAX_RESPONSE_OUTPUT_TOKENS, ...known)
-}
-
function completedToolCalls(steps: ReadonlyArray<{ toolResults: ReadonlyArray<{ toolName: string, output: unknown }> }>): ChatToolCall[] {
return steps.flatMap((step) => step.toolResults.map((result) => ({ toolName: result.toolName, output: result.output })))
}
diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/output-token-ceiling.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/output-token-ceiling.test.ts
deleted file mode 100644
index 110d145c881..00000000000
--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/output-token-ceiling.test.ts
+++ /dev/null
@@ -1,63 +0,0 @@
-import { AIProviderName } from '@activepieces/core-utils'
-import { describe, expect, it, vi } from 'vitest'
-
-const { mockLoad } = vi.hoisted(() => ({ mockLoad: vi.fn() }))
-
-vi.mock('@activepieces/server-utils', async (importOriginal) => ({
- ...(await importOriginal<Record<string, unknown>>()),
- modelCatalog: { load: mockLoad },
-}))
-
-const { affordableOutputTokens } = await import('../../../../../../src/lib/execute/jobs/ee/agent/run-agent-turn')
-
-const catalogOf = (byId: Record<string, number>) => ({
- lookup: ({ modelId }: { modelId: string }) => {
- const stripped = modelId.replace(/^(us|eu|apac|global)\./, '')
- const maxOutputTokens = byId[stripped]
- return maxOutputTokens === undefined ? undefined : { maxOutputTokens }
- },
-})
-
-describe('affordableOutputTokens', () => {
- const SMART_TIER_THINKING = 10_000
-
- it('reads through a region-prefixed Bedrock id, which is how the picker spells them', async () => {
- mockLoad.mockResolvedValue(catalogOf({ 'amazon.nova-pro-v1:0': 10_000 }))
-
- await expect(affordableOutputTokens({
- provider: AIProviderName.BEDROCK,
- modelIds: ['eu.amazon.nova-pro-v1:0'],
- thinkingBudget: SMART_TIER_THINKING,
- })).resolves.toBe(10_000)
- })
-
- it('keeps the turn inside the smaller ceiling when the fast round runs a different model', async () => {
- mockLoad.mockResolvedValue(catalogOf({ 'amazon.nova-pro-v1:0': 10_000, 'amazon.nova-micro-v1:0': 5_000 }))
-
- await expect(affordableOutputTokens({
- provider: AIProviderName.BEDROCK,
- modelIds: ['eu.amazon.nova-pro-v1:0', 'eu.amazon.nova-micro-v1:0'],
- thinkingBudget: SMART_TIER_THINKING,
- })).resolves.toBe(5_000)
- })
-
- it('leaves the budget alone for a model the catalog has never heard of', async () => {
- mockLoad.mockResolvedValue(catalogOf({}))
-
- await expect(affordableOutputTokens({
- provider: AIProviderName.BEDROCK,
- modelIds: ['eu.some.brand-new-model-v9:0'],
- thinkingBudget: SMART_TIER_THINKING,
- })).resolves.toBe(42_000)
- })
-
- it('does not shrink a turn because the catalog was unreachable', async () => {
- mockLoad.mockResolvedValue({ lookup: () => undefined })
-
- await expect(affordableOutputTokens({
- provider: AIProviderName.BEDROCK,
- modelIds: ['eu.amazon.nova-pro-v1:0', undefined],
- thinkingBudget: SMART_TIER_THINKING,
- })).resolves.toBe(42_000)
- })
-})
diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-guards.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-guards.test.ts
index 232055f5d34..cfced625699 100644
--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-guards.test.ts
+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-guards.test.ts
@@ -3,7 +3,7 @@ import { AgentRunSource, AI_PROVIDER_ENTITY_TYPES } from '@activepieces/shared'
import { APICallError, RetryError } from 'ai'
import { describe, expect, it } from 'vitest'
-import { clampOutputTokens, classifyAgentRunError, firstStepUsesFastModel, isTransientFailureText, jsonInputFrom, looksEmptyResultText } from '../../../../../../src/lib/execute/jobs/ee/agent/run-agent-turn'
+import { classifyAgentRunError, firstStepUsesFastModel, isTransientFailureText, jsonInputFrom, looksEmptyResultText } from '../../../../../../src/lib/execute/jobs/ee/agent/run-agent-turn'
function apiError({ statusCode, message, responseBody }: { statusCode: number, message: string, responseBody?: string }): APICallError {
return new APICallError({ message, url: 'https://provider.test/v1/chat', requestBodyValues: {}, statusCode, responseBody })
@@ -208,23 +208,3 @@ describe('jsonInputFrom', () => {
}
})
})
-
-describe('clampOutputTokens', () => {
- const SMART_TIER_THINKING = 10_000
-
- it('asks for the full budget when no model declares a ceiling', () => {
- expect(clampOutputTokens({ thinkingBudget: SMART_TIER_THINKING, ceilings: [undefined, undefined] })).toBe(42_000)
- })
-
- it('never asks a model for more than it accepts', () => {
- expect(clampOutputTokens({ thinkingBudget: SMART_TIER_THINKING, ceilings: [10_000] })).toBe(10_000)
- })
-
- it('respects the smaller ceiling when the turn spans two models', () => {
- expect(clampOutputTokens({ thinkingBudget: SMART_TIER_THINKING, ceilings: [64_000, 8_192] })).toBe(8_192)
- })
-
- it('leaves a generous ceiling alone rather than raising the ask to meet it', () => {
- expect(clampOutputTokens({ thinkingBudget: SMART_TIER_THINKING, ceilings: [200_000] })).toBe(42_000)
- })
-})