Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(chat): stop a message when the platform runs out of credits (#15869)

Hazem Adelc802fdefc0a0
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.md
index 6058ac3011d..8c42354255f 100644
--- a/brain/knowledge/flows-execution/chat.md
+++ b/brain/knowledge/flows-execution/chat.md
@@ -39,6 +39,8 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-
- **The context budget ignores tool schemas and reserved `max_tokens`.** Anthropic/OpenRouter count both against the 200k window; `agent-compaction.ts` budgets neither. It trims history to `COMPACTION_THRESHOLD (0.7) × 200_000 = 140_000` and its fit check looks only at message chars, while `run-agent-turn.ts` sets `maxOutputTokens: tier.thinkingBudget + 32_000` → 52k reserved on premium, plus ~12k of tool schemas (62 tools, 41 via MCP). 140k + 12k + 52k = 204k, so a conversation that compacts to just under the threshold still 400s with "maximum context length is 200000 tokens" — and it gets retried ~6× (`streamText maxRetries: 3` × `MAX_STREAM_RETRIES`), burning ~20s per turn. `maxOutputTokens` is set at the `streamText` call level, so the full thinking budget stays reserved even on step one where `prepareStep` disables thinking and swaps in haiku-4.5 (real case: 148_628 text + 11_872 tool + 52_000 output = 212_500; dropping the unused 20k reservation alone would have fit). `ESTIMATED_TOKENS_PER_MESSAGE = 200` also sizes the recent window by message *count*, so a 12-message history holding ~235k tokens of uploaded documents summarized only 1 message. When budgeting, subtract the reserved output window and tool-schema size from `getMaxContextTokens`, and don't reserve `thinkingBudget` on a thinking-disabled step.
- **A write tool in `BUILD_ONLY_TOOL_NAMES` is only reachable if something flips the phase for it.** The denylist is the consistent home for anything that writes (`ap_create_flow`, `ap_create_table`, `ap_lock_and_publish` are all in it), but the only route out of `discovery` is `ap_set_phase`, whose description tells the model to switch when it starts *building an automation*. A tool for a subject with no build guide and no sibling build-only call, the agent-building tools being the case that found this, becomes invisible in any conversation that never builds a flow: `activeToolsForPhase` filters it out and the prompt names no tool, so the model cannot discover that it exists. Classifying by "does it write" is not enough; check what would actually flip the phase in a conversation about *that* subject. The four agent *write* tools (`ap_create_agent`, `ap_update_agent`, `ap_add_agent_tool`, `ap_remove_agent_tool`) were added to the set and then reverted for exactly this, with a test pinning the choice; `ap_list_agents` is a read and was never in it, so the group is five tools and only four were ever candidates.
- **Capability notes are built where `discoveryOnly` is not known, so a prompt can promise tools the worker has stripped.** `getAgentConfig` composes the system prompt in the api, while `discoveryOnly` rides on the job data; before Aug 2026 it never crossed that boundary. Meanwhile the worker strips image tools, email tools and the agent tools on such a run, so the notes claimed all three. Two of the three had been wrong since long before anyone noticed, because each note computed its own availability term. The flag now travels with the config request and the three notes read one shared `actingRun = !dryRun && !discoveryOnly`. Whenever a tool group is gated on a run mode in the worker, the note that advertises it has to be gated on the same term, in one place.
+- **Tool output can't be trusted to waive a charge.** A user's own MCP server or `ap_run_code` controls its output's top-level keys, and they reach the persisted part unchanged. Chat billing only honours `billedAtCost` from `ap_web_search` and `ap_generate_image`; a new built-in tool that bills at cost must be added to `TOOLS_THAT_BILL_AT_COST` in `chatBilling`.
+- **Credits are checked after every step, not just before the turn.** `runAgentTurn` asks the API (`agentHasCredits`) after each tool step, sending what the turn has used so far (`chatBilling.creditsForTurn`, the same rule end-of-turn billing uses), and stops once the remaining balance no longer covers it. Flat credits are only charged when the turn ends, so without that pending count the balance never moves mid-turn. The check fails open if the RPC errors.
- **Chat files belong to a conversation, not just a project.** Conversations are private to their user, so a project check is not enough. `saveAgentFile` and `persistAgentAttachments` stamp `conversationId` into the file metadata, and `readAgentFile` refuses a file whose stamp doesn't match. Files saved before the stamp existed can't be read back (image editing can't use them). Any new tool that reads chat files by a model-supplied ID must go through `readAgentFile`.
- Local dev needs `AP_DB_TYPE=POSTGRES` + Redis; refuses PGLite. **Prefer `AP_EDITION=cloud` over `ee` for chat work.** Cloud boots locally against plain Postgres and Redis with no Autumn, Stripe or license-key config (verified Aug 2026: API healthy, migrations applied, zero billing or license errors), and on Cloud `chatVisibility` returns `planChatEnabled || cloudRolloutOpen || userHasChatted`, so chat is simply **on** while the rollout cap is unfilled. On `ee` it is gated behind `plan.chatEnabled` and you have to get a plan onto the platform first. Note SMTP is usually unset locally, which makes the auth card open on the password form rather than the email-code step. Debug a run with `npm run chat:logs -- <conversationId> [runId]` (needs `LOG_FILE=true`/`AP_LOG_FILE=true` set when the turn ran — otherwise `.evlog/logs` is empty).
- **Chat was renamed to agent in code and DB, but only the storage half.** As of release 0.87.1 (`1823000000000-AddRenamedChatTableCompatViews`) `chat_conversation` → `agent_conversation` and `user_chat_memory` → `user_memory`, the server module moved `ee/chat/` → `ee/agent/` (entry point `agentModule`), the worker dir moved `jobs/ee/chat/` → `jobs/ee/agent/`, shared types moved `core/shared/.../ee/chat/` → `.../ee/agent/`, and `server/utils/src/chat-ai-utils.ts` → `agent-ai-utils.ts`. **The rename is not uniform, and the split is the thing to learn**: files describing chat as a *user-facing surface* deliberately kept their `chat-` names inside `ee/agent/` — `chat-visibility.ts`, `chat-rollout-service.ts`, `chat-rollout-user-entity.ts`, `chat-analytics-sync.ts`, `chat-tool-billing.ts`, `chat-usage-tracker.ts`, `chat-plan-grant.ts`. So a new chat-surface concern keeps the `chat-` prefix; a new stored entity takes `agent_`. The migration also leaves `CREATE OR REPLACE VIEW` compat views at both old table names, so raw SQL against `chat_conversation` still reads fine and will NOT tell you the rename happened — grep the entity, not the database.
diff --git a/bun.lock b/bun.lock
index 4e12133c35b..fbed33b8620 100644
--- a/bun.lock
+++ b/bun.lock
@@ -120,7 +120,7 @@
},
"packages/core/execution": {
"name": "@activepieces/core-execution",
- "version": "0.23.2",
+ "version": "0.23.3",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -165,7 +165,7 @@
},
"packages/core/shared": {
"name": "@activepieces/shared",
- "version": "0.183.0",
+ "version": "0.184.0",
"dependencies": {
"@activepieces/core-execution": "workspace:*",
"@activepieces/core-formula": "workspace:*",
diff --git a/packages/core/execution/package.json b/packages/core/execution/package.json
index 1aa2bbaad48..c926a7f0a8d 100644
--- a/packages/core/execution/package.json
+++ b/packages/core/execution/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-execution",
- "version": "0.23.2",
+ "version": "0.23.3",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/execution/src/lib/workers/worker-contract.ts b/packages/core/execution/src/lib/workers/worker-contract.ts
index a3fa2263716..2d7d9179764 100644
--- a/packages/core/execution/src/lib/workers/worker-contract.ts
+++ b/packages/core/execution/src/lib/workers/worker-contract.ts
@@ -93,6 +93,7 @@ export type WorkerToApiContract = {
readAgentFile(input: ReadAgentFileRequest): Promise<ReadFlowStepFileResponse>
updateAgentProgress(input: UpdateAgentProgressRequest): Promise<void>
heartbeatAgentConversation(input: HeartbeatAgentConversationRequest): Promise<void>
+ agentCreditsLeft(input: AgentCreditsLeftRequest): Promise<number | null>
updateProjectContext(input: UpdateProjectContextRequest): Promise<void>
executeAgentTool(input: ExecuteAgentToolRequest): Promise<ExecuteAgentToolResponse>
resumeFlowStep(input: ResumeFlowStepRequest): Promise<void>
@@ -215,6 +216,12 @@ export type HeartbeatAgentConversationRequest = {
runId?: string
}
+export type AgentCreditsLeftRequest = {
+ platformId: string
+ conversationId: string
+ pendingCredits: number
+}
+
export type UpdateProjectContextRequest = {
conversationId: string
runId?: string
diff --git a/packages/core/shared/package.json b/packages/core/shared/package.json
index 21fcd1ea9dd..b07961c3173 100644
--- a/packages/core/shared/package.json
+++ b/packages/core/shared/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/shared",
- "version": "0.183.0",
+ "version": "0.184.0",
"type": "commonjs",
"sideEffects": false,
"main": "./dist/src/index.js",
diff --git a/packages/core/shared/src/lib/ee/agent/chat-billing.ts b/packages/core/shared/src/lib/ee/agent/chat-billing.ts
new file mode 100644
index 00000000000..222764525cf
--- /dev/null
+++ b/packages/core/shared/src/lib/ee/agent/chat-billing.ts
@@ -0,0 +1,48 @@
+import { AIProviderName, isObject } from '@activepieces/core-utils'
+
+function isPaidTool(toolName: string): boolean {
+ return toolName.startsWith('mcp__') || FLAT_BILLED_TOOL_NAMES.has(toolName)
+}
+
+function isFlatBilledToolCall({ toolName, output }: ChatToolCall): boolean {
+ const paidAtCost = TOOLS_THAT_BILL_AT_COST.has(toolName) && isObject(output) && output['billedAtCost'] === true
+ return isPaidTool(toolName) && !paidAtCost
+}
+
+function creditsForTurn({ provider, toolCalls }: { provider: string | null, toolCalls: ChatToolCall[] }): TurnCredits {
+ const messageCredits = provider === AIProviderName.ACTIVEPIECES ? 0 : CHAT_CREDITS_PER_OWN_KEY_TURN
+ const billedToolCalls = toolCalls.filter(isFlatBilledToolCall).length
+ return { messageCredits, billedToolCalls, total: messageCredits + billedToolCalls * CHAT_CREDITS_PER_TOOL_CALL }
+}
+
+const FLAT_BILLED_TOOL_NAMES = new Set<string>([
+ 'ap_web_search',
+ 'ap_scrape_url',
+ 'ap_generate_image',
+ 'ap_execute_action',
+ 'ap_explore_data',
+ 'ap_run_code',
+])
+
+const TOOLS_THAT_BILL_AT_COST = new Set<string>(['ap_web_search', 'ap_generate_image'])
+
+const CHAT_CREDITS_PER_OWN_KEY_TURN = 1
+
+export const CHAT_CREDITS_PER_TOOL_CALL = 1
+
+export const chatBilling = {
+ isPaidTool,
+ isFlatBilledToolCall,
+ creditsForTurn,
+}
+
+export type ChatToolCall = {
+ toolName: string
+ output: unknown
+}
+
+export type TurnCredits = {
+ messageCredits: number
+ billedToolCalls: number
+ total: number
+}
diff --git a/packages/core/shared/src/lib/ee/agent/index.ts b/packages/core/shared/src/lib/ee/agent/index.ts
index 250dca8c863..0b90c093d0f 100644
--- a/packages/core/shared/src/lib/ee/agent/index.ts
+++ b/packages/core/shared/src/lib/ee/agent/index.ts
@@ -255,8 +255,6 @@ export const UpdateAgentMemoryRequest = z.object({
})
export type UpdateAgentMemoryRequest = z.infer<typeof UpdateAgentMemoryRequest>
-export const CHAT_CREDITS_PER_TOOL_CALL = 1
-
export const ImportAgentMemoryRequest = z.object({
text: z.string(),
})
@@ -378,6 +376,7 @@ export { CHAT_ALLOWED_MIME_TYPES }
export * from './agent'
export { agentToolClassification } from './tool-classification'
+export { CHAT_CREDITS_PER_TOOL_CALL, chatBilling, type ChatToolCall, type TurnCredits } from './chat-billing'
export { AGENT_SELF_EDIT_TOOLS, AGENT_SURFACE_TOOLS, agentToolPhases, type AgentPhase } from './tool-phases'
export { chatVisibility, type ResolveChatEnabledParams } from './chat-visibility'
export * from './chat-personalization'
diff --git a/packages/server/api/src/app/ee/agent/chat-tool-billing.ts b/packages/server/api/src/app/ee/agent/chat-tool-billing.ts
index b2f097d9c19..9cfe562c477 100644
--- a/packages/server/api/src/app/ee/agent/chat-tool-billing.ts
+++ b/packages/server/api/src/app/ee/agent/chat-tool-billing.ts
@@ -1,5 +1,5 @@
-import { AIProviderName, isNil, isObject, spreadIfDefined } from '@activepieces/core-utils'
-import { AgentConversation, CHAT_CREDITS_PER_TOOL_CALL, isAppSumoCreditedPlan, PersistedAgentMessage, PersistedAgentPartType, PersistedAgentRole, PersistedToolCallStatus } from '@activepieces/shared'
+import { isNil, spreadIfDefined } from '@activepieces/core-utils'
+import { AgentConversation, chatBilling, ChatToolCall, isAppSumoCreditedPlan, PersistedAgentMessage, PersistedAgentPartType, PersistedAgentRole, PersistedToolCallStatus } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { LicenseKeyPostHogEvents } from '../../helper/telemetry.utils'
import { trackBillingAndSendTelemetry } from '../../platform/billing-and-telemetry'
@@ -8,33 +8,22 @@ import { platformPlanService } from '../platform/platform-plan/platform-plan.ser
import { agentHelpers } from './agent-helpers'
import { agentHistory } from './history/agent-history'
-const BILLABLE_EXTERNAL_TOOL_NAMES = new Set<string>([
- 'ap_web_search',
- 'ap_scrape_url',
- 'ap_generate_image',
- 'ap_execute_action',
- 'ap_explore_data',
- 'ap_run_code',
-])
-
-function isBillableChatToolCall(toolName: string): boolean {
- return toolName.startsWith('mcp__') || BILLABLE_EXTERNAL_TOOL_NAMES.has(toolName)
+function latestTurnToolCalls({ messages }: { messages: PersistedAgentMessage[] }): ChatToolCall[] {
+ const lastUserIndex = messages.map((message) => message.role).lastIndexOf(PersistedAgentRole.USER)
+ const turn = lastUserIndex === -1 ? messages : messages.slice(lastUserIndex + 1)
+ return turn.flatMap((message) => message.parts.flatMap((part) =>
+ part.type === PersistedAgentPartType.TOOL_CALL && part.status === PersistedToolCallStatus.COMPLETED
+ ? [{ toolName: part.toolName, output: part.output }]
+ : [],
+ ))
}
function countBillableToolCallsInLatestTurn({ messages }: { messages: PersistedAgentMessage[] }): number {
- const lastUserIndex = messages.map((message) => message.role).lastIndexOf(PersistedAgentRole.USER)
- const turn = lastUserIndex === -1 ? messages : messages.slice(lastUserIndex + 1)
- return turn.reduce((sum, message) => sum + message.parts.filter((part) =>
- part.type === PersistedAgentPartType.TOOL_CALL
- && part.status === PersistedToolCallStatus.COMPLETED
- && isBillableChatToolCall(part.toolName)
- && !(isObject(part.output) && part.output['billedAtCost'] === true),
- ).length, 0)
+ return latestTurnToolCalls({ messages }).filter(chatBilling.isFlatBilledToolCall).length
}
async function chargeForLatestTurn({ conversation, runId, log }: ChargeForLatestTurnParams): Promise<void> {
const messages = agentHistory.resolveMessages({ conversation, log })
- const billableToolCalls = countBillableToolCallsInLatestTurn({ messages })
const turnIndex = messages.filter((message) => message.role === PersistedAgentRole.USER).length
const idempotencyScope = runId ?? turnIndex
const provider = await agentHelpers.resolveChatProviderName({
@@ -47,11 +36,10 @@ async function chargeForLatestTurn({ conversation, runId, log }: ChargeForLatest
const tier = agentHelpers.resolveTier({ tierId: conversation.modelName ?? null, surface })
const platformPlan = await platformPlanService(log).getOrCreateForPlatform(conversation.platformId)
- const turnCredits = provider === AIProviderName.ACTIVEPIECES ? 0 : CREDITS_PER_OWN_KEY_TURN
- const creditValue = turnCredits + billableToolCalls * CHAT_CREDITS_PER_TOOL_CALL
- const charge = creditValue === 0 ? undefined : {
+ const { messageCredits, billedToolCalls, total } = chatBilling.creditsForTurn({ provider, toolCalls: latestTurnToolCalls({ messages }) })
+ const charge = total === 0 ? undefined : {
platformId: conversation.platformId,
- value: creditValue,
+ value: total,
source: CreditUsageSource.CHAT,
idempotencyKey: `${conversation.id}:chatTurn:${idempotencyScope}`,
properties: {
@@ -60,8 +48,8 @@ async function chargeForLatestTurn({ conversation, runId, log }: ChargeForLatest
userId: conversation.userId,
conversationId: conversation.id,
turnIndex,
- messages: turnCredits,
- toolCalls: billableToolCalls,
+ messages: messageCredits,
+ toolCalls: billedToolCalls,
provider,
model,
tier: tier.id,
@@ -89,20 +77,18 @@ async function chargeForLatestTurn({ conversation, runId, log }: ChargeForLatest
properties: {
provider,
model,
- toolsUsed: billableToolCalls,
+ toolsUsed: billedToolCalls,
},
},
})
}
export const chatToolBilling = {
- isBillableChatToolCall,
countBillableToolCallsInLatestTurn,
chargeForLatestTurn,
}
const PROJECTLESS_CHAT = 'chat'
-const CREDITS_PER_OWN_KEY_TURN = 1
type ChargeForLatestTurnParams = {
conversation: AgentConversation
diff --git a/packages/server/api/src/app/ee/agent/personalization/personalization-research.ts b/packages/server/api/src/app/ee/agent/personalization/personalization-research.ts
index bc69e331239..a4d0d425f76 100644
--- a/packages/server/api/src/app/ee/agent/personalization/personalization-research.ts
+++ b/packages/server/api/src/app/ee/agent/personalization/personalization-research.ts
@@ -1,4 +1,4 @@
-import { ActivepiecesError, apId, ErrorCode, isNil, tryCatch } from '@activepieces/core-utils'
+import { apId, isNil, tryCatch } from '@activepieces/core-utils'
import {
ApEdition,
ChatPersonalization,
@@ -19,7 +19,7 @@ import { aiProviderService, ProviderScope } from '../../../ai/ai-provider-servic
import { redisConnections } from '../../../database/redis-connections'
import { system } from '../../../helper/system/system'
import { AppSystemProp } from '../../../helper/system/system-props'
-import { assertCreditsAndAppSumoNotExceeded } from '../../../platform/billing-provider'
+import { hasCreditsLeft } from '../../../platform/billing-provider'
import { jobQueue, JobType } from '../../../workers/job-queue/job-queue'
import { agentHelpers } from '../agent-helpers'
import { findRow, IN_FLIGHT_STATUSES, personalizationRepo, ValidatedResult } from './personalization-rows'
@@ -194,16 +194,9 @@ export async function guardsAllowResearch({ platformId, log }: { platformId: str
log.warn({ platform: { id: platformId } }, '[chatPersonalization] No chat AI provider configured, skipping research')
return false
}
- const credits = await tryCatch(() => assertCreditsAndAppSumoNotExceeded({ platformId, log }))
- if (credits.error) {
- const exhausted = credits.error instanceof ActivepiecesError && credits.error.error.code === ErrorCode.QUOTA_EXCEEDED
- if (!exhausted) {
- log.warn({ platform: { id: platformId }, error: credits.error }, '[chatPersonalization] Credits check failed, allowing research')
- }
- else {
- log.warn({ platform: { id: platformId } }, '[chatPersonalization] Credits exhausted, skipping research')
- return false
- }
+ if (!await hasCreditsLeft({ platformId, log })) {
+ log.warn({ platform: { id: platformId } }, '[chatPersonalization] Credits exhausted, skipping research')
+ return false
}
const { allowed, count } = await agentHelpers.incrementAndCheckLimit({
key: `chat-personalization-runs:${platformId}`,
diff --git a/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts
index 6b06592780f..2df77a17ca3 100644
--- a/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts
@@ -1,5 +1,5 @@
import { ActivepiecesError, ErrorCode, isNil, sanitizeObjectForPostgresql, spreadIfDefined } from '@activepieces/core-utils'
-import { AgentConversationStatus, AgentRunSource, FileCompression, FileType, HeartbeatAgentConversationRequest, ReadAgentFileRequest, ReadFlowStepFileResponse, SaveAgentFileRequest, SaveAgentFileResponse, SaveAgentMessagesRequest, UpdateAgentProgressRequest, UpdateProjectContextRequest } from '@activepieces/shared'
+import { AgentConversationStatus, AgentCreditsLeftRequest, AgentRunSource, FileCompression, FileType, HeartbeatAgentConversationRequest, ReadAgentFileRequest, ReadFlowStepFileResponse, SaveAgentFileRequest, SaveAgentFileResponse, SaveAgentMessagesRequest, UpdateAgentProgressRequest, UpdateProjectContextRequest } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { agentHelpers } from '.././agent-helpers'
import { chatAnalyticsTelemetry } from '.././chat-analytics-sync'
@@ -7,6 +7,7 @@ import { chatToolBilling } from '.././chat-tool-billing'
import { fileService } from '../../../file/file.service'
import { filesService } from '../../../file/files-service'
import { rejectedPromiseHandler } from '../../../helper/promise-handler'
+import { creditsLeftAfter } from '../../../platform/billing-provider'
import { updateConversationForRun } from './rpc-shared'
@@ -114,6 +115,10 @@ export const conversationRpc = (log: FastifyBaseLogger) => ({
log.debug({ conversation: { id: input.conversationId }, uiMessageCount: input.uiMessages.length, messageCount: input.messages?.length }, '[agentRpc#updateAgentProgress] Progress persisted')
},
+ async agentCreditsLeft(input: AgentCreditsLeftRequest): Promise<number | null> {
+ return creditsLeftAfter({ platformId: input.platformId, pendingCredits: input.pendingCredits, log })
+ },
+
async heartbeatAgentConversation(input: HeartbeatAgentConversationRequest): Promise<void> {
// Liveness signal: bump `updated` only while STREAMING so stale-recovery in
// getConversationOrThrow doesn't flip a genuinely-working long turn to IDLE. Gate on the
diff --git a/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-billing.ts b/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-billing.ts
index 39e54d9fcf6..2988c971153 100644
--- a/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-billing.ts
+++ b/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-billing.ts
@@ -355,6 +355,7 @@ export function computeCreditState({ balance, enforced }: ComputeCreditStatePara
const exhausted = !isNil(balance) && isCreditsExhausted(balance)
return {
blocked: enforced && exhausted,
+ metered: enforced && !isNil(balance) && !balance.unlimited,
usage: balance?.usage ?? 0,
limit: balance?.granted ?? 0,
remaining: balance?.remaining ?? 0,
diff --git a/packages/server/api/src/app/platform/billing-provider.ts b/packages/server/api/src/app/platform/billing-provider.ts
index c384ccdeb61..b6a97b9b126 100644
--- a/packages/server/api/src/app/platform/billing-provider.ts
+++ b/packages/server/api/src/app/platform/billing-provider.ts
@@ -1,4 +1,4 @@
-import { ActivepiecesError, ErrorCode, PlatformUsageMetric } from '@activepieces/core-utils'
+import { ActivepiecesError, ErrorCode, isNil, PlatformUsageMetric, tryCatch } from '@activepieces/core-utils'
import { apDayjs } from '@activepieces/server-utils'
import { ApEdition, AppSumoCreditsBillableFeature, CancellationReason, ConsumableFeatureId, ConsumableProductAutoTopupParams, CreditsBillableFeature, FlowRun, PurchasablePlan, RunEnvironment, SeatsBillableFeature, UnconsumableFeatureId } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
@@ -59,8 +59,8 @@ export const billingProvider = hooksFactory.create<BillingProvider>(() => ({
},
getCreditsAndAppSumoState: async () => {
return {
- credits: { blocked: false, usage: 0, limit: 0, remaining: 0, unlimited: false },
- appSumo: { blocked: false, usage: 0, limit: 0, remaining: 0, unlimited: false },
+ credits: { blocked: false, metered: false, usage: 0, limit: 0, remaining: 0, unlimited: false },
+ appSumo: { blocked: false, metered: false, usage: 0, limit: 0, remaining: 0, unlimited: false },
}
},
getConsumablesUsage: async () => {
@@ -87,6 +87,21 @@ export async function assertCreditsAndAppSumoNotExceeded({ platformId, log }: {
}
}
+export async function creditsLeftAfter({ platformId, pendingCredits, log }: { platformId: string, pendingCredits: number, log: FastifyBaseLogger }): Promise<number | null> {
+ const { data: state, error } = await tryCatch(() => billingProvider.get(log).getCreditsAndAppSumoState(platformId))
+ if (isNil(state)) {
+ log.warn({ platform: { id: platformId }, error }, 'Credits check failed, allowing the request')
+ return null
+ }
+ const left = [state.credits, state.appSumo].filter((gate) => gate.metered).map((gate) => gate.remaining - pendingCredits)
+ return left.length === 0 ? null : Math.min(...left)
+}
+
+export async function hasCreditsLeft({ platformId, log }: { platformId: string, log: FastifyBaseLogger }): Promise<boolean> {
+ const left = await creditsLeftAfter({ platformId, pendingCredits: 0, log })
+ return isNil(left) || left > 0
+}
+
export async function shouldBlockRunOnCredits({ platformId, environment, log }: RunCreditsGateParams): Promise<boolean> {
if (system.getEdition() === ApEdition.ENTERPRISE) {
return false
@@ -267,6 +282,7 @@ export type CreditUsage = {
export type CreditsGateState = {
blocked: boolean
+ metered: boolean
usage: number
limit: number
remaining: number
diff --git a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts b/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
index dcd7dbff27c..bf6ba67b42f 100644
--- a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
+++ b/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
@@ -371,6 +371,9 @@ export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAs
async heartbeatAgentConversation(input) {
return agentRpcHandlers(agentRpcLog(log, input)).heartbeatAgentConversation(input)
},
+ async agentCreditsLeft(input) {
+ return agentRpcHandlers(agentRpcLog(log, input)).agentCreditsLeft(input)
+ },
async updateProjectContext(input) {
return agentRpcHandlers(agentRpcLog(log, input)).updateProjectContext(input)
diff --git a/packages/server/api/test/unit/app/ee/agent/chat-tool-billing.test.ts b/packages/server/api/test/unit/app/ee/agent/chat-tool-billing.test.ts
index 8f1bc8f65e3..3f8d4c68805 100644
--- a/packages/server/api/test/unit/app/ee/agent/chat-tool-billing.test.ts
+++ b/packages/server/api/test/unit/app/ee/agent/chat-tool-billing.test.ts
@@ -1,4 +1,4 @@
-import { PersistedAgentMessage, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, PersistedToolCallStatus } from '@activepieces/shared'
+import { chatBilling, PersistedAgentMessage, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, PersistedToolCallStatus } from '@activepieces/shared'
import { describe, expect, it } from 'vitest'
import { chatToolBilling } from '../../../../../src/app/ee/agent/chat-tool-billing'
import { ALL_CONTROLLABLE_TOOL_NAMES, LOCKED_TOOL_NAMES, PLATFORM_LEVEL_TOOL_NAMES } from '../../../../../src/app/mcp/tools'
@@ -9,30 +9,30 @@ const AP_NATIVE_TOOL_NAMES = [
...ALL_CONTROLLABLE_TOOL_NAMES,
]
-describe('chatToolBilling.isBillableChatToolCall', () => {
+describe('chatBilling.isFlatBilledToolCall', () => {
it('never bills an AP-native MCP tool (they are free or already billed via the run)', () => {
- const billable = AP_NATIVE_TOOL_NAMES.filter((name) => chatToolBilling.isBillableChatToolCall(name))
+ const billable = AP_NATIVE_TOOL_NAMES.filter((name) => chatBilling.isFlatBilledToolCall({ toolName: name, output: undefined }))
expect(billable, `These AP-native tools must not be billed: ${billable.join(', ')}`).toEqual([])
})
it('bills piece integration calls (mcp__<connectorUuid>__action)', () => {
- expect(chatToolBilling.isBillableChatToolCall('mcp__attio__list_records')).toBe(true)
+ expect(chatBilling.isFlatBilledToolCall({ toolName: 'mcp__attio__list_records', output: undefined })).toBe(true)
})
it('bills the paid external tools', () => {
- expect(chatToolBilling.isBillableChatToolCall('ap_web_search')).toBe(true)
- expect(chatToolBilling.isBillableChatToolCall('ap_scrape_url')).toBe(true)
- expect(chatToolBilling.isBillableChatToolCall('ap_generate_image')).toBe(true)
+ expect(chatBilling.isFlatBilledToolCall({ toolName: 'ap_web_search', output: undefined })).toBe(true)
+ expect(chatBilling.isFlatBilledToolCall({ toolName: 'ap_scrape_url', output: undefined })).toBe(true)
+ expect(chatBilling.isFlatBilledToolCall({ toolName: 'ap_generate_image', output: undefined })).toBe(true)
})
it('bills chat-initiated ad-hoc executions (not separately metered)', () => {
- expect(chatToolBilling.isBillableChatToolCall('ap_execute_action')).toBe(true)
- expect(chatToolBilling.isBillableChatToolCall('ap_explore_data')).toBe(true)
- expect(chatToolBilling.isBillableChatToolCall('ap_run_code')).toBe(true)
+ expect(chatBilling.isFlatBilledToolCall({ toolName: 'ap_execute_action', output: undefined })).toBe(true)
+ expect(chatBilling.isFlatBilledToolCall({ toolName: 'ap_explore_data', output: undefined })).toBe(true)
+ expect(chatBilling.isFlatBilledToolCall({ toolName: 'ap_run_code', output: undefined })).toBe(true)
})
it('does not bill an unknown tool (fail-safe default)', () => {
- expect(chatToolBilling.isBillableChatToolCall('ap_some_tool_added_later')).toBe(false)
+ expect(chatBilling.isFlatBilledToolCall({ toolName: 'ap_some_tool_added_later', output: undefined })).toBe(false)
})
})
@@ -98,6 +98,17 @@ describe('chatToolBilling.countBillableToolCallsInLatestTurn', () => {
expect(chatToolBilling.countBillableToolCallsInLatestTurn({ messages })).toBe(1)
})
+ it('bills an external tool that claims it was already billed', () => {
+ const messages = [
+ user('do it'),
+ assistant([
+ { ...toolCallPart({ toolName: 'mcp__evil__lookup', status: PersistedToolCallStatus.COMPLETED }), output: { billedAtCost: true } },
+ { ...toolCallPart({ toolName: 'ap_run_code', status: PersistedToolCallStatus.COMPLETED }), output: { billedAtCost: true } },
+ ]),
+ ]
+ expect(chatToolBilling.countBillableToolCallsInLatestTurn({ messages })).toBe(2)
+ })
+
it('ignores non-billable tools regardless of status', () => {
const messages = [
user('do it'),
diff --git a/packages/server/api/test/unit/app/ee/agent/conversation-rpc-credits.test.ts b/packages/server/api/test/unit/app/ee/agent/conversation-rpc-credits.test.ts
new file mode 100644
index 00000000000..f089da20d26
--- /dev/null
+++ b/packages/server/api/test/unit/app/ee/agent/conversation-rpc-credits.test.ts
@@ -0,0 +1,46 @@
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+import { conversationRpc } from '../../../../../src/app/ee/agent/rpc/conversation-rpc'
+import { billingProvider } from '../../../../../src/app/platform/billing-provider'
+
+const log = { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() } as never
+const noopProvider = billingProvider.get(log)
+const mockCreditsState = vi.fn()
+
+describe('agentCreditsLeft', () => {
+ beforeEach(() => {
+ mockCreditsState.mockReset()
+ billingProvider.set(() => ({ ...noopProvider, getCreditsAndAppSumoState: mockCreditsState }))
+ })
+
+ it('reports the credits left after what the turn has already used', async () => {
+ mockCreditsState.mockResolvedValue(creditsState({ remaining: 3 }))
+
+ expect(await check({ pendingCredits: 2 })).toBe(1)
+ expect(await check({ pendingCredits: 3 })).toBe(0)
+ expect(await check({ pendingCredits: 4 })).toBe(-1)
+ expect(mockCreditsState).toHaveBeenCalledWith('platform-1')
+ })
+
+ it('reports no limit on a platform that is not metered', async () => {
+ mockCreditsState.mockResolvedValue(creditsState({ remaining: 0, metered: false }))
+
+ expect(await check({ pendingCredits: 5 })).toBeNull()
+ })
+
+ it('reports no limit when the check itself fails, so the turn is not stopped by an outage', async () => {
+ mockCreditsState.mockRejectedValue(new Error('redis down'))
+
+ expect(await check({ pendingCredits: 1 })).toBeNull()
+ })
+})
+
+function check({ pendingCredits }: { pendingCredits: number }): Promise<number | null> {
+ return conversationRpc(log).agentCreditsLeft({ platformId: 'platform-1', conversationId: 'conv-1', pendingCredits })
+}
+
+function creditsState({ remaining, metered = true }: { remaining: number, metered?: boolean }): Awaited<ReturnType<typeof noopProvider.getCreditsAndAppSumoState>> {
+ return {
+ credits: { blocked: metered && remaining <= 0, metered, usage: 100, limit: 100, remaining, unlimited: false },
+ appSumo: { blocked: false, metered: false, usage: 0, limit: 0, remaining: 0, unlimited: false },
+ }
+}
diff --git a/packages/server/api/test/unit/app/ee/platform-plan/credits-gate.test.ts b/packages/server/api/test/unit/app/ee/platform-plan/credits-gate.test.ts
index 0e94de805fc..a567d5a4596 100644
--- a/packages/server/api/test/unit/app/ee/platform-plan/credits-gate.test.ts
+++ b/packages/server/api/test/unit/app/ee/platform-plan/credits-gate.test.ts
@@ -1,5 +1,6 @@
import { FastifyBaseLogger } from 'fastify'
import { beforeEach, describe, expect, it, vi } from 'vitest'
+import { autumnBillingProvider, computeCreditState } from '../../../../../src/app/ee/platform/platform-plan/billing-providers/autumn-billing'
import { CreditsBalanceCache } from '../../../../../src/app/ee/platform/platform-plan/billing-providers/autumn-utils'
let storedCredits: CreditsBalanceCache | null = null
@@ -56,7 +57,6 @@ vi.mock('../../../../../src/app/ee/platform/platform-plan/platform-plan.service'
assertSeatsNotBelowActiveUsers: vi.fn(),
}))
-import { autumnBillingProvider, computeCreditState } from '../../../../../src/app/ee/platform/platform-plan/billing-providers/autumn-billing'
function balance(overrides: Partial<CreditsBalanceCache>): CreditsBalanceCache {
return { granted: 1000, usage: 0, remaining: 1000, unlimited: false, nextResetAt: null, syncedAt: 0, ...overrides }
@@ -93,6 +93,13 @@ describe('computeCreditState — credits (enforcement-gated)', () => {
expect(computeCreditState({ balance: balance({ remaining: 0, unlimited: true }), enforced: true }).blocked).toBe(false)
})
+ it('is metered only when billing is enforced on a known, limited balance', () => {
+ expect(computeCreditState({ balance: balance({ remaining: 100 }), enforced: true }).metered).toBe(true)
+ expect(computeCreditState({ balance: balance({ remaining: 100 }), enforced: false }).metered).toBe(false)
+ expect(computeCreditState({ balance: balance({ remaining: 100, unlimited: true }), enforced: true }).metered).toBe(false)
+ expect(computeCreditState({ balance: null, enforced: true }).metered).toBe(false)
+ })
+
it('fails open when no balance is cached', () => {
expect(computeCreditState({ balance: null, enforced: true }).blocked).toBe(false)
})
diff --git a/packages/server/utils/src/agent-ai-utils.ts b/packages/server/utils/src/agent-ai-utils.ts
index 90368d5db5b..2f538ce405d 100644
--- a/packages/server/utils/src/agent-ai-utils.ts
+++ b/packages/server/utils/src/agent-ai-utils.ts
@@ -150,6 +150,7 @@ function toRecord(value: unknown): Record<string, unknown> {
type ContentPartLike = {
type: string
+ invalid?: boolean
text?: string
toolCallId?: string
toolName?: string
@@ -224,7 +225,7 @@ function buildStepParts({ content }: {
...spreadIfDefined('description', description),
input,
output: rawOutput,
- status: result ? PersistedToolCallStatus.COMPLETED : PersistedToolCallStatus.ERROR,
+ status: result && part.invalid !== true ? PersistedToolCallStatus.COMPLETED : PersistedToolCallStatus.ERROR,
})
if (toolName === 'ap_execute_action' && typeof rawOutput === 'object' && rawOutput !== null && 'batchProgress' in rawOutput) {
parts.push({
diff --git a/packages/server/utils/test/agent-ai-utils.test.ts b/packages/server/utils/test/agent-ai-utils.test.ts
index 0246af3a9d9..1fdb69e4409 100644
--- a/packages/server/utils/test/agent-ai-utils.test.ts
+++ b/packages/server/utils/test/agent-ai-utils.test.ts
@@ -1,5 +1,5 @@
import { AIProviderName } from '@activepieces/core-utils'
-import { ACTIVEPIECES_CHAT_TIERS } from '@activepieces/shared'
+import { ACTIVEPIECES_CHAT_TIERS, PersistedAgentPartType, PersistedToolCallStatus } from '@activepieces/shared'
import { ModelMessage } from 'ai'
import { describe, expect, it } from 'vitest'
import { agentAiUtils } from '../src/agent-ai-utils'
@@ -150,6 +150,29 @@ describe('collectStepMessages', () => {
})
})
+describe('buildStepParts — tool call status', () => {
+ it('bills a call that ran, even one that then threw, and never one the SDK refused to run', () => {
+ const content = [
+ { type: 'tool-call', toolCallId: 'ran', toolName: 'ap_web_search', input: { query: 'a' } },
+ { type: 'tool-result', toolCallId: 'ran', toolName: 'ap_web_search', output: { type: 'json', value: { answer: 'ok' } } },
+ { type: 'tool-call', toolCallId: 'threw', toolName: 'mcp__crm__create_record', input: { name: 'b' } },
+ { type: 'tool-error', toolCallId: 'threw', toolName: 'mcp__crm__create_record' },
+ { type: 'tool-call', toolCallId: 'refused', toolName: 'ap_web_search', input: { query: 'c' }, invalid: true },
+ { type: 'tool-error', toolCallId: 'refused', toolName: 'ap_web_search' },
+ { type: 'tool-call', toolCallId: 'unanswered', toolName: 'ap_web_search', input: { query: 'd' } },
+ ]
+
+ const statuses = agentAiUtils.buildStepParts({ content }).flatMap((part) => part.type === PersistedAgentPartType.TOOL_CALL ? [[part.toolCallId, part.status]] : [])
+
+ expect(statuses).toEqual([
+ ['ran', PersistedToolCallStatus.COMPLETED],
+ ['threw', PersistedToolCallStatus.COMPLETED],
+ ['refused', PersistedToolCallStatus.ERROR],
+ ['unanswered', PersistedToolCallStatus.ERROR],
+ ])
+ })
+})
+
describe('buildStepParts — ap_set_build_plan', () => {
it('emits a build-plan part carrying the buildId from output and the plan from input', () => {
const content = [
diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
index c6443f77f43..dabef7c7f83 100644
--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
@@ -249,6 +249,7 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
return runAgentTurn({
...spreadIfDefined('stepCeiling', data.maxSteps),
+ creditsLeft: (pendingCredits) => ctx.apiClient.agentCreditsLeft({ platformId, conversationId, pendingCredits }),
model,
fastModel: firstStepUsesFastModel({ source, dryRun, runsASavedAgent: !isNil(data.promptOverride) }) ? fastModel : undefined,
provider,
@@ -297,7 +298,7 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
},
})
- const { uiParts, accumulatedResponseMessages, streamError, truncatedAfterRetries, budgetExceeded, continuations, usage, totalInputTokens, totalOutputTokens } = turn
+ const { uiParts, accumulatedResponseMessages, streamError, truncatedAfterRetries, budgetExceeded, creditsExhausted, continuations, usage, totalInputTokens, totalOutputTokens } = turn
if (abortController.signal.aborted) {
if (streamError) {
@@ -364,7 +365,7 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
}
await retryWithBackoff({ fn: () => ctx.apiClient.saveAgentMessages(savePayload), description: 'Saving the transcript', throwOnExhausted: true, log })
- answer = stepResultFrom({ prompt: userMessage, uiParts, timestamp: new Date().toISOString(), tools: reportedTools, structuredOutput: structured.output, failure: incompleteReason({ truncatedAfterRetries, budgetExceeded }) })
+ answer = stepResultFrom({ prompt: userMessage, uiParts, timestamp: new Date().toISOString(), tools: reportedTools, structuredOutput: structured.output, failure: incompleteReason({ truncatedAfterRetries, budgetExceeded, creditsExhausted }) })
if (autoTitle) {
await sendEventWithRetry({
@@ -384,6 +385,12 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
})
}
+ if (creditsExhausted) {
+ await sendEventWithRetry({
+ event: { type: AgentEventType.ERROR, data: { message: 'You ran out of credits, so this message stopped early. Your progress is saved. Add credits, then send a new message to continue.' } },
+ })
+ }
+
await sendEventWithRetry({
event: { type: AgentEventType.FINISHED, data: { conversationId } },
})
@@ -442,13 +449,16 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
},
}
-function incompleteReason({ truncatedAfterRetries, budgetExceeded }: { truncatedAfterRetries: boolean, budgetExceeded: boolean }): string | undefined {
+function incompleteReason({ truncatedAfterRetries, budgetExceeded, creditsExhausted }: { truncatedAfterRetries: boolean, budgetExceeded: boolean, creditsExhausted: boolean }): string | undefined {
if (truncatedAfterRetries) {
return 'The response reached the output limit before the agent finished'
}
if (budgetExceeded) {
return 'The run reached its usage limit and was stopped'
}
+ if (creditsExhausted) {
+ return 'The run ran out of credits and was stopped'
+ }
return undefined
}
diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
index 8327dd29d30..c76da90e8b1 100644
--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
@@ -1,6 +1,6 @@
import { AIProviderName, ErrorCode, formatPieceError, isNil, isObject, isProviderBillingError, isTransientProviderError, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'
import { agentAiUtils, ContentPartLike, modelCatalog } from '@activepieces/server-utils'
-import { AgentPhase, AgentRunSource, agentToolClassification, agentToolPhases, aiProviderUtils, apErrorOf, PersistedAgentPart } from '@activepieces/shared'
+import { AgentPhase, AgentRunSource, agentToolClassification, agentToolPhases, aiProviderUtils, apErrorOf, CHAT_CREDITS_PER_TOOL_CALL, chatBilling, ChatToolCall, PersistedAgentPart } from '@activepieces/shared'
import { APICallError, generateText, isLoopFinished, isStepCount, LanguageModel, LanguageModelUsage, ModelMessage, NoSuchToolError, RetryError, StepResultPerformance, StopCondition, streamText, ToolExecutionOptions, ToolSet } from 'ai'
const MAX_RESPONSE_OUTPUT_TOKENS = 32_000
@@ -40,13 +40,30 @@ export function shouldRetryStream({ producedVisibleOutput, streamRetries }: {
return !producedVisibleOutput && streamRetries < MAX_STREAM_RETRIES
}
-export async function runAgentTurn({ model, fastModel, provider, systemPrompt, messages, tools, allToolNames, tier, modelId, fastModelId, phaseState, abortSignal, log, sinks, stopWhen, stepCeiling }: RunAgentTurnParams): Promise<AgentTurnResult> {
+export async function runAgentTurn({ model, fastModel, provider, systemPrompt, messages, tools, allToolNames, tier, modelId, fastModelId, phaseState, abortSignal, log, sinks, stopWhen, stepCeiling, creditsLeft }: RunAgentTurnParams): Promise<AgentTurnResult> {
const drainStream = sinks?.drainStream ?? (async () => {})
const onProgress = sinks?.onProgress ?? (() => {})
const baseStopCondition = stopWhen ?? isLoopFinished()
+ let creditsExhausted = false
+ let paidToolsAffordable = true
+ let earlierAttemptToolCalls: ChatToolCall[] = []
+ const creditsRanOut: StopCondition<ToolSet> = async ({ steps }) => {
+ if (isNil(creditsLeft)) {
+ return false
+ }
+ const pendingCredits = chatBilling.creditsForTurn({ provider, toolCalls: [...earlierAttemptToolCalls, ...completedToolCalls(steps)] }).total
+ const { data: left, error } = await tryCatch(() => creditsLeft(pendingCredits))
+ if (error) {
+ log.warn({ error }, 'Credit check failed mid-turn, letting the turn continue')
+ }
+ creditsExhausted = !isNil(left) && left < 0
+ paidToolsAffordable = isNil(left) || left >= CHAT_CREDITS_PER_TOOL_CALL
+ return creditsExhausted
+ }
const loopStopCondition = [
...(Array.isArray(baseStopCondition) ? baseStopCondition : [baseStopCondition]),
isStepCount(stepCeiling ?? MAX_AGENT_STEPS),
+ creditsRanOut,
]
const guardedTools = wrapToolsWithFailureGuard({ tools, log })
const maxTurnTokens = runawayTokenCeiling(provider)
@@ -110,7 +127,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
lastStepModelId = usesFastModel ? fastModelId ?? modelId : modelId
return {
...(usesFastModel ? { model: fastModel } : {}),
- activeTools: agentToolPhases.activeToolsForPhase({ phase: phaseState.phase, allToolNames }),
+ activeTools: agentToolPhases.activeToolsForPhase({ phase: phaseState.phase, allToolNames }).filter((name) => paidToolsAffordable || !chatBilling.isPaidTool(name)),
providerOptions: agentAiUtils.buildProviderOptions({ provider, tier, modelId: lastStepModelId, disableThinking }),
...boundContextForStep({ baseMessages: attemptMessages, steps, systemPrompt, provider }),
}
@@ -210,12 +227,19 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
result.finalStep,
])
const stepMessages = agentAiUtils.collectStepMessages(steps)
+ earlierAttemptToolCalls = [...earlierAttemptToolCalls, ...completedToolCalls(steps)]
usage = attemptUsage
totalInputTokens += attemptUsage.inputTokens ?? 0
totalOutputTokens += attemptUsage.outputTokens ?? 0
lastFinishReason = finishReason
logTurnPerformance({ performance: finalStep.performance, modelId: tier.modelId, stepCount: steps.length, log })
+ if (creditsExhausted) {
+ accumulatedResponseMessages.push(...stepMessages)
+ log.warn({ totalInputTokens, totalOutputTokens }, 'Chat turn stopped because the platform ran out of credits')
+ break
+ }
+
if (totalInputTokens + totalOutputTokens >= maxTurnTokens) {
accumulatedResponseMessages.push(...stepMessages)
budgetExceeded = true
@@ -256,6 +280,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
finishReason: lastFinishReason,
truncatedAfterRetries,
budgetExceeded,
+ creditsExhausted,
streamError,
continuations,
totalInputTokens,
@@ -398,6 +423,10 @@ export function clampOutputTokens({ thinkingBudget, ceilings }: { thinkingBudget
return Math.min(thinkingBudget + MAX_RESPONSE_OUTPUT_TOKENS, ...known)
}
+function completedToolCalls(steps: ReadonlyArray<{ toolResults: ReadonlyArray<{ toolName: string, output: unknown }> }>): ChatToolCall[] {
+ return steps.flatMap((step) => step.toolResults.map((result) => ({ toolName: result.toolName, output: result.output })))
+}
+
function runawayTokenCeiling(provider: AIProviderName): number {
return aiProviderUtils.getMaxContextTokens({ provider }) * RUNAWAY_TURN_CONTEXT_MULTIPLE
}
@@ -484,6 +513,7 @@ export type RunAgentTurnParams = {
sinks?: AgentTurnSinks
stopWhen?: StopCondition<ToolSet> | Array<StopCondition<ToolSet>>
stepCeiling?: number
+ creditsLeft?: (pendingCredits: number) => Promise<number | null>
}
export type AgentTurnResult = {
@@ -493,6 +523,7 @@ export type AgentTurnResult = {
finishReason: string
truncatedAfterRetries: boolean
budgetExceeded: boolean
+ creditsExhausted: boolean
streamError: Error | null
continuations: number
totalInputTokens: number
diff --git a/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts b/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts
new file mode 100644
index 00000000000..2b5467326c7
--- /dev/null
+++ b/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts
@@ -0,0 +1,103 @@
+import { AIProviderName } from '@activepieces/core-utils'
+import { PersistedAgentPartType, PersistedToolCallStatus } from '@activepieces/shared'
+import { tool } from 'ai'
+import { convertArrayToReadableStream, MockLanguageModelV3 } from 'ai/test'
+import { describe, expect, it, vi } from 'vitest'
+import { z } from 'zod'
+import { runAgentTurn } from '../../../src/lib/execute/jobs/ee/agent/run-agent-turn'
+
+describe('a turn that runs out of credits', () => {
+ it('stops after the step that took the balance below zero, instead of running every step', async () => {
+ const search = vi.fn(async () => SEARCH_RESULT)
+ const creditsLeft = vi.fn().mockResolvedValueOnce(5).mockResolvedValueOnce(5).mockResolvedValue(-1)
+
+ const turn = await runTurn({ search, creditsLeft })
+
+ expect(search).toHaveBeenCalledTimes(3)
+ expect(turn.creditsExhausted).toBe(true)
+ })
+
+ it('asks with what the turn has used so far, so credits the turn has not been billed yet still count', async () => {
+ const creditsLeft = vi.fn().mockResolvedValueOnce(5).mockResolvedValue(-1)
+
+ await runTurn({ search: async () => SEARCH_RESULT, creditsLeft })
+
+ expect(creditsLeft.mock.calls).toEqual([[2], [3]])
+ })
+
+ it('lets the agent answer on an exact balance, but without the paid tools', async () => {
+ const model = alwaysSearchingModel()
+
+ const turn = await runTurn({ search: async () => SEARCH_RESULT, creditsLeft: async () => 0, stepCeiling: 2, model })
+
+ const [firstStep, ...laterSteps] = model.doStreamCalls.map((call) => call.tools?.map((t) => t.name))
+ expect(firstStep).toEqual(['ap_web_search', 'ap_fetch_url'])
+ expect(laterSteps.length).toBeGreaterThan(0)
+ expect(laterSteps.every((names) => names?.join() === 'ap_fetch_url')).toBe(true)
+ expect(turn.creditsExhausted).toBe(false)
+ const searchStatuses = turn.uiParts.flatMap((part) => part.type === PersistedAgentPartType.TOOL_CALL ? [part.status] : [])
+ expect(searchStatuses[0]).toBe(PersistedToolCallStatus.COMPLETED)
+ expect(searchStatuses.slice(1).every((status) => status === PersistedToolCallStatus.ERROR)).toBe(true)
+ })
+
+ it('keeps going when the credit check itself fails', async () => {
+ const search = vi.fn(async () => SEARCH_RESULT)
+
+ const turn = await runTurn({ search, creditsLeft: failingCreditCheck, stepCeiling: 5 })
+
+ expect(search.mock.calls.length).toBeGreaterThan(3)
+ expect(turn.creditsExhausted).toBe(false)
+ })
+})
+
+async function runTurn({ search, creditsLeft, stepCeiling = 20, model = alwaysSearchingModel() }: {
+ search: () => Promise<unknown>
+ creditsLeft: (pendingCredits: number) => Promise<number | null>
+ stepCeiling?: number
+ model?: MockLanguageModelV3
+}): ReturnType<typeof runAgentTurn> {
+ return runAgentTurn({
+ model,
+ provider: AIProviderName.ANTHROPIC,
+ systemPrompt: 'You are a test agent.',
+ messages: [{ role: 'user', content: 'research this' }],
+ tools: {
+ ap_web_search: tool({ description: 'search the web', inputSchema: z.object({ query: z.string() }), execute: search }),
+ ap_fetch_url: tool({ description: 'read a page', inputSchema: z.object({ url: z.string() }), execute: async () => SEARCH_RESULT }),
+ },
+ allToolNames: ['ap_web_search', 'ap_fetch_url'],
+ tier: TIER,
+ modelId: TIER.modelId,
+ phaseState: { phase: 'discovery' },
+ abortSignal: new AbortController().signal,
+ log: SILENT_LOG,
+ stepCeiling,
+ creditsLeft,
+ })
+}
+
+async function failingCreditCheck(): Promise<number | null> {
+ throw new Error('rpc down')
+}
+
+function alwaysSearchingModel(): MockLanguageModelV3 {
+ let calls = 0
+ return new MockLanguageModelV3({
+ doStream: async () => {
+ calls++
+ return {
+ stream: convertArrayToReadableStream([
+ { type: 'stream-start' as const, warnings: [] },
+ { type: 'tool-call' as const, toolCallId: `call-${calls}`, toolName: 'ap_web_search', input: '{"query":"more"}' },
+ { type: 'finish' as const, finishReason: 'tool-calls' as const, usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 } },
+ ]),
+ }
+ },
+ })
+}
+
+const SEARCH_RESULT = { content: [{ type: 'text', text: 'ok' }] }
+
+const TIER = { id: 'fast', thinkingBudget: 5_000, modelId: 'anthropic/claude-haiku-4.5' }
+
+const SILENT_LOG = { debug: () => undefined, info: () => undefined, warn: () => undefined, error: () => undefined }