AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(chat): chat tries another model when a provider fails oddly (#16668)
Hazem Adel8e10e4429d03
diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.mdindex 2932e5caea9..aa306426dc3 100644--- a/brain/knowledge/ai-intelligence/ai-providers.md+++ b/brain/knowledge/ai-intelligence/ai-providers.md@@ -96,6 +96,7 @@ after a file without chat keys it seeds the chat list from the live flow list.- **Tier entries go through the same project check as specific picks.** `platformModelTierService.getForRun` takes a `ProviderScope` and applies `aiKeyScope.rowAllowsScope` + `scopeAllows` to every entry; if the stored main entry fails, the whole tier is unavailable (it is checked before the healthy-first sort, so a down main doesn't change the answer). The `configId`s still come from a live tier on the caller's platform, never from the caller. Only the label lookups (`getManyForRun`, `firstCandidates`) use platform scope, because they return no credentials.- **Agent and chat runs read the tier grant from the conversation row, never from the worker.** CHAT and AGENT_BUILDER take `modelTierId` from the row (set by create/update). AGENT and FLOW_STEP take it from the job and write it onto the row **after** `acquireStreamingLock`, including null. Tool RPCs then call `aiModelCandidates.grantedEntryConfig` with the run's project, which honours a worker-sent `providerConfigId` only as an exact `{configId, modelId}` entry of that tier or of the Fast tier that runs in that project. A tier row switches projects through `agentModelTier.assertProjectSwitchKeepsTier`: every key the run can use (after the tool-calling filter, Fast tier included) must serve the new project. Clearing the project counts as moving to *every* project, so it needs keys open to all projects; a switch from no project checks the tier's platform-wide entries, because nothing records which keys the run kept. Any other `providerConfigId` goes through the normal scope check.- **A platform tier step falls back only on provider failures, decided by `isFallbackWorthy` on a signal from `aiProviderSignal.fromError`.** 401/402/403/404/408/429/5xx, network errors (ai 7 throws them as `APICallError` with **no status** and `isRetryable: true`) and billing bodies (Anthropic sends low credit as a **400**) fall back; any other 4xx and our own errors (classify mismatch, no structured data, web search unsupported) stop. No fallback once a web-search step finished, since it was already billed.+- **Never flatten a provider error with `String(error)`; use `toError` from `@activepieces/core-utils`.** A provider can end a stream with a plain object instead of an `Error`. `String()` turns it into `"[object Object]"` and drops the status, so `isFallbackWorthy` has nothing to judge, the tier never moves to its next model, and the user sees `[object Object]` (incident 20 Oct 2026, AI-63). `toError` keeps the raw value as `cause`, which is what lets `aiProviderSignal.fromError` reach the status.- **Worker-built models report no key health through `createModel`: no reporter is installed in the worker.** Tier steps report explicitly via the `reportAiKeyOutcome` RPC, and only provider signals (`fromProvider`) plus recoveries of non-active keys. Reporting our own errors would demote healthy keys to `unreachable`. The API never writes a worker-reported status for most providers: `confirmReportedOutcome` runs in the background and only triggers a `recheck` of the real key (once per key per 10s) when the report disagrees with the stored status. The exception is providers flagged `validationSkipsModelEndpoint` (CUSTOM, Vertex): their check never reaches the model, so a recheck would always pass, and the report is recorded as-is. Any direct `aiProviderHealth.record` from concurrent callers must pass `expectVersion` (the `statusVersion` it read), or an older report landing late overwrites a newer one. The worker token is shared and a shared worker serves every platform, so a report can't be bound to a platform; a forged one can at most cause a recheck. Don't pass `platformId`/`providerConfigId` to `createModel` in the worker, or a future reporter double-counts.- **`ExecuteAiJobData` is a zod schema that strips unknown keys** (`JobData.parse` in `worker.ts` and `job-broker.ts`). A field added only to the TS type silently never reaches the worker; add it to `AiStepJobBase`.- **Builders read the slim tier list (`GET /v1/platform-model-tiers`) with no `configId` or key names; the full entries are `GET /admin` only.** Don't widen the builder view: it would show project users the ids of keys scoped away from them.diff --git a/packages/core/utils/package.json b/packages/core/utils/package.jsonindex 23d0c4897d1..bc584f10393 100644--- a/packages/core/utils/package.json+++ b/packages/core/utils/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/core-utils",- "version": "0.10.4",+ "version": "0.10.5","type": "commonjs","main": "./dist/src/index.js","scripts": {diff --git a/packages/core/utils/src/lib/friendly-piece-error.ts b/packages/core/utils/src/lib/friendly-piece-error.tsindex c713845ecdc..5ac8ea065e3 100644--- a/packages/core/utils/src/lib/friendly-piece-error.ts+++ b/packages/core/utils/src/lib/friendly-piece-error.ts@@ -203,11 +203,12 @@ const extractResponseHttpDetails = (error: Record<string, unknown>): HttpDetails}const extractClientHttpDetails = (error: Record<string, unknown>): HttpDetails | null => {- const status = toHttpStatus(error['status'])+ const thrownAsPlainValue = !(error instanceof Error)+ const status = toHttpStatus(error['status']) ?? (thrownAsPlainValue ? toHttpStatus(error['statusCode']) : undefined)if (isNil(status)) {return null}- const responseBody = error['error'] ?? error['body']+ const responseBody = error['error'] ?? error['body'] ?? (thrownAsPlainValue ? error['responseBody'] : undefined)const headersValue = error['headers']const headers = isObjectRecord(headersValue) ? headersValue : undefined@@ -289,6 +290,10 @@ const readErrorMessage = (error: Record<string, unknown>): string => {if (stringified.length > 0 && stringified !== '[object Object]') {return stringified}+ const nested = extractApiMessage(error['error'])+ if (!isNil(nested)) {+ return nested+ }return 'Unknown error'}diff --git a/packages/core/utils/src/lib/try-catch.ts b/packages/core/utils/src/lib/try-catch.tsindex ef763c1490a..46eff19df7e 100644--- a/packages/core/utils/src/lib/try-catch.ts+++ b/packages/core/utils/src/lib/try-catch.ts@@ -44,6 +44,6 @@ export function toError(value: unknown): Error {return new Error(value)}const { data: serialized } = tryCatchSync(() => JSON.stringify(value))- return new Error(serialized ?? String(value))+ return new Error(serialized ?? String(value), { cause: value })}diff --git a/packages/server/utils/test/ai-provider-signal.test.ts b/packages/server/utils/test/ai-provider-signal.test.tsindex d2682642c7e..ca21f181abb 100644--- a/packages/server/utils/test/ai-provider-signal.test.ts+++ b/packages/server/utils/test/ai-provider-signal.test.ts@@ -1,3 +1,4 @@+import { isFallbackWorthy, toError } from '@activepieces/core-utils'import { APICallError, RetryError } from 'ai'import { describe, expect, it } from 'vitest'import { aiProviderSignal } from '../src/ai-provider-signal'@@ -45,3 +46,24 @@ describe('aiProviderSignal.fromError', () => {expect(aiProviderSignal.fromError(new Error('Unable to classify the text into the provided categories.'))).toMatchObject({ fromProvider: false })})})++describe('a provider failure that is not an Error', () => {+ it.each([+ ['statusCode, the shape the ai sdk uses', { statusCode: 503, responseBody: '{"error":"upstream unavailable"}' }],+ ['status, the shape a client library uses', { status: 503, body: '{"error":"upstream unavailable"}' }],+ ])('keeps the status of a plain object carrying %s', (_name, raw) => {+ const signal = aiProviderSignal.fromError(toError(raw))+ expect(signal.statusCode).toBe(503)+ expect(isFallbackWorthy(signal)).toBe(true)+ })++ it('never reports a provider failure as [object Object]', () => {+ const signal = aiProviderSignal.fromError(toError({ error: { message: 'upstream unavailable' } }))+ expect(signal.message).not.toContain('[object Object]')+ expect(signal.message).toContain('upstream unavailable')+ })++ it('still turns a bare string into a readable message', () => {+ expect(aiProviderSignal.fromError(toError('stream dropped')).message).toBe('stream dropped')+ })+})diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.tsindex 1abecc3b161..edf59a86162 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts@@ -1,4 +1,4 @@-import { AiProviderKeyStatus, AIProviderName, ErrorCode, formatPieceError, isFallbackWorthy, isNil, isObject, isProviderBillingError, isTransientProviderError, ProviderOutcomeSignal, spreadIfDefined, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'+import { AiProviderKeyStatus, AIProviderName, ErrorCode, formatPieceError, isFallbackWorthy, isNil, isObject, isProviderBillingError, isTransientProviderError, ProviderOutcomeSignal, spreadIfDefined, toError, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'import { agentAiUtils, aiProviderSignal, ContentPartLike } from '@activepieces/server-utils'import { AgentPhase, AgentRunSource, agentToolClassification, agentToolPhases, agentToolSkills, AI_PROVIDER_ENTITY_TYPES, aiProviderUtils, apErrorOf, CHAT_CREDITS_PER_TOOL_CALL, chatBilling, ChatToolCall, LAZY_TOOL_NAME, PersistedAgentPart, SkillSurface, TASK_COMPLETION_TOOL_NAME } from '@activepieces/shared'import { APICallError, generateText, isLoopFinished, isStepCount, LanguageModel, LanguageModelUsage, ModelMessage, NoSuchToolError, RetryError, StepResultPerformance, StopCondition, streamText, ToolChoice, ToolExecutionOptions, ToolSet } from 'ai'@@ -257,7 +257,7 @@ export async function runAgentTurn({ models, fastModel, systemPrompt, messages,},onError: ({ error }) => {log.error({ error }, 'Chat streamText error')- streamError = error instanceof Error ? error : new Error(String(error))+ streamError = toError(error)},})diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.tsindex 8339fea33fa..122bd0073c4 100644--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts@@ -18,6 +18,28 @@ describe('a platform tier chat turn', () => {expect(textOf(turn)).toBe('from backup')})+ it.each([+ ['statusCode, the shape the ai sdk uses', { statusCode: 503, responseBody: '{"error":"overloaded"}' }],+ ['status, the shape a client library uses', { status: 503, body: '{"error":"overloaded"}' }],+ ['no status, only a nested message', { error: { message: 'service unavailable, try another upstream' } }],+ ])('answers on the next model when the main model fails with a plain object carrying %s', async (_name, raw) => {+ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: failingModelWithoutAnError(raw) }), tierModel({ modelId: 'backup', model: answeringModel('from backup') })] })++ expect(turn.streamError).toBeNull()+ expect(turn.answeredBy.modelId).toBe('backup')+ expect(textOf(turn)).toBe('from backup')+ })++ it('says what the provider returned instead of [object Object] when the whole tier is down', async () => {+ const raw = { error: { message: 'service unavailable, try another upstream' } }++ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: failingModelWithoutAnError(raw) }), tierModel({ modelId: 'backup', model: failingModelWithoutAnError(raw) })] })++ expect(turn.allModelsFailed).toBe(true)+ expect(turn.streamError?.message).toContain('service unavailable')+ expect(turn.streamError?.message).not.toContain('[object Object]')+ })+it('continues on the next model after the fast step already answered and ran a tool, without running it again', async () => {const search = vi.fn(async () => ({ content: [{ type: 'text', text: 'found' }] }))const backup = answeringModel('done')@@ -222,6 +244,14 @@ function failingModel({ statusCode }: { statusCode: number }): MockLanguageModel})}+function failingModelWithoutAnError(raw: unknown): MockLanguageModelV3 {+ return new MockLanguageModelV3({+ doStream: async () => {+ throw raw+ },+ })+}+function answeringModel(text: string, sentOptions?: SharedV3ProviderOptions[]): MockLanguageModelV3 {return new MockLanguageModelV3({doStream: async ({ providerOptions }) => {