Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(chat): chat tries another model when a provider fails oddly (#16668)

Hazem Adel8e10e4429d03
diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.md
index 2932e5caea9..aa306426dc3 100644
--- a/brain/knowledge/ai-intelligence/ai-providers.md
+++ b/brain/knowledge/ai-intelligence/ai-providers.md
@@ -96,6 +96,7 @@ after a file without chat keys it seeds the chat list from the live flow list.
- **Tier entries go through the same project check as specific picks.** `platformModelTierService.getForRun` takes a `ProviderScope` and applies `aiKeyScope.rowAllowsScope` + `scopeAllows` to every entry; if the stored main entry fails, the whole tier is unavailable (it is checked before the healthy-first sort, so a down main doesn't change the answer). The `configId`s still come from a live tier on the caller's platform, never from the caller. Only the label lookups (`getManyForRun`, `firstCandidates`) use platform scope, because they return no credentials.
- **Agent and chat runs read the tier grant from the conversation row, never from the worker.** CHAT and AGENT_BUILDER take `modelTierId` from the row (set by create/update). AGENT and FLOW_STEP take it from the job and write it onto the row **after** `acquireStreamingLock`, including null. Tool RPCs then call `aiModelCandidates.grantedEntryConfig` with the run's project, which honours a worker-sent `providerConfigId` only as an exact `{configId, modelId}` entry of that tier or of the Fast tier that runs in that project. A tier row switches projects through `agentModelTier.assertProjectSwitchKeepsTier`: every key the run can use (after the tool-calling filter, Fast tier included) must serve the new project. Clearing the project counts as moving to *every* project, so it needs keys open to all projects; a switch from no project checks the tier's platform-wide entries, because nothing records which keys the run kept. Any other `providerConfigId` goes through the normal scope check.
- **A platform tier step falls back only on provider failures, decided by `isFallbackWorthy` on a signal from `aiProviderSignal.fromError`.** 401/402/403/404/408/429/5xx, network errors (ai 7 throws them as `APICallError` with **no status** and `isRetryable: true`) and billing bodies (Anthropic sends low credit as a **400**) fall back; any other 4xx and our own errors (classify mismatch, no structured data, web search unsupported) stop. No fallback once a web-search step finished, since it was already billed.
+- **Never flatten a provider error with `String(error)`; use `toError` from `@activepieces/core-utils`.** A provider can end a stream with a plain object instead of an `Error`. `String()` turns it into `"[object Object]"` and drops the status, so `isFallbackWorthy` has nothing to judge, the tier never moves to its next model, and the user sees `[object Object]` (incident 20 Oct 2026, AI-63). `toError` keeps the raw value as `cause`, which is what lets `aiProviderSignal.fromError` reach the status.
- **Worker-built models report no key health through `createModel`: no reporter is installed in the worker.** Tier steps report explicitly via the `reportAiKeyOutcome` RPC, and only provider signals (`fromProvider`) plus recoveries of non-active keys. Reporting our own errors would demote healthy keys to `unreachable`. The API never writes a worker-reported status for most providers: `confirmReportedOutcome` runs in the background and only triggers a `recheck` of the real key (once per key per 10s) when the report disagrees with the stored status. The exception is providers flagged `validationSkipsModelEndpoint` (CUSTOM, Vertex): their check never reaches the model, so a recheck would always pass, and the report is recorded as-is. Any direct `aiProviderHealth.record` from concurrent callers must pass `expectVersion` (the `statusVersion` it read), or an older report landing late overwrites a newer one. The worker token is shared and a shared worker serves every platform, so a report can't be bound to a platform; a forged one can at most cause a recheck. Don't pass `platformId`/`providerConfigId` to `createModel` in the worker, or a future reporter double-counts.
- **`ExecuteAiJobData` is a zod schema that strips unknown keys** (`JobData.parse` in `worker.ts` and `job-broker.ts`). A field added only to the TS type silently never reaches the worker; add it to `AiStepJobBase`.
- **Builders read the slim tier list (`GET /v1/platform-model-tiers`) with no `configId` or key names; the full entries are `GET /admin` only.** Don't widen the builder view: it would show project users the ids of keys scoped away from them.
diff --git a/packages/core/utils/package.json b/packages/core/utils/package.json
index 23d0c4897d1..bc584f10393 100644
--- a/packages/core/utils/package.json
+++ b/packages/core/utils/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-utils",
- "version": "0.10.4",
+ "version": "0.10.5",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/utils/src/lib/friendly-piece-error.ts b/packages/core/utils/src/lib/friendly-piece-error.ts
index c713845ecdc..5ac8ea065e3 100644
--- a/packages/core/utils/src/lib/friendly-piece-error.ts
+++ b/packages/core/utils/src/lib/friendly-piece-error.ts
@@ -203,11 +203,12 @@ const extractResponseHttpDetails = (error: Record<string, unknown>): HttpDetails
}
const extractClientHttpDetails = (error: Record<string, unknown>): HttpDetails | null => {
- const status = toHttpStatus(error['status'])
+ const thrownAsPlainValue = !(error instanceof Error)
+ const status = toHttpStatus(error['status']) ?? (thrownAsPlainValue ? toHttpStatus(error['statusCode']) : undefined)
if (isNil(status)) {
return null
}
- const responseBody = error['error'] ?? error['body']
+ const responseBody = error['error'] ?? error['body'] ?? (thrownAsPlainValue ? error['responseBody'] : undefined)
const headersValue = error['headers']
const headers = isObjectRecord(headersValue) ? headersValue : undefined
@@ -289,6 +290,10 @@ const readErrorMessage = (error: Record<string, unknown>): string => {
if (stringified.length > 0 && stringified !== '[object Object]') {
return stringified
}
+ const nested = extractApiMessage(error['error'])
+ if (!isNil(nested)) {
+ return nested
+ }
return 'Unknown error'
}
diff --git a/packages/core/utils/src/lib/try-catch.ts b/packages/core/utils/src/lib/try-catch.ts
index ef763c1490a..46eff19df7e 100644
--- a/packages/core/utils/src/lib/try-catch.ts
+++ b/packages/core/utils/src/lib/try-catch.ts
@@ -44,6 +44,6 @@ export function toError(value: unknown): Error {
return new Error(value)
}
const { data: serialized } = tryCatchSync(() => JSON.stringify(value))
- return new Error(serialized ?? String(value))
+ return new Error(serialized ?? String(value), { cause: value })
}
diff --git a/packages/server/utils/test/ai-provider-signal.test.ts b/packages/server/utils/test/ai-provider-signal.test.ts
index d2682642c7e..ca21f181abb 100644
--- a/packages/server/utils/test/ai-provider-signal.test.ts
+++ b/packages/server/utils/test/ai-provider-signal.test.ts
@@ -1,3 +1,4 @@
+import { isFallbackWorthy, toError } from '@activepieces/core-utils'
import { APICallError, RetryError } from 'ai'
import { describe, expect, it } from 'vitest'
import { aiProviderSignal } from '../src/ai-provider-signal'
@@ -45,3 +46,24 @@ describe('aiProviderSignal.fromError', () => {
expect(aiProviderSignal.fromError(new Error('Unable to classify the text into the provided categories.'))).toMatchObject({ fromProvider: false })
})
})
+
+describe('a provider failure that is not an Error', () => {
+ it.each([
+ ['statusCode, the shape the ai sdk uses', { statusCode: 503, responseBody: '{"error":"upstream unavailable"}' }],
+ ['status, the shape a client library uses', { status: 503, body: '{"error":"upstream unavailable"}' }],
+ ])('keeps the status of a plain object carrying %s', (_name, raw) => {
+ const signal = aiProviderSignal.fromError(toError(raw))
+ expect(signal.statusCode).toBe(503)
+ expect(isFallbackWorthy(signal)).toBe(true)
+ })
+
+ it('never reports a provider failure as [object Object]', () => {
+ const signal = aiProviderSignal.fromError(toError({ error: { message: 'upstream unavailable' } }))
+ expect(signal.message).not.toContain('[object Object]')
+ expect(signal.message).toContain('upstream unavailable')
+ })
+
+ it('still turns a bare string into a readable message', () => {
+ expect(aiProviderSignal.fromError(toError('stream dropped')).message).toBe('stream dropped')
+ })
+})
diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
index 1abecc3b161..edf59a86162 100644
--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
@@ -1,4 +1,4 @@
-import { AiProviderKeyStatus, AIProviderName, ErrorCode, formatPieceError, isFallbackWorthy, isNil, isObject, isProviderBillingError, isTransientProviderError, ProviderOutcomeSignal, spreadIfDefined, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'
+import { AiProviderKeyStatus, AIProviderName, ErrorCode, formatPieceError, isFallbackWorthy, isNil, isObject, isProviderBillingError, isTransientProviderError, ProviderOutcomeSignal, spreadIfDefined, toError, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'
import { agentAiUtils, aiProviderSignal, ContentPartLike } from '@activepieces/server-utils'
import { AgentPhase, AgentRunSource, agentToolClassification, agentToolPhases, agentToolSkills, AI_PROVIDER_ENTITY_TYPES, aiProviderUtils, apErrorOf, CHAT_CREDITS_PER_TOOL_CALL, chatBilling, ChatToolCall, LAZY_TOOL_NAME, PersistedAgentPart, SkillSurface, TASK_COMPLETION_TOOL_NAME } from '@activepieces/shared'
import { APICallError, generateText, isLoopFinished, isStepCount, LanguageModel, LanguageModelUsage, ModelMessage, NoSuchToolError, RetryError, StepResultPerformance, StopCondition, streamText, ToolChoice, ToolExecutionOptions, ToolSet } from 'ai'
@@ -257,7 +257,7 @@ export async function runAgentTurn({ models, fastModel, systemPrompt, messages,
},
onError: ({ error }) => {
log.error({ error }, 'Chat streamText error')
- streamError = error instanceof Error ? error : new Error(String(error))
+ streamError = toError(error)
},
})
diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts
index 8339fea33fa..122bd0073c4 100644
--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts
+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts
@@ -18,6 +18,28 @@ describe('a platform tier chat turn', () => {
expect(textOf(turn)).toBe('from backup')
})
+ it.each([
+ ['statusCode, the shape the ai sdk uses', { statusCode: 503, responseBody: '{"error":"overloaded"}' }],
+ ['status, the shape a client library uses', { status: 503, body: '{"error":"overloaded"}' }],
+ ['no status, only a nested message', { error: { message: 'service unavailable, try another upstream' } }],
+ ])('answers on the next model when the main model fails with a plain object carrying %s', async (_name, raw) => {
+ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: failingModelWithoutAnError(raw) }), tierModel({ modelId: 'backup', model: answeringModel('from backup') })] })
+
+ expect(turn.streamError).toBeNull()
+ expect(turn.answeredBy.modelId).toBe('backup')
+ expect(textOf(turn)).toBe('from backup')
+ })
+
+ it('says what the provider returned instead of [object Object] when the whole tier is down', async () => {
+ const raw = { error: { message: 'service unavailable, try another upstream' } }
+
+ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: failingModelWithoutAnError(raw) }), tierModel({ modelId: 'backup', model: failingModelWithoutAnError(raw) })] })
+
+ expect(turn.allModelsFailed).toBe(true)
+ expect(turn.streamError?.message).toContain('service unavailable')
+ expect(turn.streamError?.message).not.toContain('[object Object]')
+ })
+
it('continues on the next model after the fast step already answered and ran a tool, without running it again', async () => {
const search = vi.fn(async () => ({ content: [{ type: 'text', text: 'found' }] }))
const backup = answeringModel('done')
@@ -222,6 +244,14 @@ function failingModel({ statusCode }: { statusCode: number }): MockLanguageModel
})
}
+function failingModelWithoutAnError(raw: unknown): MockLanguageModelV3 {
+ return new MockLanguageModelV3({
+ doStream: async () => {
+ throw raw
+ },
+ })
+}
+
function answeringModel(text: string, sentOptions?: SharedV3ProviderOptions[]): MockLanguageModelV3 {
return new MockLanguageModelV3({
doStream: async ({ providerOptions }) => {