Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(ai): run agent and chat turns on platform tiers with step-boundary fallback (#16093)

Louai Boumedienef05889be258b
diff --git a/brain/knowledge/ai-intelligence/ai-agents.md b/brain/knowledge/ai-intelligence/ai-agents.md
index f3942579d85..d554a59c8ed 100644
--- a/brain/knowledge/ai-intelligence/ai-agents.md
+++ b/brain/knowledge/ai-intelligence/ai-agents.md
@@ -19,7 +19,7 @@ A flow step type (the `run_agent` action of `@activepieces/piece-ai`) that runs
| `AGENT_BUILDER` | `conversation.modelName` | tier id |
| `FLOW_STEP` | `agent.published.modelName`, or the step's own `input.aiProviderModel.model` | same as `AGENT` |
-Three things follow. `run_agent` has two modes: pointed at a saved agent (`agentId` set) the step sends no model and the agent row decides; inline, it sends `input.aiProviderModel.model`. An agent row carries two configs: agent-page chat reads `agent.draft`, a `run_agent` step reads `agent.published` (via `resolvePublishedAgent`), so the same agent can already run different models through different doors. The fast model crosses every door: `firstStepUsesFastModel` (`run-agent-turn.ts`) runs the first step of almost every turn on the fast tier, flow steps included, whatever the user picked. `execute-agent-run.ts` persists `config.tier.id` onto `conversation.modelName` when the job carried no `modelName`, so conversations already hold tier ids today (inline `run_agent`, eval runs, every `AGENT_BUILDER`), read back by `chat-tool-billing.ts` and `chat-analytics-sync.ts`. **A persisted tier id is not a pin.** `pinnedModelOf` (`rpc/rpc-shared.ts`) ignores `conversation.modelName` for a conversation with an agent: that value is the resolved default, and once the console removes that tier it would reach a catalogue-less own-key provider as a literal model id. A saved agent's run model is the agent's own half (draft for AGENT, published for FLOW_STEP); only an agentless inline step reads `conversation.modelName`, which is the step's configured model.
+Three things follow. `run_agent` has two modes: pointed at a saved agent (`agentId` set) the step sends no model and the agent row decides; inline, it sends `input.aiProviderModel.model`. An agent row carries two configs: agent-page chat reads `agent.draft`, a `run_agent` step reads `agent.published` (via `resolvePublishedAgent`), so the same agent can already run different models through different doors. The fast model crosses every door: `firstStepUsesFastModel` (`run-agent-turn.ts`) runs the first step of almost every turn on the fast tier, flow steps included, whatever the user picked. `execute-agent-run.ts` persists `config.tier.id` onto `conversation.modelName` when the job carried no `modelName`, so conversations already hold tier ids today (inline `run_agent`, eval runs, every `AGENT_BUILDER`), read back by `chat-tool-billing.ts` and `chat-analytics-sync.ts`. **A persisted tier id is not a pin.** `getAgentConfig` picks `modelName ?? conversation.modelName`, so a job that carries no model reads the persisted value back (`pinnedModelOf` no longer exists). A saved agent's run model is the agent's own half (draft for AGENT, published for FLOW_STEP), copied onto the job by `jobFieldsFromConfig`.
### Tool types (AgentTool discriminated union)
@@ -73,13 +73,17 @@ Three things follow. `run_agent` has two modes: pointed at a saved agent (`agent
- **Reasoning cannot be *disabled* on a reasoning-native endpoint, and the managed catalogue is full of models nobody chose.** `buildProviderOptions` (`server/utils/src/agent-provider-options.ts`) expresses `disableThinking` as `reasoning: { enabled: false }` on the `ACTIVEPIECES`/`OPENROUTER` branch, and `prepareStep` sets `disableThinking` for the first step *and the whole discovery phase*. OpenRouter forwards the flag, and Gemini 3.x, GPT-6 Astra and Claude Fable 5.1 reject it outright: `Reasoning is mandatory for this endpoint and cannot be disabled.` The step dies before its first token. Four things this taught, in order of how easy each is to get wrong:
- **Swapping in `{ effort: 'minimal' }` unconditionally is a regression, not a fix.** OpenRouter maps `minimal` to `low` for Anthropic, which is 20% of `max_tokens`, so the *thinking-disabled* first step on the Fast tier would get a bigger reasoning allowance (7,400) than its thinking-*enabled* step asks for (5,000). Every default tier is Anthropic, so that degrades every working customer to fix the broken ones. The suppression is gated on `aiProviderUtils.isCuratedChatModelId` instead: a model we picked keeps `{ enabled: false }`, anything else gets `{ effort: 'minimal' }`.
- **`getCuratedChatModels` returns `undefined` for `ACTIVEPIECES` on purpose, so the managed provider serves the entire live OpenRouter catalogue.** Every failing model in the incident (`google/gemini-3.8-flash`, `openai/gpt-6-astra`, `openai/gpt-6-astra-pro`, `anthropic/claude-fable-5.1`) is absent from `ALLOWED_CHAT_MODELS_BY_PROVIDER`. Only a React `.filter()` narrows the picker to the three tiers; `modelName` is a free `z.string()` server-side and `agent-config-rpc` uses it verbatim for `FLOW_STEP` and `AGENT`. Hiding is not denying.
- - **The curated list is a guess about capability, so the code learns.** A rejection calls `noteReasoningIsMandatory`, which records the id **only if it is curated** so a caller-supplied model id cannot grow the set, and the existing one-shot `shouldRetryStream` re-issues the turn with a minimal budget. `gemini-3.7-flash` is curated *and* reasoning-native, which is exactly the case the list alone would miss.
+ - **The curated list is a fixed guess; nothing learns from a rejection any more.** `aiProviderUtils.canDisableReasoning` (`core/piece-types/src/lib/ai-providers.ts`) is a static check, and `shouldRetryStream` re-runs the identical request once. A curated model that turns out to be reasoning-native fails until the list is fixed.
- **`satisfies` against the vendor's own reasoning type is near-worthless here.** `{ effort: 'none' }`, `{ enabled: false, max_tokens: n }` and `{ enabled: false, effort: 'minimal' }` all satisfy it, because the vendor supports disabling and it is the *endpoints* that do not. The guard has to be a closed union of the three directives we actually send.
- **A tool's *property keys* are user text too, so they go through `mcp-tool-input.ts`.** Field labels a customer typed into the MCP trigger (`Email Sender`) are not valid schema keys on Anthropic. `modelKeyByPropertyName` sanitizes them with `mcpToolNameUtils.toProviderSafeIdentifier`, keeping them unique, with a positional fallback and a length cap. `toFlowPayload` maps the model's args back to the original labels, so `{{trigger['Email Sender']}}` keeps working. The MCP server and agent flow tools both use it; build any new flow-as-tool schema through it.
- **Do not raise `ai` past 7.0.69 — from `[email protected]` the SDK stops auto-executing tools, and it fails silently.** Bisected on 2026-09-21 while trying to reach `experimental_evaluate`. On 7.0.44 (what the repo pins) and every version through 7.0.69, a tool call executes normally. From **7.0.70** onward, including 7.0.103–7.0.107, `generateText`/`streamText` still *recognise* the call — `staticToolCalls=1`, a `tool-call` part is emitted with correctly parsed input — but `execute` is never invoked and `toolResults` stays `0`. No error, no warning. The changelog entry for 7.0.70 is `a828527: Prevent automatic tool execution when a model call ends with an unsafe finish reason`, but the gate fires for every `finishReason` tried, `tool-calls` and `stop` included. Reproduced in a clean project outside the monorepo, on both `MockLanguageModelV3` and `MockLanguageModelV4`, on zod 3 and zod 4; `needsApproval: false` and `toolApproval: () => false` do not restore it, and neither does emitting the full `tool-input-start`/`delta`/`end` lifecycle before the `tool-call`. The alarm is `packages/server/worker/test/lib/agent-eval/agent-tool-call-repair.test.ts` — 4 of its 5 tests go red, and they look like a *repair* regression, which is the wrong trail: a plain valid tool call does not execute either. **Consequence: `experimental_evaluate` (first shipped in 7.0.103) cannot be reached without breaking the agent**, so anything needing typed evaluation calls the gateway's `/v4/ai/evaluation-model` endpoint over `safeHttp` instead of taking the bump. Re-test the table above before any future `ai` upgrade; if a release fixes it, that is the one to jump to.
+- **A platform tier turn falls back per step, not per turn.** When a step fails on a provider error (`isFallbackWorthy`) before sending content, the turn continues on the next candidate from the steps already done, the way `continue_truncation` does. Tool results stay in the messages, so nothing runs twice. A fast step that sent nothing is dropped on **any** error, not only fallback-worthy ones: the Fast tier can sit on a smaller context window than the main tier (Haiku 200K under a Gemini 1M tier), and once the history outgrows it the fast model answers 400 "prompt is too long", which must not sink a turn the main model can still take. For the same reason compaction sizes its budget over the tier's candidates **and** the Fast tier, not the main tier alone. After any switch the turn stays on that candidate and has no fast model. A step that already sent text never falls back, and is not retried on the same model either (`shouldRetryStream` also reads the drain). Piece tools follow the model **running the current step** (`onStepModel` from `prepareStep`), so a fast step's tool calls use the fast key. The worker sends `answeredBy` on save, and billing labels the turn with it. Search and image stay on the pair they were resolved with, and knowledge-base search stays on the chat key, because the chunks were embedded with it.
+- **"Did the failing step send anything" is read off the UI stream, and `producedVisibleOutput` cannot answer it.** `uiParts` only grows in `onStepEnd`, after chunks have already reached the client. The UI stream sends `start`/`start-step` before any content. When the model call itself throws, there is no `start-step` at all, only `error`. A stream error arrives **before** `finish-step` (`text-delta, error, finish-step`). `trackStepContent` therefore resets on `start-step`/`finish-step` and freezes its answer at the first `error` chunk. Drive it through the real `toUIMessageStream` in tests, because a mocked drain hides the ordering.
+- **`AgentConfigResponse.tier` is always the console credits tier.** A platform tier run fills `platformTier`, `candidates` and `fastCandidate`. The single `credentials`/`modelId` fields equal `candidates[0]`. On a tier run the worker sends no `modelName` to `saveAgentMessages`, and the RPC ignores one on a tier row, because a model name clears the tier (the last write wins).
+
### Key files
Entry point: `runAgent`, the createAction in the `ai` piece registered in `packages/pieces/community/ai/src/index.ts`.
diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.md
index 318b4b2593e..34c0be786c9 100644
--- a/brain/knowledge/ai-intelligence/ai-providers.md
+++ b/brain/knowledge/ai-intelligence/ai-providers.md
@@ -78,6 +78,7 @@ after a file without chat keys it seeds the chat list from the live flow list.
- **The live-name unique index is `lower("name")`, which `EntitySchema` cannot express.** The entity declares it `synchronize: false` (so `check-migrations` skips it), which means test DBs built by `synchronize` don't have it. The case-insensitive check that tests rely on is the service's `assertNameIsFree`, which is race-free because it runs under the `platform` row lock; the index is the backstop.
- **`lockPlatform` takes `FOR NO KEY UPDATE`, not `FOR UPDATE`.** Every FK insert that references `platform` (projects, users, …) takes `FOR KEY SHARE` on the platform row, which `FOR UPDATE` blocks until commit. `FOR NO KEY UPDATE` serialises tier writes just the same without stalling those inserts.
- **The tier grant lives in one function: `aiModelCandidates.resolve` (via `platformModelTierService.getForRun`) is the only path that hands out a key without `rowAllowsScope`.** The `configId`s come from a live tier on the caller's platform, never from the caller; keep it that way. A specific pick still goes through `getConfigOrThrow` with the project scope.
+- **Agent and chat runs read the tier grant from the conversation row, never from the worker.** CHAT and AGENT_BUILDER take `modelTierId` from the row (set by create/update). AGENT and FLOW_STEP take it from the job and write it onto the row **after** `acquireStreamingLock`, including null. Tool RPCs then call `aiModelCandidates.grantedEntryConfig`, which honours a worker-sent `providerConfigId` only as an exact `{configId, modelId}` entry of that tier or of the Fast tier. A tier row also skips `assertProjectSwitchKeepsKey`. Any other `providerConfigId` goes through the normal scope check.
- **A platform tier step falls back only on provider failures, decided by `isFallbackWorthy` on a signal from `aiProviderSignal.fromError`.** 401/402/403/404/408/429/5xx, network errors (ai 7 throws them as `APICallError` with **no status** and `isRetryable: true`) and billing bodies (Anthropic sends low credit as a **400**) fall back; any other 4xx and our own errors (classify mismatch, no structured data, web search unsupported) stop. No fallback once a web-search step finished, since it was already billed.
- **Worker-built models report no key health through `createModel`: no reporter is installed in the worker.** Tier steps report explicitly via the `reportAiKeyOutcome` RPC, and only provider signals (`fromProvider`) plus recoveries of non-active keys. Reporting our own errors would demote healthy keys to `unreachable`. The API never writes a worker-reported status for most providers: `confirmReportedOutcome` runs in the background and only triggers a `recheck` of the real key (once per key per 10s) when the report disagrees with the stored status. The exception is providers flagged `validationSkipsModelEndpoint` (CUSTOM, Vertex): their check never reaches the model, so a recheck would always pass, and the report is recorded as-is. Any direct `aiProviderHealth.record` from concurrent callers must pass `expectVersion` (the `statusVersion` it read), or an older report landing late overwrites a newer one. The worker token is shared and a shared worker serves every platform, so a report can't be bound to a platform; a forged one can at most cause a recheck. Don't pass `platformId`/`providerConfigId` to `createModel` in the worker, or a future reporter double-counts.
- **`ExecuteAiJobData` is a zod schema that strips unknown keys** (`JobData.parse` in `worker.ts` and `job-broker.ts`). A field added only to the TS type silently never reaches the worker; add it to `AiStepJobBase`.
diff --git a/brain/knowledge/decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.md b/brain/knowledge/decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.md
index f5df4c7fd19..11b2ff7e603 100644
--- a/brain/knowledge/decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.md
+++ b/brain/knowledge/decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.md
@@ -17,4 +17,5 @@ If a tier were limited by each key's project scope, the same tier would run diff
- The grant is security-sensitive. The runtime must only honour a `configId` that a live tier on the same platform contains, and never a client-sent `configId`.
- A key a live tier uses cannot be deleted, and its `modelScope` cannot drop a tier's model. `aiProviderService` refuses both.
- A deleted tier stays as a soft-deleted row with a `replacedBy` pointer, so stored refs keep resolving without rewriting any flow.
+- On agent and chat runs the grant also covers the platform's **Fast** tier, because the first step runs on it. Fallback there is per step: a step that fails before sending content continues on the next candidate.
- Platform tiers are unrelated to the console **credits tiers** (`ModelTier`, `pricing.json`). They share only the word "tier".
diff --git a/brain/knowledge/engineering/ci-pr-review-hygiene.md b/brain/knowledge/engineering/ci-pr-review-hygiene.md
index ec968423190..137583b7138 100644
--- a/brain/knowledge/engineering/ci-pr-review-hygiene.md
+++ b/brain/knowledge/engineering/ci-pr-review-hygiene.md
@@ -24,7 +24,7 @@ Enforcement is the **`Codeowners review` repository ruleset** (active on the def
## Gotchas
- **When merging main, `rerere` can replay a stale `bun.lock` resolution.** "Staged 'bun.lock' using previous resolution" means git reused an old answer, which can carry workspace versions that no longer match any `package.json` (seen: core-utils `0.8.6` in the lockfile against `0.8.5` in the manifest). If the branch adds no dependencies, take main's `bun.lock` and edit only the branch's own workspace `version` lines. Expect `bun install` to then rewrite a few lines for pieces the branch never touched: piece PRs on `main` often bump a `package.json` without its `bun.lock` entry (seen 2026-10-05: grist, microsoft-todo, ticktick, zendesk). Those lines are main catching up, not your change; keep them, because the lock should match the manifests.
- **Any change to `bun.lock` runs every package's tests in `build-test`, all 700+ pieces included.** The `changes` job in `ci.yml` sets `all=true` when a changed file is outside `packages/`, `docs/`, `brain/`, `.agents/` and `.claude/` and is not a `*.md` file (so root Markdown alone does not count), and `tools/scripts/test-filters.ts` then adds every package with a `test` script **except `api`** (it lists with `--filter=!api`). API tests run in the separate `api-tests` matrix job (`api (ce)`, `api (cloud)`, `api (ee, unit, migrations)`), so look there for their results, not in `build-test`. Without that, a piece only runs when its own files or `pieces/framework` / `pieces/common` change; a `core-utils` change alone does not pull pieces in. Any workspace version bump edits `bun.lock`, so this hits most PRs that bump a package. A CPU-heavy piece test can then fail on the starved runner with no change of yours: an Odoo polling test that takes ~350ms locally passed vitest's 5s default in 3 of 3 runs, reruns included. Don't keep rerunning: give the heavy test an explicit timeout (third `it` argument). Even a test-only change needs that piece's patch bump (and its `bun.lock` line): `validate-publishable-packages` diffs the whole package folder against `origin/main`.
-- **After the base PR of a stack merges, merging main into the next PR silently keeps the base PR's old lines.** The merge base predates the base PR, so code the base PR dropped during review (fields, routes, columns) looks like a branch addition and survives with no conflict. Check with `git diff --name-only origin/main`: every file must be one the stacked PR itself changed. If not, save the stacked PR's own patch first (`git diff --binary <base-PR tip it was built on> <head before the merge> > pr.patch`; without `--binary`, an image or other binary file is written as "Binary files differ" and `read-tree` deletes it with no way back), then `git read-tree -u --reset origin/main` and `git apply --3way pr.patch`, and commit. `read-tree` also deletes files that exist only on the branch (a file the base PR added, then dropped), which `git checkout origin/main -- .` would leave behind. Use `origin/main`, not `MERGE_HEAD`: a merge with no conflicts commits itself and `MERGE_HEAD` is gone. Then retarget it (`gh pr edit <n> --base main`): a PR still pointed at the merged base branch shows every main change since then (seen: 1,285 files for a 28-file PR). No rebase or force push needed.
+- **After the base PR of a stack merges, merging main into the next PR silently keeps the base PR's old lines.** The merge base predates the base PR, so code the base PR dropped during review (fields, routes, columns) looks like a branch addition and survives with no conflict. Check with `git diff --name-only origin/main`: every file must be one the stacked PR itself changed. If not, save the stacked PR's own patch first (`git diff --binary <base-PR tip it was built on> <head before the merge> > pr.patch`; without `--binary`, an image or other binary file is written as "Binary files differ" and `read-tree` deletes it with no way back), then `git read-tree -u --reset origin/main` and `git apply --3way pr.patch`, and commit. `read-tree` also deletes files that exist only on the branch (a file the base PR added, then dropped), which `git checkout origin/main -- .` would leave behind. Use `origin/main`, not `MERGE_HEAD`: a merge with no conflicts commits itself and `MERGE_HEAD` is gone. **The rebuild is not finished until `origin/main` is an ancestor.** `read-tree` + `apply` makes a commit whose *tree* is main + patch, but the merge base with main is still the old one, so GitHub's PR diff counts every main change since then as the PR's (seen: 731 files for a 43-file PR). Follow it with `git merge origin/main`; every conflict is the same change on both sides, so `git diff --name-only --diff-filter=U | xargs git checkout --ours --` is the answer, and `git diff --cached HEAD --stat` must print nothing before you commit (the tree does not change, only the ancestry). A branch cut from a stacked branch with `git checkout -b` inherits its **upstream**, so `git status -sb` reads `feat/pr-4...origin/feat/pr-3` and a plain `git push` would push PR 4's commits onto PR 3's remote; run `git branch --unset-upstream` right after cutting it (seen 2026-10-05 on `feat/platform-tiers-admin-ui`). Then retarget it (`gh pr edit <n> --base main`): a PR still pointed at the merged base branch shows every main change since then (seen: 1,285 files for a 28-file PR). No rebase or force push needed.
- **The tests bucket keys off a path *segment*, so a feature folder named `test-step` is not mistaken for tests.** `packages/web/src/app/builder/test-step/` and `latest-test.ts` stay in `packages/web`; the regex requires `test/` as a whole segment or a `.test.`/`.spec.` suffix. The one deliberate overlap is `packages/pieces/framework/src/lib/test/index.ts` — shipped `createMockActionContext` helpers, counted as tests, and exempt either way because pieces are exempt.
- **A job-level `env:` in `ci.yml` does not reach anything turbo runs.** Turbo 2.x defaults to `--env-mode=strict`: a task's process only receives variables named in `globalPassThroughEnv` / `passThroughEnv` / `env`, plus turbo's own system list. Everything else is dropped silently, so the task falls back to whatever `packages/server/api/.env.tests` set (dotenv there runs without `override`, so a real env var would otherwise win). Symptom: the workflow clearly sets the variable and the test still uses the old value. Add the name to `globalPassThroughEnv` in `turbo.json` — that is why `AP_DB_TYPE` and `AP_POSTGRES_*` are already listed. Jobs that call `npx vitest` directly, like `tool-search-postgres`, are unaffected, which is what makes the difference easy to miss. Verify with a throwaway test that `console.log`s the variable through `npx turbo run <task> --filter=<pkg>`.
- **A GitHub service container is not reachable by its name from a non-container job.** `services:` attaches the container to a docker network with a `--network-alias`, but the runner itself is on the host, so the alias does not resolve (`getaddrinfo EAI_AGAIN redis`) and only the published `ports:` mapping on `localhost` works. `.env.tests` pins `AP_REDIS_HOST=redis` for `docker-compose`, so any test that opens a real connection needs `localhost` forced past it.
diff --git a/bun.lock b/bun.lock
index f79006079c7..60ae67471b8 100644
--- a/bun.lock
+++ b/bun.lock
@@ -122,7 +122,7 @@
},
"packages/core/execution": {
"name": "@activepieces/core-execution",
- "version": "0.26.0",
+ "version": "0.26.1",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -167,7 +167,7 @@
},
"packages/core/shared": {
"name": "@activepieces/shared",
- "version": "0.194.0",
+ "version": "0.195.0",
"dependencies": {
"@activepieces/core-execution": "workspace:*",
"@activepieces/core-formula": "workspace:*",
diff --git a/packages/core/execution/package.json b/packages/core/execution/package.json
index bb1a0317885..449d9e2ce2c 100644
--- a/packages/core/execution/package.json
+++ b/packages/core/execution/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-execution",
- "version": "0.26.0",
+ "version": "0.26.1",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/execution/src/lib/workers/job-data.ts b/packages/core/execution/src/lib/workers/job-data.ts
index d2950f86a32..cfd6c5872dc 100644
--- a/packages/core/execution/src/lib/workers/job-data.ts
+++ b/packages/core/execution/src/lib/workers/job-data.ts
@@ -328,6 +328,7 @@ export const ExecuteAgentRunJobData = z.object({
provider: z.enum(AIProviderName).optional(),
providerConfigId: z.string().optional(),
modelName: z.string().nullable(),
+ modelTierId: z.string().optional(),
files: z.array(z.object({
name: z.string(),
mimeType: z.string(),
diff --git a/packages/core/execution/src/lib/workers/worker-contract.ts b/packages/core/execution/src/lib/workers/worker-contract.ts
index 4efc1920953..c76a223607c 100644
--- a/packages/core/execution/src/lib/workers/worker-contract.ts
+++ b/packages/core/execution/src/lib/workers/worker-contract.ts
@@ -138,6 +138,7 @@ export type GetAgentConfigRequest = {
projectId?: string | null
userMessage: string
modelName: string | null
+ modelTierId?: string
files?: Array<{ name: string, mimeType: string, data: string }>
promptOverride?: AgentPromptOverride
dryRun?: boolean
@@ -178,6 +179,13 @@ export type AgentConfigResponse = {
agentsAvailable: boolean
userEmail: string
source: AgentRunSource
+ platformTier?: { id: string, name: string }
+ candidates?: AgentModelCandidate[]
+ fastCandidate?: AgentModelCandidate
+}
+
+export type AgentModelCandidate = AiModelCandidate & {
+ thinkingBudget: number
}
export type SaveAgentMessagesRequest = {
@@ -187,9 +195,15 @@ export type SaveAgentMessagesRequest = {
uiMessages: unknown[]
title?: string
modelName?: string
+ answeredBy?: AgentTurnModel
failure?: { message: string, userMessage?: string }
}
+export type AgentTurnModel = {
+ provider: AIProviderName
+ modelId: string
+}
+
export type SaveAgentFileRequest = {
platformId: string
projectId?: string
diff --git a/packages/core/shared/package.json b/packages/core/shared/package.json
index 21514f7212e..5348e9290d3 100644
--- a/packages/core/shared/package.json
+++ b/packages/core/shared/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/shared",
- "version": "0.194.0",
+ "version": "0.195.0",
"type": "commonjs",
"sideEffects": false,
"main": "./dist/src/index.js",
diff --git a/packages/core/shared/src/lib/ee/agent/agent.ts b/packages/core/shared/src/lib/ee/agent/agent.ts
index cc8bfb55b99..ccbbc9cfbba 100644
--- a/packages/core/shared/src/lib/ee/agent/agent.ts
+++ b/packages/core/shared/src/lib/ee/agent/agent.ts
@@ -44,6 +44,7 @@ const AgentConfig = z.object({
provider: Nullable(z.enum(AIProviderName)),
providerConfigId: Nullable(ApId),
modelName: Nullable(z.string().max(MAX_AGENT_NAME_LENGTH)),
+ modelTierId: Nullable(ApId),
maxSteps: z.number().int().positive().max(MAX_AGENT_STEP_BUDGET).default(DEFAULT_AGENT_MAX_STEPS),
tools: z.array(AgentTool).max(MAX_AGENT_TOOLS).default([]),
structuredOutput: z.array(AgentOutputField).max(MAX_AGENT_OUTPUT_FIELDS).default([]),
diff --git a/packages/core/shared/src/lib/ee/agent/index.ts b/packages/core/shared/src/lib/ee/agent/index.ts
index 83ad2418794..c2a4785a089 100644
--- a/packages/core/shared/src/lib/ee/agent/index.ts
+++ b/packages/core/shared/src/lib/ee/agent/index.ts
@@ -1,5 +1,5 @@
import { AgentPromptOverride, AgentRunSource } from '@activepieces/core-execution'
-import { BaseModelSchema, Nullable } from '@activepieces/core-utils'
+import { ApId, BaseModelSchema, Nullable } from '@activepieces/core-utils'
import { z } from 'zod'
import { formErrors } from '../../form-errors'
import { MAX_AGENT_TEXT_LENGTH } from './agent'
@@ -205,6 +205,7 @@ export const AgentConversation = z.object({
source: z.enum(AgentRunSource),
title: Nullable(z.string()),
modelName: Nullable(z.string()),
+ modelTierId: Nullable(ApId),
status: z.nativeEnum(AgentConversationStatus).default(AgentConversationStatus.IDLE),
activeRunId: Nullable(z.string()),
messages: z.array(z.record(z.string(), z.unknown())).default([]),
@@ -223,6 +224,7 @@ export type AgentRunListItem = z.infer<typeof AgentRunListItem>
export const CreateAgentConversationRequest = z.object({
title: z.optional(Nullable(z.string())),
modelName: z.optional(Nullable(z.string())),
+ modelTierId: z.optional(Nullable(ApId)),
agentId: z.optional(z.string()),
builder: z.optional(z.boolean()),
projectId: z.optional(z.string()),
@@ -232,6 +234,7 @@ export type CreateAgentConversationRequest = z.infer<typeof CreateAgentConversat
export const UpdateAgentConversationRequest = z.object({
title: z.optional(Nullable(z.string())),
modelName: z.optional(Nullable(z.string())),
+ modelTierId: z.optional(Nullable(ApId)),
})
export type UpdateAgentConversationRequest = z.infer<typeof UpdateAgentConversationRequest>
diff --git a/packages/core/shared/src/lib/management/ai-providers/platform-model-tier.ts b/packages/core/shared/src/lib/management/ai-providers/platform-model-tier.ts
index f3ae8caf5c5..8318318ccf9 100644
--- a/packages/core/shared/src/lib/management/ai-providers/platform-model-tier.ts
+++ b/packages/core/shared/src/lib/management/ai-providers/platform-model-tier.ts
@@ -43,6 +43,7 @@ export const PlatformModelTier = z.object({
position: z.number().int(),
entries: z.array(PlatformModelTierEntry),
isDefault: z.boolean(),
+ isFast: z.boolean(),
thinkingBudget: Nullable(z.number().int()),
deleted: Nullable(DateOrString),
replacedBy: Nullable(ApId),
@@ -55,6 +56,7 @@ export const PlatformModelTierSummary = z.object({
description: Nullable(z.string()),
position: z.number().int(),
isDefault: z.boolean(),
+ isFast: z.boolean(),
mainModel: Nullable(z.object({
provider: z.enum(AIProviderName),
modelId: z.string(),
@@ -77,6 +79,7 @@ export const UpdatePlatformModelTierRequest = z.object({
entries: TierEntries.optional(),
thinkingBudget: Nullable(TierThinkingBudget),
isDefault: z.literal(true).optional(),
+ isFast: z.literal(true).optional(),
})
export const ReorderPlatformModelTiersRequest = z.object({
diff --git a/packages/server/api/src/app/ai/ai-model-candidates.ts b/packages/server/api/src/app/ai/ai-model-candidates.ts
index 5f687544075..8ee546de98e 100644
--- a/packages/server/api/src/app/ai/ai-model-candidates.ts
+++ b/packages/server/api/src/app/ai/ai-model-candidates.ts
@@ -1,8 +1,8 @@
-import { ActivepiecesError, AIProviderName, ErrorCode, isNil, PlatformId, tryCatch } from '@activepieces/core-utils'
-import { ResolveAiModelCandidatesResponse } from '@activepieces/shared'
+import { ActivepiecesError, AiProviderKeyStatus, AIProviderName, ErrorCode, isNil, PlatformId, tryCatch } from '@activepieces/core-utils'
+import { GetProviderConfigResponse, ResolveAiModelCandidatesResponse } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { aiProviderService } from './ai-provider-service'
-import { platformModelTierService } from './platform-model-tier-service'
+import { platformModelTierService, TierForRun } from './platform-model-tier-service'
export const aiModelCandidates = (log: FastifyBaseLogger) => ({
async firstCandidate({ platformId, tierId }: { platformId: PlatformId, tierId: string }): Promise<FirstCandidate> {
@@ -10,26 +10,75 @@ export const aiModelCandidates = (log: FastifyBaseLogger) => ({
return { provider: first.key.provider, providerConfigId: first.key.id, modelId: first.modelId }
},
- async resolve({ platformId, tierId }: { platformId: PlatformId, tierId: string }): Promise<ResolveAiModelCandidatesResponse> {
- const { tier, entries } = await platformModelTierService.getForRun({ platformId, id: tierId })
- const decrypted = await Promise.all(entries.map(async ({ modelId, key }) => {
- const { data: credentials, error } = await tryCatch(() => aiProviderService(log).credentialsForTierKey({ platformId, key }))
- if (!isNil(error) || isNil(credentials)) {
- log.warn({ error, aiProvider: { id: key.id }, platformTier: { id: tier.id } }, '[aiModelCandidates] Skipping a tier entry whose key cannot be read')
- return []
- }
- return [{ ...credentials, providerConfigId: key.id, modelId, status: key.status }]
+ async firstCandidates({ platformId, tierIds }: { platformId: PlatformId, tierIds: string[] }): Promise<Map<string, FirstCandidate | null>> {
+ const tiers = await platformModelTierService.getManyForRun({ platformId, ids: tierIds })
+ return new Map([...tiers].map(([tierId, forRun]) => {
+ const first = forRun?.entries[0]
+ return [tierId, isNil(first) ? null : { provider: first.key.provider, providerConfigId: first.key.id, modelId: first.modelId }]
}))
- const candidates = decrypted.flat()
+ },
+
+ async resolve({ platformId, tierId }: { platformId: PlatformId, tierId: string }): Promise<ResolveAiModelCandidatesResponse> {
+ const { tier, candidates } = await this.resolveConfigs({ platformId, tierId })
+ return {
+ tierName: tier.name,
+ candidates: candidates.map(({ config, modelId, status }) => ({ ...config, providerConfigId: config.configId, modelId, status })),
+ }
+ },
+
+ async resolveConfigs({ platformId, tierId }: { platformId: PlatformId, tierId: string }): Promise<TierConfigs> {
+ const forRun = await platformModelTierService.getForRun({ platformId, id: tierId })
+ const candidates = await readableCandidates({ platformId, forRun, log })
if (candidates.length === 0) {
- throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: `No model in tier "${tier.name}" can run` } })
+ throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: `No model in tier "${forRun.tier.name}" can run` } })
+ }
+ return { tier: forRun.tier, candidates }
+ },
+
+ async resolveFastConfig({ platformId }: { platformId: PlatformId }): Promise<TierConfigs | null> {
+ const forRun = await platformModelTierService.getFastForRun({ platformId })
+ if (isNil(forRun)) {
+ return null
}
- return { tierName: tier.name, candidates }
+ const candidates = await readableCandidates({ platformId, forRun, log })
+ return candidates.length === 0 ? null : { tier: forRun.tier, candidates }
+ },
+
+ async grantedEntryConfig({ platformId, tierId, configId, modelId }: { platformId: PlatformId, tierId: string, configId: string, modelId: string }): Promise<GetProviderConfigResponse | null> {
+ const [run, fast] = await Promise.all([
+ platformModelTierService.getForRun({ platformId, id: tierId }),
+ platformModelTierService.getFastForRun({ platformId }),
+ ])
+ const entry = [...run.entries, ...(fast?.entries ?? [])].find((candidate) => candidate.key.id === configId && candidate.modelId === modelId)
+ return isNil(entry) ? null : aiProviderService(log).configForTierKey({ platformId, key: entry.key })
},
})
+async function readableCandidates({ platformId, forRun, log }: { platformId: PlatformId, forRun: TierForRun, log: FastifyBaseLogger }): Promise<TierConfigCandidate[]> {
+ const read = await Promise.all(forRun.entries.map(async ({ modelId, key }) => {
+ const { data: config, error } = await tryCatch(() => aiProviderService(log).configForTierKey({ platformId, key }))
+ if (!isNil(error) || isNil(config)) {
+ log.warn({ error, aiProvider: { id: key.id }, platformTier: { id: forRun.tier.id } }, '[aiModelCandidates] Skipping a tier entry whose key cannot be read')
+ return []
+ }
+ return [{ config, modelId, status: key.status }]
+ }))
+ return read.flat()
+}
+
export type FirstCandidate = {
provider: AIProviderName
providerConfigId: string
modelId: string
}
+
+export type TierConfigCandidate = {
+ config: GetProviderConfigResponse
+ modelId: string
+ status: AiProviderKeyStatus
+}
+
+export type TierConfigs = {
+ tier: TierForRun['tier']
+ candidates: TierConfigCandidate[]
+}
diff --git a/packages/server/api/src/app/ai/ai-provider-service.ts b/packages/server/api/src/app/ai/ai-provider-service.ts
index 479aba38e45..0ed8ac75cf7 100644
--- a/packages/server/api/src/app/ai/ai-provider-service.ts
+++ b/packages/server/api/src/app/ai/ai-provider-service.ts
@@ -1,4 +1,4 @@
-import { ActivepiecesError, AiProviderCredentials, AiProviderKeyStatus, AIProviderName, apId, classifyProviderOutcome, ErrorCode, isNil, PlatformId, ProviderOutcomeSignal, spreadIfDefined, spreadIfNotUndefined, toProviderOutcomeSignal, tryCatch, unique } from '@activepieces/core-utils'
+import { ActivepiecesError, AiProviderKeyStatus, AIProviderName, apId, classifyProviderOutcome, ErrorCode, isNil, PlatformId, ProviderOutcomeSignal, spreadIfDefined, spreadIfNotUndefined, toProviderOutcomeSignal, tryCatch, unique } from '@activepieces/core-utils'
import { modelCatalog, modelTierCatalog } from '@activepieces/server-utils'
import { ActivePiecesProviderAuthConfig, AI_PROVIDER_ENTITY_TYPES, AIProviderAuthConfig, AIProviderConfig, aiProviderCredentials, AIProviderModel, AIProviderModelType, AiProviderProjectScope, aiProviderUtils, AIProviderWithoutSensitiveData, CreateAIProviderRequest, GetProviderConfigResponse, ProjectAIProvider, UpdateAIProviderRequest } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
@@ -289,9 +289,9 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({
const auth = await decryptRowAuth({ aiProvider, platformId, log })
return { ...aiProviderCredentials({ provider: aiProvider.provider, auth, config: aiProvider.config }), configId: aiProvider.id, platformId, modelScope: aiProvider.modelScope, modelIds: aiProvider.modelIds }
},
- async credentialsForTierKey({ platformId, key }: { platformId: PlatformId, key: AIProviderSchema }): Promise<AiProviderCredentials> {
+ async configForTierKey({ platformId, key }: { platformId: PlatformId, key: AIProviderSchema }): Promise<GetProviderConfigResponse> {
const auth = await decryptRowAuth({ aiProvider: key, platformId, log })
- return aiProviderCredentials({ provider: key.provider, auth, config: key.config })
+ return { ...aiProviderCredentials({ provider: key.provider, auth, config: key.config }), configId: key.id, platformId, modelScope: key.modelScope, modelIds: key.modelIds }
},
async getOrCreateActivePiecesProviderAuthConfig(platformId: PlatformId): Promise<ActivePiecesProviderAuthConfig> {
await ensureManagedProviderRow({ platformId })
diff --git a/packages/server/api/src/app/ai/platform-model-tier-entity.ts b/packages/server/api/src/app/ai/platform-model-tier-entity.ts
index 7b032035167..e28a0d9a7ec 100644
--- a/packages/server/api/src/app/ai/platform-model-tier-entity.ts
+++ b/packages/server/api/src/app/ai/platform-model-tier-entity.ts
@@ -40,6 +40,11 @@ export const PlatformModelTierEntity = new EntitySchema<PlatformModelTierSchema>
nullable: false,
default: false,
},
+ isFast: {
+ type: Boolean,
+ nullable: false,
+ default: false,
+ },
thinkingBudget: {
type: Number,
nullable: true,
@@ -71,6 +76,12 @@ export const PlatformModelTierEntity = new EntitySchema<PlatformModelTierSchema>
unique: true,
where: '"isDefault" = true AND "deleted" IS NULL',
},
+ {
+ name: 'idx_platform_model_tier_platform_fast',
+ columns: ['platformId'],
+ unique: true,
+ where: '"isFast" = true AND "deleted" IS NULL',
+ },
],
relations: {
platform: {
diff --git a/packages/server/api/src/app/ai/platform-model-tier-service.ts b/packages/server/api/src/app/ai/platform-model-tier-service.ts
index 3ef06df1cab..557f6ff640c 100644
--- a/packages/server/api/src/app/ai/platform-model-tier-service.ts
+++ b/packages/server/api/src/app/ai/platform-model-tier-service.ts
@@ -1,4 +1,4 @@
-import { ActivepiecesError, AIProviderName, apId, ErrorCode, isNil, PlatformId, spreadIfDefined, spreadIfNotUndefined, unique } from '@activepieces/core-utils'
+import { ActivepiecesError, AIProviderName, apId, ErrorCode, isNil, PlatformId, spreadIfDefined, spreadIfNotUndefined, tryCatch, unique } from '@activepieces/core-utils'
import { AiProviderModelScope, CreatePlatformModelTierRequest, PlatformModelTier, PlatformModelTierEntry, PlatformModelTierSummary, UpdatePlatformModelTierRequest } from '@activepieces/shared'
import { EntityManager, In } from 'typeorm'
import { repoFactory } from '../core/db/repo-factory'
@@ -48,6 +48,7 @@ export const platformModelTierService = {
entries: request.entries,
thinkingBudget: request.thinkingBudget ?? null,
isDefault: isFirstTier,
+ isFast: isFirstTier,
deleted: null,
replacedBy: null,
})
@@ -67,6 +68,9 @@ export const platformModelTierService = {
if (request.isDefault === true && !tier.isDefault) {
await tierRepo(manager).update({ platformId, isDefault: true }, { isDefault: false })
}
+ if (request.isFast === true && !tier.isFast) {
+ await tierRepo(manager).update({ platformId, isFast: true }, { isFast: false })
+ }
await tierRepo(manager).update({ platformId, id }, {
...spreadIfDefined('name', request.name),
...spreadIfDefined('emoji', request.emoji),
@@ -74,6 +78,7 @@ export const platformModelTierService = {
...spreadIfDefined('entries', request.entries),
...spreadIfNotUndefined('thinkingBudget', request.thinkingBudget),
...spreadIfDefined('isDefault', request.isDefault),
+ ...spreadIfDefined('isFast', request.isFast),
})
return getLiveOrThrow({ manager, platformId, id })
}))
@@ -113,7 +118,7 @@ export const platformModelTierService = {
const tier = await getLiveOrThrow({ manager, platformId, id })
if (isNil(replacedBy)) {
await assertLastTierCanGo({ manager, platformId })
- await tierRepo(manager).update({ platformId, id }, { isDefault: false })
+ await tierRepo(manager).update({ platformId, id }, { isDefault: false, isFast: false })
await tierRepo(manager).softDelete({ platformId, id })
return
}
@@ -122,10 +127,13 @@ export const platformModelTierService = {
'UPDATE "platform_model_tier" SET "replacedBy" = $1 WHERE "platformId" = $2 AND ("id" = $3 OR "replacedBy" = $3)',
[replacedBy, platformId, id],
)
- await tierRepo(manager).update({ platformId, id }, { isDefault: false })
+ await tierRepo(manager).update({ platformId, id }, { isDefault: false, isFast: false })
await tierRepo(manager).softDelete({ platformId, id })
- if (tier.isDefault) {
- await tierRepo(manager).update({ platformId, id: replacedBy }, { isDefault: true })
+ if (tier.isDefault || tier.isFast) {
+ await tierRepo(manager).update({ platformId, id: replacedBy }, {
+ ...(tier.isDefault ? { isDefault: true } : {}),
+ ...(tier.isFast ? { isFast: true } : {}),
+ })
}
})
},
@@ -138,26 +146,36 @@ export const platformModelTierService = {
}
},
+ async getFastForRun({ platformId }: { platformId: PlatformId }): Promise<TierForRun | null> {
+ const fastTier = await tierRepo().findOneBy({ platformId, isFast: true })
+ if (isNil(fastTier)) {
+ return null
+ }
+ const { data } = await tryCatch(() => platformModelTierService.getForRun({ platformId, id: fastTier.id }))
+ return data ?? null
+ },
+
async getForRun({ platformId, id }: { platformId: PlatformId, id: string }): Promise<TierForRun> {
const tier = await followReplacements({ platformId, id })
const configIds = unique(tier.entries.map((entry) => entry.configId))
const keys = await aiProviderRepo().findBy({ platformId, id: In(configIds) })
- const keyById = new Map(keys.map((key) => [key.id, key]))
- const runnable = tier.entries.flatMap((entry) => {
- const key = keyById.get(entry.configId)
- if (isNil(key) || key.provider === AIProviderName.ACTIVEPIECES || !scopeAllows({ modelScope: key.modelScope, modelIds: key.modelIds, modelId: entry.modelId })) {
- return []
- }
- return [{ modelId: entry.modelId, key }]
- })
- if (runnable.length === 0) {
+ const entries = runnableEntries({ tier, keyById: new Map(keys.map((key) => [key.id, key])) })
+ if (entries.length === 0) {
throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: `No model in tier "${tier.name}" can run` } })
}
- const healthyFirst = [
- ...runnable.filter((entry) => entry.key.status === 'active'),
- ...runnable.filter((entry) => entry.key.status !== 'active'),
- ]
- return { tier, entries: healthyFirst }
+ return { tier, entries }
+ },
+
+ async getManyForRun({ platformId, ids }: { platformId: PlatformId, ids: string[] }): Promise<Map<string, TierForRun | null>> {
+ const liveByRequestedId = await followReplacementsInBulk({ platformId, ids: unique(ids) })
+ const liveTiers = unique([...liveByRequestedId.values()].filter((tier): tier is PlatformModelTier => !isNil(tier)))
+ const configIds = unique(liveTiers.flatMap((tier) => tier.entries.map((entry) => entry.configId)))
+ const keys = configIds.length === 0 ? [] : await aiProviderRepo().findBy({ platformId, id: In(configIds) })
+ const keyById = new Map(keys.map((key) => [key.id, key]))
+ return new Map([...liveByRequestedId].map(([requestedId, tier]) => {
+ const entries = isNil(tier) ? [] : runnableEntries({ tier, keyById })
+ return [requestedId, isNil(tier) || entries.length === 0 ? null : { tier, entries }]
+ }))
},
async assertKeyScopeKeepsTiers({ manager, platformId, configId, modelScope, modelIds }: AssertKeyScopeParams): Promise<void> {
@@ -185,6 +203,41 @@ async function lockPlatform({ manager, platformId }: { manager: EntityManager, p
}
}
+function runnableEntries({ tier, keyById }: { tier: PlatformModelTier, keyById: Map<string, AIProviderSchema> }): TierForRun['entries'] {
+ const runnable = tier.entries.flatMap((entry) => {
+ const key = keyById.get(entry.configId)
+ if (isNil(key) || key.provider === AIProviderName.ACTIVEPIECES || !scopeAllows({ modelScope: key.modelScope, modelIds: key.modelIds, modelId: entry.modelId })) {
+ return []
+ }
+ return [{ modelId: entry.modelId, key }]
+ })
+ return [
+ ...runnable.filter((entry) => entry.key.status === 'active'),
+ ...runnable.filter((entry) => entry.key.status !== 'active'),
+ ]
+}
+
+async function followReplacementsInBulk({ platformId, ids }: { platformId: PlatformId, ids: string[] }): Promise<Map<string, PlatformModelTier | null>> {
+ const loaded = new Map<string, PlatformModelTier>()
+ let missing = ids
+ for (let hop = 0; hop <= MAX_REPLACEMENT_HOPS && missing.length > 0; hop++) {
+ const tiers = await tierRepo().find({ where: { platformId, id: In(missing) }, withDeleted: true })
+ tiers.forEach((tier) => loaded.set(tier.id, tier))
+ missing = unique(tiers.flatMap((tier) => isNil(tier.deleted) || isNil(tier.replacedBy) || loaded.has(tier.replacedBy) ? [] : [tier.replacedBy]))
+ }
+ const liveFor = (id: string): PlatformModelTier | null => {
+ let current = loaded.get(id)
+ for (let hop = 0; hop <= MAX_REPLACEMENT_HOPS && !isNil(current); hop++) {
+ if (isNil(current.deleted)) {
+ return current
+ }
+ current = isNil(current.replacedBy) ? undefined : loaded.get(current.replacedBy)
+ }
+ return null
+ }
+ return new Map(ids.map((id) => [id, liveFor(id)]))
+}
+
async function followReplacements({ platformId, id }: { platformId: PlatformId, id: string }): Promise<PlatformModelTier> {
let currentId = id
for (let hop = 0; hop <= MAX_REPLACEMENT_HOPS; hop++) {
@@ -288,6 +341,7 @@ function toSummary({ tier, providerByConfigId }: { tier: PlatformModelTier, prov
description: tier.description ?? null,
position: tier.position,
isDefault: tier.isDefault,
+ isFast: tier.isFast,
mainModel: isNil(main) || isNil(provider) ? null : { provider, modelId: main.modelId },
fallbackCount: Math.max(tier.entries.length - 1, 0),
}
diff --git a/packages/server/api/src/app/database/migration/postgres/1864000000000-AddPlatformTierFastAndConversationTier.ts b/packages/server/api/src/app/database/migration/postgres/1864000000000-AddPlatformTierFastAndConversationTier.ts
new file mode 100644
index 00000000000..6d576d1dd75
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/1864000000000-AddPlatformTierFastAndConversationTier.ts
@@ -0,0 +1,37 @@
+import { QueryRunner } from 'typeorm'
+import { Migration } from '../../migration'
+
+export class AddPlatformTierFastAndConversationTier1864000000000 implements Migration {
+ name = 'AddPlatformTierFastAndConversationTier1864000000000'
+ breaking = false
+ release = '0.93.0'
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`
+ ALTER TABLE "platform_model_tier"
+ ADD "isFast" boolean NOT NULL DEFAULT false
+ `)
+ await queryRunner.query(`
+ UPDATE "platform_model_tier" SET "isFast" = true WHERE "isDefault" = true AND "deleted" IS NULL
+ `)
+ await queryRunner.query(`
+ CREATE UNIQUE INDEX "idx_platform_model_tier_platform_fast" ON "platform_model_tier" ("platformId") WHERE "isFast" = true AND "deleted" IS NULL
+ `)
+ await queryRunner.query(`
+ ALTER TABLE "agent_conversation"
+ ADD "modelTierId" character varying(21)
+ `)
+ }
+
+ public async down(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`
+ ALTER TABLE "agent_conversation" DROP COLUMN "modelTierId"
+ `)
+ await queryRunner.query(`
+ DROP INDEX "idx_platform_model_tier_platform_fast"
+ `)
+ await queryRunner.query(`
+ ALTER TABLE "platform_model_tier" DROP COLUMN "isFast"
+ `)
+ }
+}
diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.ts
index 65bce6bb27d..5218f65bd28 100644
--- a/packages/server/api/src/app/database/postgres-connection.ts
+++ b/packages/server/api/src/app/database/postgres-connection.ts
@@ -451,6 +451,7 @@ import { AddPlatformModelTier1859000000000 } from './migration/postgres/18590000
import { AddPieceSetRequiredActions1860000000000 } from './migration/postgres/1860000000000-AddPieceSetRequiredActions'
import { AddEventDestinationEnabledAndFormat1862000000000 } from './migration/postgres/1862000000000-AddEventDestinationEnabledAndFormat'
import { AddEventDestinationHeaders1863000000000 } from './migration/postgres/1863000000000-AddEventDestinationHeaders'
+import { AddPlatformTierFastAndConversationTier1864000000000 } from './migration/postgres/1864000000000-AddPlatformTierFastAndConversationTier'
const getSslConfig = (): boolean | TlsOptions => {
const useSsl = system.get(AppSystemProp.POSTGRES_USE_SSL)
@@ -917,6 +918,7 @@ export const getMigrations = (): (new () => Migration)[] => {
AddPieceSetRequiredActions1860000000000,
AddEventDestinationEnabledAndFormat1862000000000,
AddEventDestinationHeaders1863000000000,
+ AddPlatformTierFastAndConversationTier1864000000000,
]
return migrations
}
diff --git a/packages/server/api/src/app/ee/agent/agent-conversation-controller.ts b/packages/server/api/src/app/ee/agent/agent-conversation-controller.ts
index 7f7a358bb2a..074915871f3 100644
--- a/packages/server/api/src/app/ee/agent/agent-conversation-controller.ts
+++ b/packages/server/api/src/app/ee/agent/agent-conversation-controller.ts
@@ -114,7 +114,9 @@ export const agentConversationController: FastifyPluginAsyncZod = async (app) =>
const isBuilder = conversation.source === AgentRunSource.AGENT_BUILDER
// resolveRunProvider and the assertion below both fall through to the platform's chat
// provider when no provider is named. An agent answers on its own model or it does not run.
- if (!isNil(agent) && !isBuilder && (isNil(agentConfig?.provider) || isNil(agentConfig?.modelName))) {
+ const runsOnAgentModel = !isNil(agentConfig) && !isBuilder
+ const runTierId = runsOnAgentModel ? agentConfig.modelTierId ?? null : conversation.modelTierId ?? null
+ if (runsOnAgentModel && isNil(runTierId) && (isNil(agentConfig.provider) || isNil(agentConfig.modelName))) {
throw new ActivepiecesError({
code: ErrorCode.VALIDATION,
params: { message: 'Pick a model for this agent before talking to it' },
@@ -133,6 +135,7 @@ export const agentConversationController: FastifyPluginAsyncZod = async (app) =>
platformId,
log,
scope: runScope,
+ modelTierId: runTierId,
...spreadIfDefined('provider', agentConfig?.provider ?? undefined),
...spreadIfDefined('providerConfigId', agentConfig?.providerConfigId ?? undefined),
})
diff --git a/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts b/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts
index 2f40d1c919b..47269b50927 100644
--- a/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts
+++ b/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts
@@ -50,6 +50,10 @@ export const AgentConversationEntity = new EntitySchema<AgentConversationWithRel
type: String,
nullable: true,
},
+ modelTierId: {
+ ...ApIdSchema,
+ nullable: true,
+ },
status: {
type: String,
nullable: false,
diff --git a/packages/server/api/src/app/ee/agent/agent-conversation-service.ts b/packages/server/api/src/app/ee/agent/agent-conversation-service.ts
index c27520bb6f1..69d976b77fd 100644
--- a/packages/server/api/src/app/ee/agent/agent-conversation-service.ts
+++ b/packages/server/api/src/app/ee/agent/agent-conversation-service.ts
@@ -3,6 +3,7 @@ import { Agent, AgentConversation, AgentConversationStatus, AgentHistoryMessage,
import { ModelMessage } from 'ai'
import { FastifyBaseLogger } from 'fastify'
import { EntityManager, In } from 'typeorm'
+import { platformModelTierService } from '../../ai/platform-model-tier-service'
import { transaction } from '../../core/db/transaction'
import { databaseConnection } from '../../database/database-connection'
import { FlowRunEntity } from '../../flows/flow-run/flow-run-entity'
@@ -55,12 +56,24 @@ async function projectStillHoldingAgent({ agentId, authorisedProjectId, entityMa
return locked.projectId
}
+async function modelChoiceFrom({ platformId, modelName, modelTierId }: { platformId: string, modelName: string | null | undefined, modelTierId: string | null | undefined }): Promise<ModelChoice> {
+ if (!isNil(modelTierId)) {
+ await platformModelTierService.getForRun({ platformId, id: modelTierId })
+ return { modelTierId, modelName: null }
+ }
+ if (!isNil(modelName)) {
+ return { modelName, modelTierId: null }
+ }
+ return modelTierId === null ? { modelTierId: null } : {}
+}
+
export const agentConversationService = (log: FastifyBaseLogger) => ({
async createConversation({ platformId, userId, request, id }: CreateConversationParams): Promise<AgentConversation> {
const agent = isNil(request.agentId)
? null
: await agentService(log).getOneOrThrowByPlatform({ id: request.agentId, platformId, userId })
const builder = request.builder === true
+ const modelChoice = await modelChoiceFrom({ platformId, modelName: request.modelName, modelTierId: request.modelTierId })
const builderProjectId = builder
? await resolveBuilderProject({ agent, requestedProjectId: request.projectId, platformId, userId, log })
: null
@@ -74,7 +87,8 @@ export const agentConversationService = (log: FastifyBaseLogger) => ({
agentId: agent?.id ?? null,
source: builder ? AgentRunSource.AGENT_BUILDER : isNil(agent) ? AgentRunSource.CHAT : AgentRunSource.AGENT,
title: request.title ?? null,
- modelName: request.modelName ?? null,
+ modelName: modelChoice.modelName ?? null,
+ modelTierId: modelChoice.modelTierId ?? null,
messages: [],
}))
log.info({ conversation: { id: conversation.id }, platform: { id: platformId }, user: { id: userId } }, '[agentConversationService] Conversation created')
@@ -206,7 +220,7 @@ export const agentConversationService = (log: FastifyBaseLogger) => ({
const conversation = await this.getConversationOrThrow({ id, platformId, userId })
const updates = {
...spreadIfDefined('title', request.title),
- ...spreadIfDefined('modelName', request.modelName),
+ ...await modelChoiceFrom({ platformId, modelName: request.modelName, modelTierId: request.modelTierId }),
}
if (Object.keys(updates).length > 0) {
@@ -327,3 +341,8 @@ type SetMessageFeedbackParams = ConversationIdentifier & {
messageIndex: number
request: SetAgentMessageFeedbackRequest
}
+
+type ModelChoice = {
+ modelName?: string | null
+ modelTierId?: string | null
+}
diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.ts
index 3047b5fddfb..7a5540c5a58 100644
--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts
+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts
@@ -8,6 +8,7 @@ import { EmbeddingModel, LanguageModel } from 'ai'
import { FastifyBaseLogger } from 'fastify'
import { Repository } from 'typeorm'
import { z } from 'zod'
+import { aiModelCandidates } from '../../ai/ai-model-candidates'
import { aiProviderService, ProviderScope } from '../../ai/ai-provider-service'
import { repoFactory } from '../../core/db/repo-factory'
import { transaction } from '../../core/db/transaction'
@@ -129,7 +130,11 @@ async function resolveChatProvider({ platformId, scope, log }: { platformId: str
return chatProvider
}
-async function assertRunProviderConfigured({ platformId, provider, providerConfigId, scope, log }: { platformId: string, provider?: AIProviderName | null, providerConfigId?: string | null, scope: ProviderScope, log: FastifyBaseLogger }): Promise<void> {
+async function assertRunProviderConfigured({ platformId, provider, providerConfigId, modelTierId, scope, log }: { platformId: string, provider?: AIProviderName | null, providerConfigId?: string | null, modelTierId?: string | null, scope: ProviderScope, log: FastifyBaseLogger }): Promise<void> {
+ if (!isNil(modelTierId)) {
+ await aiModelCandidates(log).firstCandidate({ platformId, tierId: modelTierId })
+ return
+ }
if (isNil(provider)) {
const chatProvider = await aiProviderService(log).getChatProviderName({ platformId, scope })
if (isNil(chatProvider)) {
@@ -178,14 +183,16 @@ async function resolveModelId({ platformId, providerConfig, selectedModel, surfa
return picked.id
}
-async function resolveImageModelId({ platformId, providerConfig, scope, log }: { platformId: string, providerConfig: GetProviderConfigResponse, scope: ProviderScope, log: FastifyBaseLogger }): Promise<string | undefined> {
+async function resolveImageModelId({ platformId, providerConfig, scope, grantedByTier = false, log }: { platformId: string, providerConfig: GetProviderConfigResponse, scope: ProviderScope, grantedByTier?: boolean, log: FastifyBaseLogger }): Promise<string | undefined> {
const { provider, configId } = providerConfig
const preferred = AI_PROVIDER_CAPABILITIES[provider].defaultImageModel
if (isNil(preferred) || provider === AIProviderName.ACTIVEPIECES) {
return preferred
}
const listed = await Promise.race([
- tryCatch(() => aiProviderService(log).listModels({ platformId, provider, scope, configId })),
+ tryCatch(() => grantedByTier
+ ? aiProviderService(log).listModelsForConfig({ platformId, configId })
+ : aiProviderService(log).listModels({ platformId, provider, scope, configId })),
delay(IMAGE_MODEL_LOOKUP_TIMEOUT_MS, null),
])
if (isNil(listed) || isNil(listed.data)) {
@@ -336,6 +343,7 @@ function jobFieldsFromConfig({ config }: { config: AgentConfig }): AgentJobConfi
modelName: config.modelName ?? null,
...spreadIfDefined('provider', config.provider ?? undefined),
...spreadIfDefined('providerConfigId', config.providerConfigId ?? undefined),
+ ...spreadIfDefined('modelTierId', config.modelTierId ?? undefined),
promptOverride: { system: config.instructions },
}
}
@@ -458,6 +466,6 @@ export const agentHelpers = {
saveUserMemory,
}
-type AgentJobConfigFields = Pick<ExecuteAgentRunJobData, 'tools' | 'structuredOutput' | 'maxSteps' | 'modelName' | 'provider' | 'providerConfigId' | 'promptOverride'>
+type AgentJobConfigFields = Pick<ExecuteAgentRunJobData, 'tools' | 'structuredOutput' | 'maxSteps' | 'modelName' | 'provider' | 'providerConfigId' | 'modelTierId' | 'promptOverride'>
export type StreamingLockResult = 'acquired' | 'busy' | 'superseded'
diff --git a/packages/server/api/src/app/ee/agent/agent-model-tier.ts b/packages/server/api/src/app/ee/agent/agent-model-tier.ts
new file mode 100644
index 00000000000..4a0f905ce35
--- /dev/null
+++ b/packages/server/api/src/app/ee/agent/agent-model-tier.ts
@@ -0,0 +1,75 @@
+import { isNil, PlatformId, tryCatch } from '@activepieces/core-utils'
+import { ModelTierSurface } from '@activepieces/server-utils'
+import { AgentConversation, AgentModelCandidate } from '@activepieces/shared'
+import { FastifyBaseLogger } from 'fastify'
+import { aiModelCandidates, FirstCandidate, TierConfigCandidate, TierConfigs } from '../../ai/ai-model-candidates'
+import { agentModelResolution } from './agent-model-resolution'
+
+export const agentModelTier = (log: FastifyBaseLogger) => ({
+ async resolveRun({ platformId, tierId, surface }: { platformId: PlatformId, tierId: string, surface: ModelTierSurface }): Promise<AgentTierRun> {
+ const [run, fast] = await Promise.all([
+ aiModelCandidates(log).resolveConfigs({ platformId, tierId }),
+ aiModelCandidates(log).resolveFastConfig({ platformId }),
+ ])
+ const consoleBudget = agentModelResolution.resolveTier({ tierId: null, surface }).thinkingBudget
+ const candidates = withBudget({ configs: run, consoleBudget })
+ const fastIsThisTier = isNil(fast) || fast.tier.id === run.tier.id
+ return {
+ tier: { id: run.tier.id, name: run.tier.name },
+ candidates,
+ fast: fastIsThisTier ? null : withBudget({ configs: fast, consoleBudget })[0],
+ }
+ },
+
+ async mainModelOf({ conversation }: { conversation: Pick<AgentConversation, 'platformId' | 'modelTierId'> }): Promise<FirstCandidate | null> {
+ const tierId = conversation.modelTierId
+ if (isNil(tierId)) {
+ return null
+ }
+ const { data } = await tryCatch(() => aiModelCandidates(log).firstCandidates({ platformId: conversation.platformId, tierIds: [tierId] }))
+ return data?.get(tierId) ?? null
+ },
+
+ async mainModelsOf({ conversations }: { conversations: Pick<AgentConversation, 'platformId' | 'modelTierId'>[] }): Promise<Map<string, FirstCandidate | null>> {
+ const tierIdsByPlatform = new Map<string, string[]>()
+ conversations.forEach(({ platformId, modelTierId }) => {
+ if (!isNil(modelTierId)) {
+ tierIdsByPlatform.set(platformId, [...(tierIdsByPlatform.get(platformId) ?? []), modelTierId])
+ }
+ })
+ const perPlatform = await Promise.all([...tierIdsByPlatform].map(async ([platformId, tierIds]): Promise<[string, FirstCandidate | null][]> => {
+ const { data, error } = await tryCatch(() => aiModelCandidates(log).firstCandidates({ platformId, tierIds }))
+ if (!isNil(error) || isNil(data)) {
+ log.warn({ error, platform: { id: platformId }, tierCount: tierIds.length }, '[agentModelTier] Could not resolve the tiers of a platform in one pass, each conversation will look its tier up on its own')
+ return []
+ }
+ return tierIds.map((tierId) => [tierCacheKey({ platformId, tierId }), data.get(tierId) ?? null])
+ }))
+ return new Map(perPlatform.flat())
+ },
+})
+
+function tierCacheKey({ platformId, tierId }: { platformId: string, tierId: string }): string {
+ return `${platformId}:${tierId}`
+}
+
+function withBudget({ configs, consoleBudget }: { configs: TierConfigs, consoleBudget: number }): AgentTierCandidate[] {
+ const thinkingBudget = configs.tier.thinkingBudget ?? consoleBudget
+ return configs.candidates.map((candidate) => ({ ...candidate, thinkingBudget }))
+}
+
+function toWorkerCandidate({ config, modelId, status, thinkingBudget }: AgentTierCandidate): AgentModelCandidate {
+ return { ...config, providerConfigId: config.configId, modelId, status, thinkingBudget }
+}
+
+export const agentTierCandidates = { toWorkerCandidate, tierCacheKey }
+
+export type AgentTierCandidate = TierConfigCandidate & {
+ thinkingBudget: number
+}
+
+export type AgentTierRun = {
+ tier: { id: string, name: string }
+ candidates: AgentTierCandidate[]
+ fast: AgentTierCandidate | null
+}
diff --git a/packages/server/api/src/app/ee/agent/agent-run-controller.ts b/packages/server/api/src/app/ee/agent/agent-run-controller.ts
index 59032293318..45750ddd225 100644
--- a/packages/server/api/src/app/ee/agent/agent-run-controller.ts
+++ b/packages/server/api/src/app/ee/agent/agent-run-controller.ts
@@ -49,6 +49,7 @@ export const agentRunController: FastifyPluginAsyncZod = async (app) => {
modelName: request.body.modelName ?? null,
...spreadIfDefined('provider', request.body.provider),
...spreadIfDefined('providerConfigId', providerConfigId),
+ ...spreadIfDefined('modelTierId', request.body.modelTierId),
}
: agentHelpers.jobFieldsFromConfig({ config: linked.config })
const { tools, structuredOutput, provider } = runFields
@@ -71,7 +72,7 @@ export const agentRunController: FastifyPluginAsyncZod = async (app) => {
})
}
const flowTools = await agentHelpers.resolveFlowTools({ projectId, tools: tools ?? [], log: request.log })
- await agentHelpers.assertRunProviderConfigured({ platformId: platform.id, provider, providerConfigId: runFields.providerConfigId, scope: agentHelpers.runScopeOrThrow({ projectId }), log: request.log })
+ await agentHelpers.assertRunProviderConfigured({ platformId: platform.id, provider, providerConfigId: runFields.providerConfigId, modelTierId: runFields.modelTierId, scope: agentHelpers.runScopeOrThrow({ projectId }), log: request.log })
await assertCreditsAndAppSumoNotExceeded({ platformId: platform.id, log: request.log })
const { ownerId } = await projectService(request.log).getOneOrThrow(projectId)
@@ -184,6 +185,7 @@ const StartAgentRunRequest = z.object({
modelName: z.string().optional(),
provider: z.enum(AIProviderName).optional(),
providerConfigId: z.string().optional(),
+ modelTierId: z.optional(ApId),
})
const StartAgentRunResponse = z.object({
diff --git a/packages/server/api/src/app/ee/agent/agent-service.ts b/packages/server/api/src/app/ee/agent/agent-service.ts
index da6769b20ed..8a3170ea5ea 100644
--- a/packages/server/api/src/app/ee/agent/agent-service.ts
+++ b/packages/server/api/src/app/ee/agent/agent-service.ts
@@ -4,6 +4,7 @@ import { ActivepiecesError, AIProviderName, apId, ApId, connectionTemplate, Curs
import { Agent, AgentConfig, AgentFlowTool, AgentKnowledgeBaseTool, AgentListSort, AgentMoveLoss, AgentMoveLossKind, AgentMovePreview, AgentRunSource, AgentSummary, agentUtils, AgentVisibility, CreateAgentRequest, DefaultProjectRole, Project, ProjectType, UpdateAgentRequest } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { Brackets, EntityManager, In, SelectQueryBuilder } from 'typeorm'
+import { platformModelTierService } from '../../ai/platform-model-tier-service'
import { appConnectionService } from '../../app-connection/app-connection-service/app-connection-service'
import { repoFactory } from '../../core/db/repo-factory'
import { transaction } from '../../core/db/transaction'
@@ -35,7 +36,7 @@ const AGENT_MOVED_AWAY = 'That agent has just been moved somewhere else. Reload
export const agentService = (log: FastifyBaseLogger) => ({
async create({ platformId, projectId, ownerId, request }: CreateParams): Promise<Agent> {
const visibility = request.visibility ?? AgentVisibility.PROJECT
- const draft = await withDefaultModel({ draft: request.draft, platformId, projectId, log })
+ const draft = await withDefaultModel({ draft: await withTierOrModel({ draft: request.draft, platformId }), platformId, projectId, log })
return agentRepo().save({
id: apId(),
projectId,
@@ -119,7 +120,7 @@ export const agentService = (log: FastifyBaseLogger) => ({
projectId,
log,
})
- const draft = isNil(request.draft) ? agent.draft : sanitizeObjectForPostgresql(request.draft)
+ const draft = isNil(request.draft) ? agent.draft : sanitizeObjectForPostgresql(await withTierOrModel({ draft: request.draft, platformId }))
await getFolderIdFromRequest({ projectId, folderId: request.folderId ?? undefined, folderName: undefined, log })
const goingLive = goLive && agentUtils.isPublishable(draft)
const published = goingLive ? draft : agent.published
@@ -422,13 +423,21 @@ async function isProjectAdministrator({ projectId, userId, log }: { projectId: P
return role?.name === DefaultProjectRole.ADMIN
}
+async function withTierOrModel({ draft, platformId }: { draft: AgentConfig, platformId: PlatformId }): Promise<AgentConfig> {
+ if (isNil(draft.modelTierId)) {
+ return draft
+ }
+ await platformModelTierService.getForRun({ platformId, id: draft.modelTierId })
+ return { ...draft, provider: null, providerConfigId: null, modelName: null }
+}
+
async function withDefaultModel({ draft, platformId, projectId, log }: {
draft: AgentConfig
platformId: PlatformId
projectId: ProjectId
log: FastifyBaseLogger
}): Promise<AgentConfig> {
- if (!isNil(draft.modelName)) {
+ if (!isNil(draft.modelName) || !isNil(draft.modelTierId)) {
return draft
}
const provider = await agentHelpers.resolveChatProviderName({ platformId, projectId, log })
diff --git a/packages/server/api/src/app/ee/agent/chat-analytics-sync.ts b/packages/server/api/src/app/ee/agent/chat-analytics-sync.ts
index 2d5b8646d36..04a5c8cb3f1 100644
--- a/packages/server/api/src/app/ee/agent/chat-analytics-sync.ts
+++ b/packages/server/api/src/app/ee/agent/chat-analytics-sync.ts
@@ -1,6 +1,7 @@
import { AIProviderName, chunk, isNil, tryCatch } from '@activepieces/core-utils'
import { AgentConversation, AgentRunSource, ApEdition, PersistedAgentMessage, PersistedAgentPartType, PersistedToolCallStatus } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
+import { FirstCandidate } from '../../ai/ai-model-candidates'
import { isNotOneOfTheseEditions } from '../../database/database-common'
import { rejectedPromiseHandler } from '../../helper/promise-handler'
import { system } from '../../helper/system/system'
@@ -9,6 +10,7 @@ import { platformService } from '../../platform/platform.service'
import { userService } from '../../user/user-service'
import { platformPlanRepo } from '../platform/platform-plan/platform-plan.service'
import { agentHelpers } from './agent-helpers'
+import { agentModelTier, agentTierCandidates } from './agent-model-tier'
import { agentHistory } from './history/agent-history'
const CONSOLE_TELEMETRY_URL = 'https://console.activepieces.com/api/chat-analytics/external/sync'
@@ -49,13 +51,13 @@ async function syncConversations({ conversations, log }: {
const platformIds = [...new Set(conversations.map((c) => c.platformId))]
const licenseKeyByPlatform = await resolveLicenseKeysByPlatform({ platformIds })
- const { userCache, platformCache, providerCache } = await resolveLookups({ conversations, log })
+ const { userCache, platformCache, providerCache, tierCache } = await resolveLookups({ conversations, log })
let pushed = 0
let skipped = 0
let failed = 0
for (const batch of chunk(conversations, BATCH_SIZE)) {
- const result = await pushBatch({ conversations: batch, licenseKeyByPlatform, log, userCache, platformCache, providerCache })
+ const result = await pushBatch({ conversations: batch, licenseKeyByPlatform, log, userCache, platformCache, providerCache, tierCache })
pushed += result.pushed
skipped += result.skipped
if (!result.success) {
@@ -100,29 +102,32 @@ async function resolveLookups({ conversations, log }: {
const uniquePlatformIds = [...new Set(conversations.map((c) => c.platformId))]
const uniqueScopes = [...new Map(conversations.map((c) => [chatProviderCacheKey(c), c])).values()]
- const [userEntries, platformNameEntries, providerEntries] = await Promise.all([
+ const [userEntries, platformNameEntries, providerEntries, tierCache] = await Promise.all([
Promise.all(uniqueUserIds.map(async (userId): Promise<[string, string | null]> => [userId, await resolveUserEmail({ userId, log })])),
Promise.all(uniquePlatformIds.map(async (platformId): Promise<[string, string | null]> => [platformId, await resolvePlatformName({ platformId, log })])),
Promise.all(uniqueScopes.map(async (conversation): Promise<[string, AIProviderName | null]> => [
chatProviderCacheKey(conversation),
await resolveProviderName({ platformId: conversation.platformId, projectId: conversation.projectId ?? null, log }),
])),
+ agentModelTier(log).mainModelsOf({ conversations }),
])
return {
userCache: new Map(userEntries),
platformCache: new Map(platformNameEntries),
providerCache: new Map(providerEntries),
+ tierCache,
}
}
-async function pushBatch({ conversations, licenseKeyByPlatform, log, userCache, platformCache, providerCache }: {
+async function pushBatch({ conversations, licenseKeyByPlatform, log, userCache, platformCache, providerCache, tierCache }: {
conversations: AgentConversation[]
licenseKeyByPlatform: Map<string, string>
log: FastifyBaseLogger
userCache?: Map<string, string | null>
platformCache?: Map<string, string | null>
providerCache?: Map<string, AIProviderName | null>
+ tierCache?: Map<string, FirstCandidate | null>
}): Promise<{ pushed: number, skipped: number, success: boolean }> {
const secret = system.get(AppSystemProp.CONSOLE_API_SECRET_KEY)
if (isNil(secret)) {
@@ -132,7 +137,7 @@ async function pushBatch({ conversations, licenseKeyByPlatform, log, userCache,
const payloads: Record<string, unknown>[] = []
for (const conversation of conversations) {
const licenseKey = licenseKeyByPlatform.get(conversation.platformId) ?? null
- const payloadResult = await tryCatch(() => toSyncPayload({ conversation, licenseKey, log, userCache, platformCache, providerCache }))
+ const payloadResult = await tryCatch(() => toSyncPayload({ conversation, licenseKey, log, userCache, platformCache, providerCache, tierCache }))
if (payloadResult.error) {
log.error({ conversation: { id: conversation.id }, error: String(payloadResult.error) }, 'Failed to build sync payload for conversation, skipping')
continue
@@ -168,17 +173,20 @@ async function pushBatch({ conversations, licenseKeyByPlatform, log, userCache,
return { pushed: payloads.length, skipped, success: true }
}
-async function toSyncPayload({ conversation, licenseKey, log, userCache, platformCache, providerCache }: {
+async function toSyncPayload({ conversation, licenseKey, log, userCache, platformCache, providerCache, tierCache }: {
conversation: AgentConversation
licenseKey: string | null
log: FastifyBaseLogger
userCache?: Map<string, string | null>
platformCache?: Map<string, string | null>
providerCache?: Map<string, AIProviderName | null>
+ tierCache?: Map<string, FirstCandidate | null>
}): Promise<Record<string, unknown>> {
const userEmail = userCache?.get(conversation.userId) ?? await resolveUserEmail({ userId: conversation.userId, log })
const platformName = platformCache?.get(conversation.platformId) ?? await resolvePlatformName({ platformId: conversation.platformId, log })
- const provider = providerCache?.get(chatProviderCacheKey(conversation)) ?? await resolveProviderName({ platformId: conversation.platformId, projectId: conversation.projectId ?? null, log })
+ const tierKey = agentTierCandidates.tierCacheKey({ platformId: conversation.platformId, tierId: conversation.modelTierId ?? '' })
+ const tierModel = tierCache?.has(tierKey) ? tierCache.get(tierKey) ?? null : await agentModelTier(log).mainModelOf({ conversation })
+ const provider = tierModel?.provider ?? providerCache?.get(chatProviderCacheKey(conversation)) ?? await resolveProviderName({ platformId: conversation.platformId, projectId: conversation.projectId ?? null, log })
const messages = agentHistory.resolveMessages({ conversation, log })
@@ -190,7 +198,7 @@ async function toSyncPayload({ conversation, licenseKey, log, userCache, platfor
userId: conversation.userId,
userEmail,
title: conversation.title,
- modelName: agentHelpers.resolveModelIdForAnalytics({ selectedModel: conversation.modelName ?? null, provider, surface: 'chat' }),
+ modelName: tierModel?.modelId ?? agentHelpers.resolveModelIdForAnalytics({ selectedModel: conversation.modelName ?? null, provider, surface: 'chat' }),
provider,
messages,
messageCount: messages.length,
@@ -245,4 +253,5 @@ type ConversationLookups = {
userCache: Map<string, string | null>
platformCache: Map<string, string | null>
providerCache: Map<string, AIProviderName | null>
+ tierCache: Map<string, FirstCandidate | null>
}
diff --git a/packages/server/api/src/app/ee/agent/chat-tool-billing.ts b/packages/server/api/src/app/ee/agent/chat-tool-billing.ts
index 9cfe562c477..97a0162af2e 100644
--- a/packages/server/api/src/app/ee/agent/chat-tool-billing.ts
+++ b/packages/server/api/src/app/ee/agent/chat-tool-billing.ts
@@ -1,11 +1,12 @@
import { isNil, spreadIfDefined } from '@activepieces/core-utils'
-import { AgentConversation, chatBilling, ChatToolCall, isAppSumoCreditedPlan, PersistedAgentMessage, PersistedAgentPartType, PersistedAgentRole, PersistedToolCallStatus } from '@activepieces/shared'
+import { AgentConversation, AgentTurnModel, chatBilling, ChatToolCall, isAppSumoCreditedPlan, PersistedAgentMessage, PersistedAgentPartType, PersistedAgentRole, PersistedToolCallStatus } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { LicenseKeyPostHogEvents } from '../../helper/telemetry.utils'
import { trackBillingAndSendTelemetry } from '../../platform/billing-and-telemetry'
import { CreditUsageSource } from '../../platform/billing-provider'
import { platformPlanService } from '../platform/platform-plan/platform-plan.service'
import { agentHelpers } from './agent-helpers'
+import { agentModelTier } from './agent-model-tier'
import { agentHistory } from './history/agent-history'
function latestTurnToolCalls({ messages }: { messages: PersistedAgentMessage[] }): ChatToolCall[] {
@@ -22,17 +23,18 @@ function countBillableToolCallsInLatestTurn({ messages }: { messages: PersistedA
return latestTurnToolCalls({ messages }).filter(chatBilling.isFlatBilledToolCall).length
}
-async function chargeForLatestTurn({ conversation, runId, log }: ChargeForLatestTurnParams): Promise<void> {
+async function chargeForLatestTurn({ conversation, runId, answeredBy, log }: ChargeForLatestTurnParams): Promise<void> {
const messages = agentHistory.resolveMessages({ conversation, log })
const turnIndex = messages.filter((message) => message.role === PersistedAgentRole.USER).length
const idempotencyScope = runId ?? turnIndex
- const provider = await agentHelpers.resolveChatProviderName({
+ const tierModel = isNil(conversation.modelTierId) ? null : answeredBy ?? await agentModelTier(log).mainModelOf({ conversation })
+ const provider = tierModel?.provider ?? await agentHelpers.resolveChatProviderName({
platformId: conversation.platformId,
projectId: conversation.projectId ?? null,
log,
})
const surface = agentHelpers.surfaceOf({ source: conversation.source })
- const model = agentHelpers.resolveModelIdForAnalytics({ selectedModel: conversation.modelName ?? null, provider, surface })
+ const model = tierModel?.modelId ?? agentHelpers.resolveModelIdForAnalytics({ selectedModel: conversation.modelName ?? null, provider, surface })
const tier = agentHelpers.resolveTier({ tierId: conversation.modelName ?? null, surface })
const platformPlan = await platformPlanService(log).getOrCreateForPlatform(conversation.platformId)
@@ -93,5 +95,6 @@ const PROJECTLESS_CHAT = 'chat'
type ChargeForLatestTurnParams = {
conversation: AgentConversation
runId?: string
+ answeredBy?: AgentTurnModel
log: FastifyBaseLogger
}
diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
index cbc27a07ecf..3322a20bdb2 100644
--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
@@ -1,12 +1,13 @@
-import { ActivepiecesError, ErrorCode, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'
-import { agentAiUtils, aiUtils } from '@activepieces/server-utils'
-import { AgentConfigResponse, AgentRunSource, GetAgentConfigRequest, GetEnabledAiToolsResponse, PersistedAgentMessage, PersistedAgentPartType, PersistedAgentRole } from '@activepieces/shared'
-import { ModelMessage } from 'ai'
+import { ActivepiecesError, AIProviderName, ErrorCode, isFallbackWorthy, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'
+import { agentAiUtils, aiProviderSignal, aiUtils } from '@activepieces/server-utils'
+import { AgentConfigResponse, AgentConversation, AgentRunSource, aiProviderUtils, GetAgentConfigRequest, GetEnabledAiToolsResponse, GetProviderConfigResponse, PersistedAgentMessage, PersistedAgentPartType, PersistedAgentRole } from '@activepieces/shared'
+import { LanguageModel, ModelMessage } from 'ai'
import { FastifyBaseLogger } from 'fastify'
import { agentApprovalGate } from '.././agent-approval-gate'
import { agentCompaction } from '.././agent-compaction'
import { buildAttachmentNote, buildUserContentWithFiles, persistAgentAttachments } from '.././agent-file-utils'
import { agentHelpers } from '.././agent-helpers'
+import { agentModelTier, agentTierCandidates, AgentTierRun } from '.././agent-model-tier'
import { agentMcp } from '.././mcp/agent-mcp'
import { chatPersonalizationService } from '.././personalization/chat-personalization-service'
import { agentPrompt } from '.././prompt/agent-prompt'
@@ -86,7 +87,12 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
// selection above, which narrows to what the owner can still see in chat.
const runProjectId = isFlowStep ? conversation.projectId ?? null : selectedProjectId
const runScope = agentHelpers.runScopeOrThrow({ projectId: runProjectId })
- const providerConfig = await agentHelpers.resolveRunProvider({ platformId, log, scope: runScope, ...spreadIfDefined('provider', input.provider), ...spreadIfDefined('providerConfigId', input.providerConfigId) })
+ const surface = agentHelpers.surfaceOf({ source: requestedSource })
+ const modelTierId = tierIdForRun({ conversation, requestedTierId: input.modelTierId ?? null })
+ const tierRun = isNil(modelTierId) ? null : await agentModelTier(log).resolveRun({ platformId, tierId: modelTierId, surface })
+ const providerConfig = isNil(tierRun)
+ ? await agentHelpers.resolveRunProvider({ platformId, log, scope: runScope, ...spreadIfDefined('provider', input.provider), ...spreadIfDefined('providerConfigId', input.providerConfigId) })
+ : tierRun.candidates[0].config
const attachmentRefs = files && files.length > 0 && !isNil(selectedProjectId)
? await persistAgentAttachments({ files, projectId: selectedProjectId, platformId, conversationId, log })
@@ -94,7 +100,6 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
const userContent = await buildUserContentWithFiles({ text: userMessage, files, attachmentNote: buildAttachmentNote(attachmentRefs) })
const aiTools: GetEnabledAiToolsResponse = dryRun ? {} : enabledAiTools
- const surface = agentHelpers.surfaceOf({ source: requestedSource })
const chosen = dryRun
? { search: null, image: null }
: await chosenProviders.resolveForRun({ platformId, choices: providerChoices, surface, scope: runScope, log })
@@ -108,7 +113,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
const generatesImagesOnProvider = actingRun && isNil(aiTools.imageGeneration)
const imageModelId = !generatesImagesOnProvider
? undefined
- : chosen.image?.modelId ?? await agentHelpers.resolveImageModelId({ platformId, providerConfig, scope: runScope, log })
+ : chosen.image?.modelId ?? await agentHelpers.resolveImageModelId({ platformId, providerConfig, scope: runScope, grantedByTier: !isNil(tierRun), log })
const lock = await agentHelpers.acquireStreamingLock({ conversationId, ...spreadIfDefined('runId', input.runId) })
if (lock === 'superseded') {
@@ -129,14 +134,23 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
if (!dryRun && isNil(validCandidateProjectId) && !isNil(selectedProjectId)) {
await agentHelpers.conversationRepo().update(conversationId, { projectId: selectedProjectId })
}
+ if (!dryRun && (conversation.modelTierId ?? null) !== modelTierId) {
+ await agentHelpers.conversationRepo().update(conversationId, { modelTierId })
+ }
const selectedModel = modelName ?? conversation.modelName ?? null
const namesItsOwnModel = surface === 'flow'
- const tier = agentHelpers.resolveRunTier({ provider: providerConfig.provider, modelName: modelName ?? null, selectedModel, surface })
- const resolvedModelId = namesItsOwnModel && !isNil(modelName)
- ? agentHelpers.resolveNamedModelId({ provider: providerConfig.provider, modelName, surface, modelScope: providerConfig.modelScope, modelIds: providerConfig.modelIds, log })
- : await agentHelpers.resolveModelId({ platformId, providerConfig, selectedModel, surface, scope: runScope, log })
- const fastModelId = await agentHelpers.resolveFastModelId({ platformId, providerConfig, surface, scope: runScope, fallbackModelId: resolvedModelId, log })
+ const tier = isNil(tierRun)
+ ? agentHelpers.resolveRunTier({ provider: providerConfig.provider, modelName: modelName ?? null, selectedModel, surface })
+ : { ...agentHelpers.resolveRunTier({ provider: providerConfig.provider, modelName: null, selectedModel: null, surface }), thinkingBudget: tierRun.candidates[0].thinkingBudget, modelId: tierRun.candidates[0].modelId }
+ const resolvedModelId = !isNil(tierRun)
+ ? tierRun.candidates[0].modelId
+ : namesItsOwnModel && !isNil(modelName)
+ ? agentHelpers.resolveNamedModelId({ provider: providerConfig.provider, modelName, surface, modelScope: providerConfig.modelScope, modelIds: providerConfig.modelIds, log })
+ : await agentHelpers.resolveModelId({ platformId, providerConfig, selectedModel, surface, scope: runScope, log })
+ const fastModelId = isNil(tierRun)
+ ? await agentHelpers.resolveFastModelId({ platformId, providerConfig, surface, scope: runScope, fallbackModelId: resolvedModelId, log })
+ : resolvedModelId
// Inject an inventory of the project's existing connections into context so the agent
// never has to *guess* an app name to find out what's connected. Without this, discovery
@@ -204,27 +218,35 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
})
await agentApprovalGate.clearCancel({ conversationId })
- const reservedOutputTokens = await agentAiUtils.affordableOutputTokens({ provider: providerConfig.provider, modelIds: [resolvedModelId, fastModelId], thinkingBudget: tier.thinkingBudget })
+ const compactionModels = isNil(tierRun)
+ ? [{ config: providerConfig, modelId: resolvedModelId, thinkingBudget: tier.thinkingBudget }]
+ : tierRun.candidates
+ const everyModelOfTheTurn = isNil(tierRun) ? compactionModels : [...tierRun.candidates, ...(isNil(tierRun.fast) ? [] : [tierRun.fast])]
+ const contextProvider = smallestContextProvider({ configs: everyModelOfTheTurn.map((candidate) => candidate.config) })
+ const reservedOutputTokens = isNil(tierRun)
+ ? await agentAiUtils.affordableOutputTokens({ provider: providerConfig.provider, modelIds: [resolvedModelId, fastModelId], thinkingBudget: tier.thinkingBudget })
+ : Math.max(...await Promise.all(everyModelOfTheTurn.map((candidate) => agentAiUtils.affordableOutputTokens({ provider: candidate.config.provider, modelIds: [candidate.modelId], thinkingBudget: candidate.thinkingBudget }))))
const reservedTokens = reservedOutputTokens + TOOL_SCHEMA_TOKEN_ESTIMATE
const payloadReservedTokens = reservedTokens + agentAiUtils.estimateTokenCount({ messages: [], systemPromptLength: systemPromptText.length })
const estimatedTokens = agentCompaction.estimateTokenCount({ messages: llmHistory, systemPromptLength: systemPromptText.length })
let compactionState = { summary: conversation.summary ?? null, summarizedUpToIndex: conversation.summarizedUpToIndex ?? null }
- const willCompact = agentCompaction.shouldCompact({ estimatedTokens, provider: providerConfig.provider, messageCount: llmHistory.length, reservedTokens })
+ const willCompact = agentCompaction.shouldCompact({ estimatedTokens, provider: contextProvider, messageCount: llmHistory.length, reservedTokens })
log.debug({ estimatedTokens, willCompact, messageCount: llmHistory.length, systemPromptLength: systemPromptText.length }, '[agentRpc#getAgentConfig] Compaction decision')
if (willCompact) {
- const model = aiUtils.createModel({
- credentials: providerConfig,
- modelId: resolvedModelId,
- })
- compactionState = await agentCompaction.compactMessages({
- messages: llmHistory,
- existingSummary: compactionState.summary,
- summarizedUpToIndex: compactionState.summarizedUpToIndex,
- provider: providerConfig.provider,
- reservedTokens: payloadReservedTokens,
- model,
+ const previous = compactionState
+ compactionState = await onFirstModelThatAnswers({
+ models: compactionModels,
log,
+ run: (model) => agentCompaction.compactMessages({
+ messages: llmHistory,
+ existingSummary: previous.summary,
+ summarizedUpToIndex: previous.summarizedUpToIndex,
+ provider: contextProvider,
+ reservedTokens: payloadReservedTokens,
+ model,
+ log,
+ }),
})
await agentHelpers.conversationRepo().update(conversationId, {
summary: compactionState.summary,
@@ -237,7 +259,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
messages: llmHistory,
summary: compactionState.summary,
summarizedUpToIndex: compactionState.summarizedUpToIndex,
- provider: providerConfig.provider,
+ provider: contextProvider,
reservedTokens: payloadReservedTokens,
})
@@ -247,6 +269,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
model: { id: resolvedModelId },
provider: providerConfig.provider,
tier: { id: tier.id, surface },
+ ...spreadIfDefined('platformTier', tierRun?.tier),
project: selectedProjectId ? { id: selectedProjectId } : undefined,
webSearchAvailable,
}, '[agentRpc#getAgentConfig] Chat config resolved')
@@ -276,9 +299,45 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
agentsAvailable,
userEmail: runUserEmail,
source: conversation.source,
+ ...tierFieldsOf({ tierRun }),
}
},
})
+function tierIdForRun({ conversation, requestedTierId }: { conversation: AgentConversation, requestedTierId: string | null }): string | null {
+ const choiceLivesOnConversation = conversation.source === AgentRunSource.CHAT || conversation.source === AgentRunSource.AGENT_BUILDER
+ return choiceLivesOnConversation ? conversation.modelTierId ?? null : requestedTierId
+}
+
+function smallestContextProvider({ configs }: { configs: GetProviderConfigResponse[] }): AIProviderName {
+ return configs.reduce((smallest, config) => aiProviderUtils.getMaxContextTokens({ provider: config.provider }) < aiProviderUtils.getMaxContextTokens({ provider: smallest.provider }) ? config : smallest).provider
+}
+
+async function onFirstModelThatAnswers<T>({ models, run, log }: { models: { config: GetProviderConfigResponse, modelId: string }[], run: (model: LanguageModel) => Promise<T>, log: FastifyBaseLogger }): Promise<T> {
+ for (const [index, candidate] of models.entries()) {
+ const attempt = await tryCatch(() => run(aiUtils.createModel({ credentials: candidate.config, modelId: candidate.modelId })))
+ if (attempt.error === null) {
+ return attempt.data
+ }
+ const isLast = index === models.length - 1
+ if (isLast || !isFallbackWorthy(aiProviderSignal.fromError(attempt.error))) {
+ throw attempt.error
+ }
+ log.warn({ error: attempt.error, candidateIndex: index, model: { id: candidate.modelId } }, '[agentRpc#getAgentConfig] Compaction model failed, trying the next one in the tier')
+ }
+ throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: 'No model is left to compact this conversation' } })
+}
+
+function tierFieldsOf({ tierRun }: { tierRun: AgentTierRun | null }): Pick<AgentConfigResponse, 'platformTier' | 'candidates' | 'fastCandidate'> {
+ if (isNil(tierRun)) {
+ return {}
+ }
+ return {
+ platformTier: tierRun.tier,
+ candidates: tierRun.candidates.map(agentTierCandidates.toWorkerCandidate),
+ ...spreadIfDefined('fastCandidate', isNil(tierRun.fast) ? undefined : agentTierCandidates.toWorkerCandidate(tierRun.fast)),
+ }
+}
+
const TOOL_SCHEMA_TOKEN_ESTIMATE = 12_000
diff --git a/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts
index 4bbc696d458..4f6f162455a 100644
--- a/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts
@@ -75,7 +75,7 @@ export const conversationRpc = (log: FastifyBaseLogger) => ({
updates.messages = input.messages
updates.uiMessages = sanitizeObjectForPostgresql(input.uiMessages)
if (input.title) updates.title = input.title
- if (input.modelName) updates.modelName = input.modelName
+ if (input.modelName && isNil(stored?.modelTierId)) updates.modelName = input.modelName
}
const failureAppend = isNil(input.failure) ? null : failureAppendFor({ stored: stored?.uiMessages ?? [], failure: input.failure })
if (!isNil(failureAppend)) {
@@ -118,7 +118,7 @@ export const conversationRpc = (log: FastifyBaseLogger) => ({
const conversation = await agentHelpers.conversationRepo().findOneBy({ id: input.conversationId })
if (conversation) {
chatAnalyticsTelemetry(log).sendConversationUpdate({ conversation })
- rejectedPromiseHandler(chatToolBilling.chargeForLatestTurn({ conversation, runId: input.runId, log }), log)
+ rejectedPromiseHandler(chatToolBilling.chargeForLatestTurn({ conversation, runId: input.runId, log, ...spreadIfDefined('answeredBy', input.answeredBy) }), log)
}
}
},
@@ -162,7 +162,7 @@ export const conversationRpc = (log: FastifyBaseLogger) => ({
params: { message: 'A flow-step agent run cannot move to another project' },
})
}
- if (!isNil(conversation)) {
+ if (!isNil(conversation) && isNil(conversation.modelTierId)) {
await agentHelpers.assertProjectSwitchKeepsKey({
platformId: conversation.platformId,
fromProjectId: conversation.projectId ?? null,
diff --git a/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts b/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts
index b4bbe731efc..7081d0c2cb6 100644
--- a/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts
@@ -66,6 +66,7 @@ export async function configuredToolConversationOrThrow({ conversationId }: { co
platformId: conversation.platformId,
userId: conversation.userId,
source: conversation.source,
+ modelTierId: conversation.modelTierId ?? null,
...spreadIfDefined('agent', isNil(conversation.agentId) ? undefined : {
id: conversation.agentId,
...spreadIfDefined('displayName', conversation.agent?.displayName),
@@ -200,6 +201,7 @@ export type ConfiguredToolRun = {
platformId: string
userId: string
source: AgentRunSource
+ modelTierId: string | null
agent?: { id: string, displayName?: string }
}
diff --git a/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts
index d1bac09ef71..d6f293ed72d 100644
--- a/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts
@@ -2,24 +2,25 @@ import { isDeepStrictEqual } from 'node:util'
import { ActivepiecesAiConsumerSource, ActivepiecesError, ErrorCode, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'
import { aiUtils } from '@activepieces/server-utils'
import { AGENT_SELF_EDIT_TOOLS, AGENT_SURFACE_TOOLS, AgentActionOutcome, AgentRunSource, agentToolClassification, ExecuteAgentToolRequest, ExecuteAgentToolResponse, ExecuteFlowToolRequest, ExecuteFlowToolResponse, ExecuteKnowledgeBaseToolRequest, ExecuteKnowledgeBaseToolResponse, ExecutePieceToolRequest, ExecutePieceToolResponse, FlowActionType, flowStructureUtil } from '@activepieces/shared'
-import { embed } from 'ai'
+import { embed, LanguageModel } from 'ai'
import { FastifyBaseLogger } from 'fastify'
import { agentApprovalGate } from '.././agent-approval-gate'
import { agentHelpers } from '.././agent-helpers'
import { executeCrossProjectTool } from '.././tools/agent-tools'
import { pieceToolRunner } from '.././tools/piece-tool-runner'
+import { aiModelCandidates } from '../../../ai/ai-model-candidates'
import { flowService } from '../../../flows/flow/flow.service'
import { flowRunService } from '../../../flows/flow-run/flow-run-service'
import { knowledgeBaseService } from '../../../knowledge-base/knowledge-base.service'
import { extractMcpTriggerInput, resolveRunnableFlow, runFlowAsTool } from '../../../mcp/mcp-server-builder'
-import { byteLengthOf, CONFIGURED_TOOL_SOURCES, configuredToolConversationOrThrow, confinedRunFor, connectionForConfiguredTool, markTurnAsHavingRead, outcomeOfToolResult, pinConnectionToAgent, recordAgentAction, recordAgentFlowToolUse, turnHasRead } from './rpc-shared'
+import { byteLengthOf, CONFIGURED_TOOL_SOURCES, configuredToolConversationOrThrow, ConfiguredToolRun, confinedRunFor, connectionForConfiguredTool, markTurnAsHavingRead, outcomeOfToolResult, pinConnectionToAgent, recordAgentAction, recordAgentFlowToolUse, turnHasRead } from './rpc-shared'
export const toolExecutionRpc = (log: FastifyBaseLogger) => ({
async executePieceTool(input: ExecutePieceToolRequest): Promise<ExecutePieceToolResponse> {
const configuredRun = await configuredToolConversationOrThrow({ conversationId: input.conversationId })
const { projectId, platformId } = configuredRun
- const model = await agentHelpers.resolveFastModel({ platformId, surface: agentHelpers.surfaceOf({ source: configuredRun.source }), scope: { type: 'project', projectId }, log, ...spreadIfDefined('provider', input.provider), ...spreadIfDefined('providerConfigId', input.providerConfigId), fallbackModelId: input.modelId })
+ const model = await pieceToolModel({ configuredRun, input, log })
const piece = { pieceName: input.piece.pieceName, actionName: input.piece.actionName, ...spreadIfDefined('pieceVersion', input.piece.pieceVersion) }
const connection = await connectionForConfiguredTool({ piece: input.piece, projectId, platformId, log })
const { data: resolved, error: resolveError } = await tryCatch(() => pieceToolRunner.resolveInput({
@@ -321,6 +322,17 @@ const UNATTENDED_FORBIDDEN_TOOLS = ['ap_run_code', 'ap_execute_action', 'ap_expl
const KNOWLEDGE_BASE_SEARCH_LIMIT = 5
const KNOWLEDGE_BASE_SIMILARITY_THRESHOLD = 0.5
+async function pieceToolModel({ configuredRun, input, log }: { configuredRun: ConfiguredToolRun, input: ExecutePieceToolRequest, log: FastifyBaseLogger }): Promise<LanguageModel> {
+ const { projectId, platformId, modelTierId } = configuredRun
+ const granted = isNil(modelTierId) || isNil(input.providerConfigId)
+ ? null
+ : await aiModelCandidates(log).grantedEntryConfig({ platformId, tierId: modelTierId, configId: input.providerConfigId, modelId: input.modelId })
+ if (!isNil(granted)) {
+ return aiUtils.createModel({ credentials: granted, modelId: input.modelId, platformId, providerConfigId: granted.configId })
+ }
+ return agentHelpers.resolveFastModel({ platformId, surface: agentHelpers.surfaceOf({ source: configuredRun.source }), scope: { type: 'project', projectId }, log, ...spreadIfDefined('provider', input.provider), ...spreadIfDefined('providerConfigId', input.providerConfigId), fallbackModelId: input.modelId })
+}
+
async function flowOfRun({ flowRunId, projectId, log }: { flowRunId: string, projectId: string, log: FastifyBaseLogger }): Promise<{ id: string, runId: string } | undefined> {
const { data: flowRun, error } = await tryCatch(() => flowRunService(log).getOneOrThrow({ id: flowRunId, projectId }))
if (isNil(flowRun)) {
diff --git a/packages/server/api/test/integration/ce/ai-provider/ai-model-candidates.test.ts b/packages/server/api/test/integration/ce/ai-provider/ai-model-candidates.test.ts
index 395c62047ec..8b7cd1f57cc 100644
--- a/packages/server/api/test/integration/ce/ai-provider/ai-model-candidates.test.ts
+++ b/packages/server/api/test/integration/ce/ai-provider/ai-model-candidates.test.ts
@@ -3,6 +3,7 @@ import { AiStepAction, PlatformModelTier, PrincipalType } from '@activepieces/sh
import { FastifyInstance } from 'fastify'
import { StatusCodes } from 'http-status-codes'
import { vi } from 'vitest'
+import { aiModelCandidates } from '../../../../src/app/ai/ai-model-candidates'
import { aiRpcHandlers } from '../../../../src/app/ai/ai-rpc-handlers'
import { databaseConnection } from '../../../../src/app/database/database-connection'
import { generateMockToken } from '../../../helpers/auth'
@@ -92,6 +93,24 @@ describe('tier candidates', () => {
expect(candidates.map((candidate) => candidate.modelId)).toEqual(['new-model'])
})
+ it('resolves many tiers at once, following replacements and skipping removed or foreign ones', async () => {
+ const key = await seedKey({ testCtx: ctx })
+ const live = await createTier({ testCtx: ctx, name: 'Live', entries: [{ configId: key.id, modelId: 'live-model' }] })
+ const old = await createTier({ testCtx: ctx, name: 'Old', entries: [{ configId: key.id, modelId: 'old-model' }] })
+ const gone = await createTier({ testCtx: ctx, name: 'Gone', entries: [{ configId: key.id, modelId: 'gone-model' }] })
+ await ctx.delete(`${TIERS}/${old.id}`, { replacedBy: live.id })
+ await ctx.delete(`${TIERS}/${gone.id}`, { replacedBy: live.id })
+ const otherCtx = await createTestContext(app!)
+ const foreign = await createTier({ testCtx: otherCtx, name: 'Foreign', entries: [{ configId: (await seedKey({ testCtx: otherCtx })).id, modelId: 'foreign-model' }] })
+
+ const found = await aiModelCandidates(app!.log).firstCandidates({ platformId: ctx.platform.id, tierIds: [live.id, old.id, foreign.id, apId()] })
+
+ expect(found.get(live.id)?.modelId).toBe('live-model')
+ expect(found.get(old.id)?.modelId).toBe('live-model')
+ expect(found.get(foreign.id)).toBeNull()
+ expect(found.size).toBe(4)
+ })
+
it('fails a tier that was removed without a replacement', async () => {
const key = await seedKey({ testCtx: ctx })
const tier = await createTier({ testCtx: ctx, entries: [{ configId: key.id, modelId: 'gpt-4o' }] })
diff --git a/packages/server/api/test/integration/ce/ai-provider/platform-model-tier.test.ts b/packages/server/api/test/integration/ce/ai-provider/platform-model-tier.test.ts
index 77ecdb6883e..7c3a9bf6f4c 100644
--- a/packages/server/api/test/integration/ce/ai-provider/platform-model-tier.test.ts
+++ b/packages/server/api/test/integration/ce/ai-provider/platform-model-tier.test.ts
@@ -31,8 +31,8 @@ describe('Platform model tiers API', () => {
const first = await createTier({ testCtx: ctx, body: tierBody({ configId: key.id, name: 'Fast' }) })
const second = await createTier({ testCtx: ctx, body: tierBody({ configId: key.id, name: 'Expert' }) })
- expect(first).toMatchObject({ isDefault: true, position: 0, emoji: '⚡' })
- expect(second).toMatchObject({ isDefault: false, position: 1 })
+ expect(first).toMatchObject({ isDefault: true, isFast: true, position: 0, emoji: '⚡' })
+ expect(second).toMatchObject({ isDefault: false, isFast: false, position: 1 })
})
it('rejects bad entry counts and duplicate entries', async () => {
@@ -160,6 +160,19 @@ describe('Platform model tiers API', () => {
expect(tiers.find((tier) => tier.id === second.id)).toMatchObject({ isDefault: true, name: 'B2' })
})
+ it('moves the fast pointer on its own', async () => {
+ const key = await seedKey({ testCtx: ctx })
+ const first = await createTier({ testCtx: ctx, body: tierBody({ configId: key.id, name: 'A' }) })
+ const second = await createTier({ testCtx: ctx, body: tierBody({ configId: key.id, name: 'B' }) })
+
+ const response = await ctx.post(`${TIERS}/${second.id}`, { isFast: true })
+ const tiers = await listAdmin({ testCtx: ctx })
+
+ expect(response.statusCode).toBe(StatusCodes.OK)
+ expect(tiers.find((tier) => tier.id === first.id)).toMatchObject({ isDefault: true, isFast: false })
+ expect(tiers.find((tier) => tier.id === second.id)).toMatchObject({ isDefault: false, isFast: true })
+ })
+
it('reorders only with the exact live set', async () => {
const key = await seedKey({ testCtx: ctx })
const a = await createTier({ testCtx: ctx, body: tierBody({ configId: key.id, name: 'A' }) })
@@ -225,9 +238,9 @@ describe('Platform model tiers API', () => {
expect(first.statusCode).toBe(StatusCodes.NO_CONTENT)
expect(second.statusCode).toBe(StatusCodes.NO_CONTENT)
- expect(oldA).toMatchObject({ replacedBy: c.id, isDefault: false })
+ expect(oldA).toMatchObject({ replacedBy: c.id, isDefault: false, isFast: false })
expect(live.map((tier) => tier.id)).toEqual([c.id])
- expect(live[0]).toMatchObject({ isDefault: true })
+ expect(live[0]).toMatchObject({ isDefault: true, isFast: true })
})
it('refuses a bad replacement', async () => {
diff --git a/packages/server/api/test/integration/ee/agent/agent-tier-runtime.test.ts b/packages/server/api/test/integration/ee/agent/agent-tier-runtime.test.ts
new file mode 100644
index 00000000000..da6b824c8d6
--- /dev/null
+++ b/packages/server/api/test/integration/ee/agent/agent-tier-runtime.test.ts
@@ -0,0 +1,185 @@
+import { AIProviderName, apId, ErrorCode } from '@activepieces/core-utils'
+import { AgentConversation, AgentIcon, AgentRunSource, ColorName, WorkerJobType } from '@activepieces/shared'
+import { FastifyInstance } from 'fastify'
+import { StatusCodes } from 'http-status-codes'
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from 'vitest'
+import { aiModelCandidates } from '../../../../src/app/ai/ai-model-candidates'
+import { agentRpcHandlers } from '../../../../src/app/ee/agent/agent-rpc-handlers'
+import * as jobQueueModule from '../../../../src/app/workers/job-queue/job-queue'
+import { db } from '../../../helpers/db'
+import { createMockProject, mockAndSaveAIProvider } from '../../../helpers/mocks'
+import { createTestContext, TestContext } from '../../../helpers/test-context'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+
+let app: FastifyInstance
+
+const originalJobQueue = jobQueueModule.jobQueue
+
+beforeAll(async () => {
+ app = await setupTestEnvironment()
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+afterEach(() => {
+ vi.restoreAllMocks()
+})
+
+describe('a chat on a platform tier', () => {
+ it('runs on a tier key that is scoped away from the project, because the tier grants it', async () => {
+ const { ctx, tier, keyId } = await tierOnScopedAwayKey()
+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })
+
+ const config = await getConfig({ ctx, conversationId: conversation.id })
+
+ expect(config.candidates?.map((candidate) => candidate.providerConfigId)).toEqual([keyId])
+ expect(config.providerConfigId).toBe(keyId)
+ expect(config.platformTier).toEqual({ id: tier.id, name: 'Expert' })
+ })
+
+ it('cannot reach the same key as a specific model, because the key does not serve the project', async () => {
+ const { ctx, keyId } = await tierOnScopedAwayKey()
+ const conversation = await createConversation({ ctx, body: {} })
+
+ await expect(getConfig({ ctx, conversationId: conversation.id, provider: AIProviderName.OPENAI, providerConfigId: keyId })).rejects.toMatchObject({ error: { code: ErrorCode.ENTITY_NOT_FOUND } })
+ })
+
+ it('refuses a tier from another platform', async () => {
+ const { tier } = await tierOnScopedAwayKey()
+ const otherCtx = await context()
+
+ const response = await otherCtx.post(CONVERSATIONS_URL, { modelTierId: tier.id })
+
+ expect(response.statusCode).toBe(StatusCodes.NOT_FOUND)
+ })
+
+ it('keeps one model choice: a tier clears the model name and a model name clears the tier', async () => {
+ const { ctx, tier } = await tierOnScopedAwayKey()
+ const conversation = await createConversation({ ctx, body: { modelName: 'expert', modelTierId: tier.id } })
+
+ const pickedModel = await ctx.post(`${CONVERSATIONS_URL}/${conversation.id}`, { modelName: 'fast' })
+ const pickedTier = await ctx.post(`${CONVERSATIONS_URL}/${conversation.id}`, { modelTierId: tier.id })
+
+ expect(conversation).toMatchObject({ modelTierId: tier.id, modelName: null })
+ expect(pickedModel.json()).toMatchObject({ modelName: 'fast', modelTierId: null })
+ expect(pickedTier.json()).toMatchObject({ modelTierId: tier.id, modelName: null })
+ })
+
+ it('keeps the tier when the worker saves the turn with a model name', async () => {
+ const { ctx, tier } = await tierOnScopedAwayKey()
+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })
+
+ await agentRpcHandlers(app.log).saveAgentMessages({ conversationId: conversation.id, messages: [{ role: 'user', content: 'hi' }], uiMessages: [], modelName: 'smart' })
+
+ const stored = await db.findOneByOrFail<AgentConversation>('agent_conversation', { id: conversation.id })
+ expect(stored).toMatchObject({ modelTierId: tier.id, modelName: null })
+ })
+
+ it('moves to another project on a tier key that serves neither project', async () => {
+ const { ctx, tier, keyId } = await tierOnScopedAwayKey()
+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })
+ await db.update('agent_conversation', conversation.id, { projectId: ctx.project.id })
+ const target = createMockProject({ ownerId: ctx.user.id, platformId: ctx.platform.id })
+ await db.save('project', target)
+
+ await agentRpcHandlers(app.log).updateProjectContext({ conversationId: conversation.id, projectId: target.id, provider: AIProviderName.OPENAI, providerConfigId: keyId })
+
+ const stored = await db.findOneByOrFail<AgentConversation>('agent_conversation', { id: conversation.id })
+ expect(stored.projectId).toBe(target.id)
+ })
+})
+
+describe('the tier grant for tools', () => {
+ it('grants a model of the tier on its own key, and nothing else', async () => {
+ const { ctx, tier, keyId } = await tierOnScopedAwayKey()
+ const otherKey = await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })
+ const grant = (configId: string, modelId: string) => aiModelCandidates(app.log).grantedEntryConfig({ platformId: ctx.platform.id, tierId: tier.id, configId, modelId })
+
+ expect((await grant(keyId, TIER_MODEL))?.configId).toBe(keyId)
+ expect(await grant(keyId, 'another-model')).toBeNull()
+ expect(await grant(otherKey.id, TIER_MODEL)).toBeNull()
+ })
+})
+
+describe('an agent on a platform tier', () => {
+ it('keeps the tier on create instead of filling in a default model', async () => {
+ const { ctx, tier } = await tierOnScopedAwayKey()
+
+ const agent = await createAgent({ ctx, draft: { modelTierId: tier.id, provider: AIProviderName.OPENAI, modelName: 'gpt-4o' } })
+
+ expect(agent.draft).toMatchObject({ modelTierId: tier.id, provider: null, providerConfigId: null, modelName: null })
+ })
+
+ it('can be talked to, and hands the tier to the worker', async () => {
+ const { ctx, tier } = await tierOnScopedAwayKey()
+ const add = vi.fn()
+ vi.spyOn(jobQueueModule, 'jobQueue').mockImplementation((log) => ({ ...originalJobQueue(log), add }))
+ const agent = await createAgent({ ctx, draft: { modelTierId: tier.id } })
+ const conversation = await createConversation({ ctx, body: { agentId: agent.id } })
+
+ const response = await ctx.post(`${CONVERSATIONS_URL}/${conversation.id}/messages`, { content: 'hello' })
+
+ expect(response.statusCode).toBe(StatusCodes.OK)
+ const job = add.mock.calls.map(([call]) => call).find((call) => call.data.jobType === WorkerJobType.EXECUTE_AGENT_RUN)
+ expect(job.data.modelTierId).toBe(tier.id)
+ })
+
+ it('records the tier a flow step ran on, so its tools are granted the same keys', async () => {
+ const { ctx, tier } = await tierOnScopedAwayKey()
+ const conversationId = apId()
+
+ await agentRpcHandlers(app.log).getAgentConfig({ conversationId, platformId: ctx.platform.id, projectId: ctx.project.id, userId: ctx.user.id, userMessage: 'sweep the inbox', modelName: null, modelTierId: tier.id, source: AgentRunSource.FLOW_STEP })
+
+ const stored = await db.findOneByOrFail<AgentConversation>('agent_conversation', { id: conversationId })
+ expect(stored.modelTierId).toBe(tier.id)
+ })
+})
+
+async function context(): Promise<TestContext> {
+ return createTestContext(app, { plan: { agentsEnabled: true, chatEnabled: true, aiProvidersEnabled: true } })
+}
+
+async function tierOnScopedAwayKey(): Promise<{ ctx: TestContext, tier: { id: string }, keyId: string }> {
+ const ctx = await context()
+ const key = await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })
+ await db.update('ai_provider', key.id, { projectScope: 'selected', projectIds: [] })
+ const response = await ctx.post('/v1/platform-model-tiers', { name: 'Expert', emoji: '🧠', description: null, entries: [{ configId: key.id, modelId: TIER_MODEL }] })
+ expect(response.statusCode).toBe(StatusCodes.OK)
+ return { ctx, tier: response.json(), keyId: key.id }
+}
+
+async function createConversation({ ctx, body }: { ctx: TestContext, body: Record<string, unknown> }): Promise<AgentConversation> {
+ const response = await ctx.post(CONVERSATIONS_URL, body)
+ expect(response.statusCode).toBe(StatusCodes.CREATED)
+ return response.json()
+}
+
+async function createAgent({ ctx, draft }: { ctx: TestContext, draft: Record<string, unknown> }): Promise<{ id: string, draft: Record<string, unknown> }> {
+ const response = await ctx.post('/v1/agents', {
+ projectId: ctx.project.id,
+ displayName: 'Inbox sorter',
+ icon: AgentIcon.MAIL,
+ color: ColorName.BLUE,
+ draft: { instructions: 'Sort unread mail.', provider: null, modelName: null, maxSteps: 5, tools: [], structuredOutput: [], ...draft },
+ })
+ expect(response.statusCode).toBe(StatusCodes.CREATED)
+ return response.json()
+}
+
+async function getConfig({ ctx, conversationId, provider, providerConfigId }: { ctx: TestContext, conversationId: string, provider?: AIProviderName, providerConfigId?: string }): ReturnType<ReturnType<typeof agentRpcHandlers>['getAgentConfig']> {
+ return agentRpcHandlers(app.log).getAgentConfig({
+ conversationId,
+ platformId: ctx.platform.id,
+ userId: ctx.user.id,
+ userMessage: 'hello',
+ modelName: null,
+ dryRun: true,
+ ...(provider ? { provider } : {}),
+ ...(providerConfigId ? { providerConfigId } : {}),
+ })
+}
+
+const CONVERSATIONS_URL = '/v1/agents/conversations'
+const TIER_MODEL = 'gpt-4o'
diff --git a/packages/server/api/test/unit/app/ee/agent/agent-rpc-handlers.test.ts b/packages/server/api/test/unit/app/ee/agent/agent-rpc-handlers.test.ts
index ca937f6c8cc..a9064a83614 100644
--- a/packages/server/api/test/unit/app/ee/agent/agent-rpc-handlers.test.ts
+++ b/packages/server/api/test/unit/app/ee/agent/agent-rpc-handlers.test.ts
@@ -187,7 +187,7 @@ describe('agentRpcHandlers.updateAgentProgress — incremental LLM message persi
})
})
-async function callSaveChatMessages(input: { conversationId: string, runId?: string, messages: unknown[], uiMessages: unknown[] }): Promise<void> {
+async function callSaveChatMessages(input: { conversationId: string, runId?: string, messages: unknown[], uiMessages: unknown[], answeredBy?: { provider: string, modelId: string } }): Promise<void> {
await agentRpcHandlers(noopLogger as never).saveAgentMessages(input as never)
}
@@ -264,6 +264,15 @@ describe('agentRpcHandlers.saveAgentMessages — billing a row the run no longer
expect(mockTrack.mock.calls[0][0]).toMatchObject({ runId: 'run-1' })
})
+ it('bills a platform tier turn against the model that answered it', async () => {
+ mockExecute.mockResolvedValue({ raw: [{ id: 'conv-1' }] })
+ mockFindOneBy.mockResolvedValue({ id: 'conv-1', messages: [{ role: 'user' }] })
+
+ await callSaveChatMessages({ conversationId: 'conv-1', runId: 'run-1', messages: [{ role: 'user' }, { role: 'assistant' }], uiMessages: [{ role: 'assistant' }], answeredBy: { provider: 'openai', modelId: 'gpt-4o' } })
+
+ expect(mockTrack.mock.calls[0][0]).toMatchObject({ answeredBy: { provider: 'openai', modelId: 'gpt-4o' } })
+ })
+
it('does not bill when the write returned nothing, on any driver', async () => {
mockExecute.mockResolvedValue({})
mockFindOneBy.mockResolvedValue({ id: 'conv-1', messages: [{ role: 'user' }] })
diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
index 8976944d054..a8185d04842 100644
--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
@@ -1,14 +1,14 @@
import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AIProviderName, ErrorCode, formatPieceError, isNil, isObject, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'
import { agentAiUtils, aiUtils } from '@activepieces/server-utils'
-import { AgentEvent, AgentEventType, AgentKnowledgeBaseTool, AgentMcpTool, AgentOutputField, AgentPhase, AgentPieceTool, AgentResult, AgentRunSource, AgentTool, AgentToolType, AiProviderCredentials, apErrorOf, EngineResponseStatus, ExecuteAgentRunJobData, MAX_AGENT_TURN_WALL_CLOCK_MS, PersistedAgentMessage, PersistedAgentMessageSchema, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, ResolvedAgentFlowTool, WorkerJobType } from '@activepieces/shared'
-import { createUIMessageStream, generateText, ModelMessage, streamText, ToolSet, toUIMessageStream } from 'ai'
+import { AgentConfigResponse, AgentEvent, AgentEventType, AgentKnowledgeBaseTool, AgentMcpTool, AgentModelCandidate, AgentOutputField, AgentPhase, AgentPieceTool, AgentResult, AgentRunSource, AgentTool, AgentToolType, AiProviderCredentials, apErrorOf, EngineResponseStatus, ExecuteAgentRunJobData, MAX_AGENT_TURN_WALL_CLOCK_MS, PersistedAgentMessage, PersistedAgentMessageSchema, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, ResolvedAgentFlowTool, SaveAgentMessagesRequest, WorkerJobType } from '@activepieces/shared'
+import { createUIMessageStream, generateText, LanguageModel, ModelMessage, streamText, ToolSet, toUIMessageStream } from 'ai'
import { FireAndForgetJobResult, JobContext, JobHandler, JobResultKind } from '../../../types'
import { toResolvedAiFile } from '../../ai/ai-files'
import { agentMcpClient, McpConnection } from './agent-mcp-client'
import { stepResultFrom } from './agent-step-result'
import { agentToolPolicy } from './agent-tool-policy'
import { agentWorkerTools, GateDecision, TaintState } from './agent-worker-tools'
-import { classifyAgentRunError, delayWithJitter, firstStepUsesFastModel, isTransientFailureText, runAgentTurn } from './run-agent-turn'
+import { classifyAgentRunError, delayWithJitter, drainOf, firstStepUsesFastModel, isTransientFailureText, NO_STEP_CONTENT, runAgentTurn, StreamDrain, trackStepContent, TurnModel } from './run-agent-turn'
const BATCH_SIZE = 10
const BATCH_FLUSH_MS = 50
@@ -65,6 +65,7 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
let heartbeatInterval: NodeJS.Timeout | undefined
let runProvider: string | undefined
let runModelId: string | undefined
+ let tierFailureMessage: string | undefined
let answer: AgentResult | undefined
let configLoaded = false
const structured: { output?: Record<string, unknown> } = {}
@@ -98,6 +99,7 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
...spreadIfDefined('flowRunId', flowRunId),
...spreadIfDefined('provider', data.provider),
...spreadIfDefined('providerConfigId', data.providerConfigId),
+ ...spreadIfDefined('modelTierId', data.modelTierId),
...spreadIfDefined('projectId', projectId),
...spreadIfDefined('promptOverride', promptOverride),
...spreadIfDefined('dryRun', dryRun),
@@ -135,6 +137,12 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
billing,
turnAlreadyCharged: true,
})
+ const turnModels = turnModelsFor({ config, model, billing, metadata: { platformId, conversationId, runId } })
+ const fastTurnModel: TurnModel | undefined = isNil(config.platformTier)
+ ? { model: fastModel, provider, modelId: config.fastModelId, thinkingBudget: config.tier.thinkingBudget }
+ : isNil(config.fastCandidate) ? undefined : candidateTurnModel({ candidate: config.fastCandidate, billing })
+ let answering: TurnModel = turnModels[0]
+ const runKey = (): RunKey => ({ provider: answering.provider, providerConfigId: answering.key?.providerConfigId ?? config.providerConfigId, modelId: answering.modelId })
log.info({ provider, model: { id: config.modelId }, tier: { id: config.tier.id }, dryRun: dryRun ?? false, tavilySearchActive, providerSearchActive }, '[executeAgentRun] Chat config loaded')
@@ -216,9 +224,8 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
}
const allTools = buildToolSet({
- provider,
- providerConfigId: config.providerConfigId,
- modelId: config.modelId,
+ runKey,
+ knowledgeBaseKey: isNil(config.platformTier) ? { provider, providerConfigId: config.providerConfigId } : {},
ctx, eventEmitter, log, phaseState, taintState, mcpToolSet, webTools,
projects: config.projects, projectId, conversationId, runId, ...spreadIfDefined('flowRunId', flowRunId), platformId, userId, userEmail: config.userEmail,
guides: config.guides, dryRun: dryRun ?? false, discoveryOnly: discoveryOnly ?? false,
@@ -254,22 +261,34 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
return runAgentTurn({
...spreadIfDefined('stepCeiling', data.maxSteps),
creditsLeft: (pendingCredits) => ctx.apiClient.agentCreditsLeft({ platformId, conversationId, pendingCredits }),
- model,
- fastModel: firstStepUsesFastModel({ source, dryRun, runsASavedAgent: !isNil(data.promptOverride) }) ? fastModel : undefined,
- provider,
+ models: turnModels,
+ fastModel: firstStepUsesFastModel({ source, dryRun, runsASavedAgent: !isNil(data.promptOverride) }) ? fastTurnModel : undefined,
+ onStepModel: (stepModel) => {
+ if (isNil(stepModel.key)) {
+ return
+ }
+ answering = stepModel
+ runProvider = stepModel.provider
+ runModelId = stepModel.modelId
+ },
+ onModelOutcome: ({ turnModel, signal }) => {
+ if (isNil(turnModel.key)) {
+ return
+ }
+ ctx.apiClient.reportAiKeyOutcome({ platformId, providerConfigId: turnModel.key.providerConfigId, signal }).catch(() => undefined)
+ },
systemPrompt: config.systemPrompt,
messages: config.messages as ModelMessage[],
tools: mergedTools,
allToolNames,
tier: config.tier,
- modelId: config.modelId,
- ...spreadIfDefined('fastModelId', dryRun ? undefined : config.fastModelId),
phaseState,
abortSignal: abortController.signal,
log,
sinks: {
drainStream: (result) => streamChunksToClient({
result, ctx, userId, conversationId, runId, log,
+ holdErrorChunks: !isNil(config.platformTier),
abortSignal: abortController.signal,
onStreamIdle: (reason) => {
const fields = { conversation: { id: conversationId }, idleMs: STREAM_IDLE_REPORT_MS, reason }
@@ -318,6 +337,7 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
...(config.previousUiMessages as PersistedAgentMessage[]),
...(uiParts.length > 0 ? [{ role: PersistedAgentRole.ASSISTANT, parts: uiParts, thinkingDurationMs, tainted: taintState.readInThisReply() }] : []),
],
+ ...answeredByOf({ config, answeredBy: turn.answeredBy }),
}
const { error: cancelSaveError } = await tryCatch(() => ctx.apiClient.saveAgentMessages(cancelSavePayload))
if (cancelSaveError) {
@@ -338,6 +358,9 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
}
if (streamError) {
+ tierFailureMessage = turn.allModelsFailed && !isNil(config.platformTier)
+ ? `All ${turnModels.length} models in tier ${config.platformTier.name} failed (${turnModels.map((candidate) => candidate.modelId).join(', ')}). Last error: ${apErrorOf(streamError)?.message ?? formatPieceError(streamError).message}`
+ : undefined
throw streamError
}
@@ -350,7 +373,8 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
outputTokens: totalOutputTokens,
...spreadIfDefined('cacheReadTokens', usage?.inputTokenDetails?.cacheReadTokens),
...spreadIfDefined('cacheWriteTokens', usage?.inputTokenDetails?.cacheWriteTokens),
- provider: config.credentials.provider,
+ provider: turn.answeredBy.provider,
+ model: { id: turn.answeredBy.modelId },
finishReason: turn.finishReason,
truncatedAfterRetries,
}, 'Chat message completed')
@@ -365,7 +389,8 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
...(uiParts.length > 0 ? [{ role: PersistedAgentRole.ASSISTANT, parts: uiParts, thinkingDurationMs, tainted: taintState.readInThisReply() }] : []),
],
...spreadIfDefined('title', autoTitle),
- ...spreadIfDefined('modelName', isNil(data.modelName) ? config.tier.id : undefined),
+ ...spreadIfDefined('modelName', isNil(data.modelName) && isNil(config.platformTier) ? config.tier.id : undefined),
+ ...answeredByOf({ config, answeredBy: turn.answeredBy }),
}
await retryWithBackoff({ fn: () => ctx.apiClient.saveAgentMessages(savePayload), description: 'Saving the transcript', throwOnExhausted: true, log })
@@ -409,11 +434,13 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
const errorMessage = apErrorOf(err)?.message ?? formatPieceError(err).message
const isCreditError = errorClass === 'credit'
// "User not found" is OpenRouter refusing a key, and reads like a missing account.
- const clientMessage = !isCreditError && isTransientFailureText(errorMessage)
- ? 'The AI provider is temporarily unavailable. Please try again in a moment.'
- : isCreditError || isNil(runProvider)
- ? errorMessage
- : `${runProvider}${isNil(runModelId) ? '' : ` (${runModelId})`}: ${errorMessage}`
+ const clientMessage = !isNil(tierFailureMessage)
+ ? tierFailureMessage
+ : !isCreditError && isTransientFailureText(errorMessage)
+ ? 'The AI provider is temporarily unavailable. Please try again in a moment.'
+ : isCreditError || isNil(runProvider)
+ ? errorMessage
+ : `${runProvider}${isNil(runModelId) ? '' : ` (${runModelId})`}: ${errorMessage}`
const failedResult = { ...(answer ?? stepResultFrom({ prompt: userMessage, uiParts: [], timestamp: new Date().toISOString(), tools: reportedTools, structuredOutput: structured.output, failure: clientMessage })), failure: clientMessage }
reportFinal(failedResult)
const { error: releaseError } = await tryCatch(() => releaseFlowStep({ ctx, conversationId, flowRunId, waitpointId, output: failedResult, source, log }))
@@ -458,6 +485,32 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
},
}
+function answeredByOf({ config, answeredBy }: { config: AgentConfigResponse, answeredBy: TurnModel }): Pick<SaveAgentMessagesRequest, 'answeredBy'> {
+ return isNil(config.platformTier) ? {} : { answeredBy: { provider: answeredBy.provider, modelId: answeredBy.modelId } }
+}
+
+function turnModelsFor({ config, model, billing, metadata }: { config: AgentConfigResponse, model: LanguageModel, billing: ActivepiecesAiBilling, metadata: ModelMetadata }): [TurnModel, ...TurnModel[]] {
+ const single: TurnModel = { model, provider: config.credentials.provider, modelId: config.modelId, thinkingBudget: config.tier.thinkingBudget }
+ if (isNil(config.candidates) || config.candidates.length === 0) {
+ return [single]
+ }
+ const [first, ...rest] = config.candidates
+ return [
+ { ...candidateTurnModel({ candidate: first, billing }), model },
+ ...rest.map((candidate) => candidateTurnModel({ candidate, billing, metadata })),
+ ]
+}
+
+function candidateTurnModel({ candidate, billing, metadata }: { candidate: AgentModelCandidate, billing: ActivepiecesAiBilling, metadata?: ModelMetadata }): TurnModel {
+ return {
+ model: aiUtils.createModel({ credentials: candidate, modelId: candidate.modelId, billing, turnAlreadyCharged: true, ...spreadIfDefined('metadata', metadata) }),
+ provider: candidate.provider,
+ modelId: candidate.modelId,
+ thinkingBudget: candidate.thinkingBudget,
+ key: { providerConfigId: candidate.providerConfigId, status: candidate.status },
+ }
+}
+
function incompleteReason({ truncatedAfterRetries, budgetExceeded, creditsExhausted }: { truncatedAfterRetries: boolean, budgetExceeded: boolean, creditsExhausted: boolean }): string | undefined {
if (truncatedAfterRetries) {
return 'The response reached the output limit before the agent finished'
@@ -534,11 +587,10 @@ function pickImageGenerator({ falApiKey, imageModelId, credentials, billing, rea
}
}
-function buildToolSet({ ctx, eventEmitter, log, phaseState, taintState, mcpToolSet, webTools, projects, projectId, conversationId, flowRunId, runId, platformId, userId, userEmail, guides, dryRun, discoveryOnly, emailEnabled, agentsAvailable, abortSignal, source, provider, providerConfigId, modelId, configuredPieceTools, configuredFlowTools, configuredKnowledgeBaseTools, structuredOutput, captureStructured }: {
+function buildToolSet({ ctx, eventEmitter, log, phaseState, taintState, mcpToolSet, webTools, projects, projectId, conversationId, flowRunId, runId, platformId, userId, userEmail, guides, dryRun, discoveryOnly, emailEnabled, agentsAvailable, abortSignal, source, runKey, knowledgeBaseKey, configuredPieceTools, configuredFlowTools, configuredKnowledgeBaseTools, structuredOutput, captureStructured }: {
ctx: JobContext
- provider: AIProviderName
- providerConfigId: string
- modelId: string
+ runKey: () => RunKey
+ knowledgeBaseKey: { provider?: AIProviderName, providerConfigId?: string }
eventEmitter: ReturnType<typeof agentWorkerTools.createEventEmitter>
log: JobContext['log']
phaseState: { phase: AgentPhase }
@@ -634,7 +686,7 @@ function buildToolSet({ ctx, eventEmitter, log, phaseState, taintState, mcpToolS
const selectedConnectionByPiece = new Map<string, string>()
const localTools = agentWorkerTools.createLocalTools({
onSetProjectContext: async (projectId) => {
- const { error } = await tryCatch(() => ctx.apiClient.updateProjectContext({ conversationId, runId, projectId, provider, providerConfigId }))
+ const { error } = await tryCatch(() => ctx.apiClient.updateProjectContext({ conversationId, runId, projectId, provider: runKey().provider, providerConfigId: runKey().providerConfigId }))
if (!isNil(error)) {
log.warn({ error, conversation: { id: conversationId }, project: projectId ? { id: projectId } : undefined }, '[executeAgentRun] Refused a project switch')
return { success: false, error: 'Could not switch to that project on this chat — the AI provider key this chat runs on is not available there. Start a new chat in that project instead.' }
@@ -744,7 +796,7 @@ function buildToolSet({ ctx, eventEmitter, log, phaseState, taintState, mcpToolS
// answer, and an agent that asks an empty room reads the silence as a refusal and stops.
const configuredTools = agentWorkerTools.createConfiguredPieceTools({
tools: dryRun || discoveryOnly ? [] : configuredPieceTools,
- runPieceTool: ({ toolName, instruction, piece }) => ctx.apiClient.executePieceTool({ conversationId, ...spreadIfDefined('runId', runId), toolName, instruction, piece, provider, providerConfigId, modelId, ...spreadIfDefined('flowRunId', flowRunId) }),
+ runPieceTool: ({ toolName, instruction, piece }) => ctx.apiClient.executePieceTool({ conversationId, ...spreadIfDefined('runId', runId), toolName, instruction, piece, ...runKey(), ...spreadIfDefined('flowRunId', flowRunId) }),
taintState,
eventEmitter,
log,
@@ -758,7 +810,7 @@ function buildToolSet({ ctx, eventEmitter, log, phaseState, taintState, mcpToolS
const knowledgeBaseTools = agentWorkerTools.createConfiguredKnowledgeBaseTools({
taintState,
tools: dryRun || discoveryOnly ? [] : configuredKnowledgeBaseTools,
- runKnowledgeBaseTool: ({ toolName, knowledgeBaseFileId, query }) => ctx.apiClient.executeKnowledgeBaseTool({ conversationId, ...spreadIfDefined('runId', runId), toolName, knowledgeBaseFileId, query, provider, providerConfigId }),
+ runKnowledgeBaseTool: ({ toolName, knowledgeBaseFileId, query }) => ctx.apiClient.executeKnowledgeBaseTool({ conversationId, ...spreadIfDefined('runId', runId), toolName, knowledgeBaseFileId, query, ...spreadIfDefined('provider', knowledgeBaseKey.provider), ...spreadIfDefined('providerConfigId', knowledgeBaseKey.providerConfigId) }),
log,
})
const completionTool = structuredOutput.length === 0
@@ -785,7 +837,7 @@ function buildToolSet({ ctx, eventEmitter, log, phaseState, taintState, mcpToolS
})
}
-async function streamChunksToClient({ result, ctx, userId, conversationId, runId, log, abortSignal, onStreamIdle }: {
+async function streamChunksToClient({ result, ctx, userId, conversationId, runId, log, abortSignal, onStreamIdle, holdErrorChunks }: {
result: ReturnType<typeof streamText>
ctx: JobContext
userId: string
@@ -794,8 +846,10 @@ async function streamChunksToClient({ result, ctx, userId, conversationId, runId
log: JobContext['log']
abortSignal: AbortSignal
onStreamIdle: (reason: 'idle' | 'tool' | 'reasoning') => void
-}): Promise<void> {
+ holdErrorChunks: boolean
+}): Promise<StreamDrain> {
let chunkBuffer: unknown[] = []
+ let stepContent = NO_STEP_CONTENT
let flushTimer: ReturnType<typeof setTimeout> | null = null
const flushChunks = async () => {
@@ -860,6 +914,10 @@ async function streamChunksToClient({ result, ctx, userId, conversationId, runId
else if (chunkType === 'reasoning-end') {
reasoningInFlight = false
}
+ stepContent = trackStepContent({ state: stepContent, chunkType })
+ if (holdErrorChunks && chunkType === 'error') {
+ continue
+ }
chunkBuffer.push(chunk)
if (chunkBuffer.length >= BATCH_SIZE) {
if (flushTimer) {
@@ -887,6 +945,7 @@ async function streamChunksToClient({ result, ctx, userId, conversationId, runId
}
if (flushTimer) clearTimeout(flushTimer)
await flushChunks()
+ return drainOf({ state: stepContent })
}
async function generateTitleIfFirstTurn({ model, userMessage, previousUiMessages, log, conversationId, abortSignal }: {
@@ -956,3 +1015,11 @@ async function retryWithBackoff({ fn, maxAttempts = RETRY_MAX_ATTEMPTS, descript
}
type ImageToolParams = Parameters<typeof agentWorkerTools.createImageTools>[0]
+
+type ModelMetadata = Parameters<typeof aiUtils.createModel>[0]['metadata']
+
+type RunKey = {
+ provider: AIProviderName
+ providerConfigId: string
+ modelId: string
+}
diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
index 41dd461f8e3..51520d5f65f 100644
--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts
@@ -1,5 +1,5 @@
-import { AIProviderName, ErrorCode, formatPieceError, isNil, isObject, isProviderBillingError, isTransientProviderError, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'
-import { agentAiUtils, ContentPartLike } from '@activepieces/server-utils'
+import { AiProviderKeyStatus, AIProviderName, ErrorCode, formatPieceError, isFallbackWorthy, isNil, isObject, isProviderBillingError, isTransientProviderError, ProviderOutcomeSignal, spreadIfDefined, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'
+import { agentAiUtils, aiProviderSignal, ContentPartLike } from '@activepieces/server-utils'
import { AgentPhase, AgentRunSource, agentToolClassification, agentToolPhases, AI_PROVIDER_ENTITY_TYPES, aiProviderUtils, apErrorOf, CHAT_CREDITS_PER_TOOL_CALL, chatBilling, ChatToolCall, PersistedAgentPart, TASK_COMPLETION_TOOL_NAME } from '@activepieces/shared'
import { APICallError, generateText, isLoopFinished, isStepCount, LanguageModel, LanguageModelUsage, ModelMessage, NoSuchToolError, RetryError, StepResultPerformance, StopCondition, streamText, ToolChoice, ToolExecutionOptions, ToolSet } from 'ai'
@@ -10,6 +10,10 @@ const MAX_IDENTICAL_TOOL_FAILURES = 2
const IN_LOOP_COMPACTION_THRESHOLD = 0.6
const RUNAWAY_TURN_CONTEXT_MULTIPLE = 90
const STREAM_RETRY_BASE_DELAY_MS = 1_000
+const SDK_RETRIES = 3
+const SDK_RETRIES_BEFORE_FALLBACK = 1
+const MIN_THINKING_BUDGET = 1_024
+const CONTENT_CHUNK_TYPES = new Set(['text-delta', 'reasoning-delta', 'tool-input-start', 'tool-input-available', 'source-url', 'source-document', 'file'])
const USER_FAULT_STATUS_CODES = new Set([401, 403, 404])
const MODEL_UNAVAILABLE_PATTERNS = [/\bis deprecated\b/i, /no longer (available|supported)/i, /\bmodel_not_found\b/i, /\bunknown model\b/i, /\bdecommissioned\b/i]
const USER_CONFIG_ENTITY_TYPES = new Set<string>(Object.values(AI_PROVIDER_ENTITY_TYPES))
@@ -41,8 +45,25 @@ export function shouldRetryStream({ producedVisibleOutput, streamRetries }: {
return !producedVisibleOutput && streamRetries < MAX_STREAM_RETRIES
}
-export async function runAgentTurn({ model, fastModel, provider, systemPrompt, messages, tools, allToolNames, tier, modelId, fastModelId, phaseState, abortSignal, log, sinks, stopWhen, stepCeiling, creditsLeft }: RunAgentTurnParams): Promise<AgentTurnResult> {
- const drainStream = sinks?.drainStream ?? (async () => {})
+export function trackStepContent({ state, chunkType }: { state: StepContentState, chunkType: string | undefined }): StepContentState {
+ if (!isNil(state.sentWhenFailed)) {
+ return state
+ }
+ if (chunkType === 'error') {
+ return { ...state, sentWhenFailed: state.sentInStep }
+ }
+ if (chunkType === 'start-step' || chunkType === 'finish-step') {
+ return { ...state, sentInStep: false }
+ }
+ return { ...state, sentInStep: state.sentInStep || (!isNil(chunkType) && CONTENT_CHUNK_TYPES.has(chunkType)) }
+}
+
+export function drainOf({ state }: { state: StepContentState }): StreamDrain {
+ return { lastStepSentContent: state.sentWhenFailed ?? state.sentInStep }
+}
+
+export async function runAgentTurn({ models, fastModel, systemPrompt, messages, tools, allToolNames, tier, phaseState, abortSignal, log, sinks, stopWhen, stepCeiling, creditsLeft, onModelOutcome, onStepModel }: RunAgentTurnParams): Promise<AgentTurnResult> {
+ const drainStream = sinks?.drainStream ?? (async () => undefined)
const onProgress = sinks?.onProgress ?? (() => {})
const baseStopCondition = stopWhen ?? isLoopFinished()
let creditsExhausted = false
@@ -52,7 +73,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
if (isNil(creditsLeft)) {
return false
}
- const pendingCredits = chatBilling.creditsForTurn({ provider, toolCalls: [...earlierAttemptToolCalls, ...completedToolCalls(steps)] }).total
+ const pendingCredits = chatBilling.creditsForTurn({ provider: current().provider, toolCalls: [...earlierAttemptToolCalls, ...completedToolCalls(steps)] }).total
const { data: left, error } = await tryCatch(() => creditsLeft(pendingCredits))
if (error) {
log.warn({ error }, 'Credit check failed mid-turn, letting the turn continue')
@@ -67,7 +88,16 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
creditsRanOut,
]
const guardedTools = wrapToolsWithFailureGuard({ tools, log })
- const maxTurnTokens = runawayTokenCeiling(provider)
+ const maxTurnTokens = runawayTokenCeiling(models[0].provider)
+ const runsATier = models.some((candidate) => !isNil(candidate.key))
+ const peers = [...models, ...(isNil(fastModel) ? [] : [fastModel])]
+ const ranked = await Promise.all(models.map((turnModel) => withOutputBudget({ turnModel, peers })))
+ let fast = isNil(fastModel) ? undefined : await withOutputBudget({ turnModel: fastModel, peers })
+ let modelIndex = 0
+ let lastStepUsedFast = false
+ const failedModelIds = new Set<string>()
+ const current = (): BudgetedTurnModel => ranked[modelIndex]
+ const hasNextModel = (): boolean => modelIndex < ranked.length - 1
const uiParts: PersistedAgentPart[] = []
const toolCalls: AgentTurnToolCall[] = []
@@ -80,7 +110,6 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
let continuations = 0
let emptyContinuations = 0
let streamRetries = 0
- let lastStepModelId = modelId
let truncatedAfterRetries = false
let usage: LanguageModelUsage | undefined
let totalInputTokens = 0
@@ -88,16 +117,13 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
let lastFinishReason = ''
let budgetExceeded = false
- const maxOutputTokens = await agentAiUtils.affordableOutputTokens({ provider, modelIds: [modelId, fastModelId], thinkingBudget: tier.thinkingBudget })
- const maxOutputTokensWithoutThinking = agentAiUtils.clampOutputTokens({ thinkingBudget: 0, ceilings: [maxOutputTokens] })
-
const runStreamAttempt = (attemptMessages: ModelMessage[]): ReturnType<typeof streamText> => streamText({
- model,
- maxRetries: 3,
- maxOutputTokens,
+ model: current().model,
+ maxRetries: runsATier && hasNextModel() ? SDK_RETRIES_BEFORE_FALLBACK : SDK_RETRIES,
+ maxOutputTokens: current().budget.maxOutputTokens,
abortSignal,
- instructions: agentAiUtils.buildSystemPromptWithCaching({ systemPrompt, provider }),
- messages: agentAiUtils.stripThinkingBlocks(attemptMessages, provider),
+ instructions: agentAiUtils.buildSystemPromptWithCaching({ systemPrompt, provider: current().provider }),
+ messages: agentAiUtils.stripThinkingBlocks(attemptMessages, current().provider),
tools: guardedTools,
stopWhen: loopStopCondition,
telemetry: agentAiUtils.buildTelemetry({ functionId: 'agent-turn' }),
@@ -127,20 +153,24 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
? (forcesCompletion ? { type: 'tool', toolName: TASK_COMPLETION_TOOL_NAME } : 'none')
: undefined
const disableThinking = isFirstStep || phaseState.phase === 'discovery' || forcesCompletion
- const usesFastModel = isFirstStep && !isNil(fastModel)
+ const stepModel = isFirstStep && !isNil(fast) ? fast : current()
+ const usesFastModel = stepModel !== current()
+ const stepBudget = stepModel.budget
const phaseTools = agentToolPhases.activeToolsForPhase({ phase: phaseState.phase, allToolNames }).filter((name) => paidToolsAffordable || !chatBilling.isPaidTool(name))
const activeTools = forcesCompletion ? [TASK_COMPLETION_TOOL_NAME] : phaseTools
- const boundedContext = boundContextForStep({ baseMessages: attemptMessages, steps, systemPrompt, provider })
+ const boundedContext = boundContextForStep({ baseMessages: attemptMessages, steps, systemPrompt, provider: stepModel.provider })
const stepContext = isLastAllowedStep
? { messages: [...(boundedContext.messages ?? currentMessages), FINAL_STEP_MESSAGE] }
: boundedContext
- lastStepModelId = usesFastModel ? fastModelId ?? modelId : modelId
+ lastStepUsedFast = usesFastModel
+ onStepModel?.(stepModel)
+ const thinkingOff = disableThinking || stepBudget.thinkingBudget === 0
return {
- ...(usesFastModel ? { model: fastModel } : {}),
+ ...(usesFastModel ? { model: stepModel.model } : {}),
activeTools,
...spreadIfDefined('toolChoice', toolChoice),
- maxOutputTokens: disableThinking ? maxOutputTokensWithoutThinking : maxOutputTokens,
- providerOptions: agentAiUtils.buildProviderOptions({ provider, tier, modelId: lastStepModelId, disableThinking }),
+ maxOutputTokens: thinkingOff ? stepBudget.maxOutputTokensWithoutThinking : stepBudget.maxOutputTokens,
+ providerOptions: agentAiUtils.buildProviderOptions({ provider: stepModel.provider, tier: { id: tier.id, thinkingBudget: stepBudget.thinkingBudget }, modelId: stepModel.modelId, disableThinking: thinkingOff }),
...stepContext,
}
},
@@ -152,7 +182,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
log.warn({ toolName: toolCall.toolName, error }, 'Repairing malformed tool call')
const { data: repaired } = await tryCatch(async () => {
const { text } = await generateText({
- model,
+ model: current().model,
abortSignal,
telemetry: agentAiUtils.buildTelemetry({ functionId: 'agent-tool-repair' }),
prompt: `Fix this malformed JSON tool call for "${toolCall.toolName}". The error was: ${error.message}\n\nOriginal input:\n${toolCall.input}\n\nReturn ONLY the corrected JSON input, nothing else.`,
@@ -203,7 +233,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
const uiPartsCountBefore = uiParts.length
currentAttemptMessages = []
const result = runStreamAttempt(llmMessages)
- await drainStream(result)
+ const drained = await drainStream(result)
const producedVisibleOutput = uiParts.length > uiPartsCountBefore
// On abort/error we leave the loop WITHOUT reaching the clean-exit pushes below, so fold
// this attempt's completed steps in here — otherwise the turn's work is lost from the
@@ -213,7 +243,29 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
break
}
if (streamError) {
- if (shouldRetryStream({ producedVisibleOutput, streamRetries })) {
+ const signal = aiProviderSignal.fromError(streamError)
+ const failedModel = lastStepUsedFast && !isNil(fast) ? fast : current()
+ reportOutcome({ turnModel: failedModel, signal, onModelOutcome })
+ const failedFast = failedModel === fast
+ if (!failedFast) {
+ failedModelIds.add(failedModel.modelId)
+ }
+ const sentNothing = runsATier && drained?.lastStepSentContent === false
+ const dropsFast = failedFast && sentNothing
+ const movesOn = !failedFast && sentNothing && isFallbackWorthy(signal) && hasNextModel()
+ if (dropsFast || movesOn) {
+ if (!failedFast) {
+ modelIndex++
+ }
+ log.warn({ error: streamError, failedModel: { id: failedModel.modelId, fast: failedFast }, model: { id: current().modelId }, candidateIndex: modelIndex }, 'Chat model failed before sending anything — continuing the turn on the next model')
+ fast = undefined
+ accumulatedResponseMessages.push(...currentAttemptMessages)
+ llmMessages = [...llmMessages, ...currentAttemptMessages]
+ streamError = null
+ streamRetries = 0
+ continue
+ }
+ if (shouldRetryStream({ producedVisibleOutput: producedVisibleOutput || drained?.lastStepSentContent === true, streamRetries })) {
streamRetries++
log.warn({ streamRetries, error: streamError }, 'Chat stream failed before any visible output — retrying the turn')
streamError = null
@@ -239,7 +291,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
totalInputTokens += attemptUsage.inputTokens ?? 0
totalOutputTokens += attemptUsage.outputTokens ?? 0
lastFinishReason = finishReason
- logTurnPerformance({ performance: finalStep.performance, modelId: tier.modelId, stepCount: steps.length, log })
+ logTurnPerformance({ performance: finalStep.performance, modelId: current().modelId, stepCount: steps.length, log })
if (creditsExhausted) {
accumulatedResponseMessages.push(...stepMessages)
@@ -280,6 +332,13 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
llmMessages = [...llmMessages, ...sanitizedTail, { role: 'user', content: EMPTY_OUTPUT_NUDGE }]
}
+ if (isNil(streamError) && !abortSignal.aborted && current().key?.status !== 'active') {
+ reportOutcome({ turnModel: current(), signal: { statusCode: 200 }, onModelOutcome })
+ }
+ if (modelIndex > 0) {
+ log.info({ candidateIndex: modelIndex, model: { id: current().modelId } }, 'A fallback model answered the chat turn')
+ }
+
return {
accumulatedResponseMessages,
uiParts,
@@ -293,7 +352,32 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m
totalInputTokens,
totalOutputTokens,
toolCalls,
+ answeredBy: current(),
+ allModelsFailed: !isNil(streamError) && runsATier && failedModelIds.size === unique(ranked.map((candidate) => candidate.modelId)).length,
+ }
+}
+
+async function withOutputBudget({ turnModel, peers }: { turnModel: TurnModel, peers: TurnModel[] }): Promise<BudgetedTurnModel> {
+ const sameProviderIds = unique(peers.filter((peer) => peer.provider === turnModel.provider).map((peer) => peer.modelId))
+ const maxOutputTokens = await agentAiUtils.affordableOutputTokens({ provider: turnModel.provider, modelIds: sameProviderIds, thinkingBudget: turnModel.thinkingBudget })
+ const roomForThinking = Math.min(turnModel.thinkingBudget, maxOutputTokens - MIN_THINKING_BUDGET)
+ const thinkingBudget = isNil(turnModel.key) ? turnModel.thinkingBudget : roomForThinking < MIN_THINKING_BUDGET ? 0 : roomForThinking
+ return {
+ ...turnModel,
+ budget: {
+ maxOutputTokens,
+ maxOutputTokensWithoutThinking: agentAiUtils.clampOutputTokens({ thinkingBudget: 0, ceilings: [maxOutputTokens] }),
+ thinkingBudget,
+ },
+ }
+}
+
+function reportOutcome({ turnModel, signal, onModelOutcome }: { turnModel: TurnModel, signal: ProviderOutcomeSignal, onModelOutcome?: (outcome: { turnModel: TurnModel, signal: ProviderOutcomeSignal }) => void }): void {
+ const comesFromProvider = signal.fromProvider === true || signal.statusCode === 200
+ if (isNil(turnModel.key) || isNil(onModelOutcome) || !comesFromProvider) {
+ return
}
+ onModelOutcome({ turnModel, signal })
}
// timeToFirstOutputMs is the number the fast-model-first-step swap in prepareStep exists to buy;
@@ -493,21 +577,45 @@ export type AgentTurnToolCall = {
}
export type AgentTurnSinks = {
- drainStream?: (result: ReturnType<typeof streamText>) => Promise<void>
+ drainStream?: (result: ReturnType<typeof streamText>) => Promise<StreamDrain | undefined>
onProgress?: (progress: { uiParts: PersistedAgentPart[], responseMessages: ModelMessage[] }) => void
}
-export type RunAgentTurnParams = {
+export type StreamDrain = {
+ lastStepSentContent: boolean
+}
+
+export type StepContentState = {
+ sentInStep: boolean
+ sentWhenFailed: boolean | null
+}
+
+export const NO_STEP_CONTENT: StepContentState = { sentInStep: false, sentWhenFailed: null }
+
+export type TurnModel = {
model: LanguageModel
- fastModel?: LanguageModel
provider: AIProviderName
+ modelId: string
+ thinkingBudget: number
+ key?: { providerConfigId: string, status: AiProviderKeyStatus }
+}
+
+type BudgetedTurnModel = TurnModel & {
+ budget: {
+ maxOutputTokens: number
+ maxOutputTokensWithoutThinking: number
+ thinkingBudget: number
+ }
+}
+
+export type RunAgentTurnParams = {
+ models: [TurnModel, ...TurnModel[]]
+ fastModel?: TurnModel
systemPrompt: string
messages: ModelMessage[]
tools: ToolSet
allToolNames: string[]
tier: { id: string, thinkingBudget: number, modelId: string }
- modelId: string
- fastModelId?: string
phaseState: { phase: AgentPhase }
abortSignal: AbortSignal
log: AgentTurnLogger
@@ -515,6 +623,8 @@ export type RunAgentTurnParams = {
stopWhen?: StopCondition<ToolSet> | Array<StopCondition<ToolSet>>
stepCeiling?: number
creditsLeft?: (pendingCredits: number) => Promise<number | null>
+ onModelOutcome?: (outcome: { turnModel: TurnModel, signal: ProviderOutcomeSignal }) => void
+ onStepModel?: (turnModel: TurnModel) => void
}
export type AgentTurnResult = {
@@ -530,6 +640,8 @@ export type AgentTurnResult = {
totalInputTokens: number
totalOutputTokens: number
toolCalls: AgentTurnToolCall[]
+ answeredBy: TurnModel
+ allModelsFailed: boolean
}
type AgentRunErrorClass = 'credit' | 'user' | 'provider' | 'internal'
diff --git a/packages/server/worker/test/lib/agent-eval/agent-tool-call-repair.test.ts b/packages/server/worker/test/lib/agent-eval/agent-tool-call-repair.test.ts
index 65aa1793a54..e56aef5ee56 100644
--- a/packages/server/worker/test/lib/agent-eval/agent-tool-call-repair.test.ts
+++ b/packages/server/worker/test/lib/agent-eval/agent-tool-call-repair.test.ts
@@ -41,8 +41,7 @@ async function turnWhereTheModelRepairsWith(repairText: string): Promise<unknown
})
await runAgentTurn({
- model,
- provider: AIProviderName.ANTHROPIC,
+ models: [{ model, provider: AIProviderName.ANTHROPIC, modelId: TIER.modelId, thinkingBudget: TIER.thinkingBudget }],
systemPrompt: 'You are a test agent.',
messages: [{ role: 'user', content: 'search the web' }],
tools: {
@@ -57,7 +56,6 @@ async function turnWhereTheModelRepairsWith(repairText: string): Promise<unknown
} as never,
allToolNames: ['ap_web_search'],
tier: TIER,
- modelId: TIER.modelId,
phaseState: { phase: 'discovery' },
abortSignal: new AbortController().signal,
log: silentLog,
diff --git a/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts b/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts
index 3d24b60827b..0f754e86f51 100644
--- a/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts
+++ b/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts
@@ -100,8 +100,7 @@ async function runTurn({ search, creditsLeft, stepCeiling = 20, model = alwaysSe
? {}
: { updateTaskStatus: tool({ description: 'report the result', inputSchema: z.object({ output: z.object({ summary: z.string() }) }), execute: async (input) => { completion(input); return SEARCH_RESULT } }) }
return runAgentTurn({
- model,
- provider: AIProviderName.ANTHROPIC,
+ models: [{ model, provider: AIProviderName.ANTHROPIC, modelId: TIER.modelId, thinkingBudget: TIER.thinkingBudget }],
systemPrompt: 'You are a test agent.',
messages: [{ role: 'user', content: 'research this' }],
tools: {
@@ -111,12 +110,11 @@ async function runTurn({ search, creditsLeft, stepCeiling = 20, model = alwaysSe
},
allToolNames: ['ap_web_search', 'ap_fetch_url', ...Object.keys(completionTools)],
tier: TIER,
- modelId: TIER.modelId,
phaseState: { phase },
abortSignal: new AbortController().signal,
log: SILENT_LOG,
...spreadIfDefined('stepCeiling', stepCeiling ?? undefined),
- ...(drainsStream ? { sinks: { drainStream: (result) => result.consumeStream() } } : {}),
+ ...(drainsStream ? { sinks: { drainStream: async (result) => { await result.consumeStream(); return undefined } } } : {}),
creditsLeft,
})
}
diff --git a/packages/server/worker/test/lib/agent-eval/agent-turn-provider-options.test.ts b/packages/server/worker/test/lib/agent-eval/agent-turn-provider-options.test.ts
index 185cb5d9515..69456ebe156 100644
--- a/packages/server/worker/test/lib/agent-eval/agent-turn-provider-options.test.ts
+++ b/packages/server/worker/test/lib/agent-eval/agent-turn-provider-options.test.ts
@@ -26,14 +26,12 @@ async function capturedProviderOptions({ provider, modelId }: { provider: AIProv
})
await runAgentTurn({
- model,
- provider,
+ models: [{ model, provider, modelId: modelId, thinkingBudget: TIER.thinkingBudget }],
systemPrompt: 'You are a test agent.',
messages: [{ role: 'user', content: 'hello' }],
tools: {},
allToolNames: [],
tier: TIER,
- modelId,
phaseState: { phase: 'discovery' },
abortSignal: new AbortController().signal,
log: silentLog,
diff --git a/packages/server/worker/test/lib/agent-eval/core/runner.ts b/packages/server/worker/test/lib/agent-eval/core/runner.ts
index 1f7218a6f14..c84556c8fb3 100644
--- a/packages/server/worker/test/lib/agent-eval/core/runner.ts
+++ b/packages/server/worker/test/lib/agent-eval/core/runner.ts
@@ -201,14 +201,12 @@ async function runTurn({ fixture, systemPrompt, guides, auth }: { fixture: ChatE
const capturedErrors: unknown[] = []
const { data: result, error: turnError } = await tryCatch(() => runAgentTurn({
- model,
- provider: fixture.model.provider,
+ models: [{ model, provider: fixture.model.provider, modelId: fixture.model.tier.modelId, thinkingBudget: fixture.model.tier.thinkingBudget }],
systemPrompt: systemPrompt ?? evalPrompts.loadSystemPrompt(),
messages,
tools,
allToolNames: Object.keys(tools),
tier: fixture.model.tier,
- modelId: fixture.model.tier.modelId,
phaseState,
abortSignal: new AbortController().signal,
log: {
diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts
new file mode 100644
index 00000000000..3a2f852f1de
--- /dev/null
+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts
@@ -0,0 +1,252 @@
+import { AIProviderName, isObject } from '@activepieces/core-utils'
+import { SharedV3ProviderOptions } from '@ai-sdk/provider'
+import { APICallError, streamText, tool, toUIMessageStream } from 'ai'
+import { convertArrayToReadableStream, MockLanguageModelV3 } from 'ai/test'
+import { describe, expect, it, vi } from 'vitest'
+import { z } from 'zod'
+import { drainOf, NO_STEP_CONTENT, runAgentTurn, StepContentState, StreamDrain, trackStepContent, TurnModel } from '../../../../../../src/lib/execute/jobs/ee/agent/run-agent-turn'
+
+describe('a platform tier chat turn', () => {
+ it('answers on the next model when the main model fails before sending anything', async () => {
+ const onStepModel = vi.fn()
+
+ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: failingModel({ statusCode: 503 }) }), tierModel({ modelId: 'backup', model: answeringModel('from backup') })], onStepModel })
+
+ expect(turn.streamError).toBeNull()
+ expect(turn.answeredBy.modelId).toBe('backup')
+ expect(onStepModel).toHaveBeenLastCalledWith(expect.objectContaining({ modelId: 'backup' }))
+ expect(textOf(turn)).toBe('from backup')
+ })
+
+ it('continues on the next model after the fast step already answered and ran a tool, without running it again', async () => {
+ const search = vi.fn(async () => ({ content: [{ type: 'text', text: 'found' }] }))
+ const backup = answeringModel('done')
+
+ const turn = await runTurn({
+ models: [tierModel({ modelId: 'main', model: failingModel({ statusCode: 503 }) }), tierModel({ modelId: 'backup', model: backup })],
+ fastModel: tierModel({ modelId: 'fast', model: searchingModel() }),
+ search,
+ })
+
+ expect(turn.streamError).toBeNull()
+ expect(turn.answeredBy.modelId).toBe('backup')
+ expect(search).toHaveBeenCalledTimes(1)
+ expect(textOf(turn)).toBe('Looking.done')
+ expect(JSON.stringify(backup.doStreamCalls[0].prompt)).toContain('found')
+ })
+
+ it('does not fall back once the failing step already sent text to the user', async () => {
+ const backup = answeringModel('from backup')
+
+ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: textThenFailingModel() }), tierModel({ modelId: 'backup', model: backup })] })
+
+ expect(turn.streamError).not.toBeNull()
+ expect(backup.doStreamCalls).toHaveLength(0)
+ })
+
+ it('tells tools which model runs each step, so a fast step\'s tool calls use the fast key', async () => {
+ const onStepModel = vi.fn()
+
+ await runTurn({
+ models: [tierModel({ modelId: 'main', model: answeringModel('done') })],
+ fastModel: tierModel({ modelId: 'fast', model: searchingModel() }),
+ onStepModel,
+ })
+
+ expect(onStepModel.mock.calls.map(([turnModel]) => turnModel.modelId)).toEqual(['fast', 'main'])
+ })
+
+ it('does not retry a step that already sent text, even on the last model', async () => {
+ const main = textThenFailingModel()
+
+ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: main })] })
+
+ expect(turn.streamError).not.toBeNull()
+ expect(main.doStreamCalls).toHaveLength(1)
+ })
+
+ it('drops a failing fast model and answers on the same main model', async () => {
+ const turn = await runTurn({
+ models: [tierModel({ modelId: 'main', model: answeringModel('from main') }), tierModel({ modelId: 'backup', model: answeringModel('from backup') })],
+ fastModel: tierModel({ modelId: 'fast', model: failingModel({ statusCode: 429 }) }),
+ })
+
+ expect(turn.streamError).toBeNull()
+ expect(turn.answeredBy.modelId).toBe('main')
+ expect(textOf(turn)).toBe('from main')
+ })
+
+ it('drops a fast model that rejects the request outright, since the main model may still take it', async () => {
+ const turn = await runTurn({
+ models: [tierModel({ modelId: 'main', model: answeringModel('from main') })],
+ fastModel: tierModel({ modelId: 'fast', model: failingModel({ statusCode: 400 }) }),
+ })
+
+ expect(turn.streamError).toBeNull()
+ expect(textOf(turn)).toBe('from main')
+ })
+
+ it('fails on a bad request instead of trying the next model', async () => {
+ const backup = answeringModel('from backup')
+
+ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: failingModel({ statusCode: 400 }) }), tierModel({ modelId: 'backup', model: backup })] })
+
+ expect(turn.streamError).not.toBeNull()
+ expect(backup.doStreamCalls).toHaveLength(0)
+ expect(turn.allModelsFailed).toBe(false)
+ })
+
+ it('says every model failed when the whole tier is down', async () => {
+ const turn = await runTurn({ models: [tierModel({ modelId: 'main', model: failingModel({ statusCode: 503 }) }), tierModel({ modelId: 'backup', model: failingModel({ statusCode: 503 }) })] })
+
+ expect(turn.allModelsFailed).toBe(true)
+ })
+
+ it('builds the provider options for the model that runs, not the one that failed', async () => {
+ const sent: SharedV3ProviderOptions[] = []
+ const backup = answeringModel('from backup', sent)
+
+ await runTurn({
+ models: [
+ tierModel({ modelId: 'claude', model: failingModel({ statusCode: 503 }), provider: AIProviderName.ANTHROPIC }),
+ tierModel({ modelId: 'gpt', model: backup, provider: AIProviderName.OPENAI }),
+ ],
+ })
+
+ expect(sent[0]).not.toHaveProperty('anthropic')
+ })
+
+ it('reports a provider failure for the key that failed', async () => {
+ const onModelOutcome = vi.fn()
+
+ await runTurn({ models: [tierModel({ modelId: 'main', model: failingModel({ statusCode: 503 }) }), tierModel({ modelId: 'backup', model: answeringModel('ok') })], onModelOutcome })
+
+ expect(onModelOutcome).toHaveBeenCalledWith(expect.objectContaining({ turnModel: expect.objectContaining({ modelId: 'main' }), signal: expect.objectContaining({ statusCode: 503 }) }))
+ })
+
+ it('leaves a single specific model on today\'s path, with no fallback', async () => {
+ const turn = await runTurn({ models: [{ model: failingModel({ statusCode: 503 }), provider: AIProviderName.OPENAI, modelId: 'only', thinkingBudget: 0 }] })
+
+ expect(turn.streamError).not.toBeNull()
+ expect(turn.allModelsFailed).toBe(false)
+ })
+})
+
+describe('what counts as content sent in the step that failed', () => {
+ it('ignores the start markers the stream sends before any content', () => {
+ expect(sentContentAfter(['start', 'error'])).toBe(false)
+ })
+
+ it('counts a step that failed after a step that finished as clean', () => {
+ expect(sentContentAfter(['start', 'start-step', 'text-delta', 'tool-output-available', 'finish-step', 'error'])).toBe(false)
+ })
+
+ it('counts text sent before the error, even though the step is closed after it', () => {
+ expect(sentContentAfter(['start', 'start-step', 'text-start', 'text-delta', 'error', 'finish-step', 'finish'])).toBe(true)
+ })
+})
+
+async function runTurn({ models, fastModel, search, onStepModel, onModelOutcome }: {
+ models: [TurnModel, ...TurnModel[]]
+ fastModel?: TurnModel
+ search?: () => Promise<unknown>
+ onStepModel?: (turnModel: TurnModel) => void
+ onModelOutcome?: (outcome: unknown) => void
+}): ReturnType<typeof runAgentTurn> {
+ return runAgentTurn({
+ models,
+ ...(fastModel ? { fastModel } : {}),
+ systemPrompt: 'You are a test agent.',
+ messages: [{ role: 'user', content: 'hello' }],
+ tools: {
+ ap_web_search: tool({ description: 'search the web', inputSchema: z.object({ query: z.string() }), execute: search ?? (async () => ({ content: [] })) }),
+ },
+ allToolNames: ['ap_web_search'],
+ tier: TIER,
+ phaseState: { phase: 'discovery' },
+ abortSignal: new AbortController().signal,
+ log: SILENT_LOG,
+ sinks: { drainStream: drainLikeTheWorker },
+ ...(onStepModel ? { onStepModel } : {}),
+ ...(onModelOutcome ? { onModelOutcome } : {}),
+ })
+}
+
+async function drainLikeTheWorker(result: ReturnType<typeof streamText>): Promise<StreamDrain> {
+ let state: StepContentState = NO_STEP_CONTENT
+ const reader = toUIMessageStream({ stream: result.stream }).getReader()
+ for (let next = await reader.read(); !next.done; next = await reader.read()) {
+ const chunkType = isObject(next.value) && typeof next.value['type'] === 'string' ? next.value['type'] : undefined
+ state = trackStepContent({ state, chunkType })
+ }
+ return drainOf({ state })
+}
+
+function sentContentAfter(chunkTypes: string[]): boolean {
+ return drainOf({ state: chunkTypes.reduce((state, chunkType) => trackStepContent({ state, chunkType }), NO_STEP_CONTENT) }).lastStepSentContent
+}
+
+function tierModel({ modelId, model, provider = AIProviderName.OPENAI }: { modelId: string, model: MockLanguageModelV3, provider?: AIProviderName }): TurnModel {
+ return { model, provider, modelId, thinkingBudget: 0, key: { providerConfigId: `key-${modelId}`, status: 'active' } }
+}
+
+function failingModel({ statusCode }: { statusCode: number }): MockLanguageModelV3 {
+ return new MockLanguageModelV3({
+ doStream: async () => {
+ throw new APICallError({ message: `HTTP ${statusCode}`, url: 'https://llm.example/v1', requestBodyValues: {}, statusCode, isRetryable: false })
+ },
+ })
+}
+
+function answeringModel(text: string, sentOptions?: SharedV3ProviderOptions[]): MockLanguageModelV3 {
+ return new MockLanguageModelV3({
+ doStream: async ({ providerOptions }) => {
+ sentOptions?.push(providerOptions ?? {})
+ return {
+ stream: convertArrayToReadableStream([
+ { type: 'stream-start' as const, warnings: [] },
+ { type: 'text-start' as const, id: 't' },
+ { type: 'text-delta' as const, id: 't', delta: text },
+ { type: 'text-end' as const, id: 't' },
+ { type: 'finish' as const, finishReason: 'stop' as const, usage: USAGE },
+ ]),
+ }
+ },
+ })
+}
+
+function searchingModel(): MockLanguageModelV3 {
+ return new MockLanguageModelV3({
+ doStream: async () => ({
+ stream: convertArrayToReadableStream([
+ { type: 'stream-start' as const, warnings: [] },
+ { type: 'text-start' as const, id: 't' },
+ { type: 'text-delta' as const, id: 't', delta: 'Looking.' },
+ { type: 'text-end' as const, id: 't' },
+ { type: 'tool-call' as const, toolCallId: 'call-1', toolName: 'ap_web_search', input: '{"query":"x"}' },
+ { type: 'finish' as const, finishReason: 'tool-calls' as const, usage: USAGE },
+ ]),
+ }),
+ })
+}
+
+function textThenFailingModel(): MockLanguageModelV3 {
+ return new MockLanguageModelV3({
+ doStream: async () => ({
+ stream: convertArrayToReadableStream([
+ { type: 'stream-start' as const, warnings: [] },
+ { type: 'text-start' as const, id: 't' },
+ { type: 'text-delta' as const, id: 't', delta: 'Half an ans' },
+ { type: 'error' as const, error: new APICallError({ message: 'HTTP 503', url: 'https://llm.example/v1', requestBodyValues: {}, statusCode: 503, isRetryable: false }) },
+ ]),
+ }),
+ })
+}
+
+function textOf(turn: Awaited<ReturnType<typeof runAgentTurn>>): string {
+ return turn.uiParts.map((part) => isObject(part) && typeof part['text'] === 'string' ? part['text'] : '').join('')
+}
+
+const TIER = { id: 'fast', thinkingBudget: 0, modelId: 'main' }
+const USAGE = { inputTokens: 1, outputTokens: 1, totalTokens: 2 }
+const SILENT_LOG = { debug: () => undefined, info: () => undefined, warn: () => undefined, error: () => undefined }
diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-history.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-history.test.ts
index 3e7d1c8eede..3f898525f53 100644
--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-history.test.ts
+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-history.test.ts
@@ -10,8 +10,7 @@ describe('the history a turn leaves behind', () => {
const progress: ModelMessage[][] = []
const turn = await runAgentTurn({
- model: updateThenConfirm(),
- provider: AIProviderName.ANTHROPIC,
+ models: [{ model: updateThenConfirm(), provider: AIProviderName.ANTHROPIC, modelId: TIER.modelId, thinkingBudget: TIER.thinkingBudget }],
systemPrompt: 'You are a test agent.',
messages: [{ role: 'user', content: 'Change the instructions of New agent to: Reply in Arabic.' }],
tools: {
@@ -19,13 +18,12 @@ describe('the history a turn leaves behind', () => {
},
allToolNames: ['ap_update_agent'],
tier: TIER,
- modelId: TIER.modelId,
phaseState: { phase: 'build' },
abortSignal: new AbortController().signal,
log: SILENT_LOG,
creditsLeft: async () => 100,
sinks: {
- drainStream: (result) => result.consumeStream(),
+ drainStream: async (result) => { await result.consumeStream(); return undefined },
onProgress: ({ responseMessages }) => progress.push(responseMessages),
},
})
@@ -39,8 +37,7 @@ describe('the history a turn leaves behind', () => {
let saves = 0
const turn = await runAgentTurn({
- model: keepsUpdating(),
- provider: AIProviderName.ANTHROPIC,
+ models: [{ model: keepsUpdating(), provider: AIProviderName.ANTHROPIC, modelId: TIER.modelId, thinkingBudget: TIER.thinkingBudget }],
systemPrompt: 'You are a test agent.',
messages: [{ role: 'user', content: 'Update New agent three times.' }],
tools: {
@@ -58,12 +55,11 @@ describe('the history a turn leaves behind', () => {
},
allToolNames: ['ap_update_agent'],
tier: TIER,
- modelId: TIER.modelId,
phaseState: { phase: 'build' },
abortSignal: controller.signal,
log: SILENT_LOG,
creditsLeft: async () => 100,
- sinks: { drainStream: (result) => result.consumeStream() },
+ sinks: { drainStream: async (result) => { await result.consumeStream(); return undefined } },
})
const toolCalls = turn.accumulatedResponseMessages.flatMap((message) => typeof message.content === 'string'