AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
feat(ai): platform tiers follow key project scope (#16197)
Louai Boumedienee565763a8aca
diff --git a/brain/knowledge/ai-intelligence/ai-agents.md b/brain/knowledge/ai-intelligence/ai-agents.mdindex d554a59c8ed..cf6ba04b08c 100644--- a/brain/knowledge/ai-intelligence/ai-agents.md+++ b/brain/knowledge/ai-intelligence/ai-agents.md@@ -80,7 +80,7 @@ Three things follow. `run_agent` has two modes: pointed at a saved agent (`agent- **Do not raise `ai` past 7.0.69 — from `[email protected]` the SDK stops auto-executing tools, and it fails silently.** Bisected on 2026-09-21 while trying to reach `experimental_evaluate`. On 7.0.44 (what the repo pins) and every version through 7.0.69, a tool call executes normally. From **7.0.70** onward, including 7.0.103–7.0.107, `generateText`/`streamText` still *recognise* the call — `staticToolCalls=1`, a `tool-call` part is emitted with correctly parsed input — but `execute` is never invoked and `toolResults` stays `0`. No error, no warning. The changelog entry for 7.0.70 is `a828527: Prevent automatic tool execution when a model call ends with an unsafe finish reason`, but the gate fires for every `finishReason` tried, `tool-calls` and `stop` included. Reproduced in a clean project outside the monorepo, on both `MockLanguageModelV3` and `MockLanguageModelV4`, on zod 3 and zod 4; `needsApproval: false` and `toolApproval: () => false` do not restore it, and neither does emitting the full `tool-input-start`/`delta`/`end` lifecycle before the `tool-call`. The alarm is `packages/server/worker/test/lib/agent-eval/agent-tool-call-repair.test.ts` — 4 of its 5 tests go red, and they look like a *repair* regression, which is the wrong trail: a plain valid tool call does not execute either. **Consequence: `experimental_evaluate` (first shipped in 7.0.103) cannot be reached without breaking the agent**, so anything needing typed evaluation calls the gateway's `/v4/ai/evaluation-model` endpoint over `safeHttp` instead of taking the bump. Re-test the table above before any future `ai` upgrade; if a release fixes it, that is the one to jump to.-- **A platform tier turn falls back per step, not per turn.** When a step fails on a provider error (`isFallbackWorthy`) before sending content, the turn continues on the next candidate from the steps already done, the way `continue_truncation` does. Tool results stay in the messages, so nothing runs twice. A fast step that sent nothing is dropped on **any** error, not only fallback-worthy ones: the Fast tier can sit on a smaller context window than the main tier (Haiku 200K under a Gemini 1M tier), and once the history outgrows it the fast model answers 400 "prompt is too long", which must not sink a turn the main model can still take. For the same reason compaction sizes its budget over the tier's candidates **and** the Fast tier, not the main tier alone. After any switch the turn stays on that candidate and has no fast model. A step that already sent text never falls back, and is not retried on the same model either (`shouldRetryStream` also reads the drain). Piece tools follow the model **running the current step** (`onStepModel` from `prepareStep`), so a fast step's tool calls use the fast key. The worker sends `answeredBy` on save, and billing labels the turn with it. Search and image stay on the pair they were resolved with, and knowledge-base search stays on the chat key, because the chunks were embedded with it.+- **A platform tier turn falls back per step, not per turn.** When a step fails on a provider error (`isFallbackWorthy`) before sending content, the turn continues on the next candidate from the steps already done, the way `continue_truncation` does. Tool results stay in the messages, so nothing runs twice. A fast step that sent nothing is dropped on **any** error, not only fallback-worthy ones: the Fast tier can sit on a smaller context window than the main tier (Haiku 200K under a Gemini 1M tier), and once the history outgrows it the fast model answers 400 "prompt is too long", which must not sink a turn the main model can still take. For the same reason compaction sizes its budget over the tier's candidates **and** the Fast tier, not the main tier alone. After any switch the turn stays on that candidate and has no fast model. A step that already sent text never falls back, and is not retried on the same model either (`shouldRetryStream` also reads the drain). Piece tools follow the model **running the current step** (`onStepModel` from `prepareStep`), so a fast step's tool calls use the fast key. The worker sends `answeredBy` on save, and billing labels the turn with it. `answeredBy` is the model of the turn's **last** step, so a turn that ends on the fast step is billed under the fast tier's key, not the main one. Search and image stay on the pair they were resolved with, and knowledge-base search stays on the chat key, because the chunks were embedded with it.- **"Did the failing step send anything" is read off the UI stream, and `producedVisibleOutput` cannot answer it.** `uiParts` only grows in `onStepEnd`, after chunks have already reached the client. The UI stream sends `start`/`start-step` before any content. When the model call itself throws, there is no `start-step` at all, only `error`. A stream error arrives **before** `finish-step` (`text-delta, error, finish-step`). `trackStepContent` therefore resets on `start-step`/`finish-step` and freezes its answer at the first `error` chunk. Drive it through the real `toUIMessageStream` in tests, because a mocked drain hides the ordering.- **`AgentConfigResponse.tier` is always the console credits tier.** A platform tier run fills `platformTier`, `candidates` and `fastCandidate`. The single `credentials`/`modelId` fields equal `candidates[0]`. On a tier run the worker sends no `modelName` to `saveAgentMessages`, and the RPC ignores one on a tier row, because a model name clears the tier (the last write wins).diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.mdindex 34c0be786c9..49b83f4c431 100644--- a/brain/knowledge/ai-intelligence/ai-providers.md+++ b/brain/knowledge/ai-intelligence/ai-providers.md@@ -70,15 +70,15 @@ decision 000041. The console publishes both lists and both fast pointers on everafter a file without chat keys it seeds the chat list from the live flow list.### Platform tiers-**Platform tier** — an admin-built, platform-owned list of up to 5 `{configId, modelId}` entries (main + ranked fallbacks, any keys), table `platform_model_tier`, API `/v1/platform-model-tiers`. A tier grants its entries to every project; see decision [000044](../decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.md). _Avoid_: "model tier" alone — that is the console **credits tier** (`ModelTier`, above).+**Platform tier** — an admin-built, platform-owned list of up to 5 `{configId, modelId}` entries (main + ranked fallbacks, any keys), table `platform_model_tier`, API `/v1/platform-model-tiers`. A tier follows key scope: in a project it runs only the entries whose key serves that project, and it is listed there only when its main entry can run; see decision [000048](../decisions/000048-platform-tiers-follow-key-project-scope.md) (replaces 000044's grant). _Avoid_: "model tier" alone — that is the console **credits tier** (`ModelTier`, above).### Gotchas- **A key a live platform tier uses cannot be deleted, and its `modelScope` cannot drop a tier's model.** `aiProviderService.delete` and `update` call `platformModelTierService.assertKeyCanBeDeleted` / `assertKeyScopeKeepsTiers` under a `platform` row `FOR NO KEY UPDATE` lock and answer 409 with the tier names. The same lock serialises every tier write, so a tier save can't race a key delete into a dangling entry.- **Deleting a platform tier needs a replacement, except the last live tier, and there is no usage count.** Refs to a deleted tier are never rewritten; they reach a live tier through `replacedBy` (flattened to one hop on delete). The last tier may always go with no replacement: without that exit, a platform's last tier and the key it uses could never be removed, because the key guard refuses keys a live tier uses. Tier delete is also deliberately **not** plan-gated (writes are), so a platform that downgrades can still clear its tiers and then remove its key. Its refs then point at a tier with `replacedBy: null`, and the resolver must fail them with a clear "tier removed" error. An exact "used by N flows" count needs a `LIKE` scan over every flow version's `trigger` text, plus agent jsonb and conversations, on the whole platform. We dropped it rather than pay that. If counts come back, add a denormalized `flow_version.modelTierIds` array + GIN index (the `connectionIds` pattern) instead of scanning.- **The live-name unique index is `lower("name")`, which `EntitySchema` cannot express.** The entity declares it `synchronize: false` (so `check-migrations` skips it), which means test DBs built by `synchronize` don't have it. The case-insensitive check that tests rely on is the service's `assertNameIsFree`, which is race-free because it runs under the `platform` row lock; the index is the backstop.- **`lockPlatform` takes `FOR NO KEY UPDATE`, not `FOR UPDATE`.** Every FK insert that references `platform` (projects, users, …) takes `FOR KEY SHARE` on the platform row, which `FOR UPDATE` blocks until commit. `FOR NO KEY UPDATE` serialises tier writes just the same without stalling those inserts.-- **The tier grant lives in one function: `aiModelCandidates.resolve` (via `platformModelTierService.getForRun`) is the only path that hands out a key without `rowAllowsScope`.** The `configId`s come from a live tier on the caller's platform, never from the caller; keep it that way. A specific pick still goes through `getConfigOrThrow` with the project scope.-- **Agent and chat runs read the tier grant from the conversation row, never from the worker.** CHAT and AGENT_BUILDER take `modelTierId` from the row (set by create/update). AGENT and FLOW_STEP take it from the job and write it onto the row **after** `acquireStreamingLock`, including null. Tool RPCs then call `aiModelCandidates.grantedEntryConfig`, which honours a worker-sent `providerConfigId` only as an exact `{configId, modelId}` entry of that tier or of the Fast tier. A tier row also skips `assertProjectSwitchKeepsKey`. Any other `providerConfigId` goes through the normal scope check.+- **Tier entries go through the same project check as specific picks.** `platformModelTierService.getForRun` takes a `ProviderScope` and applies `aiKeyScope.rowAllowsScope` + `scopeAllows` to every entry; if the stored main entry fails, the whole tier is unavailable (it is checked before the healthy-first sort, so a down main doesn't change the answer). The `configId`s still come from a live tier on the caller's platform, never from the caller. Only the label lookups (`getManyForRun`, `firstCandidates`) use platform scope, because they return no credentials.+- **Agent and chat runs read the tier grant from the conversation row, never from the worker.** CHAT and AGENT_BUILDER take `modelTierId` from the row (set by create/update). AGENT and FLOW_STEP take it from the job and write it onto the row **after** `acquireStreamingLock`, including null. Tool RPCs then call `aiModelCandidates.grantedEntryConfig` with the run's project, which honours a worker-sent `providerConfigId` only as an exact `{configId, modelId}` entry of that tier or of the Fast tier that runs in that project. A tier row switches projects through `agentModelTier.assertProjectSwitchKeepsTier`: every key the run can use (after the tool-calling filter, Fast tier included) must serve the new project. Clearing the project counts as moving to *every* project, so it needs keys open to all projects; a switch from no project checks the tier's platform-wide entries, because nothing records which keys the run kept. Any other `providerConfigId` goes through the normal scope check.- **A platform tier step falls back only on provider failures, decided by `isFallbackWorthy` on a signal from `aiProviderSignal.fromError`.** 401/402/403/404/408/429/5xx, network errors (ai 7 throws them as `APICallError` with **no status** and `isRetryable: true`) and billing bodies (Anthropic sends low credit as a **400**) fall back; any other 4xx and our own errors (classify mismatch, no structured data, web search unsupported) stop. No fallback once a web-search step finished, since it was already billed.- **Worker-built models report no key health through `createModel`: no reporter is installed in the worker.** Tier steps report explicitly via the `reportAiKeyOutcome` RPC, and only provider signals (`fromProvider`) plus recoveries of non-active keys. Reporting our own errors would demote healthy keys to `unreachable`. The API never writes a worker-reported status for most providers: `confirmReportedOutcome` runs in the background and only triggers a `recheck` of the real key (once per key per 10s) when the report disagrees with the stored status. The exception is providers flagged `validationSkipsModelEndpoint` (CUSTOM, Vertex): their check never reaches the model, so a recheck would always pass, and the report is recorded as-is. Any direct `aiProviderHealth.record` from concurrent callers must pass `expectVersion` (the `statusVersion` it read), or an older report landing late overwrites a newer one. The worker token is shared and a shared worker serves every platform, so a report can't be bound to a platform; a forged one can at most cause a recheck. Don't pass `platformId`/`providerConfigId` to `createModel` in the worker, or a future reporter double-counts.- **`ExecuteAiJobData` is a zod schema that strips unknown keys** (`JobData.parse` in `worker.ts` and `job-broker.ts`). A field added only to the TS type silently never reaches the worker; add it to `AiStepJobBase`.diff --git a/brain/knowledge/decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.md b/brain/knowledge/decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.mdindex 11b2ff7e603..c2bad7acda1 100644--- a/brain/knowledge/decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.md+++ b/brain/knowledge/decisions/000044-platform-tiers-are-platform-owned-and-grant-key-access.md@@ -1,5 +1,5 @@----status: proposed+status: superseded by 000048-platform-tiers-follow-key-project-scope---# Platform tiers are platform-owned and grant key accessdiff --git a/brain/knowledge/decisions/000048-platform-tiers-follow-key-project-scope.md b/brain/knowledge/decisions/000048-platform-tiers-follow-key-project-scope.mdnew file mode 100644index 00000000000..4ca2e395ed4--- /dev/null+++ b/brain/knowledge/decisions/000048-platform-tiers-follow-key-project-scope.md@@ -0,0 +1,20 @@+---+status: proposed+---++# Platform tiers follow key project scope++## Decision+A platform tier runs in project P only the entries whose key serves P (the key's project scope) and allows the model (its model scope). The tier is available in P only when its main entry, the stored `entries[0]`, can run there; otherwise P's pickers don't list it, and a stored ref fails with "This tier isn't available in this project". Chat, agents and the agent builder also skip any entry the model catalog says can't call tools, and refuse a tier whose main model can't.++## Context+Decision 000044 let a tier grant its entries to every project, so a key limited to some projects ran everywhere once it sat in a tier. The key page still said "Only these projects can use this key", and nothing warned the admin. Product's call: per-project model scoping matters more than tiers, so tiers follow it.++## Why+One rule for every path (flow steps, agents, chat, tools, project switch) is easier to trust than a grant with exceptions. Gating on the main model keeps a tier's name honest: "Expert" means the same main model in every project that sees it. We rejected giving each tier its own project scope: that would be a second scope system that still contradicts the key's.++## Consequences+- Narrowing a key's projects can hide a tier from those projects. The admin UI shows the impact before saving.+- The project-scope check is shared in `ai-key-scope.ts` (`aiKeyScope.rowAllowsScope`), used by both `aiProviderService` and `platformModelTierService`.+- A tier conversation can switch projects only if every key the run could fall back to (the Fast tier's too) serves the new project.+- Labels for billing and analytics still read the tier's main model without a project: they hand out no credentials.diff --git a/packages/server/api/src/app/ai/ai-execute-controller.ts b/packages/server/api/src/app/ai/ai-execute-controller.tsindex 24c5c5a0ef3..dae0921376c 100644--- a/packages/server/api/src/app/ai/ai-execute-controller.ts+++ b/packages/server/api/src/app/ai/ai-execute-controller.ts@@ -29,7 +29,7 @@ export const aiExecuteController: FastifyPluginAsyncZod = async (app) => {const requestId = apId()const log = request.log.child({ flowRun: { id: body.flowRunId }, requestId })const execution = aiExecution(log)- const model = await pickModel({ body, platformId: platform.id, log })+ const model = await pickModel({ body, platformId: platform.id, projectId, log })const answerInThisRequest = isNil(body.waitpointId)const timeoutMs = system.getNumberOrThrow(AppSystemProp.FLOW_TIMEOUT_SECONDS) * 1000const answer = answerInThisRequest ? execution.waitForAnswer({ requestId, timeoutMs }) : undefined@@ -56,16 +56,17 @@ export const aiExecuteController: FastifyPluginAsyncZod = async (app) => {})}-async function pickModel({ body, platformId, log }: {+async function pickModel({ body, platformId, projectId, log }: {body: z.infer<typeof ExecuteAiRequest>platformId: string+ projectId: stringlog: FastifyBaseLogger}): Promise<PickedModel> {if (!isNil(body.modelTierId)) {if (body.action === AiStepAction.GENERATE_IMAGE) {throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: 'Image steps pick a specific model, not a tier' } })}- return aiModelCandidates(log).firstCandidate({ platformId, tierId: body.modelTierId })+ return aiModelCandidates(log).firstCandidate({ platformId, tierId: body.modelTierId, scope: { type: 'project', projectId } })}if (isNil(body.provider) || isNil(body.modelId)) {throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: 'Pick a tier or a provider and model' } })diff --git a/packages/server/api/src/app/ai/ai-key-scope.ts b/packages/server/api/src/app/ai/ai-key-scope.tsnew file mode 100644index 00000000000..6a82a832ff9--- /dev/null+++ b/packages/server/api/src/app/ai/ai-key-scope.ts@@ -0,0 +1,25 @@+import { AiProviderModelScope } from '@activepieces/shared'+import { AIProviderSchema } from './ai-provider-entity'++export const aiKeyScope = {+ rowAllowsScope({ row, scope }: { row: Pick<AIProviderSchema, 'projectScope' | 'projectIds'>, scope: ProviderScope }): boolean {+ if (scope.type === 'platform') {+ return true+ }+ switch (row.projectScope) {+ case 'selected':+ return row.projectIds.includes(scope.projectId)+ case 'except':+ return !row.projectIds.includes(scope.projectId)+ default:+ return true+ }+ },+ scopeAllows({ modelScope, modelIds, modelId }: { modelScope: AiProviderModelScope, modelIds: string[], modelId: string }): boolean {+ return modelScope !== 'selected' || modelIds.includes(modelId)+ },+}++export type ProviderScope =+ | { type: 'project', projectId: string }+ | { type: 'platform' }diff --git a/packages/server/api/src/app/ai/ai-model-candidates.ts b/packages/server/api/src/app/ai/ai-model-candidates.tsindex 8ee546de98e..d2cfdf047b6 100644--- a/packages/server/api/src/app/ai/ai-model-candidates.ts+++ b/packages/server/api/src/app/ai/ai-model-candidates.ts@@ -1,12 +1,13 @@import { ActivepiecesError, AiProviderKeyStatus, AIProviderName, ErrorCode, isNil, PlatformId, tryCatch } from '@activepieces/core-utils'import { GetProviderConfigResponse, ResolveAiModelCandidatesResponse } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'+import { ProviderScope } from './ai-key-scope'import { aiProviderService } from './ai-provider-service'import { platformModelTierService, TierForRun } from './platform-model-tier-service'export const aiModelCandidates = (log: FastifyBaseLogger) => ({- async firstCandidate({ platformId, tierId }: { platformId: PlatformId, tierId: string }): Promise<FirstCandidate> {- const { entries: [first] } = await platformModelTierService.getForRun({ platformId, id: tierId })+ async firstCandidate({ platformId, tierId, scope }: { platformId: PlatformId, tierId: string, scope: ProviderScope }): Promise<FirstCandidate> {+ const { entries: [first] } = await platformModelTierService.getForRun({ platformId, id: tierId, scope })return { provider: first.key.provider, providerConfigId: first.key.id, modelId: first.modelId }},@@ -18,36 +19,31 @@ export const aiModelCandidates = (log: FastifyBaseLogger) => ({}))},- async resolve({ platformId, tierId }: { platformId: PlatformId, tierId: string }): Promise<ResolveAiModelCandidatesResponse> {- const { tier, candidates } = await this.resolveConfigs({ platformId, tierId })+ async resolve({ platformId, tierId, scope }: { platformId: PlatformId, tierId: string, scope: ProviderScope }): Promise<ResolveAiModelCandidatesResponse> {+ const { tier, candidates } = await this.resolveConfigs({ platformId, tierId, scope })return {tierName: tier.name,candidates: candidates.map(({ config, modelId, status }) => ({ ...config, providerConfigId: config.configId, modelId, status })),}},- async resolveConfigs({ platformId, tierId }: { platformId: PlatformId, tierId: string }): Promise<TierConfigs> {- const forRun = await platformModelTierService.getForRun({ platformId, id: tierId })- const candidates = await readableCandidates({ platformId, forRun, log })- if (candidates.length === 0) {+ async resolveConfigs({ platformId, tierId, scope }: { platformId: PlatformId, tierId: string, scope: ProviderScope }): Promise<TierConfigs> {+ const forRun = await platformModelTierService.getForRun({ platformId, id: tierId, scope })+ const configs = await this.configsFor({ platformId, forRun })+ if (configs.candidates.length === 0) {throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: `No model in tier "${forRun.tier.name}" can run` } })}- return { tier: forRun.tier, candidates }+ return configs},- async resolveFastConfig({ platformId }: { platformId: PlatformId }): Promise<TierConfigs | null> {- const forRun = await platformModelTierService.getFastForRun({ platformId })- if (isNil(forRun)) {- return null- }- const candidates = await readableCandidates({ platformId, forRun, log })- return candidates.length === 0 ? null : { tier: forRun.tier, candidates }+ async configsFor({ platformId, forRun }: { platformId: PlatformId, forRun: TierForRun }): Promise<TierConfigs> {+ return { tier: forRun.tier, candidates: await readableCandidates({ platformId, forRun, log }) }},- async grantedEntryConfig({ platformId, tierId, configId, modelId }: { platformId: PlatformId, tierId: string, configId: string, modelId: string }): Promise<GetProviderConfigResponse | null> {+ async grantedEntryConfig({ platformId, tierId, configId, modelId, scope }: { platformId: PlatformId, tierId: string, configId: string, modelId: string, scope: ProviderScope }): Promise<GetProviderConfigResponse | null> {const [run, fast] = await Promise.all([- platformModelTierService.getForRun({ platformId, id: tierId }),- platformModelTierService.getFastForRun({ platformId }),+ platformModelTierService.getForRun({ platformId, id: tierId, scope }),+ platformModelTierService.getFastForRun({ platformId, scope }),])const entry = [...run.entries, ...(fast?.entries ?? [])].find((candidate) => candidate.key.id === configId && candidate.modelId === modelId)return isNil(entry) ? null : aiProviderService(log).configForTierKey({ platformId, key: entry.key })diff --git a/packages/server/api/src/app/ai/ai-provider-service.ts b/packages/server/api/src/app/ai/ai-provider-service.tsindex 0ed8ac75cf7..a5391120bb8 100644--- a/packages/server/api/src/app/ai/ai-provider-service.ts+++ b/packages/server/api/src/app/ai/ai-provider-service.ts@@ -12,6 +12,7 @@ import { openRouterApi } from '../ee/platform/platform-plan/openrouter/openrouteimport { flagService } from '../flags/flag.service'import { EncryptedObject, encryptUtils } from '../helper/encryption'import { platformService } from '../platform/platform.service'+import { aiKeyScope, ProviderScope } from './ai-key-scope'import { AIProviderEntity, AIProviderSchema } from './ai-provider-entity'import { aiProviderHealth } from './ai-provider-health'import { platformModelTierService } from './platform-model-tier-service'@@ -47,7 +48,7 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({async listForProject({ platformId, projectId }: { platformId: PlatformId, projectId: string }): Promise<ProjectAIProvider[]> {const rows = await listVisibleRows({ platformId, log })- const eligible = rows.filter((row) => rowAllowsScope({ row, scope: { type: 'project', projectId } }))+ const eligible = rows.filter((row) => aiKeyScope.rowAllowsScope({ row, scope: { type: 'project', projectId } }))const ranked = rankRows(eligible)const chatRow = pickChatRow(ranked)return unique(ranked.map((row) => row.provider)).map((provider) => {@@ -183,12 +184,12 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({const candidates = await findRunKeyCandidates({ platformId, provider, configId, scope: resolvedFor, log })return candidates.every((row) => target.type === 'platform'? row.projectScope === 'all'- : rowAllowsScope({ row, scope: target }))+ : aiKeyScope.rowAllowsScope({ row, scope: target }))},async exists({ platformId, provider, scope, configId }: { platformId: PlatformId, provider: AIProviderName, scope: ProviderScope, configId?: string }): Promise<boolean> {const rows = await aiProviderRepo().findBy({ platformId, provider })- return rows.some((row) => rowAllowsScope({ row, scope }) && (isNil(configId) || row.id === configId))+ return rows.some((row) => aiKeyScope.rowAllowsScope({ row, scope }) && (isNil(configId) || row.id === configId))},async delete(platformId: PlatformId, providerId: string): Promise<void> {@@ -311,20 +312,6 @@ const PROJECT_SCOPE_SPECIFICITY: Record<AiProviderProjectScope, number> = {all: 2,}-function rowAllowsScope({ row, scope }: { row: AIProviderSchema, scope: ProviderScope }): boolean {- if (scope.type === 'platform') {- return true- }- switch (row.projectScope) {- case 'selected':- return row.projectIds.includes(scope.projectId)- case 'except':- return !row.projectIds.includes(scope.projectId)- default:- return true- }-}-function rankRows(rows: AIProviderSchema[]): AIProviderSchema[] {return [...rows].sort((a, b) => {const specificityDelta = PROJECT_SCOPE_SPECIFICITY[a.projectScope] - PROJECT_SCOPE_SPECIFICITY[b.projectScope]@@ -406,7 +393,7 @@ async function findRunKeyCandidates({ platformId, provider, configId, scope, logasync function findEligibleRow({ platformId, provider, scope }: { platformId: PlatformId, provider: AIProviderName, scope: ProviderScope }): Promise<AIProviderSchema | null> {const rows = await aiProviderRepo().findBy({ platformId, provider })- const eligible = rows.filter((row) => rowAllowsScope({ row, scope }))+ const eligible = rows.filter((row) => aiKeyScope.rowAllowsScope({ row, scope }))return rankRows(eligible)[0] ?? null}@@ -431,7 +418,7 @@ async function resolveRowForScope({ platformId, provider, scope, configId }: { preturn resolveEligibleRow({ platformId, provider, scope })}const rows = await aiProviderRepo().findBy({ platformId, provider })- const row = rows.find((candidate) => candidate.id === configId && rowAllowsScope({ row: candidate, scope }))+ const row = rows.find((candidate) => candidate.id === configId && aiKeyScope.rowAllowsScope({ row: candidate, scope }))if (isNil(row)) {throw new ActivepiecesError({code: ErrorCode.ENTITY_NOT_FOUND,@@ -517,7 +504,7 @@ async function findAvailableChatProviderRow({ platformId, scope, log }: { platfo{ platformId, enabledForChat: true },{ platformId, provider: AIProviderName.ACTIVEPIECES },])- const rows = candidates.filter((row) => rowAllowsScope({ row, scope }))+ const rows = candidates.filter((row) => aiKeyScope.rowAllowsScope({ row, scope }))const chatRow = pickChatRow(rows)if (chatRow?.provider !== AIProviderName.ACTIVEPIECES) {return chatRow@@ -634,6 +621,4 @@ function getModelsCacheKey({ provider, auth, config }: { provider: AIProviderNamreturn `${provider}-${JSON.stringify(auth)}-${JSON.stringify(config)}`}-export type ProviderScope =- | { type: 'project', projectId: string }- | { type: 'platform' }+export type { ProviderScope }diff --git a/packages/server/api/src/app/ai/ai-rpc-handlers.ts b/packages/server/api/src/app/ai/ai-rpc-handlers.tsindex c6bb776fd45..2ee2045cadb 100644--- a/packages/server/api/src/app/ai/ai-rpc-handlers.ts+++ b/packages/server/api/src/app/ai/ai-rpc-handlers.ts@@ -30,7 +30,7 @@ export const aiRpcHandlers = (log: FastifyBaseLogger) => ({async resolveAiModelCandidates(input: ResolveAiModelCandidatesRequest): Promise<ResolveAiModelCandidatesResponse> {await assertProjectBelongsToPlatform({ ...input, log })- return aiModelCandidates(log).resolve({ platformId: input.platformId, tierId: input.modelTierId })+ return aiModelCandidates(log).resolve({ platformId: input.platformId, tierId: input.modelTierId, scope: { type: 'project', projectId: input.projectId } })},async reportAiKeyOutcome(input: ReportAiKeyOutcomeRequest): Promise<void> {diff --git a/packages/server/api/src/app/ai/platform-model-tier-controller.ts b/packages/server/api/src/app/ai/platform-model-tier-controller.tsindex 548c0919275..47c16f67d63 100644--- a/packages/server/api/src/app/ai/platform-model-tier-controller.ts+++ b/packages/server/api/src/app/ai/platform-model-tier-controller.ts@@ -11,7 +11,7 @@ import { platformModelTierService } from './platform-model-tier-service'export const platformModelTierController: FastifyPluginAsyncZod = async (app) => {app.get('/', ListPlatformModelTiers, async (request) => {- return platformModelTierService.listSummaries({ platformId: request.principal.platform.id })+ return platformModelTierService.listSummaries({ platformId: request.principal.platform.id, scope: { type: 'project', projectId: request.projectId } })})app.get('/admin', ListPlatformModelTiersForAdmin, async (request) => {return platformModelTierService.list({ platformId: request.principal.platform.id })diff --git a/packages/server/api/src/app/ai/platform-model-tier-service.ts b/packages/server/api/src/app/ai/platform-model-tier-service.tsindex 557f6ff640c..c1fd4b81438 100644--- a/packages/server/api/src/app/ai/platform-model-tier-service.ts+++ b/packages/server/api/src/app/ai/platform-model-tier-service.ts@@ -4,6 +4,7 @@ import { EntityManager, In } from 'typeorm'import { repoFactory } from '../core/db/repo-factory'import { transaction } from '../core/db/transaction'import { isUniqueViolation } from '../core/db/unique-violation'+import { aiKeyScope, ProviderScope } from './ai-key-scope'import { AIProviderEntity, AIProviderSchema } from './ai-provider-entity'import { PlatformModelTierEntity, PlatformModelTierSchema } from './platform-model-tier-entity'@@ -13,14 +14,19 @@ const aiProviderRepo = repoFactory<AIProviderSchema>(AIProviderEntity)const MAX_LIVE_TIERS = 50const MAX_REPLACEMENT_HOPS = 3const TIER_REMOVED_MESSAGE = 'This tier was removed. Pick a new model for this step.'+const TIER_NOT_AVAILABLE_MESSAGE = 'This tier isn\'t available in this project. Pick another model.'+const PLATFORM_SCOPE: ProviderScope = { type: 'platform' }export const platformModelTierService = {- async listSummaries({ platformId }: { platformId: PlatformId }): Promise<PlatformModelTierSummary[]> {+ async listSummaries({ platformId, scope }: { platformId: PlatformId, scope: ProviderScope }): Promise<PlatformModelTierSummary[]> {const tiers = await listLive({ platformId })const mainConfigIds = unique(tiers.flatMap((tier) => tier.entries.slice(0, 1).map((entry) => entry.configId)))const keys = mainConfigIds.length === 0 ? [] : await aiProviderRepo().findBy({ platformId, id: In(mainConfigIds) })+ const keyById = new Map(keys.map((key) => [key.id, key]))const providerByConfigId = new Map(keys.map((key) => [key.id, key.provider]))- return tiers.map((tier) => toSummary({ tier, providerByConfigId }))+ return tiers+ .filter((tier) => mainRunsIn({ tier, keyById, scope }))+ .map((tier) => toSummary({ tier, providerByConfigId }))},async list({ platformId }: { platformId: PlatformId }): Promise<PlatformModelTier[]> {@@ -146,20 +152,24 @@ export const platformModelTierService = {}},- async getFastForRun({ platformId }: { platformId: PlatformId }): Promise<TierForRun | null> {+ async getFastForRun({ platformId, scope }: { platformId: PlatformId, scope: ProviderScope }): Promise<TierForRun | null> {const fastTier = await tierRepo().findOneBy({ platformId, isFast: true })if (isNil(fastTier)) {return null}- const { data } = await tryCatch(() => platformModelTierService.getForRun({ platformId, id: fastTier.id }))+ const { data } = await tryCatch(() => platformModelTierService.getForRun({ platformId, id: fastTier.id, scope }))return data ?? null},- async getForRun({ platformId, id }: { platformId: PlatformId, id: string }): Promise<TierForRun> {+ async getForRun({ platformId, id, scope }: { platformId: PlatformId, id: string, scope: ProviderScope }): Promise<TierForRun> {const tier = await followReplacements({ platformId, id })const configIds = unique(tier.entries.map((entry) => entry.configId))const keys = await aiProviderRepo().findBy({ platformId, id: In(configIds) })- const entries = runnableEntries({ tier, keyById: new Map(keys.map((key) => [key.id, key])) })+ const keyById = new Map(keys.map((key) => [key.id, key]))+ if (!mainRunsIn({ tier, keyById, scope })) {+ throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: TIER_NOT_AVAILABLE_MESSAGE } })+ }+ const entries = runnableEntries({ tier, keyById, scope })if (entries.length === 0) {throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: `No model in tier "${tier.name}" can run` } })}@@ -173,7 +183,7 @@ export const platformModelTierService = {const keys = configIds.length === 0 ? [] : await aiProviderRepo().findBy({ platformId, id: In(configIds) })const keyById = new Map(keys.map((key) => [key.id, key]))return new Map([...liveByRequestedId].map(([requestedId, tier]) => {- const entries = isNil(tier) ? [] : runnableEntries({ tier, keyById })+ const entries = isNil(tier) ? [] : runnableEntries({ tier, keyById, scope: PLATFORM_SCOPE })return [requestedId, isNil(tier) || entries.length === 0 ? null : { tier, entries }]}))},@@ -186,7 +196,7 @@ export const platformModelTierService = {}const nextScope = { modelScope: modelScope ?? key.modelScope, modelIds: modelIds ?? key.modelIds }const tiers = await findLiveTiersUsingKey({ manager, platformId, configId })- const broken = tiers.filter((tier) => tier.entries.some((entry) => entry.configId === configId && !scopeAllows({ ...nextScope, modelId: entry.modelId })))+ const broken = tiers.filter((tier) => tier.entries.some((entry) => entry.configId === configId && !aiKeyScope.scopeAllows({ ...nextScope, modelId: entry.modelId })))if (broken.length > 0) {throw keyInUseError({ tierNames: broken.map((tier) => tier.name) })}@@ -203,13 +213,22 @@ async function lockPlatform({ manager, platformId }: { manager: EntityManager, p}}-function runnableEntries({ tier, keyById }: { tier: PlatformModelTier, keyById: Map<string, AIProviderSchema> }): TierForRun['entries'] {+function mainRunsIn({ tier, keyById, scope }: { tier: PlatformModelTier, keyById: Map<string, AIProviderSchema>, scope: ProviderScope }): boolean {+ const main = tier.entries[0]+ return !isNil(main) && entryRunsIn({ entry: main, key: keyById.get(main.configId), scope })+}++function entryRunsIn({ entry, key, scope }: { entry: PlatformModelTierEntry, key: AIProviderSchema | undefined, scope: ProviderScope }): boolean {+ return !isNil(key)+ && key.provider !== AIProviderName.ACTIVEPIECES+ && aiKeyScope.scopeAllows({ modelScope: key.modelScope, modelIds: key.modelIds, modelId: entry.modelId })+ && aiKeyScope.rowAllowsScope({ row: key, scope })+}++function runnableEntries({ tier, keyById, scope }: { tier: PlatformModelTier, keyById: Map<string, AIProviderSchema>, scope: ProviderScope }): TierForRun['entries'] {const runnable = tier.entries.flatMap((entry) => {const key = keyById.get(entry.configId)- if (isNil(key) || key.provider === AIProviderName.ACTIVEPIECES || !scopeAllows({ modelScope: key.modelScope, modelIds: key.modelIds, modelId: entry.modelId })) {- return []- }- return [{ modelId: entry.modelId, key }]+ return isNil(key) || !entryRunsIn({ entry, key, scope }) ? [] : [{ modelId: entry.modelId, key }]})return [...runnable.filter((entry) => entry.key.status === 'active'),@@ -294,7 +313,7 @@ async function assertEntriesValid({ manager, platformId, entries }: { manager: Eif (key.provider === AIProviderName.ACTIVEPIECES) {throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: 'Tiers cannot use the Activepieces credits key' } })}- if (!scopeAllows({ modelScope: key.modelScope, modelIds: key.modelIds, modelId: entry.modelId })) {+ if (!aiKeyScope.scopeAllows({ modelScope: key.modelScope, modelIds: key.modelIds, modelId: entry.modelId })) {throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: `Key "${key.displayName}" does not allow model ${entry.modelId}` } })}}@@ -320,10 +339,6 @@ async function withNameConflictAsValidation<T>(operation: () => Promise<T>): Pro}}-function scopeAllows({ modelScope, modelIds, modelId }: { modelScope: AiProviderModelScope, modelIds: string[], modelId: string }): boolean {- return modelScope !== 'selected' || modelIds.includes(modelId)-}-function keyInUseError({ tierNames }: { tierNames: string[] }): ActivepiecesError {return new ActivepiecesError({code: ErrorCode.VALIDATION,diff --git a/packages/server/api/src/app/ee/agent/agent-conversation-service.ts b/packages/server/api/src/app/ee/agent/agent-conversation-service.tsindex 69d976b77fd..5b04b672b9a 100644--- a/packages/server/api/src/app/ee/agent/agent-conversation-service.ts+++ b/packages/server/api/src/app/ee/agent/agent-conversation-service.ts@@ -3,7 +3,6 @@ import { Agent, AgentConversation, AgentConversationStatus, AgentHistoryMessage,import { ModelMessage } from 'ai'import { FastifyBaseLogger } from 'fastify'import { EntityManager, In } from 'typeorm'-import { platformModelTierService } from '../../ai/platform-model-tier-service'import { transaction } from '../../core/db/transaction'import { databaseConnection } from '../../database/database-connection'import { FlowRunEntity } from '../../flows/flow-run/flow-run-entity'@@ -15,6 +14,7 @@ import { agentApprovalGate } from './agent-approval-gate'import { AgentConversationEntity } from './agent-conversation-entity'import { AgentEntity } from './agent-entity'import { agentHelpers, EVAL_CONVERSATION_ID_PREFIX, isEvalConversationId } from './agent-helpers'+import { agentModelTier } from './agent-model-tier'import { agentService } from './agent-service'import { agentHistory } from './history/agent-history'@@ -56,15 +56,18 @@ async function projectStillHoldingAgent({ agentId, authorisedProjectId, entityMareturn locked.projectId}-async function modelChoiceFrom({ platformId, modelName, modelTierId }: { platformId: string, modelName: string | null | undefined, modelTierId: string | null | undefined }): Promise<ModelChoice> {+async function modelChoiceFrom({ platformId, projectId, modelName, modelTierId, log }: { platformId: string, projectId: string | null, modelName: string | null | undefined, modelTierId: string | null | undefined, log: FastifyBaseLogger }): Promise<ModelChoice> {if (!isNil(modelTierId)) {- await platformModelTierService.getForRun({ platformId, id: modelTierId })+ await agentModelTier(log).assertUsable({ platformId, tierId: modelTierId, scope: isNil(projectId) ? { type: 'platform' } : { type: 'project', projectId } })return { modelTierId, modelName: null }}if (!isNil(modelName)) {return { modelName, modelTierId: null }}- return modelTierId === null ? { modelTierId: null } : {}+ return {+ ...(modelName === null ? { modelName: null } : {}),+ ...(modelTierId === null ? { modelTierId: null } : {}),+ }}export const agentConversationService = (log: FastifyBaseLogger) => ({@@ -73,10 +76,10 @@ export const agentConversationService = (log: FastifyBaseLogger) => ({? null: await agentService(log).getOneOrThrowByPlatform({ id: request.agentId, platformId, userId })const builder = request.builder === true- const modelChoice = await modelChoiceFrom({ platformId, modelName: request.modelName, modelTierId: request.modelTierId })const builderProjectId = builder? await resolveBuilderProject({ agent, requestedProjectId: request.projectId, platformId, userId, log }): null+ const modelChoice = await modelChoiceFrom({ platformId, projectId: agent?.projectId ?? builderProjectId, modelName: request.modelName, modelTierId: request.modelTierId, log })const conversation = await transaction(async (entityManager) => entityManager.getRepository(AgentConversationEntity).save({id: id ?? apId(),platformId,@@ -220,7 +223,7 @@ export const agentConversationService = (log: FastifyBaseLogger) => ({const conversation = await this.getConversationOrThrow({ id, platformId, userId })const updates = {...spreadIfDefined('title', request.title),- ...await modelChoiceFrom({ platformId, modelName: request.modelName, modelTierId: request.modelTierId }),+ ...await modelChoiceFrom({ platformId, projectId: conversation.projectId ?? null, modelName: request.modelName, modelTierId: request.modelTierId, log }),}if (Object.keys(updates).length > 0) {diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.tsindex 7a5540c5a58..496455c20d5 100644--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts@@ -132,7 +132,7 @@ async function resolveChatProvider({ platformId, scope, log }: { platformId: strasync function assertRunProviderConfigured({ platformId, provider, providerConfigId, modelTierId, scope, log }: { platformId: string, provider?: AIProviderName | null, providerConfigId?: string | null, modelTierId?: string | null, scope: ProviderScope, log: FastifyBaseLogger }): Promise<void> {if (!isNil(modelTierId)) {- await aiModelCandidates(log).firstCandidate({ platformId, tierId: modelTierId })+ await aiModelCandidates(log).firstCandidate({ platformId, tierId: modelTierId, scope })return}if (isNil(provider)) {@@ -183,16 +183,14 @@ async function resolveModelId({ platformId, providerConfig, selectedModel, surfareturn picked.id}-async function resolveImageModelId({ platformId, providerConfig, scope, grantedByTier = false, log }: { platformId: string, providerConfig: GetProviderConfigResponse, scope: ProviderScope, grantedByTier?: boolean, log: FastifyBaseLogger }): Promise<string | undefined> {+async function resolveImageModelId({ platformId, providerConfig, scope, log }: { platformId: string, providerConfig: GetProviderConfigResponse, scope: ProviderScope, log: FastifyBaseLogger }): Promise<string | undefined> {const { provider, configId } = providerConfigconst preferred = AI_PROVIDER_CAPABILITIES[provider].defaultImageModelif (isNil(preferred) || provider === AIProviderName.ACTIVEPIECES) {return preferred}const listed = await Promise.race([- tryCatch(() => grantedByTier- ? aiProviderService(log).listModelsForConfig({ platformId, configId })- : aiProviderService(log).listModels({ platformId, provider, scope, configId })),+ tryCatch(() => aiProviderService(log).listModels({ platformId, provider, scope, configId })),delay(IMAGE_MODEL_LOOKUP_TIMEOUT_MS, null),])if (isNil(listed) || isNil(listed.data)) {diff --git a/packages/server/api/src/app/ee/agent/agent-model-tier.ts b/packages/server/api/src/app/ee/agent/agent-model-tier.tsindex 4a0f905ce35..2febcac7e9c 100644--- a/packages/server/api/src/app/ee/agent/agent-model-tier.ts+++ b/packages/server/api/src/app/ee/agent/agent-model-tier.ts@@ -1,19 +1,26 @@-import { isNil, PlatformId, tryCatch } from '@activepieces/core-utils'-import { ModelTierSurface } from '@activepieces/server-utils'+import { ActivepiecesError, AIProviderName, ErrorCode, isNil, PlatformId, tryCatch } from '@activepieces/core-utils'+import { modelCatalog, ModelTierSurface } from '@activepieces/server-utils'import { AgentConversation, AgentModelCandidate } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'+import { aiKeyScope, ProviderScope } from '../../ai/ai-key-scope'import { aiModelCandidates, FirstCandidate, TierConfigCandidate, TierConfigs } from '../../ai/ai-model-candidates'+import { AIProviderSchema } from '../../ai/ai-provider-entity'+import { platformModelTierService, TierForRun } from '../../ai/platform-model-tier-service'import { agentModelResolution } from './agent-model-resolution'export const agentModelTier = (log: FastifyBaseLogger) => ({- async resolveRun({ platformId, tierId, surface }: { platformId: PlatformId, tierId: string, surface: ModelTierSurface }): Promise<AgentTierRun> {+ async resolveRun({ platformId, tierId, surface, scope }: { platformId: PlatformId, tierId: string, surface: ModelTierSurface, scope: ProviderScope }): Promise<AgentTierRun> {+ const tiers = await toolCallingTiers({ platformId, tierId, scope })const [run, fast] = await Promise.all([- aiModelCandidates(log).resolveConfigs({ platformId, tierId }),- aiModelCandidates(log).resolveFastConfig({ platformId }),+ aiModelCandidates(log).configsFor({ platformId, forRun: tiers.run }),+ isNil(tiers.fast) ? null : aiModelCandidates(log).configsFor({ platformId, forRun: tiers.fast }),])+ if (run.candidates.length === 0) {+ throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: `No model in tier "${run.tier.name}" can run` } })+ }const consoleBudget = agentModelResolution.resolveTier({ tierId: null, surface }).thinkingBudgetconst candidates = withBudget({ configs: run, consoleBudget })- const fastIsThisTier = isNil(fast) || fast.tier.id === run.tier.id+ const fastIsThisTier = isNil(fast) || fast.candidates.length === 0 || fast.tier.id === run.tier.idreturn {tier: { id: run.tier.id, name: run.tier.name },candidates,@@ -21,6 +28,28 @@ export const agentModelTier = (log: FastifyBaseLogger) => ({}},+ async assertUsable({ platformId, tierId, scope }: { platformId: PlatformId, tierId: string, scope: ProviderScope }): Promise<void> {+ await toolCallingTiers({ platformId, tierId, scope })+ },++ async assertProjectSwitchKeepsTier({ platformId, tierId, fromProjectId, toProjectId }: { platformId: PlatformId, tierId: string, fromProjectId: string | null, toProjectId: string | null }): Promise<void> {+ if (fromProjectId === toProjectId) {+ return+ }+ const { run, fast } = await toolCallingTiers({ platformId, tierId, scope: isNil(fromProjectId) ? { type: 'platform' } : { type: 'project', projectId: fromProjectId } })+ const keys = [...run.entries, ...(fast?.entries ?? [])].map((entry) => entry.key)+ const servesTarget = (key: AIProviderSchema): boolean => isNil(toProjectId)+ ? key.projectScope === 'all'+ : aiKeyScope.rowAllowsScope({ row: key, scope: { type: 'project', projectId: toProjectId } })+ if (keys.every(servesTarget)) {+ return+ }+ throw new ActivepiecesError({+ code: ErrorCode.AUTHORIZATION,+ params: { message: 'a model this run can use is not available to the project it tried to switch to' },+ })+ },+async mainModelOf({ conversation }: { conversation: Pick<AgentConversation, 'platformId' | 'modelTierId'> }): Promise<FirstCandidate | null> {const tierId = conversation.modelTierIdif (isNil(tierId)) {@@ -49,6 +78,36 @@ export const agentModelTier = (log: FastifyBaseLogger) => ({},})+async function toolCallingTiers({ platformId, tierId, scope }: { platformId: PlatformId, tierId: string, scope: ProviderScope }): Promise<{ run: TierForRun, fast: TierForRun | null }> {+ const [run, fast, callsTools] = await Promise.all([+ platformModelTierService.getForRun({ platformId, id: tierId, scope }),+ platformModelTierService.getFastForRun({ platformId, scope }),+ toolCallingCheck(),+ ])+ const toolRun = withToolCallingEntries({ forRun: run, callsTools })+ if (isNil(toolRun)) {+ throw noToolCallingError({ tierName: run.tier.name })+ }+ return { run: toolRun, fast: isNil(fast) ? null : withToolCallingEntries({ forRun: fast, callsTools }) }+}++async function toolCallingCheck(): Promise<CallsTools> {+ const catalog = await modelCatalog.load()+ return ({ provider, modelId }) => catalog.lookup({ provider, modelId })?.supportsToolCalling !== false+}++function withToolCallingEntries({ forRun, callsTools }: { forRun: TierForRun, callsTools: CallsTools }): TierForRun | null {+ const main = forRun.tier.entries[0]+ const mainEntry = forRun.entries.find((entry) => entry.key.id === main.configId && entry.modelId === main.modelId)+ const entries = forRun.entries.filter((entry) => callsTools({ provider: entry.key.provider, modelId: entry.modelId }))+ const mainRefused = !isNil(mainEntry) && !entries.includes(mainEntry)+ return mainRefused || entries.length === 0 ? null : { tier: forRun.tier, entries }+}++function noToolCallingError({ tierName }: { tierName: string }): ActivepiecesError {+ return new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: `The main model of tier "${tierName}" can't call tools, so agents and chat can't use it` } })+}+function tierCacheKey({ platformId, tierId }: { platformId: string, tierId: string }): string {return `${platformId}:${tierId}`}@@ -68,6 +127,8 @@ export type AgentTierCandidate = TierConfigCandidate & {thinkingBudget: number}+type CallsTools = (model: { provider: AIProviderName, modelId: string }) => boolean+export type AgentTierRun = {tier: { id: string, name: string }candidates: AgentTierCandidate[]diff --git a/packages/server/api/src/app/ee/agent/agent-service.ts b/packages/server/api/src/app/ee/agent/agent-service.tsindex 8a3170ea5ea..8ead302af57 100644--- a/packages/server/api/src/app/ee/agent/agent-service.ts+++ b/packages/server/api/src/app/ee/agent/agent-service.ts@@ -4,7 +4,6 @@ import { ActivepiecesError, AIProviderName, apId, ApId, connectionTemplate, Cursimport { Agent, AgentConfig, AgentFlowTool, AgentKnowledgeBaseTool, AgentListSort, AgentMoveLoss, AgentMoveLossKind, AgentMovePreview, AgentRunSource, AgentSummary, agentUtils, AgentVisibility, CreateAgentRequest, DefaultProjectRole, Project, ProjectType, UpdateAgentRequest } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { Brackets, EntityManager, In, SelectQueryBuilder } from 'typeorm'-import { platformModelTierService } from '../../ai/platform-model-tier-service'import { appConnectionService } from '../../app-connection/app-connection-service/app-connection-service'import { repoFactory } from '../../core/db/repo-factory'import { transaction } from '../../core/db/transaction'@@ -22,6 +21,7 @@ import { projectMemberService } from '../projects/project-members/project-memberimport { AgentConversationEntity } from './agent-conversation-entity'import { AgentEntity, AgentWithRelations } from './agent-entity'import { agentHelpers } from './agent-helpers'+import { agentModelTier } from './agent-model-tier'const DEFAULT_PAGE_SIZE = 20const MAX_NAMED_FLOWS_IN_USE = 3@@ -36,7 +36,7 @@ const AGENT_MOVED_AWAY = 'That agent has just been moved somewhere else. Reloadexport const agentService = (log: FastifyBaseLogger) => ({async create({ platformId, projectId, ownerId, request }: CreateParams): Promise<Agent> {const visibility = request.visibility ?? AgentVisibility.PROJECT- const draft = await withDefaultModel({ draft: await withTierOrModel({ draft: request.draft, platformId }), platformId, projectId, log })+ const draft = await withDefaultModel({ draft: await withTierOrModel({ draft: request.draft, platformId, projectId, log }), platformId, projectId, log })return agentRepo().save({id: apId(),projectId,@@ -120,7 +120,7 @@ export const agentService = (log: FastifyBaseLogger) => ({projectId,log,})- const draft = isNil(request.draft) ? agent.draft : sanitizeObjectForPostgresql(await withTierOrModel({ draft: request.draft, platformId }))+ const draft = isNil(request.draft) ? agent.draft : sanitizeObjectForPostgresql(await withTierOrModel({ draft: request.draft, platformId, projectId, log }))await getFolderIdFromRequest({ projectId, folderId: request.folderId ?? undefined, folderName: undefined, log })const goingLive = goLive && agentUtils.isPublishable(draft)const published = goingLive ? draft : agent.published@@ -423,11 +423,11 @@ async function isProjectAdministrator({ projectId, userId, log }: { projectId: Preturn role?.name === DefaultProjectRole.ADMIN}-async function withTierOrModel({ draft, platformId }: { draft: AgentConfig, platformId: PlatformId }): Promise<AgentConfig> {+async function withTierOrModel({ draft, platformId, projectId, log }: { draft: AgentConfig, platformId: PlatformId, projectId: ProjectId, log: FastifyBaseLogger }): Promise<AgentConfig> {if (isNil(draft.modelTierId)) {return draft}- await platformModelTierService.getForRun({ platformId, id: draft.modelTierId })+ await agentModelTier(log).assertUsable({ platformId, tierId: draft.modelTierId, scope: { type: 'project', projectId } })return { ...draft, provider: null, providerConfigId: null, modelName: null }}diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.tsindex 3322a20bdb2..905771abb88 100644--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts@@ -89,7 +89,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({const runScope = agentHelpers.runScopeOrThrow({ projectId: runProjectId })const surface = agentHelpers.surfaceOf({ source: requestedSource })const modelTierId = tierIdForRun({ conversation, requestedTierId: input.modelTierId ?? null })- const tierRun = isNil(modelTierId) ? null : await agentModelTier(log).resolveRun({ platformId, tierId: modelTierId, surface })+ const tierRun = isNil(modelTierId) ? null : await agentModelTier(log).resolveRun({ platformId, tierId: modelTierId, surface, scope: runScope })const providerConfig = isNil(tierRun)? await agentHelpers.resolveRunProvider({ platformId, log, scope: runScope, ...spreadIfDefined('provider', input.provider), ...spreadIfDefined('providerConfigId', input.providerConfigId) }): tierRun.candidates[0].config@@ -113,7 +113,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({const generatesImagesOnProvider = actingRun && isNil(aiTools.imageGeneration)const imageModelId = !generatesImagesOnProvider? undefined- : chosen.image?.modelId ?? await agentHelpers.resolveImageModelId({ platformId, providerConfig, scope: runScope, grantedByTier: !isNil(tierRun), log })+ : chosen.image?.modelId ?? await agentHelpers.resolveImageModelId({ platformId, providerConfig, scope: runScope, log })const lock = await agentHelpers.acquireStreamingLock({ conversationId, ...spreadIfDefined('runId', input.runId) })if (lock === 'superseded') {diff --git a/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.tsindex 4f6f162455a..ca77b157c73 100644--- a/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts+++ b/packages/server/api/src/app/ee/agent/rpc/conversation-rpc.ts@@ -3,6 +3,7 @@ import { AgentConversationStatus, AgentCreditsLeftRequest, AgentRunSource, FileCimport { FastifyBaseLogger } from 'fastify'import { readConversationFile } from '.././agent-file-utils'import { agentHelpers } from '.././agent-helpers'+import { agentModelTier } from '.././agent-model-tier'import { chatAnalyticsTelemetry } from '.././chat-analytics-sync'import { chatToolBilling } from '.././chat-tool-billing'import { fileService } from '../../../file/file.service'@@ -162,6 +163,14 @@ export const conversationRpc = (log: FastifyBaseLogger) => ({params: { message: 'A flow-step agent run cannot move to another project' },})}+ if (!isNil(conversation) && !isNil(conversation.modelTierId)) {+ await agentModelTier(log).assertProjectSwitchKeepsTier({+ platformId: conversation.platformId,+ tierId: conversation.modelTierId,+ fromProjectId: conversation.projectId ?? null,+ toProjectId: input.projectId,+ })+ }if (!isNil(conversation) && isNil(conversation.modelTierId)) {await agentHelpers.assertProjectSwitchKeepsKey({platformId: conversation.platformId,diff --git a/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.tsindex d6f293ed72d..a10ac60540a 100644--- a/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts+++ b/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts@@ -326,7 +326,7 @@ async function pieceToolModel({ configuredRun, input, log }: { configuredRun: Coconst { projectId, platformId, modelTierId } = configuredRunconst granted = isNil(modelTierId) || isNil(input.providerConfigId)? null- : await aiModelCandidates(log).grantedEntryConfig({ platformId, tierId: modelTierId, configId: input.providerConfigId, modelId: input.modelId })+ : await aiModelCandidates(log).grantedEntryConfig({ platformId, tierId: modelTierId, configId: input.providerConfigId, modelId: input.modelId, scope: { type: 'project', projectId } })if (!isNil(granted)) {return aiUtils.createModel({ credentials: granted, modelId: input.modelId, platformId, providerConfigId: granted.configId })}diff --git a/packages/server/api/test/integration/ce/ai-provider/ai-model-candidates.test.ts b/packages/server/api/test/integration/ce/ai-provider/ai-model-candidates.test.tsindex 8b7cd1f57cc..423e192977f 100644--- a/packages/server/api/test/integration/ce/ai-provider/ai-model-candidates.test.ts+++ b/packages/server/api/test/integration/ce/ai-provider/ai-model-candidates.test.ts@@ -43,16 +43,33 @@ beforeEach(async () => {})describe('tier candidates', () => {- it('lets a tier run a key that is scoped away from the project, while a specific pick of it still fails', async () => {+ it('skips a fallback whose key is scoped away from the project', async () => {+ const open = await seedKey({ testCtx: ctx })const scopedAway = await seedKey({ testCtx: ctx, projectScope: 'selected', projectIds: [] })- const tier = await createTier({ testCtx: ctx, name: 'Expert', entries: [{ configId: scopedAway.id, modelId: 'gpt-4o' }] })+ const tier = await createTier({ testCtx: ctx, name: 'Expert', entries: [{ configId: open.id, modelId: 'main' }, { configId: scopedAway.id, modelId: 'fallback' }] })const { tierName, candidates } = await rpc().resolveAiModelCandidates({ projectId: ctx.project.id, platformId: ctx.platform.id, modelTierId: tier.id })- const specificPick = rpc().resolveAiProvider({ projectId: ctx.project.id, platformId: ctx.platform.id, provider: AIProviderName.CUSTOM, providerConfigId: scopedAway.id })expect(tierName).toBe('Expert')- expect(candidates).toEqual([expect.objectContaining({ providerConfigId: scopedAway.id, modelId: 'gpt-4o', provider: AIProviderName.CUSTOM, status: 'active' })])- await expect(specificPick).rejects.toThrow()+ expect(candidates).toEqual([expect.objectContaining({ providerConfigId: open.id, modelId: 'main', provider: AIProviderName.CUSTOM, status: 'active' })])+ })++ it('makes a tier unavailable in a project its main model\'s key does not serve', async () => {+ const scopedAway = await seedKey({ testCtx: ctx, projectScope: 'except', projectIds: [ctx.project.id] })+ const open = await seedKey({ testCtx: ctx })+ const tier = await createTier({ testCtx: ctx, entries: [{ configId: scopedAway.id, modelId: 'main' }, { configId: open.id, modelId: 'fallback' }] })++ await expect(rpc().resolveAiModelCandidates({ projectId: ctx.project.id, platformId: ctx.platform.id, modelTierId: tier.id }))+ .rejects.toMatchObject({ error: { code: 'VALIDATION', params: { message: TIER_NOT_AVAILABLE } } })+ })++ it('runs a tier whose keys are limited to this project', async () => {+ const selected = await seedKey({ testCtx: ctx, projectScope: 'selected', projectIds: [ctx.project.id] })+ const tier = await createTier({ testCtx: ctx, entries: [{ configId: selected.id, modelId: 'main' }] })++ const { candidates } = await rpc().resolveAiModelCandidates({ projectId: ctx.project.id, platformId: ctx.platform.id, modelTierId: tier.id })++ expect(candidates.map((candidate) => candidate.providerConfigId)).toEqual([selected.id])})it('tries healthy keys first and keeps the tier order inside each group', async () => {@@ -133,6 +150,20 @@ describe('tier candidates', () => {})})+describe('GET /v1/platform-model-tiers', () => {+ it('lists only the tiers whose main model can run in the project', async () => {+ const open = await seedKey({ testCtx: ctx })+ const scopedAway = await seedKey({ testCtx: ctx, projectScope: 'selected', projectIds: [] })+ await createTier({ testCtx: ctx, name: 'Everywhere', entries: [{ configId: open.id, modelId: 'main' }, { configId: scopedAway.id, modelId: 'fallback' }] })+ await createTier({ testCtx: ctx, name: 'Elsewhere', entries: [{ configId: scopedAway.id, modelId: 'main' }] })++ const response = await ctx.get(TIERS, { projectId: ctx.project.id })++ expect(response.statusCode).toBe(StatusCodes.OK)+ expect(response.json().map((tier: { name: string }) => tier.name)).toEqual(['Everywhere'])+ })+})+describe('reported key outcomes', () => {it('re-checks the key instead of trusting a reported success, so a forged report cannot mark a key healthy', async () => {const key = await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.AZURE, config: { resourceName: `forged-${apId()}` } })@@ -176,7 +207,7 @@ describe('reported key outcomes', () => {describe('POST /v1/ai/execute with a tier', () => {it('enqueues the tier and the first model the worker will try', async () => {- const key = await seedKey({ testCtx: ctx, projectScope: 'selected', projectIds: [] })+ const key = await seedKey({ testCtx: ctx, projectScope: 'selected', projectIds: [ctx.project.id] })const tier = await createTier({ testCtx: ctx, entries: [{ configId: key.id, modelId: 'gpt-4o' }] })const response = await executeAiStep({ body: { action: AiStepAction.ASK_AI, modelTierId: tier.id } })@@ -185,6 +216,16 @@ describe('POST /v1/ai/execute with a tier', () => {expect(enqueue.mock.calls[0][0]).toMatchObject({ modelTierId: tier.id, provider: AIProviderName.CUSTOM, providerConfigId: key.id, modelId: 'gpt-4o' })})+ it('refuses a tier whose main model\'s key does not serve the project', async () => {+ const key = await seedKey({ testCtx: ctx, projectScope: 'selected', projectIds: [] })+ const tier = await createTier({ testCtx: ctx, entries: [{ configId: key.id, modelId: 'gpt-4o' }] })++ const response = await executeAiStep({ body: { action: AiStepAction.ASK_AI, modelTierId: tier.id } })++ expect(response.statusCode).not.toBe(StatusCodes.OK)+ expect(enqueue).not.toHaveBeenCalled()+ })+it('refuses a tier on an image step', async () => {const key = await seedKey({ testCtx: ctx })const tier = await createTier({ testCtx: ctx, entries: [{ configId: key.id, modelId: 'gpt-4o' }] })@@ -223,7 +264,7 @@ function rpc(): ReturnType<typeof aiRpcHandlers> {return aiRpcHandlers(app!.log)}-async function seedKey({ testCtx, projectScope, projectIds }: { testCtx: TestContext, projectScope?: 'all' | 'selected', projectIds?: string[] }): Promise<{ id: string }> {+async function seedKey({ testCtx, projectScope, projectIds }: { testCtx: TestContext, projectScope?: 'all' | 'selected' | 'except', projectIds?: string[] }): Promise<{ id: string }> {return mockAndSaveAIProvider({platformId: testCtx.platform.id,provider: AIProviderName.CUSTOM,@@ -255,3 +296,4 @@ async function executeAiStep({ body }: { body: Record<string, unknown> }) {}const TIERS = '/v1/platform-model-tiers'+const TIER_NOT_AVAILABLE = 'This tier isn\'t available in this project. Pick another model.'diff --git a/packages/server/api/test/integration/ee/agent/agent-tier-runtime.test.ts b/packages/server/api/test/integration/ee/agent/agent-tier-runtime.test.tsindex da6b824c8d6..105e0364433 100644--- a/packages/server/api/test/integration/ee/agent/agent-tier-runtime.test.ts+++ b/packages/server/api/test/integration/ee/agent/agent-tier-runtime.test.ts@@ -1,4 +1,5 @@import { AIProviderName, apId, ErrorCode } from '@activepieces/core-utils'+import { modelCatalog } from '@activepieces/server-utils'import { AgentConversation, AgentIcon, AgentRunSource, ColorName, WorkerJobType } from '@activepieces/shared'import { FastifyInstance } from 'fastify'import { StatusCodes } from 'http-status-codes'@@ -28,8 +29,8 @@ afterEach(() => {})describe('a chat on a platform tier', () => {- it('runs on a tier key that is scoped away from the project, because the tier grants it', async () => {- const { ctx, tier, keyId } = await tierOnScopedAwayKey()+ it('runs on a tier key limited to the chat\'s project', async () => {+ const { ctx, tier, keyId } = await tierOnKey({ servesOwnProject: true })const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })const config = await getConfig({ ctx, conversationId: conversation.id })@@ -39,15 +40,25 @@ describe('a chat on a platform tier', () => {expect(config.platformTier).toEqual({ id: tier.id, name: 'Expert' })})- it('cannot reach the same key as a specific model, because the key does not serve the project', async () => {- const { ctx, keyId } = await tierOnScopedAwayKey()- const conversation = await createConversation({ ctx, body: {} })+ it('refuses to run a tier whose main model\'s key does not serve the project', async () => {+ const { ctx, tier } = await tierOnKey({ servesOwnProject: false })+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })- await expect(getConfig({ ctx, conversationId: conversation.id, provider: AIProviderName.OPENAI, providerConfigId: keyId })).rejects.toMatchObject({ error: { code: ErrorCode.ENTITY_NOT_FOUND } })+ await expect(getConfig({ ctx, conversationId: conversation.id })).rejects.toMatchObject({ error: { code: ErrorCode.VALIDATION, params: { message: TIER_NOT_AVAILABLE } } })+ })++ it('skips a fallback that cannot call tools', async () => {+ mockCatalogWithoutToolsFor({ modelId: NO_TOOLS_MODEL })+ const { ctx, tier, keyId } = await tierOnKey({ servesOwnProject: true, fallbackModelId: NO_TOOLS_MODEL })+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })++ const config = await getConfig({ ctx, conversationId: conversation.id })++ expect(config.candidates?.map((candidate) => [candidate.providerConfigId, candidate.modelId])).toEqual([[keyId, TIER_MODEL]])})it('refuses a tier from another platform', async () => {- const { tier } = await tierOnScopedAwayKey()+ const { tier } = await tierOnKey({ servesOwnProject: true })const otherCtx = await context()const response = await otherCtx.post(CONVERSATIONS_URL, { modelTierId: tier.id })@@ -56,7 +67,7 @@ describe('a chat on a platform tier', () => {})it('keeps one model choice: a tier clears the model name and a model name clears the tier', async () => {- const { ctx, tier } = await tierOnScopedAwayKey()+ const { ctx, tier } = await tierOnKey({ servesOwnProject: true })const conversation = await createConversation({ ctx, body: { modelName: 'expert', modelTierId: tier.id } })const pickedModel = await ctx.post(`${CONVERSATIONS_URL}/${conversation.id}`, { modelName: 'fast' })@@ -67,8 +78,18 @@ describe('a chat on a platform tier', () => {expect(pickedTier.json()).toMatchObject({ modelTierId: tier.id, modelName: null })})+ it('clears the model when an update sends a null model name', async () => {+ const { ctx } = await tierOnKey({ servesOwnProject: true })+ const conversation = await createConversation({ ctx, body: { modelName: 'fast' } })++ const cleared = await ctx.post(`${CONVERSATIONS_URL}/${conversation.id}`, { modelName: null })++ expect(cleared.statusCode).toBe(StatusCodes.OK)+ expect(cleared.json()).toMatchObject({ modelName: null, modelTierId: null })+ })+it('keeps the tier when the worker saves the turn with a model name', async () => {- const { ctx, tier } = await tierOnScopedAwayKey()+ const { ctx, tier } = await tierOnKey({ servesOwnProject: true })const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })await agentRpcHandlers(app.log).saveAgentMessages({ conversationId: conversation.id, messages: [{ role: 'user', content: 'hi' }], uiMessages: [], modelName: 'smart' })@@ -77,12 +98,25 @@ describe('a chat on a platform tier', () => {expect(stored).toMatchObject({ modelTierId: tier.id, modelName: null })})- it('moves to another project on a tier key that serves neither project', async () => {- const { ctx, tier, keyId } = await tierOnScopedAwayKey()+ it('refuses to move to a project the tier\'s key does not serve', async () => {+ const { ctx, tier, keyId } = await tierOnKey({ servesOwnProject: true })+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })+ await db.update('agent_conversation', conversation.id, { projectId: ctx.project.id })+ const target = await otherProject({ ctx })++ await expect(agentRpcHandlers(app.log).updateProjectContext({ conversationId: conversation.id, projectId: target.id, provider: AIProviderName.OPENAI, providerConfigId: keyId }))+ .rejects.toMatchObject({ error: { code: ErrorCode.AUTHORIZATION } })++ const stored = await db.findOneByOrFail<AgentConversation>('agent_conversation', { id: conversation.id })+ expect(stored.projectId).toBe(ctx.project.id)+ })++ it('moves to another project when every key of the tier serves it', async () => {+ const { ctx, tier, keyId } = await tierOnKey({ servesOwnProject: true })+ await db.update('ai_provider', keyId, { projectScope: 'all', projectIds: [] })const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })await db.update('agent_conversation', conversation.id, { projectId: ctx.project.id })- const target = createMockProject({ ownerId: ctx.user.id, platformId: ctx.platform.id })- await db.save('project', target)+ const target = await otherProject({ ctx })await agentRpcHandlers(app.log).updateProjectContext({ conversationId: conversation.id, projectId: target.id, provider: AIProviderName.OPENAI, providerConfigId: keyId })@@ -91,29 +125,100 @@ describe('a chat on a platform tier', () => {})})+describe('a tier chat that clears or switches its project', () => {+ it('refuses to clear the project while a kept key is limited to some projects', async () => {+ const { ctx, tier, keyId } = await tierOnKey({ servesOwnProject: true })+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })+ await db.update('agent_conversation', conversation.id, { projectId: ctx.project.id })++ await expect(agentRpcHandlers(app.log).updateProjectContext({ conversationId: conversation.id, projectId: null, provider: AIProviderName.OPENAI, providerConfigId: keyId }))+ .rejects.toMatchObject({ error: { code: ErrorCode.AUTHORIZATION } })+ })++ it('clears and re-selects a project when every key of the tier is open to all projects', async () => {+ const { ctx, tier, keyId } = await tierOnKey({ servesOwnProject: true })+ await db.update('ai_provider', keyId, { projectScope: 'all', projectIds: [] })+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })+ await db.update('agent_conversation', conversation.id, { projectId: ctx.project.id })+ const target = await otherProject({ ctx })++ await agentRpcHandlers(app.log).updateProjectContext({ conversationId: conversation.id, projectId: null, provider: AIProviderName.OPENAI, providerConfigId: keyId })+ await agentRpcHandlers(app.log).updateProjectContext({ conversationId: conversation.id, projectId: target.id, provider: AIProviderName.OPENAI, providerConfigId: keyId })++ const stored = await db.findOneByOrFail<AgentConversation>('agent_conversation', { id: conversation.id })+ expect(stored.projectId).toBe(target.id)+ })++ it('ignores a fallback the chat cannot use when checking a switch', async () => {+ mockCatalogWithoutToolsFor({ modelId: NO_TOOLS_MODEL })+ const ctx = await context()+ const openKey = await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })+ const limitedKey = await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })+ await db.update('ai_provider', limitedKey.id, { projectScope: 'selected', projectIds: [ctx.project.id] })+ const tier = await createTier({ ctx, entries: [{ configId: openKey.id, modelId: TIER_MODEL }, { configId: limitedKey.id, modelId: NO_TOOLS_MODEL }] })+ const conversation = await createConversation({ ctx, body: { modelTierId: tier.id } })+ await db.update('agent_conversation', conversation.id, { projectId: ctx.project.id })+ const target = await otherProject({ ctx })++ await agentRpcHandlers(app.log).updateProjectContext({ conversationId: conversation.id, projectId: target.id, provider: AIProviderName.OPENAI, providerConfigId: openKey.id })++ const stored = await db.findOneByOrFail<AgentConversation>('agent_conversation', { id: conversation.id })+ expect(stored.projectId).toBe(target.id)+ })+})++describe('a new-agent builder chat on a tier', () => {+ it('refuses a tier its builder project cannot use', async () => {+ const { ctx, tier } = await tierOnKey({ servesOwnProject: false })++ const response = await ctx.post(CONVERSATIONS_URL, { builder: true, projectId: ctx.project.id, modelTierId: tier.id })++ expect(response.statusCode).toBe(StatusCodes.CONFLICT)+ })+})+describe('the tier grant for tools', () => {- it('grants a model of the tier on its own key, and nothing else', async () => {- const { ctx, tier, keyId } = await tierOnScopedAwayKey()+ it('grants a model of the tier on its own key inside the project, and nothing else', async () => {+ const { ctx, tier, keyId } = await tierOnKey({ servesOwnProject: true })const otherKey = await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })- const grant = (configId: string, modelId: string) => aiModelCandidates(app.log).grantedEntryConfig({ platformId: ctx.platform.id, tierId: tier.id, configId, modelId })+ const target = await otherProject({ ctx })+ const grant = ({ configId, modelId, projectId }: { configId: string, modelId: string, projectId: string }) => aiModelCandidates(app.log).grantedEntryConfig({ platformId: ctx.platform.id, tierId: tier.id, configId, modelId, scope: { type: 'project', projectId } })- expect((await grant(keyId, TIER_MODEL))?.configId).toBe(keyId)- expect(await grant(keyId, 'another-model')).toBeNull()- expect(await grant(otherKey.id, TIER_MODEL)).toBeNull()+ expect((await grant({ configId: keyId, modelId: TIER_MODEL, projectId: ctx.project.id }))?.configId).toBe(keyId)+ expect(await grant({ configId: keyId, modelId: 'another-model', projectId: ctx.project.id })).toBeNull()+ expect(await grant({ configId: otherKey.id, modelId: TIER_MODEL, projectId: ctx.project.id })).toBeNull()+ await expect(grant({ configId: keyId, modelId: TIER_MODEL, projectId: target.id })).rejects.toMatchObject({ error: { code: ErrorCode.VALIDATION } })})})describe('an agent on a platform tier', () => {it('keeps the tier on create instead of filling in a default model', async () => {- const { ctx, tier } = await tierOnScopedAwayKey()+ const { ctx, tier } = await tierOnKey({ servesOwnProject: true })const agent = await createAgent({ ctx, draft: { modelTierId: tier.id, provider: AIProviderName.OPENAI, modelName: 'gpt-4o' } })expect(agent.draft).toMatchObject({ modelTierId: tier.id, provider: null, providerConfigId: null, modelName: null })})+ it('refuses a tier whose main model cannot call tools', async () => {+ mockCatalogWithoutToolsFor({ modelId: TIER_MODEL })+ const { ctx, tier } = await tierOnKey({ servesOwnProject: true })++ const response = await ctx.post('/v1/agents', agentBody({ ctx, draft: { modelTierId: tier.id } }))++ expect(response.statusCode).toBe(StatusCodes.CONFLICT)+ })++ it('refuses a tier whose main model\'s key does not serve the agent\'s project', async () => {+ const { ctx, tier } = await tierOnKey({ servesOwnProject: false })++ const response = await ctx.post('/v1/agents', agentBody({ ctx, draft: { modelTierId: tier.id } }))++ expect(response.statusCode).toBe(StatusCodes.CONFLICT)+ })+it('can be talked to, and hands the tier to the worker', async () => {- const { ctx, tier } = await tierOnScopedAwayKey()+ const { ctx, tier } = await tierOnKey({ servesOwnProject: true })const add = vi.fn()vi.spyOn(jobQueueModule, 'jobQueue').mockImplementation((log) => ({ ...originalJobQueue(log), add }))const agent = await createAgent({ ctx, draft: { modelTierId: tier.id } })@@ -127,7 +232,7 @@ describe('an agent on a platform tier', () => {})it('records the tier a flow step ran on, so its tools are granted the same keys', async () => {- const { ctx, tier } = await tierOnScopedAwayKey()+ const { ctx, tier } = await tierOnKey({ servesOwnProject: true })const conversationId = apId()await agentRpcHandlers(app.log).getAgentConfig({ conversationId, platformId: ctx.platform.id, projectId: ctx.project.id, userId: ctx.user.id, userMessage: 'sweep the inbox', modelName: null, modelTierId: tier.id, source: AgentRunSource.FLOW_STEP })@@ -141,15 +246,34 @@ async function context(): Promise<TestContext> {return createTestContext(app, { plan: { agentsEnabled: true, chatEnabled: true, aiProvidersEnabled: true } })}-async function tierOnScopedAwayKey(): Promise<{ ctx: TestContext, tier: { id: string }, keyId: string }> {+async function tierOnKey({ servesOwnProject, fallbackModelId }: { servesOwnProject: boolean, fallbackModelId?: string }): Promise<{ ctx: TestContext, tier: { id: string }, keyId: string }> {const ctx = await context()const key = await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })- await db.update('ai_provider', key.id, { projectScope: 'selected', projectIds: [] })- const response = await ctx.post('/v1/platform-model-tiers', { name: 'Expert', emoji: '🧠', description: null, entries: [{ configId: key.id, modelId: TIER_MODEL }] })+ await db.update('ai_provider', key.id, { projectScope: 'selected', projectIds: servesOwnProject ? [ctx.project.id] : [] })+ const entries = [{ configId: key.id, modelId: TIER_MODEL }, ...(fallbackModelId ? [{ configId: key.id, modelId: fallbackModelId }] : [])]+ const response = await ctx.post('/v1/platform-model-tiers', { name: 'Expert', emoji: '🧠', description: null, entries })expect(response.statusCode).toBe(StatusCodes.OK)return { ctx, tier: response.json(), keyId: key.id }}+async function createTier({ ctx, entries }: { ctx: TestContext, entries: { configId: string, modelId: string }[] }): Promise<{ id: string }> {+ const response = await ctx.post('/v1/platform-model-tiers', { name: 'Expert', emoji: '🧠', description: null, entries })+ expect(response.statusCode).toBe(StatusCodes.OK)+ return response.json()+}++async function otherProject({ ctx }: { ctx: TestContext }): Promise<{ id: string }> {+ const project = createMockProject({ ownerId: ctx.user.id, platformId: ctx.platform.id })+ await db.save('project', project)+ return project+}++function mockCatalogWithoutToolsFor({ modelId }: { modelId: string }): void {+ vi.spyOn(modelCatalog, 'load').mockResolvedValue({+ lookup: (model) => model.modelId === modelId ? { supportsToolCalling: false } : undefined,+ })+}+async function createConversation({ ctx, body }: { ctx: TestContext, body: Record<string, unknown> }): Promise<AgentConversation> {const response = await ctx.post(CONVERSATIONS_URL, body)expect(response.statusCode).toBe(StatusCodes.CREATED)@@ -157,15 +281,19 @@ async function createConversation({ ctx, body }: { ctx: TestContext, body: Recor}async function createAgent({ ctx, draft }: { ctx: TestContext, draft: Record<string, unknown> }): Promise<{ id: string, draft: Record<string, unknown> }> {- const response = await ctx.post('/v1/agents', {+ const response = await ctx.post('/v1/agents', agentBody({ ctx, draft }))+ expect(response.statusCode).toBe(StatusCodes.CREATED)+ return response.json()+}++function agentBody({ ctx, draft }: { ctx: TestContext, draft: Record<string, unknown> }): Record<string, unknown> {+ return {projectId: ctx.project.id,displayName: 'Inbox sorter',icon: AgentIcon.MAIL,color: ColorName.BLUE,draft: { instructions: 'Sort unread mail.', provider: null, modelName: null, maxSteps: 5, tools: [], structuredOutput: [], ...draft },- })- expect(response.statusCode).toBe(StatusCodes.CREATED)- return response.json()+ }}async function getConfig({ ctx, conversationId, provider, providerConfigId }: { ctx: TestContext, conversationId: string, provider?: AIProviderName, providerConfigId?: string }): ReturnType<ReturnType<typeof agentRpcHandlers>['getAgentConfig']> {@@ -183,3 +311,5 @@ async function getConfig({ ctx, conversationId, provider, providerConfigId }: {const CONVERSATIONS_URL = '/v1/agents/conversations'const TIER_MODEL = 'gpt-4o'+const NO_TOOLS_MODEL = 'text-only-model'+const TIER_NOT_AVAILABLE = 'This tier isn\'t available in this project. Pick another model.'diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.tsindex a8185d04842..bdc22468cc5 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts@@ -689,7 +689,7 @@ function buildToolSet({ ctx, eventEmitter, log, phaseState, taintState, mcpToolSconst { error } = await tryCatch(() => ctx.apiClient.updateProjectContext({ conversationId, runId, projectId, provider: runKey().provider, providerConfigId: runKey().providerConfigId }))if (!isNil(error)) {log.warn({ error, conversation: { id: conversationId }, project: projectId ? { id: projectId } : undefined }, '[executeAgentRun] Refused a project switch')- return { success: false, error: 'Could not switch to that project on this chat — the AI provider key this chat runs on is not available there. Start a new chat in that project instead.' }+ return { success: false, error: 'Could not switch to that project on this chat — the model this chat runs on is not available there. Start a new chat in that project instead.' }}projectState.projectId = projectIdreturn { success: true }diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.tsindex 51520d5f65f..25accac78a3 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts@@ -352,7 +352,7 @@ export async function runAgentTurn({ models, fastModel, systemPrompt, messages,totalInputTokens,totalOutputTokens,toolCalls,- answeredBy: current(),+ answeredBy: lastStepUsedFast && !isNil(fast) ? fast : current(),allModelsFailed: !isNil(streamError) && runsATier && failedModelIds.size === unique(ranked.map((candidate) => candidate.modelId)).length,}}diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.tsindex 3a2f852f1de..f9a3b805d6a 100644--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/run-agent-turn-fallback.test.ts@@ -65,6 +65,19 @@ describe('a platform tier chat turn', () => {expect(main.doStreamCalls).toHaveLength(1)})+ it('names the fast model as the one that answered when the turn ends on the fast step', async () => {+ const main = answeringModel('from main')++ const turn = await runTurn({+ models: [tierModel({ modelId: 'main', model: main })],+ fastModel: tierModel({ modelId: 'fast', model: answeringModel('from fast') }),+ })++ expect(turn.answeredBy.modelId).toBe('fast')+ expect(main.doStreamCalls).toHaveLength(0)+ expect(textOf(turn)).toBe('from fast')+ })+it('drops a failing fast model and answers on the same main model', async () => {const turn = await runTurn({models: [tierModel({ modelId: 'main', model: answeringModel('from main') }), tierModel({ modelId: 'backup', model: answeringModel('from backup') })],