AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(ai): stop managed AI credit keys from leaking on OpenRouter (#15704)
Bartosz Majewski9ad57b36ac67
diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.mdindex 8def67e8fa7..318b4b2593e 100644--- a/brain/knowledge/ai-intelligence/ai-providers.md+++ b/brain/knowledge/ai-intelligence/ai-providers.md@@ -13,7 +13,7 @@ Lets platform admins configure one or more LLM backends for AI pieces in flows.### How it works- `GET /` list (auto-creates ACTIVEPIECES when `aiCreditsEnabled`); `GET /:provider/models` (cached); `POST /` create (validates creds first); `POST /:id` update; `DELETE /:id`.-- For ACTIVEPIECES, update is blocked by an early return (only enabling `enabledForChat` is allowed); deletion is allowed and self-healing — `listProviders` recreates the managed row.+- For ACTIVEPIECES, update is blocked by an early return (only enabling `enabledForChat` is allowed); deletion is allowed and self-healing — `listProviders` recreates the managed row. The delete revokes the row's OpenRouter key, and the platform hard delete does the same through `deleteManagedProvider`, because the `ON DELETE CASCADE` from `platform` would otherwise leave the key live. The recreated row mints a fresh $500 key, so a limit raised by `increaseAiCredits` does not survive a delete.- Engine integration: the AI piece no longer fetches credentials. It POSTs `/v1/ai/execute`, the server enqueues an `EXECUTE_AI` worker job, and the worker resolves the provider over the `resolveAiProvider` RPC. `GET /:provider/config` was removed in [#15494](https://github.com/activepieces/activepieces/pull/15494).- For the managed ACTIVEPIECES provider that config route is also the **credit gate**: `assertCreditsAndAppSumoNotExceeded` (`platform/billing-provider.ts`) throws `QUOTA_EXCEEDED` when the Autumn credit or AppSumo balance is blocked — fires per AI call, but usage is only metered post-run, so in-flight spend is invisible to it. See decision 000016 (`brain/decisions/000016-managed-ai-metering-moves-to-centralized-worker-execution.md`).- Activepieces provisioning: `getOrCreateActivePiecesProviderAuthConfig()` → `enrichWithKeysIfNeeded()` mints an OpenRouter key. No system job is scheduled — renewal/top-up is driven by Autumn (`autoTopUps` in `autumn-billing.ts`), not Stripe.@@ -189,8 +189,8 @@ after a file without chat keys it seeds the chat list from the live flow list.`STORE_VALUE_MAX_SIZE`. So the history cannot grow unbounded and cannot outgrow a request body — noseparate limit is needed wherever it is passed along. An older version of the action could write a`{ messages: [] }` wrapper instead of a bare array, so any reader has to accept both shapes.-- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `chatHelpers.resolveChatProvider` → `getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. There is also no distributed lock or cache, so concurrent turns for one platform each mint a live key and only the last is saved; the orphans keep spending provisioning quota. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getChatConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.-- **`openrouter-api.ts` uses raw `fetch`** — no timeout, no retry, no `tryCatch`, and it bypasses the repo's `safeHttp` rule for outbound HTTP in `packages/server/api`. A `getKey` 408 from OpenRouter escapes the admin `increaseAiCredits` path as an unhandled rejection.+- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `chatHelpers.resolveChatProvider` → `getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. The mint runs under a per-platform `distributedLock` and re-reads the row inside it, and the write lands only if `auth` still matches the ciphertext it read (a `jsonb` compare; the row is then read back, since `affected` is unreliable on PGlite). A key that loses that write, or whose write throws, is revoked with `openRouterApi.deleteKey`. Two leaks stay open until a reconcile sweep over `listKeys` exists: a crash between `createKey` and the write, and a `createKey` that times out after OpenRouter has made the key. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getChatConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.+- **`openrouter-api.ts` has no retry, and every failure is a plain `Error`.** Calls go through `safeHttp.axios` with a 15 s timeout, and `openRouterRequest` rethrows any failure as `[OpenRouter] <METHOD> <path> error: <status> <body>`, so a caller cannot tell a 429 from a 500 without parsing the message.- **Chat model tiers are Activepieces-only.** `ACTIVEPIECES_CHAT_TIERS` (`fast`/`smart`/`premium`, shown as Fast/Expert/Heavy) hold OpenRouter-shaped Anthropic ids, so they only mean anything for the ACTIVEPIECES and OPENROUTER chat providers. Any provider that declares `ALLOWED_CHAT_MODELS_BY_PROVIDER` (openai, anthropic, google) picks a real model id from that list instead. Naively stripping the tier's vendor prefix for every provider is what once sent `claude-haiku-4-5` to OpenAI and broke every message.- **A short model list in the key's picker is the vendor's catalog, not a truncation.** `listModels` returns whatever the provider's own `/models` endpoint gives and filters nothing except the key's `modelScope` allow-list. Anthropic ships roughly a dozen models, OpenAI ~80 (mostly embeddings/tts/whisper), while OpenRouter is an aggregator and returns 400+ from every vendor it proxies — so the counts differ by an order of magnitude by design. Anthropic pages at 20 by default, which is why its request pins `?limit=1000`. If the list shows exactly three Claude models, that is the *chat* dropdown reading the curated `ANTHROPIC_CHAT_MODELS`, a different surface from the admin picker.- **Read the chat model list through `aiProviderUtils.getCuratedChatModels({ provider })`.** It is the one accessor the server resolver (`agentHelpers.resolveModelIdForProvider`) and the chat dropdown share, so the two cannot drift; it returns `{ id, label }` or `undefined` — never an empty list, so callers may treat a result as non-empty. Labels come from the hardcoded `CHAT_MODEL_LABELS` (falling back to the id) rather than `AIProviderModel.name`, because the live `listModels` response cannot supply one for every provider: anthropic returns `display_name` and google `displayName`, but OpenAI's `/v1/models` returns ids only.@@ -218,7 +218,7 @@ after a file without chat keys it seeds the chat list from the live flow list.- `packages/server/api/src/app/ai/` — backend module: provider controller, service, entity, module, plus the sibling ai-tool-config files- `packages/server/api/src/app/ai/providers/` — per-vendor strategies keyed by `AIProviderName`- `packages/server/api/src/app/platform/billing-provider.ts` — `assertCreditsAndAppSumoNotExceeded` credit gate-- `packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts` — the OpenRouter provisioning client (`createKey`/`updateKey`/`getKey`/`listKeys`)+- `packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts` — the OpenRouter provisioning client (`createKey`/`updateKey`/`getKey`/`listKeys`/`deleteKey`)- `packages/core/ai-providers/src/lib/create-language-model.ts` — the shared per-provider model factory (`createLanguageModel`, `buildOpenAICompatibleHeaders`) used by both the AI piece and the agent path- `packages/core/shared/src/lib/management/ai-providers/index.ts` — shared zod schemas, enums, request/response types- `packages/core/shared/src/lib/management/ai-tools/index.ts` — shared schemas for the AI Tool Configs siblingdiff --git a/packages/server/api/src/app/ai/ai-provider-service.ts b/packages/server/api/src/app/ai/ai-provider-service.tsindex e196cad5f8e..479aba38e45 100644--- a/packages/server/api/src/app/ai/ai-provider-service.ts+++ b/packages/server/api/src/app/ai/ai-provider-service.ts@@ -3,13 +3,14 @@ import { modelCatalog, modelTierCatalog } from '@activepieces/server-utils'import { ActivePiecesProviderAuthConfig, AI_PROVIDER_ENTITY_TYPES, AIProviderAuthConfig, AIProviderConfig, aiProviderCredentials, AIProviderModel, AIProviderModelType, AiProviderProjectScope, aiProviderUtils, AIProviderWithoutSensitiveData, CreateAIProviderRequest, GetProviderConfigResponse, ProjectAIProvider, UpdateAIProviderRequest } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import cron from 'node-cron'+import { EntityManager, FindOptionsWhere } from 'typeorm'import { repoFactory } from '../core/db/repo-factory'import { transaction } from '../core/db/transaction'-import { getAiProviderConfirmKey } from '../database/redis/keys'-import { distributedStore } from '../database/redis-connections'+import { getAiProviderConfirmKey, getManagedAiProviderKeyLockKey } from '../database/redis/keys'+import { distributedLock, distributedStore } from '../database/redis-connections'import { openRouterApi } from '../ee/platform/platform-plan/openrouter/openrouter-api'import { flagService } from '../flags/flag.service'-import { encryptUtils } from '../helper/encryption'+import { EncryptedObject, encryptUtils } from '../helper/encryption'import { platformService } from '../platform/platform.service'import { AIProviderEntity, AIProviderSchema } from './ai-provider-entity'import { aiProviderHealth } from './ai-provider-health'@@ -22,6 +23,7 @@ const modelsCache = new Map<string, AIProviderModel[]>()const MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD = 500const MANAGED_OPENROUTER_KEY_LIMIT_RESET = 'monthly'+const MANAGED_OPENROUTER_KEY_LOCK_TIMEOUT_SECONDS = 30// A passing check must not lock out the next real failure, so the claim is a floor between// checks rather than a window that swallows them. A confirmed failure needs no floor: the row@@ -128,7 +130,7 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({await this.validateProviderCredentials(aiProvider.provider, request.auth, config)}else if (!isNil(request.config)) {- const auth = await decryptRowAuth({ aiProvider, platformId })+ const auth = await decryptRowAuth({ aiProvider, platformId, log })await this.validateProviderCredentials(aiProvider.provider, auth, config)}@@ -173,7 +175,7 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({if (isNil(chatProvider)) {return null}- const auth = await decryptRowAuth({ aiProvider: chatProvider, platformId })+ const auth = await decryptRowAuth({ aiProvider: chatProvider, platformId, log })return { ...aiProviderCredentials({ provider: chatProvider.provider, auth, config: chatProvider.config }), configId: chatProvider.id, platformId, modelScope: chatProvider.modelScope, modelIds: chatProvider.modelIds }},@@ -190,13 +192,20 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({},async delete(platformId: PlatformId, providerId: string): Promise<void> {- await transaction(async (manager) => {+ const deleted = await transaction(async (manager) => {await platformModelTierService.assertKeyCanBeDeleted({ manager, platformId, configId: providerId })- await aiProviderRepo(manager).delete({- platformId,- id: providerId,- })+ return deleteRowForUpdate({ manager, where: { platformId, id: providerId } })})+ await revokeKeyIfManaged({ row: deleted, log })+ },+ async deleteManagedProvider({ platformId, inSameTransaction }: { platformId: PlatformId, inSameTransaction?: (manager: EntityManager) => Promise<unknown> }): Promise<void> {+ const deleted = await transaction(async (manager) => {+ await lockPlatformForUpdate({ manager, platformId })+ const row = await deleteRowForUpdate({ manager, where: { platformId, provider: AIProviderName.ACTIVEPIECES } })+ await inSameTransaction?.(manager)+ return row+ })+ await revokeKeyIfManaged({ row: deleted, log })},async recordKeyObservation({ platformId, providerId, signal }: { platformId: PlatformId, providerId: string, signal: ProviderOutcomeSignal }): Promise<void> {const status = classifyProviderOutcome(signal)@@ -244,7 +253,7 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({if (aiProvider.provider === AIProviderName.ACTIVEPIECES) {return aiProvider.status}- const auth = await decryptRowAuth({ aiProvider, platformId })+ const auth = await decryptRowAuth({ aiProvider, platformId, log })const { error } = await tryCatch(() => aiProviders[aiProvider.provider].validateConnection(auth, aiProvider.config, log))const signal = isNil(error) ? { statusCode: 200 } : toProviderOutcomeSignal(error)const recorded = await aiProviderHealth(log).record({ platformId, providerId, signal, throttled: false, ...spreadIfNotUndefined('expectVersion', expectVersion) })@@ -277,11 +286,11 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({},async getConfigOrThrow({ platformId, provider, scope, configId }: { platformId: PlatformId, provider: AIProviderName, scope: ProviderScope, configId?: string }): Promise<GetProviderConfigResponse> {const aiProvider = await resolveRowForScope({ platformId, provider, scope, configId })- const auth = await decryptRowAuth({ aiProvider, platformId })+ const auth = await decryptRowAuth({ aiProvider, platformId, log })return { ...aiProviderCredentials({ provider: aiProvider.provider, auth, config: aiProvider.config }), configId: aiProvider.id, platformId, modelScope: aiProvider.modelScope, modelIds: aiProvider.modelIds }},async credentialsForTierKey({ platformId, key }: { platformId: PlatformId, key: AIProviderSchema }): Promise<AiProviderCredentials> {- const auth = await decryptRowAuth({ aiProvider: key, platformId })+ const auth = await decryptRowAuth({ aiProvider: key, platformId, log })return aiProviderCredentials({ provider: key.provider, auth, config: key.config })},async getOrCreateActivePiecesProviderAuthConfig(platformId: PlatformId): Promise<ActivePiecesProviderAuthConfig> {@@ -467,7 +476,7 @@ async function getRowByIdOrThrow({ platformId, configId }: { platformId: Platforasync function fetchModels({ aiProvider, platformId, log }: { aiProvider: AIProviderSchema, platformId: PlatformId, log: FastifyBaseLogger }): Promise<AIProviderModel[]> {const { provider, config } = aiProvider- const auth = await decryptRowAuth({ aiProvider, platformId })+ const auth = await decryptRowAuth({ aiProvider, platformId, log })const cacheKey = getModelsCacheKey({ provider, auth, config })if (!modelsCache.has(cacheKey) || 'models' in config) {const { data, error } = await tryCatch(() => aiProviders[provider].listModels(auth, config))@@ -491,17 +500,18 @@ function appliesChatModelIdRule({ provider, config }: { provider: AIProviderNamereturn !('models' in config) && aiProviders[provider].modelIdsAreCustomerNamed !== true}-async function decryptRowAuth({ aiProvider, platformId }: { aiProvider: AIProviderSchema, platformId: PlatformId }): Promise<AIProviderAuthConfig> {+async function decryptRowAuth({ aiProvider, platformId, log }: { aiProvider: AIProviderSchema, platformId: PlatformId, log: FastifyBaseLogger }): Promise<AIProviderAuthConfig> {const auth = await encryptUtils.decryptObject<AIProviderAuthConfig>(aiProvider.auth)- if (aiProvider.provider === AIProviderName.ACTIVEPIECES) {- const doesHaveKeys = !isNil(auth) && 'apiKey' in auth && !isNil(auth.apiKey) && auth.apiKey !== ''- if (!doesHaveKeys) {- return enrichWithKeysIfNeeded(aiProvider, platformId)- }+ if (aiProvider.provider === AIProviderName.ACTIVEPIECES && !hasManagedKey(auth)) {+ return enrichWithKeysIfNeeded({ providerId: aiProvider.id, platformId, log })}return auth}+function hasManagedKey(auth: AIProviderAuthConfig | null): auth is AIProviderAuthConfig {+ return !isNil(auth) && 'apiKey' in auth && !isNil(auth.apiKey) && auth.apiKey !== ''+}+async function findAvailableChatProviderRow({ platformId, scope, log }: { platformId: PlatformId, scope: ProviderScope, log: FastifyBaseLogger }): Promise<AIProviderSchema | null> {const candidates = await aiProviderRepo().findBy([{ platformId, enabledForChat: true },@@ -529,22 +539,86 @@ async function isActivepiecesAiProviderHidden({ platformId, log }: { platformId:return shouldHideActivepiecesAiProvider({ platformId, log })}-async function enrichWithKeysIfNeeded(aiProvider: AIProviderSchema, platformId: PlatformId): Promise<ActivePiecesProviderAuthConfig> {- const { key, data } = await openRouterApi.createKey({- name: `Platform ${platformId}`,- limit: MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD,- limit_reset: MANAGED_OPENROUTER_KEY_LIMIT_RESET,- })- const rawAuth: ActivePiecesProviderAuthConfig = { apiKey: key, apiKeyHash: data.hash }- await aiProviderRepo().save({- id: aiProvider.id,- platformId,- provider: AIProviderName.ACTIVEPIECES,- displayName: 'Activepieces',- config: {},- auth: await encryptUtils.encryptObject(rawAuth),+async function enrichWithKeysIfNeeded({ providerId, platformId, log }: { providerId: string, platformId: PlatformId, log: FastifyBaseLogger }): Promise<AIProviderAuthConfig> {+ return distributedLock(log).runExclusive({+ key: getManagedAiProviderKeyLockKey(platformId),+ timeoutInSeconds: MANAGED_OPENROUTER_KEY_LOCK_TIMEOUT_SECONDS,+ fn: async () => {+ const current = await getRowByIdOrThrow({ platformId, configId: providerId })+ const currentAuth = await encryptUtils.decryptObject<AIProviderAuthConfig>(current.auth)+ if (hasManagedKey(currentAuth)) {+ return currentAuth+ }+ const { key, data } = await openRouterApi.createKey({+ name: `Platform ${platformId}`,+ limit: MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD,+ limit_reset: MANAGED_OPENROUTER_KEY_LIMIT_RESET,+ })+ const minted: ActivePiecesProviderAuthConfig = { apiKey: key, apiKeyHash: data.hash }+ const { data: stored, error } = await tryCatch(() => storeKeyIfAuthUnchanged({ providerId, platformId, observedAuth: current.auth, auth: minted }))+ if (isNil(stored) || managedKeyHash(stored) !== minted.apiKeyHash) {+ await revokeManagedKey({ hash: minted.apiKeyHash, platformId, log })+ }+ if (!isNil(error)) {+ throw error+ }+ if (!hasManagedKey(stored)) {+ throw new Error(`[aiProviderService#enrichWithKeysIfNeeded] Managed AI provider ${providerId} has no stored key`)+ }+ return stored+ },})- return rawAuth+}++async function storeKeyIfAuthUnchanged({ providerId, platformId, observedAuth, auth }: { providerId: string, platformId: PlatformId, observedAuth: EncryptedObject, auth: ActivePiecesProviderAuthConfig }): Promise<AIProviderAuthConfig> {+ await aiProviderRepo().createQueryBuilder()+ .update()+ .set({ auth: await encryptUtils.encryptObject(auth) })+ .where({ id: providerId, platformId })+ .andWhere('"auth"::jsonb = CAST(:observedAuth AS jsonb)', { observedAuth: JSON.stringify(observedAuth) })+ .execute()+ const row = await getRowByIdOrThrow({ platformId, configId: providerId })+ return encryptUtils.decryptObject<AIProviderAuthConfig>(row.auth)+}++async function lockPlatformForUpdate({ manager, platformId }: { manager: EntityManager, platformId: PlatformId }): Promise<void> {+ await manager.query('SELECT 1 FROM "platform" WHERE "id" = $1 FOR UPDATE', [platformId])+}++async function deleteRowForUpdate({ manager, where }: { manager: EntityManager, where: FindOptionsWhere<AIProviderSchema> }): Promise<AIProviderSchema | null> {+ const row = await aiProviderRepo(manager).findOne({ where, lock: { mode: 'pessimistic_write' } })+ if (isNil(row)) {+ return null+ }+ await aiProviderRepo(manager).delete({ platformId: row.platformId, id: row.id })+ return row+}++async function revokeKeyIfManaged({ row, log }: { row: AIProviderSchema | null, log: FastifyBaseLogger }): Promise<void> {+ if (isNil(row) || row.provider !== AIProviderName.ACTIVEPIECES) {+ return+ }+ await revokeStoredManagedKey({ auth: row.auth, platformId: row.platformId, log })+}++function managedKeyHash(auth: AIProviderAuthConfig): string | undefined {+ return 'apiKeyHash' in auth ? auth.apiKeyHash : undefined+}++async function revokeStoredManagedKey({ auth, platformId, log }: { auth: EncryptedObject, platformId: PlatformId, log: FastifyBaseLogger }): Promise<void> {+ const { data: stored } = await tryCatch(() => encryptUtils.decryptObject<AIProviderAuthConfig>(auth))+ const hash = isNil(stored) ? undefined : managedKeyHash(stored)+ if (isNil(hash) || hash === '') {+ return+ }+ await revokeManagedKey({ hash, platformId, log })+}++async function revokeManagedKey({ hash, platformId, log }: { hash: string, platformId: PlatformId, log: FastifyBaseLogger }): Promise<void> {+ const { error } = await tryCatch(() => openRouterApi.deleteKey({ hash }))+ if (!isNil(error)) {+ log.error({ error, platform: { id: platformId }, openRouterKey: { hash } }, '[aiProviderService#revokeManagedKey] Could not revoke a managed OpenRouter key, so it stays live upstream')+ }}diff --git a/packages/server/api/src/app/database/redis/keys.ts b/packages/server/api/src/app/database/redis/keys.tsindex a539d3843e7..bd972823cae 100644--- a/packages/server/api/src/app/database/redis/keys.ts+++ b/packages/server/api/src/app/database/redis/keys.ts@@ -1,6 +1,7 @@import { PlatformId, ProjectId } from '@activepieces/core-utils'export const getAiProviderConfirmKey = (providerId: string): string => `ai_provider:confirm-downgrade:${providerId}`+export const getManagedAiProviderKeyLockKey = (platformId: PlatformId): string => `ai_provider_managed_key_${platformId}`export const getPlatformPlanNameKey = (platformId: PlatformId): string => `platform_plan:plan:${platformId}`export const getCreditsBalanceKey = (platformId: PlatformId): string => `platform_plan:credits:${platformId}`export const getAppSumoAiCreditsBalanceKey = (platformId: PlatformId): string => `platform_plan:appsumo-ai-credits:${platformId}`diff --git a/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts b/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.tsindex 1608135a5f3..0fc49bc075a 100644--- a/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts+++ b/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts@@ -20,6 +20,10 @@ export const openRouterApi = {return openRouterRequest<GetKeyResponse>({ method: 'GET', path: `/keys/${hash}` })},+ async deleteKey({ hash }: DeleteKeyRequest): Promise<void> {+ await openRouterRequest<unknown>({ method: 'DELETE', path: `/keys/${hash}` })+ },+async listKeys({ offset, include_disabled }: ListKeysRequest): Promise<ListKeysResponse> {return openRouterRequest<ListKeysResponse>({method: 'GET',@@ -96,6 +100,10 @@ type GetKeyResponse = {data: OpenRouterApikey}+type DeleteKeyRequest = {+ hash: string+}+type ListKeysRequest = {offset?: numberinclude_disabled?: 'true' | 'false' // default falsediff --git a/packages/server/api/src/app/ee/platform/platform-teardown-jobs.ts b/packages/server/api/src/app/ee/platform/platform-teardown-jobs.tsindex 43a304d09e4..a0e1a68fed6 100644--- a/packages/server/api/src/app/ee/platform/platform-teardown-jobs.ts+++ b/packages/server/api/src/app/ee/platform/platform-teardown-jobs.ts@@ -2,6 +2,7 @@ import { isNil, tryCatch, unique } from '@activepieces/core-utils'import { Flow, FlowOperationType, FlowStatus, UserStatus } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { IsNull, ObjectLiteral, Repository } from 'typeorm'+import { aiProviderService } from '../../ai/ai-provider-service'import { appConnectionsRepo } from '../../app-connection/app-connection-service/app-connection-service'import { userIdentityRepository } from '../../authentication/user-identity/user-identity-service'import { repoFactory } from '../../core/db/repo-factory'@@ -85,7 +86,10 @@ export const platformTeardownJobs = (log: FastifyBaseLogger) => ({.execute()deletedAuditEvents = result.affected ?? 0} while (deletedAuditEvents > 0)- await platformRepo().delete({ id: platformId })+ await aiProviderService(log).deleteManagedProvider({+ platformId,+ inSameTransaction: (manager) => platformRepo(manager).delete({ id: platformId }),+ })const identityIds = await deletePlatformUsers(platformId)await deleteUnreferencedIdentities(identityIds)diff --git a/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts b/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.tsnew file mode 100644index 00000000000..596ce92e7ce--- /dev/null+++ b/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts@@ -0,0 +1,261 @@+import { AIProviderName, apId, isNil } from '@activepieces/core-utils'+import { FastifyInstance } from 'fastify'+import { afterAll, beforeAll, beforeEach, describe, expect, it, MockInstance, vi } from 'vitest'+import { aiProviderService } from '../../../../src/app/ai/ai-provider-service'+import { openRouterApi, OpenRouterApikey } from '../../../../src/app/ee/platform/platform-plan/openrouter/openrouter-api'+import { EncryptedObject, encryptUtils } from '../../../../src/app/helper/encryption'+import { db } from '../../../helpers/db'+import { createTestContext, TestContext } from '../../../helpers/test-context'+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'++let app: FastifyInstance+let ctx: TestContext+let deleteKey: MockInstance<typeof openRouterApi.deleteKey>++beforeAll(async () => {+ app = await setupTestEnvironment({ fresh: true })+}, 300_000)++afterAll(async () => {+ await teardownTestEnvironment()+})++beforeEach(async () => {+ vi.restoreAllMocks()+ deleteKey = vi.spyOn(openRouterApi, 'deleteKey').mockResolvedValue(undefined)+ ctx = await createTestContext(app)+})++function mockOpenRouterKey(hash: string): OpenRouterApikey {+ return {+ hash,+ name: 'test',+ label: 'test',+ disabled: false,+ limit: 500,+ limit_remaining: 500,+ limit_reset: 'monthly',+ include_byok_in_limit: false,+ usage: 0,+ usage_daily: 0,+ usage_weekly: 0,+ usage_monthly: 0,+ byok_usage: 0,+ byok_usage_daily: 0,+ byok_usage_weekly: 0,+ byok_usage_monthly: 0,+ created_at: new Date().toISOString(),+ updated_at: null,+ expires_at: null,+ }+}++function slowMintingKeys(): () => Promise<{ key: string, data: OpenRouterApikey }> {+ let minted = 0+ return async () => {+ minted += 1+ const id = minted+ await new Promise((resolve) => setTimeout(resolve, 100))+ return { key: `sk-or-${id}`, data: mockOpenRouterKey(`hash-${id}`) }+ }+}++async function findManagedRowId(platformId: string): Promise<string> {+ const row = await db.findOneByOrFail<{ id: string }>('ai_provider', {+ platformId,+ provider: AIProviderName.ACTIVEPIECES,+ })+ return row.id+}++async function readPersistedApiKey(platformId: string): Promise<string | undefined> {+ const row = await db.findOneBy<{ auth: EncryptedObject }>('ai_provider', {+ platformId,+ provider: AIProviderName.ACTIVEPIECES,+ })+ if (isNil(row)) {+ return undefined+ }+ const auth = await encryptUtils.decryptObject<{ apiKey?: string }>(row.auth)+ return auth.apiKey+}++describe('managed AI provider key provisioning', () => {+ it('mints exactly one OpenRouter key when two first reads race', async () => {+ const createKey = vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())++ const [first, second] = await Promise.all([+ aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id),+ aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id),+ ])++ expect(createKey).toHaveBeenCalledTimes(1)+ expect(deleteKey).not.toHaveBeenCalled()+ expect(first.apiKey).toBe(second.apiKey)+ expect(first.apiKeyHash).toBe(second.apiKeyHash)+ expect(await readPersistedApiKey(ctx.platform.id)).toBe(first.apiKey)+ })++ it('releases the lock and provisions on the next read when minting fails', async () => {+ const createKey = vi.spyOn(openRouterApi, 'createKey').mockRejectedValueOnce(new Error('[OpenRouter] POST /keys error: 500'))++ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow('[OpenRouter] POST /keys error: 500')+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()+ expect(deleteKey).not.toHaveBeenCalled()++ createKey.mockImplementation(slowMintingKeys())+ const recovered = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)++ expect(recovered.apiKey).toBe('sk-or-1')+ expect(await readPersistedApiKey(ctx.platform.id)).toBe('sk-or-1')+ })++ it('revokes the minted key and does not hand it back when the row is deleted mid-mint', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(async () => {+ await db.delete('ai_provider', await findManagedRowId(ctx.platform.id))+ return { key: 'sk-or-1', data: mockOpenRouterKey('hash-1') }+ })++ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow()+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })+ })++ it('keeps the key another writer stored first and revokes its own', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(async () => {+ await db.update('ai_provider', await findManagedRowId(ctx.platform.id), {+ auth: await encryptUtils.encryptObject({ apiKey: 'sk-or-other', apiKeyHash: 'hash-other' }),+ })+ return { key: 'sk-or-1', data: mockOpenRouterKey('hash-1') }+ })++ const result = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)++ expect(result.apiKey).toBe('sk-or-other')+ expect(result.apiKeyHash).toBe('hash-other')+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })+ expect(await readPersistedApiKey(ctx.platform.id)).toBe('sk-or-other')+ })++ it('revokes the minted key when storing it fails', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(async () => {+ vi.spyOn(encryptUtils, 'encryptObject').mockRejectedValueOnce(new Error('store failed'))+ return { key: 'sk-or-1', data: mockOpenRouterKey('hash-1') }+ })++ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow('store failed')+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })+ })++ it('does not mint again once the platform already has a managed key', async () => {+ const createKey = vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())++ const provisioned = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)+ const [third, fourth] = await Promise.all([+ aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id),+ aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id),+ ])++ expect(createKey).toHaveBeenCalledTimes(1)+ expect(deleteKey).not.toHaveBeenCalled()+ expect(third.apiKey).toBe(provisioned.apiKey)+ expect(fourth.apiKey).toBe(provisioned.apiKey)+ expect(await readPersistedApiKey(ctx.platform.id)).toBe(provisioned.apiKey)+ })+})++describe('managed AI provider deletion', () => {+ it('revokes the stored key when an admin deletes the managed provider', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)++ await aiProviderService(app.log).delete(ctx.platform.id, await findManagedRowId(ctx.platform.id))++ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()+ })++ it('revokes the minted key when an admin deletes the managed provider mid-mint', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(async () => {+ await aiProviderService(app.log).delete(ctx.platform.id, await findManagedRowId(ctx.platform.id))+ return { key: 'sk-or-1', data: mockOpenRouterKey('hash-1') }+ })++ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow()++ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })+ expect(await db.findOneBy('ai_provider', { platformId: ctx.platform.id, provider: AIProviderName.ACTIVEPIECES })).toBeNull()+ })++ it('removes the managed provider on teardown even when a live tier references it', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)+ const managedId = await findManagedRowId(ctx.platform.id)+ await db.save('platform_model_tier', {+ id: apId(),+ platformId: ctx.platform.id,+ name: 'Legacy tier',+ emoji: '⚡',+ position: 0,+ entries: [{ configId: managedId, modelId: 'openai/gpt-4o' }],+ isDefault: true,+ })+ await expect(aiProviderService(app.log).delete(ctx.platform.id, managedId)).rejects.toThrow()++ await aiProviderService(app.log).deleteManagedProvider({ platformId: ctx.platform.id })++ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()+ })++ it('revokes the stored key when the platform teardown removes the managed provider', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)++ await aiProviderService(app.log).deleteManagedProvider({ platformId: ctx.platform.id })++ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()+ })++ it('runs the platform delete in the same transaction and revokes only after it', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)+ const steps: string[] = []+ deleteKey.mockImplementation(async () => {+ steps.push('revoke')+ })++ await aiProviderService(app.log).deleteManagedProvider({+ platformId: ctx.platform.id,+ inSameTransaction: async (manager) => {+ const managedRows = await manager.query('SELECT 1 FROM "ai_provider" WHERE "platformId" = $1 AND "provider" = $2', [ctx.platform.id, AIProviderName.ACTIVEPIECES])+ steps.push(`platform delete sees ${managedRows.length} managed rows`)+ },+ })++ expect(steps).toEqual(['platform delete sees 0 managed rows', 'revoke'])+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })+ })++ it('revokes nothing when the managed provider never got a key', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockRejectedValue(new Error('[OpenRouter] POST /keys error: 500'))+ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow()++ await aiProviderService(app.log).deleteManagedProvider({ platformId: ctx.platform.id })++ expect(deleteKey).not.toHaveBeenCalled()+ expect(await db.findOneBy('ai_provider', { platformId: ctx.platform.id, provider: AIProviderName.ACTIVEPIECES })).toBeNull()+ })++ it('keeps the platform deletable when OpenRouter refuses the revoke', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)+ deleteKey.mockRejectedValueOnce(new Error('[OpenRouter] DELETE /keys/hash-1 error: 500'))++ await aiProviderService(app.log).deleteManagedProvider({ platformId: ctx.platform.id })++ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()+ })+})