Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(ai): stop managed AI credit keys from leaking on OpenRouter (#15704)

Bartosz Majewski9ad57b36ac67
diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.md
index 8def67e8fa7..318b4b2593e 100644
--- a/brain/knowledge/ai-intelligence/ai-providers.md
+++ b/brain/knowledge/ai-intelligence/ai-providers.md
@@ -13,7 +13,7 @@ Lets platform admins configure one or more LLM backends for AI pieces in flows.
### How it works
- `GET /` list (auto-creates ACTIVEPIECES when `aiCreditsEnabled`); `GET /:provider/models` (cached); `POST /` create (validates creds first); `POST /:id` update; `DELETE /:id`.
-- For ACTIVEPIECES, update is blocked by an early return (only enabling `enabledForChat` is allowed); deletion is allowed and self-healing — `listProviders` recreates the managed row.
+- For ACTIVEPIECES, update is blocked by an early return (only enabling `enabledForChat` is allowed); deletion is allowed and self-healing — `listProviders` recreates the managed row. The delete revokes the row's OpenRouter key, and the platform hard delete does the same through `deleteManagedProvider`, because the `ON DELETE CASCADE` from `platform` would otherwise leave the key live. The recreated row mints a fresh $500 key, so a limit raised by `increaseAiCredits` does not survive a delete.
- Engine integration: the AI piece no longer fetches credentials. It POSTs `/v1/ai/execute`, the server enqueues an `EXECUTE_AI` worker job, and the worker resolves the provider over the `resolveAiProvider` RPC. `GET /:provider/config` was removed in [#15494](https://github.com/activepieces/activepieces/pull/15494).
- For the managed ACTIVEPIECES provider that config route is also the **credit gate**: `assertCreditsAndAppSumoNotExceeded` (`platform/billing-provider.ts`) throws `QUOTA_EXCEEDED` when the Autumn credit or AppSumo balance is blocked — fires per AI call, but usage is only metered post-run, so in-flight spend is invisible to it. See decision 000016 (`brain/decisions/000016-managed-ai-metering-moves-to-centralized-worker-execution.md`).
- Activepieces provisioning: `getOrCreateActivePiecesProviderAuthConfig()` → `enrichWithKeysIfNeeded()` mints an OpenRouter key. No system job is scheduled — renewal/top-up is driven by Autumn (`autoTopUps` in `autumn-billing.ts`), not Stripe.
@@ -189,8 +189,8 @@ after a file without chat keys it seeds the chat list from the live flow list.
`STORE_VALUE_MAX_SIZE`. So the history cannot grow unbounded and cannot outgrow a request body — no
separate limit is needed wherever it is passed along. An older version of the action could write a
`{ messages: [] }` wrapper instead of a bare array, so any reader has to accept both shapes.
-- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `chatHelpers.resolveChatProvider` → `getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. There is also no distributed lock or cache, so concurrent turns for one platform each mint a live key and only the last is saved; the orphans keep spending provisioning quota. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getChatConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.
-- **`openrouter-api.ts` uses raw `fetch`** — no timeout, no retry, no `tryCatch`, and it bypasses the repo's `safeHttp` rule for outbound HTTP in `packages/server/api`. A `getKey` 408 from OpenRouter escapes the admin `increaseAiCredits` path as an unhandled rejection.
+- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `chatHelpers.resolveChatProvider` → `getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. The mint runs under a per-platform `distributedLock` and re-reads the row inside it, and the write lands only if `auth` still matches the ciphertext it read (a `jsonb` compare; the row is then read back, since `affected` is unreliable on PGlite). A key that loses that write, or whose write throws, is revoked with `openRouterApi.deleteKey`. Two leaks stay open until a reconcile sweep over `listKeys` exists: a crash between `createKey` and the write, and a `createKey` that times out after OpenRouter has made the key. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getChatConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.
+- **`openrouter-api.ts` has no retry, and every failure is a plain `Error`.** Calls go through `safeHttp.axios` with a 15 s timeout, and `openRouterRequest` rethrows any failure as `[OpenRouter] <METHOD> <path> error: <status> <body>`, so a caller cannot tell a 429 from a 500 without parsing the message.
- **Chat model tiers are Activepieces-only.** `ACTIVEPIECES_CHAT_TIERS` (`fast`/`smart`/`premium`, shown as Fast/Expert/Heavy) hold OpenRouter-shaped Anthropic ids, so they only mean anything for the ACTIVEPIECES and OPENROUTER chat providers. Any provider that declares `ALLOWED_CHAT_MODELS_BY_PROVIDER` (openai, anthropic, google) picks a real model id from that list instead. Naively stripping the tier's vendor prefix for every provider is what once sent `claude-haiku-4-5` to OpenAI and broke every message.
- **A short model list in the key's picker is the vendor's catalog, not a truncation.** `listModels` returns whatever the provider's own `/models` endpoint gives and filters nothing except the key's `modelScope` allow-list. Anthropic ships roughly a dozen models, OpenAI ~80 (mostly embeddings/tts/whisper), while OpenRouter is an aggregator and returns 400+ from every vendor it proxies — so the counts differ by an order of magnitude by design. Anthropic pages at 20 by default, which is why its request pins `?limit=1000`. If the list shows exactly three Claude models, that is the *chat* dropdown reading the curated `ANTHROPIC_CHAT_MODELS`, a different surface from the admin picker.
- **Read the chat model list through `aiProviderUtils.getCuratedChatModels({ provider })`.** It is the one accessor the server resolver (`agentHelpers.resolveModelIdForProvider`) and the chat dropdown share, so the two cannot drift; it returns `{ id, label }` or `undefined` — never an empty list, so callers may treat a result as non-empty. Labels come from the hardcoded `CHAT_MODEL_LABELS` (falling back to the id) rather than `AIProviderModel.name`, because the live `listModels` response cannot supply one for every provider: anthropic returns `display_name` and google `displayName`, but OpenAI's `/v1/models` returns ids only.
@@ -218,7 +218,7 @@ after a file without chat keys it seeds the chat list from the live flow list.
- `packages/server/api/src/app/ai/` — backend module: provider controller, service, entity, module, plus the sibling ai-tool-config files
- `packages/server/api/src/app/ai/providers/` — per-vendor strategies keyed by `AIProviderName`
- `packages/server/api/src/app/platform/billing-provider.ts` — `assertCreditsAndAppSumoNotExceeded` credit gate
-- `packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts` — the OpenRouter provisioning client (`createKey`/`updateKey`/`getKey`/`listKeys`)
+- `packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts` — the OpenRouter provisioning client (`createKey`/`updateKey`/`getKey`/`listKeys`/`deleteKey`)
- `packages/core/ai-providers/src/lib/create-language-model.ts` — the shared per-provider model factory (`createLanguageModel`, `buildOpenAICompatibleHeaders`) used by both the AI piece and the agent path
- `packages/core/shared/src/lib/management/ai-providers/index.ts` — shared zod schemas, enums, request/response types
- `packages/core/shared/src/lib/management/ai-tools/index.ts` — shared schemas for the AI Tool Configs sibling
diff --git a/packages/server/api/src/app/ai/ai-provider-service.ts b/packages/server/api/src/app/ai/ai-provider-service.ts
index e196cad5f8e..479aba38e45 100644
--- a/packages/server/api/src/app/ai/ai-provider-service.ts
+++ b/packages/server/api/src/app/ai/ai-provider-service.ts
@@ -3,13 +3,14 @@ import { modelCatalog, modelTierCatalog } from '@activepieces/server-utils'
import { ActivePiecesProviderAuthConfig, AI_PROVIDER_ENTITY_TYPES, AIProviderAuthConfig, AIProviderConfig, aiProviderCredentials, AIProviderModel, AIProviderModelType, AiProviderProjectScope, aiProviderUtils, AIProviderWithoutSensitiveData, CreateAIProviderRequest, GetProviderConfigResponse, ProjectAIProvider, UpdateAIProviderRequest } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import cron from 'node-cron'
+import { EntityManager, FindOptionsWhere } from 'typeorm'
import { repoFactory } from '../core/db/repo-factory'
import { transaction } from '../core/db/transaction'
-import { getAiProviderConfirmKey } from '../database/redis/keys'
-import { distributedStore } from '../database/redis-connections'
+import { getAiProviderConfirmKey, getManagedAiProviderKeyLockKey } from '../database/redis/keys'
+import { distributedLock, distributedStore } from '../database/redis-connections'
import { openRouterApi } from '../ee/platform/platform-plan/openrouter/openrouter-api'
import { flagService } from '../flags/flag.service'
-import { encryptUtils } from '../helper/encryption'
+import { EncryptedObject, encryptUtils } from '../helper/encryption'
import { platformService } from '../platform/platform.service'
import { AIProviderEntity, AIProviderSchema } from './ai-provider-entity'
import { aiProviderHealth } from './ai-provider-health'
@@ -22,6 +23,7 @@ const modelsCache = new Map<string, AIProviderModel[]>()
const MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD = 500
const MANAGED_OPENROUTER_KEY_LIMIT_RESET = 'monthly'
+const MANAGED_OPENROUTER_KEY_LOCK_TIMEOUT_SECONDS = 30
// A passing check must not lock out the next real failure, so the claim is a floor between
// checks rather than a window that swallows them. A confirmed failure needs no floor: the row
@@ -128,7 +130,7 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({
await this.validateProviderCredentials(aiProvider.provider, request.auth, config)
}
else if (!isNil(request.config)) {
- const auth = await decryptRowAuth({ aiProvider, platformId })
+ const auth = await decryptRowAuth({ aiProvider, platformId, log })
await this.validateProviderCredentials(aiProvider.provider, auth, config)
}
@@ -173,7 +175,7 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({
if (isNil(chatProvider)) {
return null
}
- const auth = await decryptRowAuth({ aiProvider: chatProvider, platformId })
+ const auth = await decryptRowAuth({ aiProvider: chatProvider, platformId, log })
return { ...aiProviderCredentials({ provider: chatProvider.provider, auth, config: chatProvider.config }), configId: chatProvider.id, platformId, modelScope: chatProvider.modelScope, modelIds: chatProvider.modelIds }
},
@@ -190,13 +192,20 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({
},
async delete(platformId: PlatformId, providerId: string): Promise<void> {
- await transaction(async (manager) => {
+ const deleted = await transaction(async (manager) => {
await platformModelTierService.assertKeyCanBeDeleted({ manager, platformId, configId: providerId })
- await aiProviderRepo(manager).delete({
- platformId,
- id: providerId,
- })
+ return deleteRowForUpdate({ manager, where: { platformId, id: providerId } })
})
+ await revokeKeyIfManaged({ row: deleted, log })
+ },
+ async deleteManagedProvider({ platformId, inSameTransaction }: { platformId: PlatformId, inSameTransaction?: (manager: EntityManager) => Promise<unknown> }): Promise<void> {
+ const deleted = await transaction(async (manager) => {
+ await lockPlatformForUpdate({ manager, platformId })
+ const row = await deleteRowForUpdate({ manager, where: { platformId, provider: AIProviderName.ACTIVEPIECES } })
+ await inSameTransaction?.(manager)
+ return row
+ })
+ await revokeKeyIfManaged({ row: deleted, log })
},
async recordKeyObservation({ platformId, providerId, signal }: { platformId: PlatformId, providerId: string, signal: ProviderOutcomeSignal }): Promise<void> {
const status = classifyProviderOutcome(signal)
@@ -244,7 +253,7 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({
if (aiProvider.provider === AIProviderName.ACTIVEPIECES) {
return aiProvider.status
}
- const auth = await decryptRowAuth({ aiProvider, platformId })
+ const auth = await decryptRowAuth({ aiProvider, platformId, log })
const { error } = await tryCatch(() => aiProviders[aiProvider.provider].validateConnection(auth, aiProvider.config, log))
const signal = isNil(error) ? { statusCode: 200 } : toProviderOutcomeSignal(error)
const recorded = await aiProviderHealth(log).record({ platformId, providerId, signal, throttled: false, ...spreadIfNotUndefined('expectVersion', expectVersion) })
@@ -277,11 +286,11 @@ export const aiProviderService = (log: FastifyBaseLogger) => ({
},
async getConfigOrThrow({ platformId, provider, scope, configId }: { platformId: PlatformId, provider: AIProviderName, scope: ProviderScope, configId?: string }): Promise<GetProviderConfigResponse> {
const aiProvider = await resolveRowForScope({ platformId, provider, scope, configId })
- const auth = await decryptRowAuth({ aiProvider, platformId })
+ const auth = await decryptRowAuth({ aiProvider, platformId, log })
return { ...aiProviderCredentials({ provider: aiProvider.provider, auth, config: aiProvider.config }), configId: aiProvider.id, platformId, modelScope: aiProvider.modelScope, modelIds: aiProvider.modelIds }
},
async credentialsForTierKey({ platformId, key }: { platformId: PlatformId, key: AIProviderSchema }): Promise<AiProviderCredentials> {
- const auth = await decryptRowAuth({ aiProvider: key, platformId })
+ const auth = await decryptRowAuth({ aiProvider: key, platformId, log })
return aiProviderCredentials({ provider: key.provider, auth, config: key.config })
},
async getOrCreateActivePiecesProviderAuthConfig(platformId: PlatformId): Promise<ActivePiecesProviderAuthConfig> {
@@ -467,7 +476,7 @@ async function getRowByIdOrThrow({ platformId, configId }: { platformId: Platfor
async function fetchModels({ aiProvider, platformId, log }: { aiProvider: AIProviderSchema, platformId: PlatformId, log: FastifyBaseLogger }): Promise<AIProviderModel[]> {
const { provider, config } = aiProvider
- const auth = await decryptRowAuth({ aiProvider, platformId })
+ const auth = await decryptRowAuth({ aiProvider, platformId, log })
const cacheKey = getModelsCacheKey({ provider, auth, config })
if (!modelsCache.has(cacheKey) || 'models' in config) {
const { data, error } = await tryCatch(() => aiProviders[provider].listModels(auth, config))
@@ -491,17 +500,18 @@ function appliesChatModelIdRule({ provider, config }: { provider: AIProviderName
return !('models' in config) && aiProviders[provider].modelIdsAreCustomerNamed !== true
}
-async function decryptRowAuth({ aiProvider, platformId }: { aiProvider: AIProviderSchema, platformId: PlatformId }): Promise<AIProviderAuthConfig> {
+async function decryptRowAuth({ aiProvider, platformId, log }: { aiProvider: AIProviderSchema, platformId: PlatformId, log: FastifyBaseLogger }): Promise<AIProviderAuthConfig> {
const auth = await encryptUtils.decryptObject<AIProviderAuthConfig>(aiProvider.auth)
- if (aiProvider.provider === AIProviderName.ACTIVEPIECES) {
- const doesHaveKeys = !isNil(auth) && 'apiKey' in auth && !isNil(auth.apiKey) && auth.apiKey !== ''
- if (!doesHaveKeys) {
- return enrichWithKeysIfNeeded(aiProvider, platformId)
- }
+ if (aiProvider.provider === AIProviderName.ACTIVEPIECES && !hasManagedKey(auth)) {
+ return enrichWithKeysIfNeeded({ providerId: aiProvider.id, platformId, log })
}
return auth
}
+function hasManagedKey(auth: AIProviderAuthConfig | null): auth is AIProviderAuthConfig {
+ return !isNil(auth) && 'apiKey' in auth && !isNil(auth.apiKey) && auth.apiKey !== ''
+}
+
async function findAvailableChatProviderRow({ platformId, scope, log }: { platformId: PlatformId, scope: ProviderScope, log: FastifyBaseLogger }): Promise<AIProviderSchema | null> {
const candidates = await aiProviderRepo().findBy([
{ platformId, enabledForChat: true },
@@ -529,22 +539,86 @@ async function isActivepiecesAiProviderHidden({ platformId, log }: { platformId:
return shouldHideActivepiecesAiProvider({ platformId, log })
}
-async function enrichWithKeysIfNeeded(aiProvider: AIProviderSchema, platformId: PlatformId): Promise<ActivePiecesProviderAuthConfig> {
- const { key, data } = await openRouterApi.createKey({
- name: `Platform ${platformId}`,
- limit: MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD,
- limit_reset: MANAGED_OPENROUTER_KEY_LIMIT_RESET,
- })
- const rawAuth: ActivePiecesProviderAuthConfig = { apiKey: key, apiKeyHash: data.hash }
- await aiProviderRepo().save({
- id: aiProvider.id,
- platformId,
- provider: AIProviderName.ACTIVEPIECES,
- displayName: 'Activepieces',
- config: {},
- auth: await encryptUtils.encryptObject(rawAuth),
+async function enrichWithKeysIfNeeded({ providerId, platformId, log }: { providerId: string, platformId: PlatformId, log: FastifyBaseLogger }): Promise<AIProviderAuthConfig> {
+ return distributedLock(log).runExclusive({
+ key: getManagedAiProviderKeyLockKey(platformId),
+ timeoutInSeconds: MANAGED_OPENROUTER_KEY_LOCK_TIMEOUT_SECONDS,
+ fn: async () => {
+ const current = await getRowByIdOrThrow({ platformId, configId: providerId })
+ const currentAuth = await encryptUtils.decryptObject<AIProviderAuthConfig>(current.auth)
+ if (hasManagedKey(currentAuth)) {
+ return currentAuth
+ }
+ const { key, data } = await openRouterApi.createKey({
+ name: `Platform ${platformId}`,
+ limit: MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD,
+ limit_reset: MANAGED_OPENROUTER_KEY_LIMIT_RESET,
+ })
+ const minted: ActivePiecesProviderAuthConfig = { apiKey: key, apiKeyHash: data.hash }
+ const { data: stored, error } = await tryCatch(() => storeKeyIfAuthUnchanged({ providerId, platformId, observedAuth: current.auth, auth: minted }))
+ if (isNil(stored) || managedKeyHash(stored) !== minted.apiKeyHash) {
+ await revokeManagedKey({ hash: minted.apiKeyHash, platformId, log })
+ }
+ if (!isNil(error)) {
+ throw error
+ }
+ if (!hasManagedKey(stored)) {
+ throw new Error(`[aiProviderService#enrichWithKeysIfNeeded] Managed AI provider ${providerId} has no stored key`)
+ }
+ return stored
+ },
})
- return rawAuth
+}
+
+async function storeKeyIfAuthUnchanged({ providerId, platformId, observedAuth, auth }: { providerId: string, platformId: PlatformId, observedAuth: EncryptedObject, auth: ActivePiecesProviderAuthConfig }): Promise<AIProviderAuthConfig> {
+ await aiProviderRepo().createQueryBuilder()
+ .update()
+ .set({ auth: await encryptUtils.encryptObject(auth) })
+ .where({ id: providerId, platformId })
+ .andWhere('"auth"::jsonb = CAST(:observedAuth AS jsonb)', { observedAuth: JSON.stringify(observedAuth) })
+ .execute()
+ const row = await getRowByIdOrThrow({ platformId, configId: providerId })
+ return encryptUtils.decryptObject<AIProviderAuthConfig>(row.auth)
+}
+
+async function lockPlatformForUpdate({ manager, platformId }: { manager: EntityManager, platformId: PlatformId }): Promise<void> {
+ await manager.query('SELECT 1 FROM "platform" WHERE "id" = $1 FOR UPDATE', [platformId])
+}
+
+async function deleteRowForUpdate({ manager, where }: { manager: EntityManager, where: FindOptionsWhere<AIProviderSchema> }): Promise<AIProviderSchema | null> {
+ const row = await aiProviderRepo(manager).findOne({ where, lock: { mode: 'pessimistic_write' } })
+ if (isNil(row)) {
+ return null
+ }
+ await aiProviderRepo(manager).delete({ platformId: row.platformId, id: row.id })
+ return row
+}
+
+async function revokeKeyIfManaged({ row, log }: { row: AIProviderSchema | null, log: FastifyBaseLogger }): Promise<void> {
+ if (isNil(row) || row.provider !== AIProviderName.ACTIVEPIECES) {
+ return
+ }
+ await revokeStoredManagedKey({ auth: row.auth, platformId: row.platformId, log })
+}
+
+function managedKeyHash(auth: AIProviderAuthConfig): string | undefined {
+ return 'apiKeyHash' in auth ? auth.apiKeyHash : undefined
+}
+
+async function revokeStoredManagedKey({ auth, platformId, log }: { auth: EncryptedObject, platformId: PlatformId, log: FastifyBaseLogger }): Promise<void> {
+ const { data: stored } = await tryCatch(() => encryptUtils.decryptObject<AIProviderAuthConfig>(auth))
+ const hash = isNil(stored) ? undefined : managedKeyHash(stored)
+ if (isNil(hash) || hash === '') {
+ return
+ }
+ await revokeManagedKey({ hash, platformId, log })
+}
+
+async function revokeManagedKey({ hash, platformId, log }: { hash: string, platformId: PlatformId, log: FastifyBaseLogger }): Promise<void> {
+ const { error } = await tryCatch(() => openRouterApi.deleteKey({ hash }))
+ if (!isNil(error)) {
+ log.error({ error, platform: { id: platformId }, openRouterKey: { hash } }, '[aiProviderService#revokeManagedKey] Could not revoke a managed OpenRouter key, so it stays live upstream')
+ }
}
diff --git a/packages/server/api/src/app/database/redis/keys.ts b/packages/server/api/src/app/database/redis/keys.ts
index a539d3843e7..bd972823cae 100644
--- a/packages/server/api/src/app/database/redis/keys.ts
+++ b/packages/server/api/src/app/database/redis/keys.ts
@@ -1,6 +1,7 @@
import { PlatformId, ProjectId } from '@activepieces/core-utils'
export const getAiProviderConfirmKey = (providerId: string): string => `ai_provider:confirm-downgrade:${providerId}`
+export const getManagedAiProviderKeyLockKey = (platformId: PlatformId): string => `ai_provider_managed_key_${platformId}`
export const getPlatformPlanNameKey = (platformId: PlatformId): string => `platform_plan:plan:${platformId}`
export const getCreditsBalanceKey = (platformId: PlatformId): string => `platform_plan:credits:${platformId}`
export const getAppSumoAiCreditsBalanceKey = (platformId: PlatformId): string => `platform_plan:appsumo-ai-credits:${platformId}`
diff --git a/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts b/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts
index 1608135a5f3..0fc49bc075a 100644
--- a/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts
+++ b/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts
@@ -20,6 +20,10 @@ export const openRouterApi = {
return openRouterRequest<GetKeyResponse>({ method: 'GET', path: `/keys/${hash}` })
},
+ async deleteKey({ hash }: DeleteKeyRequest): Promise<void> {
+ await openRouterRequest<unknown>({ method: 'DELETE', path: `/keys/${hash}` })
+ },
+
async listKeys({ offset, include_disabled }: ListKeysRequest): Promise<ListKeysResponse> {
return openRouterRequest<ListKeysResponse>({
method: 'GET',
@@ -96,6 +100,10 @@ type GetKeyResponse = {
data: OpenRouterApikey
}
+type DeleteKeyRequest = {
+ hash: string
+}
+
type ListKeysRequest = {
offset?: number
include_disabled?: 'true' | 'false' // default false
diff --git a/packages/server/api/src/app/ee/platform/platform-teardown-jobs.ts b/packages/server/api/src/app/ee/platform/platform-teardown-jobs.ts
index 43a304d09e4..a0e1a68fed6 100644
--- a/packages/server/api/src/app/ee/platform/platform-teardown-jobs.ts
+++ b/packages/server/api/src/app/ee/platform/platform-teardown-jobs.ts
@@ -2,6 +2,7 @@ import { isNil, tryCatch, unique } from '@activepieces/core-utils'
import { Flow, FlowOperationType, FlowStatus, UserStatus } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { IsNull, ObjectLiteral, Repository } from 'typeorm'
+import { aiProviderService } from '../../ai/ai-provider-service'
import { appConnectionsRepo } from '../../app-connection/app-connection-service/app-connection-service'
import { userIdentityRepository } from '../../authentication/user-identity/user-identity-service'
import { repoFactory } from '../../core/db/repo-factory'
@@ -85,7 +86,10 @@ export const platformTeardownJobs = (log: FastifyBaseLogger) => ({
.execute()
deletedAuditEvents = result.affected ?? 0
} while (deletedAuditEvents > 0)
- await platformRepo().delete({ id: platformId })
+ await aiProviderService(log).deleteManagedProvider({
+ platformId,
+ inSameTransaction: (manager) => platformRepo(manager).delete({ id: platformId }),
+ })
const identityIds = await deletePlatformUsers(platformId)
await deleteUnreferencedIdentities(identityIds)
diff --git a/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts b/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts
new file mode 100644
index 00000000000..596ce92e7ce
--- /dev/null
+++ b/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts
@@ -0,0 +1,261 @@
+import { AIProviderName, apId, isNil } from '@activepieces/core-utils'
+import { FastifyInstance } from 'fastify'
+import { afterAll, beforeAll, beforeEach, describe, expect, it, MockInstance, vi } from 'vitest'
+import { aiProviderService } from '../../../../src/app/ai/ai-provider-service'
+import { openRouterApi, OpenRouterApikey } from '../../../../src/app/ee/platform/platform-plan/openrouter/openrouter-api'
+import { EncryptedObject, encryptUtils } from '../../../../src/app/helper/encryption'
+import { db } from '../../../helpers/db'
+import { createTestContext, TestContext } from '../../../helpers/test-context'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+
+let app: FastifyInstance
+let ctx: TestContext
+let deleteKey: MockInstance<typeof openRouterApi.deleteKey>
+
+beforeAll(async () => {
+ app = await setupTestEnvironment({ fresh: true })
+}, 300_000)
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+beforeEach(async () => {
+ vi.restoreAllMocks()
+ deleteKey = vi.spyOn(openRouterApi, 'deleteKey').mockResolvedValue(undefined)
+ ctx = await createTestContext(app)
+})
+
+function mockOpenRouterKey(hash: string): OpenRouterApikey {
+ return {
+ hash,
+ name: 'test',
+ label: 'test',
+ disabled: false,
+ limit: 500,
+ limit_remaining: 500,
+ limit_reset: 'monthly',
+ include_byok_in_limit: false,
+ usage: 0,
+ usage_daily: 0,
+ usage_weekly: 0,
+ usage_monthly: 0,
+ byok_usage: 0,
+ byok_usage_daily: 0,
+ byok_usage_weekly: 0,
+ byok_usage_monthly: 0,
+ created_at: new Date().toISOString(),
+ updated_at: null,
+ expires_at: null,
+ }
+}
+
+function slowMintingKeys(): () => Promise<{ key: string, data: OpenRouterApikey }> {
+ let minted = 0
+ return async () => {
+ minted += 1
+ const id = minted
+ await new Promise((resolve) => setTimeout(resolve, 100))
+ return { key: `sk-or-${id}`, data: mockOpenRouterKey(`hash-${id}`) }
+ }
+}
+
+async function findManagedRowId(platformId: string): Promise<string> {
+ const row = await db.findOneByOrFail<{ id: string }>('ai_provider', {
+ platformId,
+ provider: AIProviderName.ACTIVEPIECES,
+ })
+ return row.id
+}
+
+async function readPersistedApiKey(platformId: string): Promise<string | undefined> {
+ const row = await db.findOneBy<{ auth: EncryptedObject }>('ai_provider', {
+ platformId,
+ provider: AIProviderName.ACTIVEPIECES,
+ })
+ if (isNil(row)) {
+ return undefined
+ }
+ const auth = await encryptUtils.decryptObject<{ apiKey?: string }>(row.auth)
+ return auth.apiKey
+}
+
+describe('managed AI provider key provisioning', () => {
+ it('mints exactly one OpenRouter key when two first reads race', async () => {
+ const createKey = vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+
+ const [first, second] = await Promise.all([
+ aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id),
+ aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id),
+ ])
+
+ expect(createKey).toHaveBeenCalledTimes(1)
+ expect(deleteKey).not.toHaveBeenCalled()
+ expect(first.apiKey).toBe(second.apiKey)
+ expect(first.apiKeyHash).toBe(second.apiKeyHash)
+ expect(await readPersistedApiKey(ctx.platform.id)).toBe(first.apiKey)
+ })
+
+ it('releases the lock and provisions on the next read when minting fails', async () => {
+ const createKey = vi.spyOn(openRouterApi, 'createKey').mockRejectedValueOnce(new Error('[OpenRouter] POST /keys error: 500'))
+
+ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow('[OpenRouter] POST /keys error: 500')
+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
+ expect(deleteKey).not.toHaveBeenCalled()
+
+ createKey.mockImplementation(slowMintingKeys())
+ const recovered = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+
+ expect(recovered.apiKey).toBe('sk-or-1')
+ expect(await readPersistedApiKey(ctx.platform.id)).toBe('sk-or-1')
+ })
+
+ it('revokes the minted key and does not hand it back when the row is deleted mid-mint', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(async () => {
+ await db.delete('ai_provider', await findManagedRowId(ctx.platform.id))
+ return { key: 'sk-or-1', data: mockOpenRouterKey('hash-1') }
+ })
+
+ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow()
+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })
+ })
+
+ it('keeps the key another writer stored first and revokes its own', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(async () => {
+ await db.update('ai_provider', await findManagedRowId(ctx.platform.id), {
+ auth: await encryptUtils.encryptObject({ apiKey: 'sk-or-other', apiKeyHash: 'hash-other' }),
+ })
+ return { key: 'sk-or-1', data: mockOpenRouterKey('hash-1') }
+ })
+
+ const result = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+
+ expect(result.apiKey).toBe('sk-or-other')
+ expect(result.apiKeyHash).toBe('hash-other')
+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })
+ expect(await readPersistedApiKey(ctx.platform.id)).toBe('sk-or-other')
+ })
+
+ it('revokes the minted key when storing it fails', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(async () => {
+ vi.spyOn(encryptUtils, 'encryptObject').mockRejectedValueOnce(new Error('store failed'))
+ return { key: 'sk-or-1', data: mockOpenRouterKey('hash-1') }
+ })
+
+ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow('store failed')
+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })
+ })
+
+ it('does not mint again once the platform already has a managed key', async () => {
+ const createKey = vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+
+ const provisioned = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+ const [third, fourth] = await Promise.all([
+ aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id),
+ aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id),
+ ])
+
+ expect(createKey).toHaveBeenCalledTimes(1)
+ expect(deleteKey).not.toHaveBeenCalled()
+ expect(third.apiKey).toBe(provisioned.apiKey)
+ expect(fourth.apiKey).toBe(provisioned.apiKey)
+ expect(await readPersistedApiKey(ctx.platform.id)).toBe(provisioned.apiKey)
+ })
+})
+
+describe('managed AI provider deletion', () => {
+ it('revokes the stored key when an admin deletes the managed provider', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+
+ await aiProviderService(app.log).delete(ctx.platform.id, await findManagedRowId(ctx.platform.id))
+
+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })
+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
+ })
+
+ it('revokes the minted key when an admin deletes the managed provider mid-mint', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(async () => {
+ await aiProviderService(app.log).delete(ctx.platform.id, await findManagedRowId(ctx.platform.id))
+ return { key: 'sk-or-1', data: mockOpenRouterKey('hash-1') }
+ })
+
+ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow()
+
+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })
+ expect(await db.findOneBy('ai_provider', { platformId: ctx.platform.id, provider: AIProviderName.ACTIVEPIECES })).toBeNull()
+ })
+
+ it('removes the managed provider on teardown even when a live tier references it', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+ const managedId = await findManagedRowId(ctx.platform.id)
+ await db.save('platform_model_tier', {
+ id: apId(),
+ platformId: ctx.platform.id,
+ name: 'Legacy tier',
+ emoji: '⚡',
+ position: 0,
+ entries: [{ configId: managedId, modelId: 'openai/gpt-4o' }],
+ isDefault: true,
+ })
+ await expect(aiProviderService(app.log).delete(ctx.platform.id, managedId)).rejects.toThrow()
+
+ await aiProviderService(app.log).deleteManagedProvider({ platformId: ctx.platform.id })
+
+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })
+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
+ })
+
+ it('revokes the stored key when the platform teardown removes the managed provider', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+
+ await aiProviderService(app.log).deleteManagedProvider({ platformId: ctx.platform.id })
+
+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })
+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
+ })
+
+ it('runs the platform delete in the same transaction and revokes only after it', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+ const steps: string[] = []
+ deleteKey.mockImplementation(async () => {
+ steps.push('revoke')
+ })
+
+ await aiProviderService(app.log).deleteManagedProvider({
+ platformId: ctx.platform.id,
+ inSameTransaction: async (manager) => {
+ const managedRows = await manager.query('SELECT 1 FROM "ai_provider" WHERE "platformId" = $1 AND "provider" = $2', [ctx.platform.id, AIProviderName.ACTIVEPIECES])
+ steps.push(`platform delete sees ${managedRows.length} managed rows`)
+ },
+ })
+
+ expect(steps).toEqual(['platform delete sees 0 managed rows', 'revoke'])
+ expect(deleteKey).toHaveBeenCalledExactlyOnceWith({ hash: 'hash-1' })
+ })
+
+ it('revokes nothing when the managed provider never got a key', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockRejectedValue(new Error('[OpenRouter] POST /keys error: 500'))
+ await expect(aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)).rejects.toThrow()
+
+ await aiProviderService(app.log).deleteManagedProvider({ platformId: ctx.platform.id })
+
+ expect(deleteKey).not.toHaveBeenCalled()
+ expect(await db.findOneBy('ai_provider', { platformId: ctx.platform.id, provider: AIProviderName.ACTIVEPIECES })).toBeNull()
+ })
+
+ it('keeps the platform deletable when OpenRouter refuses the revoke', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+ await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+ deleteKey.mockRejectedValueOnce(new Error('[OpenRouter] DELETE /keys/hash-1 error: 500'))
+
+ await aiProviderService(app.log).deleteManagedProvider({ platformId: ctx.platform.id })
+
+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
+ })
+})