AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(ai): a new platform's first AI call no longer fails with "User not found" (#16935)
Hazem Adel91e9fa77bb88
diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.mdindex b271f90b0ad..e54ea8330c6 100644--- a/brain/knowledge/ai-intelligence/ai-providers.md+++ b/brain/knowledge/ai-intelligence/ai-providers.md@@ -210,6 +210,7 @@ after a file without chat keys it seeds the chat list from the live flow list.separate limit is needed wherever it is passed along. An older version of the action could write a`{ messages: [] }` wrapper instead of a bare array, so any reader has to accept both shapes.- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `chatHelpers.resolveChatProvider` → `getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. The mint runs under a per-platform `distributedLock` and re-reads the row inside it, and the write lands only if `auth` still matches the ciphertext it read (a `jsonb` compare; the row is then read back, since `affected` is unreliable on PGlite). A key that loses that write, or whose write throws, is revoked with `openRouterApi.deleteKey`. Two leaks stay open until a reconcile sweep over `listKeys` exists: a crash between `createKey` and the write, and a `createKey` that times out after OpenRouter has made the key. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getChatConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.+- **A freshly minted managed OpenRouter key can answer 401 "User not found." for up to about 20s, so `enrichWithKeysIfNeeded` only saves a new key once it answers.** In prod that failed the first chat turns of about 15% of new platforms (AI-158). `GET /api/v1/key` is not a good enough check: in a 150-key run (2026-10-27) 7 keys passed it 3 times in a row and then still failed 1 to 3 completions. So `openRouterApi.isKeyLive` sends a 1-token completion and counts only a 401 or no answer as not live (so a retired or down probe model can't stall every new platform for 30s), and the wait needs 3 passes in a row 1s apart. With that, 1 of 150 keys failed one call afterwards, which chat's single retry absorbs (median wait 5.4s, p95 22s). How long a key takes varies with OpenRouter's load, and hammering it with 5+ new keys at once makes keys flap, so measure at most 3 at a time. The wait runs for up to 30s inside the mint lock, *before* the save, so a concurrent call still sees an empty row and queues on the lock instead of reading a key that isn't live (lock timeout 60s, so waiters outlast a 15s mint plus the wait plus one 10s check). On timeout it logs `waitUntilKeyIsLive` and saves the key anyway.- **`openrouter-api.ts` has no retry, and every failure is a plain `Error`.** Calls go through `safeHttp.axios` with a 15 s timeout, and `openRouterRequest` rethrows any failure as `[OpenRouter] <METHOD> <path> error: <status> <body>`, so a caller cannot tell a 429 from a 500 without parsing the message.- **Chat model tiers are Activepieces-only.** `ACTIVEPIECES_CHAT_TIERS` (`fast`/`smart`/`premium`, shown as Fast/Expert/Heavy) hold OpenRouter-shaped Anthropic ids, so they only mean anything for the ACTIVEPIECES and OPENROUTER chat providers. Any provider that declares `ALLOWED_CHAT_MODELS_BY_PROVIDER` (openai, anthropic, google) picks a real model id from that list instead. Naively stripping the tier's vendor prefix for every provider is what once sent `claude-haiku-4-5` to OpenAI and broke every message.- **A short model list in the key's picker is the vendor's catalog, not a truncation.** `listModels` returns whatever the provider's own `/models` endpoint gives and filters nothing except the key's `modelScope` allow-list. Anthropic ships roughly a dozen models, OpenAI ~80 (mostly embeddings/tts/whisper), while OpenRouter is an aggregator and returns 400+ from every vendor it proxies — so the counts differ by an order of magnitude by design. Anthropic pages at 20 by default, which is why its request pins `?limit=1000`. If the list shows exactly three Claude models, that is the *chat* dropdown reading the curated `ANTHROPIC_CHAT_MODELS`, a different surface from the admin picker.diff --git a/packages/server/api/src/app/ai/ai-provider-service.ts b/packages/server/api/src/app/ai/ai-provider-service.tsindex b00dcc05180..756cc82cd1b 100644--- a/packages/server/api/src/app/ai/ai-provider-service.ts+++ b/packages/server/api/src/app/ai/ai-provider-service.ts@@ -11,6 +11,7 @@ import { distributedLock, distributedStore } from '../database/redis-connectionsimport { openRouterApi } from '../ee/platform/platform-plan/openrouter/openrouter-api'import { flagService } from '../flags/flag.service'import { EncryptedObject, encryptUtils } from '../helper/encryption'+import { sleep } from '../helper/sleep'import { platformConfigurationService } from '../platform/platform-configuration.service'import { platformService } from '../platform/platform.service'import { aiKeyScope, ProviderScope } from './ai-key-scope'@@ -27,7 +28,10 @@ const modelsCache = new Map<string, AIProviderModel[]>()const MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD = 500const MANAGED_OPENROUTER_KEY_LIMIT_RESET = 'monthly'-const MANAGED_OPENROUTER_KEY_LOCK_TIMEOUT_SECONDS = 30+const MANAGED_OPENROUTER_KEY_LOCK_TIMEOUT_SECONDS = 60+const MANAGED_OPENROUTER_KEY_LIVE_WAIT_MS = 30_000+const MANAGED_OPENROUTER_KEY_LIVE_POLL_MS = 1_000+const MANAGED_OPENROUTER_KEY_LIVE_PASSES_IN_A_ROW = 3// A passing check must not lock out the next real failure, so the claim is a floor between// checks rather than a window that swallows them. A confirmed failure needs no floor: the row@@ -716,6 +720,7 @@ async function enrichWithKeysIfNeeded({ providerId, platformId, log }: { providelimit_reset: MANAGED_OPENROUTER_KEY_LIMIT_RESET,})const minted: ActivePiecesProviderAuthConfig = { apiKey: key, apiKeyHash: data.hash }+ await waitUntilKeyIsLive({ minted, platformId, log })const { data: stored, error } = await tryCatch(() => storeKeyIfAuthUnchanged({ providerId, platformId, observedAuth: current.auth, auth: minted }))if (isNil(stored) || managedKeyHash(stored) !== minted.apiKeyHash) {await revokeManagedKey({ hash: minted.apiKeyHash, platformId, log })@@ -731,6 +736,19 @@ async function enrichWithKeysIfNeeded({ providerId, platformId, log }: { provide})}+async function waitUntilKeyIsLive({ minted, platformId, log }: { minted: ActivePiecesProviderAuthConfig, platformId: PlatformId, log: FastifyBaseLogger }): Promise<void> {+ const deadline = Date.now() + MANAGED_OPENROUTER_KEY_LIVE_WAIT_MS+ let passesInARow = 0+ while (Date.now() < deadline) {+ passesInARow = await openRouterApi.isKeyLive({ key: minted.apiKey }) ? passesInARow + 1 : 0+ if (passesInARow === MANAGED_OPENROUTER_KEY_LIVE_PASSES_IN_A_ROW) {+ return+ }+ await sleep(MANAGED_OPENROUTER_KEY_LIVE_POLL_MS)+ }+ log.warn({ platform: { id: platformId }, openRouterKey: { hash: minted.apiKeyHash } }, '[aiProviderService#waitUntilKeyIsLive] New managed OpenRouter key is still not live, using it anyway')+}+async function storeKeyIfAuthUnchanged({ providerId, platformId, observedAuth, auth }: { providerId: string, platformId: PlatformId, observedAuth: EncryptedObject, auth: ActivePiecesProviderAuthConfig }): Promise<AIProviderAuthConfig> {await aiProviderRepo().createQueryBuilder().update()diff --git a/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts b/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.tsindex 0fc49bc075a..99772c45294 100644--- a/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts+++ b/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts@@ -6,6 +6,8 @@ import { AppSystemProp } from '../../../../helper/system/system-props'const OPENROUTER_BASE_URL = 'https://openrouter.ai/api/v1'const REQUEST_TIMEOUT_MS = 15000+const KEY_LIVENESS_TIMEOUT_MS = 10000+const KEY_LIVENESS_MODEL = 'openai/gpt-4o-mini'export const openRouterApi = {async createKey(request: CreateKeyRequest): Promise<CreateKeyResponse> {@@ -24,6 +26,19 @@ export const openRouterApi = {await openRouterRequest<unknown>({ method: 'DELETE', path: `/keys/${hash}` })},+ async isKeyLive({ key }: IsKeyLiveRequest): Promise<boolean> {+ const { data: response } = await tryCatch(() => safeHttp.axios.post(`${OPENROUTER_BASE_URL}/chat/completions`, {+ model: KEY_LIVENESS_MODEL,+ max_tokens: 1,+ messages: [{ role: 'user', content: 'hi' }],+ }, {+ timeout: KEY_LIVENESS_TIMEOUT_MS,+ headers: { Authorization: `Bearer ${key}` },+ validateStatus: () => true,+ }))+ return !isNil(response) && response.status !== 401+ },+async listKeys({ offset, include_disabled }: ListKeysRequest): Promise<ListKeysResponse> {return openRouterRequest<ListKeysResponse>({method: 'GET',@@ -100,6 +115,10 @@ type GetKeyResponse = {data: OpenRouterApikey}+type IsKeyLiveRequest = {+ key: string+}+type DeleteKeyRequest = {hash: string}diff --git a/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts b/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.tsindex 4baef1ec005..12987e381f5 100644--- a/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts+++ b/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts@@ -1,4 +1,5 @@import { AIProviderName, apId, isNil } from '@activepieces/core-utils'+import { safeHttp } from '@activepieces/server-utils'import { FastifyInstance } from 'fastify'import { afterAll, beforeAll, beforeEach, describe, expect, it, MockInstance, vi } from 'vitest'import { aiProviderService } from '../../../../src/app/ai/ai-provider-service'@@ -8,9 +9,12 @@ import { db } from '../../../helpers/db'import { createTestContext, TestContext } from '../../../helpers/test-context'import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'+vi.mock('../../../../src/app/helper/sleep', () => ({ sleep: async () => undefined }))+let app: FastifyInstancelet ctx: TestContextlet deleteKey: MockInstance<typeof openRouterApi.deleteKey>+let isKeyLive: MockInstance<typeof openRouterApi.isKeyLive>beforeAll(async () => {app = await setupTestEnvironment({ fresh: true })@@ -23,6 +27,7 @@ afterAll(async () => {beforeEach(async () => {vi.restoreAllMocks()deleteKey = vi.spyOn(openRouterApi, 'deleteKey').mockResolvedValue(undefined)+ isKeyLive = vi.spyOn(openRouterApi, 'isKeyLive').mockResolvedValue(true)ctx = await createTestContext(app)})@@ -158,11 +163,69 @@ describe('managed AI provider key provisioning', () => {])expect(createKey).toHaveBeenCalledTimes(1)+ expect(isKeyLive).toHaveBeenCalledTimes(3)expect(deleteKey).not.toHaveBeenCalled()expect(third.apiKey).toBe(provisioned.apiKey)expect(fourth.apiKey).toBe(provisioned.apiKey)expect(await readPersistedApiKey(ctx.platform.id)).toBe(provisioned.apiKey)})++ it('waits until a freshly minted key passes three checks in a row before handing it out', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())+ isKeyLive.mockResolvedValueOnce(false).mockResolvedValueOnce(true).mockResolvedValueOnce(false)++ const provisioned = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)++ expect(provisioned.apiKey).toBe('sk-or-1')+ expect(isKeyLive).toHaveBeenCalledTimes(6)+ expect(isKeyLive).toHaveBeenLastCalledWith({ key: 'sk-or-1' })+ expect(deleteKey).not.toHaveBeenCalled()+ })++ it('keeps a concurrent read waiting until the new key is live', async () => {+ const createKey = vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())+ const { promise: goesLive, resolve: markLive } = Promise.withResolvers<boolean>()+ const { promise: probing, resolve: markProbing } = Promise.withResolvers<void>()+ isKeyLive.mockImplementationOnce(async () => {+ markProbing()+ return goesLive+ })++ const first = aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)+ await probing+ const second = aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)+ await new Promise((resolve) => setTimeout(resolve, 300))++ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()+ markLive(true)+ const [firstKey, secondKey] = await Promise.all([first, second])++ expect(createKey).toHaveBeenCalledTimes(1)+ expect(firstKey.apiKey).toBe('sk-or-1')+ expect(secondKey.apiKey).toBe('sk-or-1')+ expect(deleteKey).not.toHaveBeenCalled()+ })++ it('keeps checking until the wait runs out, then hands out the minted key', async () => {+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())+ vi.useFakeTimers({ toFake: ['Date'] })+ isKeyLive.mockImplementation(async () => {+ vi.setSystemTime(Date.now() + 5_000)+ return false+ })++ try {+ const provisioned = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)++ expect(provisioned.apiKey).toBe('sk-or-1')+ expect(isKeyLive).toHaveBeenCalledTimes(6)+ expect(deleteKey).not.toHaveBeenCalled()+ expect(await readPersistedApiKey(ctx.platform.id)).toBe('sk-or-1')+ }+ finally {+ vi.useRealTimers()+ }+ })})describe('managed AI provider deletion', () => {@@ -258,3 +321,19 @@ describe('managed AI provider deletion', () => {expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()})})++describe('managed OpenRouter key liveness check', () => {+ it('treats only a 401 or no answer as not live', async () => {+ isKeyLive.mockRestore()+ const post = vi.spyOn(safeHttp.axios, 'post')++ post.mockResolvedValueOnce({ status: 401 })+ expect(await openRouterApi.isKeyLive({ key: 'sk-or-1' })).toBe(false)+ post.mockRejectedValueOnce(new Error('timeout'))+ expect(await openRouterApi.isKeyLive({ key: 'sk-or-1' })).toBe(false)+ post.mockResolvedValueOnce({ status: 200 })+ expect(await openRouterApi.isKeyLive({ key: 'sk-or-1' })).toBe(true)+ post.mockResolvedValueOnce({ status: 404 })+ expect(await openRouterApi.isKeyLive({ key: 'sk-or-1' })).toBe(true)+ })+})