Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(ai): a new platform's first AI call no longer fails with "User not found" (#16935)

Hazem Adel91e9fa77bb88
diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.md
index b271f90b0ad..e54ea8330c6 100644
--- a/brain/knowledge/ai-intelligence/ai-providers.md
+++ b/brain/knowledge/ai-intelligence/ai-providers.md
@@ -210,6 +210,7 @@ after a file without chat keys it seeds the chat list from the live flow list.
separate limit is needed wherever it is passed along. An older version of the action could write a
`{ messages: [] }` wrapper instead of a bare array, so any reader has to accept both shapes.
- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `chatHelpers.resolveChatProvider` → `getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. The mint runs under a per-platform `distributedLock` and re-reads the row inside it, and the write lands only if `auth` still matches the ciphertext it read (a `jsonb` compare; the row is then read back, since `affected` is unreliable on PGlite). A key that loses that write, or whose write throws, is revoked with `openRouterApi.deleteKey`. Two leaks stay open until a reconcile sweep over `listKeys` exists: a crash between `createKey` and the write, and a `createKey` that times out after OpenRouter has made the key. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getChatConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.
+- **A freshly minted managed OpenRouter key can answer 401 "User not found." for up to about 20s, so `enrichWithKeysIfNeeded` only saves a new key once it answers.** In prod that failed the first chat turns of about 15% of new platforms (AI-158). `GET /api/v1/key` is not a good enough check: in a 150-key run (2026-10-27) 7 keys passed it 3 times in a row and then still failed 1 to 3 completions. So `openRouterApi.isKeyLive` sends a 1-token completion and counts only a 401 or no answer as not live (so a retired or down probe model can't stall every new platform for 30s), and the wait needs 3 passes in a row 1s apart. With that, 1 of 150 keys failed one call afterwards, which chat's single retry absorbs (median wait 5.4s, p95 22s). How long a key takes varies with OpenRouter's load, and hammering it with 5+ new keys at once makes keys flap, so measure at most 3 at a time. The wait runs for up to 30s inside the mint lock, *before* the save, so a concurrent call still sees an empty row and queues on the lock instead of reading a key that isn't live (lock timeout 60s, so waiters outlast a 15s mint plus the wait plus one 10s check). On timeout it logs `waitUntilKeyIsLive` and saves the key anyway.
- **`openrouter-api.ts` has no retry, and every failure is a plain `Error`.** Calls go through `safeHttp.axios` with a 15 s timeout, and `openRouterRequest` rethrows any failure as `[OpenRouter] <METHOD> <path> error: <status> <body>`, so a caller cannot tell a 429 from a 500 without parsing the message.
- **Chat model tiers are Activepieces-only.** `ACTIVEPIECES_CHAT_TIERS` (`fast`/`smart`/`premium`, shown as Fast/Expert/Heavy) hold OpenRouter-shaped Anthropic ids, so they only mean anything for the ACTIVEPIECES and OPENROUTER chat providers. Any provider that declares `ALLOWED_CHAT_MODELS_BY_PROVIDER` (openai, anthropic, google) picks a real model id from that list instead. Naively stripping the tier's vendor prefix for every provider is what once sent `claude-haiku-4-5` to OpenAI and broke every message.
- **A short model list in the key's picker is the vendor's catalog, not a truncation.** `listModels` returns whatever the provider's own `/models` endpoint gives and filters nothing except the key's `modelScope` allow-list. Anthropic ships roughly a dozen models, OpenAI ~80 (mostly embeddings/tts/whisper), while OpenRouter is an aggregator and returns 400+ from every vendor it proxies — so the counts differ by an order of magnitude by design. Anthropic pages at 20 by default, which is why its request pins `?limit=1000`. If the list shows exactly three Claude models, that is the *chat* dropdown reading the curated `ANTHROPIC_CHAT_MODELS`, a different surface from the admin picker.
diff --git a/packages/server/api/src/app/ai/ai-provider-service.ts b/packages/server/api/src/app/ai/ai-provider-service.ts
index b00dcc05180..756cc82cd1b 100644
--- a/packages/server/api/src/app/ai/ai-provider-service.ts
+++ b/packages/server/api/src/app/ai/ai-provider-service.ts
@@ -11,6 +11,7 @@ import { distributedLock, distributedStore } from '../database/redis-connections
import { openRouterApi } from '../ee/platform/platform-plan/openrouter/openrouter-api'
import { flagService } from '../flags/flag.service'
import { EncryptedObject, encryptUtils } from '../helper/encryption'
+import { sleep } from '../helper/sleep'
import { platformConfigurationService } from '../platform/platform-configuration.service'
import { platformService } from '../platform/platform.service'
import { aiKeyScope, ProviderScope } from './ai-key-scope'
@@ -27,7 +28,10 @@ const modelsCache = new Map<string, AIProviderModel[]>()
const MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD = 500
const MANAGED_OPENROUTER_KEY_LIMIT_RESET = 'monthly'
-const MANAGED_OPENROUTER_KEY_LOCK_TIMEOUT_SECONDS = 30
+const MANAGED_OPENROUTER_KEY_LOCK_TIMEOUT_SECONDS = 60
+const MANAGED_OPENROUTER_KEY_LIVE_WAIT_MS = 30_000
+const MANAGED_OPENROUTER_KEY_LIVE_POLL_MS = 1_000
+const MANAGED_OPENROUTER_KEY_LIVE_PASSES_IN_A_ROW = 3
// A passing check must not lock out the next real failure, so the claim is a floor between
// checks rather than a window that swallows them. A confirmed failure needs no floor: the row
@@ -716,6 +720,7 @@ async function enrichWithKeysIfNeeded({ providerId, platformId, log }: { provide
limit_reset: MANAGED_OPENROUTER_KEY_LIMIT_RESET,
})
const minted: ActivePiecesProviderAuthConfig = { apiKey: key, apiKeyHash: data.hash }
+ await waitUntilKeyIsLive({ minted, platformId, log })
const { data: stored, error } = await tryCatch(() => storeKeyIfAuthUnchanged({ providerId, platformId, observedAuth: current.auth, auth: minted }))
if (isNil(stored) || managedKeyHash(stored) !== minted.apiKeyHash) {
await revokeManagedKey({ hash: minted.apiKeyHash, platformId, log })
@@ -731,6 +736,19 @@ async function enrichWithKeysIfNeeded({ providerId, platformId, log }: { provide
})
}
+async function waitUntilKeyIsLive({ minted, platformId, log }: { minted: ActivePiecesProviderAuthConfig, platformId: PlatformId, log: FastifyBaseLogger }): Promise<void> {
+ const deadline = Date.now() + MANAGED_OPENROUTER_KEY_LIVE_WAIT_MS
+ let passesInARow = 0
+ while (Date.now() < deadline) {
+ passesInARow = await openRouterApi.isKeyLive({ key: minted.apiKey }) ? passesInARow + 1 : 0
+ if (passesInARow === MANAGED_OPENROUTER_KEY_LIVE_PASSES_IN_A_ROW) {
+ return
+ }
+ await sleep(MANAGED_OPENROUTER_KEY_LIVE_POLL_MS)
+ }
+ log.warn({ platform: { id: platformId }, openRouterKey: { hash: minted.apiKeyHash } }, '[aiProviderService#waitUntilKeyIsLive] New managed OpenRouter key is still not live, using it anyway')
+}
+
async function storeKeyIfAuthUnchanged({ providerId, platformId, observedAuth, auth }: { providerId: string, platformId: PlatformId, observedAuth: EncryptedObject, auth: ActivePiecesProviderAuthConfig }): Promise<AIProviderAuthConfig> {
await aiProviderRepo().createQueryBuilder()
.update()
diff --git a/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts b/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts
index 0fc49bc075a..99772c45294 100644
--- a/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts
+++ b/packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts
@@ -6,6 +6,8 @@ import { AppSystemProp } from '../../../../helper/system/system-props'
const OPENROUTER_BASE_URL = 'https://openrouter.ai/api/v1'
const REQUEST_TIMEOUT_MS = 15000
+const KEY_LIVENESS_TIMEOUT_MS = 10000
+const KEY_LIVENESS_MODEL = 'openai/gpt-4o-mini'
export const openRouterApi = {
async createKey(request: CreateKeyRequest): Promise<CreateKeyResponse> {
@@ -24,6 +26,19 @@ export const openRouterApi = {
await openRouterRequest<unknown>({ method: 'DELETE', path: `/keys/${hash}` })
},
+ async isKeyLive({ key }: IsKeyLiveRequest): Promise<boolean> {
+ const { data: response } = await tryCatch(() => safeHttp.axios.post(`${OPENROUTER_BASE_URL}/chat/completions`, {
+ model: KEY_LIVENESS_MODEL,
+ max_tokens: 1,
+ messages: [{ role: 'user', content: 'hi' }],
+ }, {
+ timeout: KEY_LIVENESS_TIMEOUT_MS,
+ headers: { Authorization: `Bearer ${key}` },
+ validateStatus: () => true,
+ }))
+ return !isNil(response) && response.status !== 401
+ },
+
async listKeys({ offset, include_disabled }: ListKeysRequest): Promise<ListKeysResponse> {
return openRouterRequest<ListKeysResponse>({
method: 'GET',
@@ -100,6 +115,10 @@ type GetKeyResponse = {
data: OpenRouterApikey
}
+type IsKeyLiveRequest = {
+ key: string
+}
+
type DeleteKeyRequest = {
hash: string
}
diff --git a/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts b/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts
index 4baef1ec005..12987e381f5 100644
--- a/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts
+++ b/packages/server/api/test/integration/ce/ai-provider/managed-provider-key-race.test.ts
@@ -1,4 +1,5 @@
import { AIProviderName, apId, isNil } from '@activepieces/core-utils'
+import { safeHttp } from '@activepieces/server-utils'
import { FastifyInstance } from 'fastify'
import { afterAll, beforeAll, beforeEach, describe, expect, it, MockInstance, vi } from 'vitest'
import { aiProviderService } from '../../../../src/app/ai/ai-provider-service'
@@ -8,9 +9,12 @@ import { db } from '../../../helpers/db'
import { createTestContext, TestContext } from '../../../helpers/test-context'
import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+vi.mock('../../../../src/app/helper/sleep', () => ({ sleep: async () => undefined }))
+
let app: FastifyInstance
let ctx: TestContext
let deleteKey: MockInstance<typeof openRouterApi.deleteKey>
+let isKeyLive: MockInstance<typeof openRouterApi.isKeyLive>
beforeAll(async () => {
app = await setupTestEnvironment({ fresh: true })
@@ -23,6 +27,7 @@ afterAll(async () => {
beforeEach(async () => {
vi.restoreAllMocks()
deleteKey = vi.spyOn(openRouterApi, 'deleteKey').mockResolvedValue(undefined)
+ isKeyLive = vi.spyOn(openRouterApi, 'isKeyLive').mockResolvedValue(true)
ctx = await createTestContext(app)
})
@@ -158,11 +163,69 @@ describe('managed AI provider key provisioning', () => {
])
expect(createKey).toHaveBeenCalledTimes(1)
+ expect(isKeyLive).toHaveBeenCalledTimes(3)
expect(deleteKey).not.toHaveBeenCalled()
expect(third.apiKey).toBe(provisioned.apiKey)
expect(fourth.apiKey).toBe(provisioned.apiKey)
expect(await readPersistedApiKey(ctx.platform.id)).toBe(provisioned.apiKey)
})
+
+ it('waits until a freshly minted key passes three checks in a row before handing it out', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+ isKeyLive.mockResolvedValueOnce(false).mockResolvedValueOnce(true).mockResolvedValueOnce(false)
+
+ const provisioned = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+
+ expect(provisioned.apiKey).toBe('sk-or-1')
+ expect(isKeyLive).toHaveBeenCalledTimes(6)
+ expect(isKeyLive).toHaveBeenLastCalledWith({ key: 'sk-or-1' })
+ expect(deleteKey).not.toHaveBeenCalled()
+ })
+
+ it('keeps a concurrent read waiting until the new key is live', async () => {
+ const createKey = vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+ const { promise: goesLive, resolve: markLive } = Promise.withResolvers<boolean>()
+ const { promise: probing, resolve: markProbing } = Promise.withResolvers<void>()
+ isKeyLive.mockImplementationOnce(async () => {
+ markProbing()
+ return goesLive
+ })
+
+ const first = aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+ await probing
+ const second = aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+ await new Promise((resolve) => setTimeout(resolve, 300))
+
+ expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
+ markLive(true)
+ const [firstKey, secondKey] = await Promise.all([first, second])
+
+ expect(createKey).toHaveBeenCalledTimes(1)
+ expect(firstKey.apiKey).toBe('sk-or-1')
+ expect(secondKey.apiKey).toBe('sk-or-1')
+ expect(deleteKey).not.toHaveBeenCalled()
+ })
+
+ it('keeps checking until the wait runs out, then hands out the minted key', async () => {
+ vi.spyOn(openRouterApi, 'createKey').mockImplementation(slowMintingKeys())
+ vi.useFakeTimers({ toFake: ['Date'] })
+ isKeyLive.mockImplementation(async () => {
+ vi.setSystemTime(Date.now() + 5_000)
+ return false
+ })
+
+ try {
+ const provisioned = await aiProviderService(app.log).getOrCreateActivePiecesProviderAuthConfig(ctx.platform.id)
+
+ expect(provisioned.apiKey).toBe('sk-or-1')
+ expect(isKeyLive).toHaveBeenCalledTimes(6)
+ expect(deleteKey).not.toHaveBeenCalled()
+ expect(await readPersistedApiKey(ctx.platform.id)).toBe('sk-or-1')
+ }
+ finally {
+ vi.useRealTimers()
+ }
+ })
})
describe('managed AI provider deletion', () => {
@@ -258,3 +321,19 @@ describe('managed AI provider deletion', () => {
expect(await readPersistedApiKey(ctx.platform.id)).toBeUndefined()
})
})
+
+describe('managed OpenRouter key liveness check', () => {
+ it('treats only a 401 or no answer as not live', async () => {
+ isKeyLive.mockRestore()
+ const post = vi.spyOn(safeHttp.axios, 'post')
+
+ post.mockResolvedValueOnce({ status: 401 })
+ expect(await openRouterApi.isKeyLive({ key: 'sk-or-1' })).toBe(false)
+ post.mockRejectedValueOnce(new Error('timeout'))
+ expect(await openRouterApi.isKeyLive({ key: 'sk-or-1' })).toBe(false)
+ post.mockResolvedValueOnce({ status: 200 })
+ expect(await openRouterApi.isKeyLive({ key: 'sk-or-1' })).toBe(true)
+ post.mockResolvedValueOnce({ status: 404 })
+ expect(await openRouterApi.isKeyLive({ key: 'sk-or-1' })).toBe(true)
+ })
+})