AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
docs(brain): tiers change in the console, and two Claude 5.5 gotchas (#16463)
Hazem Adelc3d66a7f7601
diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.mdindex e92a499ef57..f41084309a1 100644--- a/brain/knowledge/ai-intelligence/ai-providers.md+++ b/brain/knowledge/ai-intelligence/ai-providers.md@@ -75,6 +75,12 @@ after a file without chat keys it seeds the chat list from the live flow list.**Specific model** — a model a builder picks directly from one key (`{provider, configId, model}`), as opposed to a tier. The admin switch `platform_configuration.aiSpecificModelsVisible` (Tiers tab, "Visible to builders") hides them from builders. _Avoid_: "Advanced" (an early mockup word for the same section)### Gotchas+- **The replay eval can't judge a model swap where the new model takes a different tool path.** Two separate things cause it:+ - **Replay divergence.** Every fixture gets the same tool set from `buildEvalToolSet` (local, display, cross-project and thinking tools). The recording only fixes which results come back and in what order. Calling a registered tool out of the recorded order returns `__evalDivergence`.+ - **Tools the eval doesn't have.** The flow and table tools (`ap_duplicate_flow`, `ap_build_flow`, `ap_find_records`) aren't in that set at all, so `ap_get_tool_schema` answers "There is no tool named". The tool does exist in production.++ Either one makes a candidate give up for reasons that aren't the model's, so `--candidate-model` scores read low. On 13 Oct, Sonnet 5.5 scored 9/26 against 14/26, and 4 of its 6 regressions were one of these two. Use `--baseline-model`/`--candidate-model` for cost and for judge-only cases. Check tool-heavy behaviour with `agent-evals:live` before changing a tier.+- **Change which model a tier runs in the AI Tiers console, not in code.** A model a tier points at runs on credits without being in `ANTHROPIC_OPENROUTER_CHAT_MODELS` (`assertManagedModelAllowed` allows every tier `modelId`). Keep Claude Sonnet 5.5 and Opus 5.5 out of that list: everything in it counts as reasoning-optional (`canDisableReasoning`), and their OpenRouter endpoints reject `reasoning: {enabled: false}` with "Reasoning is mandatory for this endpoint", which would fail the thinking-off first step of every turn. Unlisted, they get `effort: minimal`, which works. Haiku 5.5 accepts `enabled: false` (checked 13 Oct).- **The admin Tiers page has no sidebar or search entry until the feature launches.** The route `/platform/ai/tiers` works, but the platform sidebar and the global search `STATIC_PAGES` deliberately leave it out, so only someone with the URL finds it. To launch, add `{ to: '/platform/ai/tiers', label: t('Tiers') }` to the AI subItems in `sidebar/platform/index.tsx` and add a `page-platform-ai-tiers` entry to `static-pages.ts`.- **New shared AI-provider schemas go inside `packages/core/shared/src/lib/management/ai-providers/index.ts`, not in a sibling file that imports from it.** `index.ts` re-exports its siblings, so a sibling that imports `AIProviderModel` back from `./index` gets `undefined` when the module loads, and every importer crashes with `Cannot read properties of undefined (reading 'optional')`. `platform-model-tier.ts` works only because it imports from `core-utils`.- **The admin Tiers tab applies a key's `modelScope` itself.** `GET /v1/ai-providers/configs/:id/models` returns the unfiltered list on purpose (admins pick the allow-list from it), so the tier pickers and the Specific models list narrow it client-side in `modelMeta.scopedTextModels` (`packages/web/src/features/agents/ai-model/`). The server still refuses an out-of-scope entry on save (`assertEntriesValid`), so the client filter is convenience, not the guard.