AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
chore(ai-task): triage queue alerts with /debug-failed-run (#16449)
Hazem Adel517cd5bb5b43
diff --git a/.agents/skills/ai-task/SKILL.md b/.agents/skills/ai-task/SKILL.mdindex c0bcf10b187..3b23cfa340a 100644--- a/.agents/skills/ai-task/SKILL.md+++ b/.agents/skills/ai-task/SKILL.md@@ -39,6 +39,7 @@ Run these checks before anything else, silently when they pass. If one fails, sh> Run `box new -n data` (first time: `npm i -g @abuaboud/box` and `box login --server https://box.abuaboud.me`). It opens Claude with ClickHouse and the production replica, so run the same `/ai-task` there.> For now I'll go with what the code shows.+ - **Queue or job failures** also need the DevOps box over SSH, which `/debug-failed-run` drives. Ask for the host once.- **Rules for real data:** read only. Keep ClickHouse windows to 3 days or less. Copy only counts and ids into tickets, never customer content, emails or credentials.## Help (no arguments, "help", or "?")@@ -283,6 +284,7 @@ The cards below are examples of that voice, not forms to fill in word for word.- ClickHouse `default.otel_logs` for errors, with windows of 3 days or less;- the Postgres replica for the row in question, read only.- If the box isn't connected, say so and go on with code evidence alone.+ - **Queue or job failure** (a "Queue Alerts" ticket, a BullMQ job, a failed run id, an `EngineError@...` or `RPC [...] handler threw` signature): run `/debug-failed-run` first. Use its Step 4 to find job ids for the signature across every queue, then Steps 1 to 3 on one of them. Its verdict (product bug, config or infra) drives the decision below. The signature is often a wrapper, so read `stacktrace[0]` before naming a cause.3. **Decide one of these** and write a short comment with the evidence (files, functions, PRs, numbers). Don't post the comment without the user's go-ahead.- **Fixed already:** name the PR that fixed it, then move to Done.- **Duplicate:** link the original. Set the duplicate relation first, then the status.diff --git a/brain/knowledge/ai-intelligence/ai-agents.md b/brain/knowledge/ai-intelligence/ai-agents.mdindex d31ed35a2b9..fd063620bdb 100644--- a/brain/knowledge/ai-intelligence/ai-agents.md+++ b/brain/knowledge/ai-intelligence/ai-agents.md@@ -123,3 +123,4 @@ Paths verified 2026-07-17. An earlier version pointed at `packages/core/shared/s- **Build the agent before reading the user's data, not after.** A turn that reads the user's mail, rows or records may not then change a saved agent, and the model's instinct is to look at the real data first so the instructions fit what is actually there. It then ends the turn describing an agent it cannot create and asking the user to say "create it". The order has to be: stand it up from what the user said plus the piece catalog, add its tools, then explore and refine with `ap_update_agent` on the next turn. That rule lives in three places that must agree, since the model only ever sees whichever one it is reading: the `ap_create_agent` and `ap_add_agent_tool` descriptions in `session-tools.ts`, the agent paragraph in `chat-system-prompt.md`, and the refusal string itself, which should teach the ordering rather than just apologise. **Treat it as a nudge, not a fix.** Measured over repeated runs of the same request it holds sometimes and not others: one run created the agent first and published it with its Gmail tools in a single turn, the next reached for `ap_explore_data` to read the real labels first and was refused exactly as before. Hiding the agent-surface tools once tainted does not help either, it only turns the refusal into an absence and the agent still is not created. The deterministic fix, unimplemented, is to let a tainted turn make the change behind the approval card `waitForApproval` already provides: a human sees the diff before it lands, which keeps the property the guard exists for while removing the dead end. Do not instead make the guard advisory on the first offence, that hands an injection one free write. Agent mutations all return a `url` from `afterDraftChange`; the `<links>` block has to tell the model to use it, or a successful build ends with no way to open what was built.- **A step only persists the inputs its pinned piece version declares.** `validateProps` in `flow-version-validator-util.ts` builds `cleanInput` from `Object.keys(propsSchema.shape)`, so any key the pinned version does not know is dropped on save with no error. This bit the agent picker: `AgentLink` rendered on every Run Agent step, but `agentId` only exists from `@activepieces/[email protected]`, so on an older pinned step linking looked like it worked and was gone on reload. Anything that writes a new prop from a custom component has to gate on that prop being in the step's `selectedAction.props`, not on a plan flag. Two dev-environment corollaries: `packages/web` consumes `@activepieces/core-piece-types` from its **built dist**, so a branch that adds an enum member needs that package rebuilt before web typechecks, and the API caches dev piece metadata in memory, so a piece version bump needs the API restarted before the builder pins the new version. **You cannot fake the new version locally to test the prop**: inserting the row into `piece_metadata` by hand works for a few minutes and then `pieceSyncService` reconciles the table against the published registry and deletes a version that is not there, so the builder goes back to pinning the old one. A prop added to a piece is only exercisable end to end in the builder once that piece version is actually published, which is why the linked-step behaviour is pinned by `agent-run-link.test.ts` at the API level rather than by a browser pass.+- **A queue failure reading `RPC [getAgentConfig] handler threw: VALIDATION` hides the real reason.** The thrown `Error`'s message is only the error code, so BullMQ's `failedReason` shows the same string for every validation throw in the config path. The full text does cross the RPC in `__rpcApError`: the worker reads it with `apErrorOf(err)?.message` and writes it to the conversation's saved failure, the client error event and the flow step output, and the API logs it as `RPC handler threw`. Read one of those, not the queue. Group a backlog by `platformId` and `modelName` from `job.data` before calling it one incident: the Sep 13 to Oct 1 2026 wave of 28.7k was 3 platforms.