AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
docs(brain): how to ship AI work behind a flag, and the first open-cohort activation baseline (#16667)
Hazem Adel6d93dbcc7472
diff --git a/brain/knowledge/ai-intelligence/index.md b/brain/knowledge/ai-intelligence/index.mdindex e1882a7ae16..197d43e1b00 100644--- a/brain/knowledge/ai-intelligence/index.md+++ b/brain/knowledge/ai-intelligence/index.md@@ -39,6 +39,7 @@ The per-project endpoint that exposes Activepieces tools to an external AI assis- **Model Picker** — the builder's tier / credits / key picker and its options endpoint- **MCP Server** — the per-project endpoint, tool exposure, visibility rules- **AI & MCP** — how the AI and MCP surfaces fit together+- **Shipping Behind a Flag** — merging unfinished AI work, turned on for a test group only## Relateddiff --git a/brain/knowledge/ai-intelligence/shipping-behind-a-flag.md b/brain/knowledge/ai-intelligence/shipping-behind-a-flag.mdnew file mode 100644index 00000000000..8031af1fab3--- /dev/null+++ b/brain/knowledge/ai-intelligence/shipping-behind-a-flag.md@@ -0,0 +1,112 @@+---+icon: 🚦+---++# Shipping Behind a Flag++How to merge unfinished AI work safely, turned on only for a test group. Without this, features wait in big PRs until they are finished, which is how the cost PRs and the helper-agent PRs ended up colliding on 11 files and needing a fixed merge order.++Two shapes, and they answer different questions. Pick by asking **who decides** whether the feature is on.++| We decide, while we are still learning | The customer's plan decides |+| --- | --- |+| A rollout module: a hash bucket over a stable id | A `PlatformPlan` boolean |+| Removed when the feature goes to everyone | Lives as long as the feature does |+| Task agents, split screen, a new prompt | `chatEnabled`, `agentsEnabled`, `customAppearanceEnabled` |++**Never a new env var or system property.** Features are decided in code. An env var means a self-hoster has to find it and set it, and the feature looks enabled in the UI while being silently broken for everyone who did not.++## A test group, while we are still learning++Task agents shipped this way on 15 Oct 2026 and the gate came out on 18 Oct.++**None of it is on `main` any more**, which is the point: a rollout flag is meant to be deleted. #16574 took it out, and `taskAgentsRollout`, `isOnForConversation` and the `tasksEnabled` log field are all gone. Do not go looking for `task-agents-rollout.ts` in the tree; everything you need is below.++The whole module was this, named after the feature and owning the whole decision:++```ts+import { createHash } from 'node:crypto'+import { ApEdition, ApEnvironment } from '@activepieces/shared'++function isOnForConversation({ conversationId }: { conversationId: string }): boolean {+ if (system.get(AppSystemProp.ENVIRONMENT) === ApEnvironment.DEVELOPMENT) {+ return true+ }+ if (system.getEdition() !== ApEdition.CLOUD) {+ return false+ }+ return bucketOf({ conversationId }) < ROLLOUT_PERCENT+}++function bucketOf({ conversationId }: { conversationId: string }): number {+ return createHash('sha256').update(conversationId).digest().readUInt32BE(0) % 100+}++const ROLLOUT_PERCENT = 10++export const taskAgentsRollout = {+ isOnForConversation,+ ROLLOUT_PERCENT,+}+```++Four properties make it work, and each one was a test:++- **Deterministic.** Hash a stable id, so a conversation gets the same answer on every turn. A user must never see the feature appear and vanish mid-chat.+- **Cloud only.** Self-hosters never get a half-finished feature, and it needs no setup from them.+- **Always on in local dev**, so nobody has to opt in to test their own work.+- **One decision point.** Decide it once, server side, and let the result flow down. Task agents gated `tasksAvailable` in `agent-config-rpc.ts`; when it was false the server sent no `taskSystemPrompt`, so the worker built no task tools. Both of those still exist on `main`, ungated. Do not re-check a flag in three places.++Then **tag the turn log with the group**, as task agents did with a `tasksEnabled` field. Without it you cannot split cost or quality by group, which is the only reason to run a test group at all. That field went out with the gate, so add your own; there is no shared one to reuse.++Turning it off is setting `ROLLOUT_PERCENT` to 0 and deploying. There is no runtime switch, by design.++The four tests that pinned it, worth copying verbatim for your own flag:++- turns it on for about the rollout share of cloud conversations (assert within a few points of `ROLLOUT_PERCENT`, not an exact count)+- keeps the same answer for the same conversation on every turn+- keeps it off on self-hosted editions+- turns it on for every conversation in local development++## A feature the plan decides++Four places, and missing the last one is a compile error rather than a silent bug. `chatEnabled` and `agentsEnabled` are the ones to copy.++1. A boolean on `PlatformPlan`, in `platform.model.ts`.+2. An id on `FeatureFlagId`, in the same file.+3. Its default per plan, in the billing defaults.+4. **A line in `toPlatformPlanFlags`**, in `autumn-utils.ts`, mapping the `FeatureFlagId` onto the boolean. It returns the whole `PlatformPlanFlags`, so a flag added to the type and not mapped here fails to typecheck. `autumn-entitlement-flags.test.ts` covers this mapping.++Then gate on it:++- **Backend:** `platformMustHaveFeatureEnabled((p) => p.plan.myFlag)` on the EE module. Returns 402.+- **Frontend:** `LockedFeatureGuard`, and `enabled: platform.plan.myFlag` on queries.++This is a permanent part of the product, not a rollout. Do not use it to hide unfinished work: a plan flag says "your plan does not include this", which is a lie when the real reason is that it is half built.++## Removing it++A rollout flag is finished work that nobody has deleted yet, so delete it in one PR of its own, the way #16574 did:++1. Drop the gate and call the feature directly.+2. Delete the rollout module and its tests.+3. Drop the log field once no dashboard reads it.+4. If it had a table, the migration that drops it sets `breaking = true` and leaves `down()` empty.+5. Remove the line about it from this area's page.++The cohort-table approach (a `chat_rollout_user` row per user) was tried for the 200-user chat beta and dropped on 18 Oct. Do not bring it back: it needs a migration, an entity, an admin surface and a cleanup migration, where the hash bucket needs one file and one delete.++## Gotchas++- **Hash the thing the user experiences, not the user.** Task agents bucket by conversation id, so one user can see the feature in a new chat without their older chats changing shape underneath them. Bucketing by user id is right when the feature is a durable setting rather than a per-run behaviour.+- **A rollout share is not a sample size.** 10% of conversations is a traffic share. Check how many conversations that actually is per day before trusting any comparison between the groups.+- **`ApFlagId` is a different thing.** It is the frontend's flag list served by the flags endpoint, not a feature gate for unfinished work.++## Key files++- #16574 — the worked example being removed. There is no rollout module in the tree to read, by design; it is inlined above+- `packages/server/api/src/app/ee/agent/rpc` — `agent-config-rpc.ts`, the server-side decision point a chat flag hangs off+- `packages/server/api/src/app/ee/platform/platform-plan/billing-providers` — `toPlatformPlanFlags` in `autumn-utils.ts`, the step a new plan flag is easiest to forget+- `packages/core/shared/src/lib/management/platform` — `PlatformPlan` and `FeatureFlagId` in `platform.model.ts`+- `packages/core/shared/src/lib/ee/billing` — per-plan defaults for each flag+- `packages/server/api/src/app/ee/authentication` — `platformMustHaveFeatureEnabled` in `ee-authorization.ts`diff --git a/brain/knowledge/flows-execution/chat-activation-metric.md b/brain/knowledge/flows-execution/chat-activation-metric.mdindex ed57b0d3e57..e5d782caabb 100644--- a/brain/knowledge/flows-execution/chat-activation-metric.md+++ b/brain/knowledge/flows-execution/chat-activation-metric.md@@ -17,11 +17,19 @@ S1 first chat → S2 tried to build (`ap_set_build_plan`, `ap_build_flow` or `ap## Running it1. Open a Craftboxes box (`box new -n chat-activation`). The replica is reached through the box's `craftbox` MCP server (`postgres_query`, one read-only statement, 1000 rows max), not through `psql`.-2. The replica has a 15s statement timeout, so the single query below times out on prod. Run it in stages and join them locally:- 1. the cohort and S2 counts;- 2. the chat-built flow ids;- 3. flow state and run counts, in batches of about 50 flows passed back as `VALUES`;- 4. agents the cohort owns that were created in the window, with their published state and successful runs.+2. The replica has a 15s statement timeout, so the single query below times out on prod. **Split the cohort by first-chat week and add the halves.** Every count is per user and a user has exactly one `t0`, so disjoint `t0` ranges never double count. The `cohort` CTE takes only a lower bound, so add an upper one; setting `since` twice would overlap, not split. What ran on 2026-10-20:++ ```sql+ -- first half+ where t0 >= '2026-10-01'::timestamptz and t0 < '2026-10-07'::timestamptz+ -- second half+ where t0 >= '2026-10-07'::timestamptz and t0 < now() - interval '7 days'+ ```++ Only the **flow** half needs this. The cohort and S2 counts, and the whole agent half, still run in one pass over the full cohort. Marking `chat_parts` as `materialized` helps but is not enough on its own.++ If splitting by week ever stops being enough, fall back to pulling the chat-built flow ids out and passing them back as `VALUES` in batches of about 50. That costs several round trips and hits the 1000-row cap, which is why the split is the better route.+The single query is still the definition. Save the SQL block below as `chat-activation.sql` and it runs as-is on a local or dev database with `psql -v since=2026-06-01 -f chat-activation.sql`.3. Record the row, with the date, under Baselines.@@ -125,9 +133,16 @@ from per_user;| Date | Cohort (first chat before) | S1 | S2 | S3 | S4 | S5 | S6 | Activation || --- | --- | --- | --- | --- | --- | --- | --- | --- || 2026-09-29 | 2026-09-22 | 850 | 280 | 229 | 91 | 44 | 25 | 2.9% |+| 2026-10-20 | 2026-10-13 | 1265 | 677 | 364 | 118 | 44 | 27–29 | 2.1–2.3% |S3 counts a flow that was built in the window and later deleted. On 2026-09-29, 68 of 381 chat-built flows had been deleted since, and agents contributed nothing: 72 exist on Cloud, 3 are published, and 2 have ever run.+**2026-10-20 is the first open-cohort run, and it was measured in parts**, so read the row with three caveats. S3, S4 and S5 are the **flow path only**: 364 users built a chat-built flow, 118 published one, 44 had a successful production run. S2 is the tool-based `tried` count alone, without the `or s3_built` arm, so it is a slight undercount. S6 is a range because the flow and agent halves ran as separate queries and the definition takes a per-user OR: 27 users activated through a flow, 2 through an agent, overlap unknown.++Of 500 chat-built flows, 32 had been deleted since, down from 18% to 6.4%. The agent path, newly significant: 390 agents created by 316 of the 1265 users, 182 published, 121 with at least one good run.++**The leak is published → first good run, and it widened.** 91 → 44 on 2026-09-29 was 48% through that step; 118 → 44 is 37%. More users publish, the same absolute number ever see a flow work. Do not read 2.1–2.3% against 2.9% as a regression: the cohorts are different populations (see Gotchas), and this is the first real baseline rather than a decline.+## Gotchas- **Read `uiMessages`, not `messages`.** `messages` is the model context and compaction rewrites it; `uiMessages` is the durable transcript, and it is the only one holding `ap_build_flow` results with the flowId.@@ -135,3 +150,5 @@ S3 counts a flow that was built in the window and later deleted. On 2026-09-29,- **The cohort anchor is the first CHAT message.**- **Cohorts before and after 1 Oct 2026 are different populations.** Until then Cloud chat was capped at 200 users plus platforms whose plan had `chatEnabled`; from 1 Oct it is open to every Cloud user. Compare cohorts by first-chat date, and don't read the 2.9% baseline as what an open cohort should match.- **Small cohorts swing.** In the capped cohorts one user moved a stage by several points, so read the counts next to the percentages.+- **The flow half stopped fitting the 15s timeout after the 1 Oct opening**, even with `chat_parts` materialized, though the cohort, S2 and agent halves still run in one pass. See Running it for the cohort split that solves it.+- **Agents stopped being a rounding error.** On 20 Oct the open cohort had created 390 agents, 182 published, 121 with at least one good run. The 29 Sep note that agents contribute nothing (72 agents, 3 published, 2 ever run) is about the capped cohort and no longer holds, so run the agent half rather than assuming it is zero.