Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

docs(brain): how to ship AI work behind a flag, and the first open-cohort activation baseline (#16667)

Hazem Adel6d93dbcc7472
diff --git a/brain/knowledge/ai-intelligence/index.md b/brain/knowledge/ai-intelligence/index.md
index e1882a7ae16..197d43e1b00 100644
--- a/brain/knowledge/ai-intelligence/index.md
+++ b/brain/knowledge/ai-intelligence/index.md
@@ -39,6 +39,7 @@ The per-project endpoint that exposes Activepieces tools to an external AI assis
- **Model Picker** — the builder's tier / credits / key picker and its options endpoint
- **MCP Server** — the per-project endpoint, tool exposure, visibility rules
- **AI & MCP** — how the AI and MCP surfaces fit together
+- **Shipping Behind a Flag** — merging unfinished AI work, turned on for a test group only
## Related
diff --git a/brain/knowledge/ai-intelligence/shipping-behind-a-flag.md b/brain/knowledge/ai-intelligence/shipping-behind-a-flag.md
new file mode 100644
index 00000000000..8031af1fab3
--- /dev/null
+++ b/brain/knowledge/ai-intelligence/shipping-behind-a-flag.md
@@ -0,0 +1,112 @@
+---
+icon: 🚦
+---
+
+# Shipping Behind a Flag
+
+How to merge unfinished AI work safely, turned on only for a test group. Without this, features wait in big PRs until they are finished, which is how the cost PRs and the helper-agent PRs ended up colliding on 11 files and needing a fixed merge order.
+
+Two shapes, and they answer different questions. Pick by asking **who decides** whether the feature is on.
+
+| We decide, while we are still learning | The customer's plan decides |
+| --- | --- |
+| A rollout module: a hash bucket over a stable id | A `PlatformPlan` boolean |
+| Removed when the feature goes to everyone | Lives as long as the feature does |
+| Task agents, split screen, a new prompt | `chatEnabled`, `agentsEnabled`, `customAppearanceEnabled` |
+
+**Never a new env var or system property.** Features are decided in code. An env var means a self-hoster has to find it and set it, and the feature looks enabled in the UI while being silently broken for everyone who did not.
+
+## A test group, while we are still learning
+
+Task agents shipped this way on 15 Oct 2026 and the gate came out on 18 Oct.
+
+**None of it is on `main` any more**, which is the point: a rollout flag is meant to be deleted. #16574 took it out, and `taskAgentsRollout`, `isOnForConversation` and the `tasksEnabled` log field are all gone. Do not go looking for `task-agents-rollout.ts` in the tree; everything you need is below.
+
+The whole module was this, named after the feature and owning the whole decision:
+
+```ts
+import { createHash } from 'node:crypto'
+import { ApEdition, ApEnvironment } from '@activepieces/shared'
+
+function isOnForConversation({ conversationId }: { conversationId: string }): boolean {
+ if (system.get(AppSystemProp.ENVIRONMENT) === ApEnvironment.DEVELOPMENT) {
+ return true
+ }
+ if (system.getEdition() !== ApEdition.CLOUD) {
+ return false
+ }
+ return bucketOf({ conversationId }) < ROLLOUT_PERCENT
+}
+
+function bucketOf({ conversationId }: { conversationId: string }): number {
+ return createHash('sha256').update(conversationId).digest().readUInt32BE(0) % 100
+}
+
+const ROLLOUT_PERCENT = 10
+
+export const taskAgentsRollout = {
+ isOnForConversation,
+ ROLLOUT_PERCENT,
+}
+```
+
+Four properties make it work, and each one was a test:
+
+- **Deterministic.** Hash a stable id, so a conversation gets the same answer on every turn. A user must never see the feature appear and vanish mid-chat.
+- **Cloud only.** Self-hosters never get a half-finished feature, and it needs no setup from them.
+- **Always on in local dev**, so nobody has to opt in to test their own work.
+- **One decision point.** Decide it once, server side, and let the result flow down. Task agents gated `tasksAvailable` in `agent-config-rpc.ts`; when it was false the server sent no `taskSystemPrompt`, so the worker built no task tools. Both of those still exist on `main`, ungated. Do not re-check a flag in three places.
+
+Then **tag the turn log with the group**, as task agents did with a `tasksEnabled` field. Without it you cannot split cost or quality by group, which is the only reason to run a test group at all. That field went out with the gate, so add your own; there is no shared one to reuse.
+
+Turning it off is setting `ROLLOUT_PERCENT` to 0 and deploying. There is no runtime switch, by design.
+
+The four tests that pinned it, worth copying verbatim for your own flag:
+
+- turns it on for about the rollout share of cloud conversations (assert within a few points of `ROLLOUT_PERCENT`, not an exact count)
+- keeps the same answer for the same conversation on every turn
+- keeps it off on self-hosted editions
+- turns it on for every conversation in local development
+
+## A feature the plan decides
+
+Four places, and missing the last one is a compile error rather than a silent bug. `chatEnabled` and `agentsEnabled` are the ones to copy.
+
+1. A boolean on `PlatformPlan`, in `platform.model.ts`.
+2. An id on `FeatureFlagId`, in the same file.
+3. Its default per plan, in the billing defaults.
+4. **A line in `toPlatformPlanFlags`**, in `autumn-utils.ts`, mapping the `FeatureFlagId` onto the boolean. It returns the whole `PlatformPlanFlags`, so a flag added to the type and not mapped here fails to typecheck. `autumn-entitlement-flags.test.ts` covers this mapping.
+
+Then gate on it:
+
+- **Backend:** `platformMustHaveFeatureEnabled((p) => p.plan.myFlag)` on the EE module. Returns 402.
+- **Frontend:** `LockedFeatureGuard`, and `enabled: platform.plan.myFlag` on queries.
+
+This is a permanent part of the product, not a rollout. Do not use it to hide unfinished work: a plan flag says "your plan does not include this", which is a lie when the real reason is that it is half built.
+
+## Removing it
+
+A rollout flag is finished work that nobody has deleted yet, so delete it in one PR of its own, the way #16574 did:
+
+1. Drop the gate and call the feature directly.
+2. Delete the rollout module and its tests.
+3. Drop the log field once no dashboard reads it.
+4. If it had a table, the migration that drops it sets `breaking = true` and leaves `down()` empty.
+5. Remove the line about it from this area's page.
+
+The cohort-table approach (a `chat_rollout_user` row per user) was tried for the 200-user chat beta and dropped on 18 Oct. Do not bring it back: it needs a migration, an entity, an admin surface and a cleanup migration, where the hash bucket needs one file and one delete.
+
+## Gotchas
+
+- **Hash the thing the user experiences, not the user.** Task agents bucket by conversation id, so one user can see the feature in a new chat without their older chats changing shape underneath them. Bucketing by user id is right when the feature is a durable setting rather than a per-run behaviour.
+- **A rollout share is not a sample size.** 10% of conversations is a traffic share. Check how many conversations that actually is per day before trusting any comparison between the groups.
+- **`ApFlagId` is a different thing.** It is the frontend's flag list served by the flags endpoint, not a feature gate for unfinished work.
+
+## Key files
+
+- #16574 — the worked example being removed. There is no rollout module in the tree to read, by design; it is inlined above
+- `packages/server/api/src/app/ee/agent/rpc` — `agent-config-rpc.ts`, the server-side decision point a chat flag hangs off
+- `packages/server/api/src/app/ee/platform/platform-plan/billing-providers` — `toPlatformPlanFlags` in `autumn-utils.ts`, the step a new plan flag is easiest to forget
+- `packages/core/shared/src/lib/management/platform` — `PlatformPlan` and `FeatureFlagId` in `platform.model.ts`
+- `packages/core/shared/src/lib/ee/billing` — per-plan defaults for each flag
+- `packages/server/api/src/app/ee/authentication` — `platformMustHaveFeatureEnabled` in `ee-authorization.ts`
diff --git a/brain/knowledge/flows-execution/chat-activation-metric.md b/brain/knowledge/flows-execution/chat-activation-metric.md
index ed57b0d3e57..e5d782caabb 100644
--- a/brain/knowledge/flows-execution/chat-activation-metric.md
+++ b/brain/knowledge/flows-execution/chat-activation-metric.md
@@ -17,11 +17,19 @@ S1 first chat → S2 tried to build (`ap_set_build_plan`, `ap_build_flow` or `ap
## Running it
1. Open a Craftboxes box (`box new -n chat-activation`). The replica is reached through the box's `craftbox` MCP server (`postgres_query`, one read-only statement, 1000 rows max), not through `psql`.
-2. The replica has a 15s statement timeout, so the single query below times out on prod. Run it in stages and join them locally:
- 1. the cohort and S2 counts;
- 2. the chat-built flow ids;
- 3. flow state and run counts, in batches of about 50 flows passed back as `VALUES`;
- 4. agents the cohort owns that were created in the window, with their published state and successful runs.
+2. The replica has a 15s statement timeout, so the single query below times out on prod. **Split the cohort by first-chat week and add the halves.** Every count is per user and a user has exactly one `t0`, so disjoint `t0` ranges never double count. The `cohort` CTE takes only a lower bound, so add an upper one; setting `since` twice would overlap, not split. What ran on 2026-10-20:
+
+ ```sql
+ -- first half
+ where t0 >= '2026-10-01'::timestamptz and t0 < '2026-10-07'::timestamptz
+ -- second half
+ where t0 >= '2026-10-07'::timestamptz and t0 < now() - interval '7 days'
+ ```
+
+ Only the **flow** half needs this. The cohort and S2 counts, and the whole agent half, still run in one pass over the full cohort. Marking `chat_parts` as `materialized` helps but is not enough on its own.
+
+ If splitting by week ever stops being enough, fall back to pulling the chat-built flow ids out and passing them back as `VALUES` in batches of about 50. That costs several round trips and hits the 1000-row cap, which is why the split is the better route.
+
The single query is still the definition. Save the SQL block below as `chat-activation.sql` and it runs as-is on a local or dev database with `psql -v since=2026-06-01 -f chat-activation.sql`.
3. Record the row, with the date, under Baselines.
@@ -125,9 +133,16 @@ from per_user;
| Date | Cohort (first chat before) | S1 | S2 | S3 | S4 | S5 | S6 | Activation |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2026-09-29 | 2026-09-22 | 850 | 280 | 229 | 91 | 44 | 25 | 2.9% |
+| 2026-10-20 | 2026-10-13 | 1265 | 677 | 364 | 118 | 44 | 27–29 | 2.1–2.3% |
S3 counts a flow that was built in the window and later deleted. On 2026-09-29, 68 of 381 chat-built flows had been deleted since, and agents contributed nothing: 72 exist on Cloud, 3 are published, and 2 have ever run.
+**2026-10-20 is the first open-cohort run, and it was measured in parts**, so read the row with three caveats. S3, S4 and S5 are the **flow path only**: 364 users built a chat-built flow, 118 published one, 44 had a successful production run. S2 is the tool-based `tried` count alone, without the `or s3_built` arm, so it is a slight undercount. S6 is a range because the flow and agent halves ran as separate queries and the definition takes a per-user OR: 27 users activated through a flow, 2 through an agent, overlap unknown.
+
+Of 500 chat-built flows, 32 had been deleted since, down from 18% to 6.4%. The agent path, newly significant: 390 agents created by 316 of the 1265 users, 182 published, 121 with at least one good run.
+
+**The leak is published → first good run, and it widened.** 91 → 44 on 2026-09-29 was 48% through that step; 118 → 44 is 37%. More users publish, the same absolute number ever see a flow work. Do not read 2.1–2.3% against 2.9% as a regression: the cohorts are different populations (see Gotchas), and this is the first real baseline rather than a decline.
+
## Gotchas
- **Read `uiMessages`, not `messages`.** `messages` is the model context and compaction rewrites it; `uiMessages` is the durable transcript, and it is the only one holding `ap_build_flow` results with the flowId.
@@ -135,3 +150,5 @@ S3 counts a flow that was built in the window and later deleted. On 2026-09-29,
- **The cohort anchor is the first CHAT message.**
- **Cohorts before and after 1 Oct 2026 are different populations.** Until then Cloud chat was capped at 200 users plus platforms whose plan had `chatEnabled`; from 1 Oct it is open to every Cloud user. Compare cohorts by first-chat date, and don't read the 2.9% baseline as what an open cohort should match.
- **Small cohorts swing.** In the capped cohorts one user moved a stage by several points, so read the counts next to the percentages.
+- **The flow half stopped fitting the 15s timeout after the 1 Oct opening**, even with `chat_parts` materialized, though the cohort, S2 and agent halves still run in one pass. See Running it for the cohort split that solves it.
+- **Agents stopped being a rounding error.** On 20 Oct the open cohort had created 390 agents, 182 published, 121 with at least one good run. The 29 Sep note that agents contribute nothing (72 agents, 3 published, 2 ever run) is about the capped cohort and no longer holds, so run the agent half rather than assuming it is zero.