Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(flow-runs): attribute a child run to the barrier that awaits it (#14888)

Othman Abu Ajamieh84f01299174e
diff --git a/brain/knowledge/data-storage-observability/database.md b/brain/knowledge/data-storage-observability/database.md
index e06262897e9..a2641ea4a2c 100644
--- a/brain/knowledge/data-storage-observability/database.md
+++ b/brain/knowledge/data-storage-observability/database.md
@@ -262,7 +262,7 @@ public async down(q: QueryRunner): Promise<void> {
}
```
-Deleting the `background_migrations` row is a per-migration choice — tells re-upgrade to re-run the reshape. Omit if `down()` leaves the background work's effects intact.
+Deleting the `background_migrations` row is a per-migration choice — tells re-upgrade to re-run the reshape. Omit if `down()` leaves the background work's effects intact. Either way, assume the background migration can run again after `down()`: it must never drop an object the previous release needs (see Gotchas).
**Caveats:**
- Lock wait is bounded at ~4 min (`distributedLockFactory` passes `retryCount * retryDelay = (timeout / 200) * 200ms` into RedLock). If `down()` races a long in-flight backfill, the helper throws rather than waits for the full duration. Fix: scale API pods to 0 (quiesces the runner), re-run `rollback-migrations.ts`, redeploy.
@@ -291,6 +291,7 @@ A background migration that fails every retry stops the app from booting. The gu
- **Contract DDL in the same release as its backfill, skip-release upgrades, and fresh installs are all handled by boot catchup.** `runMigrationsWithCatchup` wraps TypeORM's `runMigrations`: when a blocking migration's guard throws `BackgroundMigrationNotCompleteError`, the catchup catches it, runs that one backfill synchronously, and retries. Max 10 catchup attempts (one per distinct dependency), then boot fails with the failing migration name. Normal upgrade pays nothing; skip-release pays a one-time slow boot while prior backfills catch up.
- **A brand-new hot-path index in the same release as the code that needs it can leave the app on seq scans while background CIC builds.** Only applies when there's *no* prior covering index for the query. Fixes: (a) ship the CIC in a prior release so it's built by the time the code lands (verify via `pendingCount` on `/v1/health/system`), or (b) put the CIC in the blocking track so the index is present when the app starts serving. Reshapes (replacing existing indexes) don't hit this — the old index keeps queries fast while new one builds.
- **Same-name migration in both folders would run twice.** The two history tables are independent — a rename or folder move without a seed-row migration double-executes on customers that already ran it in the old track.
+- **A rollback does not stop the paired background migration from running again.** On Cloud, `rollback.js` runs inside a live new-image container (`continuous-delivery-rollback.yml`) and the image switch comes after it. A new-image pod that boots in between re-applies every rolled-back blocking `up()`, then runs any background migration without a completed row, whether `down()` deleted that row or the migration never finished. Whatever it drops is gone under the old image. So a same-release reshape drops only an index whose queries a kept index still serves, and leaves the rest to a later release. `ReshapeFlowRunListIndexesForBarrierChildren` keeps four of the five old `flow_run` list indexes for this reason. Self-hosted rollback stops the app first.
- **Background migrations run on API pods only.** Workers don't register system-job handlers. Deployments with zero API pods will never finish backfills; not a supported topology, worth flagging if it comes up.
- **`check-migrations` and `db-migration` build their database with `src/migrate.ts`, not `typeorm migration:run`.** The CLI runs only the blocking track, so it reports every entity index that a background migration builds as drift. `migrate.ts` runs `runMigrationsWithCatchup` and then the background track, the boot order, through the same `distributedLock` as boot. It ends with `process.exit`: on memory Redis the lock starts redis-memory-server, whose open connection would keep the step from exiting, and the library's killer process stops `redis-server` once the parent is gone.
- **A role-level `statement_timeout` on the main app role kills background migrations, and runtime `SET` can't substitute through a pooler.** Managed Postgres (DO, Supabase) caps the app role, which cancels every long `CREATE/DROP/REINDEX CONCURRENTLY` and chunked backfill. The retry loop leaves 0-byte INVALID index artifacts behind that `IF NOT EXISTS` on the next deploy silently skips (that's how `idx_run_project_id_environment_created_status_archived_at` and `idx_todo_run_id` sat on Cloud `flow_run`/`todo` as orphans). Runtime `SET statement_timeout = 0` is unsound on a transaction-mode pooler: the `SET` lands on one backend, the DDL on another that still has the cap, and the `SET` leaks to normal app traffic on the first backend (empirically verified against DO's pool). Fix: dedicated role (`ALTER ROLE ap_migrator SET statement_timeout = 0`) + `AP_POSTGRES_MIGRATION_URL` pointing the runner at a pool for that role. `migrationDatabaseConnection()` resolves this; unset or non-Postgres falls back to the main DS. Full SQL in `docs/install/reference/breaking-changes-unreleased.mdx`.
diff --git a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
index 3018b1404f9..df374ef9687 100644
--- a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
+++ b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
@@ -105,6 +105,14 @@ all, purely because child rows appear later than the dispatch that created them.
- **`result` is the first place unauthenticated free text reaches a jsonb column.** Bounded to 2 000
characters **server-side** (rejected at the page, never truncated), passed through
`sanitizeObjectForPostgresql()` before the insert, stored as text and rendered escaped.
+- **A child run signals its barrier through `flow_run.parentWaitpointId`, and `dispatchIndex` is a separate
+ column rather than a reuse of it.** The terminal-run hook maps the run's status onto a
+ `BarrierSignalStatus` and calls `receive({ refId: runId })`, so the barrier never polls its children.
+ `parentWaitpointId` doubles as the discriminator that keeps children off the runs list, the status counts,
+ the billing daily count and the analytics rollup — and stays out of `healthMetricsService`, which counts
+ them by design. `dispatchIndex` exists because attribution can legitimately be dropped while the run is
+ still a child: it is the "started mid-graph" flag retry refuses on, and the ordering key for listing one
+ barrier's children.
- **Accepted cost: ~30k row writes per 10 000-signal barrier** — insert, receive, delete — on top of the
children themselves. Bounded, on a narrow table, and gone the moment the barrier resolves.
- **A returning approver still sees a bare "already responded".** Signals die with the release, so there is
diff --git a/brain/knowledge/decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md b/brain/knowledge/decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md
index a6d563b19ed..a261d1aaaf1 100644
--- a/brain/knowledge/decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md
+++ b/brain/knowledge/decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md
@@ -9,8 +9,8 @@ status: accepted
Delivering a resume no longer means deleting the waitpoint row. `waitpointService.consume` owns the
rule: a `BARRIER` is tombstoned at a third status, `CONSUMED`, and every other type is still deleted.
-The tombstone lives until the run reaches a terminal status, where the existing `deleteByFlowRunId`
-sweep clears it. The legacy by-run resume guard (`hasBarrier`) therefore asks whether the run holds a
+The tombstone lives until the run reaches a terminal status, where `onFinish`
+(`waitpointService.deleteForFinishedRun`) clears it; that call deletes a barrier whatever the run's status. The legacy by-run resume guard (`hasBarrier`) therefore asks whether the run holds a
barrier row **at all**, in any status, instead of whether one is `PENDING`.
## Context
diff --git a/brain/knowledge/flows-execution/flow-runs.md b/brain/knowledge/flows-execution/flow-runs.md
index 5476e851216..b9fc52a2697 100644
--- a/brain/knowledge/flows-execution/flow-runs.md
+++ b/brain/knowledge/flows-execution/flow-runs.md
@@ -7,7 +7,7 @@ icon: 🏃
A Flow Run records one execution of a specific flow version, from trigger to terminal state. It stores compressed step-by-step logs, supports pause/resume for delay and webhook waits, offers retry strategies, and emits WebSocket + application events for real-time UI.
### Entities & services
-- **FlowRun** — id, projectId, flowId, flowVersionId, environment (PRODUCTION/TESTING), status, logsFileId, parentRunId (subflows), failedStep (JSONB `{name, displayName, message?}`), timeline (JSONB), archivedAt (soft delete).
+- **FlowRun** — id, projectId, flowId, flowVersionId, environment (PRODUCTION/TESTING), status, logsFileId, parentRunId (subflows), `parentWaitpointId` + `dispatchIndex` (barrier attribution — which barrier counts this run, and its ordinal within that barrier), failedStep (JSONB `{name, displayName, message?}`), timeline (JSONB), archivedAt (soft delete).
- **12 statuses**: 3 non-terminal (QUEUED, RUNNING, PAUSED) + 9 terminal (SUCCEEDED, FAILED, TIMEOUT, CANCELED, QUOTA_EXCEEDED, MEMORY_LIMIT_EXCEEDED, INTERNAL_ERROR, LOG_SIZE_EXCEEDED).
- **Waitpoint** — row per paused step: `type` (DELAY|WEBHOOK|BARRIER), `version` (V0|V1), `status` (PENDING|COMPLETED), unique on `(flow_run_id, step_name)`. A BARRIER adds `sealed` (bool) and a nullable `policy` jsonb, and any row may carry `deadLetteredAt`. A partial index on `resumeDateTime` (`idx_waitpoint_live_deadline`, `WHERE status = 'PENDING' AND "resumeDateTime" IS NOT NULL AND "deadLetteredAt" IS NULL`) serves the deadline sweep; every read path leads with `flowRunId` or the primary key, but `idx_waitpoint_project_id` stays because `fk_waitpoint_project_id` is `ON DELETE CASCADE` and a project delete seq-scans the table without it.
- **WaitpointSignal** — one row per thing a BARRIER awaits, created up front: `status` (PENDING|SUCCEEDED|FAILED|REJECTED|CANCELED|NOT_DISPATCHED), `refId` (the child run or link it stands for), nullable `sequence` (the producer's ordinal, partial-unique per barrier), nullable `label`, small `result` jsonb. Release is the pure `shouldReleaseBarrier({ policy, sealed, counts })` in `core-execution` — see decision [000015](../decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md).
@@ -61,8 +61,19 @@ A Flow Run records one execution of a specific flow version, from trigger to ter
- **A cancelled run's terminal status is only *queued*, so `flow_run.status` still reads `PAUSED` for a moment — read the Redis intent, not the row.** `cancelSingleRun` holds `runs_metadata_${id}`, removes the jobs and deletes every waitpoint, then hands `CANCELED` to `runsMetadataQueue` rather than writing it. For that window a resume request sees a `PAUSED` run with no waitpoint and cannot tell it from a legacy pre-waitpoint pause, and nothing downstream stops it — `execute-flow.ts` has no terminal-status guard, so the resume actually runs and overwrites the cancellation. The intent *is* observable: `runsMetadataQueue.add` writes the pending status into the `runs_metadata:<runId>` hash synchronously, inside that same lock, before enqueuing, and the metadata worker clears the key with `deleteKeyIfFieldValueMatches` once it lands. So `resumeService` refuses when `hgetJson(redisMetadataKey(id))` already holds a terminal status, and every resume path — the legacy by-run ones included — takes `runs_metadata_${id}` so the two orderings are both safe: cancel-first makes the resume wait and then see it, resume-first lets cancel's `removeAllFlowRunJobs` reap the job it just enqueued. The lock alone is not enough, and neither is re-reading `flow_run` inside it.
- **The `RESUME_DELAY_WAITPOINT` job id is keyed per waitpoint, and it has to stay that way.** `waitpointService.createForPause` schedules it as `resumeDelayJobId(waitpoint.id)` — never per run — because `systemJobsSchedule.upsertJob` only adds when that id is free (`if (isNil(existingJob))`), so a run-keyed id means the first waitpoint to arm owns the run's **only** timer and every later one silently gets none. Since decision 000015 one run legitimately holds several waitpoints, so that is a hang, not an edge case: `create_approval_links` (WEBHOOK, +30d) followed by `delayFor 45s` (DELAY) left the delay with no timer and the run sat `PAUSED` until the 30-day webhook expiry marked it FAILED. Two `WEBHOOK` rows hide it — identical `AP_PAUSED_FLOW_TIMEOUT_DAYS` deadlines, and each is resumed by its own callback anyway — which is why flow 4 of `/test-waitpoints` never caught it and flow 9 was added. Found and fixed 2026-09-10; the bug had been live on `main`, not introduced by the barrier work. Per-waitpoint keying also fixes the `removeOnFail: { age: ONE_MONTH }` case, where `upsertJob` retries a month-old failed job and would otherwise replay it against a **different** waitpoint's id. Pinned by "should give each waitpoint of one run its own resume job id" in `waitpoint.test.ts`.
- **Post-run metering window**: AI usage is metered only at `onFinish`, so a long run can spend past the credit limit before anything lands; interim by design — see decision 000016.
-- **A step that arms a waitpoint and then asks the API to fill it can leave the row behind if that second call fails, and the leftover is safe only while the run stays out of PAUSED.** Both the AI pieces (`createWaitpoint` then `POST /v1/ai/execute`) and the AI Router (`waitpointClient.create` then `POST /v1/engine/ai-router`) create the row first, because the request has to carry the waitpoint id. A failed second call fails the step and the run, but the PENDING row and its `RESUME_DELAY_WAITPOINT` timer survive; nothing on the failure path deletes them, only `cancel` does (`cancelSingleRun` calls `deleteByFlowRunId`). `handleResumeDelayWaitpoint` skips a timer whose run is not PAUSED or whose row is not PENDING, so a failed run is never touched. The one corner: a retry of that run within the deadline (10 min for the router) that pauses somewhere else is PAUSED when the stale WEBHOOK timer fires, and `isWebhookExpiry` marks it FAILED. Reviewed on #15707 and left as is; the fix, if it ever bites, is a delete of the row on the failure path or a retry that clears pending waitpoints.
+- **A step that arms a waitpoint and then asks the API to fill it can leave the row behind if that second call fails, and the leftover is safe only while the run stays out of PAUSED.** Both the AI pieces (`createWaitpoint` then `POST /v1/ai/execute`) and the AI Router (`waitpointClient.create` then `POST /v1/engine/ai-router`) create the row first, because the request has to carry the waitpoint id. A failed second call fails the step and the run, and `onFinish` then deletes the PENDING row and its `RESUME_DELAY_WAITPOINT` timer, because the row belongs to the step that failed (see the next bullet for what a failed run keeps). `onFinish` never runs for `INTERNAL_ERROR` (it calls `isFlowRunStateTerminal` with `ignoreInternalError: true`); `cancel` deletes every row (`cancelSingleRun` calls `deleteByFlowRunId`). `handleResumeDelayWaitpoint` skips a timer whose run is not PAUSED or whose row is not PENDING, so a failed run is never touched. The one corner is a run that ends `INTERNAL_ERROR` and so keeps its rows: a retry of that run within the deadline (10 min for the router) that pauses somewhere else is PAUSED when the stale WEBHOOK timer fires, and `isWebhookExpiry` marks it FAILED. Reviewed on #15707 and left as is. A retry that clears every waitpoint was tried on #14888 and dropped, because it also kills an approval link an earlier step handed out; the fix, if this ever bites, is a delete of the row on the failure path.
+- **A failed run keeps the waitpoints of the steps that finished, so a retry can still honour a link it handed out.** `FROM_FAILED_STEP` keeps every earlier step's output, including a `create_approval_links` link that a later step may already have emailed. So `onFinish` (`waitpointService.deleteForFinishedRun`) deletes all rows only for a run that succeeded, was cancelled or names no failed step; a run in a failed state (`isFailedState`) loses its barriers and the rows of its failed step — matched on the last segment of the row's `stepName`, so every loop iteration counts — and keeps the rest with their timers. The retry leaves the rows alone, and `createForPause` re-arms a step through `.orIgnore()` on `(flowRunId, stepName)`, so a step that runs again gets its old row and its old link back. Kept rows are inert while the run is terminal: the deadline sweep and `handleResumeDelayWaitpoint` act only on PAUSED runs, and a click answers "expired". Accepted cost: a failed run that is never retried keeps those rows for good — `waitpoint` has no foreign key to `flow_run` and nothing sweeps them — which is one row per handed-out link, not per failure, because a failed step's own rows (an expired approval, an AI leftover) are still deleted. A barrier parent needs a fresh barrier on retry; that is why barriers always go, and any re-dispatch reset belongs next to the code that re-dispatches its children. `ON_LATEST_VERSION` copies `parentRunId` but must never copy `parentWaitpointId`: the barrier already counted the original child, so attributing the retry to it would inflate `terminal` and risk a premature release.
+- **`parentWaitpointId IS NOT NULL` is the one test for "this run is a barrier child". `dispatchIndex` is ordering data, never an identity test.** The two columns are written together by the single dispatcher and are set together or not at all; treat that as the invariant and keep any new producer to it. The only place that reads both is `flowRunService.retry`, which refuses a run carrying *either* column — belt and braces, because a retry that slipped through would re-signal a barrier that already counted the original child. Retrying a child is meaningless in both strategies anyway: `FROM_FAILED_STEP` resumes from the trigger, so it would re-run the *parent's* whole prefix inside the child, and `ON_LATEST_VERSION` starts a fresh top-level run that no barrier is counting. `retry` throws `VALIDATION` before the retention check.
+- **The bulk helper hides children from a filter, never from an explicit id.** `filterFlowRunsAndApplyFilters` backs `cancel`, `bulkArchive` *and* `bulkRetry`, so an unconditional `parentWaitpointId IS NULL` there made `cancel({ flowRunIds: [childId] })` return 200 and do nothing while the child ran on and the barrier waited. Cancelling a stuck child directly is a legitimate operator action, and it is already safe end to end: `cancelSingleRun` hands `CANCELED` to `runsMetadataQueue`, which is terminal, so the barrier receives the signal. The child filter therefore applies only when the caller passes no `flowRunIds`. It is not load-bearing for retry safety — `bulkRetry` calls `retry()` per run, and that guard is the real lock; a filter-driven bulk retry would otherwise return one error row per child after a wide fan-out.
+- **`parentWaitpointId` (and any new run field) must be added to `RUNS_METADATA_UPSERT_KEYS`** in `runs-metadata-queue-factory.ts`. It is an allowlist — an unlisted key is silently dropped between the worker and the row with no error, the same failure class as forgetting `getEntities()`.
+- **A new index on a mostly-NULL `flow_run` column must be partial.** Postgres indexes NULLs (unlike Oracle), and a non-HOT UPDATE inserts a tuple into *every* index on the table including ones on columns it did not touch — `flow_run` updates are essentially all non-HOT already (`status` sits in five composite indexes, `logsFileId` is indexed and the 15s log backup touches it). So a full index on a column that is NULL for ~every row adds permanent write amplification to the hottest write path in the product for reads that almost never happen. `WHERE <col> IS NOT NULL` makes that maintenance **zero**, not smaller, and the planner still uses the index for `col = $1` because `x = c` implies `x IS NOT NULL`. `idx_run_parent_waitpoint_id` is the worked example: partial, and widened to `(parentWaitpointId, projectId, status)` so the barrier's existence probe and release aggregate are index-only. `idx_run_parent_run_id`, `idx_run_logs_file_id` and `idx_run_triggered_by` are still full indexes on mostly-NULL columns.
+- **Barrier children are unlisted, never invisible — and the discriminator is `parentWaitpointId IS NOT NULL`, nothing cheaper.** Children share the parent's flow id, flow version id *and* display name. Of the eight run list/count surfaces, five filter them out and three deliberately keep them: **user-facing and billing surfaces exclude children, ops and load surfaces keep them.** The five are `flowRunService.list` (one edit covering the runs endpoint, the builder's list, the agent tool and the MCP tool), `countByStatus`, `filterFlowRunsAndApplyFilters`, the billing usage report's daily-execution count and `platformAnalyticsReportService`'s `listRuns`. The three that keep counting are `healthMetricsService`, the telemetry daily rollup in `flow-run-module.ts` and the admin platform retry. Two of the five are correctness, not consistency: the shared filter-and-apply helper is the *only* thing stopping a filter-driven bulk retry from being a second door onto the refused mid-graph retry path, and the analytics filter multiplies a user-set per-trigger constant, so counting children makes a flow batching ten items claim eleven times the time saved. Reaching a child by id is the zero-work default (`getOnePopulatedOrThrow` already scopes by project) and `ListFlowRunsRequestQuery.parentWaitpointId` is the opt-back-in, ordered by `dispatchIndex` — a filter on the existing endpoint, not a new one. That listing also requires `dispatchIndex IS NOT NULL`, and the reason is the cursor, not taste: `Paginator` slices its composite cursor at the **first field missing from the decoded cursor object**, so a page ending on a NULL-index child encodes no `dispatchIndex`, emits *no* cursor predicate at all, and returns page 1 again forever. Any ordering added on a nullable column has this failure mode — either exclude the NULLs or do not sort on it. A child with no dispatch index is not a batch (only the webhook header path can make one) and stays reachable by id. `archivedAt` is a separate user-driven axis and is **not** reused for hiding.
+- **The status-count query plans on `idx_run_project_id_environment_status_created_archived_at`, not the `created`-leading one — but only once the visibility map is healthy.** `EXPLAIN` it on a table whose VM is stale (right after a bulk `UPDATE`) and the planner refuses the index-only scan, falls back to a bitmap heap scan on `idx_run_project_id_environment_created_archived_at`, and you extend the wrong index. `VACUUM ANALYZE` first, then `EXPLAIN`. Appending `parentWaitpointId` as the trailing column keeps the scan index-only and pushes the child exclusion into the Index Cond instead of a post-heap Filter — measured on 1M rows / 50k per project: 42ms and 17.8k buffers → 5.5ms and 4k buffers, `Heap Fetches: 0`. A btree cannot be extended in place, so the migration creates the widened index under a new name **concurrently first** and only then drops the old one; dropping first would leave the hottest table in the product without that index for the length of a concurrent build. The children-listing query needs no new index — the partial `idx_run_parent_waitpoint_id` already covers it. **But the widened index only covers the status-count query, and the runs *list* was left behind.** `flowRunService.list` with no status filter orders by `created DESC` and plans on `idx_run_project_id_environment_created_archived_at`, which has no `parentWaitpointId` — so there the child exclusion is exactly the post-heap Filter this bullet says was eliminated, and every child row created since the newest visible parent is fetched and discarded before the page fills. Cost scales with children ever created, not with page size: right after a wide fan-out the first page walks the whole batch to return ten rows, and the table re-polls it every 15s while anything is non-terminal. **Resolved by making the three `created`-leading list indexes partial on `WHERE "parentWaitpointId" IS NULL`** — the only shape the list asks for, and a child costs nothing to maintain there (see the mostly-NULL bullet above): children are absent from the index entirely, so cost is proportional to page size instead of fan-out size, and a fan-out writes zero entries into them. A normal run is in every one of them, so until the four old list indexes are dropped, each insert and non-HOT update of a normal run maintains eight list indexes instead of five. Measured on 1.02M rows / 200 projects / a 20k-child fan-out as the newest activity: the no-filter list 4.5ms → 0.11ms, flow+status 6.2ms → 0.79ms, `countByStatus` 20.5ms → 1.2ms and the billing daily count 10.7ms → 2.3ms (both now `Index Only Scan`, `Heap Fetches: 0`). Trailing the column instead of a partial predicate was measured too and is the weaker fix — it moves the exclusion into the Index Cond but still walks every child's leaf pages, so it only bought 2-3x where partial buys 40x.
+- **The status-leading run index must stay NON-partial, and that asymmetry is deliberate.** It is the only index left that can serve the ops aggregates in `healthMetricsService`, which by design count children (see the unlisted-children bullet) and therefore carry no `parentWaitpointId` predicate. Make all four partial and those seven aggregates fall to a `Parallel Seq Scan` on the whole table. The alternative — adding the predicate to `healthMetricsService` — was rejected because it silently reclassifies an ops surface as a user-facing one. So: `created`-leading and `flowId`-leading indexes are partial (list-only), the `status`-leading one is full and keeps `parentWaitpointId` as a trailing column so `countByStatus` stays index-only.
+- **The `flow_run` list-index reshape runs in the background track, not at boot.** `AddBarrierChildAttribution` (blocking) only adds the two nullable columns; `ReshapeFlowRunListIndexesForBarrierChildren` (background) builds the five new indexes after the app listens, so a large `flow_run` no longer holds every replica's boot for five concurrent builds. It then drops only `idx_run_project_id_environment_created_archived_at`: with that narrow index present, the planner picks it over the partial `created` index and reads every child again (PGlite, 1M rows and a 20k fan-out: 0.14ms → 3.2ms). Its queries stay covered under the old image, because `idx_run_project_id_environment_created_status_archived_at` has the same leading columns. The other four old indexes stay declared on the entity until a later release drops them, because a new-image pod that boots during a Cloud rollback re-runs the reshape (see the rollback gotcha on the Database page). The new code is fast on the old indexes meanwhile: the list query keeps its index and adds only a `parentWaitpointId IS NULL` filter that matches every row until children exist. **The blocking `down()` restores the old indexes, drops the new ones and deletes the background row before it drops the columns**, because the background track is forward-only and `DROP COLUMN` silently drops all five new indexes with it (each one references `parentWaitpointId`), which would leave `flow_run` with only its primary key. Deleting the row makes a re-upgrade run the reshape again.
+- **A run read must never widen its `flow_version` join, and the paginator is why it costs double.** `queryBuilderForFlowRun` is the single builder behind `list`, `getOne`, `getOnePopulatedOrThrow` *and* `findFlowRunOrThrow`, and the only thing any caller ever reads off the relation is `displayName` — the runs table column and the failed-step dialog. It used to be `leftJoinAndSelect`, which selects all thirteen `flow_version` columns including three `jsonb` ones: `trigger` (the whole flow graph, routinely TOASTed), `backupFiles`, `notes`. The `.addSelect(['"flowVersion"."displayName"'])` sitting beside it was raw SQL, so TypeORM mapped it to nothing — a duplicate column in the SELECT list that read as intent and did no work. Three things compounded it: `Paginator.paginate` uses `getRawAndEntities()` and `mergeAdditionalColumns` copies **every raw key not prefixed `<rootAlias>_` or `ap_cursor_`** onto the entity, so each row carried the graph twice (nested `flowVersion.trigger` *and* flattened `flowVersion_trigger`) — that flattening applies to any paginated list with a join, not just runs; `convertDatesToStrings` in `server.ts` then deep-walks both copies before zod strips them, so the wire payload stayed clean while the DB read, the heap and the per-request CPU did not; and `releaseBarrierWithoutLock` calls `findFlowRunOrThrow` up to 3× per barrier release just to read `status`. The fix is `leftJoin` + `addSelect(['flowVersion.id', 'flowVersion.displayName'])`, and **the PK is load-bearing, not decoration** — `RawSqlResultsToEntityTransformer.group()` keys joined rows by the alias's primary columns, so a partial relation select without the id silently fails to hydrate. Use entity-property notation (`'flowVersion.displayName'`); the quoted raw form is not mapped back onto the relation. Pinned by `list-flow-runs.test.ts`, which asserts on the **service** result, not the HTTP response — zod strips the heavy keys at serialization, so an HTTP-only assertion passes against the wide query too and proves nothing. Measured on 1M runs / 200 projects / 10k flow versions with a ~6.4 KB-on-disk TOASTed trigger, fully warm: a 50-row page went **1.59 MB → 15 KB** returned and 4.63ms → 0.56ms median, and the single-row read behind `findFlowRunOrThrow` 0.63ms → 0.30ms. **`EXPLAIN (ANALYZE, BUFFERS)` cannot measure this and will tell you there is nothing to fix** — it discards output rows, so it never detoasts the wide columns; the before/after execution-buffer difference was 9 pages and the only four-figure number in the plan was *planning* buffers. Time the query with its rows actually materialized (`\timing` + `\o /dev/null`). Two further traps in seeding it: give each project its own flow versions, or a page's lookups collapse into a handful of `Memoize` hits and the join looks free, and fill `trigger` with random text rather than repeated characters, or pglz compresses the payload away.
+- **Every path that creates a *terminal* run must carry `parentWaitpointId`, not just the paths that execute one.** The barrier releases from a `GROUP BY` over `flow_run.parentWaitpointId`, so a child row that lands terminal without it is invisible: `terminal >= expected` never holds and the parent hangs to the barrier's `timeoutAt`. The `QUOTA_EXCEEDED` admission gate is the trap — `createQuotaExceededRun` writes a finished row straight to `runsMetadataQueue`, bypassing `flowRunService.start`, so it needs **both** `parentWaitpointId` and `dispatchIndex` threaded explicitly at all three call sites (worker RPC `submitPayloads`, sync webhook, manual trigger) — `dispatchIndex` is not in `RUNS_METADATA_UPSERT_KEYS`, so add it there first or the allowlist drops it and the row lands half-attributed. Found while merging the gate into the fan-in branch: neither side was wrong alone. Any future admit-without-executing path inherits the same obligation.
### Editions
CE has full run tracking. Cloud may enforce retention windows; bulk-retry admin endpoint is Cloud-only.
diff --git a/bun.lock b/bun.lock
index f52360309ea..fef073b6d13 100644
--- a/bun.lock
+++ b/bun.lock
@@ -122,7 +122,7 @@
},
"packages/core/execution": {
"name": "@activepieces/core-execution",
- "version": "0.26.3",
+ "version": "0.27.0",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
diff --git a/packages/core/execution/package.json b/packages/core/execution/package.json
index 535cc909247..b7ec7e97a07 100644
--- a/packages/core/execution/package.json
+++ b/packages/core/execution/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-execution",
- "version": "0.26.4",
+ "version": "0.27.0",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/execution/src/lib/flow-run/dto/list-flow-runs-request.ts b/packages/core/execution/src/lib/flow-run/dto/list-flow-runs-request.ts
index ea7575e6527..cb7e59cc12a 100755
--- a/packages/core/execution/src/lib/flow-run/dto/list-flow-runs-request.ts
+++ b/packages/core/execution/src/lib/flow-run/dto/list-flow-runs-request.ts
@@ -16,6 +16,7 @@ export const ListFlowRunsRequestQuery = z.object({
failedStepMessage: z.string().optional(),
flowRunIds: OptionalArrayFromQuery(ApId),
includeArchived: OptionalBooleanFromQuery,
+ parentWaitpointId: z.optional(ApId),
})
export type ListFlowRunsRequestQuery = z.infer<typeof ListFlowRunsRequestQuery>
diff --git a/packages/core/execution/src/lib/flow-run/flow-run.ts b/packages/core/execution/src/lib/flow-run/flow-run.ts
index 1cc35da2880..d6ee99d27f5 100755
--- a/packages/core/execution/src/lib/flow-run/flow-run.ts
+++ b/packages/core/execution/src/lib/flow-run/flow-run.ts
@@ -61,6 +61,8 @@ export const FlowRun = z.object({
projectId: z.string(),
flowId: z.string(),
parentRunId: z.string().optional(),
+ parentWaitpointId: z.string().optional(),
+ dispatchIndex: z.number().int().nullish(),
failParentOnFailure: z.boolean(),
triggeredBy: z.string().optional(),
tags: z.array(z.string()).optional(),
diff --git a/packages/core/execution/src/lib/workers/job-data.ts b/packages/core/execution/src/lib/workers/job-data.ts
index cfd6c5872dc..76f2542f475 100644
--- a/packages/core/execution/src/lib/workers/job-data.ts
+++ b/packages/core/execution/src/lib/workers/job-data.ts
@@ -37,14 +37,18 @@ export const JOB_PRIORITY = {
const TESTING_EXECUTE_FLOW_PRIORITY: keyof typeof JOB_PRIORITY = 'high'
const ASYNC_EXECUTE_FLOW_PRIORITY: keyof typeof JOB_PRIORITY = 'medium'
const SYNC_EXECUTE_FLOW_PRIORITY: keyof typeof JOB_PRIORITY = 'high'
+const FAN_IN_CHILD_EXECUTE_FLOW_PRIORITY: keyof typeof JOB_PRIORITY = 'low'
export const RATE_LIMIT_PRIORITY: keyof typeof JOB_PRIORITY = 'lowest'
-function getExecuteFlowPriority(environment: RunEnvironment, workerHandlerId: string | undefined | null): keyof typeof JOB_PRIORITY {
- switch (environment) {
+function getExecuteFlowPriority(job: ExecuteFlowJobData): keyof typeof JOB_PRIORITY {
+ switch (job.environment) {
case RunEnvironment.TESTING:
return TESTING_EXECUTE_FLOW_PRIORITY
case RunEnvironment.PRODUCTION:
- return isNil(workerHandlerId) ? ASYNC_EXECUTE_FLOW_PRIORITY : SYNC_EXECUTE_FLOW_PRIORITY
+ if (!isNil(job.parentWaitpointId)) {
+ return FAN_IN_CHILD_EXECUTE_FLOW_PRIORITY
+ }
+ return isNil(job.workerHandlerId) ? ASYNC_EXECUTE_FLOW_PRIORITY : SYNC_EXECUTE_FLOW_PRIORITY
}
}
@@ -58,7 +62,7 @@ export function getDefaultJobPriority(job: JobData): keyof typeof JOB_PRIORITY {
case WorkerJobType.EXECUTE_PERSONALIZATION_RESEARCH:
return 'medium'
case WorkerJobType.EXECUTE_FLOW:
- return getExecuteFlowPriority(job.environment, job.workerHandlerId)
+ return getExecuteFlowPriority(job)
case WorkerJobType.EXECUTE_PROPERTY:
case WorkerJobType.EXECUTE_EXTRACT_PIECE_INFORMATION:
case WorkerJobType.EXECUTE_VALIDATION:
@@ -129,6 +133,7 @@ const ExecuteFlowJobDataCommon = z.object({
stepNameToTest: z.string().optional(),
sampleData: z.record(z.string(), z.unknown()).optional(),
logsFileId: z.string(),
+ parentWaitpointId: z.string().optional(),
})
export const BeginExecuteFlowJobData = ExecuteFlowJobDataCommon.extend({
diff --git a/packages/core/execution/test/workers/job-priority.test.ts b/packages/core/execution/test/workers/job-priority.test.ts
new file mode 100644
index 00000000000..0e13be9f3df
--- /dev/null
+++ b/packages/core/execution/test/workers/job-priority.test.ts
@@ -0,0 +1,42 @@
+import { describe, expect, it } from 'vitest'
+import { ExecutionType } from '../../src/lib/flow-run/execution/execution-output'
+import { RunEnvironment } from '../../src/lib/flow-run/flow-run'
+import { getDefaultJobPriority, JOB_PRIORITY, WorkerJobType } from '../../src/lib/workers/job-data'
+import { ResumeReason, StreamStepProgress } from '../../src/lib/engine'
+
+const baseJob = {
+ projectId: 'project-id',
+ platformId: 'platform-id',
+ jobType: WorkerJobType.EXECUTE_FLOW as const,
+ environment: RunEnvironment.PRODUCTION,
+ schemaVersion: 1,
+ flowId: 'flow-id',
+ flowVersionId: 'flow-version-id',
+ runId: 'run-id',
+ payload: { type: 'inline' as const, value: {} },
+ streamStepProgress: StreamStepProgress.NONE,
+ logsFileId: 'logs-file-id',
+ executionType: ExecutionType.BEGIN as const,
+}
+
+describe('getDefaultJobPriority', () => {
+ it('demotes a fan-in child below an ordinary async run', () => {
+ const child = getDefaultJobPriority({ ...baseJob, parentWaitpointId: 'waitpoint-id' })
+ const ordinary = getDefaultJobPriority(baseJob)
+
+ expect(JOB_PRIORITY[child]).toBeGreaterThan(JOB_PRIORITY[ordinary])
+ })
+
+ it('keeps a fan-in child above polling so triggers cannot starve', () => {
+ const child = getDefaultJobPriority({ ...baseJob, parentWaitpointId: 'waitpoint-id' })
+
+ expect(JOB_PRIORITY[child]).toBeLessThan(JOB_PRIORITY.veryLow)
+ })
+
+ it('leaves a resumed parent in the ordinary lane, ahead of its children', () => {
+ const parentResume = getDefaultJobPriority({ ...baseJob, executionType: ExecutionType.RESUME, resumeReason: ResumeReason.WAITPOINT})
+ const child = getDefaultJobPriority({ ...baseJob, parentWaitpointId: 'waitpoint-id' })
+
+ expect(JOB_PRIORITY[parentResume]).toBeLessThan(JOB_PRIORITY[child])
+ })
+})
diff --git a/packages/server/api/src/app/analytics/platform-analytics-report.service.ts b/packages/server/api/src/app/analytics/platform-analytics-report.service.ts
index 611e51fadb4..9fcc3610634 100644
--- a/packages/server/api/src/app/analytics/platform-analytics-report.service.ts
+++ b/packages/server/api/src/app/analytics/platform-analytics-report.service.ts
@@ -132,6 +132,7 @@ async function listRuns(projectIds: string[], afterDate: string | null, currentD
.addSelect('COUNT(*)', 'runs')
.where('flow_run.projectId IN (:...projectIds)', { projectIds })
.andWhere('flow_run.environment = :environment', { environment: RunEnvironment.PRODUCTION })
+ .andWhere('flow_run.parentWaitpointId IS NULL')
.groupBy('flow_run.flowId')
.addGroupBy('DATE_TRUNC(\'day\', flow_run.created)')
diff --git a/packages/server/api/src/app/database/migration/postgres/1871000000000-AddBarrierChildAttribution.ts b/packages/server/api/src/app/database/migration/postgres/1871000000000-AddBarrierChildAttribution.ts
new file mode 100644
index 00000000000..53874723854
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/1871000000000-AddBarrierChildAttribution.ts
@@ -0,0 +1,49 @@
+import { QueryRunner } from 'typeorm'
+import { system } from '../../../helper/system/system'
+import { BACKGROUND_MIGRATIONS_TABLE } from '../../background-migration-runner'
+import { Migration } from '../../migration'
+import { migrationHelpers } from '../../migration-helpers'
+import {
+ FLOW_RUN_BARRIER_INDEXES,
+ FLOW_RUN_LEGACY_LIST_INDEXES,
+ ReshapeFlowRunListIndexesForBarrierChildren1871000000000,
+} from './background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren'
+
+export class AddBarrierChildAttribution1871000000000 implements Migration {
+ name = 'AddBarrierChildAttribution1871000000000'
+ breaking = false
+ release = '0.93.0'
+ transaction = false
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`
+ ALTER TABLE "flow_run"
+ ADD COLUMN IF NOT EXISTS "parentWaitpointId" character varying(21)
+ `)
+ await queryRunner.query(`
+ ALTER TABLE "flow_run"
+ ADD COLUMN IF NOT EXISTS "dispatchIndex" integer
+ `)
+ }
+
+ public async down(queryRunner: QueryRunner): Promise<void> {
+ await migrationHelpers.withBackgroundMigrationLock({
+ log: system.globalLogger(),
+ migration: ReshapeFlowRunListIndexesForBarrierChildren1871000000000,
+ fn: async () => {
+ for (const index of FLOW_RUN_LEGACY_LIST_INDEXES) {
+ await migrationHelpers.createIndexConcurrently({ queryRunner, table: 'flow_run', ...index })
+ }
+ for (const index of FLOW_RUN_BARRIER_INDEXES) {
+ await migrationHelpers.dropIndexConcurrently({ queryRunner, name: index.name })
+ }
+ await queryRunner.query(
+ `DELETE FROM "${BACKGROUND_MIGRATIONS_TABLE}" WHERE "name" = $1`,
+ [new ReshapeFlowRunListIndexesForBarrierChildren1871000000000().name],
+ )
+ await queryRunner.query('ALTER TABLE "flow_run" DROP COLUMN IF EXISTS "dispatchIndex"')
+ await queryRunner.query('ALTER TABLE "flow_run" DROP COLUMN IF EXISTS "parentWaitpointId"')
+ },
+ })
+ }
+}
diff --git a/packages/server/api/src/app/database/migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren.ts b/packages/server/api/src/app/database/migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren.ts
new file mode 100644
index 00000000000..81f6d35df9c
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren.ts
@@ -0,0 +1,76 @@
+import { QueryRunner } from 'typeorm'
+import { BackgroundMigration } from '../../../background-migration'
+import { migrationHelpers } from '../../../migration-helpers'
+
+export class ReshapeFlowRunListIndexesForBarrierChildren1871000000000 implements BackgroundMigration {
+ name = 'ReshapeFlowRunListIndexesForBarrierChildren1871000000000'
+ release = '0.93.0'
+ transaction = false as const
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ for (const index of FLOW_RUN_BARRIER_INDEXES) {
+ await migrationHelpers.createIndexConcurrently({ queryRunner, table: 'flow_run', ...index })
+ }
+ await migrationHelpers.dropIndexConcurrently({ queryRunner, name: FLOW_RUN_SHADOWING_LEGACY_INDEX_NAME })
+ }
+
+ public async down(): Promise<void> {}
+}
+
+export const FLOW_RUN_SHADOWING_LEGACY_INDEX_NAME = 'idx_run_project_id_environment_created_archived_at'
+
+export const FLOW_RUN_BARRIER_INDEXES: FlowRunIndex[] = [
+ {
+ name: 'idx_run_parent_waitpoint_id',
+ columns: '"parentWaitpointId", "projectId", "dispatchIndex", "status"',
+ where: '"parentWaitpointId" IS NOT NULL',
+ },
+ {
+ name: 'idx_run_project_env_status_created_archived_parent_waitpoint',
+ columns: '"projectId", "environment", "status", "created" DESC, "archivedAt", "parentWaitpointId"',
+ },
+ {
+ name: 'idx_run_project_env_created_archived_status_parent_waitpoint',
+ columns: '"projectId", "environment", "created" DESC, "archivedAt", "status"',
+ where: '"parentWaitpointId" IS NULL',
+ },
+ {
+ name: 'idx_run_project_env_flow_created_archived_parent_waitpoint',
+ columns: '"projectId", "environment", "flowId", "created" DESC, "archivedAt"',
+ where: '"parentWaitpointId" IS NULL',
+ },
+ {
+ name: 'idx_run_project_env_flow_status_created_parent_waitpoint',
+ columns: '"projectId", "environment", "flowId", "status", "created" DESC, "archivedAt"',
+ where: '"parentWaitpointId" IS NULL',
+ },
+]
+
+export const FLOW_RUN_LEGACY_LIST_INDEXES: FlowRunIndex[] = [
+ {
+ name: 'idx_run_project_id_environment_status_created_archived_at',
+ columns: '"projectId", "environment", "status", "created" DESC, "archivedAt"',
+ },
+ {
+ name: 'idx_run_project_id_environment_created_status_archived_at',
+ columns: '"projectId", "environment", "created" DESC, "archivedAt", "status"',
+ },
+ {
+ name: FLOW_RUN_SHADOWING_LEGACY_INDEX_NAME,
+ columns: '"projectId", "environment", "created" DESC, "archivedAt"',
+ },
+ {
+ name: 'idx_run_project_id_environment_flow_id_created_archived_at',
+ columns: '"projectId", "environment", "flowId", "created" DESC, "archivedAt"',
+ },
+ {
+ name: 'idx_run_project_id_environment_flow_id_status_created_archived_',
+ columns: '"projectId", "environment", "flowId", "status", "created" DESC, "archivedAt"',
+ },
+]
+
+type FlowRunIndex = {
+ name: string
+ columns: string
+ where?: string
+}
diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.ts
index ec14f497383..7d4bd0160ee 100644
--- a/packages/server/api/src/app/database/postgres-connection.ts
+++ b/packages/server/api/src/app/database/postgres-connection.ts
@@ -458,7 +458,9 @@ import { AddAiSpecificModelsVisibleToPlatformConfiguration1866000000000 } from '
import { AddAgentTask1867000000000 } from './migration/postgres/1867000000000-AddAgentTask'
import { AddProviderConfigIdToAgentConversation1868000000000 } from './migration/postgres/1868000000000-AddProviderConfigIdToAgentConversation'
import { DropChatRolloutUserTable1870000000000 } from './migration/postgres/1870000000000-DropChatRolloutUserTable'
+import { AddBarrierChildAttribution1871000000000 } from './migration/postgres/1871000000000-AddBarrierChildAttribution'
import { AddAgentConversationProviderConfigIdIndex1869000000000 } from './migration/postgres/background/1869000000000-AddAgentConversationProviderConfigIdIndex'
+import { ReshapeFlowRunListIndexesForBarrierChildren1871000000000 } from './migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren'
const getSslConfig = (): boolean | TlsOptions => {
const useSsl = system.get(AppSystemProp.POSTGRES_USE_SSL)
@@ -473,6 +475,7 @@ const getSslConfig = (): boolean | TlsOptions => {
export const getBackgroundMigrations = (): (new () => BackgroundMigration)[] => {
return [
AddAgentConversationProviderConfigIdIndex1869000000000,
+ ReshapeFlowRunListIndexesForBarrierChildren1871000000000,
]
}
@@ -937,6 +940,7 @@ export const getMigrations = (): (new () => Migration)[] => {
AddAgentTask1867000000000,
AddProviderConfigIdToAgentConversation1868000000000,
DropChatRolloutUserTable1870000000000,
+ AddBarrierChildAttribution1871000000000,
]
return migrations
}
diff --git a/packages/server/api/src/app/ee/license-key-usage-report/license-key-usage-report-service.ts b/packages/server/api/src/app/ee/license-key-usage-report/license-key-usage-report-service.ts
index b36bf226218..090466a37ab 100644
--- a/packages/server/api/src/app/ee/license-key-usage-report/license-key-usage-report-service.ts
+++ b/packages/server/api/src/app/ee/license-key-usage-report/license-key-usage-report-service.ts
@@ -183,6 +183,7 @@ async function countDailyProductionRunsByPlatform({ platformIds, dayStartInclusi
.andWhere('flow_run.environment = :environment', { environment: RunEnvironment.PRODUCTION })
.andWhere('flow_run.created >= :dayStartInclusive', { dayStartInclusive })
.andWhere('flow_run.created < :dayEndExclusive', { dayEndExclusive })
+ .andWhere('flow_run.parentWaitpointId IS NULL')
.groupBy('flow_run.projectId')
.addGroupBy('to_char(flow_run.created AT TIME ZONE \'UTC\', \'YYYY-MM-DD\')')
.getRawMany<{ projectId: string, runDay: string, runCount: string }>()
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-controller.ts b/packages/server/api/src/app/flows/flow-run/flow-run-controller.ts
index 95a8f936e0c..526f603bf3f 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-controller.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-controller.ts
@@ -28,6 +28,7 @@ export const flowRunController: FastifyPluginAsyncZod = async (app) => {
createdBefore: request.query.createdBefore,
flowRunIds: request.query.flowRunIds,
includeArchived: request.query.includeArchived,
+ parentWaitpointId: request.query.parentWaitpointId,
environment: RunEnvironment.PRODUCTION,
})
})
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts b/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts
index 45dcbb1406c..f4b0aa9f616 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts
@@ -12,7 +12,7 @@ import {
BaseColumnSchemaPart,
} from '../../database/database-common'
-type FlowRunSchema = FlowRun & {
+type FlowRunSchema = FlowRunWithDispatchIndex & {
project: Project
flow: Flow
flowVersion: FlowVersion
@@ -41,6 +41,14 @@ export const FlowRunEntity = new EntitySchema<FlowRunSchema>({
...ApIdSchema,
nullable: true,
},
+ parentWaitpointId: {
+ ...ApIdSchema,
+ nullable: true,
+ },
+ dispatchIndex: {
+ type: Number,
+ nullable: true,
+ },
failParentOnFailure: {
type: Boolean,
nullable: false,
@@ -103,10 +111,6 @@ export const FlowRunEntity = new EntitySchema<FlowRunSchema>({
name: 'idx_run_project_id_environment_status_created_archived_at',
columns: ['projectId', 'environment', 'status', 'created', 'archivedAt'],
},
- {
- name: 'idx_run_project_id_environment_created_archived_at',
- columns: ['projectId', 'environment', 'created', 'archivedAt'],
- },
{
name: 'idx_run_project_id_environment_created_status_archived_at',
columns: ['projectId', 'environment', 'created', 'archivedAt', 'status'],
@@ -115,6 +119,25 @@ export const FlowRunEntity = new EntitySchema<FlowRunSchema>({
name: 'idx_run_project_id_environment_flow_id_created_archived_at',
columns: ['projectId', 'environment', 'flowId', 'created', 'archivedAt'],
},
+ {
+ name: 'idx_run_project_env_status_created_archived_parent_waitpoint',
+ columns: ['projectId', 'environment', 'status', 'created', 'archivedAt', 'parentWaitpointId'],
+ },
+ {
+ name: 'idx_run_project_env_created_archived_status_parent_waitpoint',
+ columns: ['projectId', 'environment', 'created', 'archivedAt', 'status'],
+ where: '"parentWaitpointId" IS NULL',
+ },
+ {
+ name: 'idx_run_project_env_flow_created_archived_parent_waitpoint',
+ columns: ['projectId', 'environment', 'flowId', 'created', 'archivedAt'],
+ where: '"parentWaitpointId" IS NULL',
+ },
+ {
+ name: 'idx_run_project_env_flow_status_created_parent_waitpoint',
+ columns: ['projectId', 'environment', 'flowId', 'status', 'created', 'archivedAt'],
+ where: '"parentWaitpointId" IS NULL',
+ },
{
name: 'idx_run_flow_id',
columns: ['flowId'],
@@ -127,6 +150,11 @@ export const FlowRunEntity = new EntitySchema<FlowRunSchema>({
name: 'idx_run_parent_run_id',
columns: ['parentRunId'],
},
+ {
+ name: 'idx_run_parent_waitpoint_id',
+ columns: ['parentWaitpointId', 'projectId', 'dispatchIndex', 'status'],
+ where: '"parentWaitpointId" IS NOT NULL',
+ },
{
name: 'idx_run_flow_version_id',
columns: ['flowVersionId'],
@@ -188,4 +216,8 @@ export const FlowRunEntity = new EntitySchema<FlowRunSchema>({
},
},
},
-})
\ No newline at end of file
+})
+
+export type FlowRunWithDispatchIndex = FlowRun & {
+ dispatchIndex?: number | null
+}
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
index 06971fdeab4..fc51add6d64 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
@@ -1,4 +1,4 @@
-import { ActivepiecesError, apId, Cursor, ErrorCode, FlowId, FlowRunId, FlowVersionId, isNil, PlatformId, ProjectId, SeekPage } from '@activepieces/core-utils'
+import { ActivepiecesError, apId, ApId, Cursor, ErrorCode, FlowId, FlowRunId, FlowVersionId, isNil, PlatformId, ProjectId, SeekPage } from '@activepieces/core-utils'
import { apDayjs, apDayjsDuration, wideEvent } from '@activepieces/server-utils'
import { ExecuteFlowJobData, ExecutionType, ExecutioOutputFile, FileCompression, FileType, FlowRetryStrategy, FlowRun, FlowRunCountByStatus, FlowRunStatus, FlowRunWithRetryError, flowStructureUtil, FlowVersion, GenericStepOutput, isFlowRunStateTerminal, JobPayload, LATEST_JOB_DATA_SCHEMA_VERSION, logSerializer, LogSliceRef, ResumeReason, RunEnvironment, RunInternalError, SampleDataFileType, StepOutput, StepOutputStatus, StepOutputType, StreamStepProgress, WorkerJobType } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
@@ -46,10 +46,15 @@ export const flowRunService = (log: FastifyBaseLogger) => ({
entity: FlowRunEntity,
query: {
limit: params.limit,
- orderBy: [
- { field: 'created', order: Order.DESC },
- { field: 'id', order: Order.DESC },
- ],
+ orderBy: isNil(params.parentWaitpointId)
+ ? [
+ { field: 'created', order: Order.DESC },
+ { field: 'id', order: Order.DESC },
+ ]
+ : [
+ { field: 'dispatchIndex', order: Order.ASC },
+ { field: 'id', order: Order.ASC },
+ ],
afterCursor: decodedCursor.nextCursor,
beforeCursor: decodedCursor.previousCursor,
},
@@ -62,7 +67,13 @@ export const flowRunService = (log: FastifyBaseLogger) => ({
if (!isNil(params.environment)) {
whereClause.environment = params.environment
}
- let query = queryBuilderForFlowRun(flowRunRepo()).where(whereClause)
+ let query = queryBuilderForFlowRun(flowRunRepo())
+ .where(whereClause)
+ .andWhere({ parentWaitpointId: params.parentWaitpointId ?? IsNull() })
+
+ if (!isNil(params.parentWaitpointId)) {
+ query = query.andWhere({ dispatchIndex: Not(IsNull()) })
+ }
if (!params.includeArchived) {
query = query.andWhere({
@@ -120,6 +131,14 @@ export const flowRunService = (log: FastifyBaseLogger) => ({
})
log.info({ flowRun: { id: flowRunId }, flow: { id: oldFlowRun.flowId }, strategy }, 'Flow run retry initiated')
+ if (!isNil(oldFlowRun.dispatchIndex) || !isNil(oldFlowRun.parentWaitpointId)) {
+ const message = `The run ${flowRunId} started from a step inside its flow rather than from the trigger, so it cannot be retried. Retry the run that dispatched it instead.`
+ throw new ActivepiecesError({
+ code: ErrorCode.VALIDATION,
+ params: { message },
+ }, message)
+ }
+
const project = await projectService(log).getOneOrThrow(oldFlowRun.projectId)
const retentionDays = getEffectiveExecutionDataRetentionDays(project.executionDataRetentionDays)
if (
@@ -468,6 +487,7 @@ export const flowRunService = (log: FastifyBaseLogger) => ({
projectId: params.projectId,
environment: RunEnvironment.PRODUCTION,
archivedAt: IsNull(),
+ parentWaitpointId: IsNull(),
})
.groupBy('flow_run.status')
@@ -556,14 +576,20 @@ async function getAllChildRuns(parentRunIds: string[]): Promise<FlowRun[]> {
async function filterFlowRunsAndApplyFilters(
params: FilterFlowRunsAndApplyFiltersParams,
): Promise<FlowRun[]> {
+ const explicitFlowRunIds = params.flowRunIds ?? []
let query = flowRunRepo().createQueryBuilder('flow_run').where({
projectId: params.projectId,
environment: RunEnvironment.PRODUCTION,
})
- if (!isNil(params.flowRunIds) && params.flowRunIds.length > 0) {
+ if (explicitFlowRunIds.length > 0) {
+ query = query.andWhere({
+ id: In(explicitFlowRunIds),
+ })
+ }
+ else {
query = query.andWhere({
- id: In(params.flowRunIds),
+ parentWaitpointId: IsNull(),
})
}
@@ -642,6 +668,7 @@ export async function addToQueue(params: AddToQueueParams, log: FastifyBaseLogge
stepNameToTest: params.flowRun.stepNameToTest ?? undefined,
sampleData: params.sampleData,
logsFileId,
+ parentWaitpointId: params.flowRun.parentWaitpointId ?? undefined,
}
const data: ExecuteFlowJobData = params.executionType === ExecutionType.RESUME
? {
@@ -681,8 +708,8 @@ export async function findFlowRunOrThrow(flowRunId: FlowRunId): Promise<FlowRun>
function queryBuilderForFlowRun(repo: Repository<FlowRun>): SelectQueryBuilder<FlowRun> {
return repo.createQueryBuilder('flow_run')
- .leftJoinAndSelect('flow_run.flowVersion', 'flowVersion')
- .addSelect(['"flowVersion"."displayName"'])
+ .leftJoin('flow_run.flowVersion', 'flowVersion')
+ .addSelect(['flowVersion.id', 'flowVersion.displayName'])
}
async function resolveStepOutput({ step, flowRun, log }: ResolveStepOutputParams): Promise<unknown> {
@@ -822,6 +849,7 @@ type ListParams = {
flowRunIds?: FlowRunId[]
includeArchived?: boolean
environment?: RunEnvironment
+ parentWaitpointId?: ApId
}
type GetOneParams = {
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-side-effects.ts b/packages/server/api/src/app/flows/flow-run/flow-run-side-effects.ts
index cefe5f747a0..8a975e4dbe5 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-side-effects.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-side-effects.ts
@@ -16,7 +16,7 @@ export const flowRunSideEffects = (log: FastifyBaseLogger) => ({
})) {
return
}
- await waitpointService(log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: flowRun.projectId })
+ await waitpointService(log).deleteForFinishedRun({ flowRun })
await flowRunHooks(log).onFinish(flowRun)
applicationEvents(log).sendWorkerEvent({
projectId: flowRun.projectId,
diff --git a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
index f76b6b24dca..ea4363aaf1e 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
@@ -1,5 +1,5 @@
import { apId, assertNotNullOrUndefined, isNil, sanitizeObjectForPostgresql, spreadIfDefined } from '@activepieces/core-utils'
-import { FlowRun, FlowRunStatus, isFlowRunStateTerminal, RunTimeline } from '@activepieces/shared'
+import { BarrierSignalStatus, FlowRun, FlowRunStatus, isFlowRunStateTerminal, RunTimeline } from '@activepieces/shared'
import { Queue, Worker } from 'bullmq'
import { FastifyBaseLogger } from 'fastify'
import { distributedLock, distributedStore, redisConnections } from '../../database/redis-connections'
@@ -8,6 +8,7 @@ import { exceptionHandler } from '../../helper/exception-handler'
import { system } from '../../helper/system/system'
import { AppSystemProp } from '../../helper/system/system-props'
import { projectService } from '../../project/project-service'
+import { barrierService } from '../../waitpoints/barrier-service'
import { resumeService } from '../../waitpoints/resume-service'
import { waitpointService } from '../../waitpoints/waitpoint-service'
import { legacyRedisMetadataKey, QueueName, redisMetadataKey, runsMetadataDeduplicationId, RunsMetadataJobData, RunsMetadataQueueConfig, runsMetadataQueueFactory, RunsMetadataUpsertData } from '../../workers/job'
@@ -73,6 +74,7 @@ export const runsMetadataQueue = (log: FastifyBaseLogger) => ({
...spreadIfDefined('failedStep', runMetadata.failedStep),
...spreadIfDefined('stepNameToTest', runMetadata.stepNameToTest),
...spreadIfDefined('parentRunId', runMetadata.parentRunId),
+ ...spreadIfDefined('parentWaitpointId', runMetadata.parentWaitpointId),
...spreadIfDefined('failParentOnFailure', runMetadata.failParentOnFailure),
...spreadIfDefined('logsFileId', runMetadata.logsFileId),
...spreadIfDefined('updated', runMetadata.updated),
@@ -118,7 +120,15 @@ export const runsMetadataQueue = (log: FastifyBaseLogger) => ({
failParentOnFailure: savedFlowRun.failParentOnFailure,
willRetry: runMetadata.willRetry,
})
- if (shouldMarkParentAsFailed) {
+ if (!isNil(savedFlowRun.parentWaitpointId) && isFlowRunStateTerminal({ status: savedFlowRun.status, ignoreInternalError: runMetadata.willRetry === true })) {
+ await barrierService(log).receiveSignal({
+ refId: savedFlowRun.id,
+ projectId: savedFlowRun.projectId,
+ status: toSignalStatus(savedFlowRun.status),
+ result: { runId: savedFlowRun.id, status: savedFlowRun.status },
+ })
+ }
+ else if (shouldMarkParentAsFailed) {
await markParentRunAsFailed({
parentRunId,
childRunId: savedFlowRun.id,
@@ -290,6 +300,16 @@ export async function markParentRunAsFailed({
}
}
+function toSignalStatus(status: FlowRunStatus): BarrierSignalStatus {
+ if (status === FlowRunStatus.SUCCEEDED) {
+ return BarrierSignalStatus.SUCCEEDED
+ }
+ if (status === FlowRunStatus.CANCELED) {
+ return BarrierSignalStatus.CANCELED
+ }
+ return BarrierSignalStatus.FAILED
+}
+
type ChildRunFailsParentParams = {
status: FlowRunStatus
failParentOnFailure: boolean
diff --git a/packages/server/api/src/app/waitpoints/barrier-queue.ts b/packages/server/api/src/app/waitpoints/barrier-queue.ts
index a920e488912..59354d1ddfe 100644
--- a/packages/server/api/src/app/waitpoints/barrier-queue.ts
+++ b/packages/server/api/src/app/waitpoints/barrier-queue.ts
@@ -25,7 +25,7 @@ export const barrierQueue = (log: FastifyBaseLogger) => ({
const jobLogger = createLogger({
event: 'barrier.job',
job: { id: job.id, type: job.name },
- fanIn: { barrierId: job.data.barrierId },
+ waitpoint: { id: job.data.barrierId },
})
return wideEvent.run({
logger: jobLogger,
diff --git a/packages/server/api/src/app/waitpoints/barrier-service.ts b/packages/server/api/src/app/waitpoints/barrier-service.ts
index 205b920af08..5ed9cfabbbc 100644
--- a/packages/server/api/src/app/waitpoints/barrier-service.ts
+++ b/packages/server/api/src/app/waitpoints/barrier-service.ts
@@ -149,8 +149,8 @@ export const barrierService = (log: FastifyBaseLogger) => ({
return null
}
wideEvent.set({
- fanIn: {
- barrierId: barrier.id,
+ waitpoint: {
+ id: barrier.id,
signalCount: summary.total,
releaseReason,
stillRunning: summary.stillRunning,
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-service.ts b/packages/server/api/src/app/waitpoints/waitpoint-service.ts
index 50f7745643a..6536da15d55 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-service.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-service.ts
@@ -1,8 +1,8 @@
import { ActivepiecesError, apId, ErrorCode, isNil } from '@activepieces/core-utils'
-import { FlowRunStatus, PauseType } from '@activepieces/shared'
+import { FlowRun, FlowRunStatus, isFailedState, PauseType } from '@activepieces/shared'
import dayjs from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
-import { EntityManager, Not } from 'typeorm'
+import { EntityManager, FindOptionsWhere, In, Not } from 'typeorm'
import { z } from 'zod'
import { repoFactory } from '../core/db/repo-factory'
import { transaction } from '../core/db/transaction'
@@ -214,18 +214,45 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
},
async deleteByFlowRunId({ flowRunId, projectId }: DeleteByFlowRunIdParams): Promise<void> {
- const result = await waitpointRepo()
- .createQueryBuilder()
- .delete()
- .where({ flowRunId, projectId })
- .returning(['id', 'resumeDateTime'])
- .execute()
- const timed = DeletedWaitpointRows.parse(result.raw ?? []).filter((waitpoint) => !isNil(waitpoint.resumeDateTime))
- await Promise.all(timed.map((waitpoint) => waitpointTimeoutJob.remove({ waitpointId: waitpoint.id, flowRunId, log })))
+ await deleteWaitpointsWithTimers({ where: { flowRunId, projectId }, flowRunId, log })
log.info({ flowRun: { id: flowRunId } }, '[waitpointService#deleteByFlowRunId] Waitpoint deleted')
},
+
+ async deleteForFinishedRun({ flowRun }: DeleteForFinishedRunParams): Promise<void> {
+ const failedStepName = flowRun.failedStep?.name
+ if (!isFailedState(flowRun.status) || isNil(failedStepName)) {
+ await this.deleteByFlowRunId({ flowRunId: flowRun.id, projectId: flowRun.projectId })
+ return
+ }
+ const waitpoints = await waitpointRepo().find({
+ select: { id: true, stepName: true, type: true },
+ where: { flowRunId: flowRun.id, projectId: flowRun.projectId },
+ })
+ const discardedIds = waitpoints
+ .filter((waitpoint) => waitpoint.type === PauseType.BARRIER || stepNameOfWaitpointKey({ waitpointKey: waitpoint.stepName }) === failedStepName)
+ .map((waitpoint) => waitpoint.id)
+ if (discardedIds.length > 0) {
+ await deleteWaitpointsWithTimers({ where: { id: In(discardedIds), flowRunId: flowRun.id, projectId: flowRun.projectId }, flowRunId: flowRun.id, log })
+ }
+ log.info({ flowRun: { id: flowRun.id } }, '[waitpointService#deleteForFinishedRun] Deleted the failed step\'s waitpoints and any barrier, kept the waitpoints of the steps that finished')
+ },
})
+async function deleteWaitpointsWithTimers({ where, flowRunId, log }: DeleteWaitpointsWithTimersParams): Promise<void> {
+ const result = await waitpointRepo()
+ .createQueryBuilder()
+ .delete()
+ .where(where)
+ .returning(['id', 'resumeDateTime'])
+ .execute()
+ const timed = DeletedWaitpointRows.parse(result.raw ?? []).filter((waitpoint) => !isNil(waitpoint.resumeDateTime))
+ await Promise.all(timed.map((waitpoint) => waitpointTimeoutJob.remove({ waitpointId: waitpoint.id, flowRunId, log })))
+}
+
+function stepNameOfWaitpointKey({ waitpointKey }: { waitpointKey: string }): string {
+ return waitpointKey.split('/').at(-1) ?? waitpointKey
+}
+
function clampWaitpointResumeDeadline({ requested, type, flowRunCreated, flowRunId }: ClampWaitpointResumeDeadlineParams): string | undefined {
const pauseTimeoutDays = system.getNumberOrThrow(AppSystemProp.PAUSED_FLOW_TIMEOUT_DAYS)
const runDeadline = dayjs(flowRunCreated).add(pauseTimeoutDays, 'day')
@@ -260,6 +287,16 @@ type DeleteByFlowRunIdParams = {
projectId: string
}
+type DeleteForFinishedRunParams = {
+ flowRun: FlowRun
+}
+
+type DeleteWaitpointsWithTimersParams = {
+ where: FindOptionsWhere<Waitpoint>
+ flowRunId: string
+ log: FastifyBaseLogger
+}
+
type HasPendingBarrierParams = {
flowRunId: string
projectId: string
diff --git a/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts b/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts
index 6ccc6fbee15..f4e6fcb7082 100644
--- a/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts
+++ b/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts
@@ -73,7 +73,7 @@ export const runsMetadataQueueFactory = ({
const RUNS_METADATA_UPSERT_KEYS: (keyof RunsMetadataUpsertData)[] = [
'id', 'projectId', 'created', 'flowId', 'flowVersionId', 'environment',
'triggeredBy', 'startTime', 'finishTime', 'status', 'tags',
- 'failedStep', 'stepNameToTest', 'parentRunId', 'failParentOnFailure',
+ 'failedStep', 'stepNameToTest', 'parentRunId', 'parentWaitpointId', 'failParentOnFailure',
'logsFileId', 'updated', 'stepsCount', 'requestId',
'provisionMs', 'bootMs', 'runMs', 'willRetry',
]
@@ -123,6 +123,7 @@ export type RunsMetadataUpsertData = {
failedStep?: FailedStep
stepNameToTest?: string
parentRunId?: string
+ parentWaitpointId?: string
failParentOnFailure?: boolean
logsFileId?: string | null
updated?: string
diff --git a/packages/server/api/test/helpers/db.ts b/packages/server/api/test/helpers/db.ts
index e88784f3907..329c92381a7 100644
--- a/packages/server/api/test/helpers/db.ts
+++ b/packages/server/api/test/helpers/db.ts
@@ -10,7 +10,7 @@ export const db = {
return databaseConnection().getRepository(entity).update(id, data)
},
- delete(entity: string, id: string): Promise<unknown> {
+ delete(entity: string, id: string | string[]): Promise<unknown> {
return databaseConnection().getRepository(entity).delete(id)
},
diff --git a/packages/server/api/test/helpers/mocks/index.ts b/packages/server/api/test/helpers/mocks/index.ts
index de6203ab615..c8ccbc90bd9 100644
--- a/packages/server/api/test/helpers/mocks/index.ts
+++ b/packages/server/api/test/helpers/mocks/index.ts
@@ -396,6 +396,7 @@ export const createMockFlowRun = (flowRun?: Partial<FlowRun>): FlowRun => {
steps: {},
failParentOnFailure: flowRun?.failParentOnFailure ?? false,
parentRunId: flowRun?.parentRunId ?? undefined,
+ parentWaitpointId: flowRun?.parentWaitpointId ?? undefined,
flowVersionId: flowRun?.flowVersionId ?? apId(),
flowVersion: flowRun?.flowVersion,
logsFileId: flowRun?.logsFileId ?? null,
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/barrier-children-visibility.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/barrier-children-visibility.test.ts
new file mode 100644
index 00000000000..69309f805af
--- /dev/null
+++ b/packages/server/api/test/integration/ce/flows/flow-run/barrier-children-visibility.test.ts
@@ -0,0 +1,300 @@
+import { apId } from '@activepieces/core-utils'
+import { FlowRetryStrategy, FlowRunStatus, FlowStatus, FlowVersionState, RunEnvironment } from '@activepieces/shared'
+import dayjs from 'dayjs'
+import { FastifyInstance } from 'fastify'
+import { platformAnalyticsReportService } from '../../../../../src/app/analytics/platform-analytics-report.service'
+import { databaseConnection } from '../../../../../src/app/database/database-connection'
+import { healthMetricsService } from '../../../../../src/app/health/health-metrics.service'
+import { db } from '../../../../helpers/db'
+import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'
+import { createTestContext, TestContext } from '../../../../helpers/test-context'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'
+
+let app: FastifyInstance
+let ctx: TestContext
+
+beforeAll(async () => {
+ app = await setupTestEnvironment()
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+beforeEach(async () => {
+ ctx = await createTestContext(app)
+})
+
+async function seedFanOut({ childStatuses, parentStatus = FlowRunStatus.PAUSED, created }: {
+ childStatuses: FlowRunStatus[]
+ parentStatus?: FlowRunStatus
+ created?: string
+}) {
+ const flow = createMockFlow({ projectId: ctx.project.id, status: FlowStatus.ENABLED })
+ await db.save('flow', flow)
+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })
+ await db.save('flow_version', flowVersion)
+ await db.save('flow_version', createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.DRAFT }))
+ await db.update('flow', flow.id, { publishedVersionId: flowVersion.id })
+
+ const barrierId = apId()
+ const parent = createMockFlowRun({
+ projectId: ctx.project.id,
+ flowId: flow.id,
+ flowVersionId: flowVersion.id,
+ status: parentStatus,
+ environment: RunEnvironment.PRODUCTION,
+ ...(created ? { created } : {}),
+ })
+ await db.save('flow_run', parent)
+
+ const children = childStatuses.map((status, index) => ({
+ ...createMockFlowRun({
+ projectId: ctx.project.id,
+ flowId: flow.id,
+ flowVersionId: flowVersion.id,
+ status,
+ parentRunId: parent.id,
+ parentWaitpointId: barrierId,
+ environment: RunEnvironment.PRODUCTION,
+ ...(created ? { created } : {}),
+ }),
+ dispatchIndex: index,
+ }))
+ for (const child of children) {
+ await db.save('flow_run', child)
+ }
+
+ return { flow, flowVersion, parent, children, barrierId }
+}
+
+describe('Barrier children on user-facing surfaces', () => {
+ it('lists the parent but none of its children', async () => {
+ const { parent } = await seedFanOut({
+ childStatuses: [FlowRunStatus.SUCCEEDED, FlowRunStatus.SUCCEEDED, FlowRunStatus.FAILED],
+ })
+
+ const response = await ctx.get('/v1/flow-runs', { projectId: ctx.project.id, limit: 100 })
+
+ expect(response.statusCode).toBe(200)
+ expect(response.json().data.map((run: { id: string }) => run.id)).toEqual([parent.id])
+ })
+
+ it('counts by status agree exactly with the rows the list returns', async () => {
+ await seedFanOut({
+ childStatuses: [FlowRunStatus.SUCCEEDED, FlowRunStatus.SUCCEEDED, FlowRunStatus.FAILED],
+ parentStatus: FlowRunStatus.SUCCEEDED,
+ })
+
+ const list = await ctx.get('/v1/flow-runs', { projectId: ctx.project.id, limit: 100 })
+ const counts = await ctx.get('/v1/flow-runs/count-by-status', { projectId: ctx.project.id })
+
+ expect(counts.statusCode).toBe(200)
+ const totalCounted = counts.json().data.reduce((sum: number, row: { count: number }) => sum + row.count, 0)
+ expect(totalCounted).toBe(list.json().data.length)
+ expect(counts.json().data).toEqual([{ status: FlowRunStatus.SUCCEEDED, count: 1 }])
+ })
+
+ it('cannot select a child through a filter-driven bulk retry', async () => {
+ await seedFanOut({
+ childStatuses: [FlowRunStatus.FAILED, FlowRunStatus.FAILED],
+ parentStatus: FlowRunStatus.FAILED,
+ })
+
+ const response = await ctx.post('/v1/flow-runs/retry', {
+ projectId: ctx.project.id,
+ strategy: FlowRetryStrategy.ON_LATEST_VERSION,
+ status: [FlowRunStatus.FAILED],
+ })
+
+ expect(response.statusCode).toBe(200)
+ await waitForRunCountForProject({ projectId: ctx.project.id, expected: 4 })
+ })
+
+ it('excludes children from the analytics report', async () => {
+ const { flow } = await seedFanOut({
+ childStatuses: [FlowRunStatus.SUCCEEDED, FlowRunStatus.SUCCEEDED, FlowRunStatus.SUCCEEDED],
+ parentStatus: FlowRunStatus.SUCCEEDED,
+ })
+
+ const report = await platformAnalyticsReportService(app.log).refreshReport(ctx.platform.id)
+
+ const runsForFlow = report.runs.filter((run) => run.flowId === flow.id)
+ expect(runsForFlow.reduce((sum, run) => sum + run.runs, 0)).toBe(1)
+ })
+})
+
+describe('Barrier children on ops surfaces', () => {
+ it('keeps counting children in the platform health metrics', async () => {
+ await seedFanOut({
+ childStatuses: [FlowRunStatus.SUCCEEDED, FlowRunStatus.SUCCEEDED, FlowRunStatus.SUCCEEDED],
+ parentStatus: FlowRunStatus.SUCCEEDED,
+ })
+
+ const report = await healthMetricsService(app.log).getRunMetrics(ctx.platform.id, {
+ createdAfter: dayjs().subtract(1, 'day').toISOString(),
+ createdBefore: dayjs().add(1, 'day').toISOString(),
+ })
+
+ expect(report.summary.completed).toBe(4)
+ })
+})
+
+describe('Reaching a barrier child on request', () => {
+ it('returns a child by id under the ordinary project read permission', async () => {
+ const { children } = await seedFanOut({ childStatuses: [FlowRunStatus.SUCCEEDED] })
+
+ const response = await ctx.get(`/v1/flow-runs/${children[0].id}`)
+
+ expect(response.statusCode).toBe(200)
+ expect(response.json().id).toBe(children[0].id)
+ expect(response.json().steps).toEqual({})
+ })
+
+ it('returns a barrier\'s children in dispatch order when asked for them explicitly', async () => {
+ const { children, barrierId } = await seedFanOut({
+ childStatuses: [FlowRunStatus.SUCCEEDED, FlowRunStatus.FAILED, FlowRunStatus.SUCCEEDED],
+ })
+
+ const response = await ctx.get('/v1/flow-runs', {
+ projectId: ctx.project.id,
+ parentWaitpointId: barrierId,
+ limit: 100,
+ })
+
+ expect(response.statusCode).toBe(200)
+ expect(response.json().data.map((run: { id: string }) => run.id)).toEqual(children.map((child) => child.id))
+ })
+
+ it('pages through a barrier\'s children in dispatch order', async () => {
+ const { children, barrierId } = await seedFanOut({
+ childStatuses: Array.from({ length: 5 }, () => FlowRunStatus.SUCCEEDED),
+ })
+
+ const firstPage = await ctx.get('/v1/flow-runs', {
+ projectId: ctx.project.id,
+ parentWaitpointId: barrierId,
+ limit: 2,
+ })
+ const secondPage = await ctx.get('/v1/flow-runs', {
+ projectId: ctx.project.id,
+ parentWaitpointId: barrierId,
+ limit: 2,
+ cursor: firstPage.json().next,
+ })
+
+ expect(firstPage.json().data.map((run: { id: string }) => run.id)).toEqual(children.slice(0, 2).map((child) => child.id))
+ expect(secondPage.json().data.map((run: { id: string }) => run.id)).toEqual(children.slice(2, 4).map((child) => child.id))
+ })
+
+ it('leaves a child with no dispatch index out of the dispatch-ordered listing rather than stalling the cursor', async () => {
+ const { children, barrierId } = await seedFanOut({
+ childStatuses: [FlowRunStatus.SUCCEEDED, FlowRunStatus.SUCCEEDED],
+ })
+ await databaseConnection().query('UPDATE flow_run SET "dispatchIndex" = NULL WHERE id = ANY($1)', [children.map((child) => child.id)])
+
+ const firstPage = await ctx.get('/v1/flow-runs', {
+ projectId: ctx.project.id,
+ parentWaitpointId: barrierId,
+ limit: 1,
+ })
+ const secondPage = await ctx.get('/v1/flow-runs', {
+ projectId: ctx.project.id,
+ parentWaitpointId: barrierId,
+ limit: 1,
+ cursor: firstPage.json().next,
+ })
+
+ expect(firstPage.json().data).toEqual([])
+ expect(secondPage.json().data).toEqual([])
+ })
+
+ it('never returns another barrier\'s children', async () => {
+ const { barrierId } = await seedFanOut({ childStatuses: [FlowRunStatus.SUCCEEDED] })
+ await seedFanOut({ childStatuses: [FlowRunStatus.SUCCEEDED, FlowRunStatus.SUCCEEDED] })
+
+ const response = await ctx.get('/v1/flow-runs', {
+ projectId: ctx.project.id,
+ parentWaitpointId: barrierId,
+ limit: 100,
+ })
+
+ expect(response.json().data).toHaveLength(1)
+ })
+})
+
+describe('Barrier children named by id', () => {
+ it('cancels a child the caller names by id', async () => {
+ const { children } = await seedFanOut({
+ childStatuses: [FlowRunStatus.QUEUED],
+ })
+
+ const response = await ctx.post('/v1/flow-runs/cancel', {
+ projectId: ctx.project.id,
+ flowRunIds: [children[0].id],
+ })
+
+ expect(response.statusCode).toBe(200)
+ await waitForRunStatus({ flowRunId: children[0].id, expected: FlowRunStatus.CANCELED })
+ })
+
+ it('does not select a child through a filter-driven cancel', async () => {
+ const { flow, children } = await seedFanOut({
+ childStatuses: [FlowRunStatus.QUEUED, FlowRunStatus.QUEUED],
+ parentStatus: FlowRunStatus.SUCCEEDED,
+ })
+
+ const response = await ctx.post('/v1/flow-runs/cancel', {
+ projectId: ctx.project.id,
+ flowId: [flow.id],
+ })
+
+ expect(response.statusCode).toBe(200)
+ for (const child of children) {
+ const row = await db.findOneByOrFail<{ id: string, status: string }>('flow_run', { id: child.id })
+ expect(row.status).toBe(FlowRunStatus.QUEUED)
+ }
+ })
+
+ it('reports an error instead of silence when a bulk retry names a child by id', async () => {
+ const { children } = await seedFanOut({
+ childStatuses: [FlowRunStatus.FAILED],
+ parentStatus: FlowRunStatus.FAILED,
+ })
+
+ const response = await ctx.post('/v1/flow-runs/retry', {
+ projectId: ctx.project.id,
+ strategy: FlowRetryStrategy.ON_LATEST_VERSION,
+ flowRunIds: [children[0].id],
+ })
+
+ expect(response.statusCode).toBe(200)
+ const body = response.json()
+ expect(body).toHaveLength(1)
+ expect(body[0].id).toBe(children[0].id)
+ expect(body[0].error.errorMessage).toContain('started from a step inside its flow')
+ await waitForRunCountForProject({ projectId: ctx.project.id, expected: 2 })
+ })
+})
+
+async function waitForRunStatus({ flowRunId, expected }: { flowRunId: string, expected: FlowRunStatus }): Promise<void> {
+ const read = async () => db.findOneByOrFail<{ id: string, status: string }>('flow_run', { id: flowRunId })
+ const start = Date.now()
+ let last = await read()
+ while (last.status !== expected && Date.now() - start < 10_000) {
+ await new Promise((resolve) => setTimeout(resolve, 100))
+ last = await read()
+ }
+ expect(last.status).toBe(expected)
+}
+
+async function waitForRunCountForProject({ projectId, expected }: { projectId: string, expected: number }): Promise<void> {
+ const read = async () => databaseConnection().getRepository('flow_run').count({ where: { projectId } })
+ const start = Date.now()
+ let last = await read()
+ while (last !== expected && Date.now() - start < 10_000) {
+ await new Promise((resolve) => setTimeout(resolve, 100))
+ last = await read()
+ }
+ expect(last).toBe(expected)
+}
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/list-flow-runs.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/list-flow-runs.test.ts
index 8956d9131af..2e5d5dc594f 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/list-flow-runs.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/list-flow-runs.test.ts
@@ -1,5 +1,9 @@
+import { FlowRunStatus, FlowVersionState, RunEnvironment } from '@activepieces/shared'
import { FastifyInstance } from 'fastify'
+import { flowRunService } from '../../../../../src/app/flows/flow-run/flow-run-service'
+import { db } from '../../../../helpers/db'
import { describeWithAuth } from '../../../../helpers/describe-with-auth'
+import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'
import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'
let app: FastifyInstance | null = null
@@ -12,6 +16,28 @@ afterAll(async () => {
await teardownTestEnvironment()
})
+async function createProductionRun(projectId: string) {
+ const flow = createMockFlow({ projectId })
+ await db.save('flow', flow)
+
+ const flowVersion = createMockFlowVersion({
+ flowId: flow.id,
+ state: FlowVersionState.LOCKED,
+ })
+ await db.save('flow_version', flowVersion)
+
+ const flowRun = createMockFlowRun({
+ projectId,
+ flowId: flow.id,
+ flowVersionId: flowVersion.id,
+ status: FlowRunStatus.SUCCEEDED,
+ environment: RunEnvironment.PRODUCTION,
+ })
+ await db.save('flow_run', flowRun)
+
+ return { flow, flowVersion, flowRun }
+}
+
describeWithAuth('List flow runs endpoint', () => app!, (setup) => {
it('should return empty list with correct structure', async () => {
const ctx = await setup()
@@ -25,4 +51,39 @@ describeWithAuth('List flow runs endpoint', () => app!, (setup) => {
expect(body.data).toEqual([])
expect(body.cursor).toBeUndefined()
})
+
+ it('should return the flow version display name', async () => {
+ const ctx = await setup()
+ const { flowVersion } = await createProductionRun(ctx.project.id)
+
+ const response = await ctx.get('/v1/flow-runs', {
+ projectId: ctx.project.id,
+ })
+
+ expect(response?.statusCode).toBe(200)
+ const body = response?.json()
+ expect(body.data).toHaveLength(1)
+ expect(body.data[0].flowVersion.displayName).toBe(flowVersion.displayName)
+ })
+
+ it('should not read the flow definition into listed runs', async () => {
+ const ctx = await setup()
+ const { flowVersion } = await createProductionRun(ctx.project.id)
+
+ const { data } = await flowRunService(app!.log).list({
+ projectId: ctx.project.id,
+ flowId: undefined,
+ status: undefined,
+ cursor: null,
+ limit: 10,
+ environment: RunEnvironment.PRODUCTION,
+ })
+
+ expect(data).toHaveLength(1)
+ expect(data[0].flowVersion?.displayName).toBe(flowVersion.displayName)
+ expect(data[0].flowVersion).not.toHaveProperty('trigger')
+ expect(data[0].flowVersion).not.toHaveProperty('notes')
+ expect(data[0].flowVersion).not.toHaveProperty('backupFiles')
+ expect(data[0]).not.toHaveProperty('flowVersion_trigger')
+ })
})
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/retry-flow-run.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/retry-flow-run.test.ts
index 14560ea22e4..ac64a144437 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/retry-flow-run.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/retry-flow-run.test.ts
@@ -1,3 +1,4 @@
+import { apId } from '@activepieces/core-utils'
import { FileCompression, FileType, FlowRetryStrategy, FlowRunStatus, FlowTriggerType, FlowVersionState, RunEnvironment, StepOutputStatus, StepOutputType } from '@activepieces/shared'
import { FastifyInstance } from 'fastify'
import { fileService } from '../../../../../src/app/file/file.service'
@@ -105,6 +106,71 @@ describe('Retry flow run', () => {
expect(body.flowId).toBe(flowRun.flowId)
})
+ it('should keep parentRunId when retrying a subflow run on latest version', async () => {
+ const { flowRun } = await createFailedFlowRun({
+ projectId: ctx.project.id,
+ })
+ const parentRunId = apId()
+ await db.update('flow_run', flowRun.id, { parentRunId })
+
+ const response = await ctx.post(`/v1/flow-runs/${flowRun.id}/retry`, {
+ strategy: FlowRetryStrategy.ON_LATEST_VERSION,
+ projectId: ctx.project.id,
+ })
+
+ expect(response.statusCode).toBe(200)
+ const retried = response.json()
+ expect(retried.id).not.toBe(flowRun.id)
+ expect(retried.parentRunId).toBe(parentRunId)
+ expect(retried.parentWaitpointId).toBeUndefined()
+ })
+
+ it('should refuse to retry a barrier child on either strategy', async () => {
+ const { flowRun } = await createFailedFlowRun({
+ projectId: ctx.project.id,
+ })
+ await db.update('flow_run', flowRun.id, { parentWaitpointId: apId(), dispatchIndex: 3 })
+
+ const fromFailedStep = await ctx.post(`/v1/flow-runs/${flowRun.id}/retry`, {
+ strategy: FlowRetryStrategy.FROM_FAILED_STEP,
+ projectId: ctx.project.id,
+ })
+ expect(fromFailedStep.statusCode).toBe(409)
+
+ const onLatestVersion = await ctx.post(`/v1/flow-runs/${flowRun.id}/retry`, {
+ strategy: FlowRetryStrategy.ON_LATEST_VERSION,
+ projectId: ctx.project.id,
+ })
+ expect(onLatestVersion.statusCode).toBe(409)
+
+ const notRetried = await db.findOneByOrFail<{ id: string, status: string, parentWaitpointId: string | null }>('flow_run', { id: flowRun.id })
+ expect(notRetried.status).toBe(FlowRunStatus.FAILED)
+ expect(notRetried.parentWaitpointId).not.toBeNull()
+ })
+
+ it('should refuse to retry a run that started mid-graph', async () => {
+ const { flowRun } = await createFailedFlowRun({
+ projectId: ctx.project.id,
+ })
+ await db.update('flow_run', flowRun.id, { dispatchIndex: 3 })
+
+ const fromFailedStep = await ctx.post(`/v1/flow-runs/${flowRun.id}/retry`, {
+ strategy: FlowRetryStrategy.FROM_FAILED_STEP,
+ projectId: ctx.project.id,
+ })
+ expect(fromFailedStep.statusCode).toBe(409)
+ expect(fromFailedStep.json().params.message).toContain('started from a step inside its flow')
+
+ const onLatestVersion = await ctx.post(`/v1/flow-runs/${flowRun.id}/retry`, {
+ strategy: FlowRetryStrategy.ON_LATEST_VERSION,
+ projectId: ctx.project.id,
+ })
+ expect(onLatestVersion.statusCode).toBe(409)
+
+ const notRetried = await db.findOneByOrFail<{ id: string, status: string }>('flow_run', { id: flowRun.id })
+ expect(notRetried.status).toBe(FlowRunStatus.FAILED)
+ })
+
it('should return 400 for invalid flow run id', async () => {
const response = await ctx.post('/v1/flow-runs/non-existent-id/retry', {
strategy: FlowRetryStrategy.FROM_FAILED_STEP,
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
index 7f6289fd0c8..8fbc5b9ed8b 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
@@ -3,6 +3,7 @@ import { FlowRunStatus, FlowVersionState, PauseType, RunEnvironment } from '@act
import { FastifyInstance } from 'fastify'
import { systemJobIds } from '../../../../../src/app/helper/system-jobs/common'
import * as systemJobModule from '../../../../../src/app/helper/system-jobs/system-job'
+import { barrierService } from '../../../../../src/app/waitpoints/barrier-service'
import { handleResumeDelayWaitpoint } from '../../../../../src/app/waitpoints/resume-delay-handler'
import { waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
import { WaitpointStatus } from '../../../../../src/app/waitpoints/waitpoint-types'
@@ -546,6 +547,103 @@ describe('Waitpoint service', () => {
})
})
+ describe('deleteForFinishedRun', () => {
+ it('keeps the link a finished step handed out when the run fails, so a retry from the failed step can still use it', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+ const { waitpoint: handedOut } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'step_1',
+ type: PauseType.WEBHOOK,
+ })
+ const { waitpoint: failedStepWaitpoint } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'step_3',
+ type: PauseType.WEBHOOK,
+ })
+
+ await waitpointService(app.log).deleteForFinishedRun({
+ flowRun: { ...flowRun, status: FlowRunStatus.FAILED, failedStep: { name: 'step_3', displayName: 'Wait for Approval', message: 'failed' } },
+ })
+
+ expect(await db.findOneBy('waitpoint', { id: handedOut.id })).not.toBeNull()
+ expect(await db.findOneBy('waitpoint', { id: failedStepWaitpoint.id })).toBeNull()
+ expect(await originalSystemJobsSchedule(app.log).getJob(systemJobIds.resumeDelay({ waitpointId: handedOut.id }))).toBeDefined()
+ expect(await originalSystemJobsSchedule(app.log).getJob(systemJobIds.resumeDelay({ waitpointId: failedStepWaitpoint.id }))).toBeUndefined()
+ })
+
+ it('matches the failed step in every loop iteration, not only at the top level', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+ const { waitpoint: handedOut } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'step_1:0/step_2',
+ type: PauseType.WEBHOOK,
+ })
+ const { waitpoint: failedInLoop } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'step_1:1/step_3',
+ type: PauseType.WEBHOOK,
+ })
+
+ await waitpointService(app.log).deleteForFinishedRun({
+ flowRun: { ...flowRun, status: FlowRunStatus.FAILED, failedStep: { name: 'step_3', displayName: 'Ask the AI', message: 'failed' } },
+ })
+
+ expect(await db.findOneBy('waitpoint', { id: handedOut.id })).not.toBeNull()
+ expect(await db.findOneBy('waitpoint', { id: failedInLoop.id })).toBeNull()
+ })
+
+ it('deletes the barrier of a failed run so a retry starts a fresh one', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+ const { barrier } = await barrierService(app.log).create({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ platformId: ctx.platform.id,
+ stepName: 'step_2',
+ version: 'V1',
+ policy: { requiredSuccesses: 1 },
+ signalLabels: ['[email protected]'],
+ })
+
+ await waitpointService(app.log).deleteForFinishedRun({
+ flowRun: { ...flowRun, status: FlowRunStatus.FAILED, failedStep: { name: 'step_3', displayName: 'Send Email', message: 'failed' } },
+ })
+
+ expect(await db.findOneBy('waitpoint', { id: barrier.id })).toBeNull()
+ })
+
+ it('deletes every waitpoint of a run that succeeded', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+ await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'step_1',
+ type: PauseType.WEBHOOK,
+ })
+
+ await waitpointService(app.log).deleteForFinishedRun({ flowRun: { ...flowRun, status: FlowRunStatus.SUCCEEDED } })
+
+ expect(await db.findOneBy('waitpoint', { flowRunId: flowRun.id })).toBeNull()
+ })
+
+ it('deletes every waitpoint of a failed run that names no failed step', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+ await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'step_1',
+ type: PauseType.WEBHOOK,
+ })
+
+ await waitpointService(app.log).deleteForFinishedRun({ flowRun: { ...flowRun, status: FlowRunStatus.TIMEOUT, failedStep: undefined } })
+
+ expect(await db.findOneBy('waitpoint', { flowRunId: flowRun.id })).toBeNull()
+ })
+ })
+
describe('findUndeliveredCompletedWaitpoint', () => {
it('should return null when no waitpoint exists', async () => {
const result = await waitpointService(app.log).findUndeliveredCompletedWaitpoint({ flowRunId: apId(), projectId: ctx.project.id })
diff --git a/packages/server/api/test/integration/ee/billing-usage-report/daily-executions.test.ts b/packages/server/api/test/integration/ee/billing-usage-report/daily-executions.test.ts
new file mode 100644
index 00000000000..dac74dfa5e9
--- /dev/null
+++ b/packages/server/api/test/integration/ee/billing-usage-report/daily-executions.test.ts
@@ -0,0 +1,92 @@
+import { apId } from '@activepieces/core-utils'
+import { FlowRunStatus, FlowVersionState, RunEnvironment } from '@activepieces/shared'
+import { FastifyInstance } from 'fastify'
+import { vi } from 'vitest'
+import { databaseConnection } from '../../../../src/app/database/database-connection'
+import { licenseKeyUsageReportService } from '../../../../src/app/ee/license-key-usage-report/license-key-usage-report-service'
+import { db } from '../../../helpers/db'
+import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../helpers/mocks'
+import { createTestContext, TestContext } from '../../../helpers/test-context'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+
+const { mockCaptureLicenseKeyEvent } = vi.hoisted(() => ({ mockCaptureLicenseKeyEvent: vi.fn() }))
+
+vi.mock('../../../../src/app/helper/telemetry.utils', async (importOriginal) => {
+ const original = await importOriginal<typeof import('../../../../src/app/helper/telemetry.utils')>()
+ return {
+ ...original,
+ captureLicenseKeyEvent: mockCaptureLicenseKeyEvent,
+ flushLicenseKeyPostHogEvents: vi.fn().mockResolvedValue(undefined),
+ }
+})
+
+vi.mock('../../../../src/app/helper/sleep', () => ({
+ sleep: vi.fn().mockResolvedValue(undefined),
+}))
+
+let app: FastifyInstance
+let ctx: TestContext
+
+beforeAll(async () => {
+ app = await setupTestEnvironment({ fresh: true })
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+beforeEach(async () => {
+ ctx = await createTestContext(app)
+ mockCaptureLicenseKeyEvent.mockClear()
+})
+
+function yesterdayNoonUtc(): Date {
+ const date = new Date()
+ date.setUTCDate(date.getUTCDate() - 1)
+ date.setUTCHours(12, 0, 0, 0)
+ return date
+}
+
+describe('Daily platform execution tracking', () => {
+ it('excludes barrier children from the reported daily execution count', async () => {
+ const flow = createMockFlow({ projectId: ctx.project.id })
+ await db.save('flow', flow)
+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })
+ await db.save('flow_version', flowVersion)
+
+ const runDay = yesterdayNoonUtc()
+ const barrierId = apId()
+ const runs = [undefined, barrierId, barrierId, barrierId].map((parentWaitpointId, index) => ({
+ ...createMockFlowRun({
+ projectId: ctx.project.id,
+ flowId: flow.id,
+ flowVersionId: flowVersion.id,
+ status: FlowRunStatus.SUCCEEDED,
+ environment: RunEnvironment.PRODUCTION,
+ parentWaitpointId,
+ }),
+ dispatchIndex: index === 0 ? null : index - 1,
+ }))
+ for (const run of runs) {
+ await db.save('flow_run', run)
+ await databaseConnection().query('UPDATE flow_run SET created = $1 WHERE id = $2', [runDay.toISOString(), run.id])
+ }
+
+ await licenseKeyUsageReportService(app.log).reportAllPlatforms()
+
+ const reported = mockCaptureLicenseKeyEvent.mock.calls
+ .map(([event]): CapturedLicenseKeyEvent => event)
+ .find((event) => event.properties.platform_id === ctx.platform.id)
+
+ expect(reported?.properties.daily_executions).toEqual([
+ { date: runDay.toISOString().slice(0, 10), count: 1 },
+ ])
+ })
+})
+
+type CapturedLicenseKeyEvent = {
+ properties: {
+ platform_id: string
+ daily_executions: { date: string, count: number }[]
+ }
+}
diff --git a/packages/server/worker/src/lib/worker.ts b/packages/server/worker/src/lib/worker.ts
index 9e29fc4caf8..56d0c9e0b90 100644
--- a/packages/server/worker/src/lib/worker.ts
+++ b/packages/server/worker/src/lib/worker.ts
@@ -355,6 +355,7 @@ async function executeJob(apiClient: WorkerToApiContract, job: ConsumeJobRequest
...spreadIfDefined('run', isAgentJob && 'runId' in jobData ? { id: jobData.runId } : undefined),
...spreadIfDefined('conversation', 'conversationId' in jobData ? { id: jobData.conversationId } : undefined),
...spreadIfDefined('flowVersion', 'flowVersionId' in jobData ? { id: jobData.flowVersionId } : undefined),
+ ...spreadIfDefined('waitpoint', 'parentWaitpointId' in jobData && jobData.parentWaitpointId != null ? { id: jobData.parentWaitpointId } : undefined),
})
return wideEvent.run({
logger: jobLogger,