Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(waitpoints): give a looped pause its own row, and release barriers on the predicate (#14777)

Othman Abu Ajamieh32413bf656cf
diff --git a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
index c8c9c5e2111..94095a8c881 100644
--- a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
+++ b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
@@ -14,7 +14,7 @@ producer's ordinal), nullable `label` (the human name in the summary) and a smal
Release is an **unconditional floor rule**: a sealed barrier releases once no signal is still `PENDING`. No
configuration can produce a hang; `policy` only ever releases *sooner* — `requiredSuccesses` (K-of-N
approvals) and `releaseOnFirstFailure` (veto). Evaluation is a coalesced `BarrierJobName.EVALUATE` job on a
-dedicated `BARRIER_JOBS` queue, deduplicated on the barrier id, owned by the waitpoints module. Counts are
+dedicated `BARRIER_EVALUATION` queue, deduplicated on the barrier id, owned by the waitpoints module. Counts are
taken once, from a single `GROUP BY status`, and handed to a pure predicate. Signals are deleted on release,
in the same transaction that completes the waitpoint — summary first, then complete, then delete.
@@ -51,7 +51,7 @@ all, purely because child rows appear later than the dispatch that created them.
already returns the ceiling, and a shorter deadline is a `policy` input, so both are known at create.
- **Evaluation must be coalesced, and coalescing must not swallow its own last signal.** At 10 000 signals,
evaluating per signal is ~100M row reads. BullMQ holds a deduplication key while the job is queued *and*
- active, so the handler's **first statement** is `clearEvaluationDeduplication`, and every producer commits
+ active, so the handler's **first statement** is `clearEvaluationDedupKey`, and every producer commits
its signal row **before** enqueuing. Otherwise the final signal lands while the job meant to see it is
already running, its enqueue is dropped as a duplicate, and the barrier waits out its deadline holding a
run that was ready to resume.
@@ -61,6 +61,31 @@ all, purely because child rows appear later than the dispatch that created them.
`shouldReleaseBarrier({ policy, sealed, counts })` takes counts rather than issuing its own queries, so one
`GROUP BY status` replaces up to three round-trips and the policy matrix is unit-testable without a
database.
+- **Identity is path-keyed, and that needed no new column.** The engine sends `loop_1:3/loop_2:0/approval`
+ in the existing `stepName` field, which is only ever an identity key inside `waitpoint-service`. Which
+ fixes a live bug on the way past: a delay or approval **inside a loop** used to reuse one waitpoint row
+ across every iteration, so iteration 2 found iteration 1's COMPLETED row in `createForPause`'s
+ pre-completed check and skipped its pause entirely.
+- **Two per-run waitpoint reads stopped being sound and were fixed here.** `getByFlowRunId` became
+ `findCompletedWaitpointIfRunIsIdle` — **null when any PENDING row exists for the run**, else the newest COMPLETED
+ one — because per-iteration rows mean a run can hold a COMPLETED leftover *and* an open PENDING pause at
+ once, and both callers delete the row they find and enqueue a resume. `findNonFanInByFlowRunId` became
+ `findSubflowWaitpoint` and prefers PENDING, newest first. `findPendingByVersion` is **still** unsound
+ for parallel branches; nothing here creates that case.
+- **The resume guards moved to the entry point rather than becoming a flag.** `resumeFromWaitpoint`
+ **refuses barriers, always** — by addressed waitpoint type, or, on the by-run legacy routes, by "does this
+ run hold any PENDING barrier". Every unscoped route goes through it, so there is no `false` to forget and
+ no parameter to default wrong. What the guard is *not* is structural: `resumeTrusted` and
+ `resumeTrustedWithoutLock` are ordinary public members of `resumeService`, one `if` away from the guard
+ they skip. `flow-runs-queue`'s pre-completed recovery calls the unlocked variant from outside the module,
+ so a module-private `resumeTrusted` would not have worked as written. Treated as acceptable because the
+ reachable surface — the HTTP routes — is closed; if a third caller appears, make the boundary real rather
+ than adding a second convention.
+- **`release` is what happens to a barrier; `resume` is what happens to a run.** The two verbs sit one layer
+ apart — `barrierService.release` closes the barrier, then asks `resumeService` to resume the run — so a
+ `resumeService.releaseBarrier` named the wrong layer and hid that the same method also serves plain `DELAY`
+ and `WEBHOOK` waitpoints, which is exactly what `flow-runs-queue` calls it for. It is `resumeTrusted`:
+ resume, skipping the barrier guard, because the caller is internal rather than an HTTP route.
- **An external actor never addresses the waitpoint.** Their link carries a **signal id** on
`/v1/flow-runs/:id/signals/:signalId/confirm` — in the path, not the query string, because `resumePayload`
is built from `{ body, headers, queryParams }` and persisted with the run. The link must never carry
@@ -76,7 +101,7 @@ all, purely because child rows appear later than the dispatch that created them.
- **The machinery ships able to carry K-of-N approvals; no approval piece is rewired yet.** Per-signal links
and `reasonRequiredOn` exist now because retrofitting them means another migration.
- **`policy` carries no deadline override.** The plan allowed "or the policy's shorter value"; nothing
- produces one, so `resolveDeadline()` is `now + AP_PAUSED_FLOW_TIMEOUT_DAYS` and takes no arguments. Add the
+ produces one, so `defaultBarrierDeadline()` is `now + AP_PAUSED_FLOW_TIMEOUT_DAYS` and takes no arguments. Add the
field when a caller for it exists, not before.
- **The 2 000-character reason bound stays a handler check, not a request schema.** The confirm route is
`app.all` serving an HTML form: a Zod `body` schema would also apply to the GET that renders the page, and
diff --git a/brain/knowledge/decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md b/brain/knowledge/decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md
new file mode 100644
index 00000000000..a6d563b19ed
--- /dev/null
+++ b/brain/knowledge/decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md
@@ -0,0 +1,57 @@
+---
+icon: 🪦
+status: accepted
+---
+
+# A released barrier leaves a tombstone until the run ends
+
+## Decision
+
+Delivering a resume no longer means deleting the waitpoint row. `waitpointService.consume` owns the
+rule: a `BARRIER` is tombstoned at a third status, `CONSUMED`, and every other type is still deleted.
+The tombstone lives until the run reaches a terminal status, where the existing `deleteByFlowRunId`
+sweep clears it. The legacy by-run resume guard (`hasBarrier`) therefore asks whether the run holds a
+barrier row **at all**, in any status, instead of whether one is `PENDING`.
+
+## Context
+
+A barrier's release boundary is predicate-owned: only its own predicate or its deadline may release
+it. The guard enforcing that on the deprecated V0 `/:id/requests/:requestId` routes asked
+`hasPendingBarrier`, which filters `status: PENDING`, so it stopped guarding the moment the barrier
+was released. Two windows followed — between `closeBarrier` committing `COMPLETED` and the trusted
+resume consuming it, and the much wider one after delivery, since a run stays `PAUSED` until the
+engine's next progress upload. In both, a legacy resume was accepted and enqueued a second job with
+a request-controlled payload, under a job id BullMQ would not collapse against the trusted one.
+
+The root cause was that "this resume was delivered" was encoded as *row-absence*, which makes a
+delivered barrier and a pre-waitpoint legacy pause indistinguishable — both are "no rows" — while
+the legacy path may only serve the second.
+
+## Why
+
+Making delivery explicit is what lets the two be told apart, and `waitpoint.status` is
+`character varying` with a TypeScript-only enum, so a third value costs no migration.
+
+Rejected: a `flow_run.pausedWaitpointId` column — a real migration, and `pauseMetadata` was not
+available to reuse (it is deprecated and pending removal). A Redis marker with a TTL — puts a
+security boundary on an expiry and a flush reopens the hole. Deleting the legacy path outright —
+tempting at 4.5 months past deprecation with a 30-day default pause timeout, but a self-hoster
+jumping many versions would silently lose every in-flight approval.
+
+Tombstoning is scoped to barriers rather than all waitpoints because the barrier is the only
+waitpoint whose release boundary is predicate-owned; for `DELAY` / `WEBHOOK` the unguessable link
+*is* the authorization. Widening it would leak `CONSUMED` rows into `findSubflowWaitpoint`'s
+any-status fallback and into `createForPause`'s unique `(flowRunId, stepName)` slot.
+
+## Consequences
+
+A run that has ever held a barrier can never use the legacy by-run resume path again. That is
+correct, not a side effect: that path serves only runs which never had a waitpoint row at all.
+
+`hasPendingBarrier` survives alongside `hasBarrier` and `findUndeliveredCompletedWaitpoint` must keep
+calling the *pending* one — switching it to existence would return null whenever a `CONSUMED` barrier
+exists and hang a later non-barrier delivery on the same run.
+
+The invariant that a row still sitting at `COMPLETED` was never delivered is preserved, now carried
+by an explicit status rather than by absence. Cost is one small row per barrier for the life of the
+run; `barrierService.create` already deletes any non-`PENDING` row occupying its step's slot.
diff --git a/brain/knowledge/decisions/000035-the-barrier-signal-cap-is-a-platform-configuration-row-not-an-env-var.md b/brain/knowledge/decisions/000035-the-barrier-signal-cap-is-a-platform-configuration-row-not-an-env-var.md
new file mode 100644
index 00000000000..b611aba3834
--- /dev/null
+++ b/brain/knowledge/decisions/000035-the-barrier-signal-cap-is-a-platform-configuration-row-not-an-env-var.md
@@ -0,0 +1,78 @@
+---
+status: accepted
+---
+
+# The barrier signal cap is a platform configuration row, not an env var
+
+## Decision
+
+`AP_MAX_BARRIER_SIGNALS` becomes `maxBarrierSignals`, an `integer NOT NULL DEFAULT 10000` column on
+`platform_configuration`, with a `Limits` section on the Configurations page to edit it. The row is
+authoritative and born from the env var, exactly as [000033](000033-platform-configuration-rows-are-authoritative-and-created-on-first-read.md)
+does it. The env var survives only as the birth default for a platform that has no row yet.
+
+Cloud is short-circuited **in the service**: `maxBarrierSignals({ platformId })` returns the system prop
+and ignores the row when `edition === CLOUD`, the same shape `isProductTelemetryEnabled` already uses.
+`platformId` is added to `CreateBarrierParams` so the caller supplies it rather than
+`barrierService.create` resolving it from `projectId`. There is no unlimited value and no nullable
+column.
+
+## Context
+
+The variable is new on `feat/barrier-service` and has never shipped: it is absent from `main`, from the
+published env var table, and from every deploy surface. So this is a choice about where a dial lives
+before release, with no upgrade path to protect, no breaking-change label and no docs migration.
+
+Its only read site is `assertSignalCountWithinLimit` in `waitpoints/barrier-service.ts`, and
+`barrierService.create` has no production caller yet, so the signature is free to change.
+
+## Why
+
+000033 draws a line that this decision deliberately walks up to: configurations hold behaviour,
+`platform_plan` holds entitlements, and a **cap** is the `MAX_RECORDS_PER_TABLE` shape that decision
+keeps off the page, because `POST /v1/platform-configurations` carries no edition guard and a limit
+living there is a self-serve quota bypass the day the page stops being Cloud-hidden.
+
+The cap goes there anyway because the bypass is closed where 000033 says invariants must be closed: in
+the server, on the value. Hiding the page was never the enforcement, and the Cloud short-circuit in the
+service makes the row unreadable as a cap on Cloud regardless of what an admin writes.
+
+**Rejected: `platform_plan`.** It is the correct home for a limit that is sold, and it is CE-readable
+via `platformService.getOneWithPlanOrThrow`. It was rejected because nobody sells barrier width, and a
+plan column would put a self-hoster's own protective cap behind an entitlement they cannot edit.
+
+**Rejected: env var as a `min(row, env)` ceiling.** It preserves an operator guarantee on a
+multi-platform EE install, where a platform admin is not the operator and could raise a cap that
+consumes shared Postgres and Redis. It was rejected as one concept too many for a case that does not
+yet exist, and because it reintroduces the two-sources-of-truth shape 000033 rejected for telemetry.
+This is the variant to revisit if multi-platform installs start mattering here.
+
+## Consequences
+
+- The error message must stop saying "raise `AP_MAX_BARRIER_SIGNALS`". It is read by a project member
+ who may not be a platform admin, so it names the limit and the page without implying the reader can
+ change it.
+- `platformId` on `CreateBarrierParams` means every future caller has to hold platform scope. The
+ waitpoints module already resolves it the other way in `resume-service.ts`
+ (`projectService.getPlatformId`), so the two paths in one module now differ on purpose: the check runs
+ before the transaction that inserts up to 10 000 signal rows, and the caller was cheaper to widen
+ while it has no production implementation.
+- **The number reaches the browser whether or not the builder uses it.** The GET response is the whole
+ `PlatformConfiguration` object, and `TelemetryProvider` wraps the router and already caches that row
+ for every signed-in user, so any builder component can read it with an existing hook and no new
+ endpoint. Nothing is built on that now: no fan-out step exists, and the real fan-out shape dispatches
+ over a runtime array, so a design-time count does not exist for the general case.
+- Because Cloud ignores the row, the raw column is **not** the effective cap. Whoever adds a builder
+ surface must serve a computed effective value rather than recompute the edition rule in the browser,
+ which is how 000033's split-brain bug happened.
+- **No `tryCatch` around the read, because every failure that could reach it is already fatal to
+ `barrierService.create`.** Postgres down kills the transaction and the signal inserts, Redis down kills
+ `enqueueEvaluation`, and the `distributedLock` is taken only on the create path, once per platform for
+ the life of the install. Fail-closed is also the do-nothing option, so there is no code to write.
+- **The `Limits` section is written but not rendered.** `limits-section.tsx` exists and is wired to the
+ same form schema, but `configurations/index.tsx` does not mount it, so no edition shows the field yet.
+ The row, the API and the cap are all live regardless: the value every platform runs on is the one
+ `AP_MAX_BARRIER_SIGNALS` seeded, and an existing platform's row can only be changed through
+ `POST /v1/platform-configurations` until the section is mounted. Rendering it is one import and one
+ line.
+- Postgres only. There is no sqlite migration for `platform_configuration`.
diff --git a/brain/knowledge/engineering/ci-pr-review-hygiene.md b/brain/knowledge/engineering/ci-pr-review-hygiene.md
index 93ae61d793b..f8f33538c53 100644
--- a/brain/knowledge/engineering/ci-pr-review-hygiene.md
+++ b/brain/knowledge/engineering/ci-pr-review-hygiene.md
@@ -44,7 +44,7 @@ Enforcement is the **`Codeowners review` repository ruleset** (active on the def
- **Linting a single server test file OOMs node at its default heap — pass `NODE_OPTIONS=--max-old-space-size=8192`.** `npx eslint packages/server/api/test/.../<file>.test.ts` on one file died with `FATAL ERROR: Reached heap limit` after ~23s at 2GB, because the type-aware config loads the whole `packages/server/api` program regardless of how few files you name. It reads as a broken lint setup, not as a memory ceiling. The same run with an 8GB heap finishes and reports normally.
- **`bun install` on a recent bun adds `"configVersion": 0` to `bun.lock`, which is not on `main`.** It rides along in any commit that touches the lockfile and reads as an unrelated change; drop the line and re-run `bun install --frozen-lockfile` to confirm the lockfile is still consistent without it. **You do not have to run `bun install` to get it** — a bare `npx vitest run <file>` in `packages/web` was enough to rewrite the lockfile with `configVersion` *plus* every workspace version already bumped in the branch, so `git status` after any verification run is worth a glance before committing.
-- **A version bump that merges cleanly can still be wrong — check what `main`'s number *means*, not whether it conflicts.** Two branches bumping the same package to the same number do not conflict, so git takes it silently; but if `main`'s copy of `0.5.0` is another PR's content and yours adds further exports on top, you ship new exports under an already-published version and nothing catches it. Seen merging [#15001](https://github.com/activepieces/activepieces/pull/15001) after the six-providers PR landed: `core-piece-types` and `pieces-framework` auto-merged at `0.5.0` / `0.37.0` and both needed a further bump. Only a *conflicting* version (like `core/shared` `0.140.0` vs `0.141.0`) forces you to think; the clean ones are the dangerous ones. After any merge, re-check every package you bumped against `git show origin/main:<pkg>/package.json`. The reverse also happens: when review makes you *delete* code, the bump it justified can become dead — after acting on review, `git diff origin/main...HEAD -- <pkg>/src` and drop the bump if it is empty. On #15001 two packages ended up byte-identical to `main` while still carrying a bump, which is noise at best and a version collision at worst.
+- **A version bump that merges cleanly can still be wrong — check what `main`'s number *means*, not whether it conflicts.** Two branches bumping the same package to the same number do not conflict, so git takes it silently; but if `main`'s copy of `0.5.0` is another PR's content and yours adds further exports on top, you ship new exports under an already-published version and nothing catches it. Seen merging [#15001](https://github.com/activepieces/activepieces/pull/15001) after the six-providers PR landed: `core-piece-types` and `pieces-framework` auto-merged at `0.5.0` / `0.37.0` and both needed a further bump. Only a *conflicting* version (like `core/shared` `0.140.0` vs `0.141.0`) forces you to think; the clean ones are the dangerous ones. After any merge, re-check every package you bumped against `git show origin/main:<pkg>/package.json`. The reverse also happens: when review makes you *delete* code, the bump it justified can become dead — after acting on review, `git diff origin/main...HEAD -- <pkg>/src` and drop the bump if it is empty. On #15001 two packages ended up byte-identical to `main` while still carrying a bump, which is noise at best and a version collision at worst. **Nothing in CI catches a *missing* bump outside `packages/pieces/`.** `validate-publishable-packages` is the only bump check and it exempts every non-piece package, so a `core/shared` change with no bump goes green on every check. Seen on [#15078](https://github.com/activepieces/activepieces/pull/15078): nine new exports added to `core/shared/src/lib/automation/mcp/mcp-oauth.ts` with `package.json` still at `0.152.0` — the same number `main` had just reached via a *different* PR (#15162), so the collision was invisible and all 17 checks passed. Reviewing a PR that touches `core/shared`, diff the version by hand. `bun.lock` mirrors workspace versions and drifts behind — it read `0.151.0` while `main` was on `0.152.0`, because bumps routinely land without it (#15162 changed `package.json` alone). Sync it with **`bun install --lockfile-only`**, which needs no `node_modules` and so works in a throwaway worktree; on #15078 that produced exactly one line and swept up no unrelated drift. The 103-line sweep noted above is a *pieces* phenomenon — don't assume it for `core/*`, just read the diff before staging.
- **`@activepieces/shared` re-exports from `@activepieces/core-execution`, so a partial rebuild produces phantom "has no exported member" errors in unrelated files.** Rebuilding `core/shared` against a stale `core/execution` dist drops those re-exports, and the API typecheck then fails in `ee/agent/*` on symbols like `GetPersonalizationConfigRequest` — which live in `core/execution/src/lib/workers/worker-contract.ts`, not in shared at all. It reads exactly like a bad merge. The dependency order that actually works is `core/utils` → `core/piece-types` → `core/formula` → `core/execution` → `core/shared` → `server/utils` → `pieces/framework` → `core/ai-providers`; skipping a link silently poisons everything downstream of it. The same staleness makes an editor report missing enum members that exist in the source. **There is no api typecheck "error baseline" — rebuild first, then trust it.** A stale `core/shared/dist` reliably yields a handful of `ee/agent` errors that look permanent enough to be waved off as known, which is how a real error hides among them; after `npx turbo run build --filter=@activepieces/shared`, `tsc --noEmit -p packages/server/api/tsconfig.app.json` comes back completely clean. The same staleness invents errors in files a rebase just touched, so during a conflict resolution rebuild before concluding you resolved it wrong.
- **To pull a file back out of a PR, restore it from the merge-base, never from `origin/main`.** A PR's diff is computed against the merge-base, so `git checkout origin/main -- <file>` does not "revert" the file — it imports every change `main` made to it since the fork and attributes them to you. Dropping one web file from [#15001](https://github.com/activepieces/activepieces/pull/15001) that way would have silently added 82 insertions / 44 deletions of somebody else's work. `git checkout $(git merge-base origin/main HEAD) -- <file>` makes it byte-identical to where the branch started, so it leaves the diff entirely and merges cleanly instead of conflicting. Verify with `git diff --quiet $(git merge-base origin/main HEAD) -- <file>` before committing, and read `git status` first — a `bun.lock` left dirty by an earlier `bun install` loves to ride along on a commit like this.
- **Retargeting a stacked PR to `main` does not drop its base branch — it merges the whole thing.** A PR opened against a long-lived feature branch shows a small diff *relative to that base*, but `gh pr edit --base main` only moves the target; the branch still contains every commit of its old base. [#14593](https://github.com/activepieces/activepieces/pull/14593) read as 2 docs files against `feat/autumn-billing-integration` and as 198 commits / 211 files / +12k lines against `main`. Check with `git diff --stat origin/main...<branch>` **before** retargeting, and if it disagrees with the PR page, cherry-pick that PR's own commits onto `main` and force-push instead. A "conflict" on such a PR is often against the feature base only — those same commits can apply to `main` cleanly.
@@ -67,5 +67,10 @@ Enforcement is the **`Codeowners review` repository ruleset** (active on the def
- **Nothing rolls `## Unreleased` over at release time, and the docs site is unversioned — so a breaking-changes entry has to name its own version.** No workflow or script writes to `docs/install/reference/breaking-changes.mdx` (`breaking-change-check.ts` only reads it), and `git log -S"## 0.88"` on the file comes back empty: the heading has not moved since 0.87.0, so entries for work that shipped months ago still sit under "Unreleased" (PM2 removal in 0.88.2, cache pre-warm gate and workspace naming in 0.89.0, …). `docs/docs.json` has no versioning either, so there is one live page for every self-hoster whatever version they run, published on merge rather than on release — the version heading is the *only* thing telling a reader whether a change is already in their build. So before adding an entry, run `git tag --contains <commit>` on the change it describes and file it under the release that actually shipped it; only genuinely unshipped work belongs under "Unreleased". What points self-hosters at the page in the first place is `release-drafter.yml`, which appends a "review the Breaking Changes page" line to every release body and groups `⛓️‍💥 breaking-change` PRs under their own heading — which also means a docs-only PR back-documenting an old change shows up in the *next* release's breaking-change list.
- **Greptile enforces the file-order rule on *private* constants too, which CLAUDE.md only states for exported ones.** CLAUDE.md says "Exported types and constants must be placed at the end of the file" and gives the order as imports → exports → helpers → types; Greptile reads that as covering module-private constants as well, and flags a `const` sitting above the file's exported symbol (P2 on [#15226](https://github.com/activepieces/activepieces/pull/15226), for two constants only read inside the service they sat above). It has that as a stored custom-context memory, so it will keep raising it. Put private constants in the helpers section below the export — hoisting is a non-issue when they are only read at call time.
- **`turbo run lint --filter=<pkg>` never typechecks that package, so "0 errors" is no proof it compiles — run `turbo run build --filter=<pkg>` as well.** A package's `lint` script is `eslint 'src/**/*.ts'` with no `tsc`, and the turbo `lint` task's `dependsOn: ["^build"]` builds the package's *upstream dependencies* only, never itself — the `^` is the whole story. So a type error in the package you are editing is invisible to a filtered lint run. CI still catches it, but confusingly under the **lint** job as well as **main**, because linting the whole repo makes your package an upstream of something else whose `^build` finally compiles it. A branch can therefore sit red on two jobs for one `tsc` error that a local filtered lint reported clean.
+- **Never set `maxWorkers` in `packages/server/api/vitest.config.ts` — the api suites are not alone on the runner.** `ci.yml` runs `turbo run test-ce test-ee test-cloud check-migrations --filter=api` as concurrent turbo tasks, in parallel with a second `turbo run test` line and `web#build`'s vite run, all on one 4-vCPU / 16 GB `ubuntu-latest` box. Each vitest fork boots a Fastify server *and* a PGlite database, so the config's per-project worker count is really that number **times three editions**. A stray `maxWorkers: 4` on [#14777](https://github.com/activepieces/activepieces/pull/14777) raised the forks pool from its default `availableParallelism() - 1` (3) to 4 — 12 forks instead of 9 — and took the job down twice in a row with failures that never named the cause: first `execute-flow-e2e`'s `beforeAll` timing out while **1028 tests passed**, then the whole runner SIGKILLed mid-build (`web#build exited (137)`, `check-migrations` SIGTERM, api suites never finishing). Both read as "a test broke". Neither was. The tell is `The runner has received a shutdown signal` immediately above the 137, and the fix is to delete the line, not to raise a timeout. Same reason the config's `hookTimeout` is 120s: a loaded runner pushes a server boot past a minute, so a file that pins its own smaller `beforeAll` timeout (`}, 30_000)`) silently opts out of the protection and becomes the first thing to fail. Check the blast radius before calling it fixed: a config mistake made on a long-lived stack branch replicates by merge into every descendant, so deleting it from the PR in front of you leaves the rest of the stack to reintroduce it on the next rebase. This one originated once on `feat/subflow-fan-in-barrier` (2026-08-12) and reached **14 branches, 13 of them open PRs** across three unrelated stacks — `git log -S maxWorkers` over `origin/main` proved it never landed there, and a loop of `git show <branch>:<path> | grep` found every branch still carrying it.
+- **A red `api#test-ce` names one failing test, not all of them — the script runs `--bail 1`.** `test-ce-command` is `vitest run test/integration/ce --bail 1`, so CI stops at the first failure and the report is a floor, not a count. On `feat/barrier-edges` it showed one failing assertion; the file actually had three, all the same root cause. Before concluding a branch has one problem, re-run the suspect file locally *without* `--bail` — `npx vitest run <file>` — and fix the set.
+- **Fixing an infra failure can turn a branch from red to a different red, and that is progress, not a regression.** A runner killed mid-run (`shutdown signal` above `exited (137)`) reports no test results at all, so genuine failures underneath it are invisible. When the OOM is fixed the suites finally run to completion and those failures surface for the first time — on this stack, thirteen branches went from an OOM red to a real `barrier.test.ts` red the moment the runner survived. Check whether the *previous* run reached a test summary at all before blaming the change that unmasked it.
+- **A `beforeAll` that overrides the vitest timeout *downwards* is a slow-motion flake, and it fails in a way that blames your diff.** [`execute-flow-e2e.test.ts`](../../../packages/server/api/test/integration/ce/flows/flow-run/execute-flow-e2e.test.ts) passes `30_000` to `beforeAll` even though `packages/server/api/vitest.config.ts` sets `hookTimeout: 60000`, and spends 5s of that budget on a hardcoded `setTimeout`. Inside the remaining ~25s it boots a fresh app, runs the **whole migration chain**, starts queue consumers and a worker. That chain has grown 293 → 376 postgres migrations since the hook was written (`d2c42816013`, "feat: worker v2", #11608, 2026-03-15 — authored *after* the 60s default was already in place), so on a loaded runner it now times out. When it does, all 9 of its tests report **skipped**, not failed: the summary reads `1 failed | 84 passed` with no failing assertion anywhere, and `--bail 1` kills the rest of the CE task. Seen on [#15078](https://github.com/activepieces/activepieces/pull/15078), an MCP-OAuth-only PR that cannot reach that code. No single commit broke it, so there is nobody to blame and nothing to revert — the fix is to delete the override and poll for worker readiness instead of sleeping blind. When triaging a red CE task, check whether the failure is an assertion or a hook timeout **before** blaming the diff.
+- **Git hooks cannot run from a `git worktree`: `.husky/_/husky.sh` is an install artifact that only exists in the tree where `bun install` ran.** Every commit and push from a worktree dies on `.: cannot open .husky/_/husky.sh`, before the hook's own logic is reached. Don't reach for `--no-verify` — push the worktree's commit from the main checkout (`git push origin <local-branch>:<remote-branch>`; worktrees share the object store), so `pre-push` actually bootstraps and its "no direct pushes to `main`" guard runs. `SKIP_CHECK=1` is the repo's own sanctioned way to skip the lint/test prompt. Note `pre-push` special-cases `CLAUDE_PUSH=yes` into running the **full** lint + unit + API suite, so an agent that silently bypasses the hook is skipping a check the repo added deliberately for it — say so out loud when you do.
- **Docs videos belong on `cdn.activepieces.com/videos/docs/`, never in the repo — and `docs/resources/` is the legacy path that suggests otherwise.** Every `<video>` already in the wiki points at that CDN prefix, but `docs/resources/` still holds ~43 MB of in-repo gifs from before the convention, so the tree itself gives mixed signals and a contributor reasonably copies the wrong one. [#15399](https://github.com/activepieces/activepieces/pull/15399) added a fresh `docs/videos/` folder with 155 MB of `.mp4` against an 880 MB `.git`, including one byte-identical duplicate. Nothing in CI catches it: `PR size` exempts unmatched areas and counts lines, not bytes, and a binary blob contributes zero lines. Review the *file list* of a docs PR, not just its diff. Once such a branch is pushed the blobs are already on the remote — dropping them at the tip only keeps them out of `main` if the PR is **squash**-merged and the branch deleted, so a merge commit ships them permanently. Because the blobs stay reachable until the branch is deleted, nobody has to re-request the assets from whoever recorded them — pull them straight back out of history with `git archive <commit> docs/videos | tar -x --strip-components=2 -C <dir>`, picking the commit *before* the removal, which is also how you get them under the post-rename filenames the pages already reference.
- **Mintlify-editor PRs arrive with `untitled-page*.mdx` at the docs root and `(1)`/`(2)`-suffixed media, wired into `docs.json` under those names.** The bot commits whatever the web editor autosaved, so pages land as `docs/untitled-page-5.mdx` rather than in the folder their nav group implies, and re-uploaded assets keep the browser's download-collision suffix (`data-flow-(1).mp4`, `01---insert-panel-(1.2x).mp4`). Mintlify's own link check passes — the names are internally consistent — so nothing flags it. Rename the files, fix the `docs.json` entries, then diff the *flattened nav tree* against `main` rather than reading the raw JSON diff: the editor also reindents, which buried a real change (the Enterprise Control `Security` group demoted to a child of `Guides`) inside 242 lines of whitespace churn on that PR.
diff --git a/brain/knowledge/flows-execution/flow-runs.md b/brain/knowledge/flows-execution/flow-runs.md
index 14754aa8a15..a9d738711b4 100644
--- a/brain/knowledge/flows-execution/flow-runs.md
+++ b/brain/knowledge/flows-execution/flow-runs.md
@@ -24,7 +24,19 @@ A Flow Run records one execution of a specific flow version, from trigger to ter
- **A Delay inside a Loop pauses and requeues the whole run once per iteration, so no fixed sync-webhook timeout can cover it.** The delay is not a sleep inside the step: each iteration arms a waitpoint, the run goes `PAUSED`, and the resume comes back through the queue, so every item costs its delay plus queue latency and the total scales with the item count. Worked case on dev: `Catch Webhook → Code → Loop { Delay For 8s } → Return Response` over 6 items reported `stepsCount` 9 (Code + Loop + 6 delays + Return Response) and ran 53.6s, of which 48s was delay and 3s was the initial queue leg. With `AP_WEBHOOK_TIMEOUT_SECONDS` at 30 the `/sync` caller was answered 408 mid-loop; Return Response then ran ~23s later and published to a listener that had already resolved and been deleted, so it was a no-op. Putting the response step *after* slow work is the bug: respond before it (respond-and-continue rather than `stop`) or go async with a callback, because raising the timeout only works until someone sends more items. See the sync-response gotchas in [webhooks](../eventing-webhooks/webhooks.md).
- **A worker OOM-kill leaves the run stuck in RUNNING forever, and Cancel is greyed out.** The flow timeout is enforced *inside* the worker, so if the pod dies (OOM) nothing ever transitions the run to a terminal state; Cancel only applies to paused/queued runs, so the UI offers no way out and the run can't be retried either. Bug: activepieces#14372, fix PR #14374. Manual unblock on the customer's Postgres: `UPDATE flow_run SET status = 'CANCELED', "finishTime" = NOW(), updated = NOW() WHERE id = '<run id>' AND status = 'RUNNING';` (run id = last path segment of the run URL), then "Retry on latest version" replays the original payload.
- **Resume Confirmation Page (scanner guard)**: the `/confirm` route serves an HTML Approve/Disapprove page on `GET`/`HEAD` (never consumes) and only resumes on `POST` — stops email security scanners (Safe Links, Mimecast, Proofpoint) prefetching approval links. The deprecated bare `GET /:id/waitpoints/:waitpointId` still resumes for old emails. Slack is unchanged (server-side POST from webhook).
+- **`POST /confirm` with an empty JSON body is rejected before it reaches the resume code.** Sending `Content-Type: application/json` with no body gets Fastify's `FST_ERR_CTP_EMPTY_JSON_BODY` 400, so the waitpoint is never consumed and the run just stays PAUSED — which reads exactly like a resume regression. `curl -X POST .../confirm?action=approve` is fine because curl sends no content-type at all; an HTTP client that defaults to JSON is not. Either omit the content-type or send a real body (`{}`). Hit while scripting the `/test-waitpoints` flow 4 race ordering, 2026-08-30.
- **Cross-project isolation (subflow parent-fail)**: `markParentRunAsFailed` scopes its parent lookup to `{ id: parentRunId, projectId }` using the child run's authenticated `projectId`. `parentRunId`/`failParentOnFailure` arrive from spoofable webhook headers (`ap-parent-run-id`/`ap-fail-parent-on-failure`) on the public webhook endpoint, so without the scope a failed child in project A could complete a paused parent's waitpoint and resume it in project B. A cross-project parent id now matches nothing and the fail is a no-op; legitimate subflows are always same-project (Call Flow only targets flows in the caller's project).
+- **A resume link resumes the run, not the step it was minted for.** Waitpoints are unique on `(flowRunId, stepName)`, so a flow that runs `create_approval_links` and then `wait_for_approval` holds **two PENDING rows at once** — and the link handed out by `create_approval_links` points at the *first* row while the run blocks on the *second*. `handleResumeSignal`'s PAUSED branch looks the row up by `{ id, flowRunId }` with no check that it belongs to the step being waited on, so that link resumes the run anyway and its payload lands on whichever step is paused. Verified 2026-08-30 on a local instance: POSTing the step_1 link while PAUSED on step_3 resumed the run and gave step_3 `approved: true`. Upshot: an approval link is a **run-level** capability, not a step-level one.
+- **A resume that arrives while the run is still RUNNING is parked, and delivered later by the `runsMetadataQueue` worker.** `handleResumeSignal`'s RUNNING/QUEUED branch takes `complete()`, which marks the row COMPLETED rather than deleting it; the worker then picks it up on writing PAUSED via `findUndeliveredCompletedWaitpoint` (`flow-runs-queue.ts`) and enqueues the resume. Because delivering a resume never leaves the row at COMPLETED (both `handleResumeSignal`'s PAUSED branch and the race branch in `resumeTrustedWithoutLock` go through `waitpointService.consume`, which deletes a non-barrier and tombstones a barrier as CONSUMED), any row still sitting at COMPLETED is by construction one whose signal was never delivered — which is why that lookup does not need to care how many other waitpoints the run holds. It declines only when the run holds a PENDING **barrier**, since a barrier may be released solely by its predicate or deadline. **Gotcha for anyone narrowing this check:** an earlier form returned null whenever *any* PENDING row existed, which hung every run holding a second waitpoint — `create_approval_links` + `wait_for_approval` sat in PAUSED forever with the approval already consumed (`step_3` output `{approved: true}`, status `PAUSED`). Reproduced and fixed 2026-08-30 on `feat/barrier-service` with flow 4 of `/test-waitpoints`. Single-waitpoint runs (the common subflow-callback case) never showed it, so unit tests and every other flow in the suite stayed green.
+- **`create_approval_links` hands out an approval and a disapproval URL that are byte-identical** — both are the bare `/confirm` route with no `?action=`. The verdict comes from the button the human clicks on the confirmation page (which POSTs back with the action), not from which link they were given, and `wait_for_approval` reads `queryParams.action` with anything other than `approve` counting as a disapproval.
+- **A run can only have ONE scheduled DELAY resume at a time, and arming a second one is silently dropped.** `createForPause` schedules the resume job with `jobId: resume-delay-${flowRunId}` — keyed on the *run*, not the waitpoint — and `systemJobsSchedule.upsertJob` only calls `queue.add` when no job with that id exists (it retries an existing *failed* job, and otherwise does nothing at all, with no log beyond "Upserting job"). So if a run's first DELAY waitpoint is removed **before its job fires** and the run then arms another DELAY, the second schedule is a no-op: the queue still holds the first job, pointing at the now-deleted waitpoint. That job fires on the old date, `handleResumeSignal` finds nothing for the stale id, logs "Stale waitpointId, ignoring", and the job is consumed — leaving the run **PAUSED forever** with an overdue `resumeDateTime` and nothing in `bull:system-job-queue:delayed`. Reproduced 2026-08-30 by resuming a loop's iteration-0 delay early via `POST /:id/waitpoints/:waitpointId`: iteration 1 armed at +1s but the job kept iteration 0's id and fire time, and the run never completed. **This is on `main`, not a barrier-branch regression.** Normal delay flows are unaffected only because `removeOnComplete: true` frees the id once a job fires. Note `removeOnFail` keeps failed jobs for a month, so a single failed delay job also blocks every later DELAY in that run — and `upsertJob`'s retry re-runs the *old* `waitpointId`. Any path that resumes a run parked on a DELAY (the deprecated bare `GET` resume link, a subflow callback via `findSubflowWaitpoint`, a deadline sweep) can set this up. Fix direction: key the job on the waitpoint (`resume-delay-${waitpoint.id}`), which also needs `refill-paused-jobs.ts` updated since it reconstructs the same id.
+- **A barrier's waitpoint row survives its release, and the legacy by-run resume guard keys off that row's existence.** Until 2026-08-30 `legacyResume` / `legacySyncResume` (the deprecated V0 `/:id/requests/:requestId` routes, reached whenever `findPendingByVersion('V0')` returns null) checked `hasPendingBarrier`, which filters `status: PENDING` — so the guard stopped guarding the instant the barrier was released. `closeBarrier` commits COMPLETED in its own transaction and only *then* does `release()` call the trusted resume under the `runs_metadata_${flowRunId}` lock, and the legacy paths take no lock at all. Through that gap **and** the much wider one after delivery (the run stays PAUSED until the engine's next progress upload — `flow_run.status` is written only by the `runsMetadataWorker`), the run was PAUSED with no PENDING barrier, so a legacy resume was accepted and enqueued a **second** job with a request-controlled payload: `enqueueResume` namespaces job ids as `${runId}-resume-legacy` vs `${runId}-resume-${barrierId}` and the worker queue has no BullMQ `deduplication`, so nothing collapsed them. Reproduced against the dev DB on `feat/barrier-service`, and identical on all four stacked branches. **Fixed** by making delivery explicit rather than encoded as row-absence: a delivered barrier is tombstoned `CONSUMED` (see [decision 000033](../decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md)) and the legacy guard became `hasBarrier` — existence, no status filter. **Gotcha for anyone touching these two methods:** `hasPendingBarrier` still exists and `findUndeliveredCompletedWaitpoint` must keep calling *it*, not `hasBarrier`; switching that call site to existence would return null whenever a CONSUMED barrier is present and hang a later non-barrier delivery on the same run. Two different questions, two methods. The V1 addressed path never had the hole — `addressedWaitpointIsBarrier` rejects on `waitpoint.type === BARRIER` whatever the status.
+- **`handleResumeSignal`'s PAUSED branch must reject `CONSUMED` only — never require `PENDING`.** The three statuses are a delivery state machine, not a lifecycle: PENDING is *not yet owed*, COMPLETED is *owed*, CONSUMED is *delivered*. Requiring PENDING therefore breaks the pre-completed path, because `findUndeliveredCompletedWaitpoint` hands this branch a COMPLETED row on purpose whenever a callback beat the pause. Tried on `feat/barrier-service` 2026-09-22 while answering a review that proposed exactly that: it failed **nine** tests, four of which predate the barrier work, including `consumes a COMPLETED waitpoint when race recovery fires` and `does not leave a stale COMPLETED row to poison the next pause cycle`. Before that fix the branch had no status filter at all, so a CONSUMED barrier tombstone could be dispatched a second time.
+- **That `CONSUMED` check does not need a conditional `UPDATE ... WHERE status = ... RETURNING`, and two reviewers have now asked for one.** The branch already runs `SELECT ... FOR UPDATE` inside `transaction()`, and TypeORM/Postgres default to READ COMMITTED, where a `FOR UPDATE` that blocks on a concurrent writer does **not** return its original snapshot on release — EvalPlanQual follows the update chain and re-checks the `WHERE` against the newest committed version. The predicate is `id` + `flowRunId` + `projectId`, all immutable, so the row comes back with the committed status. Read-then-branch-then-write in that transaction has exactly the atomicity of a conditional UPDATE. A conditional UPDATE is also actively worse here: `consume` is polymorphic (DELETE for a non-barrier, UPDATE for a barrier), and `onReady` has to run *between* the status check and the status write, which a single `RETURNING` statement cannot host.
+- **`onReady` runs inside that transaction deliberately — do not move it after the commit.** The order is dispatch → consume → commit, so a crash in the window leaves a resume job plus an unconsumed row: recoverable, and the row is the evidence that recovery is owed. Committing first would leave a `CONSUMED` tombstone (or, for a non-barrier, nothing at all) with no job, which is an unrecoverable hang and a tombstone that lies. The real cost of the current order is that `enqueueResume` does a platform lookup and an unconditional payload offload — taking a *second* pool connection — while holding `FOR UPDATE`. Fix that by hoisting the offload above the transaction, never by reordering the commit.
+- **Widening `releaseIfReady`'s lookup to COMPLETED recovers nothing on its own.** It still routes into `release` → `closeBarrier`, which filters `status: PENDING`, returns null, and makes `release` return *before* `resumeTrusted`. A crash between `closeBarrier`'s commit and `resumeTrusted` otherwise strands the run PAUSED with a COMPLETED barrier forever: `releaseIfReady` used to filter PENDING, the deadline handler returns unless the waitpoint is PENDING, and `closeBarrier` deleted the signals so nothing re-enqueues an evaluation. The recovery has to branch on status and re-dispatch the summary already persisted on the row, bypassing `closeBarrier` — routed through the same `resumeTrusted` call the normal path uses, so it inherits the run lock and the CONSUMED guard instead of enqueuing beside them.
+- **That recovery is bounded, not complete.** The only automatic re-invoker is BullMQ's stalled check on `BARRIER_EVALUATION`, and the worker takes the default `maxStalledCount: 1` — exactly one free redelivery after a SIGKILL. A second stall retires the job to `failed` permanently, and a Redis flush removes the trigger altogether while Postgres still holds a COMPLETED barrier on a PAUSED run. `attempts: 5` and the 8-minute backoff do not help: they govern *failed* jobs, and a retry just re-runs `releaseIfReady`. The durable backstop is a sweep, and a deadline sweep only closes this if it selects `type = BARRIER AND status = COMPLETED` — the natural `status = PENDING` reading never sees the state. The same hole exists for non-barrier waitpoints (`complete()` commits COMPLETED, the caller dies before dispatch) and is covered only opportunistically by the metadata worker.
+- **A waitpoint inside a loop is keyed by its full execution path**, e.g. `step_1:0/step_2`, not the bare `step_2` — so each iteration gets its own row under the `(flowRunId, stepName)` unique constraint instead of colliding with the previous iteration's. Set by `step-execution-path.ts`.
- **ResumeReason** (`WAITPOINT`|`RETRY`) discriminates whether FAILED steps are restored on resume: waitpoint resumes preserve them, retry resumes drop them so the failed step re-executes.
- **Pause timeout is cumulative from `flowRun.created`, not per-pause.** `waitpointService.createForPause` (`clampWaitpointResumeDeadline`) rejects any resume date past `flowRun.created + AP_PAUSED_FLOW_TIMEOUT_DAYS`, throws `ErrorCode.PAUSED_FLOW_TIMEOUT_EXCEEDED`, and `waitpointClient` in the engine translates that to `PausedFlowTimeoutError` (USER → clean FAILED). This catches stacked delays (e.g. `1 + 10 + 10 + 25 + 25 = 71 days` on a 30-day timeout) that individually pass the engine-side per-pause `assertDelayWithinTimeout` but cumulatively outlive retention. Webhook waitpoints without an explicit `resumeDateTime` are defaulted to the same run deadline so they can't wait indefinitely past retention; when their timer fires the `RESUME_DELAY_WAITPOINT` handler branches on `PauseType.WEBHOOK` and marks the run FAILED (expiry, not resume). The same handler also marks FAILED if the timer fires past pause-timeout for a `DELAY` waitpoint — catches BullMQ dispatch lag on ancient queued jobs. `AP_EXECUTION_DATA_RETENTION_DAYS` must be strictly greater than `AP_PAUSED_FLOW_TIMEOUT_DAYS` (a 1+ day margin) or a valid resume firing on the retention boundary can still race the file-cleanup job.
- **`executionJournal.upsertStep` must stay immutable, and step retry is why.** `runWithExponentialBackoff` re-uses the *same* `executionState` for every attempt, so a failed attempt writing its `FAILED` output must not reach back into that state. While the journal still mutated the shared `steps` map (fixed in #14453), the write cleared the step's `PAUSED` status, the next attempt read `isPaused === false` and ran `BEGIN` instead of `RESUME`, armed a **new** waitpoint and paused again — a fresh engine run per resume means `attemptCount` restarts at 1, so `maxAttempts` is unreachable. For Call Flow with wait-for-response that re-invoked the subflow every 4-5s forever with the parent stuck `PAUSED` (GIT-1712, ≤0.86.3, same defect at the old `packages/shared/...` path in 0.85.5). Pinned by the retry-on-failure subflow case in `execute-flow-e2e.test.ts`.
@@ -36,6 +48,8 @@ A Flow Run records one execution of a specific flow version, from trigger to ter
- Retries only allowed on terminal states within `EXECUTION_DATA_RETENTION_DAYS`.
- **Credit metering (Autumn)**: on terminal runs (paid editions), `onFinish` does two tryCatch-wrapped billing steps that never break run completion. (1) A PRODUCTION run not in `QUOTA_EXCEEDED` charges +1 apCredit via `billingProvider.trackCredits` with idempotency key `{runId}:run`. (2) `flowRunAiUsageTracker` pre-scans the flow version for `@activepieces/piece-ai` steps, extracts per-provider/model usage from step outputs (`flow-run-ai-usage-extractor` — recurses into loops, fetches `FLOW_RUN_LOG_SLICE` files, falls back to flow-version settings on `**REDACTED**` models), meters `Σ(messages × model credit weight) + toolCalls` to Autumn (`{runId}:ai`, plus `{runId}:appSumoAi` for the managed-ACTIVEPIECES AppSumo cap), then emits the `AI_USAGE_PER_RUN` PostHog event — the license key is only the PostHog distinctId, no longer a gate on metering.
- **Credit gate is fail-open at admission**: the worker RPC `submitPayloads` checks `shouldBlockOnCredits` (blocks only when the platform is `billingEnforced` AND the cached balance is exhausted; CE default and Autumn-outage behavior is false). A blocked run is still admitted — as a `QUOTA_EXCEEDED` run with the trigger payload persisted in its log — so it stays retryable once credits return instead of being dropped. **`AP_EDITION=ee` skips the gate entirely** (`shouldBlockRunOnCredits` returns `false` before any provider call) so self-hosters pay no Redis/Autumn latency on admission — a temporary measure, see decision 000020.
+- **A cancelled run's terminal status is only *queued*, so `flow_run.status` still reads `PAUSED` for a moment — read the Redis intent, not the row.** `cancelSingleRun` holds `runs_metadata_${id}`, removes the jobs and deletes every waitpoint, then hands `CANCELED` to `runsMetadataQueue` rather than writing it. For that window a resume request sees a `PAUSED` run with no waitpoint and cannot tell it from a legacy pre-waitpoint pause, and nothing downstream stops it — `execute-flow.ts` has no terminal-status guard, so the resume actually runs and overwrites the cancellation. The intent *is* observable: `runsMetadataQueue.add` writes the pending status into the `runs_metadata:<runId>` hash synchronously, inside that same lock, before enqueuing, and the metadata worker clears the key with `deleteKeyIfFieldValueMatches` once it lands. So `resumeService` refuses when `hgetJson(redisMetadataKey(id))` already holds a terminal status, and every resume path — the legacy by-run ones included — takes `runs_metadata_${id}` so the two orderings are both safe: cancel-first makes the resume wait and then see it, resume-first lets cancel's `removeAllFlowRunJobs` reap the job it just enqueued. The lock alone is not enough, and neither is re-reading `flow_run` inside it.
+- **The `RESUME_DELAY_WAITPOINT` job id is keyed per waitpoint, and it has to stay that way.** `waitpointService.createForPause` schedules it as `resumeDelayJobId(waitpoint.id)` — never per run — because `systemJobsSchedule.upsertJob` only adds when that id is free (`if (isNil(existingJob))`), so a run-keyed id means the first waitpoint to arm owns the run's **only** timer and every later one silently gets none. Since decision 000015 one run legitimately holds several waitpoints, so that is a hang, not an edge case: `create_approval_links` (WEBHOOK, +30d) followed by `delayFor 45s` (DELAY) left the delay with no timer and the run sat `PAUSED` until the 30-day webhook expiry marked it FAILED. Two `WEBHOOK` rows hide it — identical `AP_PAUSED_FLOW_TIMEOUT_DAYS` deadlines, and each is resumed by its own callback anyway — which is why flow 4 of `/test-waitpoints` never caught it and flow 9 was added. Found and fixed 2026-09-10; the bug had been live on `main`, not introduced by the barrier work. Per-waitpoint keying also fixes the `removeOnFail: { age: ONE_MONTH }` case, where `upsertJob` retries a month-old failed job and would otherwise replay it against a **different** waitpoint's id. Pinned by "should give each waitpoint of one run its own resume job id" in `waitpoint.test.ts`.
- **Post-run metering window**: AI usage is metered only at `onFinish`, so a long run can spend past the credit limit before anything lands; interim by design — see decision 000016.
### Editions
diff --git a/brain/knowledge/flows-execution/index.md b/brain/knowledge/flows-execution/index.md
index 6b8580a64ac..6ccaad002d3 100644
--- a/brain/knowledge/flows-execution/index.md
+++ b/brain/knowledge/flows-execution/index.md
@@ -53,6 +53,17 @@ A **Subflow** is a flow invoked by another flow rather than by its own external
- **Batch** — the rows carried by one fan-out call: `{ batchIndex, headers, rows, extraData }`. *Avoid:* chunk, csv table, sub-table, shard.
- Parent linkage is two headers (`x-parent-run-id`, `x-fail-parent-on-failure`); fan-out sets the latter false. Streaming bounds memory, not time — the step is still capped by `FLOW_TIMEOUT_SECONDS`.
+### Waitpoints
+A **waitpoint** is a row recording something a run waits on — `DELAY` / `WEBHOOK` / `BARRIER`. A run may hold several at once, and a barrier's row outlives its own delivery, so "the waitpoint a run is parked on" is never a safe singular. Reference lives in the public docs; what belongs here is the vocabulary, because the verbs are easy to swap.
+- **Release** — what happens to a **barrier**: close it, write its summary, delete its signals. `barrierService.release`. *Avoid:* "resume the barrier".
+- **Resume** — what happens to a **run**: enqueue the job that re-invokes the step. `resumeService.resume*`. A release ends by asking for a resume; the reverse never happens. *Avoid:* "release the run".
+- **Trusted resume** — `resumeTrusted` / `resumeTrustedWithoutLock`: resume skipping the barrier guard, for internal callers only. Every HTTP route goes through `resumeFromWaitpoint`, which refuses barriers always.
+- **Waitpoint key** — the path-qualified step identity the engine sends in `stepName`, e.g. `loop_1:3/approval`. It is only ever an identity key inside `waitpoint-service`. Gives each loop iteration its own row; without it a pause inside a loop fired on iteration 1 only.
+- **Consume** — what happens to a **waitpoint** once its resume has been enqueued: a barrier is tombstoned `CONSUMED` so the run stays provably barrier-owned until it ends, any other type is deleted. `waitpointService.consume`, the single owner of the rule. *Avoid:* "close the waitpoint".
+- **Signal** — one `waitpoint_signal` row per awaited thing on a barrier, created up front. Deleted on release, so a returning approver sees only "already responded".
+- Gotcha: a run can hold a COMPLETED leftover *and* an open PENDING pause at once, so any per-run waitpoint lookup must say which it wants — `findUndeliveredCompletedWaitpoint` declines only while a PENDING **barrier** is held, since delivering a resume never leaves a row at COMPLETED. An earlier form declined while *anything* was pending and hung every run holding a second waitpoint.
+- See [decision 000015](../decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md), [decision 000033](../decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md) and the public [Waitpoints](https://www.activepieces.com/docs/install/architecture/waitpoints) page.
+
### Folders
Lightweight per-project grouping for flows and tables. Name unique case-insensitively per project.
- `folder` entity (displayName, projectId, displayOrder). List returns `numberOfFlows`/`numberOfTables` via correlated subqueries. Create is an upsert by name.
@@ -71,6 +82,7 @@ Reusable flow/table blueprints. Types: OFFICIAL (Activepieces-curated, platformI
- **Triggers** — POLLING / WEBHOOK / APP_WEBHOOK / MANUAL
- **Human Input** — forms, approvals, the resume confirmation page
- **Subflows** — flow-calls-flow: Callable Flow, Call Flow, streaming fan-out
+- **Waitpoints** — how a paused run is parked; release vs resume, barriers, signals
- **Folders** — flow organization; the uncategorized sentinel
- **Templates** — OFFICIAL / CUSTOM / SHARED blueprints
- **Variables** — project-scoped values referenced from steps
diff --git a/brain/knowledge/platform-editions-ee/platform-configuration.md b/brain/knowledge/platform-editions-ee/platform-configuration.md
index 4c098e411f6..1072f75adfb 100644
--- a/brain/knowledge/platform-editions-ee/platform-configuration.md
+++ b/brain/knowledge/platform-editions-ee/platform-configuration.md
@@ -27,6 +27,12 @@ A **Platform** is the top-level tenant namespace in Activepieces. Every install
- **What blocks `DELETE FROM platform`** is now two constraints: `project` and `signing_key`, both `RESTRICT`. It used to be four — `tag` and `piece_tag` were `NO ACTION` and were dropped once it was clear the tables had outlived their feature. The other 14 FKs already `CASCADE`. Platform deletion failing for one tenant and not another almost always means a new blocking FK, not a member count.
- **Delete order is forced by `platform.ownerId → user` being `RESTRICT`**: the platform row must go before its owner's `user` row, never the reverse. `project.ownerId → user` is `NO ACTION`, so projects must also be gone before any user is deleted.
- Any new entity carrying `platformId` should declare its FK `ON DELETE CASCADE`. Otherwise add it by name to the teardown job (`ee/platform/platform-teardown-jobs.ts`), which deletes the two blockers and the twelve unconstrained tables itself — nothing in CI checks either. See [000026](../../decisions/000026-delete-platform-is-a-cloud-owner-action-purged-by-one-cascading-job.md).
+- **`getOrCreateForPlatform` takes a `distributedLock` only on the create path, and that lock waits up to 60 seconds before it throws.** `runExclusive({ timeoutInSeconds: 60 })` becomes redlock `retryCount: 300, retryDelay: 200ms` in `distributed-lock-factory.ts`, so the timeout is both the lock TTL and the acquisition budget: a contended first touch blocks, it does not fail fast. This only ever applies to the first read for a platform, since a platform that already has a row is served by a bare `findOneBy` on the unique `platformId` index with no lock and no Redis. Worth knowing before putting a configuration read on a latency-sensitive path, and worth not defending against on a path that already needs Postgres and Redis to do its job.
+- **A `platform_configuration` column has three fallbacks for a missing row and none for a failed read, and the two shipped columns do not use the same ones.** `isProductTelemetryEnabled` is backed by `AP_TELEMETRY_ENABLED` at creation (`createInitialConfiguration`, via `spreadIfNotUndefined`) *and* at read time in the bulk path (`COALESCE(..., :fallback)` in `filterProjectsWithProductTelemetryEnabled`). `isInfraSetupTelemetryEnabled` has **no env var behind it at all**: nothing writes it on create, `filterPlatformsWithInfraSetupTelemetryEnabled` selects only the rows that say `false` and counts every row-less platform as enabled, so its only fallback is the schema `DEFAULT true`. Reading [000033](../../decisions/000033-platform-configuration-rows-are-authoritative-and-created-on-first-read.md) leaves the impression that "the insert reads the env var" holds for the table; it holds for one column. Separately, the single-platform read (`getOrCreateForPlatform`) is wrapped in no `tryCatch` at any call site, so a Postgres outage or a `distributedLock` timeout throws into the caller. `platformPlanService.getOrCreateForPlatform` has the same shape but its callers do defend (`ee/agent/agent-helpers.ts`). Any new consumer on a user-facing path has to choose fail-closed or fallback deliberately.
+- **`ActivepiecesError`'s `.message` is only the error code, so `rejects.toThrow('some text')` never matches the message you wrote.** The constructor is `super(error.code + ...)`, so a `VALIDATION` error thrown with `params: { message: 'This step waits on 3 things...' }` reports a `.message` of exactly `VALIDATION`, and a `toThrow` on the readable text fails with `expected [Function] to throw error including '...' but got 'VALIDATION'`. Assert on the structure instead: `rejects.toMatchObject({ error: { code: ErrorCode.VALIDATION, params: { message: expect.stringContaining('...') } } })`. Applies to every service-level throw in the API, not just configuration reads.
+- **Before moving an env var to `platform_configuration`, count the *processes* that read it, not the call sites, and check whether it is a cap rather than a toggle.** Two barrier vars were assessed and split cleanly. `AP_MAX_BARRIER_SIGNALS` is movable: one read site (`assertSignalCountWithinLimit` in `waitpoints/barrier-service.ts`), app-side only, and `barrierService.create` has no production caller yet. Its check site holds a `projectId` and not a `platformId`, and the waitpoints module already resolves that with `projectService.getPlatformId` (see `resume-service.ts`), so a per-platform read costs two indexed queries on a path that is about to insert up to 10 000 signal rows. `AP_PAUSED_FLOW_TIMEOUT_DAYS` is **not** movable despite also setting a barrier deadline: the engine reads it from `process.env` at module load (`handler/piece-executor.ts`), `machine-service.ts` ships it to workers inside the machine-settings zod contract, `flag.service.ts` republishes it to the browser, file cleanup sizes retention from it, and `system-validator` cross-checks it against `AP_EXECUTION_DATA_RETENTION_DAYS`, and a per-platform row cannot answer any of those. Separately, even the movable one is a *protective cap* on tenant fan-out, the `MAX_RECORDS_PER_TABLE` shape [000033](../../decisions/000033-platform-configuration-rows-are-authoritative-and-created-on-first-read.md) keeps off this page: `POST /v1/platform-configurations` has no edition guard, so the day the page stops being Cloud-hidden a cap living here is a self-serve limit bypass. If it should vary per tenant, it is a `platform_plan` entitlement.
+- **`maxBarrierSignals` has no UI.** `limits-section.tsx` renders it and `configurations/index.tsx` deliberately does not mount that component, so the Configurations page shows telemetry only. The column, the zod bounds and the `POST` body all accept it, so it is editable over the API and not in the product. Do not point a user-facing message at the Configurations page for this setting until the section is mounted.
+- **`maxBarrierSignals` is bounded on both ends, and the env var is clamped rather than trusted.** `maxBarrierSignalsBounds` in `@activepieces/shared` (1 … 10 000) backs the zod `.min`/`.max` on `PlatformConfiguration`, so the same numbers guard the request body, the response serialization and the `min`/`max` on the settings input. Because `PlatformConfiguration` is also the GET response schema, an `AP_MAX_BARRIER_SIGNALS` above the ceiling would seed a row that no longer serializes — `createInitialConfiguration` and the Cloud read path therefore clamp the env value into the bounds, and `system-validator` only warns (validator failures never block boot), so clamping is the actual protection, not the warning.
- **Setting a user `INACTIVE` stops logins and nothing else.** The trigger scheduler, the BullMQ queue, and the `SERVICE` principal an API key produces never read `user.status`. Cutting a platform off means disabling every flow through `CHANGE_STATUS` (so `triggerSourceService.disable` unregisters webhooks and drops schedules), draining queued work with `batchDeleteByFlowId`, and deleting the platform's `api_key` rows — deactivating members alone leaves all of it running.
### Key files
diff --git a/bun.lock b/bun.lock
index 6186e2d4138..8ad96b2c8eb 100644
--- a/bun.lock
+++ b/bun.lock
@@ -120,7 +120,7 @@
},
"packages/core/execution": {
"name": "@activepieces/core-execution",
- "version": "0.20.2",
+ "version": "0.21.0",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -165,7 +165,7 @@
},
"packages/core/shared": {
"name": "@activepieces/shared",
- "version": "0.176.0",
+ "version": "0.177.0",
"dependencies": {
"@activepieces/core-execution": "workspace:*",
"@activepieces/core-formula": "workspace:*",
diff --git a/docs/install/architecture/waitpoints.mdx b/docs/install/architecture/waitpoints.mdx
index 20adcf77e53..7ed01c7bf4e 100644
--- a/docs/install/architecture/waitpoints.mdx
+++ b/docs/install/architecture/waitpoints.mdx
@@ -17,7 +17,7 @@ stateDiagram-v2
| Field | Meaning |
| --- | --- |
-| `flowRunId`, `stepName` | Which run/step is paused. Unique together: a step has at most one waitpoint. |
+| `flowRunId`, `stepName` | Which run/step is paused. Unique together. `stepName` is **path-qualified** for a step inside a loop (`loop_1:3/approval`), so each iteration gets its own row. |
| `type` | `DELAY`, `WEBHOOK` or `BARRIER`. |
| `status` | `PENDING` until the resume signal arrives, then `COMPLETED`. |
| `resumeDateTime` | For `DELAY`: when to fire the resume. |
@@ -62,7 +62,8 @@ The step resumes with one shape, so an expression never breaks on fan-out width:
The `signals` array is present only when `total <= 100`; above that only the counts travel and
`signalsTruncated` is set. `AP_MAX_BARRIER_SIGNALS` (default 10 000) caps how many things one barrier may
-wait on. A barrier's deadline is set when it is created, from `AP_PAUSED_FLOW_TIMEOUT_DAYS`, and never moves.
+wait on. The cap is stored per platform, created from the env var the first time that platform's
+configuration is read. 10 000 is also the ceiling: the env var cannot raise it higher. A barrier's deadline is set when it is created, from `AP_PAUSED_FLOW_TIMEOUT_DAYS`, and never moves.
## Lifecycle
@@ -95,7 +96,7 @@ sequenceDiagram
A callback can arrive before the flow run has finished writing `PAUSED` to disk. The protocol absorbs the race:
- Completing a waitpoint takes a write lock on the `PENDING` row. If it exists, it flips to `COMPLETED` and stores the `resumePayload`. If it does not, a **pre-completed** row is inserted instead.
-- When the flow run transitions to `PAUSED`, the server checks for a matching `COMPLETED` waitpoint and enqueues the resume job immediately.
+- When the flow run transitions to `PAUSED`, the server checks for a `COMPLETED` waitpoint on the run and enqueues the resume job immediately — but only once **no** waitpoint for that run is still `PENDING`. A run holding a pause per loop iteration can carry a completed leftover and an open pause at the same time, and resuming off the leftover would skip the open one.
Duplicate callbacks are absorbed by the uniqueness constraint; the engine never processes a resume twice.
@@ -111,3 +112,8 @@ Duplicate callbacks are absorbed by the uniqueness constraint; the engine never
Piece authors create waitpoints with `ctx.run.createWaitpoint` / `ctx.run.waitForWaitpoint`. Patterns for `WEBHOOK`, `DELAY`, and `responseToSend` are in Flow Control.
+## Known limitation
+
+`findPendingByVersion` — the lookup behind the deprecated V0 `/:id/requests/:requestId` resume routes —
+returns an arbitrary pending V0 waitpoint for a run. That is exact today because nothing gives one run two
+pending V0 waitpoints at once. Parallel branches would, and must fix this lookup first.
diff --git a/docs/install/reference/environment-variables.mdx b/docs/install/reference/environment-variables.mdx
index 00dca0b93be..3d65b000982 100644
--- a/docs/install/reference/environment-variables.mdx
+++ b/docs/install/reference/environment-variables.mdx
@@ -178,6 +178,7 @@ run data is kept. The **Cloud** values and how these interact are covered in
| `AP_MAX_WEBHOOK_PAYLOAD_SIZE_MB` | Maximum incoming webhook payload size (MB). Larger payloads are rejected with HTTP 413. | `25` |
| `AP_WEBHOOK_PAYLOAD_INLINE_THRESHOLD_KB` | Webhook payloads below this (KB) are stored inline in Redis; larger ones are offloaded to file storage to protect Redis memory. | `512` |
| `AP_WEBHOOK_TIMEOUT_SECONDS` | Default timeout for synchronous webhooks. Maximum 15 minutes; note Cloudflare caps it at 30 seconds. | `30` |
+| `AP_MAX_BARRIER_SIGNALS` | Maximum number of things a single barrier may wait on — the approvers on one approval, or the batches a fan-out dispatched. The cap is stored per platform and created from this value the first time that platform's configuration is read, so changing it later only affects platforms that have not been read yet. Capped at `10000`; a higher value is clamped down to it. | `10000` |
| `AP_MAX_RECORDS_PER_TABLE` | Maximum number of records per table. | `10000` |
| `AP_MAX_FIELDS_PER_TABLE` | Maximum number of fields per table. | `100` |
diff --git a/packages/core/execution/package.json b/packages/core/execution/package.json
index 099a27a8946..7b1be0219e5 100644
--- a/packages/core/execution/package.json
+++ b/packages/core/execution/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-execution",
- "version": "0.20.2",
+ "version": "0.21.0",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/execution/src/lib/flow-run/waitpoint/index.ts b/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
index 7f3005206b6..7fc7a006104 100644
--- a/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
+++ b/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
@@ -101,6 +101,8 @@ export const BarrierSummary = z.object({
})
export type BarrierSummary = z.infer<typeof BarrierSummary>
+export const MAX_INLINE_BARRIER_SIGNALS = 100
+
export const UNFAVOURABLE_SIGNAL_STATUSES = [
BarrierSignalStatus.FAILED,
BarrierSignalStatus.REJECTED,
diff --git a/packages/core/shared/package.json b/packages/core/shared/package.json
index ff909b6dece..303ade598a5 100644
--- a/packages/core/shared/package.json
+++ b/packages/core/shared/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/shared",
- "version": "0.176.0",
+ "version": "0.177.0",
"type": "commonjs",
"sideEffects": false,
"main": "./dist/src/index.js",
diff --git a/packages/core/shared/src/lib/management/platform/platform-configuration.ts b/packages/core/shared/src/lib/management/platform/platform-configuration.ts
index 08e92b33282..ca7a006bdd9 100644
--- a/packages/core/shared/src/lib/management/platform/platform-configuration.ts
+++ b/packages/core/shared/src/lib/management/platform/platform-configuration.ts
@@ -1,16 +1,33 @@
-import { BaseModelSchema } from '@activepieces/core-utils'
+import { BaseModelSchema, isNil } from '@activepieces/core-utils'
import { z } from 'zod'
+import { formErrors } from '../../form-errors'
+
+const MIN_BARRIER_SIGNALS = 1
+const MAX_BARRIER_SIGNALS = 10_000
export const PlatformConfiguration = z.object({
...BaseModelSchema,
platformId: z.string(),
isProductTelemetryEnabled: z.boolean(),
isInfraSetupTelemetryEnabled: z.boolean(),
+ maxBarrierSignals: z
+ .int({ error: (issue) => isNil(issue.input) ? formErrors.required : formErrors.wholeNumber })
+ .min(MIN_BARRIER_SIGNALS, formErrors.atLeastOne)
+ .max(MAX_BARRIER_SIGNALS, formErrors.atMostTenThousand),
})
export type PlatformConfiguration = z.infer<typeof PlatformConfiguration>
-export const UpdatePlatformConfigurationRequestBody = PlatformConfiguration.pick({
+export const PlatformConfigurationSettings = PlatformConfiguration.pick({
isProductTelemetryEnabled: true,
isInfraSetupTelemetryEnabled: true,
-}).partial()
+ maxBarrierSignals: true,
+})
+export type PlatformConfigurationSettings = z.infer<typeof PlatformConfigurationSettings>
+
+export const UpdatePlatformConfigurationRequestBody = PlatformConfigurationSettings.partial()
export type UpdatePlatformConfigurationRequestBody = z.infer<typeof UpdatePlatformConfigurationRequestBody>
+
+export const maxBarrierSignalsBounds = {
+ min: MIN_BARRIER_SIGNALS,
+ max: MAX_BARRIER_SIGNALS,
+}
diff --git a/packages/core/utils/src/lib/form-errors.ts b/packages/core/utils/src/lib/form-errors.ts
index d4f1545d740..ffd87bbfb5d 100644
--- a/packages/core/utils/src/lib/form-errors.ts
+++ b/packages/core/utils/src/lib/form-errors.ts
@@ -8,6 +8,9 @@ export const formErrors = {
messageRequiresContentOrFiles: 'messageRequiresContentOrFiles',
agentConfigTooLarge: 'agentConfigTooLarge',
invalidGcpResourceId: 'invalidGcpResourceId',
+ atLeastOne: 'atLeastOne',
+ wholeNumber: 'wholeNumber',
+ atMostTenThousand: 'atMostTenThousand',
} as const
export const SAFE_EXTERNAL_ID_PATTERN = /^(?!\.{1,2}$)[A-Za-z0-9._-]{1,128}$/
diff --git a/packages/server/api/src/app/database/migration/postgres/1845000000000-AddMaxBarrierSignalsToPlatformConfiguration.ts b/packages/server/api/src/app/database/migration/postgres/1845000000000-AddMaxBarrierSignalsToPlatformConfiguration.ts
new file mode 100644
index 00000000000..a0ce699e6bd
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/1845000000000-AddMaxBarrierSignalsToPlatformConfiguration.ts
@@ -0,0 +1,23 @@
+import { QueryRunner } from 'typeorm'
+import { Migration } from '../../migration'
+
+export class AddMaxBarrierSignalsToPlatformConfiguration1845000000000 implements Migration {
+ name = 'AddMaxBarrierSignalsToPlatformConfiguration1845000000000'
+ breaking = false
+ release = '0.91.0'
+ transaction = true
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`
+ ALTER TABLE "platform_configuration"
+ ADD COLUMN "maxBarrierSignals" integer NOT NULL DEFAULT 10000
+ `)
+ }
+
+ public async down(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`
+ ALTER TABLE "platform_configuration"
+ DROP COLUMN "maxBarrierSignals"
+ `)
+ }
+}
diff --git a/packages/server/api/src/app/database/migration/postgres/1847000000000-IndexWaitpointLookups.ts b/packages/server/api/src/app/database/migration/postgres/1847000000000-IndexWaitpointLookups.ts
new file mode 100644
index 00000000000..4c58c59ce1a
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/1847000000000-IndexWaitpointLookups.ts
@@ -0,0 +1,40 @@
+import { QueryRunner } from 'typeorm'
+import { system } from '../../../helper/system/system'
+import { AppSystemProp } from '../../../helper/system/system-props'
+import { DatabaseType } from '../../database-type'
+import { Migration } from '../../migration'
+
+export class IndexWaitpointLookups1847000000000 implements Migration {
+ name = 'IndexWaitpointLookups1847000000000'
+ breaking = false
+ release = '0.92.0'
+ transaction = false
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`DROP INDEX ${concurrently()} IF EXISTS "idx_waitpoint_signal_ref_id"`)
+ await queryRunner.query(`
+ CREATE INDEX ${concurrently()} IF NOT EXISTS "idx_waitpoint_signal_ref_id"
+ ON "waitpoint_signal" ("refId", "projectId")
+ `)
+ await queryRunner.query(`
+ CREATE INDEX ${concurrently()} IF NOT EXISTS "idx_waitpoint_signal_waitpoint_id_project_id_sequence"
+ ON "waitpoint_signal" ("waitpointId", "projectId", "sequence", "created")
+ `)
+ await queryRunner.query(`
+ CREATE INDEX ${concurrently()} IF NOT EXISTS "idx_waitpoint_flow_run_id_status_created"
+ ON "waitpoint" ("flowRunId", "status", "created")
+ `)
+ }
+
+ public async down(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`DROP INDEX ${concurrently()} IF EXISTS "idx_waitpoint_flow_run_id_status_created"`)
+ await queryRunner.query(`DROP INDEX ${concurrently()} IF EXISTS "idx_waitpoint_signal_waitpoint_id_project_id_sequence"`)
+ await queryRunner.query(`DROP INDEX ${concurrently()} IF EXISTS "idx_waitpoint_signal_ref_id"`)
+ await queryRunner.query(`
+ CREATE INDEX ${concurrently()} IF NOT EXISTS "idx_waitpoint_signal_ref_id"
+ ON "waitpoint_signal" ("refId")
+ `)
+ }
+}
+
+const concurrently = (): string => system.get(AppSystemProp.DB_TYPE) === DatabaseType.PGLITE ? '' : 'CONCURRENTLY'
diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.ts
index 1a62566fef1..3c2826766b9 100644
--- a/packages/server/api/src/app/database/postgres-connection.ts
+++ b/packages/server/api/src/app/database/postgres-connection.ts
@@ -437,7 +437,9 @@ import { AddPlatformConfiguration1841000000000 } from './migration/postgres/1841
import { AddAgentRunsIndex1842000000000 } from './migration/postgres/1842000000000-AddAgentRunsIndex'
import { AddAgentConversationFlowRunId1843000000000 } from './migration/postgres/1843000000000-AddAgentConversationFlowRunId'
import { AddMcpActivity1844000000000 } from './migration/postgres/1844000000000-AddMcpActivity'
+import { AddMaxBarrierSignalsToPlatformConfiguration1845000000000 } from './migration/postgres/1845000000000-AddMaxBarrierSignalsToPlatformConfiguration'
import { NullOrphanFolderReferences1846000000000 } from './migration/postgres/1846000000000-NullOrphanFolderReferences'
+import { IndexWaitpointLookups1847000000000 } from './migration/postgres/1847000000000-IndexWaitpointLookups'
import { AddAgentConversationAiCredits1850000000000 } from './migration/postgres/1850000000000-AddAgentConversationAiCredits'
import { AddMcpOAuthCodeNonce1851000000000 } from './migration/postgres/1851000000000-AddMcpOAuthCodeNonce'
import { FlowExternalIdUniqueIgnoresDeleting1855000000000 } from './migration/postgres/1855000000000-FlowExternalIdUniqueIgnoresDeleting'
@@ -894,7 +896,9 @@ export const getMigrations = (): (new () => Migration)[] => {
AddAgentRunsIndex1842000000000,
AddAgentConversationFlowRunId1843000000000,
AddMcpActivity1844000000000,
+ AddMaxBarrierSignalsToPlatformConfiguration1845000000000,
NullOrphanFolderReferences1846000000000,
+ IndexWaitpointLookups1847000000000,
AddAgentConversationAiCredits1850000000000,
AddMcpOAuthCodeNonce1851000000000,
FlowExternalIdUniqueIgnoresDeleting1855000000000,
diff --git a/packages/server/api/src/app/ee/agent/agent-run-controller.ts b/packages/server/api/src/app/ee/agent/agent-run-controller.ts
index 53dee9b6745..e94d4d13927 100644
--- a/packages/server/api/src/app/ee/agent/agent-run-controller.ts
+++ b/packages/server/api/src/app/ee/agent/agent-run-controller.ts
@@ -124,7 +124,7 @@ async function resolvePublishedAgent({ projectId, externalId, flowRunId, waitpoi
log: FastifyBaseLogger
}): Promise<{ id: string, config: AgentConfig }> {
const flowRun = await flowRunService(log).getOneOrThrow({ id: flowRunId, projectId })
- const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId })
+ const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId, projectId })
if (isNil(waitpoint)) {
throw new ActivepiecesError({ code: ErrorCode.VALIDATION, params: { message: 'That waitpoint does not belong to this run, so there is no step to run an agent for' } })
}
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
index ce8560d1069..06971fdeab4 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
@@ -507,7 +507,7 @@ async function cancelSingleRun(log: FastifyBaseLogger, flowRun: FlowRun, platfor
timeoutInSeconds: 30,
fn: async () => {
await jobQueue(log).removeAllFlowRunJobs({ flowRunId: flowRun.id, platformId, projectId: flowRun.projectId })
- await waitpointService(log).deleteByFlowRunId(flowRun.id)
+ await waitpointService(log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: flowRun.projectId })
await runsMetadataQueue(log).add({
id: flowRun.id,
projectId: flowRun.projectId,
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-side-effects.ts b/packages/server/api/src/app/flows/flow-run/flow-run-side-effects.ts
index 70a2173c1b0..cefe5f747a0 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-side-effects.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-side-effects.ts
@@ -16,7 +16,7 @@ export const flowRunSideEffects = (log: FastifyBaseLogger) => ({
})) {
return
}
- await waitpointService(log).deleteByFlowRunId(flowRun.id)
+ await waitpointService(log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: flowRun.projectId })
await flowRunHooks(log).onFinish(flowRun)
applicationEvents(log).sendWorkerEvent({
projectId: flowRun.projectId,
diff --git a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
index 2759eddc806..f76b6b24dca 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
@@ -10,7 +10,6 @@ import { AppSystemProp } from '../../helper/system/system-props'
import { projectService } from '../../project/project-service'
import { resumeService } from '../../waitpoints/resume-service'
import { waitpointService } from '../../waitpoints/waitpoint-service'
-import { WaitpointStatus } from '../../waitpoints/waitpoint-types'
import { legacyRedisMetadataKey, QueueName, redisMetadataKey, runsMetadataDeduplicationId, RunsMetadataJobData, RunsMetadataQueueConfig, runsMetadataQueueFactory, RunsMetadataUpsertData } from '../../workers/job'
import { jobFailureLogger } from '../../workers/job-queue/job-failure-logger'
import { flowService } from '../flow/flow.service'
@@ -139,14 +138,12 @@ export const runsMetadataQueue = (log: FastifyBaseLogger) => ({
}
if (savedFlowRun.status === FlowRunStatus.PAUSED) {
- const latestWaitpoint = await waitpointService(log).getByFlowRunId(savedFlowRun.id)
- const isPreCompleted = !isNil(latestWaitpoint)
- && latestWaitpoint.status === WaitpointStatus.COMPLETED
- if (isPreCompleted) {
- await resumeService(log).resumeFromWaitpointWithoutLock({
+ const undeliveredWaitpoint = await waitpointService(log).findUndeliveredCompletedWaitpoint({ flowRunId: savedFlowRun.id, projectId: savedFlowRun.projectId })
+ if (!isNil(undeliveredWaitpoint)) {
+ await resumeService(log).resumeTrustedWithoutLock({
flowRunId: savedFlowRun.id,
- waitpointId: latestWaitpoint.id,
- resumePayload: latestWaitpoint.resumePayload,
+ waitpointId: undeliveredWaitpoint.id,
+ resumePayload: undeliveredWaitpoint.resumePayload,
})
}
}
@@ -276,7 +273,7 @@ export async function markParentRunAsFailed({
queryParams: {},
}
- const existingWaitpoint = await waitpointService(log).getByFlowRunId(parentRunId)
+ const existingWaitpoint = await waitpointService(log).findSubflowWaitpoint({ flowRunId: parentRunId, projectId: flowRun.projectId })
const result = await waitpointService(log).complete({
flowRunId: parentRunId,
projectId: flowRun.projectId,
diff --git a/packages/server/api/src/app/helper/system-validator.ts b/packages/server/api/src/app/helper/system-validator.ts
index 6b0c97bc583..2f148c38751 100644
--- a/packages/server/api/src/app/helper/system-validator.ts
+++ b/packages/server/api/src/app/helper/system-validator.ts
@@ -1,6 +1,6 @@
import { inspect } from 'util'
import { isNil } from '@activepieces/core-utils'
-import { ApEdition, ApEnvironment, DefaultProjectRole, ExecutionMode, FileLocation, NetworkMode, PieceSyncMode } from '@activepieces/shared'
+import { ApEdition, ApEnvironment, DefaultProjectRole, ExecutionMode, FileLocation, maxBarrierSignalsBounds, NetworkMode, PieceSyncMode } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { DatabaseType } from '../database/database-type'
import { RedisType } from '../database/redis/types'
@@ -39,6 +39,14 @@ function positiveFiniteNumberValidator(value: string | undefined) {
return isValid ? true : 'Value must be a finite number greater than zero'
}
+function boundedNumberValidator({ min, max }: { min: number, max: number }) {
+ return (value: string | undefined) => {
+ const parsed = Number(value)
+ const isValid = !isNil(value) && Number.isInteger(parsed) && parsed >= min && parsed <= max
+ return isValid ? true : `Value must be a whole number between ${min} and ${max}`
+ }
+}
+
function stringValidator(value: string) {
const isValid = typeof value === 'string' && value.length > 0
return isValid ? true : 'Value must be a non-empty string'
@@ -75,6 +83,7 @@ const systemPropValidators: {
[AppSystemProp.EVENT_DESTINATION_TIMEOUT_SECONDS]: numberValidator,
[AppSystemProp.PAUSED_FLOW_TIMEOUT_DAYS]: numberValidator,
[AppSystemProp.APP_WEBHOOK_SECRETS]: stringValidator,
+ [AppSystemProp.MAX_BARRIER_SIGNALS]: boundedNumberValidator(maxBarrierSignalsBounds),
[AppSystemProp.MAX_FILE_SIZE_MB]: numberValidator,
[AppSystemProp.AI_CREDIT_USD_VALUE]: positiveFiniteNumberValidator,
[AppSystemProp.MAX_FLOW_RUN_LOG_SIZE_MB]: numberValidator,
diff --git a/packages/server/api/src/app/helper/system/system-props.ts b/packages/server/api/src/app/helper/system/system-props.ts
index 23978e21483..c1c2086e283 100644
--- a/packages/server/api/src/app/helper/system/system-props.ts
+++ b/packages/server/api/src/app/helper/system/system-props.ts
@@ -57,6 +57,7 @@ export enum AppSystemProp {
LOKI_PASSWORD = 'LOKI_PASSWORD',
LOKI_URL = 'LOKI_URL',
LOKI_USERNAME = 'LOKI_USERNAME',
+ MAX_BARRIER_SIGNALS = 'MAX_BARRIER_SIGNALS',
MAX_FIELDS_PER_TABLE = 'MAX_FIELDS_PER_TABLE',
MAX_FILE_SIZE_MB = 'MAX_FILE_SIZE_MB',
AI_CREDIT_USD_VALUE = 'AI_CREDIT_USD_VALUE',
diff --git a/packages/server/api/src/app/helper/system/system.ts b/packages/server/api/src/app/helper/system/system.ts
index 45ae8d697c2..16e33a073cd 100644
--- a/packages/server/api/src/app/helper/system/system.ts
+++ b/packages/server/api/src/app/helper/system/system.ts
@@ -36,6 +36,7 @@ const systemPropDefaultValues: Partial<Record<SystemProp, string>> = {
[AppSystemProp.LOG_LEVEL]: 'info',
[AppSystemProp.LOG_PRETTY]: 'false',
[AppSystemProp.S3_USE_SIGNED_URLS]: 'false',
+ [AppSystemProp.MAX_BARRIER_SIGNALS]: '10000',
[AppSystemProp.MAX_FILE_SIZE_MB]: '25',
[AppSystemProp.AI_CREDIT_USD_VALUE]: '0.0005',
[AppSystemProp.MAX_FLOW_RUN_LOG_SIZE_MB]: '50',
diff --git a/packages/server/api/src/app/platform/platform-configuration.controller.ts b/packages/server/api/src/app/platform/platform-configuration.controller.ts
index 588cb794f72..dfeb73aa126 100644
--- a/packages/server/api/src/app/platform/platform-configuration.controller.ts
+++ b/packages/server/api/src/app/platform/platform-configuration.controller.ts
@@ -16,6 +16,7 @@ export const platformConfigurationController: FastifyPluginAsyncZod = async (app
platformId: request.principal.platform.id,
isProductTelemetryEnabled: request.body.isProductTelemetryEnabled,
isInfraSetupTelemetryEnabled: request.body.isInfraSetupTelemetryEnabled,
+ maxBarrierSignals: request.body.maxBarrierSignals,
})
})
}
diff --git a/packages/server/api/src/app/platform/platform-configuration.entity.ts b/packages/server/api/src/app/platform/platform-configuration.entity.ts
index 51d71f49b12..fc31eed0245 100644
--- a/packages/server/api/src/app/platform/platform-configuration.entity.ts
+++ b/packages/server/api/src/app/platform/platform-configuration.entity.ts
@@ -24,6 +24,11 @@ export const PlatformConfigurationEntity = new EntitySchema<PlatformConfiguratio
nullable: false,
default: true,
},
+ maxBarrierSignals: {
+ type: Number,
+ nullable: false,
+ default: 10000,
+ },
},
indices: [
{
diff --git a/packages/server/api/src/app/platform/platform-configuration.service.ts b/packages/server/api/src/app/platform/platform-configuration.service.ts
index a982af04d0a..b711a384f65 100644
--- a/packages/server/api/src/app/platform/platform-configuration.service.ts
+++ b/packages/server/api/src/app/platform/platform-configuration.service.ts
@@ -1,5 +1,5 @@
import { apId, chunk, isEmpty, isNil, spreadIfNotUndefined } from '@activepieces/core-utils'
-import { ApEdition, PlatformConfiguration, UpdatePlatformConfigurationRequestBody } from '@activepieces/shared'
+import { ApEdition, maxBarrierSignalsBounds, PlatformConfiguration, UpdatePlatformConfigurationRequestBody } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { In } from 'typeorm'
import { repoFactory } from '../core/db/repo-factory'
@@ -41,6 +41,14 @@ export const platformConfigurationService = (log: FastifyBaseLogger) => ({
return configuration.isProductTelemetryEnabled
},
+ async maxBarrierSignals({ platformId }: GetOrCreateParams): Promise<number> {
+ if (system.getEdition() === ApEdition.CLOUD) {
+ return clampBarrierSignalCap(system.getNumberOrThrow(AppSystemProp.MAX_BARRIER_SIGNALS))
+ }
+ const configuration = await this.getOrCreateForPlatform({ platformId })
+ return configuration.maxBarrierSignals
+ },
+
async filterProjectsWithProductTelemetryEnabled({ projectIds }: FilterProjectsParams): Promise<string[]> {
if (projectIds.length === 0) {
return []
@@ -83,11 +91,12 @@ export const platformConfigurationService = (log: FastifyBaseLogger) => ({
return platformIds.filter((platformId) => !optedOutIds.has(platformId))
},
- async update({ platformId, isProductTelemetryEnabled, isInfraSetupTelemetryEnabled }: UpdateParams): Promise<PlatformConfiguration> {
+ async update({ platformId, isProductTelemetryEnabled, isInfraSetupTelemetryEnabled, maxBarrierSignals }: UpdateParams): Promise<PlatformConfiguration> {
await this.getOrCreateForPlatform({ platformId })
const patch = {
...spreadIfNotUndefined('isProductTelemetryEnabled', isProductTelemetryEnabled),
...spreadIfNotUndefined('isInfraSetupTelemetryEnabled', isInfraSetupTelemetryEnabled),
+ ...spreadIfNotUndefined('maxBarrierSignals', maxBarrierSignals),
}
if (!isEmpty(patch)) {
await platformConfigurationRepo().update({ platformId }, patch)
@@ -98,13 +107,20 @@ export const platformConfigurationService = (log: FastifyBaseLogger) => ({
async function createInitialConfiguration({ platformId }: GetOrCreateParams): Promise<PlatformConfiguration> {
const isProductTelemetryEnabled = system.getBoolean(AppSystemProp.TELEMETRY_ENABLED)
+ const configuredMaxBarrierSignals = system.getNumber(AppSystemProp.MAX_BARRIER_SIGNALS)
+ const maxBarrierSignals = isNil(configuredMaxBarrierSignals) ? undefined : clampBarrierSignalCap(configuredMaxBarrierSignals)
return platformConfigurationRepo().save({
id: apId(),
platformId,
...spreadIfNotUndefined('isProductTelemetryEnabled', isProductTelemetryEnabled),
+ ...spreadIfNotUndefined('maxBarrierSignals', maxBarrierSignals),
})
}
+function clampBarrierSignalCap(configured: number): number {
+ return Math.min(Math.max(configured, maxBarrierSignalsBounds.min), maxBarrierSignalsBounds.max)
+}
+
type GetOrCreateParams = {
platformId: string
}
diff --git a/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts b/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts
new file mode 100644
index 00000000000..d357b8d2f76
--- /dev/null
+++ b/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts
@@ -0,0 +1,74 @@
+import { apDayjsDuration } from '@activepieces/server-utils'
+import { Queue } from 'bullmq'
+import Redis from 'ioredis'
+import { QueueName } from '../workers/job'
+
+export const evaluationDedupKey = (barrierId: string): string => `evaluate-${barrierId}`
+
+export const barrierQueueFactory = ({ createRedisConnection }: BarrierQueueFactoryParams) => {
+ let queueInstance: Queue<BarrierJobData> | undefined = undefined
+
+ const requireQueue = (): Queue<BarrierJobData> => {
+ if (!queueInstance) {
+ throw new Error('Barrier queue not initialized')
+ }
+ return queueInstance
+ }
+
+ return {
+ async init(config: BarrierQueueConfig): Promise<void> {
+ queueInstance = new Queue<BarrierJobData>(QueueName.BARRIER_EVALUATION, {
+ connection: await createRedisConnection(),
+ defaultJobOptions: {
+ attempts: 5,
+ backoff: {
+ type: 'exponential',
+ delay: apDayjsDuration(8, 'minute').asMilliseconds(),
+ },
+ removeOnComplete: true,
+ removeOnFail: {
+ age: apDayjsDuration(config.redisFailedJobRetentionDays, 'day').asSeconds(),
+ count: config.redisFailedJobRetentionMaxCount,
+ },
+ },
+ })
+ await queueInstance.waitUntilReady()
+ },
+
+ async enqueueEvaluation(params: BarrierJobData): Promise<void> {
+ await requireQueue().add(BarrierJobName.EVALUATE, params, {
+ deduplication: { id: evaluationDedupKey(params.barrierId) },
+ })
+ },
+
+ async clearEvaluationDedupKey(barrierId: string): Promise<void> {
+ await requireQueue().removeDeduplicationKey(evaluationDedupKey(barrierId))
+ },
+
+ get(): Queue<BarrierJobData> {
+ return requireQueue()
+ },
+
+ isInitialized(): boolean {
+ return queueInstance !== undefined
+ },
+ }
+}
+
+export enum BarrierJobName {
+ EVALUATE = 'evaluate-barrier',
+}
+
+export type BarrierJobData = {
+ barrierId: string
+ projectId: string
+}
+
+export type BarrierQueueConfig = {
+ redisFailedJobRetentionDays: number
+ redisFailedJobRetentionMaxCount: number
+}
+
+type BarrierQueueFactoryParams = {
+ createRedisConnection: () => Promise<Redis>
+}
diff --git a/packages/server/api/src/app/waitpoints/barrier-queue.ts b/packages/server/api/src/app/waitpoints/barrier-queue.ts
new file mode 100644
index 00000000000..a920e488912
--- /dev/null
+++ b/packages/server/api/src/app/waitpoints/barrier-queue.ts
@@ -0,0 +1,85 @@
+import { createLogger, wideEvent } from '@activepieces/server-utils'
+import { Job, Queue, Worker } from 'bullmq'
+import { FastifyBaseLogger } from 'fastify'
+import { redisConnections } from '../database/redis-connections'
+import { system } from '../helper/system/system'
+import { AppSystemProp } from '../helper/system/system-props'
+import { QueueName } from '../workers/job'
+import { BarrierJobData, BarrierJobName, BarrierQueueConfig, barrierQueueFactory } from './barrier-queue-factory'
+import { barrierService } from './barrier-service'
+
+let barrierWorker: Worker<BarrierJobData> | undefined = undefined
+
+const barrierQueueInstance = barrierQueueFactory({ createRedisConnection: redisConnections.create })
+
+export const barrierQueue = (log: FastifyBaseLogger) => ({
+ async init(): Promise<void> {
+ const config: BarrierQueueConfig = {
+ redisFailedJobRetentionDays: system.getNumberOrThrow(AppSystemProp.REDIS_FAILED_JOB_RETENTION_DAYS),
+ redisFailedJobRetentionMaxCount: system.getNumberOrThrow(AppSystemProp.REDIS_FAILED_JOB_RETENTION_MAX_COUNT),
+ }
+ await barrierQueueInstance.init(config)
+ barrierWorker = new Worker<BarrierJobData>(
+ QueueName.BARRIER_EVALUATION,
+ async (job) => {
+ const jobLogger = createLogger({
+ event: 'barrier.job',
+ job: { id: job.id, type: job.name },
+ fanIn: { barrierId: job.data.barrierId },
+ })
+ return wideEvent.run({
+ logger: jobLogger,
+ fn: () => processBarrierJob({ job, log })
+ .catch((error: unknown) => {
+ wideEvent.error(error)
+ throw error
+ })
+ .finally(() => jobLogger.emit()),
+ })
+ },
+ {
+ connection: await redisConnections.create(),
+ concurrency: system.getNumberOrThrow(AppSystemProp.RUNS_METADATA_UPDATE_CONCURRENCY),
+ autorun: true,
+ },
+ )
+ await barrierWorker.waitUntilReady()
+ },
+
+ async enqueueEvaluation(params: BarrierJobData): Promise<void> {
+ await barrierQueueInstance.enqueueEvaluation(params)
+ },
+
+ async clearEvaluationDedupKey(barrierId: string): Promise<void> {
+ await barrierQueueInstance.clearEvaluationDedupKey(barrierId)
+ },
+
+ get(): Queue<BarrierJobData> {
+ return barrierQueueInstance.get()
+ },
+
+ async close(): Promise<void> {
+ if (barrierQueueInstance.isInitialized()) {
+ await barrierQueueInstance.get().close()
+ }
+ if (barrierWorker) {
+ await barrierWorker.close()
+ }
+ },
+})
+
+async function processBarrierJob({ job, log }: ProcessBarrierJobParams): Promise<void> {
+ switch (job.name) {
+ case BarrierJobName.EVALUATE:
+ await barrierQueueInstance.clearEvaluationDedupKey(job.data.barrierId)
+ await barrierService(log).releaseIfReady({ barrierId: job.data.barrierId, projectId: job.data.projectId })
+ return
+ default:
+ log.warn({ job: { id: job.id, type: job.name } }, '[barrierQueue#worker] Unknown barrier job name, dropping it')
+ }
+}
+
+type ProcessBarrierJobParams = {
+ job: Job<BarrierJobData>
+ log: FastifyBaseLogger
+}
diff --git a/packages/server/api/src/app/waitpoints/barrier-service.ts b/packages/server/api/src/app/waitpoints/barrier-service.ts
new file mode 100644
index 00000000000..7431fc6406e
--- /dev/null
+++ b/packages/server/api/src/app/waitpoints/barrier-service.ts
@@ -0,0 +1,342 @@
+import { apId, chunk, isNil, sanitizeObjectForPostgresql, tryCatch } from '@activepieces/core-utils'
+import { wideEvent } from '@activepieces/server-utils'
+import { ActivepiecesError, BarrierPolicy, barrierReleasesOnLastPendingSignal, BarrierSignalCounts, BarrierSignalStatus, BarrierSummary, ErrorCode, MAX_INLINE_BARRIER_SIGNALS, PauseType, RespondResponse, shouldReleaseBarrier, WaitpointVersion } from '@activepieces/shared'
+import dayjs from 'dayjs'
+import { FastifyBaseLogger } from 'fastify'
+import { EntityManager } from 'typeorm'
+import { repoFactory } from '../core/db/repo-factory'
+import { transaction } from '../core/db/transaction'
+import { system } from '../helper/system/system'
+import { AppSystemProp } from '../helper/system/system-props'
+import { platformConfigurationService } from '../platform/platform-configuration.service'
+import { barrierQueue } from './barrier-queue'
+import { resumeService } from './resume-service'
+import { WaitpointEntity } from './waitpoint-entity'
+import { WaitpointSignalEntity } from './waitpoint-signal-entity'
+import { Waitpoint, WaitpointSignal, WaitpointStatus } from './waitpoint-types'
+
+const waitpointRepo = repoFactory(WaitpointEntity)
+const signalRepo = repoFactory(WaitpointSignalEntity)
+
+export const barrierService = (log: FastifyBaseLogger) => ({
+ async create(params: CreateBarrierParams): Promise<CreateBarrierResult> {
+ const labels = params.signalLabels ?? []
+ await assertSignalCountWithinLimit({ signalCount: labels.length, platformId: params.platformId, log })
+
+ const creation = await transaction(async (entityManager) => {
+ const repo = waitpointRepo(entityManager)
+ const existing = await repo.findOneBy({ flowRunId: params.flowRunId, projectId: params.projectId, stepName: params.stepName })
+ if (!isNil(existing) && existing.status === WaitpointStatus.PENDING) {
+ log.info({ flowRun: { id: params.flowRunId }, waitpoint: { id: existing.id } }, '[barrierService#create] Barrier already open for this step, reusing it')
+ return { inserted: false, barrier: existing, signals: await signalRepo(entityManager).findBy({ waitpointId: existing.id, projectId: params.projectId }) }
+ }
+ if (!isNil(existing)) {
+ await repo.delete({ id: existing.id, projectId: params.projectId })
+ }
+ const now = dayjs().toISOString()
+ const barrier: Waitpoint = {
+ id: apId(),
+ created: now,
+ updated: now,
+ flowRunId: params.flowRunId,
+ projectId: params.projectId,
+ stepName: params.stepName,
+ type: PauseType.BARRIER,
+ version: params.version,
+ status: WaitpointStatus.PENDING,
+ resumeDateTime: defaultBarrierDeadline(),
+ responseToSend: params.responseToSend ?? null,
+ workerHandlerId: params.workerHandlerId ?? null,
+ httpRequestId: params.httpRequestId ?? null,
+ resumePayload: null,
+ sealed: true,
+ policy: params.policy ?? null,
+ }
+ await repo.createQueryBuilder().insert().into('waitpoint').values(barrier).execute()
+ const signals = buildPendingSignals({ barrierId: barrier.id, projectId: params.projectId, labels })
+ for (const rows of chunk(signals, SIGNAL_INSERT_BATCH_SIZE)) {
+ await signalRepo(entityManager).createQueryBuilder().insert().into('waitpoint_signal').values(rows).execute()
+ }
+ return { inserted: true, barrier, signals }
+ })
+
+ if (creation.inserted) {
+ await barrierQueue(log).enqueueEvaluation({ barrierId: creation.barrier.id, projectId: params.projectId })
+ }
+
+ return { barrier: creation.barrier, signals: creation.signals, signalCount: creation.signals.length }
+ },
+
+ async findById({ barrierId, projectId }: FindByIdParams): Promise<Waitpoint | null> {
+ return waitpointRepo().findOneBy({ id: barrierId, projectId, type: PauseType.BARRIER })
+ },
+
+ async findSignalById({ signalId, projectId }: FindSignalByIdParams): Promise<WaitpointSignal | null> {
+ return signalRepo().findOneBy({ id: signalId, projectId })
+ },
+
+ async receiveSignal(params: ReceiveSignalParams): Promise<WaitpointSignal | null> {
+ if (isNil(params.signalId) && isNil(params.refId)) {
+ return null
+ }
+ const signal = isNil(params.signalId)
+ ? await signalRepo().findOneBy({ refId: params.refId, projectId: params.projectId })
+ : await signalRepo().findOneBy({ id: params.signalId, projectId: params.projectId })
+ if (isNil(signal)) {
+ return null
+ }
+ await signalRepo().save({
+ ...signal,
+ status: params.status,
+ result: isNil(params.result) ? null : sanitizeObjectForPostgresql(params.result),
+ })
+ await barrierQueue(log).enqueueEvaluation({ barrierId: signal.waitpointId, projectId: signal.projectId })
+ return signal
+ },
+
+ async releaseIfReady({ barrierId, projectId }: ReleaseIfReadyParams): Promise<void> {
+ const barrier = await waitpointRepo().findOneBy({ id: barrierId, projectId, type: PauseType.BARRIER })
+ if (isNil(barrier)) {
+ return
+ }
+ if (barrier.status === WaitpointStatus.COMPLETED) {
+ await redeliverClosedBarrier({ barrier, log })
+ return
+ }
+ if (barrier.status !== WaitpointStatus.PENDING) {
+ return
+ }
+ if (!await isReadyToRelease({ barrier, projectId })) {
+ return
+ }
+ await this.release({ barrier, timedOut: false, releaseReason: 'predicate' })
+ },
+
+ async release({ barrier, timedOut, releaseReason }: ReleaseParams): Promise<BarrierSummary | null> {
+ const summary = await closeBarrier({ barrier, timedOut })
+ if (isNil(summary)) {
+ log.info({ waitpoint: { id: barrier.id }, flowRun: { id: barrier.flowRunId } }, '[barrierService#release] Barrier was already closed by another release; leaving the resume to the release that closed it')
+ return null
+ }
+ wideEvent.set({
+ fanIn: {
+ barrierId: barrier.id,
+ signalCount: summary.total,
+ releaseReason,
+ stillRunning: summary.stillRunning,
+ },
+ })
+ await resumeService(log).resumeTrusted({
+ flowRunId: barrier.flowRunId,
+ waitpointId: barrier.id,
+ resumePayload: { body: summary, headers: {}, queryParams: {} },
+ })
+ return summary
+ },
+})
+
+function buildPendingSignals({ barrierId, projectId, labels }: BuildPendingSignalsParams): WaitpointSignal[] {
+ const now = new Date().toISOString()
+ return labels.map((label) => ({
+ id: apId(),
+ created: now,
+ updated: now,
+ waitpointId: barrierId,
+ projectId,
+ status: BarrierSignalStatus.PENDING,
+ refId: null,
+ sequence: null,
+ label,
+ result: null,
+ }))
+}
+
+async function closeBarrier({ barrier, timedOut }: CloseBarrierParams): Promise<BarrierSummary | null> {
+ return transaction(async (entityManager) => {
+ const repo = waitpointRepo(entityManager)
+ const lockedBarrier = await repo
+ .createQueryBuilder('waitpoint')
+ .setLock('pessimistic_write')
+ .where({ id: barrier.id, projectId: barrier.projectId, status: WaitpointStatus.PENDING })
+ .getOne()
+ if (isNil(lockedBarrier)) {
+ return null
+ }
+ const summary = await buildSummary({ barrierId: lockedBarrier.id, projectId: lockedBarrier.projectId, timedOut, entityManager })
+ await repo.save({
+ ...lockedBarrier,
+ status: WaitpointStatus.COMPLETED,
+ resumePayload: { body: summary, headers: {}, queryParams: {} },
+ })
+ await signalRepo(entityManager).delete({ waitpointId: lockedBarrier.id, projectId: lockedBarrier.projectId })
+ return summary
+ })
+}
+
+async function redeliverClosedBarrier({ barrier, log }: RedeliverClosedBarrierParams): Promise<void> {
+ log.warn({ waitpoint: { id: barrier.id }, flowRun: { id: barrier.flowRunId } }, '[barrierService#redeliverClosedBarrier] Barrier was closed but never delivered; re-dispatching the persisted summary')
+ const { error } = await tryCatch(() => resumeService(log).resumeTrusted({
+ flowRunId: barrier.flowRunId,
+ waitpointId: barrier.id,
+ resumePayload: barrier.resumePayload,
+ }))
+ if (!isNil(error)) {
+ log.error({ error, waitpoint: { id: barrier.id }, flowRun: { id: barrier.flowRunId } }, '[barrierService#redeliverClosedBarrier] Re-dispatching a closed barrier failed')
+ throw error
+ }
+}
+
+async function isReadyToRelease({ barrier, projectId }: IsReadyToReleaseParams): Promise<boolean> {
+ const { policy, sealed } = barrier
+ if (barrierReleasesOnLastPendingSignal({ policy, sealed })) {
+ return !await signalRepo().existsBy({ waitpointId: barrier.id, projectId, status: BarrierSignalStatus.PENDING })
+ }
+ const counts = await countSignalsByStatus({ barrierId: barrier.id, projectId })
+ return shouldReleaseBarrier({ policy, sealed, counts })
+}
+
+async function countSignalsByStatus({ barrierId, projectId, entityManager }: CountSignalsByStatusParams): Promise<BarrierSignalCounts> {
+ const rows = await signalRepo(entityManager)
+ .createQueryBuilder('signal')
+ .select('signal."status"', 'status')
+ .addSelect('COUNT(*)', 'count')
+ .where('signal."waitpointId" = :barrierId', { barrierId })
+ .andWhere('signal."projectId" = :projectId', { projectId })
+ .groupBy('signal."status"')
+ .getRawMany<{ status: BarrierSignalStatus, count: string }>()
+ return rows.reduce<BarrierSignalCounts>((accumulated, row) => ({ ...accumulated, [row.status]: Number(row.count) }), {})
+}
+
+async function buildSummary({ barrierId, projectId, timedOut, entityManager }: BuildSummaryParams): Promise<BarrierSummary> {
+ const counts = await countSignalsByStatus({ barrierId, projectId, entityManager })
+ const total = Object.values(counts).reduce((sum, count) => sum + count, 0)
+ const inlineSignals = total <= MAX_INLINE_BARRIER_SIGNALS
+ ? await signalRepo(entityManager).find({ where: { waitpointId: barrierId, projectId }, order: { sequence: 'ASC', created: 'ASC' } })
+ : null
+
+ return {
+ total,
+ succeeded: counts[BarrierSignalStatus.SUCCEEDED] ?? 0,
+ failed: counts[BarrierSignalStatus.FAILED] ?? 0,
+ rejected: counts[BarrierSignalStatus.REJECTED] ?? 0,
+ canceled: counts[BarrierSignalStatus.CANCELED] ?? 0,
+ notDispatched: counts[BarrierSignalStatus.NOT_DISPATCHED] ?? 0,
+ stillRunning: counts[BarrierSignalStatus.PENDING] ?? 0,
+ timedOut,
+ ...(isNil(inlineSignals)
+ ? { signalsTruncated: true }
+ : {
+ signals: inlineSignals.map((signal) => ({
+ sequence: signal.sequence,
+ label: signal.label,
+ outcome: signal.status,
+ result: signal.result ?? null,
+ runId: signal.refId,
+ })),
+ }),
+ }
+}
+
+async function assertSignalCountWithinLimit({ signalCount, platformId, log }: AssertSignalCountWithinLimitParams): Promise<void> {
+ const maxSignals = await platformConfigurationService(log).maxBarrierSignals({ platformId })
+ if (signalCount > maxSignals) {
+ throw new ActivepiecesError({
+ code: ErrorCode.VALIDATION,
+ params: { message: `This step waits on ${signalCount} things, which exceeds the maximum of ${maxSignals}. Wait on fewer things.` },
+ })
+ }
+}
+
+function defaultBarrierDeadline(): string {
+ const maxDurationInDays = system.getNumberOrThrow(AppSystemProp.PAUSED_FLOW_TIMEOUT_DAYS)
+ return dayjs().add(maxDurationInDays, 'day').toISOString()
+}
+
+const SIGNAL_INSERT_BATCH_SIZE = 500
+
+export type BarrierReleaseReason = 'predicate' | 'timeout'
+
+export type CreateBarrierParams = {
+ flowRunId: string
+ projectId: string
+ platformId: string
+ stepName: string
+ version: WaitpointVersion
+ responseToSend?: RespondResponse
+ workerHandlerId?: string
+ httpRequestId?: string
+ policy?: BarrierPolicy
+ signalLabels?: (string | null)[]
+}
+
+export type CreateBarrierResult = {
+ barrier: Waitpoint
+ signals: WaitpointSignal[]
+ signalCount: number
+}
+
+export type ReceiveSignalParams = {
+ signalId?: string
+ refId?: string
+ projectId: string
+ status: BarrierSignalStatus
+ result?: Record<string, unknown>
+}
+
+type FindSignalByIdParams = {
+ signalId: string
+ projectId: string
+}
+
+type ReleaseIfReadyParams = {
+ barrierId: string
+ projectId: string
+}
+
+type FindByIdParams = {
+ barrierId: string
+ projectId: string
+}
+
+type ReleaseParams = {
+ barrier: Waitpoint
+ timedOut: boolean
+ releaseReason: BarrierReleaseReason
+}
+
+type CloseBarrierParams = {
+ barrier: Waitpoint
+ timedOut: boolean
+}
+
+type RedeliverClosedBarrierParams = {
+ barrier: Waitpoint
+ log: FastifyBaseLogger
+}
+
+type IsReadyToReleaseParams = {
+ barrier: Waitpoint
+ projectId: string
+}
+
+type CountSignalsByStatusParams = {
+ barrierId: string
+ projectId: string
+ entityManager?: EntityManager
+}
+
+type BuildSummaryParams = CountSignalsByStatusParams & {
+ timedOut: boolean
+}
+
+type BuildPendingSignalsParams = {
+ barrierId: string
+ projectId: string
+ labels: (string | null)[]
+}
+
+type AssertSignalCountWithinLimitParams = {
+ signalCount: number
+ platformId: string
+ log: FastifyBaseLogger
+}
diff --git a/packages/server/api/src/app/waitpoints/resume-controller.ts b/packages/server/api/src/app/waitpoints/resume-controller.ts
index e1391948a21..f221de63cbd 100644
--- a/packages/server/api/src/app/waitpoints/resume-controller.ts
+++ b/packages/server/api/src/app/waitpoints/resume-controller.ts
@@ -54,7 +54,8 @@ export const resumeController: FastifyPluginAsyncZod = async (app) => {
app.all('/:id/requests/:requestId', V0ResumeFlowRunRequest, async (req, reply) => {
const headers = req.headers as Record<string, string>
const queryParams = req.query as Record<string, string>
- const waitpoint = await waitpointService(req.log).findPendingByVersion({ flowRunId: req.params.id, version: 'V0' })
+ const flowRun = await findFlowRunOrThrow(req.params.id)
+ const waitpoint = await waitpointService(req.log).findPendingByVersion({ flowRunId: flowRun.id, projectId: flowRun.projectId, version: 'V0' })
if (waitpoint) {
await handleAsyncResume({ flowRunId: req.params.id, waitpointId: waitpoint.id, body: req.body, headers, queryParams, log: req.log, reply })
}
@@ -69,7 +70,8 @@ export const resumeController: FastifyPluginAsyncZod = async (app) => {
app.all('/:id/requests/:requestId/sync', V0ResumeFlowRunRequest, async (req, reply) => {
const headers = req.headers as Record<string, string>
const queryParams = req.query as Record<string, string>
- const waitpoint = await waitpointService(req.log).findPendingByVersion({ flowRunId: req.params.id, version: 'V0' })
+ const flowRun = await findFlowRunOrThrow(req.params.id)
+ const waitpoint = await waitpointService(req.log).findPendingByVersion({ flowRunId: flowRun.id, projectId: flowRun.projectId, version: 'V0' })
if (waitpoint) {
await handleSyncResume({ flowRunId: req.params.id, waitpointId: waitpoint.id, body: req.body, headers, queryParams, log: req.log, reply, correlationId: waitpoint.workerHandlerId ?? waitpoint.id })
}
@@ -82,7 +84,7 @@ export const resumeController: FastifyPluginAsyncZod = async (app) => {
async function serveConfirmationPage({ flowRunId, waitpointId, url, queryParams, log, reply }: ConfirmationPageParams): Promise<void> {
const flowRun = await findFlowRunOrThrow(flowRunId)
const theme = await resolveResumePageTheme({ projectId: flowRun.projectId, log })
- const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId })
+ const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId, projectId: flowRun.projectId })
const isOpen = !isNil(waitpoint) && waitpoint.status === WaitpointStatus.PENDING && flowRun.status === FlowRunStatus.PAUSED
if (!isOpen) {
await replyWithHtml({ reply, html: Mustache.render(STATUS_HTML_TEMPLATE, buildThemeView({ theme, extra: { title: ALREADY_TITLE, message: ALREADY_MESSAGE, success: false } })) })
diff --git a/packages/server/api/src/app/waitpoints/resume-delay-handler.ts b/packages/server/api/src/app/waitpoints/resume-delay-handler.ts
index 922e1e3bab4..b04f0f3d436 100644
--- a/packages/server/api/src/app/waitpoints/resume-delay-handler.ts
+++ b/packages/server/api/src/app/waitpoints/resume-delay-handler.ts
@@ -24,7 +24,7 @@ export async function handleResumeDelayWaitpoint({ data, log }: HandleResumeDela
'[RESUME_DELAY_WAITPOINT] Flow not PAUSED, skipping')
return
}
- const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId: data.waitpointId, flowRunId: data.flowRunId })
+ const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId: data.waitpointId, flowRunId: data.flowRunId, projectId: flowRun.projectId })
if (isNil(waitpoint) || waitpoint.status !== WaitpointStatus.PENDING) {
log.info({ flowRun: { id: data.flowRunId }, waitpoint: { id: data.waitpointId }, waitpointStatus: waitpoint?.status },
'[RESUME_DELAY_WAITPOINT] Waitpoint no longer PENDING (stale timer from completed/deleted waitpoint), skipping')
diff --git a/packages/server/api/src/app/waitpoints/resume-service.ts b/packages/server/api/src/app/waitpoints/resume-service.ts
index 15413acf038..5c069fb1c5d 100644
--- a/packages/server/api/src/app/waitpoints/resume-service.ts
+++ b/packages/server/api/src/app/waitpoints/resume-service.ts
@@ -1,5 +1,5 @@
import { apId, FlowRunId, isNil, tryCatch } from '@activepieces/core-utils'
-import { EngineHttpResponse, ExecutionType, FlowRun, FlowRunStatus, isFlowRunStateTerminal, ResumeReason, RunEnvironment, StreamStepProgress } from '@activepieces/shared'
+import { EngineHttpResponse, ExecutionType, FlowRun, FlowRunStatus, isFlowRunStateTerminal, PauseType, ResumeReason, RunEnvironment, StreamStepProgress } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { StatusCodes } from 'http-status-codes'
import { distributedLock, distributedStore } from '../database/redis-connections'
@@ -7,20 +7,38 @@ import { addToQueue, findFlowRunOrThrow, flowRunService, WEBHOOK_TIMEOUT_MS } fr
import { flowRunSideEffects } from '../flows/flow-run/flow-run-side-effects'
import { projectService } from '../project/project-service'
import { engineResponseWatcher } from '../workers/engine-response-watcher'
+import { redisMetadataKey, RunsMetadataUpsertData } from '../workers/job'
import { waitpointService } from './waitpoint-service'
import { Waitpoint, WaitpointResumePayload, WaitpointStatus } from './waitpoint-types'
export const resumeService = (log: FastifyBaseLogger) => ({
async resumeFromWaitpoint({ flowRunId, waitpointId, resumePayload, workerHandlerId, httpRequestId }: ResumeFromWaitpointParams): Promise<ResumeFromWaitpointResult> {
+ const flowRun = await findFlowRunOrThrow(flowRunId)
+ const addressed = await addressedWaitpointIsBarrier({ flowRunId, waitpointId, projectId: flowRun.projectId, log })
+ const refuse = addressed === 'no-such-waitpoint'
+ ? await waitpointService(log).hasBarrier({ flowRunId, projectId: flowRun.projectId })
+ : addressed
+ if (refuse) {
+ log.warn({ flowRun: { id: flowRunId }, waitpoint: { id: waitpointId } }, '[resumeService#resumeFromWaitpoint] Refused an external resume of a barrier; only the barrier predicate and its deadline may release it')
+ return { flowRun, stale: true }
+ }
+ return this.resumeTrusted({ flowRunId, waitpointId, resumePayload, workerHandlerId, httpRequestId })
+ },
+
+ async resumeTrusted({ flowRunId, waitpointId, resumePayload, workerHandlerId, httpRequestId }: ResumeFromWaitpointParams): Promise<ResumeFromWaitpointResult> {
return distributedLock(log).runExclusive({
key: `runs_metadata_${flowRunId}`,
timeoutInSeconds: 30,
- fn: () => this.resumeFromWaitpointWithoutLock({ flowRunId, waitpointId, resumePayload, workerHandlerId, httpRequestId }),
+ fn: () => this.resumeTrustedWithoutLock({ flowRunId, waitpointId, resumePayload, workerHandlerId, httpRequestId }),
})
},
- async resumeFromWaitpointWithoutLock({ flowRunId, waitpointId, resumePayload, workerHandlerId, httpRequestId }: ResumeFromWaitpointParams): Promise<ResumeFromWaitpointResult> {
+ async resumeTrustedWithoutLock({ flowRunId, waitpointId, resumePayload, workerHandlerId, httpRequestId }: ResumeFromWaitpointParams): Promise<ResumeFromWaitpointResult> {
const flowRun = await findFlowRunOrThrow(flowRunId)
+ if (await hasPendingTerminalStatus({ flowRunId, projectId: flowRun.projectId })) {
+ log.warn({ flowRun: { id: flowRunId } }, '[resumeService#resumeTrustedWithoutLock] Refused a resume of a run whose terminal status is already queued')
+ return { flowRun, stale: true }
+ }
const processed = await waitpointService(log).handleResumeSignal({
flowRunId,
waitpointId,
@@ -43,20 +61,18 @@ export const resumeService = (log: FastifyBaseLogger) => ({
if (processed) {
const currentFlowRun = await findFlowRunOrThrow(flowRunId)
if (currentFlowRun.status === FlowRunStatus.PAUSED) {
- const latestWaitpoint = await waitpointService(log).getByFlowRunId(flowRunId)
- if (!isNil(latestWaitpoint) && latestWaitpoint.status === WaitpointStatus.COMPLETED) {
- log.info({ flowRun: { id: flowRunId } }, '[resumeService#resumeFromWaitpointWithoutLock] Race detected: metadata worker wrote PAUSED after callback completed waitpoint; consuming waitpoint and enqueuing resume')
- // Consume the stale COMPLETED waitpoint under the lock so it cannot
- // poison the next createForPause call on a subsequent loop iteration
+ const addressedWaitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId, projectId: currentFlowRun.projectId })
+ if (!isNil(addressedWaitpoint) && addressedWaitpoint.status === WaitpointStatus.COMPLETED) {
+ log.info({ flowRun: { id: flowRunId } }, '[resumeService#resumeTrustedWithoutLock] Race detected: metadata worker wrote PAUSED after callback completed waitpoint; consuming waitpoint and enqueuing resume')
+ await waitpointService(log).consume({ waitpoint: addressedWaitpoint })
await enqueueResume({
flowRun: currentFlowRun,
- waitpoint: latestWaitpoint,
+ waitpoint: addressedWaitpoint,
resumePayload: resumePayload ?? null,
workerHandlerId,
httpRequestId,
}, log)
- await recordWaitpointConsumed({ waitpointId: latestWaitpoint.id, log })
- await waitpointService(log).delete({ id: latestWaitpoint.id })
+ await recordWaitpointConsumed({ waitpointId: addressedWaitpoint.id, log })
}
}
}
@@ -65,9 +81,12 @@ export const resumeService = (log: FastifyBaseLogger) => ({
},
async waitpointWasConsumed({ waitpointId, flowRunId }: WaitpointWasConsumedParams): Promise<boolean> {
- const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId })
- if (!isNil(waitpoint)) {
- return waitpoint.status === WaitpointStatus.COMPLETED
+ const { data: flowRun } = await tryCatch(() => findFlowRunOrThrow(flowRunId))
+ if (!isNil(flowRun)) {
+ const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId, projectId: flowRun.projectId })
+ if (!isNil(waitpoint)) {
+ return waitpoint.status === WaitpointStatus.COMPLETED
+ }
}
const { data, error } = await tryCatch(() => distributedStore.get<boolean>(waitpointConsumedKey(waitpointId)))
if (error) {
@@ -78,12 +97,26 @@ export const resumeService = (log: FastifyBaseLogger) => ({
},
async legacyResume({ flowRunId, resumePayload, workerHandlerId }: LegacyResumeParams): Promise<ResumeFromWaitpointResult> {
- const flowRun = await findFlowRunOrThrow(flowRunId)
- if (flowRun.status !== FlowRunStatus.PAUSED) {
- return { flowRun, stale: true }
- }
- await enqueueResume({ flowRun, resumePayload, workerHandlerId }, log)
- return { flowRun, stale: false }
+ return distributedLock(log).runExclusive({
+ key: `runs_metadata_${flowRunId}`,
+ timeoutInSeconds: 30,
+ fn: async () => {
+ const flowRun = await findFlowRunOrThrow(flowRunId)
+ if (flowRun.status !== FlowRunStatus.PAUSED) {
+ return { flowRun, stale: true }
+ }
+ if (await hasPendingTerminalStatus({ flowRunId, projectId: flowRun.projectId })) {
+ log.warn({ flowRun: { id: flowRunId } }, '[resumeService#legacyResume] Refused a resume of a run whose terminal status is already queued')
+ return { flowRun, stale: true }
+ }
+ if (await waitpointService(log).hasBarrier({ flowRunId, projectId: flowRun.projectId })) {
+ log.warn({ flowRun: { id: flowRunId } }, '[resumeService#legacyResume] Refused an external resume of a barrier; only the barrier predicate and its deadline may release it')
+ return { flowRun, stale: true }
+ }
+ await enqueueResume({ flowRun, resumePayload, workerHandlerId }, log)
+ return { flowRun, stale: false }
+ },
+ })
},
async handleSyncResumeFlow({ runId, waitpointId, payload, correlationId }: HandleSyncResumeFlowParams): Promise<EngineHttpResponse> {
@@ -125,16 +158,42 @@ export const resumeService = (log: FastifyBaseLogger) => ({
},
async legacySyncResume({ runId, payload, correlationId }: LegacySyncResumeParams): Promise<EngineHttpResponse> {
- const flowRun = await findFlowRunOrThrow(runId)
- if (flowRun.status !== FlowRunStatus.PAUSED) {
- return {
- status: StatusCodes.CONFLICT,
- body: { message: 'Flow run is not paused', flowRunStatus: flowRun.status },
- headers: {},
- }
+ const refusal = await distributedLock(log).runExclusive({
+ key: `runs_metadata_${runId}`,
+ timeoutInSeconds: 30,
+ fn: async (): Promise<EngineHttpResponse | null> => {
+ const flowRun = await findFlowRunOrThrow(runId)
+ if (flowRun.status !== FlowRunStatus.PAUSED) {
+ return {
+ status: StatusCodes.CONFLICT,
+ body: { message: 'Flow run is not paused', flowRunStatus: flowRun.status },
+ headers: {},
+ }
+ }
+ if (await hasPendingTerminalStatus({ flowRunId: runId, projectId: flowRun.projectId })) {
+ log.warn({ flowRun: { id: runId } }, '[resumeService#legacySyncResume] Refused a resume of a run whose terminal status is already queued')
+ return {
+ status: StatusCodes.GONE,
+ body: { message: 'This link has expired. The action may have already been processed.' },
+ headers: {},
+ }
+ }
+ if (await waitpointService(log).hasBarrier({ flowRunId: runId, projectId: flowRun.projectId })) {
+ log.warn({ flowRun: { id: runId } }, '[resumeService#legacySyncResume] Refused an external resume of a barrier; only the barrier predicate and its deadline may release it')
+ return {
+ status: StatusCodes.GONE,
+ body: { message: 'This link has expired. The action may have already been processed.' },
+ headers: {},
+ }
+ }
+ const syncServerId = engineResponseWatcher(log).getServerId()
+ await enqueueResume({ flowRun, resumePayload: payload, workerHandlerId: syncServerId, httpRequestId: correlationId }, log)
+ return null
+ },
+ })
+ if (!isNil(refusal)) {
+ return refusal
}
- const syncServerId = engineResponseWatcher(log).getServerId()
- await enqueueResume({ flowRun, resumePayload: payload, workerHandlerId: syncServerId, httpRequestId: correlationId }, log)
return engineResponseWatcher(log).oneTimeListener<EngineHttpResponse>(correlationId, true, WEBHOOK_TIMEOUT_MS, {
status: StatusCodes.NO_CONTENT,
body: {},
@@ -143,6 +202,22 @@ export const resumeService = (log: FastifyBaseLogger) => ({
},
})
+async function hasPendingTerminalStatus({ flowRunId, projectId }: HasPendingTerminalStatusParams): Promise<boolean> {
+ const pending = await distributedStore.hgetJson<RunsMetadataUpsertData>(redisMetadataKey({ projectId, runId: flowRunId }))
+ if (isNil(pending) || isNil(pending.status)) {
+ return false
+ }
+ return isFlowRunStateTerminal({ status: pending.status, ignoreInternalError: false })
+}
+
+async function addressedWaitpointIsBarrier({ flowRunId, waitpointId, projectId, log }: AddressedWaitpointIsBarrierParams): Promise<AddressedWaitpointVerdict> {
+ const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId, projectId })
+ if (isNil(waitpoint)) {
+ return 'no-such-waitpoint'
+ }
+ return waitpoint.type === PauseType.BARRIER
+}
+
async function recordWaitpointConsumed({ waitpointId, log }: RecordWaitpointConsumedParams): Promise<void> {
const { error } = await tryCatch(() => distributedStore.put(waitpointConsumedKey(waitpointId), true, WAITPOINT_CONSUMED_TTL_SECONDS))
if (isNil(error)) {
@@ -214,6 +289,20 @@ type ResumeFromWaitpointParams = {
httpRequestId?: string
}
+type HasPendingTerminalStatusParams = {
+ flowRunId: FlowRunId
+ projectId: string
+}
+
+type AddressedWaitpointVerdict = boolean | 'no-such-waitpoint'
+
+type AddressedWaitpointIsBarrierParams = {
+ flowRunId: FlowRunId
+ waitpointId: string
+ projectId: string
+ log: FastifyBaseLogger
+}
+
type LegacyResumeParams = {
flowRunId: FlowRunId
resumePayload: WaitpointResumePayload
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-entity.ts b/packages/server/api/src/app/waitpoints/waitpoint-entity.ts
index 1527ce0c5eb..92ea40f692c 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-entity.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-entity.ts
@@ -80,6 +80,10 @@ export const WaitpointEntity = new EntitySchema<WaitpointSchema>({
name: 'idx_waitpoint_project_id',
columns: ['projectId'],
},
+ {
+ name: 'idx_waitpoint_flow_run_id_status_created',
+ columns: ['flowRunId', 'status', 'created'],
+ },
{
name: 'idx_waitpoint_pending_resume_date_time',
columns: ['resumeDateTime'],
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-service.ts b/packages/server/api/src/app/waitpoints/waitpoint-service.ts
index 4c8029e92bb..e72ecfcdc89 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-service.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-service.ts
@@ -2,6 +2,7 @@ import { ActivepiecesError, apId, ErrorCode, isNil } from '@activepieces/core-ut
import { FlowRunStatus, PauseType } from '@activepieces/shared'
import dayjs from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
+import { EntityManager, Not } from 'typeorm'
import { repoFactory } from '../core/db/repo-factory'
import { transaction } from '../core/db/transaction'
import { flowRunRepo } from '../flows/flow-run/flow-run-service'
@@ -18,6 +19,7 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
async createForPause(params: CreateForPauseParams): Promise<CreateForPauseResult> {
const preCompleted = await waitpointRepo().findOneBy({
flowRunId: params.flowRunId,
+ projectId: params.projectId,
stepName: params.stepName,
status: WaitpointStatus.COMPLETED,
})
@@ -47,11 +49,13 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
workerHandlerId: params.workerHandlerId ?? null,
httpRequestId: params.httpRequestId ?? null,
resumePayload: null,
+ sealed: false,
+ policy: null,
})
.orIgnore()
.execute()
- const waitpoint = await waitpointRepo().findOneByOrFail({ flowRunId: params.flowRunId, stepName: params.stepName })
+ const waitpoint = await waitpointRepo().findOneByOrFail({ flowRunId: params.flowRunId, projectId: params.projectId, stepName: params.stepName })
const inserted = waitpoint.id === id
if (inserted) {
log.info({ flowRun: { id: params.flowRunId }, waitpoint: { id } }, '[waitpointService#createForPause] Waitpoint created')
@@ -64,7 +68,7 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
job: {
name: SystemJobName.RESUME_DELAY_WAITPOINT,
data: { flowRunId: params.flowRunId, projectId: params.projectId, waitpointId: waitpoint.id },
- jobId: `resume-delay-${params.flowRunId}`,
+ jobId: resumeDelayJobId(waitpoint.id),
},
schedule: {
type: 'one-time',
@@ -82,7 +86,7 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
const pending = await repo
.createQueryBuilder('waitpoint')
.setLock('pessimistic_write')
- .where({ id: params.waitpointId, flowRunId: params.flowRunId, status: WaitpointStatus.PENDING })
+ .where({ id: params.waitpointId, flowRunId: params.flowRunId, projectId: params.projectId, status: WaitpointStatus.PENDING })
.getOne()
if (isNil(pending)) {
@@ -106,24 +110,31 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
const { flowRunId, waitpointId, flowRunStatus, projectId, resumePayload, workerHandlerId, onReady } = params
if (flowRunStatus === FlowRunStatus.PAUSED) {
- const waitpoint = await transaction(async (entityManager) => {
+ const delivery = await transaction<PausedDelivery>(async (entityManager) => {
const repo = waitpointRepo(entityManager)
const found = await repo
.createQueryBuilder('waitpoint')
.setLock('pessimistic_write')
- .where({ id: waitpointId, flowRunId })
+ .where({ id: waitpointId, flowRunId, projectId })
.getOne()
if (isNil(found)) {
- return null
+ return 'unknown-waitpoint'
+ }
+ if (found.status === WaitpointStatus.CONSUMED) {
+ return 'already-delivered'
}
await onReady(found)
- await repo.delete({ id: found.id })
- return found
+ await this.consume({ waitpoint: found, entityManager })
+ return 'dispatched'
})
- if (isNil(waitpoint)) {
+ if (delivery === 'unknown-waitpoint') {
log.info({ flowRun: { id: flowRunId }, waitpoint: { id: waitpointId } }, '[waitpointService#handleResumeSignal] Stale waitpointId, ignoring')
return false
}
+ if (delivery === 'already-delivered') {
+ log.info({ flowRun: { id: flowRunId }, waitpoint: { id: waitpointId } }, '[waitpointService#handleResumeSignal] Waitpoint was already delivered, ignoring')
+ return false
+ }
log.info({ flowRun: { id: flowRunId }, waitpoint: { id: waitpointId } }, '[waitpointService#handleResumeSignal] Resume triggered')
return true
}
@@ -142,32 +153,75 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
return false
},
- async findPendingByVersion({ flowRunId, version }: FindPendingByVersionParams): Promise<Waitpoint | null> {
+ async findPendingByVersion({ flowRunId, projectId, version }: FindPendingByVersionParams): Promise<Waitpoint | null> {
return waitpointRepo().findOne({
- where: { flowRunId, status: WaitpointStatus.PENDING, version },
+ where: { flowRunId, projectId, status: WaitpointStatus.PENDING, version },
})
},
- async findByIdAndFlowRunId({ waitpointId, flowRunId }: { waitpointId: string, flowRunId: string }): Promise<Waitpoint | null> {
- return waitpointRepo().findOneBy({ id: waitpointId, flowRunId })
+ async findByIdAndFlowRunId({ waitpointId, flowRunId, projectId }: FindByIdAndFlowRunIdParams): Promise<Waitpoint | null> {
+ return waitpointRepo().findOneBy({ id: waitpointId, flowRunId, projectId })
+ },
+
+ async findUndeliveredCompletedWaitpoint({ flowRunId, projectId }: FindUndeliveredCompletedWaitpointParams): Promise<Waitpoint | null> {
+ const barrierPending = await this.hasPendingBarrier({ flowRunId, projectId })
+ if (barrierPending) {
+ return null
+ }
+ return waitpointRepo().findOne({
+ where: { flowRunId, projectId, status: WaitpointStatus.COMPLETED },
+ order: { created: 'DESC' },
+ })
+ },
+
+ async hasPendingBarrier({ flowRunId, projectId }: HasPendingBarrierParams): Promise<boolean> {
+ return waitpointRepo().existsBy({ flowRunId, projectId, type: PauseType.BARRIER, status: WaitpointStatus.PENDING })
+ },
+
+ async hasBarrier({ flowRunId, projectId }: HasBarrierParams): Promise<boolean> {
+ return waitpointRepo().existsBy({ flowRunId, projectId, type: PauseType.BARRIER })
},
- async getByFlowRunId(flowRunId: string): Promise<Waitpoint | null> {
- const completed = await waitpointRepo().findOneBy({ flowRunId, status: WaitpointStatus.COMPLETED })
- return completed ?? waitpointRepo().findOneBy({ flowRunId })
+ async consume({ waitpoint, entityManager }: ConsumeParams): Promise<void> {
+ const repo = waitpointRepo(entityManager)
+ if (waitpoint.type === PauseType.BARRIER) {
+ await repo.save({ ...waitpoint, status: WaitpointStatus.CONSUMED })
+ log.info({ waitpoint: { id: waitpoint.id }, flowRun: { id: waitpoint.flowRunId } }, '[waitpointService#consume] Barrier kept as CONSUMED so the run stays barrier-owned until it ends')
+ return
+ }
+ await repo.delete({ id: waitpoint.id, projectId: waitpoint.projectId })
+ log.info({ waitpoint: { id: waitpoint.id }, flowRun: { id: waitpoint.flowRunId } }, '[waitpointService#consume] Waitpoint consumed and deleted')
},
- async delete({ id }: { id: string }): Promise<void> {
- await waitpointRepo().delete({ id })
+ async findSubflowWaitpoint({ flowRunId, projectId }: FindSubflowWaitpointParams): Promise<Waitpoint | null> {
+ const pending = await waitpointRepo().findOne({
+ where: { flowRunId, projectId, status: WaitpointStatus.PENDING, type: Not(PauseType.BARRIER) },
+ order: { created: 'DESC' },
+ })
+ if (!isNil(pending)) {
+ return pending
+ }
+ return waitpointRepo().findOne({
+ where: { flowRunId, projectId, type: Not(PauseType.BARRIER) },
+ order: { created: 'DESC' },
+ })
+ },
+
+ async delete({ id, projectId }: DeleteWaitpointParams): Promise<void> {
+ await waitpointRepo().delete({ id, projectId })
log.info({ waitpoint: { id } }, '[waitpointService#delete] Waitpoint deleted')
},
- async deleteByFlowRunId(flowRunId: string): Promise<void> {
- await waitpointRepo().delete({ flowRunId })
+ async deleteByFlowRunId({ flowRunId, projectId }: DeleteByFlowRunIdParams): Promise<void> {
+ await waitpointRepo().delete({ flowRunId, projectId })
log.info({ flowRun: { id: flowRunId } }, '[waitpointService#deleteByFlowRunId] Waitpoint deleted')
},
})
+export function resumeDelayJobId(waitpointId: string): string {
+ return `resume-delay-${waitpointId}`
+}
+
function clampWaitpointResumeDeadline({ requested, type, flowRunCreated, flowRunId }: ClampWaitpointResumeDeadlineParams): string | undefined {
const pauseTimeoutDays = system.getNumberOrThrow(AppSystemProp.PAUSED_FLOW_TIMEOUT_DAYS)
const runDeadline = dayjs(flowRunCreated).add(pauseTimeoutDays, 'day')
@@ -183,9 +237,52 @@ function clampWaitpointResumeDeadline({ requested, type, flowRunCreated, flowRun
return requested
}
+type PausedDelivery = 'dispatched' | 'already-delivered' | 'unknown-waitpoint'
+
type ClampWaitpointResumeDeadlineParams = {
requested: string | undefined
type: `${PauseType}`
flowRunCreated: string
flowRunId: string
}
+
+type HasPendingBarrierParams = {
+ flowRunId: string
+ projectId: string
+}
+
+type HasBarrierParams = {
+ flowRunId: string
+ projectId: string
+}
+
+type ConsumeParams = {
+ waitpoint: Waitpoint
+ entityManager?: EntityManager
+}
+
+type FindByIdAndFlowRunIdParams = {
+ waitpointId: string
+ flowRunId: string
+ projectId: string
+}
+
+type FindUndeliveredCompletedWaitpointParams = {
+ flowRunId: string
+ projectId: string
+}
+
+type FindSubflowWaitpointParams = {
+ flowRunId: string
+ projectId: string
+}
+
+type DeleteWaitpointParams = {
+ id: string
+ projectId: string
+}
+
+type DeleteByFlowRunIdParams = {
+ flowRunId: string
+ projectId: string
+}
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-signal-entity.ts b/packages/server/api/src/app/waitpoints/waitpoint-signal-entity.ts
index d3a40a5f77d..83433513b90 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-signal-entity.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-signal-entity.ts
@@ -48,7 +48,11 @@ export const WaitpointSignalEntity = new EntitySchema<WaitpointSignalSchema>({
},
{
name: 'idx_waitpoint_signal_ref_id',
- columns: ['refId'],
+ columns: ['refId', 'projectId'],
+ },
+ {
+ name: 'idx_waitpoint_signal_waitpoint_id_project_id_sequence',
+ columns: ['waitpointId', 'projectId', 'sequence', 'created'],
},
{
name: 'idx_waitpoint_signal_waitpoint_id_sequence',
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-types.ts b/packages/server/api/src/app/waitpoints/waitpoint-types.ts
index ee4f90850c0..bc669e2c962 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-types.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-types.ts
@@ -4,6 +4,7 @@ import { BarrierPolicy, BarrierSignalStatus, FlowRunStatus, PauseType, RespondRe
enum WaitpointStatus {
PENDING = 'PENDING',
COMPLETED = 'COMPLETED',
+ CONSUMED = 'CONSUMED',
}
enum WaitpointVersionEnum {
@@ -91,6 +92,7 @@ type HandleResumeSignalParams = {
type FindPendingByVersionParams = {
flowRunId: ApId
+ projectId: ApId
version: WaitpointVersion
}
diff --git a/packages/server/api/src/app/workers/job/index.ts b/packages/server/api/src/app/workers/job/index.ts
index e94f3d04606..78fd58116a8 100644
--- a/packages/server/api/src/app/workers/job/index.ts
+++ b/packages/server/api/src/app/workers/job/index.ts
@@ -27,6 +27,7 @@ export enum JobStatus {
export enum QueueName {
WORKER_JOBS = 'workerJobs',
RUNS_METADATA = 'runsMetadata',
+ BARRIER_EVALUATION = 'barrierEvaluation',
}
export const getPlatformGroupQueueName = (workerGroupId: string): string => {
diff --git a/packages/server/api/src/app/workers/migrations/refill-paused-jobs.ts b/packages/server/api/src/app/workers/migrations/refill-paused-jobs.ts
index 5eaadd285af..6a8d380c184 100644
--- a/packages/server/api/src/app/workers/migrations/refill-paused-jobs.ts
+++ b/packages/server/api/src/app/workers/migrations/refill-paused-jobs.ts
@@ -2,7 +2,7 @@ import { isNil } from '@activepieces/core-utils'
import { FlowRunStatus, PauseType } from '@activepieces/shared'
import dayjs from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
-import { In, MoreThan } from 'typeorm'
+import { In, IsNull, MoreThan, Not } from 'typeorm'
import { repoFactory } from '../../core/db/repo-factory'
import { redisConnections } from '../../database/redis-connections'
import { flowRunRepo } from '../../flows/flow-run/flow-run-service'
@@ -11,6 +11,8 @@ import { AppSystemProp } from '../../helper/system/system-props'
import { SystemJobName } from '../../helper/system-jobs/common'
import { systemJobsSchedule } from '../../helper/system-jobs/system-job'
import { WaitpointEntity } from '../../waitpoints/waitpoint-entity'
+import { resumeDelayJobId } from '../../waitpoints/waitpoint-service'
+import { WaitpointStatus } from '../../waitpoints/waitpoint-types'
import { jobQueue } from '../job-queue/job-queue'
const REFILL_PAUSED_RUNS_KEY = 'refill_paused_runs_v7'
@@ -41,14 +43,19 @@ export const refillPausedRuns = (log: FastifyBaseLogger) => ({
}
const flowRunIds = pausedRuns.map(r => r.id)
- const waitpoints = await waitpointRepo().findBy({ flowRunId: In(flowRunIds) })
- const waitpointByRunId = Object.fromEntries(waitpoints.map(w => [w.flowRunId, w]))
+ const delayWaitpoints = await waitpointRepo().findBy({
+ flowRunId: In(flowRunIds),
+ type: PauseType.DELAY,
+ status: WaitpointStatus.PENDING,
+ resumeDateTime: Not(IsNull()),
+ })
+ const pausedRunById = Object.fromEntries(pausedRuns.map(run => [run.id, run]))
let migratedCount = 0
- for (const pausedRun of pausedRuns) {
- const waitpoint = waitpointByRunId[pausedRun.id]
- if (isNil(waitpoint) || waitpoint.type !== PauseType.DELAY || isNil(waitpoint.resumeDateTime)) {
+ for (const waitpoint of delayWaitpoints) {
+ const pausedRun = pausedRunById[waitpoint.flowRunId]
+ if (isNil(pausedRun) || isNil(waitpoint.resumeDateTime)) {
continue
}
if (dayjs(pausedRun.created).isBefore(dayjs().subtract(executionRetentionDays, 'day'))) {
@@ -68,7 +75,7 @@ export const refillPausedRuns = (log: FastifyBaseLogger) => ({
job: {
name: SystemJobName.RESUME_DELAY_WAITPOINT,
data: { flowRunId: pausedRun.id, projectId: pausedRun.projectId, waitpointId: waitpoint.id },
- jobId: `resume-delay-${pausedRun.id}`,
+ jobId: resumeDelayJobId(waitpoint.id),
},
schedule: {
type: 'one-time',
diff --git a/packages/server/api/src/app/workers/worker-module.ts b/packages/server/api/src/app/workers/worker-module.ts
index 1f30302e890..b222b338a12 100644
--- a/packages/server/api/src/app/workers/worker-module.ts
+++ b/packages/server/api/src/app/workers/worker-module.ts
@@ -4,6 +4,7 @@ import { runsMetadataQueue } from '../flows/flow-run/flow-runs-queue'
import { pubsub } from '../helper/pubsub'
import { system } from '../helper/system/system'
import { AppSystemProp } from '../helper/system/system-props'
+import { barrierQueue } from '../waitpoints/barrier-queue'
import { flowEngineWorker } from './engine-controller'
import { setupBullMQBoard } from './job-queue/bullboard'
import { jobBroker } from './job-queue/job-broker'
@@ -21,11 +22,14 @@ export const workerModule: FastifyPluginAsyncZod = async (app) => {
await runsMetadataQueue(app.log).init()
+ await barrierQueue(app.log).init()
+
await setupBullMQBoard(app)
app.addHook('onClose', async () => {
await jobBroker(app.log).close()
await runsMetadataQueue(app.log).close()
+ await barrierQueue(app.log).close()
await jobQueue(app.log).close()
await pubsub.close()
})
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts
new file mode 100644
index 00000000000..92c101dddf1
--- /dev/null
+++ b/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts
@@ -0,0 +1,551 @@
+import { apId, isNil } from '@activepieces/core-utils'
+import { BarrierSignalStatus, BarrierSummary, ErrorCode, FlowRunStatus, FlowVersionState, PauseType, RunEnvironment } from '@activepieces/shared'
+import { Queue } from 'bullmq'
+import { FastifyInstance } from 'fastify'
+import { StatusCodes } from 'http-status-codes'
+import { databaseConnection } from '../../../../../src/app/database/database-connection'
+import { platformConfigurationService } from '../../../../../src/app/platform/platform-configuration.service'
+import { jobQueue } from '../../../../../src/app/workers/job-queue/job-queue'
+import { barrierQueue } from '../../../../../src/app/waitpoints/barrier-queue'
+import { BarrierJobData } from '../../../../../src/app/waitpoints/barrier-queue-factory'
+import { barrierService } from '../../../../../src/app/waitpoints/barrier-service'
+import { resumeService } from '../../../../../src/app/waitpoints/resume-service'
+import { waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
+import { Waitpoint, WaitpointStatus } from '../../../../../src/app/waitpoints/waitpoint-types'
+import { db } from '../../../../helpers/db'
+import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'
+import { createTestContext, TestContext } from '../../../../helpers/test-context'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'
+
+let app: FastifyInstance
+let ctx: TestContext
+
+beforeAll(async () => {
+ app = await setupTestEnvironment()
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+beforeEach(async () => {
+ ctx = await createTestContext(app)
+})
+
+async function createParentRun(status: FlowRunStatus = FlowRunStatus.PAUSED) {
+ const flow = createMockFlow({ projectId: ctx.project.id })
+ await db.save('flow', flow)
+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })
+ await db.save('flow_version', flowVersion)
+ const flowRun = createMockFlowRun({
+ projectId: ctx.project.id,
+ flowId: flow.id,
+ flowVersionId: flowVersion.id,
+ status,
+ environment: RunEnvironment.PRODUCTION,
+ })
+ await db.save('flow_run', flowRun)
+ return { flow, flowVersion, flowRun }
+}
+
+async function createBarrier({ flowRunId, signalLabels, policy, stepName }: {
+ flowRunId: string
+ signalLabels: (string | null)[]
+ policy?: { requiredSuccesses?: number, releaseOnFirstFailure?: boolean }
+ stepName?: string
+}) {
+ return barrierService(app.log).create({
+ flowRunId,
+ projectId: ctx.project.id,
+ platformId: ctx.platform.id,
+ stepName: stepName ?? 'approval',
+ version: 'V1',
+ policy,
+ signalLabels,
+ })
+}
+
+async function listSignals(barrierId: string) {
+ return databaseConnection().getRepository('waitpoint_signal').findBy({ waitpointId: barrierId })
+}
+
+async function readSummary(barrierId: string): Promise<BarrierSummary> {
+ const barrier = await databaseConnection().getRepository('waitpoint').findOneByOrFail({ id: barrierId })
+ return BarrierSummary.parse(barrier.resumePayload?.body)
+}
+
+async function receiveSignal({ signalId, status, result }: { signalId: string, status: BarrierSignalStatus, result?: Record<string, unknown> }) {
+ return barrierService(app.log).receiveSignal({ signalId, projectId: ctx.project.id, status, result })
+}
+
+async function releaseIfReady(barrierId: string) {
+ return barrierService(app.log).releaseIfReady({ barrierId, projectId: ctx.project.id })
+}
+
+async function releaseNow(barrier: Waitpoint) {
+ return barrierService(app.log).release({ barrier, timedOut: false, releaseReason: 'predicate' })
+}
+
+async function completeWithoutConsuming(barrierId: string) {
+ await databaseConnection().getRepository('waitpoint').update({ id: barrierId }, {
+ status: WaitpointStatus.COMPLETED,
+ resumePayload: { body: { total: 1 }, headers: {}, queryParams: {} },
+ })
+ await databaseConnection().getRepository('waitpoint_signal').delete({ waitpointId: barrierId })
+}
+
+async function countPendingEvaluations({ queue, barrierId }: { queue: Queue<BarrierJobData>, barrierId: string }): Promise<number> {
+ const jobs = await queue.getJobs(['waiting', 'delayed', 'prioritized', 'paused'])
+ return jobs.filter((job) => job.data.barrierId === barrierId).length
+}
+
+async function closeWithoutConsuming(barrierId: string) {
+ await databaseConnection().getRepository('waitpoint').update({ id: barrierId }, {
+ status: WaitpointStatus.COMPLETED,
+ resumePayload: {
+ body: {
+ total: 1,
+ succeeded: 1,
+ failed: 0,
+ rejected: 0,
+ canceled: 0,
+ notDispatched: 0,
+ stillRunning: 0,
+ timedOut: false,
+ signals: [],
+ },
+ headers: {},
+ queryParams: {},
+ },
+ })
+ await databaseConnection().getRepository('waitpoint_signal').delete({ waitpointId: barrierId })
+}
+
+async function listResumeJobs(flowRunId: string) {
+ const queue = jobQueue(app.log).getSharedQueue()
+ const jobs = await queue.getJobs(['waiting', 'delayed', 'prioritized', 'paused', 'active'])
+ return jobs.filter((job) => job.id?.startsWith(`${flowRunId}-resume-`))
+}
+
+async function dropResumeJobs(flowRunId: string) {
+ const jobs = await listResumeJobs(flowRunId)
+ await Promise.all(jobs.map((job) => job.remove()))
+}
+
+async function readStatus(barrierId: string): Promise<WaitpointStatus> {
+ const barrier = await db.findOneByOrFail<{ status: WaitpointStatus }>('waitpoint', { id: barrierId })
+ return barrier.status
+}
+
+async function recoverUndeliveredResume(flowRunId: string) {
+ const undelivered = await waitpointService(app.log).findUndeliveredCompletedWaitpoint({ flowRunId, projectId: ctx.project.id })
+ if (isNil(undelivered)) {
+ return null
+ }
+ await resumeService(app.log).resumeTrustedWithoutLock({
+ flowRunId,
+ waitpointId: undelivered.id,
+ resumePayload: undelivered.resumePayload,
+ })
+ return undelivered
+}
+
+describe('barrier release predicate', () => {
+ it('releases once every signal has been received, and not before', async () => {
+ const { flowRun } = await createParentRun(FlowRunStatus.RUNNING)
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+ const signals = await listSignals(barrier.id)
+ expect(signals).toHaveLength(2)
+
+ await receiveSignal({ signalId: signals[0].id, status: BarrierSignalStatus.SUCCEEDED })
+ await releaseIfReady(barrier.id)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.PENDING)
+
+ await receiveSignal({ signalId: signals[1].id, status: BarrierSignalStatus.SUCCEEDED })
+ await releaseIfReady(barrier.id)
+
+ const summary = await readSummary(barrier.id)
+ expect(summary).toMatchObject({ total: 2, succeeded: 2, failed: 0, stillRunning: 0, timedOut: false })
+ expect(await listSignals(barrier.id)).toHaveLength(0)
+ })
+
+ it('releases early once requiredSuccesses favourable signals have landed', async () => {
+ const { flowRun } = await createParentRun(FlowRunStatus.RUNNING)
+ const { barrier } = await createBarrier({
+ flowRunId: flowRun.id,
+ policy: { requiredSuccesses: 2 },
+ })
+ const signals = await listSignals(barrier.id)
+
+ await receiveSignal({ signalId: signals[0].id, status: BarrierSignalStatus.SUCCEEDED })
+ await releaseIfReady(barrier.id)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.PENDING)
+
+ await receiveSignal({ signalId: signals[1].id, status: BarrierSignalStatus.SUCCEEDED })
+ await releaseIfReady(barrier.id)
+
+ const summary = await readSummary(barrier.id)
+ expect(summary).toMatchObject({ total: 3, succeeded: 2, stillRunning: 1 })
+ })
+
+ it('releases on the first unfavourable signal when releaseOnFirstFailure is set', async () => {
+ const { flowRun } = await createParentRun(FlowRunStatus.RUNNING)
+ const { barrier } = await createBarrier({
+ flowRunId: flowRun.id,
+ policy: { releaseOnFirstFailure: true },
+ })
+ const signals = await listSignals(barrier.id)
+
+ await receiveSignal({ signalId: signals[0].id, status: BarrierSignalStatus.REJECTED })
+ await releaseIfReady(barrier.id)
+
+ const summary = await readSummary(barrier.id)
+ expect(summary).toMatchObject({ total: 3, rejected: 1, stillRunning: 2 })
+ })
+
+ it('reports the inline signals so every awaited thing is named, not only the failures', async () => {
+ const { flowRun } = await createParentRun(FlowRunStatus.RUNNING)
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+ const signals = await listSignals(barrier.id)
+
+ await receiveSignal({ signalId: signals[0].id, status: BarrierSignalStatus.SUCCEEDED })
+ await receiveSignal({ signalId: signals[1].id, status: BarrierSignalStatus.FAILED, result: { reason: 'nope' } })
+ await releaseIfReady(barrier.id)
+
+ const summary = await readSummary(barrier.id)
+ expect(summary.failed).toBe(1)
+ expect(summary.signals?.map((signal) => signal.label).sort()).toEqual(['[email protected]', '[email protected]'])
+ expect(summary.signals?.map((signal) => signal.outcome).sort()).toEqual([BarrierSignalStatus.FAILED, BarrierSignalStatus.SUCCEEDED])
+ })
+})
+
+describe('signal identity', () => {
+ it('rejects a second signal on the same (waitpointId, sequence)', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ const row = {
+ waitpointId: barrier.id,
+ projectId: ctx.project.id,
+ status: BarrierSignalStatus.PENDING,
+ refId: null,
+ sequence: 0,
+ label: null,
+ result: null,
+ }
+ await databaseConnection().getRepository('waitpoint_signal').insert({ id: apId(), ...row })
+
+ await expect(databaseConnection().getRepository('waitpoint_signal').insert({ id: apId(), ...row })).rejects.toThrow()
+ })
+
+ it('lets two null sequences coexist on one barrier', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+
+ const signals = await listSignals(barrier.id)
+ expect(signals).toHaveLength(2)
+ expect(signals.every((signal) => signal.sequence === null)).toBe(true)
+ })
+
+ it('overwrites the outcome when the same signal is received twice', async () => {
+ const { flowRun } = await createParentRun(FlowRunStatus.RUNNING)
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+ const [signal] = await listSignals(barrier.id)
+
+ await receiveSignal({ signalId: signal.id, status: BarrierSignalStatus.FAILED })
+ await receiveSignal({ signalId: signal.id, status: BarrierSignalStatus.SUCCEEDED })
+
+ const reread = (await listSignals(barrier.id)).find((row) => row.id === signal.id)
+ expect(reread.status).toBe(BarrierSignalStatus.SUCCEEDED)
+ })
+})
+
+describe('evaluation coalescing', () => {
+ it('clears the deduplication key before evaluating, so a signal landing mid-job is not swallowed', async () => {
+ const { flowRun } = await createParentRun()
+ const queue = barrierQueue(app.log).get()
+ await queue.pause()
+ try {
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+
+ await barrierQueue(app.log).enqueueEvaluation({ barrierId: barrier.id, projectId: ctx.project.id })
+ const beforeHandling = await countPendingEvaluations({ queue, barrierId: barrier.id })
+
+ await barrierQueue(app.log).clearEvaluationDedupKey(barrier.id)
+ await barrierQueue(app.log).enqueueEvaluation({ barrierId: barrier.id, projectId: ctx.project.id })
+ const afterHandling = await countPendingEvaluations({ queue, barrierId: barrier.id })
+
+ expect(beforeHandling).toBe(1)
+ expect(afterHandling).toBe(2)
+ }
+ finally {
+ await queue.resume()
+ }
+ })
+})
+
+describe('resume guards', () => {
+ it('refuses an external resume that addresses a barrier waitpoint', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+
+ const { stale } = await resumeService(app.log).resumeFromWaitpoint({
+ flowRunId: flowRun.id,
+ waitpointId: barrier.id,
+ resumePayload: { body: { forged: true } },
+ })
+
+ expect(stale).toBe(true)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.PENDING)
+ })
+
+ it('refuses a by-run resume while the run holds a pending barrier', async () => {
+ const { flowRun } = await createParentRun()
+ await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+
+ const { stale } = await resumeService(app.log).legacyResume({
+ flowRunId: flowRun.id,
+ resumePayload: { body: { forged: true } },
+ })
+
+ expect(stale).toBe(true)
+ })
+
+ it('refuses a resume addressed at a waitpoint id the run does not own while a barrier is open', async () => {
+ const { flowRun } = await createParentRun()
+ await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+
+ const { stale } = await resumeService(app.log).resumeFromWaitpoint({
+ flowRunId: flowRun.id,
+ waitpointId: apId(),
+ resumePayload: { body: { forged: true } },
+ })
+
+ expect(stale).toBe(true)
+ })
+
+ it('lets the module release a barrier and consumes the waitpoint exactly once', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ const [signal] = await listSignals(barrier.id)
+
+ await receiveSignal({ signalId: signal.id, status: BarrierSignalStatus.SUCCEEDED })
+ await releaseIfReady(barrier.id)
+
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)
+ expect(await listSignals(barrier.id)).toHaveLength(0)
+ })
+
+ it('leaves the resume to the release that closed the barrier, even inside the window before that release consumes it', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await closeWithoutConsuming(barrier.id)
+ await dropResumeJobs(flowRun.id)
+
+ const lost = await releaseNow(barrier)
+
+ expect(lost).toBeNull()
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(0)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.COMPLETED)
+ })
+
+ it('refuses a by-run resume between the barrier closing and the trusted resume consuming it', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await completeWithoutConsuming(barrier.id)
+
+ const { stale } = await resumeService(app.log).legacyResume({
+ flowRunId: flowRun.id,
+ resumePayload: { body: { forged: true }, headers: {}, queryParams: {} },
+ })
+
+ expect(stale).toBe(true)
+ })
+
+ it('refuses a by-run resume after the barrier was released while the run is still PAUSED', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+
+ await releaseNow(barrier)
+
+ const run = await db.findOneByOrFail<{ status: FlowRunStatus }>('flow_run', { id: flowRun.id })
+ expect(run.status).toBe(FlowRunStatus.PAUSED)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)
+
+ const { stale } = await resumeService(app.log).legacyResume({
+ flowRunId: flowRun.id,
+ resumePayload: { body: { forged: true }, headers: {}, queryParams: {} },
+ })
+
+ expect(stale).toBe(true)
+ })
+
+ it('refuses a by-run sync resume through both barrier windows', async () => {
+ const { flowRun: closedRun } = await createParentRun()
+ const { barrier: closedBarrier } = await createBarrier({ flowRunId: closedRun.id, signalLabels: ['[email protected]'] })
+ await completeWithoutConsuming(closedBarrier.id)
+
+ const closedResponse = await resumeService(app.log).legacySyncResume({
+ runId: closedRun.id,
+ payload: { body: { forged: true }, headers: {}, queryParams: {} },
+ correlationId: apId(),
+ })
+ expect(closedResponse.status).toBe(StatusCodes.GONE)
+
+ const { flowRun: releasedRun } = await createParentRun()
+ const { barrier: releasedBarrier } = await createBarrier({ flowRunId: releasedRun.id, signalLabels: ['[email protected]'] })
+ await releaseNow(releasedBarrier)
+
+ const releasedResponse = await resumeService(app.log).legacySyncResume({
+ runId: releasedRun.id,
+ payload: { body: { forged: true }, headers: {}, queryParams: {} },
+ correlationId: apId(),
+ })
+ expect(releasedResponse.status).toBe(StatusCodes.GONE)
+ })
+
+ it('still resumes a paused run that never held a waitpoint at all', async () => {
+ const { flowRun } = await createParentRun()
+
+ const { stale } = await resumeService(app.log).legacyResume({
+ flowRunId: flowRun.id,
+ resumePayload: { body: { approved: true }, headers: {}, queryParams: {} },
+ })
+
+ expect(stale).toBe(false)
+ })
+
+ it('leaves a non-barrier waitpoint resumable', async () => {
+ const { flowRun } = await createParentRun()
+ const { waitpoint } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'approval',
+ type: PauseType.WEBHOOK,
+ version: 'V1',
+ })
+
+ const { stale } = await resumeService(app.log).resumeFromWaitpoint({
+ flowRunId: flowRun.id,
+ waitpointId: waitpoint.id,
+ resumePayload: { body: { approved: true } },
+ })
+
+ expect(stale).toBe(false)
+ })
+
+ it('refuses a trusted resume of a barrier that was already delivered', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await releaseNow(barrier)
+ await dropResumeJobs(flowRun.id)
+
+ const { stale } = await resumeService(app.log).resumeTrustedWithoutLock({
+ flowRunId: flowRun.id,
+ waitpointId: barrier.id,
+ resumePayload: { body: { forged: true }, headers: {}, queryParams: {} },
+ })
+
+ expect(stale).toBe(true)
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(0)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)
+ })
+
+ it('does not dispatch a second resume for a barrier that was already consumed', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await releaseNow(barrier)
+ await dropResumeJobs(flowRun.id)
+
+ const { stale } = await resumeService(app.log).resumeFromWaitpoint({
+ flowRunId: flowRun.id,
+ waitpointId: barrier.id,
+ resumePayload: { body: { forged: true } },
+ })
+
+ expect(stale).toBe(true)
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(0)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)
+ })
+})
+
+describe('undelivered resume recovery', () => {
+ it('dispatches the resume for a waitpoint that was already completed before the run reached PAUSED', async () => {
+ const { flowRun } = await createParentRun()
+ const { waitpoint } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'approval',
+ type: PauseType.WEBHOOK,
+ version: 'V1',
+ })
+ await completeWithoutConsuming(waitpoint.id)
+ await dropResumeJobs(flowRun.id)
+
+ await recoverUndeliveredResume(flowRun.id)
+
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(1)
+ expect(await db.findOneBy('waitpoint', { id: waitpoint.id })).toBeNull()
+ })
+
+ it('still resumes when the release that closed the barrier died before dispatching', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await closeWithoutConsuming(barrier.id)
+ await dropResumeJobs(flowRun.id)
+
+ await releaseIfReady(barrier.id)
+
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(1)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)
+ })
+
+ it('fails the evaluation when the re-dispatch fails, so the queue spends its remaining attempts', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await closeWithoutConsuming(barrier.id)
+ await dropResumeJobs(flowRun.id)
+ await db.update('waitpoint', barrier.id, { flowRunId: apId() })
+
+ await expect(releaseIfReady(barrier.id)).rejects.toThrow()
+
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(0)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.COMPLETED)
+ })
+
+ it('enqueues nothing when recovery runs a second time on a consumed barrier', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await releaseNow(barrier)
+ await dropResumeJobs(flowRun.id)
+
+ await releaseIfReady(barrier.id)
+
+ expect(await recoverUndeliveredResume(flowRun.id)).toBeNull()
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(0)
+ })
+})
+
+describe('signal count limit', () => {
+ it('refuses a barrier that waits on more than the platform allows', async () => {
+ const { flowRun } = await createParentRun()
+ await platformConfigurationService(app.log).update({ platformId: ctx.platform.id, maxBarrierSignals: 2 })
+
+ await expect(createBarrier({ flowRunId: flowRun.id, signalLabels: ['a', 'b', 'c'] })).rejects.toMatchObject({
+ error: {
+ code: ErrorCode.VALIDATION,
+ params: { message: expect.stringContaining('exceeds the maximum of 2') },
+ },
+ })
+ })
+
+ it('allows a barrier at exactly the platform limit', async () => {
+ const { flowRun } = await createParentRun()
+ await platformConfigurationService(app.log).update({ platformId: ctx.platform.id, maxBarrierSignals: 2 })
+
+ const { signalCount } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['a', 'b'] })
+
+ expect(signalCount).toBe(2)
+ })
+})
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/execute-flow-e2e.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/execute-flow-e2e.test.ts
index 51bfee484ed..4d518b5a188 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/execute-flow-e2e.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/execute-flow-e2e.test.ts
@@ -22,6 +22,7 @@ import {
FlowTriggerType,
FlowVersionState,
PackageType,
+ PauseType,
PieceScope,
PieceType,
RunEnvironment,
@@ -60,7 +61,7 @@ beforeAll(async () => {
workerToken: ctx.workerToken,
})
await new Promise((resolve) => setTimeout(resolve, 5000))
-}, 30_000)
+})
afterAll(async () => {
worker.stop()
@@ -402,6 +403,117 @@ async function setupSubflowWithWebhookResponseFixtures() {
return { parentFlow, parentFlowVersion, mockPlatform, mockProject }
}
+const APPROVAL_PIECE_VERSION = '0.1.28'
+const DELAY_PIECE_VERSION = '0.3.34'
+const WEBHOOK_PIECE_VERSION = '0.1.29'
+
+async function saveWaitpointPieces(): Promise<void> {
+ const pieces = [
+ { name: '@activepieces/piece-webhook', version: WEBHOOK_PIECE_VERSION },
+ { name: '@activepieces/piece-approval', version: APPROVAL_PIECE_VERSION },
+ { name: '@activepieces/piece-delay', version: DELAY_PIECE_VERSION },
+ ].map(({ name, version }) => createMockPieceMetadata({
+ name,
+ version,
+ platformId: undefined,
+ packageType: PackageType.REGISTRY,
+ pieceType: PieceType.OFFICIAL,
+ }))
+ await databaseConnection().getRepository('piece_metadata').save(pieces)
+}
+
+function approvalAction({ name, displayName, nextAction }: { name: string, displayName: string, nextAction?: unknown }) {
+ return {
+ type: FlowActionType.PIECE as const,
+ name,
+ displayName,
+ valid: true,
+ settings: {
+ pieceName: '@activepieces/piece-approval',
+ pieceVersion: APPROVAL_PIECE_VERSION,
+ actionName: 'wait_for_approval',
+ input: {},
+ propertySettings: {},
+ errorHandlingOptions: {},
+ },
+ ...(nextAction ? { nextAction } : {}),
+ }
+}
+
+async function startWaitpointRun({ mockPlatform, mockProject, firstAction, payload }: {
+ mockPlatform: { id: string }
+ mockProject: { id: string }
+ firstAction: unknown
+ payload?: Record<string, unknown>
+}) {
+ const mockFlow = createMockFlow({ projectId: mockProject.id })
+ await db.save('flow', mockFlow)
+ const mockFlowVersion = createMockFlowVersion({
+ flowId: mockFlow.id,
+ state: FlowVersionState.DRAFT,
+ trigger: {
+ type: FlowTriggerType.PIECE,
+ name: 'trigger',
+ displayName: 'Catch Webhook',
+ valid: true,
+ lastUpdatedDate: new Date().toISOString(),
+ settings: {
+ pieceName: '@activepieces/piece-webhook',
+ pieceVersion: WEBHOOK_PIECE_VERSION,
+ triggerName: 'catch_webhook',
+ input: { authType: 'none' },
+ propertySettings: {},
+ },
+ nextAction: firstAction,
+ },
+ })
+ await db.save('flow_version', mockFlowVersion)
+
+ return flowRunService(app.log).start({
+ flowId: mockFlow.id,
+ payload: { body: payload ?? { test: true } },
+ platformId: mockPlatform.id,
+ executionType: ExecutionType.BEGIN,
+ environment: RunEnvironment.TESTING,
+ streamStepProgress: StreamStepProgress.NONE,
+ executeTrigger: false,
+ flowVersionId: mockFlowVersion.id,
+ projectId: mockProject.id,
+ workerHandlerId: undefined,
+ httpRequestId: undefined,
+ failParentOnFailure: undefined,
+ })
+}
+
+async function waitForPendingWaitpoint(flowRunId: string, excludeIds: string[] = [], timeoutMs = 90_000): Promise<WaitpointRow> {
+ const start = Date.now()
+ while (Date.now() - start < timeoutMs) {
+ const rows = await db.findBy<WaitpointRow>('waitpoint', { flowRunId, status: 'PENDING' })
+ const fresh = rows.filter((row) => !excludeIds.includes(row.id))
+ if (fresh.length > 0) {
+ return fresh[0]
+ }
+ await new Promise((resolve) => setTimeout(resolve, 500))
+ }
+ const all = await db.findBy('waitpoint', { flowRunId })
+ throw new Error(`no fresh PENDING waitpoint for run ${flowRunId}; rows=${JSON.stringify(all)}`)
+}
+
+async function resumeWaitpoint({ flowRunId, waitpointId }: { flowRunId: string, waitpointId: string }): Promise<number> {
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRunId}/waitpoints/${waitpointId}?action=approve`,
+ })
+ return response.statusCode
+}
+
+type WaitpointRow = {
+ id: string
+ stepName: string
+ type: string
+ status: string
+}
+
async function pollFlowRunToCompletion(flowRunId: string, projectId: string) {
const maxWaitMs = 120_000
const pollIntervalMs = 500
@@ -1270,4 +1382,145 @@ describe('Execute Flow E2E', () => {
expect(response.statusCode).toBe(StatusCodes.OK)
expect(response.headers['x-webhook-id']).toBeDefined()
}, 180_000)
+
+ it('gives every loop iteration its own path-keyed waitpoint row', async () => {
+ const { mockPlatform, mockProject } = await mockAndSaveBasicSetup()
+ await saveWaitpointPieces()
+
+ const loopAction = {
+ type: FlowActionType.LOOP_ON_ITEMS as const,
+ name: 'step_1',
+ displayName: 'Loop',
+ valid: true,
+ settings: { items: '{{trigger[\'output\'].body.items}}' },
+ firstLoopAction: approvalAction({ name: 'step_2', displayName: 'Wait for Approval' }),
+ nextAction: {
+ type: FlowActionType.CODE as const,
+ name: 'step_3',
+ displayName: 'After Loop',
+ valid: true,
+ settings: {
+ sourceCode: { code: 'export const code = async () => ({ done: true });', packageJson: '{}' },
+ input: {},
+ errorHandlingOptions: {},
+ },
+ },
+ }
+
+ const flowRun = await startWaitpointRun({ mockPlatform, mockProject, firstAction: loopAction, payload: { items: [1, 2, 3] } })
+
+ const seenStepNames: string[] = []
+ const seenWaitpointIds: string[] = []
+ for (let iteration = 0; iteration < 3; iteration++) {
+ const waitpoint = await waitForPendingWaitpoint(flowRun.id, seenWaitpointIds)
+ seenStepNames.push(waitpoint.stepName)
+ seenWaitpointIds.push(waitpoint.id)
+ await resumeWaitpoint({ flowRunId: flowRun.id, waitpointId: waitpoint.id })
+ }
+
+ const result = await pollFlowRunToCompletion(flowRun.id, mockProject.id)
+
+ expect(seenStepNames).toEqual(['step_1:0/step_2', 'step_1:1/step_2', 'step_1:2/step_2'])
+ expect(result.status).toBe(FlowRunStatus.SUCCEEDED)
+ }, 180_000)
+
+ it('advances one step per approval in a three-link approval chain', async () => {
+ const { mockPlatform, mockProject } = await mockAndSaveBasicSetup()
+ await saveWaitpointPieces()
+
+ const chain = approvalAction({
+ name: 'step_1',
+ displayName: 'Approval One',
+ nextAction: approvalAction({
+ name: 'step_2',
+ displayName: 'Approval Two',
+ nextAction: approvalAction({
+ name: 'step_3',
+ displayName: 'Approval Three',
+ nextAction: {
+ type: FlowActionType.CODE as const,
+ name: 'step_4',
+ displayName: 'Done',
+ valid: true,
+ settings: {
+ sourceCode: { code: 'export const code = async () => ({ done: true });', packageJson: '{}' },
+ input: {},
+ errorHandlingOptions: {},
+ },
+ },
+ }),
+ }),
+ })
+
+ const flowRun = await startWaitpointRun({ mockPlatform, mockProject, firstAction: chain })
+
+ const seenStepNames: string[] = []
+ const consumedWaitpointIds: string[] = []
+ for (let link = 0; link < 3; link++) {
+ const waitpoint = await waitForPendingWaitpoint(flowRun.id, consumedWaitpointIds)
+ seenStepNames.push(waitpoint.stepName)
+ await resumeWaitpoint({ flowRunId: flowRun.id, waitpointId: waitpoint.id })
+ consumedWaitpointIds.push(waitpoint.id)
+ }
+
+ const replayStatuses = await Promise.all(
+ consumedWaitpointIds.map((waitpointId) => resumeWaitpoint({ flowRunId: flowRun.id, waitpointId })),
+ )
+ const result = await pollFlowRunToCompletion(flowRun.id, mockProject.id)
+
+ expect(seenStepNames).toEqual(['step_1', 'step_2', 'step_3'])
+ expect(result.status).toBe(FlowRunStatus.SUCCEEDED)
+ expect(replayStatuses.every((status) => status < 500)).toBe(true)
+ }, 180_000)
+
+ it('delivers a DELAY pause and a WEBHOOK pause in one run and leaves no waitpoint behind', async () => {
+ const { mockPlatform, mockProject } = await mockAndSaveBasicSetup()
+ await saveWaitpointPieces()
+
+ const delayThenApproval = {
+ type: FlowActionType.PIECE as const,
+ name: 'step_1',
+ displayName: 'Delay For',
+ valid: true,
+ settings: {
+ pieceName: '@activepieces/piece-delay',
+ pieceVersion: DELAY_PIECE_VERSION,
+ actionName: 'delayFor',
+ input: { unit: 'seconds', delayFor: 11 },
+ propertySettings: {},
+ errorHandlingOptions: {},
+ },
+ nextAction: approvalAction({
+ name: 'step_2',
+ displayName: 'Wait for Approval',
+ nextAction: {
+ type: FlowActionType.CODE as const,
+ name: 'step_3',
+ displayName: 'Done',
+ valid: true,
+ settings: {
+ sourceCode: { code: 'export const code = async () => ({ done: true });', packageJson: '{}' },
+ input: {},
+ errorHandlingOptions: {},
+ },
+ },
+ }),
+ }
+
+ const flowRun = await startWaitpointRun({ mockPlatform, mockProject, firstAction: delayThenApproval })
+
+ const delayWaitpoint = await waitForPendingWaitpoint(flowRun.id)
+ expect(delayWaitpoint.type).toBe(PauseType.DELAY)
+ expect(delayWaitpoint.stepName).toBe('step_1')
+
+ const approvalWaitpoint = await waitForPendingWaitpoint(flowRun.id, [delayWaitpoint.id], 120_000)
+ expect(approvalWaitpoint.type).toBe(PauseType.WEBHOOK)
+ expect(approvalWaitpoint.stepName).toBe('step_2')
+
+ await resumeWaitpoint({ flowRunId: flowRun.id, waitpointId: approvalWaitpoint.id })
+ const result = await pollFlowRunToCompletion(flowRun.id, mockProject.id)
+
+ expect(result.status).toBe(FlowRunStatus.SUCCEEDED)
+ expect(await db.findBy('waitpoint', { flowRunId: flowRun.id })).toHaveLength(0)
+ }, 180_000)
})
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/resume-service.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/resume-service.test.ts
index 926f968200b..1d48e4361e5 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/resume-service.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/resume-service.test.ts
@@ -1,9 +1,12 @@
import { apId } from '@activepieces/core-utils'
import { FlowRunStatus, FlowVersionState, PauseType, RunEnvironment } from '@activepieces/shared'
import { FastifyInstance } from 'fastify'
+import { StatusCodes } from 'http-status-codes'
+import { distributedStore } from '../../../../../src/app/database/redis-connections'
import { resumeService } from '../../../../../src/app/waitpoints/resume-service'
import { waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
import { WaitpointStatus } from '../../../../../src/app/waitpoints/waitpoint-types'
+import { redisMetadataKey } from '../../../../../src/app/workers/job'
import { db } from '../../../../helpers/db'
import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'
import { createTestContext, TestContext } from '../../../../helpers/test-context'
@@ -62,7 +65,16 @@ async function createFlowRunAndWaitpoint(params: {
return { flow, flowVersion, flowRun, waitpointId }
}
-describe('resumeService resumeFromWaitpointWithoutLock', () => {
+async function queueTerminalStatus({ flowRunId, projectId }: { flowRunId: string, projectId: string }) {
+ await distributedStore.merge(redisMetadataKey({ projectId, runId: flowRunId }), {
+ id: flowRunId,
+ projectId,
+ status: FlowRunStatus.CANCELED,
+ requestId: apId(),
+ })
+}
+
+describe('resumeService resumeTrustedWithoutLock', () => {
it('consumes a PENDING waitpoint and enqueues resume when flow is PAUSED (worker-before-callback ordering)', async () => {
const { flowRun, waitpointId } = await createFlowRunAndWaitpoint({
projectId: ctx.project.id,
@@ -70,7 +82,7 @@ describe('resumeService resumeFromWaitpointWithoutLock', () => {
waitpointStatus: WaitpointStatus.PENDING,
})
- const result = await resumeService(app.log).resumeFromWaitpointWithoutLock({
+ const result = await resumeService(app.log).resumeTrustedWithoutLock({
flowRunId: flowRun.id,
waitpointId,
resumePayload: { body: { status: 'approved' } },
@@ -98,7 +110,7 @@ describe('resumeService resumeFromWaitpointWithoutLock', () => {
await db.update('flow_run', flowRun.id, { status: FlowRunStatus.PAUSED })
- const result = await resumeService(app.log).resumeFromWaitpointWithoutLock({
+ const result = await resumeService(app.log).resumeTrustedWithoutLock({
flowRunId: flowRun.id,
waitpointId,
resumePayload: { body: { status: 'early' } },
@@ -126,7 +138,7 @@ describe('resumeService resumeFromWaitpointWithoutLock', () => {
await db.update('flow_run', flowRun.id, { status: FlowRunStatus.PAUSED })
- await resumeService(app.log).resumeFromWaitpointWithoutLock({
+ await resumeService(app.log).resumeTrustedWithoutLock({
flowRunId: flowRun.id,
waitpointId,
resumePayload: { body: { status: 'quick' } },
@@ -154,7 +166,7 @@ describe('resumeService resumeFromWaitpointWithoutLock', () => {
})
const bogusWaitpointId = apId()
- const result = await resumeService(app.log).resumeFromWaitpointWithoutLock({
+ const result = await resumeService(app.log).resumeTrustedWithoutLock({
flowRunId: flowRun.id,
waitpointId: bogusWaitpointId,
resumePayload: { body: { status: 'stale' } },
@@ -166,3 +178,71 @@ describe('resumeService resumeFromWaitpointWithoutLock', () => {
expect(waitpoint).not.toBeNull()
})
})
+
+describe('resume guards against a cancellation in flight', () => {
+ it('refuses a by-run resume once a terminal status is queued but not yet persisted', async () => {
+ const { flowRun } = await createFlowRunAndWaitpoint({
+ projectId: ctx.project.id,
+ flowRunStatus: FlowRunStatus.PAUSED,
+ })
+ await waitpointService(app.log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: ctx.project.id })
+ await queueTerminalStatus({ flowRunId: flowRun.id, projectId: ctx.project.id })
+
+ const { stale } = await resumeService(app.log).legacyResume({
+ flowRunId: flowRun.id,
+ resumePayload: { body: { forged: true }, headers: {}, queryParams: {} },
+ })
+
+ expect(stale).toBe(true)
+ })
+
+ it('refuses a by-run sync resume in the same window', async () => {
+ const { flowRun } = await createFlowRunAndWaitpoint({
+ projectId: ctx.project.id,
+ flowRunStatus: FlowRunStatus.PAUSED,
+ })
+ await waitpointService(app.log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: ctx.project.id })
+ await queueTerminalStatus({ flowRunId: flowRun.id, projectId: ctx.project.id })
+
+ const response = await resumeService(app.log).legacySyncResume({
+ runId: flowRun.id,
+ payload: { body: { forged: true }, headers: {}, queryParams: {} },
+ correlationId: apId(),
+ })
+
+ expect(response.status).toBe(StatusCodes.GONE)
+ })
+
+ it('refuses an addressed resume of a live waitpoint in the same window, leaving the row intact', async () => {
+ const { flowRun, waitpointId } = await createFlowRunAndWaitpoint({
+ projectId: ctx.project.id,
+ flowRunStatus: FlowRunStatus.PAUSED,
+ waitpointStatus: WaitpointStatus.PENDING,
+ })
+ await queueTerminalStatus({ flowRunId: flowRun.id, projectId: ctx.project.id })
+
+ const { stale } = await resumeService(app.log).resumeFromWaitpoint({
+ flowRunId: flowRun.id,
+ waitpointId,
+ resumePayload: { body: { forged: true }, headers: {}, queryParams: {} },
+ })
+
+ expect(stale).toBe(true)
+ expect(await db.findOneBy('waitpoint', { id: waitpointId })).not.toBeNull()
+ })
+
+ it('still resumes a paused run with nothing queued against it', async () => {
+ const { flowRun } = await createFlowRunAndWaitpoint({
+ projectId: ctx.project.id,
+ flowRunStatus: FlowRunStatus.PAUSED,
+ })
+ await waitpointService(app.log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: ctx.project.id })
+
+ const { stale } = await resumeService(app.log).legacyResume({
+ flowRunId: flowRun.id,
+ resumePayload: { body: { approved: true }, headers: {}, queryParams: {} },
+ })
+
+ expect(stale).toBe(false)
+ })
+})
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
index 8a7a7516b92..40b86721a72 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
@@ -3,10 +3,10 @@ import { FlowRunStatus, FlowVersionState, PauseType, RunEnvironment } from '@act
import { FastifyInstance } from 'fastify'
import * as systemJobModule from '../../../../../src/app/helper/system-jobs/system-job'
import { handleResumeDelayWaitpoint } from '../../../../../src/app/waitpoints/resume-delay-handler'
-import { waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
+import { resumeDelayJobId, waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
import { WaitpointStatus } from '../../../../../src/app/waitpoints/waitpoint-types'
import { db } from '../../../../helpers/db'
-import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'
+import { createMockFlow, createMockFlowRun, createMockFlowVersion, mockAndSaveBasicSetup } from '../../../../helpers/mocks'
import { createTestContext, TestContext } from '../../../../helpers/test-context'
import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'
@@ -130,6 +130,31 @@ describe('Waitpoint service', () => {
expect(result.waitpoint.resumePayload).toEqual({ body: { data: 'test' } })
})
+ it('should not hand back a pre-completed waitpoint from another project', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+ const { mockProject: otherProject } = await mockAndSaveBasicSetup()
+
+ const pause = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'approval',
+ type: PauseType.WEBHOOK,
+ })
+ await waitpointService(app.log).complete({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ waitpointId: pause.waitpoint.id,
+ resumePayload: { body: { data: 'test' } },
+ })
+
+ await expect(waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: otherProject.id,
+ stepName: 'approval',
+ type: PauseType.WEBHOOK,
+ })).rejects.toThrow()
+ })
+
it('should correctly map DELAY pause fields', async () => {
const { flowRun } = await createFlowRun()
const resumeAt = new Date(Date.now() + 60000).toISOString()
@@ -174,6 +199,40 @@ describe('Waitpoint service', () => {
expect(retried.waitpoint.id).toBe(first.waitpoint.id)
expect(upsertJobSpy).toHaveBeenCalledTimes(2)
expect(upsertJobSpy.mock.calls[1][0].job.data.waitpointId).toBe(first.waitpoint.id)
+ expect(upsertJobSpy.mock.calls[1][0].job.jobId).toBe(upsertJobSpy.mock.calls[0][0].job.jobId)
+ })
+
+ it('should give each waitpoint of one run its own resume job id', async () => {
+ const { flowRun } = await createFlowRun()
+ const upsertJobSpy = vi.fn()
+ vi.spyOn(systemJobModule, 'systemJobsSchedule').mockImplementation((log) => ({
+ ...originalSystemJobsSchedule(log),
+ upsertJob: upsertJobSpy,
+ }))
+
+ const approval = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'approval_link',
+ type: PauseType.WEBHOOK,
+ })
+ const delay = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'delay_step',
+ type: PauseType.DELAY,
+ resumeDateTime: new Date(Date.now() + 45000).toISOString(),
+ })
+
+ const scheduled = upsertJobSpy.mock.calls.map(([params]) => ({
+ jobId: params.job.jobId,
+ waitpointId: params.job.data.waitpointId,
+ }))
+ expect(scheduled).toEqual([
+ { jobId: resumeDelayJobId(approval.waitpoint.id), waitpointId: approval.waitpoint.id },
+ { jobId: resumeDelayJobId(delay.waitpoint.id), waitpointId: delay.waitpoint.id },
+ ])
+ expect(scheduled[0].jobId).not.toBe(scheduled[1].jobId)
})
it('should reject a resumeDateTime past flowRun.created + AP_PAUSED_FLOW_TIMEOUT_DAYS (cumulative check)', async () => {
@@ -414,7 +473,7 @@ describe('Waitpoint service', () => {
type: PauseType.WEBHOOK,
})
- await waitpointService(app.log).deleteByFlowRunId(flowRun.id)
+ await waitpointService(app.log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: ctx.project.id })
const deleted = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })
expect(deleted).toBeNull()
@@ -432,25 +491,82 @@ describe('Waitpoint service', () => {
})
})
- describe('getByFlowRunId', () => {
+ describe('findUndeliveredCompletedWaitpoint', () => {
it('should return null when no waitpoint exists', async () => {
- const result = await waitpointService(app.log).getByFlowRunId(apId())
+ const result = await waitpointService(app.log).findUndeliveredCompletedWaitpoint({ flowRunId: apId(), projectId: ctx.project.id })
expect(result).toBeNull()
})
- it('should return the waitpoint when it exists', async () => {
- const { flowRun } = await createFlowRun()
+ it('should return the COMPLETED waitpoint when it is the only one left', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+
+ const pause = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'approval',
+ type: PauseType.WEBHOOK,
+ })
+ await waitpointService(app.log).complete({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ waitpointId: pause.waitpoint.id,
+ resumePayload: { body: { ok: true } },
+ })
+ const result = await waitpointService(app.log).findUndeliveredCompletedWaitpoint({ flowRunId: flowRun.id, projectId: ctx.project.id })
+ expect(result?.id).toBe(pause.waitpoint.id)
+ })
+
+ it('should return the COMPLETED waitpoint even while another step of the same run is still PENDING', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+
+ const minted = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'create_approval_links',
+ type: PauseType.WEBHOOK,
+ })
+ await waitpointService(app.log).complete({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ waitpointId: minted.waitpoint.id,
+ resumePayload: { body: { approved: true } },
+ })
await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'wait_for_approval',
+ type: PauseType.WEBHOOK,
+ })
+
+ const result = await waitpointService(app.log).findUndeliveredCompletedWaitpoint({ flowRunId: flowRun.id, projectId: ctx.project.id })
+ expect(result?.id).toBe(minted.waitpoint.id)
+ })
+
+ it('should return null while the run holds a PENDING barrier', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+
+ const webhook = await waitpointService(app.log).createForPause({
flowRunId: flowRun.id,
projectId: ctx.project.id,
stepName: 'approval',
type: PauseType.WEBHOOK,
})
+ await waitpointService(app.log).complete({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ waitpointId: webhook.waitpoint.id,
+ resumePayload: { body: { ok: true } },
+ })
+ await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'fan_out',
+ type: PauseType.BARRIER,
+ })
- const result = await waitpointService(app.log).getByFlowRunId(flowRun.id)
- expect(result).not.toBeNull()
- expect(result!.flowRunId).toBe(flowRun.id)
+ const result = await waitpointService(app.log).findUndeliveredCompletedWaitpoint({ flowRunId: flowRun.id, projectId: ctx.project.id })
+ expect(result).toBeNull()
})
})
@@ -656,7 +772,7 @@ describe('Waitpoint service', () => {
const staleWaitpointId = delayPause.waitpoint.id
// Simulate: delay resolved early, flow continued and paused on approval (new waitpoint)
- await waitpointService(app.log).deleteByFlowRunId(flowRun.id)
+ await waitpointService(app.log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: ctx.project.id })
const approvalPause = await waitpointService(app.log).createForPause({
flowRunId: flowRun.id,
projectId: ctx.project.id,
@@ -725,7 +841,7 @@ describe('Waitpoint service', () => {
workerHandlerId: null,
})
- const result = await waitpointService(app.log).findPendingByVersion({ flowRunId: flowRun.id, version: 'V0' })
+ const result = await waitpointService(app.log).findPendingByVersion({ flowRunId: flowRun.id, projectId: ctx.project.id, version: 'V0' })
expect(result).not.toBeNull()
expect(result!.flowRunId).toBe(flowRun.id)
expect(result!.version).toBe('V0')
@@ -746,7 +862,7 @@ describe('Waitpoint service', () => {
workerHandlerId: null,
})
- const result = await waitpointService(app.log).findPendingByVersion({ flowRunId: flowRun.id, version: 'V0' })
+ const result = await waitpointService(app.log).findPendingByVersion({ flowRunId: flowRun.id, projectId: ctx.project.id, version: 'V0' })
expect(result).toBeNull()
})
@@ -765,7 +881,7 @@ describe('Waitpoint service', () => {
workerHandlerId: null,
})
- const result = await waitpointService(app.log).findPendingByVersion({ flowRunId: flowRun.id, version: 'V0' })
+ const result = await waitpointService(app.log).findPendingByVersion({ flowRunId: flowRun.id, projectId: ctx.project.id, version: 'V0' })
expect(result).toBeNull()
})
})
diff --git a/packages/server/api/test/integration/ce/platform/platform-configuration.test.ts b/packages/server/api/test/integration/ce/platform/platform-configuration.test.ts
index e29fbe868ed..8e832c79b02 100644
--- a/packages/server/api/test/integration/ce/platform/platform-configuration.test.ts
+++ b/packages/server/api/test/integration/ce/platform/platform-configuration.test.ts
@@ -1,4 +1,4 @@
-import { ApFlagId, DefaultProjectRole } from '@activepieces/shared'
+import { ApFlagId, DefaultProjectRole, maxBarrierSignalsBounds } from '@activepieces/shared'
import { FastifyInstance } from 'fastify'
import { StatusCodes } from 'http-status-codes'
import { system } from '../../../../src/app/helper/system/system'
@@ -39,6 +39,50 @@ describe('platform configuration', () => {
expect(configuration.isInfraSetupTelemetryEnabled).toBe(true)
})
+ it('creates the barrier signal cap from AP_MAX_BARRIER_SIGNALS', async () => {
+ const ctx = await createTestContext(app!)
+
+ const configuration = (await ctx.get('/v1/platform-configurations'))!.json()
+
+ expect(configuration.maxBarrierSignals).toBe(system.getNumberOrThrow(AppSystemProp.MAX_BARRIER_SIGNALS))
+ })
+
+ it('persists the barrier signal cap without disturbing telemetry', async () => {
+ const ctx = await createTestContext(app!)
+
+ await ctx.post('/v1/platform-configurations', { maxBarrierSignals: 5 })
+
+ const configuration = (await ctx.get('/v1/platform-configurations'))!.json()
+ expect(configuration.maxBarrierSignals).toBe(5)
+ expect(configuration.isInfraSetupTelemetryEnabled).toBe(true)
+ })
+
+ it('refuses a barrier signal cap below one', async () => {
+ const ctx = await createTestContext(app!)
+
+ const response = await ctx.post('/v1/platform-configurations', { maxBarrierSignals: 0 })
+
+ expect(response?.statusCode).toBe(StatusCodes.BAD_REQUEST)
+ expect((await ctx.get('/v1/platform-configurations'))!.json().maxBarrierSignals).toBe(system.getNumberOrThrow(AppSystemProp.MAX_BARRIER_SIGNALS))
+ })
+
+ it('refuses a barrier signal cap above the ten thousand ceiling', async () => {
+ const ctx = await createTestContext(app!)
+
+ const response = await ctx.post('/v1/platform-configurations', { maxBarrierSignals: maxBarrierSignalsBounds.max + 1 })
+
+ expect(response?.statusCode).toBe(StatusCodes.BAD_REQUEST)
+ expect((await ctx.get('/v1/platform-configurations'))!.json().maxBarrierSignals).toBe(system.getNumberOrThrow(AppSystemProp.MAX_BARRIER_SIGNALS))
+ })
+
+ it('accepts a barrier signal cap sitting exactly on the ceiling', async () => {
+ const ctx = await createTestContext(app!)
+
+ await ctx.post('/v1/platform-configurations', { maxBarrierSignals: maxBarrierSignalsBounds.max })
+
+ expect((await ctx.get('/v1/platform-configurations'))!.json().maxBarrierSignals).toBe(maxBarrierSignalsBounds.max)
+ })
+
it('switches each setting without disturbing the other', async () => {
const ctx = await createTestContext(app!)
diff --git a/packages/server/api/test/integration/ce/workers/refill-paused-jobs.test.ts b/packages/server/api/test/integration/ce/workers/refill-paused-jobs.test.ts
new file mode 100644
index 00000000000..b23d723ead8
--- /dev/null
+++ b/packages/server/api/test/integration/ce/workers/refill-paused-jobs.test.ts
@@ -0,0 +1,171 @@
+import { apId } from '@activepieces/core-utils'
+import { FlowRunStatus, FlowVersionState, PauseType, RunEnvironment } from '@activepieces/shared'
+import dayjs from 'dayjs'
+import { FastifyInstance } from 'fastify'
+import { databaseConnection } from '../../../../src/app/database/database-connection'
+import { redisConnections } from '../../../../src/app/database/redis-connections'
+import { systemJobsSchedule } from '../../../../src/app/helper/system-jobs/system-job'
+import { resumeDelayJobId, waitpointService } from '../../../../src/app/waitpoints/waitpoint-service'
+import { WaitpointStatus } from '../../../../src/app/waitpoints/waitpoint-types'
+import { refillPausedRuns } from '../../../../src/app/workers/migrations/refill-paused-jobs'
+import { db } from '../../../helpers/db'
+import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../helpers/mocks'
+import { createTestContext, TestContext } from '../../../helpers/test-context'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+
+const REFILL_MIGRATION_KEY = 'refill_paused_runs_v7'
+
+let app: FastifyInstance
+let ctx: TestContext
+
+beforeAll(async () => {
+ app = await setupTestEnvironment()
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+beforeEach(async () => {
+ ctx = await createTestContext(app)
+})
+
+async function createPausedRun(): Promise<{ id: string }> {
+ const flow = createMockFlow({ projectId: ctx.project.id })
+ await db.save('flow', flow)
+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })
+ await db.save('flow_version', flowVersion)
+ const flowRun = createMockFlowRun({
+ projectId: ctx.project.id,
+ flowId: flow.id,
+ flowVersionId: flowVersion.id,
+ status: FlowRunStatus.PAUSED,
+ environment: RunEnvironment.PRODUCTION,
+ })
+ await db.save('flow_run', flowRun)
+ return flowRun
+}
+
+async function insertConsumedBarrier({ flowRunId }: { flowRunId: string }): Promise<void> {
+ const now = dayjs().toISOString()
+ await databaseConnection().getRepository('waitpoint').insert({
+ id: apId(),
+ created: now,
+ updated: now,
+ flowRunId,
+ projectId: ctx.project.id,
+ stepName: 'fan_in',
+ type: PauseType.BARRIER,
+ version: 'V1',
+ status: WaitpointStatus.CONSUMED,
+ resumeDateTime: dayjs().add(30, 'day').toISOString(),
+ responseToSend: null,
+ workerHandlerId: null,
+ httpRequestId: null,
+ resumePayload: null,
+ sealed: true,
+ policy: null,
+ })
+}
+
+async function createPendingDelay({ flowRunId, stepName }: { flowRunId: string, stepName: string }): Promise<{ id: string }> {
+ const { waitpoint } = await waitpointService(app.log).createForPause({
+ flowRunId,
+ projectId: ctx.project.id,
+ stepName,
+ type: PauseType.DELAY,
+ version: 'V1',
+ resumeDateTime: dayjs().add(1, 'hour').toISOString(),
+ })
+ return waitpoint
+}
+
+async function loseScheduledTimers({ waitpointIds }: { waitpointIds: string[] }): Promise<void> {
+ for (const waitpointId of waitpointIds) {
+ const job = await systemJobsSchedule(app.log).getJob(resumeDelayJobId(waitpointId))
+ await job?.remove()
+ }
+ const redis = await redisConnections.useExisting()
+ await redis.del(REFILL_MIGRATION_KEY)
+}
+
+async function timerIsArmed({ waitpointId }: { waitpointId: string }): Promise<boolean> {
+ const job = await systemJobsSchedule(app.log).getJob(resumeDelayJobId(waitpointId))
+ return !isNilJob(job)
+}
+
+function isNilJob(job: unknown): boolean {
+ return job === undefined || job === null
+}
+
+describe('refillPausedRuns', () => {
+ it('re-arms the timer of a run whose only waitpoint is a pending DELAY', async () => {
+ const flowRun = await createPausedRun()
+ const delay = await createPendingDelay({ flowRunId: flowRun.id, stepName: 'delay_step' })
+
+ await loseScheduledTimers({ waitpointIds: [delay.id] })
+ expect(await timerIsArmed({ waitpointId: delay.id })).toBe(false)
+
+ await refillPausedRuns(app.log).run()
+
+ expect(await timerIsArmed({ waitpointId: delay.id })).toBe(true)
+ })
+
+ it('re-arms the timer when a consumed barrier tombstone shares the run', async () => {
+ const flowRun = await createPausedRun()
+ await insertConsumedBarrier({ flowRunId: flowRun.id })
+ const delay = await createPendingDelay({ flowRunId: flowRun.id, stepName: 'delay_step' })
+
+ await loseScheduledTimers({ waitpointIds: [delay.id] })
+ await refillPausedRuns(app.log).run()
+
+ expect(await timerIsArmed({ waitpointId: delay.id })).toBe(true)
+ })
+
+ it('re-arms the timer when the tombstone was written after the delay', async () => {
+ const flowRun = await createPausedRun()
+ const delay = await createPendingDelay({ flowRunId: flowRun.id, stepName: 'delay_step' })
+ await insertConsumedBarrier({ flowRunId: flowRun.id })
+
+ await loseScheduledTimers({ waitpointIds: [delay.id] })
+ expect(await timerIsArmed({ waitpointId: delay.id })).toBe(false)
+
+ await refillPausedRuns(app.log).run()
+
+ expect(await timerIsArmed({ waitpointId: delay.id })).toBe(true)
+ })
+
+ it('re-arms every timer when one run holds several pending delays', async () => {
+ const flowRun = await createPausedRun()
+ const first = await createPendingDelay({ flowRunId: flowRun.id, stepName: 'loop_1:0/delay_step' })
+ const second = await createPendingDelay({ flowRunId: flowRun.id, stepName: 'loop_1:1/delay_step' })
+
+ await loseScheduledTimers({ waitpointIds: [first.id, second.id] })
+ await refillPausedRuns(app.log).run()
+
+ expect(await timerIsArmed({ waitpointId: first.id })).toBe(true)
+ expect(await timerIsArmed({ waitpointId: second.id })).toBe(true)
+ })
+
+ it('leaves a delivered waitpoint alone', async () => {
+ const flowRun = await createPausedRun()
+ const delay = await createPendingDelay({ flowRunId: flowRun.id, stepName: 'delay_step' })
+ await databaseConnection().getRepository('waitpoint').update({ id: delay.id }, { status: WaitpointStatus.COMPLETED })
+
+ await loseScheduledTimers({ waitpointIds: [delay.id] })
+ await refillPausedRuns(app.log).run()
+
+ expect(await timerIsArmed({ waitpointId: delay.id })).toBe(false)
+ })
+
+ it('runs once and then leaves the migration key set', async () => {
+ const flowRun = await createPausedRun()
+ const delay = await createPendingDelay({ flowRunId: flowRun.id, stepName: 'delay_step' })
+
+ await loseScheduledTimers({ waitpointIds: [delay.id] })
+ await refillPausedRuns(app.log).run()
+
+ const redis = await redisConnections.useExisting()
+ expect(await redis.get(REFILL_MIGRATION_KEY)).toBe('true')
+ })
+})
diff --git a/packages/server/engine/src/lib/handler/context/step-execution-path.ts b/packages/server/engine/src/lib/handler/context/step-execution-path.ts
index 956e48b9010..6198f05b396 100644
--- a/packages/server/engine/src/lib/handler/context/step-execution-path.ts
+++ b/packages/server/engine/src/lib/handler/context/step-execution-path.ts
@@ -13,6 +13,10 @@ export class StepExecutionPath {
return new StepExecutionPath([])
}
+ waitpointKeyFor({ stepName }: { stepName: string }): string {
+ return [...this.path.map(([loopName, iteration]) => `${loopName}:${iteration}`), stepName].join('/')
+ }
+
removeLast(): StepExecutionPath {
const newPath = this.path.slice(0, -1)
return new StepExecutionPath(newPath)
diff --git a/packages/server/engine/src/lib/handler/piece-executor.ts b/packages/server/engine/src/lib/handler/piece-executor.ts
index 45b03d1e0ff..55693b9f09d 100644
--- a/packages/server/engine/src/lib/handler/piece-executor.ts
+++ b/packages/server/engine/src/lib/handler/piece-executor.ts
@@ -131,7 +131,7 @@ const executeAction: ActionHandler<PieceAction> = async ({ action, executionStat
canPause: !constants.actionRunMode,
stop: createStopHook(params),
respond: createRespondHook(params),
- createWaitpoint: createWaitpointHook({ constants, stepName: action.name, hookParams: params }),
+ createWaitpoint: createWaitpointHook({ constants, waitpointKey: executionState.currentPath.waitpointKeyFor({ stepName: action.name }), hookParams: params }),
waitForWaitpoint: createWaitForWaitpointHook({ constants, hookParams: params }),
},
project: {
@@ -258,14 +258,14 @@ type CreateRespondHookParams = {
hookResponse: HookResponse
}
-function createWaitpointHook({ constants, stepName, hookParams }: { constants: EngineConstants, stepName: string, hookParams: { hookResponse: HookResponse } }): CreateWaitpointHook {
+function createWaitpointHook({ constants, waitpointKey, hookParams }: { constants: EngineConstants, waitpointKey: string, hookParams: { hookResponse: HookResponse } }): CreateWaitpointHook {
return (req: CreateWaitpointParams): Promise<CreateWaitpointResult> => {
assertActionRunCannotSuspend(constants)
- return submitWaitpoint({ constants, stepName, hookParams, req })
+ return submitWaitpoint({ constants, waitpointKey, hookParams, req })
}
}
-async function submitWaitpoint({ constants, stepName, hookParams, req }: { constants: EngineConstants, stepName: string, hookParams: { hookResponse: HookResponse }, req: CreateWaitpointParams }): Promise<CreateWaitpointResult> {
+async function submitWaitpoint({ constants, waitpointKey, hookParams, req }: { constants: EngineConstants, waitpointKey: string, hookParams: { hookResponse: HookResponse }, req: CreateWaitpointParams }): Promise<CreateWaitpointResult> {
const resumeDateTime = cappedForTestFlow({ resumeDateTime: req.resumeDateTime, maxTestWaitMs: req.maxTestWaitMs, isTestFlow: constants.isTestFlow })
assertDelayWithinTimeout(resumeDateTime)
if (!isNil(req.responseToSend)) {
@@ -276,7 +276,7 @@ async function submitWaitpoint({ constants, stepName, hookParams, req }: { const
engineToken: constants.engineToken,
flowRunId: constants.flowRunId,
projectId: constants.projectId,
- stepName,
+ stepName: waitpointKey,
type: req.type,
version: req.version ?? 'V1',
resumeDateTime,
diff --git a/packages/server/engine/test/handler/context/step-execution-path.test.ts b/packages/server/engine/test/handler/context/step-execution-path.test.ts
new file mode 100644
index 00000000000..22482dd140d
--- /dev/null
+++ b/packages/server/engine/test/handler/context/step-execution-path.test.ts
@@ -0,0 +1,22 @@
+import { StepExecutionPath } from '../../../src/lib/handler/context/step-execution-path'
+
+describe('StepExecutionPath#waitpointKeyFor', () => {
+ it('returns the bare step name outside a loop', () => {
+ expect(StepExecutionPath.empty().waitpointKeyFor({ stepName: 'approval' })).toBe('approval')
+ })
+
+ it('qualifies the step name with every enclosing iteration, so each iteration gets its own waitpoint', () => {
+ const path = StepExecutionPath.empty()
+ .loopIteration({ loopName: 'loop_1', iteration: 3 })
+ .loopIteration({ loopName: 'loop_2', iteration: 0 })
+
+ expect(path.waitpointKeyFor({ stepName: 'approval' })).toBe('loop_1:3/loop_2:0/approval')
+ })
+
+ it('gives two iterations of the same loop different keys', () => {
+ const first = StepExecutionPath.empty().loopIteration({ loopName: 'loop_1', iteration: 0 })
+ const second = StepExecutionPath.empty().loopIteration({ loopName: 'loop_1', iteration: 1 })
+
+ expect(first.waitpointKeyFor({ stepName: 'approval' })).not.toBe(second.waitpointKeyFor({ stepName: 'approval' }))
+ })
+})
diff --git a/packages/web/public/locales/en/translation.json b/packages/web/public/locales/en/translation.json
index 8f0499542da..d3f607d4983 100644
--- a/packages/web/public/locales/en/translation.json
+++ b/packages/web/public/locales/en/translation.json
@@ -2848,6 +2848,13 @@
"Responses": "Responses",
"Chat completions suits most gateways. Pick Responses for endpoints that only serve the newer OpenAI Responses API, such as Amazon Bedrock.": "Chat completions suits most gateways. Pick Responses for endpoints that only serve the newer OpenAI Responses API, such as Amazon Bedrock.",
"Configurations": "Configurations",
+ "atLeastOne": "Must be at least 1",
+ "atMostTenThousand": "Must be at most 10,000",
+ "wholeNumber": "Must be a whole number",
+ "Limits": "Limits",
+ "Guardrails that apply to every project on this platform.": "Guardrails that apply to every project on this platform.",
+ "Max things one step can wait on": "Max things one step can wait on",
+ "A step that waits on more than this fails when it runs. Every thing waited on costs database rows, so raise it only as far as your database can carry.": "A step that waits on more than this fails when it runs. Every thing waited on costs database rows, so raise it only as far as your database can carry.",
"Telemetry": "Telemetry",
"Product analytics": "Product analytics",
"Shares usage events so we can see which features are used and fix what breaks.": "Shares usage events so we can see which features are used and fix what breaks.",
diff --git a/packages/web/src/app/routes/platform/infra/configurations/index.tsx b/packages/web/src/app/routes/platform/infra/configurations/index.tsx
index fd9322119de..1679d059559 100644
--- a/packages/web/src/app/routes/platform/infra/configurations/index.tsx
+++ b/packages/web/src/app/routes/platform/infra/configurations/index.tsx
@@ -3,7 +3,7 @@ import {
ApEdition,
ApFlagId,
PlatformConfiguration,
- UpdatePlatformConfigurationRequestBody,
+ PlatformConfigurationSettings,
} from '@activepieces/shared';
import { zodResolver } from '@hookform/resolvers/zod';
import { useMutation, useQueryClient } from '@tanstack/react-query';
@@ -11,7 +11,6 @@ import { t } from 'i18next';
import { useForm } from 'react-hook-form';
import { Navigate } from 'react-router-dom';
import { toast } from 'sonner';
-import { z } from 'zod';
import { platformConfigurationApi } from '@/api/platform-configuration-api';
import { CenteredPage } from '@/app/components/centered-page';
@@ -47,14 +46,14 @@ const ConfigurationsContent = ({
}: ConfigurationsContentProps) => {
const queryClient = useQueryClient();
- const form = useForm<ConfigurationsFormValues>({
+ const form = useForm<PlatformConfigurationSettings>({
defaultValues: toFormValues(configuration),
- resolver: zodResolver(ConfigurationsFormValues),
+ resolver: zodResolver(PlatformConfigurationSettings),
mode: 'onChange',
});
const { mutate: saveConfiguration, isPending } = useMutation({
- mutationFn: (values: ConfigurationsFormValues) =>
+ mutationFn: (values: PlatformConfigurationSettings) =>
platformConfigurationApi.update(values),
onSuccess: async (saved) => {
const productAnalyticsChanged =
@@ -121,16 +120,12 @@ const ConfigurationsSkeleton = () => {
const toFormValues = (
configuration: PlatformConfiguration,
-): ConfigurationsFormValues => ({
+): PlatformConfigurationSettings => ({
isProductTelemetryEnabled: configuration.isProductTelemetryEnabled,
isInfraSetupTelemetryEnabled: configuration.isInfraSetupTelemetryEnabled,
+ maxBarrierSignals: configuration.maxBarrierSignals,
});
-export const ConfigurationsFormValues =
- UpdatePlatformConfigurationRequestBody.required();
-
-export type ConfigurationsFormValues = z.infer<typeof ConfigurationsFormValues>;
-
type ConfigurationsContentProps = {
configuration: PlatformConfiguration;
};
diff --git a/packages/web/src/app/routes/platform/infra/configurations/limits-section.tsx b/packages/web/src/app/routes/platform/infra/configurations/limits-section.tsx
new file mode 100644
index 00000000000..8d69ee2bae7
--- /dev/null
+++ b/packages/web/src/app/routes/platform/infra/configurations/limits-section.tsx
@@ -0,0 +1,77 @@
+import {
+ maxBarrierSignalsBounds,
+ PlatformConfigurationSettings,
+} from '@activepieces/shared';
+import { t } from 'i18next';
+import { Split } from 'lucide-react';
+import { Control } from 'react-hook-form';
+
+import { FormField, FormItem, FormMessage } from '@/components/ui/form';
+import { Input } from '@/components/ui/input';
+import {
+ Item,
+ ItemMedia,
+ ItemContent,
+ ItemTitle,
+ ItemDescription,
+ ItemActions,
+} from '@/components/ui/item';
+
+export const LimitsSection = ({ control, disabled }: LimitsSectionProps) => {
+ return (
+ <div className="flex flex-col gap-3">
+ <div className="flex flex-col gap-1">
+ <h2 className="text-base font-semibold">{t('Limits')}</h2>
+ <p className="text-sm text-muted-foreground">
+ {t('Guardrails that apply to every project on this platform.')}
+ </p>
+ </div>
+ <FormField
+ control={control}
+ name="maxBarrierSignals"
+ render={({ field }) => (
+ <FormItem>
+ <Item variant="outline">
+ <ItemMedia variant="icon">
+ <Split />
+ </ItemMedia>
+ <ItemContent>
+ <ItemTitle>{t('Max things one step can wait on')}</ItemTitle>
+ <ItemDescription className="line-clamp-none">
+ {t(
+ 'A step that waits on more than this fails when it runs. Every thing waited on costs database rows, so raise it only as far as your database can carry.',
+ )}
+ </ItemDescription>
+ </ItemContent>
+ <ItemActions>
+ <Input
+ {...field}
+ id="maxBarrierSignals"
+ type="number"
+ min={maxBarrierSignalsBounds.min}
+ max={maxBarrierSignalsBounds.max}
+ className="w-28"
+ value={field.value ?? ''}
+ onChange={(e) =>
+ field.onChange(
+ e.target.value === ''
+ ? undefined
+ : Number(e.target.value),
+ )
+ }
+ disabled={disabled}
+ />
+ </ItemActions>
+ </Item>
+ <FormMessage />
+ </FormItem>
+ )}
+ />
+ </div>
+ );
+};
+
+type LimitsSectionProps = {
+ control: Control<PlatformConfigurationSettings>;
+ disabled: boolean;
+};
diff --git a/packages/web/src/app/routes/platform/infra/configurations/telemetry-section.tsx b/packages/web/src/app/routes/platform/infra/configurations/telemetry-section.tsx
index e45348d4705..b70e2ef0602 100644
--- a/packages/web/src/app/routes/platform/infra/configurations/telemetry-section.tsx
+++ b/packages/web/src/app/routes/platform/infra/configurations/telemetry-section.tsx
@@ -1,3 +1,4 @@
+import { PlatformConfigurationSettings } from '@activepieces/shared';
import { t } from 'i18next';
import { Activity, Server } from 'lucide-react';
import { Control } from 'react-hook-form';
@@ -15,8 +16,6 @@ import { Switch } from '@/components/ui/switch';
import { TrackedEventsDialog } from './tracked-events-dialog';
-import type { ConfigurationsFormValues } from './index';
-
export const TelemetrySection = ({
control,
disabled,
@@ -95,6 +94,6 @@ export const TelemetrySection = ({
};
type TelemetrySectionProps = {
- control: Control<ConfigurationsFormValues>;
+ control: Control<PlatformConfigurationSettings>;
disabled: boolean;
};
diff --git a/packages/web/test/app/routes/platform/infra/configurations/index.test.ts b/packages/web/test/app/routes/platform/infra/configurations/index.test.ts
new file mode 100644
index 00000000000..e6693f7be9b
--- /dev/null
+++ b/packages/web/test/app/routes/platform/infra/configurations/index.test.ts
@@ -0,0 +1,54 @@
+import { readFileSync } from 'fs';
+import path from 'path';
+
+import { PlatformConfigurationSettings } from '@activepieces/shared';
+import { describe, expect, it } from 'vitest';
+
+const translations: Record<string, string> = JSON.parse(
+ readFileSync(
+ path.resolve(
+ __dirname,
+ '../../../../../../public/locales/en/translation.json',
+ ),
+ 'utf-8',
+ ),
+);
+
+const messageFor = (maxBarrierSignals: unknown): string => {
+ const result = PlatformConfigurationSettings.safeParse({
+ isProductTelemetryEnabled: true,
+ isInfraSetupTelemetryEnabled: true,
+ maxBarrierSignals,
+ });
+ expect(result.success).toBe(false);
+ return result.error!.issues[0].message;
+};
+
+describe('configurations form schema', () => {
+ it.each([
+ ['an empty field', undefined, 'Required'],
+ ['a fractional cap', 1.5, 'Must be a whole number'],
+ ['a non-numeric cap', 'abc', 'Must be a whole number'],
+ ['a cap below the floor', 0, 'Must be at least 1'],
+ ['a cap above the ceiling', 10_001, 'Must be at most 10,000'],
+ ])(
+ 'renders readable English for %s instead of a raw zod message',
+ (_case, input, expected) => {
+ const message = messageFor(input);
+
+ expect(translations[message]).toBe(expected);
+ },
+ );
+
+ it('accepts a cap sitting on each bound', () => {
+ for (const maxBarrierSignals of [1, 10_000]) {
+ expect(
+ PlatformConfigurationSettings.safeParse({
+ isProductTelemetryEnabled: true,
+ isInfraSetupTelemetryEnabled: true,
+ maxBarrierSignals,
+ }).success,
+ ).toBe(true);
+ }
+ });
+});