Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(waitpoints): the barrier's external edges — deadline, create endpoint, confirm links (#14778)

Othman Abu Ajamieh4b91d941567e
diff --git a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
index 94095a8c881..3018b1404f9 100644
--- a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
+++ b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
@@ -45,10 +45,21 @@ all, purely because child rows appear later than the dispatch that created them.
- **Receiving is an upsert, last write wins.** It is not a state machine and does not reject a second write:
a retried child reports its terminal status again — possibly a different one — and the barrier must reflect
the latest truth, not the first. That also makes redelivery of the receive path free.
+- **A human decision is the one exception: it is compare-and-set.** `receiveSignal` keeps the upsert above for
+ machine producers; the confirm link goes through `recordDecision`, whose `UPDATE` carries
+ `AND status = PENDING` and answers "already responded" when it matches nothing. A double-click, an email
+ client prefetching on hover, or two people on a shared inbox must not silently flip a recorded approve into
+ a reject. Both paths write with an `UPDATE … RETURNING "waitpointId"`, never read-then-`save()`, which
+ *re-inserted* a signal row the release had already deleted. `recordDecision` also takes `FOR SHARE` on the
+ barrier first, so `closeBarrier`'s `FOR UPDATE` waits for it: a decision is either counted in the summary or
+ refused, never dropped between the count and the signal delete. `receiveSignal` takes no lock; a machine
+ signal that commits during a close can miss the summary, which "last write wins" already accepts.
- **The deadline is set at create and nothing moves it.** The floor rule needs an evaluation to fire; a
barrier nobody ever signals gets none, so the deadline is the only thing between "the producer died hard"
and a run paused until retention deletes it. Seal does not touch it — the clamp with no requested value
already returns the ceiling, and a shorter deadline is a `policy` input, so both are known at create.
+ `defaultBarrierDeadline({ flowRunCreated })` anchors it to the run's start, not to the moment the barrier
+ opened, so a barrier inside a long-running run cannot outlive `AP_PAUSED_FLOW_TIMEOUT_DAYS`.
- **Evaluation must be coalesced, and coalescing must not swallow its own last signal.** At 10 000 signals,
evaluating per signal is ~100M row reads. BullMQ holds a deduplication key while the job is queued *and*
active, so the handler's **first statement** is `clearEvaluationDedupKey`, and every producer commits
diff --git a/brain/knowledge/engineering/ci-pr-review-hygiene.md b/brain/knowledge/engineering/ci-pr-review-hygiene.md
index f8f33538c53..00a16b5ebce 100644
--- a/brain/knowledge/engineering/ci-pr-review-hygiene.md
+++ b/brain/knowledge/engineering/ci-pr-review-hygiene.md
@@ -72,5 +72,6 @@ Enforcement is the **`Codeowners review` repository ruleset** (active on the def
- **Fixing an infra failure can turn a branch from red to a different red, and that is progress, not a regression.** A runner killed mid-run (`shutdown signal` above `exited (137)`) reports no test results at all, so genuine failures underneath it are invisible. When the OOM is fixed the suites finally run to completion and those failures surface for the first time — on this stack, thirteen branches went from an OOM red to a real `barrier.test.ts` red the moment the runner survived. Check whether the *previous* run reached a test summary at all before blaming the change that unmasked it.
- **A `beforeAll` that overrides the vitest timeout *downwards* is a slow-motion flake, and it fails in a way that blames your diff.** [`execute-flow-e2e.test.ts`](../../../packages/server/api/test/integration/ce/flows/flow-run/execute-flow-e2e.test.ts) passes `30_000` to `beforeAll` even though `packages/server/api/vitest.config.ts` sets `hookTimeout: 60000`, and spends 5s of that budget on a hardcoded `setTimeout`. Inside the remaining ~25s it boots a fresh app, runs the **whole migration chain**, starts queue consumers and a worker. That chain has grown 293 → 376 postgres migrations since the hook was written (`d2c42816013`, "feat: worker v2", #11608, 2026-03-15 — authored *after* the 60s default was already in place), so on a loaded runner it now times out. When it does, all 9 of its tests report **skipped**, not failed: the summary reads `1 failed | 84 passed` with no failing assertion anywhere, and `--bail 1` kills the rest of the CE task. Seen on [#15078](https://github.com/activepieces/activepieces/pull/15078), an MCP-OAuth-only PR that cannot reach that code. No single commit broke it, so there is nobody to blame and nothing to revert — the fix is to delete the override and poll for worker readiness instead of sleeping blind. When triaging a red CE task, check whether the failure is an assertion or a hook timeout **before** blaming the diff.
- **Git hooks cannot run from a `git worktree`: `.husky/_/husky.sh` is an install artifact that only exists in the tree where `bun install` ran.** Every commit and push from a worktree dies on `.: cannot open .husky/_/husky.sh`, before the hook's own logic is reached. Don't reach for `--no-verify` — push the worktree's commit from the main checkout (`git push origin <local-branch>:<remote-branch>`; worktrees share the object store), so `pre-push` actually bootstraps and its "no direct pushes to `main`" guard runs. `SKIP_CHECK=1` is the repo's own sanctioned way to skip the lint/test prompt. Note `pre-push` special-cases `CLAUDE_PUSH=yes` into running the **full** lint + unit + API suite, so an agent that silently bypasses the hook is skipping a check the repo added deliberately for it — say so out loud when you do.
+- **Merging up a stacked chain, a parent's rename reaches a child's call sites as a clean auto-merge — only the typecheck finds it.** Git conflicts on lines both sides edited, so a symbol renamed on the parent merges silently into a child that merely *calls* the old name: `feat/barrier-service` renamed `barrierService.receive` → `receiveSignal`, and `feat/barrier-edges`' `resume-controller.ts` kept calling `.receive(...)` with no conflict marker. Worse, both sides can each be right and still merge wrong — the same commit renamed `findPreCompletedByFlowRunId` while the child had rewritten that function's *body* to fix a real bug, so the conflict exposed only the signature and taking either side whole silently dropped work. Two habits: typecheck **every** branch in the chain after its merge rather than trusting a conflict-free result, and before picking a side run `git log --oneline $(git merge-base HEAD MERGE_HEAD)..HEAD -- <file>` on both to read what each side was actually doing. Stacks here run deep (the waitpoints → process-in-batches chain was 17 PRs), so one missed rename propagates through every branch above it.
- **Docs videos belong on `cdn.activepieces.com/videos/docs/`, never in the repo — and `docs/resources/` is the legacy path that suggests otherwise.** Every `<video>` already in the wiki points at that CDN prefix, but `docs/resources/` still holds ~43 MB of in-repo gifs from before the convention, so the tree itself gives mixed signals and a contributor reasonably copies the wrong one. [#15399](https://github.com/activepieces/activepieces/pull/15399) added a fresh `docs/videos/` folder with 155 MB of `.mp4` against an 880 MB `.git`, including one byte-identical duplicate. Nothing in CI catches it: `PR size` exempts unmatched areas and counts lines, not bytes, and a binary blob contributes zero lines. Review the *file list* of a docs PR, not just its diff. Once such a branch is pushed the blobs are already on the remote — dropping them at the tip only keeps them out of `main` if the PR is **squash**-merged and the branch deleted, so a merge commit ships them permanently. Because the blobs stay reachable until the branch is deleted, nobody has to re-request the assets from whoever recorded them — pull them straight back out of history with `git archive <commit> docs/videos | tar -x --strip-components=2 -C <dir>`, picking the commit *before* the removal, which is also how you get them under the post-rename filenames the pages already reference.
- **Mintlify-editor PRs arrive with `untitled-page*.mdx` at the docs root and `(1)`/`(2)`-suffixed media, wired into `docs.json` under those names.** The bot commits whatever the web editor autosaved, so pages land as `docs/untitled-page-5.mdx` rather than in the folder their nav group implies, and re-uploaded assets keep the browser's download-collision suffix (`data-flow-(1).mp4`, `01---insert-panel-(1.2x).mp4`). Mintlify's own link check passes — the names are internally consistent — so nothing flags it. Rename the files, fix the `docs.json` entries, then diff the *flattened nav tree* against `main` rather than reading the raw JSON diff: the editor also reindents, which buried a real change (the Enterprise Control `Security` group demoted to a child of `Guides`) inside 242 lines of whitespace churn on that PR.
diff --git a/brain/knowledge/flows-execution/flow-runs.md b/brain/knowledge/flows-execution/flow-runs.md
index a9d738711b4..9659c1c166b 100644
--- a/brain/knowledge/flows-execution/flow-runs.md
+++ b/brain/knowledge/flows-execution/flow-runs.md
@@ -9,33 +9,43 @@ A Flow Run records one execution of a specific flow version, from trigger to ter
### Entities & services
- **FlowRun** — id, projectId, flowId, flowVersionId, environment (PRODUCTION/TESTING), status, logsFileId, parentRunId (subflows), failedStep (JSONB `{name, displayName, message?}`), timeline (JSONB), archivedAt (soft delete).
- **12 statuses**: 3 non-terminal (QUEUED, RUNNING, PAUSED) + 9 terminal (SUCCEEDED, FAILED, TIMEOUT, CANCELED, QUOTA_EXCEEDED, MEMORY_LIMIT_EXCEEDED, INTERNAL_ERROR, LOG_SIZE_EXCEEDED).
-- **Waitpoint** — row per paused step: `type` (DELAY|WEBHOOK|BARRIER), `version` (V0|V1), `status` (PENDING|COMPLETED), unique on `(flow_run_id, step_name)`. A BARRIER adds `sealed` (bool) and a nullable `policy` jsonb. A partial index on `resumeDateTime` (`WHERE status = 'PENDING' AND "resumeDateTime" IS NOT NULL`) serves the deadline sweep — every other read path leads with `flowRunId` or the primary key.
+- **Waitpoint** — row per paused step: `type` (DELAY|WEBHOOK|BARRIER), `version` (V0|V1), `status` (PENDING|COMPLETED), unique on `(flow_run_id, step_name)`. A BARRIER adds `sealed` (bool) and a nullable `policy` jsonb, and any row may carry `deadLetteredAt`. A partial index on `resumeDateTime` (`idx_waitpoint_live_deadline`, `WHERE status = 'PENDING' AND "resumeDateTime" IS NOT NULL AND "deadLetteredAt" IS NULL`) serves the deadline sweep; every read path leads with `flowRunId` or the primary key, but `idx_waitpoint_project_id` stays because `fk_waitpoint_project_id` is `ON DELETE CASCADE` and a project delete seq-scans the table without it.
- **WaitpointSignal** — one row per thing a BARRIER awaits, created up front: `status` (PENDING|SUCCEEDED|FAILED|REJECTED|CANCELED|NOT_DISPATCHED), `refId` (the child run or link it stands for), nullable `sequence` (the producer's ordinal, partial-unique per barrier), nullable `label`, small `result` jsonb. Release is the pure `shouldReleaseBarrier({ policy, sealed, counts })` in `core-execution` — see decision [000015](../decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md).
- **LogsFile** — zstd-compressed File (type FLOW_RUN_LOG) holding the full executor context.
### How it works
- **Endpoints**: `GET /` (cursor paginated by composite `(created DESC, id DESC)`, filters incl. `failedStepMessage` ILIKE), `GET /:id`, `POST /:id/retry`, `POST /retry|cancel|archive` (bulk), waitpoint resume routes.
- **Retry strategies**: `FROM_FAILED_STEP` (rebuild context from logs, re-run from failure, prior outputs kept) or `ON_LATEST_VERSION` (fresh run on current published version). Both resolve the trigger payload via `resolveStepOutput`. If the trigger itself failed, they switch to `executeTrigger: true` to reprocess the raw event.
-- **Pause/resume (V1 waitpoints)**: pieces call `createWaitpoint` + `waitForWaitpoint`. DELAY upserts a `RESUME_DELAY_WAITPOINT` BullMQ job; WEBHOOK resumes on an HTTP call to `/:id/waitpoints/:waitpointId[/sync]`.
+- **Pause/resume (V1 waitpoints)**: pieces call `createWaitpoint` + `waitForWaitpoint`. Any waitpoint carrying a `resumeDateTime` upserts a `RESUME_DELAY_WAITPOINT` job keyed **per waitpoint** (`systemJobIds.resumeDelay`), removed when it resumes or is deleted; WEBHOOK resumes on an HTTP call to `/:id/waitpoints/:waitpointId[/sync]`; a BARRIER resumes on its own predicate, or on its deadline via the one-a-minute `WAITPOINT_DEADLINE_SWEEP`.
- Logs backed up every 15s during execution for crash recovery; uploaded via 7-day JWT-signed URLs.
- **RUN_TELEMETRY job**: `flow-run-module.ts` registers a BullMQ system job (cron `50 23 * * *`, once daily at 23:50 UTC) that aggregates the day's run counts by `(projectId, flowId, environment)` in one transaction (5-minute statement timeout) and emits a `FLOW_RUN_CREATED` telemetry event per group. No-op when telemetry is disabled. The cron was `0/50 23 * * *` until GIT-1632, which also fired at 23:00 with partial counts.
### Gotchas
- **A Delay inside a Loop pauses and requeues the whole run once per iteration, so no fixed sync-webhook timeout can cover it.** The delay is not a sleep inside the step: each iteration arms a waitpoint, the run goes `PAUSED`, and the resume comes back through the queue, so every item costs its delay plus queue latency and the total scales with the item count. Worked case on dev: `Catch Webhook → Code → Loop { Delay For 8s } → Return Response` over 6 items reported `stepsCount` 9 (Code + Loop + 6 delays + Return Response) and ran 53.6s, of which 48s was delay and 3s was the initial queue leg. With `AP_WEBHOOK_TIMEOUT_SECONDS` at 30 the `/sync` caller was answered 408 mid-loop; Return Response then ran ~23s later and published to a listener that had already resolved and been deleted, so it was a no-op. Putting the response step *after* slow work is the bug: respond before it (respond-and-continue rather than `stop`) or go async with a callback, because raising the timeout only works until someone sends more items. See the sync-response gotchas in [webhooks](../eventing-webhooks/webhooks.md).
- **A worker OOM-kill leaves the run stuck in RUNNING forever, and Cancel is greyed out.** The flow timeout is enforced *inside* the worker, so if the pod dies (OOM) nothing ever transitions the run to a terminal state; Cancel only applies to paused/queued runs, so the UI offers no way out and the run can't be retried either. Bug: activepieces#14372, fix PR #14374. Manual unblock on the customer's Postgres: `UPDATE flow_run SET status = 'CANCELED', "finishTime" = NOW(), updated = NOW() WHERE id = '<run id>' AND status = 'RUNNING';` (run id = last path segment of the run URL), then "Retry on latest version" replays the original payload.
+- **A per-signal confirm link's only guard is that every segment of it is unguessable.** `/:id/signals/:signalId/confirm` is `securityAccess.unscoped(ALL_PRINCIPAL_TYPES)`, so the URL carries the signal's random `apId` and never a caller-supplied key — a semantic segment (`approver-a`, an email, an index) is guessable by string substitution, and **one legitimate approver holding their own link could cast a whole K-of-N quorum**. `label` is the audit name in the summary, never the credential.
+- **A barrier's deadline is anchored to the run's start, and reaching it releases the barrier rather than failing the run.** `defaultBarrierDeadline` takes `flowRun.created + PAUSED_FLOW_TIMEOUT_DAYS`, not `now + PAUSED_FLOW_TIMEOUT_DAYS`: a barrier opens partway through a run, so a creation-anchored deadline always lands *after* the run's own pause-timeout boundary. That ordering made `handleResumeDelayWaitpoint`'s run-age guard unconditionally true for every naturally scheduled barrier deadline, so the run was marked `FAILED` and the `release({ timedOut: true })` branch below it was dead code — the timeout summary, and every signal already collected, were thrown away. The handler now routes `PauseType.BARRIER` to `release` *before* the run-age guard; the guard still governs `DELAY` and `WEBHOOK`. Tests that backdate only `waitpoint.resumeDateTime` and leave `flowRun.created` fresh cannot see this — backdate the run too.
+- **A waitpoint's deadline is set when it is created, never at seal.** A barrier that took its deadline at seal was invisible to the sweep until sealed — safe only while the parent went PAUSED *after* the seal returned. The moment a producer pauses the parent before its awaited things exist, that accident ends: a PAUSED run, an unsealed barrier, no deadline, and no sweep coverage until retention deletes it.
+- **The `RESUME_DELAY_WAITPOINT` job id is per waitpoint, not per run.** It was `resume-delay-${flowRunId}`, which collided the moment one run held two timed waitpoints — a delay inside a loop upserted iteration 2's timer over iteration 1's, and BullMQ keeps the *existing* one-time job, so the second iteration's deadline silently never fired. `systemJobIds.resumeDelay({ waitpointId })` is the fix; `systemJobIds.legacyResumeDelay` exists only so `remove` can still find a job scheduled under the old id.
+- **A timeout job is usually removed by itself, while it is still running.** `handleResumeSignal` removes the waitpoint's timeout job after every dispatch, and when the timer is what fired, that job is locked by the worker running it. `Job.remove()` throws on a locked job, which logged a false warning on every timer-driven resume; `systemJobsSchedule.removeJob` uses `Queue.remove()`, which answers 0 instead, and leaves the job to `removeOnComplete`.
+- **The deadline sweep pages on a keyset cursor, because its rows survive the pass.** An overdue waitpoint stays `PENDING` with the same `resumeDateTime`, so the repo's delete-and-repeat drain loop would re-read the first page forever. `sweepOverdueDeadlines` pages oldest-first on `("resumeDateTime", "id")`, and `idx_waitpoint_live_deadline` carries both columns so a page inside a large tie group stays cheap. A tick that spends its arm quota (`MAX_ARMED_PER_TICK`) or page budget (`MAX_SCAN_PAGES_PER_TICK`) stores the last row it handled in `distributedStore` (`waitpoint:deadline-sweep:cursor`, 10-minute TTL) and the next tick resumes there; a tick that reaches the end deletes it. The scan has no lower bound on purpose: a row overdue past `PAUSED_FLOW_TIMEOUT_DAYS` is exactly the stuck run it exists to find, and nothing deletes waitpoint rows on a schedule.
+- **A dead-lettered deadline is stamped in Postgres so the scan can skip it.** Whether a job exhausted its attempts lives in Redis, which a `WHERE` cannot read, so the probe stamps `waitpoint.deadLetteredAt` and the partial `idx_waitpoint_live_deadline` (`deadLetteredAt IS NULL`) drops the row; without it, a burst of failed resumes fills the head of every scan. `createForPause` clears the stamp when its re-arm retries the failed job or finds the row stamped, and the stamp is guarded on `updated <= <scan start>` read from the database clock (`SELECT now()`), so a probe that ran before that retry cannot re-hide the row. It is not grounds to fail the run: two attempts cannot tell a dead flow from a database outage, and for a barrier the deadline is the timeout branch.
+- **`armed` counts only what `upsertJob` reports as `'added'`.** `upsertJob` keeps an existing live one-time job and silently drops the requested schedule, so pushing to the result list unconditionally reported re-arms that never happened — and made the sweep's own test vacuous, since `barrierService.create` already schedules a job at the 30-day deadline and the test only backdated the DB column. `upsertJob` now returns a `UpsertJobResult` status (`added` | `kept` | `retried` | `scheduler-upserted`). Note the sweep still probes with its own `getJob` first rather than reading `'retried'`: `upsertJob` calls `retry()` *before* it returns, which is the very thing the probe exists to prevent.
+- **A confirm link is open while the run is `PAUSED`, `RUNNING` *or* `QUEUED` — not `PAUSED` alone.** The piece emails the links during the step, so the barrier and its signals exist and are `PENDING` for as long as it takes the engine to finish and the metadata worker to write `PAUSED`. Gating the link on `PAUSED` meant a fast approver got a 200 "Already responded" page and their decision was never recorded or logged. Those three are exactly the states `waitpointService.handleResumeSignal` treats as resumable (it completes the waitpoint in place while `RUNNING`/`QUEUED` and lets the `PAUSED` upload trigger the resume), so the link's gate and the resume path now agree.
- **Resume Confirmation Page (scanner guard)**: the `/confirm` route serves an HTML Approve/Disapprove page on `GET`/`HEAD` (never consumes) and only resumes on `POST` — stops email security scanners (Safe Links, Mimecast, Proofpoint) prefetching approval links. The deprecated bare `GET /:id/waitpoints/:waitpointId` still resumes for old emails. Slack is unchanged (server-side POST from webhook).
- **`POST /confirm` with an empty JSON body is rejected before it reaches the resume code.** Sending `Content-Type: application/json` with no body gets Fastify's `FST_ERR_CTP_EMPTY_JSON_BODY` 400, so the waitpoint is never consumed and the run just stays PAUSED — which reads exactly like a resume regression. `curl -X POST .../confirm?action=approve` is fine because curl sends no content-type at all; an HTTP client that defaults to JSON is not. Either omit the content-type or send a real body (`{}`). Hit while scripting the `/test-waitpoints` flow 4 race ordering, 2026-08-30.
- **Cross-project isolation (subflow parent-fail)**: `markParentRunAsFailed` scopes its parent lookup to `{ id: parentRunId, projectId }` using the child run's authenticated `projectId`. `parentRunId`/`failParentOnFailure` arrive from spoofable webhook headers (`ap-parent-run-id`/`ap-fail-parent-on-failure`) on the public webhook endpoint, so without the scope a failed child in project A could complete a paused parent's waitpoint and resume it in project B. A cross-project parent id now matches nothing and the fail is a no-op; legitimate subflows are always same-project (Call Flow only targets flows in the caller's project).
- **A resume link resumes the run, not the step it was minted for.** Waitpoints are unique on `(flowRunId, stepName)`, so a flow that runs `create_approval_links` and then `wait_for_approval` holds **two PENDING rows at once** — and the link handed out by `create_approval_links` points at the *first* row while the run blocks on the *second*. `handleResumeSignal`'s PAUSED branch looks the row up by `{ id, flowRunId }` with no check that it belongs to the step being waited on, so that link resumes the run anyway and its payload lands on whichever step is paused. Verified 2026-08-30 on a local instance: POSTing the step_1 link while PAUSED on step_3 resumed the run and gave step_3 `approved: true`. Upshot: an approval link is a **run-level** capability, not a step-level one.
- **A resume that arrives while the run is still RUNNING is parked, and delivered later by the `runsMetadataQueue` worker.** `handleResumeSignal`'s RUNNING/QUEUED branch takes `complete()`, which marks the row COMPLETED rather than deleting it; the worker then picks it up on writing PAUSED via `findUndeliveredCompletedWaitpoint` (`flow-runs-queue.ts`) and enqueues the resume. Because delivering a resume never leaves the row at COMPLETED (both `handleResumeSignal`'s PAUSED branch and the race branch in `resumeTrustedWithoutLock` go through `waitpointService.consume`, which deletes a non-barrier and tombstones a barrier as CONSUMED), any row still sitting at COMPLETED is by construction one whose signal was never delivered — which is why that lookup does not need to care how many other waitpoints the run holds. It declines only when the run holds a PENDING **barrier**, since a barrier may be released solely by its predicate or deadline. **Gotcha for anyone narrowing this check:** an earlier form returned null whenever *any* PENDING row existed, which hung every run holding a second waitpoint — `create_approval_links` + `wait_for_approval` sat in PAUSED forever with the approval already consumed (`step_3` output `{approved: true}`, status `PAUSED`). Reproduced and fixed 2026-08-30 on `feat/barrier-service` with flow 4 of `/test-waitpoints`. Single-waitpoint runs (the common subflow-callback case) never showed it, so unit tests and every other flow in the suite stayed green.
- **`create_approval_links` hands out an approval and a disapproval URL that are byte-identical** — both are the bare `/confirm` route with no `?action=`. The verdict comes from the button the human clicks on the confirmation page (which POSTs back with the action), not from which link they were given, and `wait_for_approval` reads `queryParams.action` with anything other than `approve` counting as a disapproval.
-- **A run can only have ONE scheduled DELAY resume at a time, and arming a second one is silently dropped.** `createForPause` schedules the resume job with `jobId: resume-delay-${flowRunId}` — keyed on the *run*, not the waitpoint — and `systemJobsSchedule.upsertJob` only calls `queue.add` when no job with that id exists (it retries an existing *failed* job, and otherwise does nothing at all, with no log beyond "Upserting job"). So if a run's first DELAY waitpoint is removed **before its job fires** and the run then arms another DELAY, the second schedule is a no-op: the queue still holds the first job, pointing at the now-deleted waitpoint. That job fires on the old date, `handleResumeSignal` finds nothing for the stale id, logs "Stale waitpointId, ignoring", and the job is consumed — leaving the run **PAUSED forever** with an overdue `resumeDateTime` and nothing in `bull:system-job-queue:delayed`. Reproduced 2026-08-30 by resuming a loop's iteration-0 delay early via `POST /:id/waitpoints/:waitpointId`: iteration 1 armed at +1s but the job kept iteration 0's id and fire time, and the run never completed. **This is on `main`, not a barrier-branch regression.** Normal delay flows are unaffected only because `removeOnComplete: true` frees the id once a job fires. Note `removeOnFail` keeps failed jobs for a month, so a single failed delay job also blocks every later DELAY in that run — and `upsertJob`'s retry re-runs the *old* `waitpointId`. Any path that resumes a run parked on a DELAY (the deprecated bare `GET` resume link, a subflow callback via `findSubflowWaitpoint`, a deadline sweep) can set this up. Fix direction: key the job on the waitpoint (`resume-delay-${waitpoint.id}`), which also needs `refill-paused-jobs.ts` updated since it reconstructs the same id.
- **A barrier's waitpoint row survives its release, and the legacy by-run resume guard keys off that row's existence.** Until 2026-08-30 `legacyResume` / `legacySyncResume` (the deprecated V0 `/:id/requests/:requestId` routes, reached whenever `findPendingByVersion('V0')` returns null) checked `hasPendingBarrier`, which filters `status: PENDING` — so the guard stopped guarding the instant the barrier was released. `closeBarrier` commits COMPLETED in its own transaction and only *then* does `release()` call the trusted resume under the `runs_metadata_${flowRunId}` lock, and the legacy paths take no lock at all. Through that gap **and** the much wider one after delivery (the run stays PAUSED until the engine's next progress upload — `flow_run.status` is written only by the `runsMetadataWorker`), the run was PAUSED with no PENDING barrier, so a legacy resume was accepted and enqueued a **second** job with a request-controlled payload: `enqueueResume` namespaces job ids as `${runId}-resume-legacy` vs `${runId}-resume-${barrierId}` and the worker queue has no BullMQ `deduplication`, so nothing collapsed them. Reproduced against the dev DB on `feat/barrier-service`, and identical on all four stacked branches. **Fixed** by making delivery explicit rather than encoded as row-absence: a delivered barrier is tombstoned `CONSUMED` (see [decision 000033](../decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md)) and the legacy guard became `hasBarrier` — existence, no status filter. **Gotcha for anyone touching these two methods:** `hasPendingBarrier` still exists and `findUndeliveredCompletedWaitpoint` must keep calling *it*, not `hasBarrier`; switching that call site to existence would return null whenever a CONSUMED barrier is present and hang a later non-barrier delivery on the same run. Two different questions, two methods. The V1 addressed path never had the hole — `addressedWaitpointIsBarrier` rejects on `waitpoint.type === BARRIER` whatever the status.
- **`handleResumeSignal`'s PAUSED branch must reject `CONSUMED` only — never require `PENDING`.** The three statuses are a delivery state machine, not a lifecycle: PENDING is *not yet owed*, COMPLETED is *owed*, CONSUMED is *delivered*. Requiring PENDING therefore breaks the pre-completed path, because `findUndeliveredCompletedWaitpoint` hands this branch a COMPLETED row on purpose whenever a callback beat the pause. Tried on `feat/barrier-service` 2026-09-22 while answering a review that proposed exactly that: it failed **nine** tests, four of which predate the barrier work, including `consumes a COMPLETED waitpoint when race recovery fires` and `does not leave a stale COMPLETED row to poison the next pause cycle`. Before that fix the branch had no status filter at all, so a CONSUMED barrier tombstone could be dispatched a second time.
- **That `CONSUMED` check does not need a conditional `UPDATE ... WHERE status = ... RETURNING`, and two reviewers have now asked for one.** The branch already runs `SELECT ... FOR UPDATE` inside `transaction()`, and TypeORM/Postgres default to READ COMMITTED, where a `FOR UPDATE` that blocks on a concurrent writer does **not** return its original snapshot on release — EvalPlanQual follows the update chain and re-checks the `WHERE` against the newest committed version. The predicate is `id` + `flowRunId` + `projectId`, all immutable, so the row comes back with the committed status. Read-then-branch-then-write in that transaction has exactly the atomicity of a conditional UPDATE. A conditional UPDATE is also actively worse here: `consume` is polymorphic (DELETE for a non-barrier, UPDATE for a barrier), and `onReady` has to run *between* the status check and the status write, which a single `RETURNING` statement cannot host.
- **`onReady` runs inside that transaction deliberately — do not move it after the commit.** The order is dispatch → consume → commit, so a crash in the window leaves a resume job plus an unconsumed row: recoverable, and the row is the evidence that recovery is owed. Committing first would leave a `CONSUMED` tombstone (or, for a non-barrier, nothing at all) with no job, which is an unrecoverable hang and a tombstone that lies. The real cost of the current order is that `enqueueResume` does a platform lookup and an unconditional payload offload — taking a *second* pool connection — while holding `FOR UPDATE`. Fix that by hoisting the offload above the transaction, never by reordering the commit.
- **Widening `releaseIfReady`'s lookup to COMPLETED recovers nothing on its own.** It still routes into `release` → `closeBarrier`, which filters `status: PENDING`, returns null, and makes `release` return *before* `resumeTrusted`. A crash between `closeBarrier`'s commit and `resumeTrusted` otherwise strands the run PAUSED with a COMPLETED barrier forever: `releaseIfReady` used to filter PENDING, the deadline handler returns unless the waitpoint is PENDING, and `closeBarrier` deleted the signals so nothing re-enqueues an evaluation. The recovery has to branch on status and re-dispatch the summary already persisted on the row, bypassing `closeBarrier` — routed through the same `resumeTrusted` call the normal path uses, so it inherits the run lock and the CONSUMED guard instead of enqueuing beside them.
- **That recovery is bounded, not complete.** The only automatic re-invoker is BullMQ's stalled check on `BARRIER_EVALUATION`, and the worker takes the default `maxStalledCount: 1` — exactly one free redelivery after a SIGKILL. A second stall retires the job to `failed` permanently, and a Redis flush removes the trigger altogether while Postgres still holds a COMPLETED barrier on a PAUSED run. `attempts: 5` and the 8-minute backoff do not help: they govern *failed* jobs, and a retry just re-runs `releaseIfReady`. The durable backstop is a sweep, and a deadline sweep only closes this if it selects `type = BARRIER AND status = COMPLETED` — the natural `status = PENDING` reading never sees the state. The same hole exists for non-barrier waitpoints (`complete()` commits COMPLETED, the caller dies before dispatch) and is covered only opportunistically by the metadata worker.
+- **The deadline sweep carries a second, separate scan for undelivered barriers, and it cannot be folded into the first.** `findOverdueWaitpoints` selects `status = PENDING AND resumeDateTime < now`; a barrier closed a second ago is COMPLETED with its deadline still 30 days out, so widening that query would not see it until the deadline passed. `redeliverUndeliveredBarriers` therefore scans `type = BARRIER AND status = COMPLETED` on a PAUSED run, keyed on `updated` rather than `resumeDateTime`, and calls `releaseIfReady` — which re-dispatches the stored summary. A two-minute grace window keeps it off barriers the normal release is still delivering (that path takes milliseconds), and the scan is covered by the partial index `idx_waitpoint_undelivered_barrier`. A barrier that cannot be delivered at all — its run row gone, say — stays COMPLETED and is retried every tick within the per-tick cap rather than dead-lettered; it is loud in the logs and rare, and retrying is what makes a transient failure self-heal.
+- **`releaseIfReady` rethrows a failed re-dispatch, so every loop that calls it in a batch must guard each row.** `redeliverClosedBarrier` logs and rethrows: that is what spends the `BARRIER_EVALUATION` job's `attempts: 5`, and a swallowed error there marks the job successful and throws the retry budget away. The cost is that the sweep calls the same method for up to a hundred rows, so a bare rethrow aborts the tick on the first bad row — and since the scan orders by `updated ASC`, every later tick meets that same row and aborts again, starving every other undelivered barrier. `redeliverUndeliveredBarriers` therefore wraps each row in `tryCatch` and logs. The two belong together: the rethrow is the queue's signal, the per-row guard is what keeps it from becoming a head-of-line block.
- **A waitpoint inside a loop is keyed by its full execution path**, e.g. `step_1:0/step_2`, not the bare `step_2` — so each iteration gets its own row under the `(flowRunId, stepName)` unique constraint instead of colliding with the previous iteration's. Set by `step-execution-path.ts`.
- **ResumeReason** (`WAITPOINT`|`RETRY`) discriminates whether FAILED steps are restored on resume: waitpoint resumes preserve them, retry resumes drop them so the failed step re-executes.
- **Pause timeout is cumulative from `flowRun.created`, not per-pause.** `waitpointService.createForPause` (`clampWaitpointResumeDeadline`) rejects any resume date past `flowRun.created + AP_PAUSED_FLOW_TIMEOUT_DAYS`, throws `ErrorCode.PAUSED_FLOW_TIMEOUT_EXCEEDED`, and `waitpointClient` in the engine translates that to `PausedFlowTimeoutError` (USER → clean FAILED). This catches stacked delays (e.g. `1 + 10 + 10 + 25 + 25 = 71 days` on a 30-day timeout) that individually pass the engine-side per-pause `assertDelayWithinTimeout` but cumulatively outlive retention. Webhook waitpoints without an explicit `resumeDateTime` are defaulted to the same run deadline so they can't wait indefinitely past retention; when their timer fires the `RESUME_DELAY_WAITPOINT` handler branches on `PauseType.WEBHOOK` and marks the run FAILED (expiry, not resume). The same handler also marks FAILED if the timer fires past pause-timeout for a `DELAY` waitpoint — catches BullMQ dispatch lag on ancient queued jobs. `AP_EXECUTION_DATA_RETENTION_DAYS` must be strictly greater than `AP_PAUSED_FLOW_TIMEOUT_DAYS` (a 1+ day margin) or a valid resume firing on the retention boundary can still race the file-cleanup job.
diff --git a/bun.lock b/bun.lock
index 8ad96b2c8eb..21147384caa 100644
--- a/bun.lock
+++ b/bun.lock
@@ -120,7 +120,7 @@
},
"packages/core/execution": {
"name": "@activepieces/core-execution",
- "version": "0.21.0",
+ "version": "0.22.0",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -3797,7 +3797,7 @@
},
"packages/pieces/community/google-contacts": {
"name": "@activepieces/piece-google-contacts",
- "version": "0.5.0",
+ "version": "0.5.1",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -7316,7 +7316,7 @@
},
"packages/pieces/community/pushover": {
"name": "@activepieces/piece-pushover",
- "version": "0.2.8",
+ "version": "0.2.9",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
diff --git a/docs/install/architecture/waitpoints.mdx b/docs/install/architecture/waitpoints.mdx
index 7ed01c7bf4e..8fc46ba9dd1 100644
--- a/docs/install/architecture/waitpoints.mdx
+++ b/docs/install/architecture/waitpoints.mdx
@@ -105,6 +105,7 @@ Duplicate callbacks are absorbed by the uniqueness constraint; the engine never
| Method | Path | Notes |
| --- | --- | --- |
| `POST` | `/v1/waitpoints` | Engine-only. Creates a `PENDING` waitpoint, returns its resume URL. For `BARRIER`, also creates its signals. |
+| `ALL` | `/v1/flow-runs/:id/signals/:signalId/confirm` | Records one signal's outcome. `GET`/`HEAD` serve the page and never decide; only `POST` does. |
| `ALL` | `/v1/flow-runs/:id/waitpoints/:waitpointId` | Resume, async. |
| `ALL` | `/v1/flow-runs/:id/waitpoints/:waitpointId/sync` | Resume, sync. The HTTP response is whatever the flow produces after resuming. |
diff --git a/packages/core/execution/package.json b/packages/core/execution/package.json
index 7b1be0219e5..36b79910954 100644
--- a/packages/core/execution/package.json
+++ b/packages/core/execution/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-execution",
- "version": "0.21.0",
+ "version": "0.22.0",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/execution/src/lib/engine/execution-errors.ts b/packages/core/execution/src/lib/engine/execution-errors.ts
index 49680320d56..7f6a3d20d99 100644
--- a/packages/core/execution/src/lib/engine/execution-errors.ts
+++ b/packages/core/execution/src/lib/engine/execution-errors.ts
@@ -130,6 +130,12 @@ export class FormulaEvaluationError extends ExecutionError {
}
}
+export class WaitpointRejectedError extends ExecutionError {
+ constructor(message: string, cause?: unknown) {
+ super('WaitpointRejectedError', formatMessage(message), ExecutionErrorType.USER, cause)
+ }
+}
+
export class EngineGenericError extends ExecutionError {
constructor(name: string, message: string, cause?: unknown) {
super(name, formatMessage(message), ExecutionErrorType.ENGINE, cause)
diff --git a/packages/core/execution/src/lib/flow-run/waitpoint/index.ts b/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
index 7fc7a006104..c4a57199f6f 100644
--- a/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
+++ b/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
@@ -68,6 +68,7 @@ export const BarrierCreatedState = z.object({
label: z.string().nullable(),
confirmUrl: z.string(),
})).optional(),
+ signalsTruncated: z.boolean().optional(),
})
export type BarrierCreatedState = z.infer<typeof BarrierCreatedState>
@@ -103,6 +104,8 @@ export type BarrierSummary = z.infer<typeof BarrierSummary>
export const MAX_INLINE_BARRIER_SIGNALS = 100
+export const MAX_SIGNAL_REASON_LENGTH = 2000
+
export const UNFAVOURABLE_SIGNAL_STATUSES = [
BarrierSignalStatus.FAILED,
BarrierSignalStatus.REJECTED,
diff --git a/packages/server/api/src/app/app.ts b/packages/server/api/src/app/app.ts
index df82eb46e4c..eb72b56968d 100644
--- a/packages/server/api/src/app/app.ts
+++ b/packages/server/api/src/app/app.ts
@@ -121,6 +121,7 @@ import { platformUserModule } from './user/platform/platform-user-module'
import { invitationModule } from './user-invitations/user-invitation.module'
import { variableModule } from './variable/variable.module'
import { resumePageHooks } from './waitpoints/resume-page-hooks'
+import { sweepOverdueDeadlines } from './waitpoints/waitpoint-deadline-sweep'
import { webhookModule } from './webhooks/webhook-module'
import { engineResponseWatcher } from './workers/engine-response-watcher'
@@ -297,6 +298,21 @@ export const setupApp = async (app: FastifyInstance): Promise<FastifyInstance> =
},
})
+ systemJobHandlers.registerJobHandler(SystemJobName.WAITPOINT_DEADLINE_SWEEP, async () => {
+ await sweepOverdueDeadlines({ log: app.log })
+ })
+ await systemJobsSchedule(app.log).upsertJob({
+ job: {
+ name: SystemJobName.WAITPOINT_DEADLINE_SWEEP,
+ data: {},
+ jobId: SystemJobName.WAITPOINT_DEADLINE_SWEEP,
+ },
+ schedule: {
+ type: 'repeated',
+ cron: '* * * * *',
+ },
+ })
+
app.get(
'/redirect',
async (
diff --git a/packages/server/api/src/app/database/migration/postgres/1857000000000-AddWaitpointDeadLetteredAt.ts b/packages/server/api/src/app/database/migration/postgres/1857000000000-AddWaitpointDeadLetteredAt.ts
new file mode 100644
index 00000000000..fac25b061eb
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/1857000000000-AddWaitpointDeadLetteredAt.ts
@@ -0,0 +1,62 @@
+import { QueryRunner } from 'typeorm'
+import { system } from '../../../helper/system/system'
+import { AppSystemProp } from '../../../helper/system/system-props'
+import { DatabaseType } from '../../database-type'
+import { Migration } from '../../migration'
+
+export class AddWaitpointDeadLetteredAt1857000000000 implements Migration {
+ name = 'AddWaitpointDeadLetteredAt1857000000000'
+ breaking = false
+ release = '0.92.0'
+ transaction = false
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ const concurrently = isPGlite() ? '' : 'CONCURRENTLY'
+
+ await queryRunner.query(`
+ ALTER TABLE "waitpoint"
+ ADD COLUMN IF NOT EXISTS "deadLetteredAt" TIMESTAMP WITH TIME ZONE
+ `)
+
+ await queryRunner.query(`
+ CREATE INDEX ${concurrently} IF NOT EXISTS "idx_waitpoint_live_deadline"
+ ON "waitpoint" ("resumeDateTime", "id")
+ WHERE "status" = 'PENDING' AND "resumeDateTime" IS NOT NULL AND "deadLetteredAt" IS NULL
+ `)
+
+ await queryRunner.query(`
+ CREATE INDEX ${concurrently} IF NOT EXISTS "idx_waitpoint_undelivered_barrier"
+ ON "waitpoint" ("updated", "id")
+ WHERE "type" = 'BARRIER' AND "status" = 'COMPLETED'
+ `)
+
+ await queryRunner.query(`DROP INDEX ${concurrently} IF EXISTS "idx_waitpoint_pending_resume_date_time"`)
+
+ await queryRunner.query(`DROP INDEX ${concurrently} IF EXISTS "idx_waitpoint_signal_ref_id"`)
+ await queryRunner.query(`
+ CREATE INDEX ${concurrently} IF NOT EXISTS "idx_waitpoint_signal_ref_id"
+ ON "waitpoint_signal" ("refId", "projectId")
+ WHERE "refId" IS NOT NULL
+ `)
+ }
+
+ public async down(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query('DROP INDEX IF EXISTS "idx_waitpoint_signal_ref_id"')
+ await queryRunner.query(`
+ CREATE INDEX IF NOT EXISTS "idx_waitpoint_signal_ref_id"
+ ON "waitpoint_signal" ("refId", "projectId")
+ `)
+
+ await queryRunner.query(`
+ CREATE INDEX IF NOT EXISTS "idx_waitpoint_pending_resume_date_time"
+ ON "waitpoint" ("resumeDateTime")
+ WHERE "status" = 'PENDING' AND "resumeDateTime" IS NOT NULL
+ `)
+
+ await queryRunner.query('DROP INDEX IF EXISTS "idx_waitpoint_undelivered_barrier"')
+ await queryRunner.query('DROP INDEX IF EXISTS "idx_waitpoint_live_deadline"')
+ await queryRunner.query('ALTER TABLE "waitpoint" DROP COLUMN IF EXISTS "deadLetteredAt"')
+ }
+}
+
+const isPGlite = (): boolean => system.get(AppSystemProp.DB_TYPE) === DatabaseType.PGLITE
diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.ts
index 3c2826766b9..4ce9bc834ea 100644
--- a/packages/server/api/src/app/database/postgres-connection.ts
+++ b/packages/server/api/src/app/database/postgres-connection.ts
@@ -444,6 +444,7 @@ import { AddAgentConversationAiCredits1850000000000 } from './migration/postgres
import { AddMcpOAuthCodeNonce1851000000000 } from './migration/postgres/1851000000000-AddMcpOAuthCodeNonce'
import { FlowExternalIdUniqueIgnoresDeleting1855000000000 } from './migration/postgres/1855000000000-FlowExternalIdUniqueIgnoresDeleting'
import { AddFlowTombstoneIndex1856000000000 } from './migration/postgres/1856000000000-AddFlowTombstoneIndex'
+import { AddWaitpointDeadLetteredAt1857000000000 } from './migration/postgres/1857000000000-AddWaitpointDeadLetteredAt'
const getSslConfig = (): boolean | TlsOptions => {
const useSsl = system.get(AppSystemProp.POSTGRES_USE_SSL)
@@ -903,6 +904,7 @@ export const getMigrations = (): (new () => Migration)[] => {
AddMcpOAuthCodeNonce1851000000000,
FlowExternalIdUniqueIgnoresDeleting1855000000000,
AddFlowTombstoneIndex1856000000000,
+ AddWaitpointDeadLetteredAt1857000000000,
]
return migrations
}
diff --git a/packages/server/api/src/app/helper/system-jobs/common.ts b/packages/server/api/src/app/helper/system-jobs/common.ts
index 09989673705..c04514726f7 100644
--- a/packages/server/api/src/app/helper/system-jobs/common.ts
+++ b/packages/server/api/src/app/helper/system-jobs/common.ts
@@ -1,6 +1,6 @@
import { FlowId, FlowRunId, PlatformId, ProjectId } from '@activepieces/core-utils'
import { Flow } from '@activepieces/shared'
-import { Job, JobsOptions } from 'bullmq'
+import { Job, JobsOptions, JobState } from 'bullmq'
import { Dayjs } from 'dayjs'
export enum SystemJobName {
@@ -15,6 +15,7 @@ export enum SystemJobName {
RESUME_DELAY_WAITPOINT = 'resume-delay-waitpoint',
TOOL_SEARCH_REINDEX = 'tool-search-reindex',
CHAT_STALE_SWEEP = 'chat-stale-sweep',
+ WAITPOINT_DEADLINE_SWEEP = 'waitpoint-deadline-sweep',
}
type DeleteFlowDurableSystemJobData = {
@@ -56,6 +57,7 @@ type SystemJobDataMap = {
[SystemJobName.RESUME_DELAY_WAITPOINT]: ResumeDelayWaitpointSystemJobData
[SystemJobName.TOOL_SEARCH_REINDEX]: ToolSearchReindexSystemJobData
[SystemJobName.CHAT_STALE_SWEEP]: Record<string, never>
+ [SystemJobName.WAITPOINT_DEADLINE_SWEEP]: Record<string, never>
}
export type SystemJobData<T extends SystemJobName = SystemJobName> = T extends SystemJobName ? SystemJobDataMap[T] : never
@@ -89,7 +91,18 @@ type UpsertJobParams<T extends SystemJobName> = {
export type SystemJobSchedule = {
init(): Promise<void>
startWorker(): Promise<void>
- upsertJob<T extends SystemJobName>(params: UpsertJobParams<T>): Promise<void>
+ upsertJob<T extends SystemJobName>(params: UpsertJobParams<T>): Promise<UpsertJobResult>
getJob<T extends SystemJobName>(jobId: string): Promise<Job<SystemJobData<T>> | undefined>
+ getJobState(jobId: string): Promise<JobState | 'unknown'>
+ removeJob(params: { jobId: string }): Promise<void>
close(): Promise<void>
}
+
+export const systemJobIds = {
+ resumeDelay: ({ waitpointId }: { waitpointId: string }): string => `resume-delay-${waitpointId}`,
+ legacyResumeDelay: ({ flowRunId }: { flowRunId: FlowRunId }): string => `resume-delay-${flowRunId}`,
+}
+
+export type UpsertJobResult = {
+ status: 'added' | 'kept' | 'retried' | 'scheduler-upserted'
+}
diff --git a/packages/server/api/src/app/helper/system-jobs/system-job.ts b/packages/server/api/src/app/helper/system-jobs/system-job.ts
index 7635bc426f9..5de92693b7c 100644
--- a/packages/server/api/src/app/helper/system-jobs/system-job.ts
+++ b/packages/server/api/src/app/helper/system-jobs/system-job.ts
@@ -1,12 +1,12 @@
import { isNil, tryCatch } from '@activepieces/core-utils'
-import { apDayjs, apDayjsDuration } from '@activepieces/server-utils'
-import { Job, JobsOptions, Queue, Worker } from 'bullmq'
+import { apDayjs, apDayjsDuration, createLogger, wideEvent } from '@activepieces/server-utils'
+import { Job, JobsOptions, JobState, Queue, Worker } from 'bullmq'
import { Dayjs } from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
import { redisConnections } from '../../database/redis-connections'
import { jobFailureLogger } from '../../workers/job-queue/job-failure-logger'
import { exceptionHandler } from '../exception-handler'
-import { SystemJobData, SystemJobName, SystemJobSchedule } from './common'
+import { SystemJobData, SystemJobName, SystemJobSchedule, UpsertJobResult } from './common'
import { systemJobHandlers } from './job-handlers'
const FIFTEEN_MINUTES = apDayjsDuration(15, 'minute').asMilliseconds()
@@ -47,9 +47,21 @@ export const systemJobsSchedule = (log: FastifyBaseLogger): SystemJobSchedule =>
SYSTEM_JOB_QUEUE,
async (job) => {
log.debug({ jobName: job.name }, '[systemJob#worker] Executing job')
-
- const jobHandler = systemJobHandlers.getJobHandler(job.name)
- await jobHandler(job.data)
+ const jobLogger = createLogger({
+ event: 'system-job.execute',
+ job: { id: job.id, type: job.name },
+ })
+ return wideEvent.run({
+ logger: jobLogger,
+ fn: () => processSystemJob(job)
+ .then(() => wideEvent.set({ outcome: 'success' }))
+ .catch((error: unknown) => {
+ wideEvent.error(error)
+ wideEvent.set({ outcome: 'failed' })
+ throw error
+ })
+ .finally(() => jobLogger.emit()),
+ })
},
{
connection: await redisConnections.create(),
@@ -75,7 +87,7 @@ export const systemJobsSchedule = (log: FastifyBaseLogger): SystemJobSchedule =>
await systemJobWorker.waitUntilReady()
},
- async upsertJob({ job, schedule, customConfig }): Promise<void> {
+ async upsertJob({ job, schedule, customConfig }): Promise<UpsertJobResult> {
log.info({ jobName: job.name }, '[systemJob#upsertJob] Upserting job')
if (schedule.type === 'repeated') {
const schedulers = await systemJobsQueue.getJobSchedulers()
@@ -85,23 +97,45 @@ export const systemJobsSchedule = (log: FastifyBaseLogger): SystemJobSchedule =>
await systemJobsQueue.removeJobScheduler(scheduler.id ?? scheduler.key)
}
await systemJobsQueue.upsertJobScheduler(job.name, { pattern: schedule.cron, tz: 'UTC' }, { name: job.name, data: job.data, opts: customConfig })
- return
+ return { status: 'scheduler-upserted' }
}
const existingJob = await getJobByNameAndJobId(job.name, job.jobId)
if (!isNil(existingJob) && await existingJob.isFailed()) {
log.info({ jobName: job.name }, '[systemJob#upsertJob] Retrying failed job')
await existingJob.retry()
+ return { status: 'retried' }
}
if (isNil(existingJob)) {
log.info({ jobName: job.name }, '[systemJob#upsertJob] Adding job to queue')
await systemJobsQueue.add(job.name, job.data, configureJobOptions({ date: schedule.date, jobId: job.jobId, customConfig }))
+ return { status: 'added' }
}
+ log.warn({ job: { id: job.jobId, type: job.name }, existingDelayMs: existingJob.opts.delay },
+ '[systemJob#upsertJob] A one-time job already exists under this id; kept its schedule')
+ return { status: 'kept' }
},
async getJob<T extends SystemJobName>(jobId: string) {
return await systemJobsQueue.getJob(jobId) as Job<SystemJobData<T>> | undefined
},
+ async getJobState(jobId: string): Promise<JobState | 'unknown'> {
+ return systemJobsQueue.getJobState(jobId)
+ },
+
+ async removeJob({ jobId }): Promise<void> {
+ const { data: removed, error } = await tryCatch(async () => systemJobsQueue.remove(jobId))
+ if (!isNil(error)) {
+ log.warn({ job: { id: jobId }, error }, '[systemJob#removeJob] Could not remove job')
+ return
+ }
+ if (removed === 0) {
+ log.info({ job: { id: jobId } }, '[systemJob#removeJob] Job is running, left to finish')
+ return
+ }
+ log.debug({ job: { id: jobId } }, '[systemJob#removeJob] Job removed or already gone')
+ },
+
async close(): Promise<void> {
if (isNil(systemJobsQueue)) {
return
@@ -114,6 +148,11 @@ export const systemJobsSchedule = (log: FastifyBaseLogger): SystemJobSchedule =>
},
})
+async function processSystemJob(job: Job<SystemJobData, unknown, SystemJobName>): Promise<void> {
+ const jobHandler = systemJobHandlers.getJobHandler(job.name)
+ await jobHandler(job.data)
+}
+
async function removeDeprecatedJobs(log: FastifyBaseLogger): Promise<void> {
const deprecatedJobs = [
'trigger-data-cleaner',
diff --git a/packages/server/api/src/app/waitpoints/barrier-service.ts b/packages/server/api/src/app/waitpoints/barrier-service.ts
index 7431fc6406e..205b920af08 100644
--- a/packages/server/api/src/app/waitpoints/barrier-service.ts
+++ b/packages/server/api/src/app/waitpoints/barrier-service.ts
@@ -1,11 +1,13 @@
-import { apId, chunk, isNil, sanitizeObjectForPostgresql, tryCatch } from '@activepieces/core-utils'
+import { apId, chunk, isNil, sanitizeObjectForPostgresql, tryCatch, unique } from '@activepieces/core-utils'
import { wideEvent } from '@activepieces/server-utils'
import { ActivepiecesError, BarrierPolicy, barrierReleasesOnLastPendingSignal, BarrierSignalCounts, BarrierSignalStatus, BarrierSummary, ErrorCode, MAX_INLINE_BARRIER_SIGNALS, PauseType, RespondResponse, shouldReleaseBarrier, WaitpointVersion } from '@activepieces/shared'
import dayjs from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
import { EntityManager } from 'typeorm'
+import { z } from 'zod'
import { repoFactory } from '../core/db/repo-factory'
import { transaction } from '../core/db/transaction'
+import { flowRunRepo } from '../flows/flow-run/flow-run-service'
import { system } from '../helper/system/system'
import { AppSystemProp } from '../helper/system/system-props'
import { platformConfigurationService } from '../platform/platform-configuration.service'
@@ -13,6 +15,7 @@ import { barrierQueue } from './barrier-queue'
import { resumeService } from './resume-service'
import { WaitpointEntity } from './waitpoint-entity'
import { WaitpointSignalEntity } from './waitpoint-signal-entity'
+import { waitpointTimeoutJob } from './waitpoint-timeout-job'
import { Waitpoint, WaitpointSignal, WaitpointStatus } from './waitpoint-types'
const waitpointRepo = repoFactory(WaitpointEntity)
@@ -22,6 +25,7 @@ export const barrierService = (log: FastifyBaseLogger) => ({
async create(params: CreateBarrierParams): Promise<CreateBarrierResult> {
const labels = params.signalLabels ?? []
await assertSignalCountWithinLimit({ signalCount: labels.length, platformId: params.platformId, log })
+ const flowRun = await flowRunRepo().findOneByOrFail({ id: params.flowRunId, projectId: params.projectId })
const creation = await transaction(async (entityManager) => {
const repo = waitpointRepo(entityManager)
@@ -44,13 +48,14 @@ export const barrierService = (log: FastifyBaseLogger) => ({
type: PauseType.BARRIER,
version: params.version,
status: WaitpointStatus.PENDING,
- resumeDateTime: defaultBarrierDeadline(),
+ resumeDateTime: defaultBarrierDeadline({ flowRunCreated: flowRun.created }),
responseToSend: params.responseToSend ?? null,
workerHandlerId: params.workerHandlerId ?? null,
httpRequestId: params.httpRequestId ?? null,
resumePayload: null,
sealed: true,
policy: params.policy ?? null,
+ deadLetteredAt: null,
}
await repo.createQueryBuilder().insert().into('waitpoint').values(barrier).execute()
const signals = buildPendingSignals({ barrierId: barrier.id, projectId: params.projectId, labels })
@@ -61,6 +66,15 @@ export const barrierService = (log: FastifyBaseLogger) => ({
})
if (creation.inserted) {
+ if (!isNil(creation.barrier.resumeDateTime)) {
+ await waitpointTimeoutJob.schedule({
+ flowRunId: params.flowRunId,
+ projectId: params.projectId,
+ waitpointId: creation.barrier.id,
+ resumeDateTime: creation.barrier.resumeDateTime,
+ log,
+ })
+ }
await barrierQueue(log).enqueueEvaluation({ barrierId: creation.barrier.id, projectId: params.projectId })
}
@@ -75,23 +89,39 @@ export const barrierService = (log: FastifyBaseLogger) => ({
return signalRepo().findOneBy({ id: signalId, projectId })
},
- async receiveSignal(params: ReceiveSignalParams): Promise<WaitpointSignal | null> {
- if (isNil(params.signalId) && isNil(params.refId)) {
- return null
- }
- const signal = isNil(params.signalId)
- ? await signalRepo().findOneBy({ refId: params.refId, projectId: params.projectId })
- : await signalRepo().findOneBy({ id: params.signalId, projectId: params.projectId })
- if (isNil(signal)) {
- return null
+ async receiveSignal({ signalId, refId, projectId, status, result }: ReceiveSignalParams): Promise<boolean> {
+ if (isNil(signalId) && isNil(refId)) {
+ return false
}
- await signalRepo().save({
- ...signal,
- status: params.status,
- result: isNil(params.result) ? null : sanitizeObjectForPostgresql(params.result),
+ const identity: SignalIdentity = isNil(signalId) ? { refId } : { id: signalId }
+ const barrierIds = await updateSignalOutcome({ where: { ...identity, projectId }, status, result })
+ await Promise.all(barrierIds.map((barrierId) => barrierQueue(log).enqueueEvaluation({ barrierId, projectId })))
+ return barrierIds.length > 0
+ },
+
+ async recordDecision({ barrierId, signalId, projectId, status, result }: RecordDecisionParams): Promise<boolean> {
+ const recorded = await transaction(async (entityManager) => {
+ const openBarrier = await waitpointRepo(entityManager)
+ .createQueryBuilder('waitpoint')
+ .setLock('pessimistic_read')
+ .where({ id: barrierId, projectId, type: PauseType.BARRIER, status: WaitpointStatus.PENDING })
+ .getOne()
+ if (isNil(openBarrier)) {
+ return false
+ }
+ const decidedBarrierIds = await updateSignalOutcome({
+ where: { id: signalId, waitpointId: barrierId, projectId, status: BarrierSignalStatus.PENDING },
+ status,
+ result,
+ entityManager,
+ })
+ return decidedBarrierIds.length > 0
})
- await barrierQueue(log).enqueueEvaluation({ barrierId: signal.waitpointId, projectId: signal.projectId })
- return signal
+ if (!recorded) {
+ return false
+ }
+ await barrierQueue(log).enqueueEvaluation({ barrierId, projectId })
+ return true
},
async releaseIfReady({ barrierId, projectId }: ReleaseIfReadyParams): Promise<void> {
@@ -135,6 +165,21 @@ export const barrierService = (log: FastifyBaseLogger) => ({
},
})
+async function updateSignalOutcome({ where, status, result, entityManager }: UpdateSignalOutcomeParams): Promise<string[]> {
+ const updateResult = await signalRepo(entityManager)
+ .createQueryBuilder()
+ .update()
+ .set({
+ status,
+ result: () => 'CAST(:signalResult AS jsonb)',
+ })
+ .setParameter('signalResult', isNil(result) ? null : JSON.stringify(sanitizeObjectForPostgresql(result)))
+ .where(where)
+ .returning(['waitpointId'])
+ .execute()
+ return unique(UpdatedSignalRows.parse(updateResult.raw ?? []).map((row) => row.waitpointId))
+}
+
function buildPendingSignals({ barrierId, projectId, labels }: BuildPendingSignalsParams): WaitpointSignal[] {
const now = new Date().toISOString()
return labels.map((label) => ({
@@ -247,13 +292,15 @@ async function assertSignalCountWithinLimit({ signalCount, platformId, log }: As
}
}
-function defaultBarrierDeadline(): string {
+function defaultBarrierDeadline({ flowRunCreated }: DefaultBarrierDeadlineParams): string {
const maxDurationInDays = system.getNumberOrThrow(AppSystemProp.PAUSED_FLOW_TIMEOUT_DAYS)
- return dayjs().add(maxDurationInDays, 'day').toISOString()
+ return dayjs(flowRunCreated).add(maxDurationInDays, 'day').toISOString()
}
const SIGNAL_INSERT_BATCH_SIZE = 500
+const UpdatedSignalRows = z.array(z.object({ waitpointId: z.string() }))
+
export type BarrierReleaseReason = 'predicate' | 'timeout'
export type CreateBarrierParams = {
@@ -283,6 +330,30 @@ export type ReceiveSignalParams = {
result?: Record<string, unknown>
}
+export type RecordDecisionParams = {
+ barrierId: string
+ signalId: string
+ projectId: string
+ status: BarrierSignalStatus
+ result?: Record<string, unknown>
+}
+
+type SignalIdentity = {
+ id?: string
+ refId?: string
+}
+
+type UpdateSignalOutcomeParams = {
+ where: SignalIdentity & {
+ projectId: string
+ waitpointId?: string
+ status?: BarrierSignalStatus
+ }
+ status: BarrierSignalStatus
+ result?: Record<string, unknown>
+ entityManager?: EntityManager
+}
+
type FindSignalByIdParams = {
signalId: string
projectId: string
@@ -340,3 +411,7 @@ type AssertSignalCountWithinLimitParams = {
platformId: string
log: FastifyBaseLogger
}
+
+type DefaultBarrierDeadlineParams = {
+ flowRunCreated: string
+}
diff --git a/packages/server/api/src/app/waitpoints/resume-controller.ts b/packages/server/api/src/app/waitpoints/resume-controller.ts
index f221de63cbd..9aaf3246ef0 100644
--- a/packages/server/api/src/app/waitpoints/resume-controller.ts
+++ b/packages/server/api/src/app/waitpoints/resume-controller.ts
@@ -1,17 +1,19 @@
import { ApId, isNil } from '@activepieces/core-utils'
-import { ALL_PRINCIPAL_TYPES, FlowRunStatus } from '@activepieces/shared'
+import { ALL_PRINCIPAL_TYPES, BarrierSignalStatus, FlowRun, FlowRunStatus, MAX_SIGNAL_REASON_LENGTH, PauseType } from '@activepieces/shared'
import { FastifyBaseLogger, FastifyReply } from 'fastify'
import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'
+import { StatusCodes } from 'http-status-codes'
import Mustache from 'mustache'
import { z } from 'zod'
import { securityAccess } from '../core/security/authorization/fastify-security'
import { findFlowRunOrThrow } from '../flows/flow-run/flow-run-service'
import { flowVersionService } from '../flows/flow-version/flow-version.service'
import { projectService } from '../project/project-service'
+import { barrierService } from './barrier-service'
import { resumePageHooks, ResumePageTheme } from './resume-page-hooks'
import { resumeService } from './resume-service'
import { waitpointService } from './waitpoint-service'
-import { WaitpointStatus } from './waitpoint-types'
+import { Waitpoint, WaitpointSignal, WaitpointStatus } from './waitpoint-types'
export const resumeController: FastifyPluginAsyncZod = async (app) => {
/**
@@ -46,6 +48,15 @@ export const resumeController: FastifyPluginAsyncZod = async (app) => {
await handleConfirmResume({ flowRunId: req.params.id, waitpointId: req.params.waitpointId, action: queryParams.action, body: req.body, headers, queryParams, log: req.log, reply })
})
+ app.all('/:id/signals/:signalId/confirm', ConfirmSignalRequest, async (req, reply) => {
+ const queryParams = req.query as Record<string, string>
+ if (req.method !== 'POST') {
+ await serveSignalConfirmationPage({ flowRunId: req.params.id, signalId: req.params.signalId, url: req.url, queryParams, log: req.log, reply })
+ return
+ }
+ await handleSignalDecision({ flowRunId: req.params.id, signalId: req.params.signalId, action: queryParams.action, body: req.body, headers: req.headers as Record<string, string>, log: req.log, reply })
+ })
+
/**
* @deprecated Deprecated since 2026-04-13. can be only removed after all paused jobs after deployment of this version to sink.
* Handles resume for V0 waitpoints created by legacy pieces using run.pause() + generateResumeUrl().
@@ -83,21 +94,35 @@ export const resumeController: FastifyPluginAsyncZod = async (app) => {
async function serveConfirmationPage({ flowRunId, waitpointId, url, queryParams, log, reply }: ConfirmationPageParams): Promise<void> {
const flowRun = await findFlowRunOrThrow(flowRunId)
- const theme = await resolveResumePageTheme({ projectId: flowRun.projectId, log })
const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId, projectId: flowRun.projectId })
- const isOpen = !isNil(waitpoint) && waitpoint.status === WaitpointStatus.PENDING && flowRun.status === FlowRunStatus.PAUSED
+ const isOpen = !isNil(waitpoint) && waitpoint.type !== PauseType.BARRIER && waitpoint.status === WaitpointStatus.PENDING && isResumableRunStatus(flowRun.status)
if (!isOpen) {
- await replyWithHtml({ reply, html: Mustache.render(STATUS_HTML_TEMPLATE, buildThemeView({ theme, extra: { title: ALREADY_TITLE, message: ALREADY_MESSAGE, success: false } })) })
+ await replyWithAlreadyResponded({ projectId: flowRun.projectId, log, reply })
return
}
+ await replyWithConfirmationPage({ flowRun, url, queryParams, log, reply })
+}
+
+async function replyWithConfirmationPage({ flowRun, url, queryParams, log, reply, reasonField }: ReplyWithConfirmationPageParams): Promise<void> {
+ const theme = await resolveResumePageTheme({ projectId: flowRun.projectId, log })
const flowName = await resolveFlowName({ flowVersionId: flowRun.flowVersionId, log })
const path = url.split('?')[0]
const extra = {
+ title: CONFIRM_TITLE,
+ message: CONFIRM_MESSAGE,
flowName,
- approveUrl: buildActionUrl({ path, queryParams, action: 'approve' }),
- disapproveUrl: buildActionUrl({ path, queryParams, action: 'disapprove' }),
+ actions: {
+ ...reasonField,
+ approveUrl: buildActionUrl({ path, queryParams, action: 'approve' }),
+ disapproveUrl: buildActionUrl({ path, queryParams, action: 'disapprove' }),
+ },
}
- await replyWithHtml({ reply, html: Mustache.render(CONFIRM_HTML_TEMPLATE, buildThemeView({ theme, extra })) })
+ await replyWithHtml({ reply, html: renderPage({ theme, extra }) })
+}
+
+async function replyWithAlreadyResponded({ projectId, log, reply }: ReplyWithAlreadyRespondedParams): Promise<void> {
+ const theme = await resolveResumePageTheme({ projectId, log })
+ await replyWithHtml({ reply, html: renderPage({ theme, extra: ALREADY_RESPONDED }) })
}
function buildActionUrl({ path, queryParams, action }: { path: string, queryParams: Record<string, string>, action: string }): string {
@@ -118,9 +143,107 @@ async function handleConfirmResume({ flowRunId, waitpointId, action, body, heade
}
const theme = await resolveResumePageTheme({ projectId: flowRun.projectId, log })
const extra = stale
- ? { title: ALREADY_TITLE, message: ALREADY_MESSAGE, success: false }
+ ? ALREADY_RESPONDED
: { title: RECORDED_TITLE, message: recordedMessageForAction(action), success: true }
- await replyWithHtml({ reply, html: Mustache.render(STATUS_HTML_TEMPLATE, buildThemeView({ theme, extra })) })
+ await replyWithHtml({ reply, html: renderPage({ theme, extra }) })
+}
+
+async function serveSignalConfirmationPage({ flowRunId, signalId, url, queryParams, log, reply }: SignalConfirmationPageParams): Promise<void> {
+ const flowRun = await findFlowRunOrThrow(flowRunId)
+ const open = await resolveOpenSignal({ flowRunId, signalId, projectId: flowRun.projectId, flowRunStatus: flowRun.status, log })
+ if (isNil(open)) {
+ await replyWithAlreadyResponded({ projectId: flowRun.projectId, log, reply })
+ return
+ }
+ await replyWithConfirmationPage({ flowRun, url, queryParams, log, reply, reasonField: { maxReasonLength: MAX_SIGNAL_REASON_LENGTH } })
+}
+
+async function handleSignalDecision({ flowRunId, signalId, action, body, headers, log, reply }: SignalDecisionParams): Promise<void> {
+ const flowRun = await findFlowRunOrThrow(flowRunId)
+ const projectId = flowRun.projectId
+ if (action !== 'approve' && action !== 'disapprove') {
+ await respondToSignalDecision({ reply, headers, projectId, log, status: StatusCodes.BAD_REQUEST, extra: { title: UNKNOWN_ACTION_TITLE, message: UNKNOWN_ACTION_MESSAGE, success: false } })
+ return
+ }
+ const open = await resolveOpenSignal({ flowRunId, signalId, projectId, flowRunStatus: flowRun.status, log })
+ if (isNil(open)) {
+ await respondToSignalDecision({ reply, headers, projectId, log, status: StatusCodes.OK, extra: ALREADY_RESPONDED })
+ return
+ }
+ const approved = action === 'approve'
+ const reason = readReason(body)
+ if (!isNil(reason) && reason.length > MAX_SIGNAL_REASON_LENGTH) {
+ await respondToSignalDecision({ reply, headers, projectId, log, status: StatusCodes.BAD_REQUEST, extra: { title: REASON_TOO_LONG_TITLE, message: reasonTooLongMessage(), success: false } })
+ return
+ }
+ if (isReasonMissing({ reasonRequiredOn: open.barrier.policy?.reasonRequiredOn, approved, reason })) {
+ await respondToSignalDecision({ reply, headers, projectId, log, status: StatusCodes.BAD_REQUEST, extra: { title: REASON_REQUIRED_TITLE, message: REASON_REQUIRED_MESSAGE, success: false } })
+ return
+ }
+
+ const recorded = await barrierService(log).recordDecision({
+ barrierId: open.barrier.id,
+ signalId,
+ projectId,
+ status: approved ? BarrierSignalStatus.SUCCEEDED : BarrierSignalStatus.REJECTED,
+ result: { outcome: approved ? 'approved' : 'rejected', reason: reason ?? null, decidedBy: open.signal.label },
+ })
+ if (!recorded) {
+ await respondToSignalDecision({ reply, headers, projectId, log, status: StatusCodes.OK, extra: ALREADY_RESPONDED })
+ return
+ }
+ await respondToSignalDecision({ reply, headers, projectId, log, status: StatusCodes.OK, extra: { title: RECORDED_TITLE, message: recordedMessageForAction(approved ? 'approve' : 'disapprove'), success: true } })
+}
+
+async function resolveOpenSignal({ flowRunId, signalId, projectId, flowRunStatus, log }: ResolveOpenSignalParams): Promise<OpenSignal | null> {
+ if (!isResumableRunStatus(flowRunStatus)) {
+ return null
+ }
+ const signal = await barrierService(log).findSignalById({ signalId, projectId })
+ if (isNil(signal) || signal.status !== BarrierSignalStatus.PENDING) {
+ return null
+ }
+ const barrier = await waitpointService(log).findByIdAndFlowRunId({ waitpointId: signal.waitpointId, flowRunId, projectId })
+ if (isNil(barrier) || barrier.type !== PauseType.BARRIER || barrier.status !== WaitpointStatus.PENDING) {
+ return null
+ }
+ return { signal, barrier }
+}
+
+function isResumableRunStatus(status: FlowRunStatus): boolean {
+ return RESUMABLE_RUN_STATUSES.includes(status)
+}
+
+function isReasonMissing({ reasonRequiredOn, approved, reason }: IsReasonMissingParams): boolean {
+ const required = reasonRequiredOn ?? 'none'
+ if (required === 'none') {
+ return false
+ }
+ if (required === 'reject' && approved) {
+ return false
+ }
+ return isNil(reason) || reason.trim().length === 0
+}
+
+function readReason(body: unknown): string | undefined {
+ const parsed = SignalDecisionBody.safeParse(body)
+ if (!parsed.success || isNil(parsed.data.reason)) {
+ return undefined
+ }
+ return parsed.data.reason.replace(/\r\n/g, '\n')
+}
+
+function reasonTooLongMessage(): string {
+ return `Your reason is longer than the ${MAX_SIGNAL_REASON_LENGTH} characters this form accepts. Shorten it and submit again — nothing was recorded.`
+}
+
+async function respondToSignalDecision({ reply, headers, projectId, log, status, extra }: RespondToSignalDecisionParams): Promise<void> {
+ if (!acceptsHtml(headers)) {
+ await reply.status(status).send({ message: extra.message })
+ return
+ }
+ const theme = await resolveResumePageTheme({ projectId, log })
+ await replyWithHtml({ reply, status, html: renderPage({ theme, extra }) })
}
async function handleAsyncResume({ flowRunId, waitpointId, body, headers, queryParams, log, reply }: AsyncResumeHandlerParams): Promise<void> {
@@ -191,21 +314,22 @@ function recordedMessageForAction(action: string | undefined): string {
return RECORDED_MESSAGE
}
-function buildThemeView({ theme, extra }: { theme: ResumePageTheme, extra: Record<string, unknown> }): Record<string, unknown> {
- return {
+function renderPage({ theme, extra }: { theme: ResumePageTheme, extra: Record<string, unknown> }): string {
+ return Mustache.render(PAGE_HTML_TEMPLATE, {
websiteName: theme.websiteName,
fullLogoUrl: theme.logos.fullLogoUrl,
primaryColor: theme.colors.primary.default,
...extra,
- }
+ })
}
function acceptsHtml(headers: Record<string, string>): boolean {
return (headers['accept'] ?? '').includes('text/html')
}
-async function replyWithHtml({ reply, html }: { reply: FastifyReply, html: string }): Promise<void> {
+async function replyWithHtml({ reply, html, status }: { reply: FastifyReply, html: string, status?: number }): Promise<void> {
await reply
+ .status(status ?? StatusCodes.OK)
.type('text/html')
.header('Content-Security-Policy', RESUME_PAGE_CSP)
.header('X-Content-Type-Options', 'nosniff')
@@ -225,6 +349,22 @@ const ResumeByWaitpointRequest = {
},
}
+const ConfirmSignalRequest = {
+ config: {
+ security: securityAccess.unscoped(ALL_PRINCIPAL_TYPES),
+ },
+ schema: {
+ params: z.object({
+ id: ApId,
+ signalId: ApId,
+ }),
+ },
+}
+
+const SignalDecisionBody = z.object({
+ reason: z.string().optional(),
+})
+
const V0ResumeFlowRunRequest = {
config: {
security: securityAccess.unscoped(ALL_PRINCIPAL_TYPES),
@@ -237,12 +377,26 @@ const V0ResumeFlowRunRequest = {
},
}
+const RESUMABLE_RUN_STATUSES = [FlowRunStatus.PAUSED, FlowRunStatus.RUNNING, FlowRunStatus.QUEUED]
+
+const CONFIRM_TITLE = 'Confirm your response'
+const CONFIRM_MESSAGE = 'A flow is paused and waiting for your response. Please confirm to continue.'
+
+const REASON_REQUIRED_TITLE = 'A reason is required'
+const REASON_REQUIRED_MESSAGE = 'This request needs a reason before it can be recorded. Go back, write one, and submit again.'
+const REASON_TOO_LONG_TITLE = 'Your reason is too long'
+
+const UNKNOWN_ACTION_TITLE = 'Nothing was recorded'
+const UNKNOWN_ACTION_MESSAGE = 'This link needs to say whether you approve or disapprove. Open it in a browser and use one of the buttons — nothing was recorded.'
+
const RECORDED_MESSAGE = 'Your response has been recorded. You can close this page now.'
const EXPIRED_MESSAGE = 'This link has expired. The action may have already been processed.'
const RECORDED_TITLE = 'Response recorded'
const ALREADY_TITLE = 'Already responded'
const ALREADY_MESSAGE = 'This request has already been responded to. There is nothing left to do here.'
+const ALREADY_RESPONDED = { title: ALREADY_TITLE, message: ALREADY_MESSAGE, success: false }
+
const RESUME_PAGE_CSP = 'default-src \'none\'; style-src \'unsafe-inline\'; form-action \'self\'; img-src \'self\' https: http: data:'
const RESUME_PAGE_STYLE = `
@@ -268,6 +422,10 @@ const RESUME_PAGE_STYLE = `
.btn { flex: 1; height: 36px; border: none; border-radius: 8px; font-size: 14px; font-weight: 500; color: #fff; cursor: pointer; display: inline-flex; align-items: center; justify-content: center; gap: 6px; }
.btn svg { width: 14px; height: 14px; }
.btn-destructive { background: #ef4444; }
+ .reason { padding: 20px 32px 0; display: flex; flex-direction: column; gap: 6px; }
+ .reason label { font-size: 12.5px; font-weight: 500; color: #404040; }
+ .reason textarea { width: 100%; min-height: 76px; resize: vertical; padding: 10px 12px; border: 1px solid #e5e5e5; border-radius: 8px;
+ font-family: inherit; font-size: 13px; line-height: 1.5; color: #0a0a0a; background: #fff; }
`
const CLOCK_SVG = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><polyline points="12 6 12 12 16 14"/></svg>'
@@ -276,7 +434,7 @@ const ACTIVITY_SVG = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor"
const X_SVG = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>'
const CHECK_SVG = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="20 6 9 17 4 12"/></svg>'
-const CONFIRM_HTML_TEMPLATE = `<!DOCTYPE html>
+const PAGE_HTML_TEMPLATE = `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
@@ -288,41 +446,36 @@ const CONFIRM_HTML_TEMPLATE = `<!DOCTYPE html>
<div class="card">
<div class="head">
{{#fullLogoUrl}}<img class="logo" src="{{fullLogoUrl}}" alt="{{websiteName}}">{{/fullLogoUrl}}
-<div class="badge badge-amber">${CLOCK_SVG}</div>
-<h1>Confirm your response</h1>
-<p>A flow is paused and waiting for your response. Please confirm to continue.</p>
+{{#success}}<div class="badge badge-green">${CHECK_CIRCLE_SVG}</div>{{/success}}
+{{^success}}<div class="badge badge-amber">${CLOCK_SVG}</div>{{/success}}
+<h1>{{title}}</h1>
+<p>{{message}}</p>
</div>
{{#flowName}}
<div class="chip">${ACTIVITY_SVG}<span class="name">{{flowName}}</span><span class="state">Paused</span></div>
{{/flowName}}
-<form method="POST" class="actions">
+{{#actions}}
+<form method="POST">
+{{#maxReasonLength}}
+<div class="reason">
+<label for="reason">Reason</label>
+<textarea id="reason" name="reason" maxlength="{{maxReasonLength}}" placeholder="Why?"></textarea>
+</div>
+{{/maxReasonLength}}
+<div class="actions">
<button type="submit" class="btn btn-destructive" formaction="{{disapproveUrl}}">${X_SVG}Disapprove</button>
<button type="submit" class="btn" style="background:{{primaryColor}}" formaction="{{approveUrl}}">${CHECK_SVG}Approve</button>
+</div>
</form>
+{{/actions}}
</div>
</body>
</html>`
-const STATUS_HTML_TEMPLATE = `<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="utf-8">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<title>{{websiteName}}</title>
-<style>${RESUME_PAGE_STYLE}</style>
-</head>
-<body>
-<div class="card">
-<div class="head">
-{{#fullLogoUrl}}<img class="logo" src="{{fullLogoUrl}}" alt="{{websiteName}}">{{/fullLogoUrl}}
-{{#success}}<div class="badge badge-green">${CHECK_CIRCLE_SVG}</div>{{/success}}
-{{^success}}<div class="badge badge-amber">${CLOCK_SVG}</div>{{/success}}
-<h1>{{title}}</h1>
-<p>{{message}}</p>
-</div>
-</div>
-</body>
-</html>`
+type OpenSignal = {
+ signal: WaitpointSignal
+ barrier: Waitpoint
+}
type ResponseWasDiscardedParams = {
flowRunId: string
@@ -340,13 +493,17 @@ type ConfirmationPageParams = {
reply: FastifyReply
}
-type ConfirmResumeParams = {
- flowRunId: string
- waitpointId: string
- action: string | undefined
- body: unknown
- headers: Record<string, string>
- queryParams: Record<string, string>
+type SignalConfirmationPageParams = Omit<ConfirmationPageParams, 'waitpointId'> & {
+ signalId: string
+}
+
+type ReplyWithConfirmationPageParams = Omit<ConfirmationPageParams, 'flowRunId' | 'waitpointId'> & {
+ flowRun: FlowRun
+ reasonField?: { maxReasonLength: number }
+}
+
+type ReplyWithAlreadyRespondedParams = {
+ projectId: string
log: FastifyBaseLogger
reply: FastifyReply
}
@@ -361,11 +518,35 @@ type AsyncResumeHandlerParams = {
reply: FastifyReply
}
-type LegacyResumeHandlerParams = {
+type ConfirmResumeParams = AsyncResumeHandlerParams & {
+ action: string | undefined
+}
+
+type LegacyResumeHandlerParams = Omit<AsyncResumeHandlerParams, 'waitpointId'>
+
+type SignalDecisionParams = Omit<ConfirmResumeParams, 'waitpointId' | 'queryParams'> & {
+ signalId: string
+}
+
+type ResolveOpenSignalParams = {
flowRunId: string
- body: unknown
- headers: Record<string, string>
- queryParams: Record<string, string>
+ signalId: string
+ projectId: string
+ flowRunStatus: FlowRunStatus
log: FastifyBaseLogger
+}
+
+type IsReasonMissingParams = {
+ reasonRequiredOn: 'none' | 'reject' | 'both' | undefined
+ approved: boolean
+ reason: string | undefined
+}
+
+type RespondToSignalDecisionParams = {
reply: FastifyReply
+ headers: Record<string, string>
+ projectId: string
+ log: FastifyBaseLogger
+ status: number
+ extra: { title: string, message: string, success: boolean }
}
diff --git a/packages/server/api/src/app/waitpoints/resume-delay-handler.ts b/packages/server/api/src/app/waitpoints/resume-delay-handler.ts
index b04f0f3d436..612ba9f8b82 100644
--- a/packages/server/api/src/app/waitpoints/resume-delay-handler.ts
+++ b/packages/server/api/src/app/waitpoints/resume-delay-handler.ts
@@ -1,4 +1,5 @@
import { isNil } from '@activepieces/core-utils'
+import { wideEvent } from '@activepieces/server-utils'
import { FlowRunStatus, flowStructureUtil, PauseType } from '@activepieces/shared'
import dayjs from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
@@ -8,11 +9,17 @@ import { flowVersionService } from '../flows/flow-version/flow-version.service'
import { system } from '../helper/system/system'
import { AppSystemProp } from '../helper/system/system-props'
import { SystemJobData, SystemJobName } from '../helper/system-jobs/common'
+import { barrierService } from './barrier-service'
import { resumeService } from './resume-service'
import { waitpointService } from './waitpoint-service'
import { WaitpointStatus } from './waitpoint-types'
export async function handleResumeDelayWaitpoint({ data, log }: HandleResumeDelayWaitpointParams): Promise<void> {
+ wideEvent.set({
+ project: { id: data.projectId },
+ flowRun: { id: data.flowRunId },
+ waitpoint: { id: data.waitpointId },
+ })
const flowRun = await flowRunService(log).getOne({ id: data.flowRunId, projectId: data.projectId })
if (isNil(flowRun)) {
log.info({ flowRun: { id: data.flowRunId }, waitpoint: { id: data.waitpointId } },
@@ -30,6 +37,13 @@ export async function handleResumeDelayWaitpoint({ data, log }: HandleResumeDela
'[RESUME_DELAY_WAITPOINT] Waitpoint no longer PENDING (stale timer from completed/deleted waitpoint), skipping')
return
}
+ if (waitpoint.type === PauseType.BARRIER) {
+ log.info({ flowRun: { id: data.flowRunId }, waitpoint: { id: data.waitpointId } },
+ '[RESUME_DELAY_WAITPOINT] Barrier reached its deadline, releasing it with a timed-out summary')
+ await barrierService(log).release({ barrier: waitpoint, timedOut: true, releaseReason: 'timeout' })
+ return
+ }
+
const pauseTimeoutDays = system.getNumberOrThrow(AppSystemProp.PAUSED_FLOW_TIMEOUT_DAYS)
const pastPauseTimeout = dayjs().isAfter(dayjs(flowRun.created).add(pauseTimeoutDays, 'day'))
const isWebhookExpiry = waitpoint.type === PauseType.WEBHOOK
@@ -50,8 +64,10 @@ export async function handleResumeDelayWaitpoint({ data, log }: HandleResumeDela
})
return
}
+
log.info({ flowRun: { id: data.flowRunId }, waitpoint: { id: data.waitpointId } },
'[RESUME_DELAY_WAITPOINT] Resuming flow')
+
await resumeService(log).resumeFromWaitpoint({
flowRunId: data.flowRunId,
waitpointId: data.waitpointId,
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-controller.ts b/packages/server/api/src/app/waitpoints/waitpoint-controller.ts
index 3348a0ceab6..b86db4d59a1 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-controller.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-controller.ts
@@ -1,13 +1,39 @@
-import { CreateWaitpointRequest, CreateWaitpointResponse } from '@activepieces/shared'
+import { isNil } from '@activepieces/core-utils'
+import { ActivepiecesError, BarrierCreatedState, CreateWaitpointRequest, CreateWaitpointResponse, ErrorCode, MAX_INLINE_BARRIER_SIGNALS, PauseType } from '@activepieces/shared'
import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'
import { StatusCodes } from 'http-status-codes'
import { securityAccess } from '../core/security/authorization/fastify-security'
import { domainHelper } from '../helper/domain-helper'
+import { projectService } from '../project/project-service'
+import { barrierService, CreateBarrierResult } from './barrier-service'
import { waitpointService } from './waitpoint-service'
+import { WaitpointSignal } from './waitpoint-types'
export const waitpointController: FastifyPluginAsyncZod = async (app) => {
app.post('/', CreateWaitpointParams, async (request, reply) => {
- const { flowRunId, projectId, stepName, type, version, resumeDateTime, responseToSend, workerHandlerId, httpRequestId } = request.body
+ const { flowRunId, projectId, stepName, type, version, resumeDateTime, responseToSend, workerHandlerId, httpRequestId, barrier } = request.body
+
+ if (type === PauseType.BARRIER) {
+ assertBarrierCarriesNoDeadline({ resumeDateTime })
+ const created = await barrierService(request.log).create({
+ flowRunId,
+ projectId,
+ platformId: await projectService(request.log).getPlatformId(projectId),
+ stepName,
+ version,
+ responseToSend: responseToSend ?? undefined,
+ workerHandlerId: workerHandlerId ?? undefined,
+ httpRequestId: httpRequestId ?? undefined,
+ policy: barrier?.policy,
+ signalLabels: barrier?.signals?.map((signal) => signal.label ?? null),
+ })
+ return reply.status(StatusCodes.CREATED).send({
+ id: created.barrier.id,
+ resumeUrl: await buildResumeUrl({ flowRunId, waitpointId: created.barrier.id }),
+ barrier: await buildBarrierState({ flowRunId, created }),
+ })
+ }
+
const { waitpoint } = await waitpointService(request.log).createForPause({
flowRunId,
projectId,
@@ -19,16 +45,44 @@ export const waitpointController: FastifyPluginAsyncZod = async (app) => {
workerHandlerId: workerHandlerId ?? undefined,
httpRequestId: httpRequestId ?? undefined,
})
- const resumeUrl = await domainHelper.getPublicApiUrl({
- path: `v1/flow-runs/${flowRunId}/waitpoints/${waitpoint.id}`,
- })
return reply.status(StatusCodes.CREATED).send({
id: waitpoint.id,
- resumeUrl,
+ resumeUrl: await buildResumeUrl({ flowRunId, waitpointId: waitpoint.id }),
})
})
}
+async function buildResumeUrl({ flowRunId, waitpointId }: { flowRunId: string, waitpointId: string }): Promise<string> {
+ return domainHelper.getPublicApiUrl({ path: `v1/flow-runs/${flowRunId}/waitpoints/${waitpointId}` })
+}
+
+async function buildBarrierState({ flowRunId, created }: BuildBarrierStateParams): Promise<BarrierCreatedState> {
+ const inlineable = created.signals.length > 0 && created.signals.length <= MAX_INLINE_BARRIER_SIGNALS && created.signals.every((signal) => isNil(signal.sequence))
+ const truncatedForSize = created.signals.length > MAX_INLINE_BARRIER_SIGNALS
+ return {
+ signalCount: created.signalCount,
+ ...(truncatedForSize ? { signalsTruncated: true } : {}),
+ ...(inlineable ? { signals: await Promise.all(created.signals.map((signal) => toSignalLink({ flowRunId, signal }))) } : {}),
+ }
+}
+
+async function toSignalLink({ flowRunId, signal }: { flowRunId: string, signal: WaitpointSignal }): Promise<{ label: string | null, confirmUrl: string }> {
+ return {
+ label: signal.label,
+ confirmUrl: await domainHelper.getPublicApiUrl({ path: `v1/flow-runs/${flowRunId}/signals/${signal.id}/confirm` }),
+ }
+}
+
+function assertBarrierCarriesNoDeadline({ resumeDateTime }: { resumeDateTime: string | undefined }): void {
+ if (isNil(resumeDateTime)) {
+ return
+ }
+ throw new ActivepiecesError({
+ code: ErrorCode.VALIDATION,
+ params: { message: 'A barrier sets its own deadline at creation, so resumeDateTime is not accepted here. Remove it, or open a plain DELAY waitpoint instead.' },
+ })
+}
+
const CreateWaitpointParams = {
config: {
security: securityAccess.engine(),
@@ -40,3 +94,8 @@ const CreateWaitpointParams = {
},
},
}
+
+type BuildBarrierStateParams = {
+ flowRunId: string
+ created: CreateBarrierResult
+}
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-deadline-sweep.ts b/packages/server/api/src/app/waitpoints/waitpoint-deadline-sweep.ts
new file mode 100644
index 00000000000..5489f9ad734
--- /dev/null
+++ b/packages/server/api/src/app/waitpoints/waitpoint-deadline-sweep.ts
@@ -0,0 +1,384 @@
+import { chunk, isNil, tryCatch } from '@activepieces/core-utils'
+import { wideEvent } from '@activepieces/server-utils'
+import { FlowRunStatus, PauseType } from '@activepieces/shared'
+import dayjs from 'dayjs'
+import { FastifyBaseLogger } from 'fastify'
+import { In, LessThanOrEqual } from 'typeorm'
+import { repoFactory } from '../core/db/repo-factory'
+import { distributedStore } from '../database/redis-connections'
+import { systemJobIds } from '../helper/system-jobs/common'
+import { systemJobsSchedule } from '../helper/system-jobs/system-job'
+import { barrierQueue } from './barrier-queue'
+import { WaitpointEntity } from './waitpoint-entity'
+import { waitpointTimeoutJob } from './waitpoint-timeout-job'
+import { Waitpoint, WaitpointStatus } from './waitpoint-types'
+
+const waitpointRepo = repoFactory(WaitpointEntity)
+
+export async function sweepOverdueDeadlines({ log, pageSize, maxPages, maxRedelivered }: SweepOverdueDeadlinesParams): Promise<string[]> {
+ const sweep = await runSweepPages({
+ log,
+ pageSize: pageSize ?? SCAN_PAGE_SIZE,
+ maxPages: maxPages ?? MAX_SCAN_PAGES_PER_TICK,
+ })
+
+ if (sweep.armed.length > 0) {
+ log.info({ armedCount: sweep.armed.length, scannedCount: sweep.scannedCount }, '[sweepOverdueDeadlines] Re-armed overdue waitpoint deadlines')
+ }
+ if (sweep.deadLettered.length > 0) {
+ log.warn({
+ deadLetteredCount: sweep.deadLettered.length,
+ scannedCount: sweep.scannedCount,
+ sample: sweep.deadLettered.slice(0, DEAD_LETTER_SAMPLE_SIZE).map((waitpoint) => waitpoint.id),
+ }, '[sweepOverdueDeadlines] Dead-lettered deadlines whose jobs exhausted their attempts; their runs stay paused')
+ }
+ if (!isNil(sweep.resumeFrom)) {
+ log.warn({
+ stopReason: sweep.stopReason,
+ scannedCount: sweep.scannedCount,
+ armedCount: sweep.armed.length,
+ resumeFrom: sweep.resumeFrom,
+ }, '[sweepOverdueDeadlines] Tick budget spent before the end of the backlog; the next tick resumes from the cursor')
+ }
+
+ const redelivery = await redeliverUndeliveredBarriers({ log, maxRedelivered: maxRedelivered ?? MAX_REDELIVERED_PER_TICK })
+
+ wideEvent.set({
+ waitpointSweep: {
+ scannedCount: sweep.scannedCount,
+ armedCount: sweep.armed.length,
+ deadLetteredCount: sweep.deadLettered.length,
+ stopReason: sweep.stopReason,
+ deadlineBacklogCarried: !isNil(sweep.resumeFrom),
+ barriersEnqueuedCount: redelivery.enqueued,
+ barriersEnqueueFailedCount: redelivery.enqueueFailed,
+ barrierBacklogCarried: redelivery.backlogCarried,
+ },
+ })
+ return sweep.armed
+}
+
+async function redeliverUndeliveredBarriers({ log, maxRedelivered }: RedeliverUndeliveredBarriersParams): Promise<RedeliveryOutcome> {
+ const cursor = await readBarrierCursor()
+ const undelivered = await findUndeliveredBarriers({
+ staleBefore: dayjs().subtract(UNDELIVERED_BARRIER_GRACE_MINUTES, 'minute').toISOString(),
+ cursor,
+ limit: maxRedelivered,
+ })
+ if (undelivered.length === 0) {
+ await rememberBarrierCursor(undefined)
+ return { enqueued: 0, enqueueFailed: 0, backlogCarried: false }
+ }
+ log.warn({
+ undeliveredCount: undelivered.length,
+ resumedFrom: cursor,
+ sample: undelivered.slice(0, DEAD_LETTER_SAMPLE_SIZE).map((barrier) => barrier.id),
+ }, '[redeliverUndeliveredBarriers] Found closed barriers that were never delivered; handing them to the barrier queue')
+ const { enqueued, enqueueFailed } = await enqueueBarrierEvaluations({ undelivered, log })
+ const backlogCarried = undelivered.length >= maxRedelivered
+ const resumeFrom = backlogCarried ? toBarrierCursor(undelivered[undelivered.length - 1]) : undefined
+ await rememberBarrierCursor(resumeFrom)
+ if (!isNil(resumeFrom)) {
+ log.warn({
+ enqueuedCount: enqueued,
+ enqueueFailedCount: enqueueFailed,
+ resumeFrom,
+ }, '[redeliverUndeliveredBarriers] Batch full before the end of the backlog; the next tick resumes from the cursor')
+ }
+ return { enqueued, enqueueFailed, backlogCarried }
+}
+
+async function enqueueBarrierEvaluations({ undelivered, log }: EnqueueBarrierEvaluationsParams): Promise<EnqueueOutcome> {
+ const outcomes = await Promise.all(undelivered.map(async (barrier) => {
+ const { error } = await tryCatch(() => barrierQueue(log).enqueueEvaluation({ barrierId: barrier.id, projectId: barrier.projectId }))
+ if (!isNil(error)) {
+ log.error({ error, waitpoint: { id: barrier.id }, flowRun: { id: barrier.flowRunId } }, '[redeliverUndeliveredBarriers] Could not enqueue an undelivered barrier; the next tick retries it')
+ return false
+ }
+ return true
+ }))
+ const enqueued = outcomes.filter(Boolean).length
+ return { enqueued, enqueueFailed: outcomes.length - enqueued }
+}
+
+async function findUndeliveredBarriers({ staleBefore, cursor, limit }: FindUndeliveredBarriersParams): Promise<Waitpoint[]> {
+ const query = waitpointRepo()
+ .createQueryBuilder('waitpoint')
+ .innerJoin('flow_run', 'flowRun', '"flowRun"."id" = "waitpoint"."flowRunId"')
+ .where('"waitpoint"."type" = :type', { type: PauseType.BARRIER })
+ .andWhere('"waitpoint"."status" = :status', { status: WaitpointStatus.COMPLETED })
+ .andWhere('"waitpoint"."updated" < :staleBefore', { staleBefore })
+ .andWhere('"flowRun"."status" = :runStatus', { runStatus: FlowRunStatus.PAUSED })
+ if (!isNil(cursor)) {
+ query.andWhere('("waitpoint"."updated", "waitpoint"."id") > (:cursorUpdated, :cursorId)', {
+ cursorUpdated: cursor.updated,
+ cursorId: cursor.id,
+ })
+ }
+ return query
+ .orderBy('"waitpoint"."updated"', 'ASC')
+ .addOrderBy('"waitpoint"."id"', 'ASC')
+ .limit(limit)
+ .getMany()
+}
+
+async function readBarrierCursor(): Promise<BarrierRecoveryCursor | undefined> {
+ const stored = await distributedStore.get<BarrierRecoveryCursor>(BARRIER_RECOVERY_CURSOR_KEY)
+ return stored ?? undefined
+}
+
+async function rememberBarrierCursor(cursor: BarrierRecoveryCursor | undefined): Promise<void> {
+ if (isNil(cursor)) {
+ await distributedStore.delete(BARRIER_RECOVERY_CURSOR_KEY)
+ return
+ }
+ await distributedStore.put(BARRIER_RECOVERY_CURSOR_KEY, cursor, DEADLINE_SWEEP_CURSOR_TTL_SECONDS)
+}
+
+function toBarrierCursor(barrier: Waitpoint): BarrierRecoveryCursor {
+ return { updated: barrier.updated, id: barrier.id }
+}
+
+async function runSweepPages({ log, pageSize, maxPages }: RunSweepPagesParams): Promise<SweepOutcome> {
+ const scanStartedAt = await readDatabaseTime()
+ const armed: string[] = []
+ const deadLettered: Waitpoint[] = []
+ let scannedCount = 0
+ let cursor = await readStoredCursor()
+
+ for (let page = 0; page < maxPages; page++) {
+ const overdue = await findOverdueWaitpoints({ now: scanStartedAt, cursor, pageSize })
+ if (overdue.length === 0) {
+ return finishSweep({ armed, deadLettered, scannedCount, resumeFrom: undefined, stopReason: 'drained' })
+ }
+ scannedCount += overdue.length
+
+ const probes = await probeDeadlineJobs({ overdue, log })
+ const pageDeadLettered = probes.filter((probe) => probe.state === 'dead-lettered').map((probe) => probe.waitpoint)
+ await stampDeadLettered({ deadLettered: pageDeadLettered, scanStartedAt })
+ deadLettered.push(...pageDeadLettered)
+
+ const unarmed = probes.filter((probe) => probe.state === 'unarmed').map((probe) => probe.waitpoint)
+ const arming = await armDeadlines({ unarmed, remainingQuota: MAX_ARMED_PER_TICK - armed.length, log })
+ armed.push(...arming.armed)
+
+ if (arming.quotaExhausted) {
+ const resumeFrom = isNil(arming.lastHandled) ? cursor : toCursor(arming.lastHandled)
+ return finishSweep({ armed, deadLettered, scannedCount, resumeFrom, stopReason: 'arm-quota' })
+ }
+
+ cursor = toCursor(overdue[overdue.length - 1])
+ if (overdue.length < pageSize) {
+ return finishSweep({ armed, deadLettered, scannedCount, resumeFrom: undefined, stopReason: 'drained' })
+ }
+ }
+ return finishSweep({ armed, deadLettered, scannedCount, resumeFrom: cursor, stopReason: 'page-budget' })
+}
+
+async function finishSweep({ armed, deadLettered, scannedCount, resumeFrom, stopReason }: SweepOutcome): Promise<SweepOutcome> {
+ await rememberCursor(resumeFrom)
+ return { armed, deadLettered, scannedCount, resumeFrom, stopReason }
+}
+
+async function readStoredCursor(): Promise<DeadlineCursor | undefined> {
+ const stored = await distributedStore.get<DeadlineCursor>(DEADLINE_SWEEP_CURSOR_KEY)
+ return stored ?? undefined
+}
+
+async function rememberCursor(cursor: DeadlineCursor | undefined): Promise<void> {
+ if (isNil(cursor)) {
+ await distributedStore.delete(DEADLINE_SWEEP_CURSOR_KEY)
+ return
+ }
+ await distributedStore.put(DEADLINE_SWEEP_CURSOR_KEY, cursor, DEADLINE_SWEEP_CURSOR_TTL_SECONDS)
+}
+
+async function readDatabaseTime(): Promise<string> {
+ const rows = await waitpointRepo().query<{ now: Date }[]>('SELECT now() AS now')
+ return dayjs(rows[0].now).toISOString()
+}
+
+function toCursor(waitpoint: Waitpoint): DeadlineCursor | undefined {
+ return isNil(waitpoint.resumeDateTime) ? undefined : { resumeDateTime: waitpoint.resumeDateTime, id: waitpoint.id }
+}
+
+async function findOverdueWaitpoints({ now, cursor, pageSize }: FindOverdueWaitpointsParams): Promise<Waitpoint[]> {
+ const query = waitpointRepo()
+ .createQueryBuilder('waitpoint')
+ .innerJoin('flow_run', 'flowRun', '"flowRun"."id" = "waitpoint"."flowRunId"')
+ .where('"waitpoint"."status" = :status', { status: WaitpointStatus.PENDING })
+ .andWhere('"waitpoint"."resumeDateTime" < :now', { now })
+ .andWhere('"waitpoint"."deadLetteredAt" IS NULL')
+ .andWhere('"flowRun"."status" = :runStatus', { runStatus: FlowRunStatus.PAUSED })
+ if (!isNil(cursor)) {
+ query.andWhere('("waitpoint"."resumeDateTime", "waitpoint"."id") > (:cursorResumeDateTime, :cursorId)', {
+ cursorResumeDateTime: cursor.resumeDateTime,
+ cursorId: cursor.id,
+ })
+ }
+ return query
+ .orderBy('"waitpoint"."resumeDateTime"', 'ASC')
+ .addOrderBy('"waitpoint"."id"', 'ASC')
+ .limit(pageSize)
+ .getMany()
+}
+
+async function probeDeadlineJobs({ overdue, log }: ProbeDeadlineJobsParams): Promise<DeadlineProbe[]> {
+ const resolvedChunks: DeadlineProbe[][] = []
+ for (const batch of chunk(overdue, DEADLINE_PROBE_CHUNK_SIZE)) {
+ resolvedChunks.push(await Promise.all(batch.map((waitpoint) => probeDeadlineJob({ waitpoint, log }))))
+ }
+ return resolvedChunks.flat()
+}
+
+async function probeDeadlineJob({ waitpoint, log }: ProbeDeadlineJobParams): Promise<DeadlineProbe> {
+ const jobState = await systemJobsSchedule(log).getJobState(systemJobIds.resumeDelay({ waitpointId: waitpoint.id }))
+ if (jobState === 'unknown') {
+ return { waitpoint, state: 'unarmed' }
+ }
+ return { waitpoint, state: jobState === 'failed' ? 'dead-lettered' : 'armed' }
+}
+
+async function stampDeadLettered({ deadLettered, scanStartedAt }: StampDeadLetteredParams): Promise<void> {
+ const stampedAt = dayjs().toISOString()
+ const idsByProject = deadLettered.reduce<Record<string, string[]>>((grouped, waitpoint) => ({
+ ...grouped,
+ [waitpoint.projectId]: [...grouped[waitpoint.projectId] ?? [], waitpoint.id],
+ }), {})
+ await Promise.all(Object.entries(idsByProject).map(([projectId, ids]) =>
+ waitpointRepo().update({ id: In(ids), projectId, updated: LessThanOrEqual(scanStartedAt) }, { deadLetteredAt: stampedAt }),
+ ))
+}
+
+async function armDeadlines({ unarmed, remainingQuota, log }: ArmDeadlinesParams): Promise<ArmOutcome> {
+ const armed: string[] = []
+ let lastHandled: Waitpoint | undefined = undefined
+ for (const waitpoint of unarmed) {
+ if (armed.length >= remainingQuota) {
+ return { armed, quotaExhausted: true, lastHandled }
+ }
+ lastHandled = waitpoint
+ if (isNil(waitpoint.resumeDateTime)) {
+ continue
+ }
+ const { status } = await waitpointTimeoutJob.schedule({
+ flowRunId: waitpoint.flowRunId,
+ projectId: waitpoint.projectId,
+ waitpointId: waitpoint.id,
+ resumeDateTime: waitpoint.resumeDateTime,
+ log,
+ })
+ if (status === 'added') {
+ armed.push(waitpoint.id)
+ }
+ }
+ return { armed, quotaExhausted: false, lastHandled }
+}
+
+const SCAN_PAGE_SIZE = 500
+const MAX_SCAN_PAGES_PER_TICK = 8
+const MAX_ARMED_PER_TICK = 500
+const DEADLINE_PROBE_CHUNK_SIZE = 50
+const DEAD_LETTER_SAMPLE_SIZE = 10
+const DEADLINE_SWEEP_CURSOR_TTL_SECONDS = 600
+const UNDELIVERED_BARRIER_GRACE_MINUTES = 2
+const MAX_REDELIVERED_PER_TICK = 100
+
+export const DEADLINE_SWEEP_CURSOR_KEY = 'waitpoint:deadline-sweep:cursor'
+export const BARRIER_RECOVERY_CURSOR_KEY = 'waitpoint:barrier-recovery:cursor'
+
+type SweepOverdueDeadlinesParams = {
+ log: FastifyBaseLogger
+ pageSize?: number
+ maxPages?: number
+ maxRedelivered?: number
+}
+
+type RedeliverUndeliveredBarriersParams = {
+ log: FastifyBaseLogger
+ maxRedelivered: number
+}
+
+type RedeliveryOutcome = {
+ enqueued: number
+ enqueueFailed: number
+ backlogCarried: boolean
+}
+
+type EnqueueOutcome = {
+ enqueued: number
+ enqueueFailed: number
+}
+
+type EnqueueBarrierEvaluationsParams = {
+ undelivered: Waitpoint[]
+ log: FastifyBaseLogger
+}
+
+type FindUndeliveredBarriersParams = {
+ staleBefore: string
+ cursor: BarrierRecoveryCursor | undefined
+ limit: number
+}
+
+type BarrierRecoveryCursor = {
+ updated: string
+ id: string
+}
+
+type RunSweepPagesParams = {
+ log: FastifyBaseLogger
+ pageSize: number
+ maxPages: number
+}
+
+type SweepOutcome = {
+ armed: string[]
+ deadLettered: Waitpoint[]
+ scannedCount: number
+ resumeFrom: DeadlineCursor | undefined
+ stopReason: 'drained' | 'arm-quota' | 'page-budget'
+}
+
+type DeadlineCursor = {
+ resumeDateTime: string
+ id: string
+}
+
+type DeadlineProbe = {
+ waitpoint: Waitpoint
+ state: 'unarmed' | 'armed' | 'dead-lettered'
+}
+
+type FindOverdueWaitpointsParams = {
+ now: string
+ cursor: DeadlineCursor | undefined
+ pageSize: number
+}
+
+type ProbeDeadlineJobsParams = {
+ overdue: Waitpoint[]
+ log: FastifyBaseLogger
+}
+
+type ProbeDeadlineJobParams = {
+ waitpoint: Waitpoint
+ log: FastifyBaseLogger
+}
+
+type StampDeadLetteredParams = {
+ deadLettered: Waitpoint[]
+ scanStartedAt: string
+}
+
+type ArmDeadlinesParams = {
+ unarmed: Waitpoint[]
+ remainingQuota: number
+ log: FastifyBaseLogger
+}
+
+type ArmOutcome = {
+ armed: string[]
+ quotaExhausted: boolean
+ lastHandled: Waitpoint | undefined
+}
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-entity.ts b/packages/server/api/src/app/waitpoints/waitpoint-entity.ts
index 92ea40f692c..8026739130d 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-entity.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-entity.ts
@@ -69,6 +69,10 @@ export const WaitpointEntity = new EntitySchema<WaitpointSchema>({
type: 'jsonb',
nullable: true,
},
+ deadLetteredAt: {
+ type: 'timestamp with time zone',
+ nullable: true,
+ },
},
indices: [
{
@@ -85,9 +89,14 @@ export const WaitpointEntity = new EntitySchema<WaitpointSchema>({
columns: ['flowRunId', 'status', 'created'],
},
{
- name: 'idx_waitpoint_pending_resume_date_time',
- columns: ['resumeDateTime'],
- where: '"status" = \'PENDING\' AND "resumeDateTime" IS NOT NULL',
+ name: 'idx_waitpoint_live_deadline',
+ columns: ['resumeDateTime', 'id'],
+ where: '"status" = \'PENDING\' AND "resumeDateTime" IS NOT NULL AND "deadLetteredAt" IS NULL',
+ },
+ {
+ name: 'idx_waitpoint_undelivered_barrier',
+ columns: ['updated', 'id'],
+ where: '"type" = \'BARRIER\' AND "status" = \'COMPLETED\'',
},
],
relations: {
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-service.ts b/packages/server/api/src/app/waitpoints/waitpoint-service.ts
index e72ecfcdc89..50f7745643a 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-service.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-service.ts
@@ -3,14 +3,14 @@ import { FlowRunStatus, PauseType } from '@activepieces/shared'
import dayjs from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
import { EntityManager, Not } from 'typeorm'
+import { z } from 'zod'
import { repoFactory } from '../core/db/repo-factory'
import { transaction } from '../core/db/transaction'
import { flowRunRepo } from '../flows/flow-run/flow-run-service'
import { system } from '../helper/system/system'
import { AppSystemProp } from '../helper/system/system-props'
-import { SystemJobName } from '../helper/system-jobs/common'
-import { systemJobsSchedule } from '../helper/system-jobs/system-job'
import { WaitpointEntity } from './waitpoint-entity'
+import { waitpointTimeoutJob } from './waitpoint-timeout-job'
import { CompleteParams, CompleteResult, CreateForPauseParams, CreateForPauseResult, FindPendingByVersionParams, HandleResumeSignalParams, Waitpoint, WaitpointStatus } from './waitpoint-types'
const waitpointRepo = repoFactory(WaitpointEntity)
@@ -63,20 +63,25 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
else {
log.info({ flowRun: { id: params.flowRunId }, existingStatus: waitpoint.status }, '[waitpointService#createForPause] Waitpoint already exists')
}
- if (!isNil(resumeDateTime)) {
- await systemJobsSchedule(log).upsertJob({
- job: {
- name: SystemJobName.RESUME_DELAY_WAITPOINT,
- data: { flowRunId: params.flowRunId, projectId: params.projectId, waitpointId: waitpoint.id },
- jobId: resumeDelayJobId(waitpoint.id),
- },
- schedule: {
- type: 'one-time',
- date: dayjs(resumeDateTime),
- },
- })
+ if (isNil(resumeDateTime)) {
+ return { inserted, waitpoint }
}
- return { inserted, waitpoint }
+ const { status } = await waitpointTimeoutJob.schedule({
+ flowRunId: params.flowRunId,
+ projectId: params.projectId,
+ waitpointId: waitpoint.id,
+ resumeDateTime,
+ log,
+ })
+ const revivesDeadLetteredDeadline = status === 'retried' || !isNil(waitpoint.deadLetteredAt)
+ if (!revivesDeadLetteredDeadline) {
+ return { inserted, waitpoint }
+ }
+ await waitpointRepo().update({ id: waitpoint.id, projectId: params.projectId }, { deadLetteredAt: null })
+ if (!isNil(waitpoint.deadLetteredAt)) {
+ log.info({ waitpoint: { id: waitpoint.id }, flowRun: { id: params.flowRunId } }, '[waitpointService#createForPause] Re-armed a dead-lettered deadline, so the sweep covers it again')
+ }
+ return { inserted, waitpoint: { ...waitpoint, deadLetteredAt: null } }
},
async complete(params: CompleteParams): Promise<CompleteResult> {
@@ -135,6 +140,7 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
log.info({ flowRun: { id: flowRunId }, waitpoint: { id: waitpointId } }, '[waitpointService#handleResumeSignal] Waitpoint was already delivered, ignoring')
return false
}
+ await waitpointTimeoutJob.remove({ waitpointId, flowRunId, log })
log.info({ flowRun: { id: flowRunId }, waitpoint: { id: waitpointId } }, '[waitpointService#handleResumeSignal] Resume triggered')
return true
}
@@ -207,21 +213,19 @@ export const waitpointService = (log: FastifyBaseLogger) => ({
})
},
- async delete({ id, projectId }: DeleteWaitpointParams): Promise<void> {
- await waitpointRepo().delete({ id, projectId })
- log.info({ waitpoint: { id } }, '[waitpointService#delete] Waitpoint deleted')
- },
-
async deleteByFlowRunId({ flowRunId, projectId }: DeleteByFlowRunIdParams): Promise<void> {
- await waitpointRepo().delete({ flowRunId, projectId })
+ const result = await waitpointRepo()
+ .createQueryBuilder()
+ .delete()
+ .where({ flowRunId, projectId })
+ .returning(['id', 'resumeDateTime'])
+ .execute()
+ const timed = DeletedWaitpointRows.parse(result.raw ?? []).filter((waitpoint) => !isNil(waitpoint.resumeDateTime))
+ await Promise.all(timed.map((waitpoint) => waitpointTimeoutJob.remove({ waitpointId: waitpoint.id, flowRunId, log })))
log.info({ flowRun: { id: flowRunId } }, '[waitpointService#deleteByFlowRunId] Waitpoint deleted')
},
})
-export function resumeDelayJobId(waitpointId: string): string {
- return `resume-delay-${waitpointId}`
-}
-
function clampWaitpointResumeDeadline({ requested, type, flowRunCreated, flowRunId }: ClampWaitpointResumeDeadlineParams): string | undefined {
const pauseTimeoutDays = system.getNumberOrThrow(AppSystemProp.PAUSED_FLOW_TIMEOUT_DAYS)
const runDeadline = dayjs(flowRunCreated).add(pauseTimeoutDays, 'day')
@@ -237,6 +241,11 @@ function clampWaitpointResumeDeadline({ requested, type, flowRunCreated, flowRun
return requested
}
+const DeletedWaitpointRows = z.array(z.object({
+ id: z.string(),
+ resumeDateTime: z.coerce.date().nullable(),
+}))
+
type PausedDelivery = 'dispatched' | 'already-delivered' | 'unknown-waitpoint'
type ClampWaitpointResumeDeadlineParams = {
@@ -246,6 +255,11 @@ type ClampWaitpointResumeDeadlineParams = {
flowRunId: string
}
+type DeleteByFlowRunIdParams = {
+ flowRunId: string
+ projectId: string
+}
+
type HasPendingBarrierParams = {
flowRunId: string
projectId: string
@@ -276,13 +290,3 @@ type FindSubflowWaitpointParams = {
flowRunId: string
projectId: string
}
-
-type DeleteWaitpointParams = {
- id: string
- projectId: string
-}
-
-type DeleteByFlowRunIdParams = {
- flowRunId: string
- projectId: string
-}
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-signal-entity.ts b/packages/server/api/src/app/waitpoints/waitpoint-signal-entity.ts
index 83433513b90..4c9d00ad788 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-signal-entity.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-signal-entity.ts
@@ -49,6 +49,7 @@ export const WaitpointSignalEntity = new EntitySchema<WaitpointSignalSchema>({
{
name: 'idx_waitpoint_signal_ref_id',
columns: ['refId', 'projectId'],
+ where: '"refId" IS NOT NULL',
},
{
name: 'idx_waitpoint_signal_waitpoint_id_project_id_sequence',
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-timeout-job.ts b/packages/server/api/src/app/waitpoints/waitpoint-timeout-job.ts
new file mode 100644
index 00000000000..13a08b7bf52
--- /dev/null
+++ b/packages/server/api/src/app/waitpoints/waitpoint-timeout-job.ts
@@ -0,0 +1,44 @@
+import { isNil } from '@activepieces/core-utils'
+import dayjs from 'dayjs'
+import { FastifyBaseLogger } from 'fastify'
+import { systemJobIds, SystemJobName, UpsertJobResult } from '../helper/system-jobs/common'
+import { systemJobsSchedule } from '../helper/system-jobs/system-job'
+
+async function schedule({ flowRunId, projectId, waitpointId, resumeDateTime, log }: ScheduleParams): Promise<UpsertJobResult> {
+ return systemJobsSchedule(log).upsertJob({
+ job: {
+ name: SystemJobName.RESUME_DELAY_WAITPOINT,
+ data: { flowRunId, projectId, waitpointId },
+ jobId: systemJobIds.resumeDelay({ waitpointId }),
+ },
+ schedule: {
+ type: 'one-time',
+ date: dayjs(resumeDateTime),
+ },
+ })
+}
+
+async function remove({ waitpointId, flowRunId, log }: RemoveParams): Promise<void> {
+ await systemJobsSchedule(log).removeJob({ jobId: systemJobIds.resumeDelay({ waitpointId }) })
+ const legacyJobId = systemJobIds.legacyResumeDelay({ flowRunId })
+ const legacyJob = await systemJobsSchedule(log).getJob<SystemJobName.RESUME_DELAY_WAITPOINT>(legacyJobId)
+ if (!isNil(legacyJob) && legacyJob.data.waitpointId === waitpointId) {
+ await systemJobsSchedule(log).removeJob({ jobId: legacyJobId })
+ }
+}
+
+export const waitpointTimeoutJob = { schedule, remove }
+
+type ScheduleParams = {
+ flowRunId: string
+ projectId: string
+ waitpointId: string
+ resumeDateTime: string
+ log: FastifyBaseLogger
+}
+
+type RemoveParams = {
+ waitpointId: string
+ flowRunId: string
+ log: FastifyBaseLogger
+}
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-types.ts b/packages/server/api/src/app/waitpoints/waitpoint-types.ts
index bc669e2c962..e3e455813c1 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-types.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-types.ts
@@ -35,6 +35,7 @@ type Waitpoint = {
resumePayload: WaitpointResumePayload | null
sealed: boolean
policy: BarrierPolicy | null
+ deadLetteredAt: string | null
}
type WaitpointSignal = {
diff --git a/packages/server/api/src/app/workers/migrations/refill-paused-jobs.ts b/packages/server/api/src/app/workers/migrations/refill-paused-jobs.ts
index 6a8d380c184..659f7d16f7b 100644
--- a/packages/server/api/src/app/workers/migrations/refill-paused-jobs.ts
+++ b/packages/server/api/src/app/workers/migrations/refill-paused-jobs.ts
@@ -8,10 +8,8 @@ import { redisConnections } from '../../database/redis-connections'
import { flowRunRepo } from '../../flows/flow-run/flow-run-service'
import { system } from '../../helper/system/system'
import { AppSystemProp } from '../../helper/system/system-props'
-import { SystemJobName } from '../../helper/system-jobs/common'
-import { systemJobsSchedule } from '../../helper/system-jobs/system-job'
import { WaitpointEntity } from '../../waitpoints/waitpoint-entity'
-import { resumeDelayJobId } from '../../waitpoints/waitpoint-service'
+import { waitpointTimeoutJob } from '../../waitpoints/waitpoint-timeout-job'
import { WaitpointStatus } from '../../waitpoints/waitpoint-types'
import { jobQueue } from '../job-queue/job-queue'
@@ -71,16 +69,12 @@ export const refillPausedRuns = (log: FastifyBaseLogger) => ({
log.error({ error: e, pausedRunId: pausedRun.id }, '[refillPausedRuns] Error removing job')
}
- await systemJobsSchedule(log).upsertJob({
- job: {
- name: SystemJobName.RESUME_DELAY_WAITPOINT,
- data: { flowRunId: pausedRun.id, projectId: pausedRun.projectId, waitpointId: waitpoint.id },
- jobId: resumeDelayJobId(waitpoint.id),
- },
- schedule: {
- type: 'one-time',
- date: dayjs(waitpoint.resumeDateTime),
- },
+ await waitpointTimeoutJob.schedule({
+ flowRunId: pausedRun.id,
+ projectId: pausedRun.projectId,
+ waitpointId: waitpoint.id,
+ resumeDateTime: waitpoint.resumeDateTime,
+ log,
})
migratedCount++
}
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts
index 92c101dddf1..49807ca5007 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts
@@ -1,16 +1,25 @@
import { apId, isNil } from '@activepieces/core-utils'
-import { BarrierSignalStatus, BarrierSummary, ErrorCode, FlowRunStatus, FlowVersionState, PauseType, RunEnvironment } from '@activepieces/shared'
+import { wideEvent } from '@activepieces/server-utils'
+import { BarrierSignalStatus, BarrierSummary, ErrorCode, FlowRunStatus, FlowVersionState, MAX_SIGNAL_REASON_LENGTH, PauseType, RunEnvironment } from '@activepieces/shared'
import { Queue } from 'bullmq'
+import dayjs from 'dayjs'
import { FastifyInstance } from 'fastify'
import { StatusCodes } from 'http-status-codes'
import { databaseConnection } from '../../../../../src/app/database/database-connection'
+import { distributedStore, redisConnections } from '../../../../../src/app/database/redis-connections'
+import { systemJobIds } from '../../../../../src/app/helper/system-jobs/common'
+import { systemJobsSchedule } from '../../../../../src/app/helper/system-jobs/system-job'
import { platformConfigurationService } from '../../../../../src/app/platform/platform-configuration.service'
import { jobQueue } from '../../../../../src/app/workers/job-queue/job-queue'
import { barrierQueue } from '../../../../../src/app/waitpoints/barrier-queue'
+import * as barrierQueueModule from '../../../../../src/app/waitpoints/barrier-queue'
import { BarrierJobData } from '../../../../../src/app/waitpoints/barrier-queue-factory'
import { barrierService } from '../../../../../src/app/waitpoints/barrier-service'
+import { handleResumeDelayWaitpoint } from '../../../../../src/app/waitpoints/resume-delay-handler'
import { resumeService } from '../../../../../src/app/waitpoints/resume-service'
+import { BARRIER_RECOVERY_CURSOR_KEY, DEADLINE_SWEEP_CURSOR_KEY, sweepOverdueDeadlines } from '../../../../../src/app/waitpoints/waitpoint-deadline-sweep'
import { waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
+import { waitpointTimeoutJob } from '../../../../../src/app/waitpoints/waitpoint-timeout-job'
import { Waitpoint, WaitpointStatus } from '../../../../../src/app/waitpoints/waitpoint-types'
import { db } from '../../../../helpers/db'
import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'
@@ -99,6 +108,11 @@ async function countPendingEvaluations({ queue, barrierId }: { queue: Queue<Barr
return jobs.filter((job) => job.data.barrierId === barrierId).length
}
+async function drainEvaluations(queue: Queue<BarrierJobData>): Promise<void> {
+ const jobs = await queue.getJobs(['waiting', 'delayed', 'prioritized', 'paused'])
+ await Promise.all(jobs.map((job) => job.remove()))
+}
+
async function closeWithoutConsuming(barrierId: string) {
await databaseConnection().getRepository('waitpoint').update({ id: barrierId }, {
status: WaitpointStatus.COMPLETED,
@@ -132,6 +146,18 @@ async function dropResumeJobs(flowRunId: string) {
await Promise.all(jobs.map((job) => job.remove()))
}
+async function waitFor(condition: () => Promise<boolean>): Promise<void> {
+ for (let attempt = 0; attempt < 100; attempt++) {
+ if (await condition()) {
+ return
+ }
+ await new Promise((resolve) => setTimeout(resolve, 50))
+ }
+ throw new Error('Timed out waiting for the barrier to settle')
+}
+
+const PAUSE_TIMEOUT_DAYS = Number(process.env.AP_PAUSED_FLOW_TIMEOUT_DAYS ?? '30')
+
async function readStatus(barrierId: string): Promise<WaitpointStatus> {
const barrier = await db.findOneByOrFail<{ status: WaitpointStatus }>('waitpoint', { id: barrierId })
return barrier.status
@@ -259,6 +285,65 @@ describe('signal identity', () => {
const reread = (await listSignals(barrier.id)).find((row) => row.id === signal.id)
expect(reread.status).toBe(BarrierSignalStatus.SUCCEEDED)
})
+
+ it('hands a refId outcome to every barrier holding a signal for that refId', async () => {
+ const queue = barrierQueue(app.log).get()
+ await queue.pause()
+ try {
+ const refId = apId()
+ const first = await createBarrierAwaiting({ refId })
+ const second = await createBarrierAwaiting({ refId })
+ await drainEvaluations(queue)
+ await barrierQueue(app.log).clearEvaluationDedupKey(first.id)
+ await barrierQueue(app.log).clearEvaluationDedupKey(second.id)
+
+ const matched = await barrierService(app.log).receiveSignal({ refId, projectId: ctx.project.id, status: BarrierSignalStatus.SUCCEEDED })
+
+ expect(matched).toBe(true)
+ for (const barrier of [first, second]) {
+ const [signal] = await listSignals(barrier.id)
+ expect(signal.status).toBe(BarrierSignalStatus.SUCCEEDED)
+ expect(await countPendingEvaluations({ queue, barrierId: barrier.id })).toBe(1)
+ }
+ }
+ finally {
+ await drainEvaluations(queue)
+ await queue.resume()
+ }
+ })
+
+ it('refuses a decision once its barrier has closed, so it cannot land after the summary was taken', async () => {
+ const queue = barrierQueue(app.log).get()
+ await queue.pause()
+ try {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ const [signal] = await listSignals(barrier.id)
+ await db.update('waitpoint', barrier.id, { status: WaitpointStatus.COMPLETED })
+
+ const recorded = await barrierService(app.log).recordDecision({
+ barrierId: barrier.id,
+ signalId: signal.id,
+ projectId: ctx.project.id,
+ status: BarrierSignalStatus.SUCCEEDED,
+ })
+
+ expect(recorded).toBe(false)
+ const [reread] = await listSignals(barrier.id)
+ expect(reread.status).toBe(BarrierSignalStatus.PENDING)
+ }
+ finally {
+ await drainEvaluations(queue)
+ await queue.resume()
+ }
+ })
+
+ async function createBarrierAwaiting({ refId }: { refId: string }): Promise<Waitpoint> {
+ const { flowRun } = await createParentRun(FlowRunStatus.RUNNING)
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['child-run'] })
+ await databaseConnection().getRepository('waitpoint_signal').update({ waitpointId: barrier.id }, { refId })
+ return barrier
+ }
})
describe('evaluation coalescing', () => {
@@ -549,3 +634,595 @@ describe('signal count limit', () => {
expect(signalCount).toBe(2)
})
})
+
+describe('barrier deadline', () => {
+ beforeEach(async () => {
+ await distributedStore.delete([DEADLINE_SWEEP_CURSOR_KEY, BARRIER_RECOVERY_CURSOR_KEY])
+ await databaseConnection().query('DELETE FROM "waitpoint"')
+ })
+
+ async function createUndeliveredBarrier({ staleByMinutes }: { staleByMinutes: number }) {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await closeWithoutConsuming(barrier.id)
+ await db.update('waitpoint', barrier.id, { updated: dayjs().subtract(staleByMinutes, 'minute').toISOString() })
+ await dropResumeJobs(flowRun.id)
+ return { flowRun, barrier }
+ }
+
+ async function createOverdueBarrier({ overdueByMinutes = 5 }: { overdueByMinutes?: number } = {}) {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+ await db.update('waitpoint', barrier.id, { resumeDateTime: dayjs().subtract(overdueByMinutes, 'minute').toISOString() })
+ return { flowRun, barrier }
+ }
+
+ async function dropDeadlineJob(waitpointId: string): Promise<void> {
+ await systemJobsSchedule(app.log).removeJob({ jobId: systemJobIds.resumeDelay({ waitpointId }) })
+ }
+
+ async function exhaustDeadlineJobAttempts(waitpointId: string): Promise<void> {
+ const redis = await redisConnections.useExisting()
+ const jobId = systemJobIds.resumeDelay({ waitpointId })
+ await redis.zrem('bull:system-job-queue:delayed', jobId)
+ await redis.zadd('bull:system-job-queue:failed', Date.now(), jobId)
+ }
+
+ async function readDeadLetteredAt(waitpointId: string): Promise<string | null> {
+ const waitpoint = await db.findOneByOrFail<{ deadLetteredAt: string | null }>('waitpoint', { id: waitpointId })
+ return waitpoint.deadLetteredAt
+ }
+
+ it('carries a deadline from creation, so a barrier nobody signals is still swept and released', async () => {
+ const { flowRun, barrier } = await createOverdueBarrier()
+ expect(barrier.resumeDateTime).not.toBeNull()
+ await dropDeadlineJob(barrier.id)
+
+ const armed = await sweepOverdueDeadlines({ log: app.log })
+ expect(armed).toContain(barrier.id)
+
+ await waitFor(async () => await readStatus(barrier.id) === WaitpointStatus.CONSUMED)
+
+ expect(await listSignals(barrier.id)).toHaveLength(0)
+ expect(await listResumeJobs(flowRun.id)).not.toHaveLength(0)
+ })
+
+ it('hands a barrier it finds closed but never delivered back to the barrier queue, which then delivers it', async () => {
+ const queue = barrierQueue(app.log).get()
+ await queue.pause()
+
+ try {
+ const { flowRun, barrier } = await createUndeliveredBarrier({ staleByMinutes: 10 })
+ await drainEvaluations(queue)
+
+ await sweepOverdueDeadlines({ log: app.log })
+
+ expect(await countPendingEvaluations({ queue, barrierId: barrier.id })).toBe(1)
+
+ await barrierService(app.log).releaseIfReady({ barrierId: barrier.id, projectId: ctx.project.id })
+
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(1)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)
+ }
+ finally {
+ await drainEvaluations(queue)
+ await queue.resume()
+ }
+ })
+
+ it('leaves a barrier closed moments ago to the release that closed it', async () => {
+ const queue = barrierQueue(app.log).get()
+ await queue.pause()
+
+ try {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await closeWithoutConsuming(barrier.id)
+ await dropResumeJobs(flowRun.id)
+ await drainEvaluations(queue)
+
+ await sweepOverdueDeadlines({ log: app.log })
+
+ expect(await countPendingEvaluations({ queue, barrierId: barrier.id })).toBe(0)
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(0)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.COMPLETED)
+ }
+ finally {
+ await drainEvaluations(queue)
+ await queue.resume()
+ }
+ })
+
+ it('carries its cursor past barriers it could not hand back, so a newer one is still recovered', async () => {
+ const queue = barrierQueue(app.log).get()
+ await queue.pause()
+ const { barrier: poisoned } = await createUndeliveredBarrier({ staleByMinutes: 20 })
+ const { barrier: healthy } = await createUndeliveredBarrier({ staleByMinutes: 10 })
+ await drainEvaluations(queue)
+
+ const realBarrierQueue = barrierQueueModule.barrierQueue
+ vi.spyOn(barrierQueueModule, 'barrierQueue').mockImplementation((log) => ({
+ ...realBarrierQueue(log),
+ enqueueEvaluation: async (params) => {
+ if (params.barrierId === poisoned.id) {
+ throw new Error('enqueue failed')
+ }
+ return realBarrierQueue(log).enqueueEvaluation(params)
+ },
+ }))
+
+ try {
+ await sweepOverdueDeadlines({ log: app.log, maxRedelivered: 1 })
+
+ expect(await countPendingEvaluations({ queue, barrierId: poisoned.id })).toBe(0)
+ expect(await countPendingEvaluations({ queue, barrierId: healthy.id })).toBe(0)
+
+ await sweepOverdueDeadlines({ log: app.log, maxRedelivered: 1 })
+
+ expect(await countPendingEvaluations({ queue, barrierId: healthy.id })).toBe(1)
+ }
+ finally {
+ vi.restoreAllMocks()
+ await drainEvaluations(queue)
+ await queue.resume()
+ }
+ })
+
+ it('carries on through the batch when one undelivered barrier cannot be handed back', async () => {
+ const queue = barrierQueue(app.log).get()
+ await queue.pause()
+ const { barrier: broken } = await createUndeliveredBarrier({ staleByMinutes: 20 })
+ const { barrier: healthy } = await createUndeliveredBarrier({ staleByMinutes: 10 })
+ await drainEvaluations(queue)
+
+ const realBarrierQueue = barrierQueueModule.barrierQueue
+ vi.spyOn(barrierQueueModule, 'barrierQueue').mockImplementation((log) => ({
+ ...realBarrierQueue(log),
+ enqueueEvaluation: async (params) => {
+ if (params.barrierId === broken.id) {
+ throw new Error('enqueue failed')
+ }
+ return realBarrierQueue(log).enqueueEvaluation(params)
+ },
+ }))
+ const sweepFields = vi.spyOn(wideEvent, 'set')
+
+ try {
+ await sweepOverdueDeadlines({ log: app.log })
+
+ expect(await countPendingEvaluations({ queue, barrierId: broken.id })).toBe(0)
+ expect(await countPendingEvaluations({ queue, barrierId: healthy.id })).toBe(1)
+ expect(await readStatus(broken.id)).toBe(WaitpointStatus.COMPLETED)
+ expect(await readStatus(healthy.id)).toBe(WaitpointStatus.COMPLETED)
+ expect(sweepFields).toHaveBeenCalledWith(expect.objectContaining({
+ waitpointSweep: expect.objectContaining({
+ barriersEnqueuedCount: 1,
+ barriersEnqueueFailedCount: 1,
+ }),
+ }))
+ }
+ finally {
+ vi.restoreAllMocks()
+ await drainEvaluations(queue)
+ await queue.resume()
+ }
+ })
+
+ it('does not report a barrier whose deadline job is still live as re-armed', async () => {
+ const { barrier } = await createOverdueBarrier()
+
+ const armed = await sweepOverdueDeadlines({ log: app.log })
+
+ expect(armed).not.toContain(barrier.id)
+ })
+
+ it('re-arms the oldest overdue deadlines first', async () => {
+ const oldest = await createOverdueBarrier({ overdueByMinutes: 30 })
+ const middle = await createOverdueBarrier({ overdueByMinutes: 20 })
+ const newest = await createOverdueBarrier({ overdueByMinutes: 10 })
+ for (const { barrier } of [oldest, middle, newest]) {
+ await dropDeadlineJob(barrier.id)
+ }
+
+ const armed = await sweepOverdueDeadlines({ log: app.log })
+
+ expect(armed).toEqual([oldest.barrier.id, middle.barrier.id, newest.barrier.id])
+ })
+
+ it('marks a deadline whose job exhausted its attempts, so the sweep stops reading it every tick', async () => {
+ const { barrier } = await createOverdueBarrier()
+ await exhaustDeadlineJobAttempts(barrier.id)
+
+ const armed = await sweepOverdueDeadlines({ log: app.log })
+
+ expect(armed).not.toContain(barrier.id)
+ expect(await readDeadLetteredAt(barrier.id)).not.toBeNull()
+ })
+
+ it('re-arms a newer deadline even when older dead-lettered ones would fill the scan ahead of it', async () => {
+ const stuck = await createOverdueBarrier({ overdueByMinutes: 30 })
+ const newer = await createOverdueBarrier({ overdueByMinutes: 10 })
+ await dropDeadlineJob(stuck.barrier.id)
+ await dropDeadlineJob(newer.barrier.id)
+ await db.update('waitpoint', stuck.barrier.id, { deadLetteredAt: dayjs().toISOString() })
+
+ const armed = await sweepOverdueDeadlines({ log: app.log })
+
+ expect(armed).toEqual([newer.barrier.id])
+ })
+
+ it('pages past deadlines that are already armed to reach a newer unarmed one', async () => {
+ const alreadyArmed = await createOverdueBarrier({ overdueByMinutes: 30 })
+ const newer = await createOverdueBarrier({ overdueByMinutes: 10 })
+ await dropDeadlineJob(newer.barrier.id)
+
+ const armed = await sweepOverdueDeadlines({ log: app.log, pageSize: 1 })
+
+ expect(armed).toEqual([newer.barrier.id])
+ expect(await readDeadLetteredAt(alreadyArmed.barrier.id)).toBeNull()
+ })
+
+ it('carries its cursor to the next tick, so an armed prefix cannot starve a newer deadline', async () => {
+ for (const overdueByMinutes of [40, 30, 20]) {
+ await createOverdueBarrier({ overdueByMinutes })
+ }
+ const newest = await createOverdueBarrier({ overdueByMinutes: 10 })
+ await dropDeadlineJob(newest.barrier.id)
+
+ const firstTick = await sweepOverdueDeadlines({ log: app.log, pageSize: 1, maxPages: 2 })
+ const secondTick = await sweepOverdueDeadlines({ log: app.log, pageSize: 1, maxPages: 2 })
+
+ expect(firstTick).toEqual([])
+ expect(secondTick).toEqual([newest.barrier.id])
+ })
+
+ it('starts over from the oldest once a tick reaches the end of the backlog', async () => {
+ const oldest = await createOverdueBarrier({ overdueByMinutes: 40 })
+ await dropDeadlineJob(oldest.barrier.id)
+
+ await sweepOverdueDeadlines({ log: app.log })
+
+ expect(await distributedStore.get(DEADLINE_SWEEP_CURSOR_KEY)).toBeNull()
+ })
+
+ it('leaves a deadline unmarked when it was re-armed after the scan began', async () => {
+ const { barrier } = await createOverdueBarrier()
+ await exhaustDeadlineJobAttempts(barrier.id)
+ await db.update('waitpoint', barrier.id, { updated: dayjs().add(1, 'minute').toISOString() })
+
+ const armed = await sweepOverdueDeadlines({ log: app.log })
+
+ expect(armed).not.toContain(barrier.id)
+ expect(await readDeadLetteredAt(barrier.id)).toBeNull()
+ })
+
+ it('clears the mark when a fresh pause re-arms the same deadline', async () => {
+ const { flowRun } = await createParentRun()
+ const resumeDateTime = dayjs().add(1, 'hour').toISOString()
+ const { waitpoint } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'delay',
+ type: PauseType.DELAY,
+ version: 'V1',
+ resumeDateTime,
+ })
+ await db.update('waitpoint', waitpoint.id, { deadLetteredAt: dayjs().toISOString() })
+
+ const { waitpoint: reArmed } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'delay',
+ type: PauseType.DELAY,
+ version: 'V1',
+ resumeDateTime,
+ })
+
+ expect(reArmed.deadLetteredAt).toBeNull()
+ expect(await readDeadLetteredAt(waitpoint.id)).toBeNull()
+ })
+
+ it('releases a barrier whose deadline lands past the run-age window instead of failing the run', async () => {
+ const { flowRun, barrier } = await createOverdueBarrier()
+ await db.update('flow_run', flowRun.id, { created: dayjs().subtract(PAUSE_TIMEOUT_DAYS + 1, 'day').toISOString() })
+ await dropDeadlineJob(barrier.id)
+
+ await handleResumeDelayWaitpoint({
+ data: { flowRunId: flowRun.id, projectId: ctx.project.id, waitpointId: barrier.id },
+ log: app.log,
+ })
+
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)
+ expect(await readSummary(barrier.id)).toMatchObject({ timedOut: true, stillRunning: 2 })
+ })
+
+ it('anchors the barrier deadline to the run start, not to the moment the barrier opened', async () => {
+ const { flowRun } = await createParentRun()
+ const runCreated = dayjs().subtract(10, 'day').toISOString()
+ await db.update('flow_run', flowRun.id, { created: runCreated })
+
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+
+ expect(dayjs(barrier.resumeDateTime).toISOString()).toBe(dayjs(runCreated).add(PAUSE_TIMEOUT_DAYS, 'day').toISOString())
+ })
+
+ it('re-arms a deadline that fell behind the run-age window instead of leaving its run paused for good', async () => {
+ const { barrier } = await createOverdueBarrier({ overdueByMinutes: (PAUSE_TIMEOUT_DAYS + 5) * 24 * 60 })
+ await dropDeadlineJob(barrier.id)
+
+ const armed = await sweepOverdueDeadlines({ log: app.log })
+
+ expect(armed).toContain(barrier.id)
+ })
+
+ it('counts the signals nobody answered as still running and marks the release as timed out', async () => {
+ const { flowRun } = await createParentRun(FlowRunStatus.RUNNING)
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]', '[email protected]'] })
+
+ await barrierService(app.log).release({ barrier, timedOut: true, releaseReason: 'timeout' })
+
+ const summary = await readSummary(barrier.id)
+ expect(summary.timedOut).toBe(true)
+ expect(summary.stillRunning).toBe(2)
+ })
+
+ it('leaves a deadline job that is running right now to finish, without a warning', async () => {
+ const { flowRun, barrier } = await createOverdueBarrier()
+ const jobId = systemJobIds.resumeDelay({ waitpointId: barrier.id })
+ const lockKey = `bull:system-job-queue:${jobId}:lock`
+ const redis = await redisConnections.useExisting()
+ await redis.set(lockKey, 'held-by-the-worker-running-this-job')
+ const warn = vi.spyOn(app.log, 'warn')
+
+ try {
+ await waitpointTimeoutJob.remove({ waitpointId: barrier.id, flowRunId: flowRun.id, log: app.log })
+
+ expect(warn).not.toHaveBeenCalled()
+ expect(await systemJobsSchedule(app.log).getJob(jobId)).toBeDefined()
+ }
+ finally {
+ vi.restoreAllMocks()
+ await redis.del(lockKey)
+ }
+ })
+})
+
+describe('multi-approval confirm page', () => {
+ async function createApprovalBarrier({ reasonRequiredOn, requiredSuccesses, runStatus }: { reasonRequiredOn?: 'none' | 'reject' | 'both', requiredSuccesses?: number, runStatus?: FlowRunStatus } = {}) {
+ const { flowRun } = await createParentRun(runStatus ?? FlowRunStatus.PAUSED)
+ const created = await barrierService(app.log).create({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ platformId: ctx.platform.id,
+ stepName: 'approval',
+ version: 'V1',
+ policy: { requiredSuccesses: requiredSuccesses ?? 2, ...(reasonRequiredOn ? { reasonRequiredOn } : {}) },
+ })
+ return { flowRun, created, signals: await listSignals(created.barrier.id) }
+ }
+
+ it('records each approver\'s decision and reason on their own signal', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier({ requiredSuccesses: 3 })
+
+ for (const signal of signals.slice(0, 2)) {
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signal.id}/confirm?action=approve`,
+ payload: { reason: `looks good from ${signal.label}` },
+ })
+ expect(response.statusCode).toBe(200)
+ }
+
+ const decided = (await listSignals(created.barrier.id)).filter((signal) => signal.status === BarrierSignalStatus.SUCCEEDED)
+ expect(decided).toHaveLength(2)
+ expect(decided.map((signal) => signal.result.reason).sort()).toEqual(['looks good from [email protected]', 'looks good from [email protected]'])
+ })
+
+ it('releases once the required approvals have landed, leaving the third link closed', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier()
+
+ for (const signal of signals.slice(0, 2)) {
+ await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signal.id}/confirm?action=approve`,
+ payload: { reason: 'ok' },
+ })
+ }
+
+ await waitFor(async () => await readStatus(created.barrier.id) === WaitpointStatus.CONSUMED)
+ expect(await listSignals(created.barrier.id)).toHaveLength(0)
+ })
+
+ it('records a decision that lands before the run has flipped to PAUSED', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier({ runStatus: FlowRunStatus.RUNNING })
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=approve`,
+ payload: { reason: 'approving from the email before the engine paused' },
+ })
+
+ expect(response.statusCode).toBe(200)
+ const stored = (await listSignals(created.barrier.id)).find((signal) => signal.id === signals[0].id)
+ expect(stored?.status).toBe(BarrierSignalStatus.SUCCEEDED)
+ })
+
+ it('keeps the first decision when the same link is posted twice', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier({ requiredSuccesses: 3 })
+
+ const approve = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=approve`,
+ payload: { reason: 'yes' },
+ })
+ const flip = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=disapprove`,
+ payload: { reason: 'changed my mind' },
+ })
+
+ expect(approve.statusCode).toBe(200)
+ expect(flip.statusCode).toBe(200)
+ const stored = (await listSignals(created.barrier.id)).find((signal) => signal.id === signals[0].id)
+ expect(stored?.status).toBe(BarrierSignalStatus.SUCCEEDED)
+ expect(stored?.result.reason).toBe('yes')
+ })
+
+ it('does not resurrect a signal the release already deleted', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier()
+
+ for (const signal of signals.slice(0, 2)) {
+ await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signal.id}/confirm?action=approve`,
+ payload: { reason: 'ok' },
+ })
+ }
+ await waitFor(async () => await readStatus(created.barrier.id) === WaitpointStatus.CONSUMED)
+
+ const late = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[2].id}/confirm?action=approve`,
+ payload: { reason: 'late to the party' },
+ })
+
+ expect(late.statusCode).toBe(200)
+ expect(await listSignals(created.barrier.id)).toHaveLength(0)
+ })
+
+ it('records a disapprove with no reason when the policy never asked for one', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier()
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=disapprove`,
+ payload: {},
+ })
+
+ expect(response.statusCode).toBe(200)
+ const stored = (await listSignals(created.barrier.id)).find((signal) => signal.id === signals[0].id)
+ expect(stored?.status).toBe(BarrierSignalStatus.REJECTED)
+ })
+
+ it('rejects a reject with no reason when reasonRequiredOn is reject', async () => {
+ const { flowRun, signals } = await createApprovalBarrier({ reasonRequiredOn: 'reject' })
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=disapprove`,
+ payload: {},
+ })
+
+ expect(response.statusCode).toBe(400)
+ expect((await listSignals(signals[0].waitpointId))[0].status).toBe(BarrierSignalStatus.PENDING)
+ })
+
+ it('rejects an over-long reason rather than truncating it', async () => {
+ const { flowRun, signals } = await createApprovalBarrier()
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=approve`,
+ payload: { reason: 'x'.repeat(MAX_SIGNAL_REASON_LENGTH + 1) },
+ })
+
+ expect(response.statusCode).toBe(400)
+ })
+
+ it('counts a line break in a form-posted reason once, the way the textarea limit does', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier()
+ const lines = ['line one', 'line two', 'line three']
+ const lastLine = 'x'.repeat(MAX_SIGNAL_REASON_LENGTH - lines.join('\n').length - 1)
+ const typedReason = [...lines, lastLine].join('\n')
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=approve`,
+ headers: { 'content-type': 'application/x-www-form-urlencoded' },
+ payload: `reason=${encodeURIComponent(typedReason.replace(/\n/g, '\r\n'))}`,
+ })
+
+ expect(typedReason).toHaveLength(MAX_SIGNAL_REASON_LENGTH)
+ expect(response.statusCode).toBe(200)
+ const stored = (await listSignals(created.barrier.id)).find((signal) => signal.id === signals[0].id)
+ expect(stored?.result).toMatchObject({ reason: typedReason })
+ })
+
+ it('stores a reason carrying a NUL byte sanitised rather than failing the write', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier()
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=approve`,
+ payload: { reason: `fine ${String.fromCharCode(0)} by me` },
+ })
+
+ expect(response.statusCode).toBe(200)
+ const stored = (await listSignals(created.barrier.id)).find((signal) => signal.id === signals[0].id)
+ expect(stored?.status).toBe(BarrierSignalStatus.SUCCEEDED)
+ expect(JSON.stringify(stored?.result)).not.toContain('\\u0000')
+ })
+
+ it('tells the third approver the request is already closed once the barrier released', async () => {
+ const { flowRun, created, signals } = await createApprovalBarrier()
+
+ for (const signal of signals.slice(0, 2)) {
+ await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signal.id}/confirm?action=approve`,
+ payload: { reason: 'ok' },
+ })
+ }
+ await waitFor(async () => await readStatus(created.barrier.id) === WaitpointStatus.CONSUMED)
+
+ const response = await app.inject({
+ method: 'GET',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[2].id}/confirm`,
+ headers: { accept: 'text/html' },
+ })
+
+ expect(response.statusCode).toBe(200)
+ expect(response.body).toContain('Already responded')
+ })
+
+ it('records nothing when the confirm url is posted with no action at all', async () => {
+ const { flowRun, signals } = await createApprovalBarrier()
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm`,
+ payload: { reason: 'posted the bare link' },
+ })
+
+ expect(response.statusCode).toBe(400)
+ expect((await listSignals(signals[0].waitpointId))[0].status).toBe(BarrierSignalStatus.PENDING)
+ })
+
+ it('records nothing when the action is misspelled rather than treating it as an approval', async () => {
+ const { flowRun, signals } = await createApprovalBarrier()
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/signals/${signals[0].id}/confirm?action=aprove`,
+ payload: { reason: 'typo' },
+ })
+
+ expect(response.statusCode).toBe(400)
+ expect((await listSignals(signals[0].waitpointId))[0].status).toBe(BarrierSignalStatus.PENDING)
+ })
+
+ it('refuses a signal id that belongs to another run', async () => {
+ const { signals } = await createApprovalBarrier()
+ const { flowRun: otherRun } = await createParentRun()
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${otherRun.id}/signals/${signals[0].id}/confirm?action=approve`,
+ payload: { reason: 'not mine' },
+ })
+
+ expect(response.statusCode).toBe(200)
+ expect((await listSignals(signals[0].waitpointId))[0].status).toBe(BarrierSignalStatus.PENDING)
+ })
+})
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
index 40b86721a72..7f6289fd0c8 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/waitpoint.test.ts
@@ -1,9 +1,10 @@
import { apId } from '@activepieces/core-utils'
import { FlowRunStatus, FlowVersionState, PauseType, RunEnvironment } from '@activepieces/shared'
import { FastifyInstance } from 'fastify'
+import { systemJobIds } from '../../../../../src/app/helper/system-jobs/common'
import * as systemJobModule from '../../../../../src/app/helper/system-jobs/system-job'
import { handleResumeDelayWaitpoint } from '../../../../../src/app/waitpoints/resume-delay-handler'
-import { resumeDelayJobId, waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
+import { waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
import { WaitpointStatus } from '../../../../../src/app/waitpoints/waitpoint-types'
import { db } from '../../../../helpers/db'
import { createMockFlow, createMockFlowRun, createMockFlowVersion, mockAndSaveBasicSetup } from '../../../../helpers/mocks'
@@ -54,6 +55,19 @@ async function createFlowRun(params?: { status?: FlowRunStatus }) {
describe('Waitpoint service', () => {
describe('createForPause', () => {
+ it('should write a fresh timed waitpoint row only once', async () => {
+ const { flowRun } = await createFlowRun()
+ const { waitpoint } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'approval',
+ type: PauseType.WEBHOOK,
+ })
+
+ const stored = await db.findOneByOrFail<{ created: string, updated: string }>('waitpoint', { id: waitpoint.id })
+ expect(new Date(stored.updated).getTime()).toBe(new Date(stored.created).getTime())
+ })
+
it('should create a PENDING waitpoint when none exists', async () => {
const { flowRun } = await createFlowRun()
@@ -185,7 +199,7 @@ describe('Waitpoint service', () => {
type: PauseType.DELAY,
resumeDateTime: new Date(Date.now() + 60000).toISOString(),
}
- const upsertJobSpy = vi.fn()
+ const upsertJobSpy = vi.fn().mockResolvedValue({ status: 'added' })
vi.spyOn(systemJobModule, 'systemJobsSchedule').mockImplementation((log) => ({
...originalSystemJobsSchedule(log),
upsertJob: upsertJobSpy,
@@ -204,7 +218,7 @@ describe('Waitpoint service', () => {
it('should give each waitpoint of one run its own resume job id', async () => {
const { flowRun } = await createFlowRun()
- const upsertJobSpy = vi.fn()
+ const upsertJobSpy = vi.fn().mockResolvedValue({ status: 'added' })
vi.spyOn(systemJobModule, 'systemJobsSchedule').mockImplementation((log) => ({
...originalSystemJobsSchedule(log),
upsertJob: upsertJobSpy,
@@ -229,8 +243,8 @@ describe('Waitpoint service', () => {
waitpointId: params.job.data.waitpointId,
}))
expect(scheduled).toEqual([
- { jobId: resumeDelayJobId(approval.waitpoint.id), waitpointId: approval.waitpoint.id },
- { jobId: resumeDelayJobId(delay.waitpoint.id), waitpointId: delay.waitpoint.id },
+ { jobId: systemJobIds.resumeDelay({ waitpointId: approval.waitpoint.id }), waitpointId: approval.waitpoint.id },
+ { jobId: systemJobIds.resumeDelay({ waitpointId: delay.waitpoint.id }), waitpointId: delay.waitpoint.id },
])
expect(scheduled[0].jobId).not.toBe(scheduled[1].jobId)
})
@@ -326,6 +340,30 @@ describe('Waitpoint service', () => {
expect(result.waitpoint!.resumePayload).toEqual({ body: { greeting: 'Hello' } })
})
+ it('should refuse to complete a waitpoint belonging to another project', async () => {
+ const { flowRun } = await createFlowRun()
+ const pauseResult = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'approval',
+ type: PauseType.WEBHOOK,
+ version: 'V1',
+ })
+
+ const result = await waitpointService(app.log).complete({
+ flowRunId: flowRun.id,
+ projectId: apId(),
+ waitpointId: pauseResult.waitpoint.id,
+ resumePayload: { body: { forged: true } },
+ })
+
+ expect(result.completedExisting).toBe(false)
+ expect(result.waitpoint).toBeNull()
+
+ const stored = await db.findOneByOrFail<{ status: string }>('waitpoint', { id: pauseResult.waitpoint.id })
+ expect(stored.status).toBe(WaitpointStatus.PENDING)
+ })
+
it('should drop stale resume signal when no PENDING waitpoint exists', async () => {
const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
@@ -489,6 +527,23 @@ describe('Waitpoint service', () => {
expect(result.inserted).toBe(true)
expect(result.waitpoint.type).toBe(PauseType.DELAY)
})
+
+ it('should remove the timeout job of a timed waitpoint it deletes', async () => {
+ const { flowRun } = await createFlowRun()
+ const { waitpoint } = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'delay_step',
+ type: PauseType.DELAY,
+ resumeDateTime: new Date(Date.now() + 60000).toISOString(),
+ })
+ const jobId = systemJobIds.resumeDelay({ waitpointId: waitpoint.id })
+ expect(await originalSystemJobsSchedule(app.log).getJob(jobId)).toBeDefined()
+
+ await waitpointService(app.log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: ctx.project.id })
+
+ expect(await originalSystemJobsSchedule(app.log).getJob(jobId)).toBeUndefined()
+ })
})
describe('findUndeliveredCompletedWaitpoint', () => {
@@ -568,6 +623,32 @@ describe('Waitpoint service', () => {
const result = await waitpointService(app.log).findUndeliveredCompletedWaitpoint({ flowRunId: flowRun.id, projectId: ctx.project.id })
expect(result).toBeNull()
})
+
+ it('should return the COMPLETED waitpoint even when an older unconsumed PENDING one lingers', async () => {
+ const { flowRun } = await createFlowRun({ status: FlowRunStatus.RUNNING })
+
+ await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'create_approval_links',
+ type: PauseType.WEBHOOK,
+ })
+ const pause = await waitpointService(app.log).createForPause({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ stepName: 'wait_for_approval',
+ type: PauseType.WEBHOOK,
+ })
+ await waitpointService(app.log).complete({
+ flowRunId: flowRun.id,
+ projectId: ctx.project.id,
+ waitpointId: pause.waitpoint.id,
+ resumePayload: { body: { approved: true } },
+ })
+
+ const result = await waitpointService(app.log).findUndeliveredCompletedWaitpoint({ flowRunId: flowRun.id, projectId: ctx.project.id })
+ expect(result?.id).toBe(pause.waitpoint.id)
+ })
})
describe('concurrent complete calls', () => {
diff --git a/packages/server/api/test/integration/ce/workers/refill-paused-jobs.test.ts b/packages/server/api/test/integration/ce/workers/refill-paused-jobs.test.ts
index b23d723ead8..c3b8bb4c311 100644
--- a/packages/server/api/test/integration/ce/workers/refill-paused-jobs.test.ts
+++ b/packages/server/api/test/integration/ce/workers/refill-paused-jobs.test.ts
@@ -4,8 +4,9 @@ import dayjs from 'dayjs'
import { FastifyInstance } from 'fastify'
import { databaseConnection } from '../../../../src/app/database/database-connection'
import { redisConnections } from '../../../../src/app/database/redis-connections'
+import { systemJobIds } from '../../../../src/app/helper/system-jobs/common'
import { systemJobsSchedule } from '../../../../src/app/helper/system-jobs/system-job'
-import { resumeDelayJobId, waitpointService } from '../../../../src/app/waitpoints/waitpoint-service'
+import { waitpointService } from '../../../../src/app/waitpoints/waitpoint-service'
import { WaitpointStatus } from '../../../../src/app/waitpoints/waitpoint-types'
import { refillPausedRuns } from '../../../../src/app/workers/migrations/refill-paused-jobs'
import { db } from '../../../helpers/db'
@@ -82,7 +83,7 @@ async function createPendingDelay({ flowRunId, stepName }: { flowRunId: string,
async function loseScheduledTimers({ waitpointIds }: { waitpointIds: string[] }): Promise<void> {
for (const waitpointId of waitpointIds) {
- const job = await systemJobsSchedule(app.log).getJob(resumeDelayJobId(waitpointId))
+ const job = await systemJobsSchedule(app.log).getJob(systemJobIds.resumeDelay({ waitpointId }))
await job?.remove()
}
const redis = await redisConnections.useExisting()
@@ -90,7 +91,7 @@ async function loseScheduledTimers({ waitpointIds }: { waitpointIds: string[] })
}
async function timerIsArmed({ waitpointId }: { waitpointId: string }): Promise<boolean> {
- const job = await systemJobsSchedule(app.log).getJob(resumeDelayJobId(waitpointId))
+ const job = await systemJobsSchedule(app.log).getJob(systemJobIds.resumeDelay({ waitpointId }))
return !isNilJob(job)
}
diff --git a/packages/server/engine/src/lib/piece-context/rejected-request.ts b/packages/server/engine/src/lib/piece-context/rejected-request.ts
new file mode 100644
index 00000000000..4776d2b38b0
--- /dev/null
+++ b/packages/server/engine/src/lib/piece-context/rejected-request.ts
@@ -0,0 +1,37 @@
+import { ErrorCode, tryCatch, tryCatchSync } from '@activepieces/core-utils'
+import { EngineGenericError, PausedFlowTimeoutError, WaitpointRejectedError } from '@activepieces/shared'
+import { z } from 'zod'
+
+export async function throwForRejectedRequest({ response, name, summary }: ThrowForRejectedRequestParams): Promise<never> {
+ const parsed = await readRejectionBody(response)
+ const statusSummary = `${summary}: ${response.status} ${response.statusText}`
+ if (parsed?.code === ErrorCode.PAUSED_FLOW_TIMEOUT_EXCEEDED) {
+ throw new PausedFlowTimeoutError(undefined, parsed.params?.pauseTimeoutDays)
+ }
+ if (parsed?.code === ErrorCode.VALIDATION) {
+ throw new WaitpointRejectedError(parsed.params?.message ?? statusSummary)
+ }
+ throw new EngineGenericError(name, statusSummary)
+}
+
+const rejectionBodySchema = z.object({
+ code: z.string().optional(),
+ params: z.object({
+ message: z.string().optional(),
+ pauseTimeoutDays: z.number().optional(),
+ }).optional(),
+})
+
+async function readRejectionBody(response: Response): Promise<RejectionBody | null> {
+ const { data: body } = await tryCatch(() => response.text())
+ const { data: parsed } = tryCatchSync(() => rejectionBodySchema.parse(JSON.parse(body ?? '')))
+ return parsed ?? null
+}
+
+type RejectionBody = z.infer<typeof rejectionBodySchema>
+
+type ThrowForRejectedRequestParams = {
+ response: Response
+ name: string
+ summary: string
+}
diff --git a/packages/server/engine/src/lib/piece-context/waitpoint-client.ts b/packages/server/engine/src/lib/piece-context/waitpoint-client.ts
index 1ef2ca7d067..17b9fcc1155 100644
--- a/packages/server/engine/src/lib/piece-context/waitpoint-client.ts
+++ b/packages/server/engine/src/lib/piece-context/waitpoint-client.ts
@@ -1,6 +1,6 @@
-import { ErrorCode } from '@activepieces/core-utils'
-import { CreateWaitpointRequest, CreateWaitpointResponse, EngineGenericError, PausedFlowTimeoutError } from '@activepieces/shared'
+import { CreateWaitpointRequest, CreateWaitpointResponse } from '@activepieces/shared'
import { retryFetch } from '../api/retry-fetch'
+import { throwForRejectedRequest } from './rejected-request'
export const waitpointClient = {
create: async ({ apiUrl, engineToken, ...body }: CreateWaitpointClientRequest): Promise<CreateWaitpointResponse> => {
@@ -13,41 +13,12 @@ export const waitpointClient = {
body: JSON.stringify(body),
})
if (!response.ok) {
- const parsed = await parseErrorBody(response)
- if (parsed?.code === ErrorCode.PAUSED_FLOW_TIMEOUT_EXCEEDED) {
- throw new PausedFlowTimeoutError(undefined, parsed.params?.pauseTimeoutDays)
- }
- throw new EngineGenericError('WaitpointCreationError', `Failed to create waitpoint: ${response.status} ${response.statusText}`)
+ await throwForRejectedRequest({ response, name: 'WaitpointCreationError', summary: 'Failed to create waitpoint' })
}
return response.json() as Promise<CreateWaitpointResponse>
},
}
-async function parseErrorBody(response: Response): Promise<ApiErrorBody | null> {
- try {
- const body: unknown = await response.json()
- if (isApiErrorBody(body)) {
- return body
- }
- return null
- }
- catch {
- return null
- }
-}
-
-function isApiErrorBody(value: unknown): value is ApiErrorBody {
- if (typeof value !== 'object' || value === null || !('code' in value)) {
- return false
- }
- return typeof value.code === 'string'
-}
-
-type ApiErrorBody = {
- code: string
- params?: { pauseTimeoutDays?: number }
-}
-
type CreateWaitpointClientRequest = CreateWaitpointRequest & {
apiUrl: string
engineToken: string
diff --git a/packages/server/engine/test/piece-context/rejected-request.test.ts b/packages/server/engine/test/piece-context/rejected-request.test.ts
new file mode 100644
index 00000000000..ac6ea0e43b3
--- /dev/null
+++ b/packages/server/engine/test/piece-context/rejected-request.test.ts
@@ -0,0 +1,52 @@
+import { ErrorCode } from '@activepieces/core-utils'
+import { EngineGenericError, PausedFlowTimeoutError, WaitpointRejectedError } from '@activepieces/shared'
+import { throwForRejectedRequest } from '../../src/lib/piece-context/rejected-request'
+
+describe('throwForRejectedRequest', () => {
+ it('fails the step as a user error when the server refuses the request as invalid', async () => {
+ const response = jsonResponse({
+ status: 400,
+ body: { code: ErrorCode.VALIDATION, params: { message: 'This step waits on 12 things, which exceeds the maximum of 10.' } },
+ })
+
+ const rejection = throwForRejectedRequest({ response, name: 'WaitpointCreationError', summary: 'Failed to create waitpoint' })
+
+ await expect(rejection).rejects.toBeInstanceOf(WaitpointRejectedError)
+ await expect(rejection).rejects.toThrow('exceeds the maximum of 10')
+ })
+
+ it('keeps an authentication failure an engine error', async () => {
+ const response = jsonResponse({
+ status: 401,
+ body: { code: ErrorCode.AUTHENTICATION, params: { message: 'invalid bearer token' } },
+ })
+
+ const rejection = throwForRejectedRequest({ response, name: 'WaitpointCreationError', summary: 'Failed to create waitpoint' })
+
+ await expect(rejection).rejects.toBeInstanceOf(EngineGenericError)
+ })
+
+ it('keeps a missing route an engine error', async () => {
+ const response = new Response('Not Found', { status: 404, statusText: 'Not Found' })
+
+ const rejection = throwForRejectedRequest({ response, name: 'WaitpointCreationError', summary: 'Failed to create waitpoint' })
+
+ await expect(rejection).rejects.toBeInstanceOf(EngineGenericError)
+ await expect(rejection).rejects.toThrow('Failed to create waitpoint: 404 Not Found')
+ })
+
+ it('reports a pause past the run deadline as a pause timeout', async () => {
+ const response = jsonResponse({
+ status: 400,
+ body: { code: ErrorCode.PAUSED_FLOW_TIMEOUT_EXCEEDED, params: { pauseTimeoutDays: 30 } },
+ })
+
+ const rejection = throwForRejectedRequest({ response, name: 'WaitpointCreationError', summary: 'Failed to create waitpoint' })
+
+ await expect(rejection).rejects.toBeInstanceOf(PausedFlowTimeoutError)
+ })
+})
+
+function jsonResponse({ status, body }: { status: number, body: unknown }): Response {
+ return new Response(JSON.stringify(body), { status, headers: { 'content-type': 'application/json' } })
+}