AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(flows): deleted flows no longer get stuck hidden and still running (#15693)
Hazem Adel735fc110100d
diff --git a/CLAUDE.md b/CLAUDE.mdindex 3bb2019f434..3d02a89cf98 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -100,9 +100,14 @@ When running in `--mode=cloud`, do not use OAuth2 connections — the OAuth prov- If the PR includes any contributions to pieces (integrations under `packages/pieces`), also add the appropriate pieces label (in addition to the primary label above):- **`🧩 area/third-party-pieces`** — for third-party integrations (most pieces under `packages/pieces/community/`)- **`🧩 area/core-pieces`** — for core pieces (under `packages/pieces/core/`)-- **Always fill the "Breaking change?" section of the PR template** — tick exactly one box (the `breaking-change-check` CI job fails if it is left unedited). A change is breaking if a self-hoster or API consumer must take action: removed/renamed API fields or endpoints, dropped columns, new required fields, removed/required env vars, or default/limit/behaviour changes. If it is breaking:- - also apply the **`⛓️💥 breaking-change`** label (in addition to the primary label above), and- - add an entry to `docs/install/reference/breaking-changes.mdx` describing what changed and the action required. CI enforces that the label and the docs entry travel together.+- **The PR template has TWO required sections and `breaking-change-check` fails if either is left unedited** — `### Breaking change?` *and* `### Security impact?`. Tick exactly one box in each (ticking two in one section also fails). Copy the option text from `.github/pull_request_template.md`; the parser matches `- [x] no` / `- [x] yes` under each heading, so keep the `no`/`yes` as the first word after the checkbox.+ - `Breaking change?` has **three** options: `no`, `yes — technical` (removed/renamed API field or endpoint, dropped column, new required field, removed/required env var), `yes — functional` (default/limit/behaviour change, new self-hosted setup step).+ - `Security impact?` has **two**: `no`, `yes — security-sensitive` (touches authentication, secrets, permissions/roles, cryptography, SSRF / outbound HTTP, or file handling). This one is a **declaration only** — no label, no docs entry, nothing else is coupled to it.+- **If the change is breaking, the label and the docs entry must travel together** — the check's coupling is bidirectional and it also cross-checks the template answer, so label, docs entry and ticked box must all agree:+ - apply the **`⛓️💥 breaking-change`** label (in addition to the primary label above), **and**+ - add an entry to `docs/install/reference/breaking-changes.mdx`. It must have a `####` title **and** at least one body line — a bare heading, a `---`, or loose prose does not count as an entry.+ - Ticking `no` while carrying the label fails, and so does adding a docs entry without the label.+- **Never `gh run rerun` a `pull_request`-triggered check to pick up a PR description edit.** A rerun replays the *original* event payload, so it re-reads the stale body and fails again — and because it is a later attempt on an older run, its failure shadows a newer run that already passed. Edit the body (which fires `edited`, a trigger these workflows listen for) and let the fresh run supersede. To check the answer locally without any run at all: `PR_BODY="$(gh pr view <n> --json body --jq .body)" PR_LABELS="$(gh pr view <n> --json labels --jq '[.labels[].name]')" BASE_REF=main npx tsx tools/scripts/breaking-change-check.ts`- **Non-rollbackable migrations are a separate axis** from customer-facing breaking changes: a migration that runs destructive DDL (`DROP TABLE`/`DROP COLUMN`, `ADD ... NOT NULL` without `DEFAULT`, etc.) must set `breaking = true` on the migration class — this is the rollback-safety flag (used by `rollback-migrations.ts` and the release rollback note), enforced by `check-migration-rollback.ts`. It does **not** by itself require the `⛓️💥 breaking-change` label; decide that from the upgrade-impact question above.## Database Migrationsdiff --git a/brain/knowledge/flows-execution/flows.md b/brain/knowledge/flows-execution/flows.mdindex b8700f2ae25..ea6732d5baf 100644--- a/brain/knowledge/flows-execution/flows.md+++ b/brain/knowledge/flows-execution/flows.md@@ -36,7 +36,8 @@ Flows are the core automation primitive: a versioned directed graph of trigger +- **`flowVersionSideEffects.preApplyOperation` writes on the default connection, so it escapes any caller transaction.** `handleSampleDataDeletion` and `handleUpdateTriggerWebhookSimulation` take no `entityManager`; a write they make from inside a `transaction()` survives its rollback. They early-return for `IMPORT_FLOW`/`UPDATE_SAMPLE_DATA_INFO`, so the transactional path above is safe today — but adding an operation type to it silently reintroduces partial commits. `updateLastModified` sits outside that swallow-all catch on purpose: a swallowed statement failure inside a transaction poisons it and resurfaces as a confusing "transaction is aborted" error on the next statement.- **`transaction()` (`core/db/transaction.ts`) is a bare `dataSource.transaction()`** — it acquires a *new* connection, not a savepoint. Nesting it deadlocks, so check every caller before wrapping a service method that others may already call inside a transaction.- Step settings split a piece's props into an always-visible **essential** set and a collapsed **Advanced** section: a prop is Advanced only when it sets `advanced: true` (everything else — incl. `MARKDOWN`, tab/section group members, and checkbox reveal targets — stays essential). `propertyGroups` render as tabs, sectioned cards, or the "Add filter" builder.-- **Flows stuck in `DELETING` keep eating the active-flow limit.** Deletion is a durable BullMQ system job (`delete-flow-<flowId>`), not synchronous: `delete()` sets `operationStatus=DELETING` and enqueues, and the row plus `status=ENABLED` only go away when the job finishes. That job runs `sampleDataService.deleteForFlow`, whose `DELETE FROM file … metadata->>'flowId'=?` had no index — on the large prod `file` table it seq-scans, blows `statement_timeout`, exhausts its 2 attempts and lands **permanently** in the failed set. The flow is then hidden from the UI list (which filters `!=DELETING`) but still counted by the active-flows quota (`getUsage` counts `status=ENABLED`), so Publish silently shows the "Purchase Extra Active Flows" dialog instead of publishing — this is what breaks the `webhook-should-return-response` e2e monitor. Stuck flows are functionally dead (`preDelete` disables the trigger before the failing delete), so forcing their rows away is safe. Fixes on `fix/flow-delete-sample-data-timeout`: a partial expression index `idx_file_sample_data_flow_id` on `file (type, (metadata->>'flowId'))`, plus `operationStatus != DELETING` in the active-flow counts so the quota stops depending on delete-job success.+- **Flows stuck in `DELETING` keep eating the active-flow limit.** Deletion is a durable BullMQ system job (`delete-flow-<flowId>`), not synchronous: `delete()` sets `operationStatus=DELETING` and enqueues, and the row plus `status=ENABLED` only go away when the job finishes. That job runs `sampleDataService.deleteForFlow`, whose `DELETE FROM file … metadata->>'flowId'=?` had no index — on the large prod `file` table it seq-scans, blows `statement_timeout`, exhausts its 2 attempts and lands **permanently** in the failed set. The flow is then hidden from the UI list (which filters `!=DELETING`) but still counted by the active-flows quota (`getUsage` counts `status=ENABLED`), so Publish silently shows the "Purchase Extra Active Flows" dialog instead of publishing — this is what breaks the `webhook-should-return-response` e2e monitor. Fixes on `fix/flow-delete-sample-data-timeout`: a partial expression index `idx_file_sample_data_flow_id` on `file (type, (metadata->>'flowId'))`, plus `operationStatus != DELETING` in the active-flow counts so the quota stops depending on delete-job success. **Do not assume a stuck flow is functionally dead** — that was the read at the time, and it is wrong whenever the job died *before* `preDelete` ran: cloud prod held 52 stranded rows, 6 still `ENABLED`, the oldest 176 days, and the polling ones were still firing production runs (GIT-1872 / Pylon #5899). GIT-1872 made the state recoverable instead of terminal: `delete()` writes `status=DISABLED` with `DELETING` and invalidates the execution cache, `delete()` accepts a flow already in `DELETING` so a retry re-enqueues (`upsertJob` already retries a failed job and adds a missing one), and a 15-minute `STRANDED_FLOW_DELETION_SWEEP` re-enqueues rows older than 15 minutes — which is the only mechanism that recovers a row whose redis job is gone, since `removeOnFail: { age: ONE_MONTH }` garbage-collects the failed job and no `attempts` value can outlive that.+- **`FlowStatus` does not gate the polling path — `trigger_source` does, and only the delete job removes it.** `executePollingJob` and `renewWebhookJob` never read `flow.status`; their sole gate is `zombiePollingInterceptor.preDispatch`, which allows the job iff a non-soft-deleted `trigger_source` row exists for the `flowVersionId`. That row is soft-deleted inside `flowSideEffects.preDelete`, which runs *inside* the `DELETE_FLOW` job — so for a flow whose delete job is stranded, writing `status=DISABLED` buys nothing and it keeps creating PRODUCTION runs via `worker-rpc-service.submitPayloads` → `flowRunService.start`, neither of which checks a status either. The webhook family *is* gated on `FlowStatus` (`webhook.service.ts`, `app-event-routing.module.ts`), which is why "disable it and the runs stop" reads as true until you try it on a schedule trigger. GIT-1872 added the `operationStatus = DELETING` check to the interceptor rather than to `flowRunService.start`, to keep the per-run hot path free of an extra query.- **The flows list page (`automations/index.tsx`) fetches a capped window and sorts it client-side, so any new "sort by X" has to be pushed into SQL to be correct.** `use-automations-data.ts` calls `flowsApi.list`/`tablesApi.list` with `limit: 1000` (1500 for folder contents, one budget shared across *all* folders) and passes `cursor: undefined` at every call site; all merging, sorting and page-slicing then happens client-side in `automations/lib/utils.ts` (`mergeAndSortItems`, `buildTreeItems`, `buildFilteredTreeItems`, plus a second folder-children sort inside `buildFilteredTreeItems` that is easy to miss). Because the window is capped, re-sorting it in the browser only reorders a *recency-biased slice* — a rarely-touched flow at row 1001 by `updated` never reaches the client, so it can never appear under A–Z. Ordering therefore has to happen before the `LIMIT`, in Postgres.- **The cursor `Paginator` can be made to order by a joined column without denormalizing anything — `addOrderBy` registered on the query builder *before* `paginate()` leads.** `appendPagingQuery` does `new SelectQueryBuilder(builder)` and then only ever calls `addOrderBy`, so a caller's own criteria comes first and the paginator's `status/updated/id` become the tiebreak chain. That means `queryBuilder.addSelect('LOWER(COALESCE(latest_version."displayName", ''))', alias).addOrderBy(alias, order)` sorts flows by name with **no new column, no migration, no index, no paginator change** — despite `OrderByConfig.field` itself being unable to target a join (`buildOrder` and `buildCursorQuery` both hard-prefix `${this.alias}.`). Three things to respect: (1) the keyset `WHERE` is still built from `status/updated/id`, so this is only sound while no cursor is passed — reject `sortBy` + `cursor` with a 400 and return `createPage(data, null)` so no meaningless `next` is minted; (2) the sort alias must be **all-lowercase**, because TypeORM's DISTINCT path re-emits the order criteria *unquoted* in its second `id IN (...)` query, so a mixed-case alias fails only on that second query; (3) that same DISTINCT path (`SelectQueryBuilder` clears the inner ORDER BY and nulls the inner limit) means **no index on `flow` can serve this ORDER BY** — which is exactly why denormalizing a sort column onto `FlowEntity` buys nothing here.- **Any query that reads `flow_version.trigger` across a whole platform is TOAST-bound, not index-bound.** Trigger blobs are jsonb and TOAST'd on rows past ~2 KB, so a report that walks piece steps across all published flows on a platform pays ~2–5 ms of TOAST fetch + JSON parse per flow no matter how tight the WHERE clause is — a 10k-published-flow platform lands at 30 s–1 min. Endpoints of this shape (`platform/pieces-report/pieces-report.controller.ts` is the current example) must page + stream (`Readable.from(async iterable)`) so memory is bounded even when wall clock isn't; the escape hatch for the biggest platforms is a background job with async delivery, unlocked when a real sync request times out. Postgres jsonpath in-DB is not a shortcut — it still reads the whole TOASTed blob and the branch schema (`nextAction`/`children.*`/`onSuccessAction`/`firstLoopAction`/router children) drifts every time the flow shape changes, which is what `flowStructureUtil.getAllSteps` is authoritative over.diff --git a/packages/server/api/src/app/database/migration/postgres/1855000000000-FlowExternalIdUniqueIgnoresDeleting.ts b/packages/server/api/src/app/database/migration/postgres/1855000000000-FlowExternalIdUniqueIgnoresDeleting.tsnew file mode 100644index 00000000000..8b5120c8ff1--- /dev/null+++ b/packages/server/api/src/app/database/migration/postgres/1855000000000-FlowExternalIdUniqueIgnoresDeleting.ts@@ -0,0 +1,49 @@+import { QueryRunner } from 'typeorm'+import { system } from '../../../helper/system/system'+import { AppSystemProp } from '../../../helper/system/system-props'+import { DatabaseType } from '../../database-type'+import { Migration } from '../../migration'++export class FlowExternalIdUniqueIgnoresDeleting1855000000000 implements Migration {+ name = 'FlowExternalIdUniqueIgnoresDeleting1855000000000'+ breaking = false+ release = '0.92.0'+ transaction = false++ public async up(queryRunner: QueryRunner): Promise<void> {+ const concurrently = isPGlite() ? '' : 'CONCURRENTLY '+ await queryRunner.query('DROP INDEX IF EXISTS "idx_flow_project_id_external_id_partial"')+ await queryRunner.query(`+ CREATE UNIQUE INDEX ${concurrently}"idx_flow_project_id_external_id_partial"+ ON "flow" ("projectId", "externalId")+ WHERE "operationStatus" != 'DELETING'+ `)+ await queryRunner.query('DROP INDEX IF EXISTS "idx_flow_project_id_external_id"')+ await queryRunner.query('ALTER INDEX "idx_flow_project_id_external_id_partial" RENAME TO "idx_flow_project_id_external_id"')+ }++ public async down(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query(`+ UPDATE "flow"+ SET "externalId" = "id"+ WHERE "id" IN (+ SELECT "id" FROM (+ SELECT "id",+ ROW_NUMBER() OVER (+ PARTITION BY "projectId", "externalId"+ ORDER BY "updated" DESC, "created" DESC, "id" DESC+ ) AS rn+ FROM "flow"+ ) ranked+ WHERE ranked.rn > 1+ )+ `)+ await queryRunner.query('DROP INDEX IF EXISTS "idx_flow_project_id_external_id"')+ await queryRunner.query(`+ CREATE UNIQUE INDEX IF NOT EXISTS "idx_flow_project_id_external_id"+ ON "flow" ("projectId", "externalId")+ `)+ }+}++const isPGlite = (): boolean => system.get(AppSystemProp.DB_TYPE) === DatabaseType.PGLITEdiff --git a/packages/server/api/src/app/database/migration/postgres/1856000000000-AddFlowTombstoneIndex.ts b/packages/server/api/src/app/database/migration/postgres/1856000000000-AddFlowTombstoneIndex.tsnew file mode 100644index 00000000000..7643151627a--- /dev/null+++ b/packages/server/api/src/app/database/migration/postgres/1856000000000-AddFlowTombstoneIndex.ts@@ -0,0 +1,27 @@+import { QueryRunner } from 'typeorm'+import { system } from '../../../helper/system/system'+import { AppSystemProp } from '../../../helper/system/system-props'+import { DatabaseType } from '../../database-type'+import { Migration } from '../../migration'++export class AddFlowTombstoneIndex1856000000000 implements Migration {+ name = 'AddFlowTombstoneIndex1856000000000'+ breaking = false+ release = '0.92.0'+ transaction = false++ public async up(queryRunner: QueryRunner): Promise<void> {+ const concurrently = isPGlite() ? '' : 'CONCURRENTLY '+ await queryRunner.query(`+ CREATE INDEX ${concurrently}IF NOT EXISTS "idx_flow_deleting_updated"+ ON "flow" ("updated")+ WHERE "operationStatus" = 'DELETING'+ `)+ }++ public async down(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query('DROP INDEX IF EXISTS "idx_flow_deleting_updated"')+ }+}++const isPGlite = (): boolean => system.get(AppSystemProp.DB_TYPE) === DatabaseType.PGLITEdiff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.tsindex 063b86280d7..1a62566fef1 100644--- a/packages/server/api/src/app/database/postgres-connection.ts+++ b/packages/server/api/src/app/database/postgres-connection.ts@@ -440,6 +440,8 @@ import { AddMcpActivity1844000000000 } from './migration/postgres/1844000000000-import { NullOrphanFolderReferences1846000000000 } from './migration/postgres/1846000000000-NullOrphanFolderReferences'import { AddAgentConversationAiCredits1850000000000 } from './migration/postgres/1850000000000-AddAgentConversationAiCredits'import { AddMcpOAuthCodeNonce1851000000000 } from './migration/postgres/1851000000000-AddMcpOAuthCodeNonce'+import { FlowExternalIdUniqueIgnoresDeleting1855000000000 } from './migration/postgres/1855000000000-FlowExternalIdUniqueIgnoresDeleting'+import { AddFlowTombstoneIndex1856000000000 } from './migration/postgres/1856000000000-AddFlowTombstoneIndex'const getSslConfig = (): boolean | TlsOptions => {const useSsl = system.get(AppSystemProp.POSTGRES_USE_SSL)@@ -895,6 +897,8 @@ export const getMigrations = (): (new () => Migration)[] => {NullOrphanFolderReferences1846000000000,AddAgentConversationAiCredits1850000000000,AddMcpOAuthCodeNonce1851000000000,+ FlowExternalIdUniqueIgnoresDeleting1855000000000,+ AddFlowTombstoneIndex1856000000000,]return migrations}diff --git a/packages/server/api/src/app/file/file.module.ts b/packages/server/api/src/app/file/file.module.tsindex a34d0956e35..f0f9aef29da 100644--- a/packages/server/api/src/app/file/file.module.ts+++ b/packages/server/api/src/app/file/file.module.ts@@ -2,6 +2,7 @@ import { FileType } from '@activepieces/shared'import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'import { entitiesMustBeOwnedByCurrentProject } from '../authentication/authorization'import { agentRetention } from '../ee/agent/agent-retention'+import { flowBackgroundJobs } from '../flows/flow/flow.jobs'import { SystemJobName } from '../helper/system-jobs/common'import { systemJobHandlers } from '../helper/system-jobs/job-handlers'import { systemJobsSchedule } from '../helper/system-jobs/system-job'@@ -15,6 +16,7 @@ export const fileModule: FastifyPluginAsyncZod = async (app) => {await fileService(app.log).deleteStaleBulk([FileType.FLOW_RUN_LOG, FileType.FLOW_RUN_LOG_SLICE, FileType.FLOW_STEP_FILE, FileType.TRIGGER_EVENT_FILE, FileType.TRIGGER_PAYLOAD, FileType.WEBHOOK_PAYLOAD, FileType.MCP_CALL_PAYLOAD])await agentRetention(app.log).deleteStaleFlowStepConversations()await mcpActivityService(app.log).deleteStale()+ await flowBackgroundJobs(app.log).reapTombstonedFlows()})await systemJobsSchedule(app.log).upsertJob({job: {diff --git a/packages/server/api/src/app/flows/flow/flow-service-side-effects.ts b/packages/server/api/src/app/flows/flow/flow-service-side-effects.tsindex 19f368c5d21..a3d891e2b9e 100644--- a/packages/server/api/src/app/flows/flow/flow-service-side-effects.ts+++ b/packages/server/api/src/app/flows/flow/flow-service-side-effects.ts@@ -38,10 +38,7 @@ export const flowSideEffects = (log: FastifyBaseLogger) => ({},async preDelete({ flowToDelete }: PreDeleteParams): Promise<void> {- if (- flowToDelete.status === FlowStatus.DISABLED ||- isNil(flowToDelete.publishedVersionId)- ) {+ if (isNil(flowToDelete.publishedVersionId)) {return}await triggerSourceService(log).disable({diff --git a/packages/server/api/src/app/flows/flow/flow.controller.ts b/packages/server/api/src/app/flows/flow/flow.controller.tsindex 9edc131f37d..f4812adf393 100644--- a/packages/server/api/src/app/flows/flow/flow.controller.ts+++ b/packages/server/api/src/app/flows/flow/flow.controller.ts@@ -145,6 +145,7 @@ export const flowController: FastifyPluginAsyncZod = async (app) => {const flow = await flowService(request.log).getOnePopulatedOrThrow({id: request.params.id,projectId: request.projectId,+ includeDeleting: true,})await gitRepoService(request.log).onDeleted({type: GitPushOperationType.DELETE_FLOW,diff --git a/packages/server/api/src/app/flows/flow/flow.entity.ts b/packages/server/api/src/app/flows/flow/flow.entity.tsindex 475f3ae8d68..17cbd372f99 100644--- a/packages/server/api/src/app/flows/flow/flow.entity.ts+++ b/packages/server/api/src/app/flows/flow/flow.entity.ts@@ -105,6 +105,12 @@ export const FlowEntity = new EntitySchema<FlowSchema>({name: 'idx_flow_project_id_external_id',columns: ['projectId', 'externalId'],unique: true,+ where: `"operationStatus" != '${FlowOperationStatus.DELETING}'`,+ },+ {+ name: 'idx_flow_deleting_updated',+ columns: ['updated'],+ where: `"operationStatus" = '${FlowOperationStatus.DELETING}'`,},],relations: {diff --git a/packages/server/api/src/app/flows/flow/flow.jobs.ts b/packages/server/api/src/app/flows/flow/flow.jobs.tsindex 987f48768dc..2fe8dbfb2b9 100644--- a/packages/server/api/src/app/flows/flow/flow.jobs.ts+++ b/packages/server/api/src/app/flows/flow/flow.jobs.ts@@ -1,4 +1,5 @@import { assertNotNullOrUndefined } from '@activepieces/core-utils'+import { FlowOperationStatus } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { repoFactory } from '../../core/db/repo-factory'import { SystemJobData, SystemJobName } from '../../helper/system-jobs/common'@@ -9,10 +10,12 @@ import { flowVersionRepo } from '../flow-version/flow-version.service'import { flowExecutionCache } from './flow-execution-cache'import { flowSideEffects } from './flow-service-side-effects'import { flowRepo } from './flow.repo'+import { flowService } from './flow.service'const waitpointRepo = repoFactory(WaitpointEntity)const BATCH_SIZE = 1000+const TOMBSTONE_REAP_LIMIT = 50export async function batchDeleteByFlowId(flowId: string): Promise<void> {while (true) {@@ -75,4 +78,17 @@ export const flowBackgroundJobs = (log: FastifyBaseLogger) => ({await flowExecutionCache(log).invalidate(flow.id)},-})\ No newline at end of file+ reapTombstonedFlows: async () => {+ const tombstoned = await flowRepo().find({+ where: { operationStatus: FlowOperationStatus.DELETING },+ order: { updated: 'ASC' },+ take: TOMBSTONE_REAP_LIMIT,+ })+ if (tombstoned.length === 0) {+ return+ }+ log.warn({ flowCount: tombstoned.length }, '[reapTombstonedFlows] Re-queueing deletions that never finished')+ await Promise.all(tombstoned.map((flow) => flowService(log).addDeleteFlowJob(flow)))+ },++})diff --git a/packages/server/api/src/app/flows/flow/flow.service.ts b/packages/server/api/src/app/flows/flow/flow.service.tsindex 97a068aa05b..8a5af5b4700 100644--- a/packages/server/api/src/app/flows/flow/flow.service.ts+++ b/packages/server/api/src/app/flows/flow/flow.service.ts@@ -29,6 +29,8 @@ import { flowRepo } from './flow.repo'+const notDeleting = { operationStatus: Not(FlowOperationStatus.DELETING) }+export const flowService = (log: FastifyBaseLogger) => ({async create({ projectId, request, externalId, ownerId, templateId, createdBy, ip, emitEvents = true }: CreateParams): Promise<PopulatedFlow> {await assertExternalIdIsUnique({ projectId, externalId })@@ -121,7 +123,7 @@ export const flowService = (log: FastifyBaseLogger) => ({},})- const queryBuilder = flowRepo().createQueryBuilder('ff').where({ operationStatus: Not(FlowOperationStatus.DELETING) })+ const queryBuilder = flowRepo().createQueryBuilder('ff').where(notDeleting)if (projectIds) {queryBuilder.andWhere({ projectId: In(projectIds) })@@ -234,11 +236,13 @@ export const flowService = (log: FastifyBaseLogger) => ({return flowRepo().existsBy({id,projectId,+ ...notDeleting,})},async getOneById(id: string): Promise<Flow | null> {const flow = await flowRepo().findOneBy({id,+ ...notDeleting,})if (isNil(flow)) {return null@@ -251,7 +255,7 @@ export const flowService = (log: FastifyBaseLogger) => ({}return flow},- async getOne({ id, projectId, entityManager }: GetOneParams): Promise<Flow | null> {+ async getOne({ id, projectId, entityManager, includeDeleting = false }: GetOneParams): Promise<Flow | null> {const projectExists = await projectService(log).exists({projectId,})@@ -261,6 +265,7 @@ export const flowService = (log: FastifyBaseLogger) => ({return flowRepo(entityManager).findOneBy({id,projectId,+ ...(includeDeleting ? {} : notDeleting),})},@@ -277,18 +282,10 @@ export const flowService = (log: FastifyBaseLogger) => ({removeConnectionsName = false,removeSampleData = false,entityManager,+ includeDeleting = false,}: GetOnePopulatedParams): Promise<PopulatedFlow | null> {- const flow = await flowRepo(entityManager).findOne({- where: {- id,- projectId,- },- })-- const projectExists = await projectService(log).exists({- projectId,- })- if (isNil(flow) || !projectExists) {+ const flow = await this.getOne({ id, projectId, entityManager, includeDeleting })+ if (isNil(flow)) {return null}@@ -316,22 +313,8 @@ export const flowService = (log: FastifyBaseLogger) => ({}},- async getOnePopulatedOrThrow({- id,- projectId,- versionId,- removeConnectionsName = false,- removeSampleData = false,- entityManager,- }: GetOnePopulatedParams): Promise<PopulatedFlow> {- const flow = await this.getOnePopulated({- id,- projectId,- versionId,- removeConnectionsName,- removeSampleData,- entityManager,- })+ async getOnePopulatedOrThrow(params: GetOnePopulatedParams): Promise<PopulatedFlow> {+ const flow = await this.getOnePopulated(params)assertFlowIsNotNull(flow)return flow@@ -357,14 +340,6 @@ export const flowService = (log: FastifyBaseLogger) => ({id,projectId,})- if (flow.operationStatus === FlowOperationStatus.DELETING) {- throw new ActivepiecesError({- code: ErrorCode.FLOW_OPERATION_IN_PROGRESS,- params: {- message: 'This flow is getting deleted.',- },- })- }if (operation.type === FlowOperationType.LOCK_AND_PUBLISH && flow.status === FlowStatus.ENABLED && !isNil(flow.publishedVersionId)) {previouslyPublishedVersion = await flowVersionService(log).getFlowVersionOrThrow({ flowId: id, versionId: flow.publishedVersionId })}@@ -570,24 +545,19 @@ export const flowService = (log: FastifyBaseLogger) => ({async delete({ id, projectId, previousFlow, userId, ip, emitEvents = true }: DeleteParams): Promise<void> {const deletedFlow = emitEvents- ? previousFlow ?? await this.getOnePopulatedOrThrow({ id, projectId })+ ? previousFlow ?? await this.getOnePopulatedOrThrow({ id, projectId, includeDeleting: true }): undefinedconst flow = await this.getOneOrThrow({id,projectId,+ includeDeleting: true,})- if (flow.operationStatus !== FlowOperationStatus.NONE) {- throw new ActivepiecesError({- code: ErrorCode.FLOW_OPERATION_IN_PROGRESS,- params: {- message: `Flow ${id} is already being ${flow.operationStatus}`,- },- })- }- await this.addDeleteFlowJob(flow)- await flowRepo().update(id, {+ await flowRepo().update({ id, projectId }, {+ status: FlowStatus.DISABLED,operationStatus: FlowOperationStatus.DELETING,})+ await flowExecutionCache(log).invalidate(id)+ await this.addDeleteFlowJob(flow)log.info({ flow: { id }, project: { id: projectId } }, 'Flow deletion requested')if (!isNil(deletedFlow)) {flowSideEffects(log).onDeleted({@@ -873,7 +843,7 @@ async function assertExternalIdIsUnique({ projectId, externalId }: { projectId:if (isNil(externalId)) {return}- const exists = await flowRepo().existsBy({ projectId, externalId })+ const exists = await flowRepo().existsBy({ projectId, externalId, ...notDeleting })if (exists) {throw new ActivepiecesError({code: ErrorCode.FLOW_EXTERNAL_ID_ALREADY_EXISTS,@@ -916,6 +886,7 @@ type GetOneParams = {id: FlowIdprojectId: ProjectIdentityManager?: EntityManager+ includeDeleting?: boolean}type GetOnePopulatedParams = GetOneParams & {diff --git a/packages/server/api/src/app/workers/job-queue/interceptors/zombie-polling-interceptor.ts b/packages/server/api/src/app/workers/job-queue/interceptors/zombie-polling-interceptor.tsindex 9b382c23e5f..a0d17b3c976 100644--- a/packages/server/api/src/app/workers/job-queue/interceptors/zombie-polling-interceptor.ts+++ b/packages/server/api/src/app/workers/job-queue/interceptors/zombie-polling-interceptor.ts@@ -1,6 +1,7 @@import { isNil } from '@activepieces/core-utils'-import { JobData, PollingJobData, RenewWebhookJobData, WorkerJobType } from '@activepieces/shared'+import { FlowOperationStatus, JobData, PollingJobData, RenewWebhookJobData, WorkerJobType } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'+import { flowRepo } from '../../../flows/flow/flow.repo'import { triggerSourceRepo } from '../../../trigger/trigger-source/trigger-source-service'import { InterceptorResult, InterceptorVerdict, JobInterceptor } from '../job-interceptor'import { jobQueue } from '../job-queue'@@ -12,14 +13,16 @@ export const zombiePollingInterceptor: JobInterceptor = {if (!ZOMBIE_REPEATING_JOB_TYPES.includes(jobData.jobType)) {return { verdict: InterceptorVerdict.ALLOW }}- const { flowVersionId } = jobData as PollingJobData | RenewWebhookJobData+ const { flowId, flowVersionId } = jobData as PollingJobData | RenewWebhookJobData// An active trigger source exists only when the flow is enabled and this exact version is current.// If soft-deleted (disabled or re-published to a new version), findOneBy returns null.const activeTriggerSource = await triggerSourceRepo().findOneBy({ flowVersionId })- if (!isNil(activeTriggerSource)) {+ const flowIsBeingDeleted = await flowRepo().existsBy({ id: flowId, operationStatus: FlowOperationStatus.DELETING })+ const triggerIsStillLive = !isNil(activeTriggerSource) && !flowIsBeingDeleted+ if (triggerIsStillLive) {return { verdict: InterceptorVerdict.ALLOW }}- log.warn({ flowVersion: { id: flowVersionId } }, '[zombiePollingInterceptor] No active trigger source — discarding repeat job (flow disabled, re-published, or deleted)')+ log.warn({ flow: { id: flowId }, flowVersion: { id: flowVersionId } }, '[zombiePollingInterceptor] No live trigger source — discarding repeat job (flow disabled, re-published, or deleted)')await jobQueue(log).removeRepeatingJob({ flowVersionId })return { verdict: InterceptorVerdict.DISCARD }},diff --git a/packages/server/api/test/integration/ce/flows/flow/flow-delete-recovery.test.ts b/packages/server/api/test/integration/ce/flows/flow/flow-delete-recovery.test.tsnew file mode 100644index 00000000000..945a0701e0d--- /dev/null+++ b/packages/server/api/test/integration/ce/flows/flow/flow-delete-recovery.test.ts@@ -0,0 +1,254 @@+import { apId } from '@activepieces/core-utils'+import { TriggerStrategy } from '@activepieces/pieces-framework'+import { Flow, FlowOperationStatus, FlowStatus, FlowTriggerType, LATEST_JOB_DATA_SCHEMA_VERSION, WorkerJobType } from '@activepieces/shared'+import dayjs from 'dayjs'+import { Job } from 'bullmq'+import { FastifyInstance } from 'fastify'+import { StatusCodes } from 'http-status-codes'+import { flowSideEffects } from '../../../../../src/app/flows/flow/flow-service-side-effects'+import { flowBackgroundJobs } from '../../../../../src/app/flows/flow/flow.jobs'+import { InterceptorVerdict } from '../../../../../src/app/workers/job-queue/job-interceptor'+import { zombiePollingInterceptor } from '../../../../../src/app/workers/job-queue/interceptors/zombie-polling-interceptor'+import { db } from '../../../../helpers/db'+import { createMockFlow, createMockFlowVersion } from '../../../../helpers/mocks'+import { createTestContext, TestContext } from '../../../../helpers/test-context'+import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'++let app: FastifyInstance | null = null++beforeAll(async () => {+ app = await setupTestEnvironment()+})++afterAll(async () => {+ await teardownTestEnvironment()+})++async function savePublishedFlow(ctx: TestContext, flow?: Partial<Flow>): Promise<Flow> {+ const mockFlow = createMockFlow({+ projectId: ctx.project.id,+ status: FlowStatus.ENABLED,+ ...flow,+ })+ await db.save('flow', mockFlow)+ const mockVersion = createMockFlowVersion({ flowId: mockFlow.id })+ await db.save('flow_version', mockVersion)+ await db.update('flow', mockFlow.id, { publishedVersionId: mockVersion.id })+ return mockFlow+}++async function waitForRowToDisappear(flowId: string): Promise<Flow | null> {+ for (let attempt = 0; attempt < 40; attempt++) {+ const row = await db.findOneBy<Flow>('flow', { id: flowId })+ if (row === null) {+ return null+ }+ await new Promise(resolve => setTimeout(resolve, 100))+ }+ return db.findOneBy<Flow>('flow', { id: flowId })+}++async function savePollingFlowWithLiveTrigger(ctx: TestContext, flow?: Partial<Flow>): Promise<{ flowId: string, flowVersionId: string }> {+ const mockFlow = createMockFlow({ projectId: ctx.project.id, status: FlowStatus.ENABLED, ...flow })+ await db.save('flow', mockFlow)+ const mockVersion = createMockFlowVersion({ flowId: mockFlow.id })+ await db.save('flow_version', mockVersion)+ await db.update('flow', mockFlow.id, { publishedVersionId: mockVersion.id })+ await db.save('trigger_source', {+ id: apId(),+ created: new Date().toISOString(),+ updated: new Date().toISOString(),+ flowId: mockFlow.id,+ flowVersionId: mockVersion.id,+ projectId: ctx.project.id,+ pieceName: '@activepieces/piece-schedule',+ pieceVersion: '0.1.0',+ triggerName: 'every_hour',+ type: TriggerStrategy.POLLING,+ simulate: false,+ schedule: null,+ deleted: null,+ })+ return { flowId: mockFlow.id, flowVersionId: mockVersion.id }+}++async function dispatchPollingJob(ctx: TestContext, flowId: string, flowVersionId: string): Promise<InterceptorVerdict> {+ const result = await zombiePollingInterceptor.preDispatch({+ jobId: apId(),+ jobData: {+ jobType: WorkerJobType.EXECUTE_POLLING,+ projectId: ctx.project.id,+ platformId: ctx.platform.id,+ schemaVersion: LATEST_JOB_DATA_SCHEMA_VERSION,+ flowId,+ flowVersionId,+ triggerType: FlowTriggerType.PIECE,+ },+ job: {} as Job,+ log: app!.log,+ })+ return result.verdict+}++describe('Flow deletion recovery', () => {+ it('disables the flow before the delete job runs, so its trigger stops firing', async () => {+ const ctx = await createTestContext(app!)+ const flow = await savePublishedFlow(ctx)++ const response = await ctx.delete(`/v1/flows/${flow.id}`)++ expect(response?.statusCode).toBe(StatusCodes.NO_CONTENT)+ const row = await db.findOneBy<Flow>('flow', { id: flow.id })+ expect(row?.status ?? FlowStatus.DISABLED).toBe(FlowStatus.DISABLED)+ })++ it('accepts a second delete for a flow already stranded in DELETING', async () => {+ const ctx = await createTestContext(app!)+ const flow = await savePublishedFlow(ctx, { operationStatus: FlowOperationStatus.DELETING })++ const response = await ctx.delete(`/v1/flows/${flow.id}`)++ expect(response?.statusCode).toBe(StatusCodes.NO_CONTENT)+ expect(await waitForRowToDisappear(flow.id)).toBeNull()+ })++ it('reaps a flow whose delete job was lost, without a user asking again', async () => {+ const ctx = await createTestContext(app!)+ const flow = await savePublishedFlow(ctx, { operationStatus: FlowOperationStatus.DELETING })+ await db.update('flow', flow.id, { updated: dayjs().subtract(1, 'day').toISOString() })++ await flowBackgroundJobs(app!.log).reapTombstonedFlows()++ expect(await waitForRowToDisappear(flow.id)).toBeNull()+ })++ it('discards the polling job of a flow being deleted, so it stops firing runs', async () => {+ const ctx = await createTestContext(app!)+ const { flowId, flowVersionId } = await savePollingFlowWithLiveTrigger(ctx, { operationStatus: FlowOperationStatus.DELETING })++ expect(await dispatchPollingJob(ctx, flowId, flowVersionId)).toBe(InterceptorVerdict.DISCARD)+ })++ it('still dispatches the polling job of a flow that is not being deleted', async () => {+ const ctx = await createTestContext(app!)+ const { flowId, flowVersionId } = await savePollingFlowWithLiveTrigger(ctx)++ expect(await dispatchPollingJob(ctx, flowId, flowVersionId)).toBe(InterceptorVerdict.ALLOW)+ })++ it('tears the trigger source down even though delete() already wrote DISABLED', async () => {+ const ctx = await createTestContext(app!)+ const { flowId } = await savePollingFlowWithLiveTrigger(ctx, { status: FlowStatus.DISABLED })+ const flowToDelete = await db.findOneByOrFail<Flow>('flow', { id: flowId })++ await flowSideEffects(app!.log).preDelete({ flowToDelete })++ const triggerSource = await db.findOneBy<{ deleted: string | null }>('trigger_source', { flowId })+ expect(triggerSource).toBeNull()+ })++ it('keeps reaping the rest when one flow cannot be reclaimed', async () => {+ const ctx = await createTestContext(app!)+ const doomed = createMockFlow({+ projectId: ctx.project.id,+ status: FlowStatus.DISABLED,+ operationStatus: FlowOperationStatus.DELETING,+ })+ await db.save('flow', doomed)+ const doomedVersion = createMockFlowVersion({ flowId: doomed.id })+ await db.save('flow_version', doomedVersion)+ await db.update('flow', doomed.id, { publishedVersionId: doomedVersion.id })+ await db.save('trigger_source', {+ id: apId(),+ created: new Date().toISOString(),+ updated: new Date().toISOString(),+ flowId: doomed.id,+ flowVersionId: apId(),+ projectId: ctx.project.id,+ pieceName: '@activepieces/piece-schedule',+ pieceVersion: '0.1.0',+ triggerName: 'every_hour',+ type: TriggerStrategy.POLLING,+ simulate: false,+ schedule: null,+ })+ const healthy = await savePublishedFlow(ctx, { operationStatus: FlowOperationStatus.DELETING })+ await db.update('flow', doomed.id, { updated: dayjs().subtract(2, 'hour').toISOString() })+ await db.update('flow', healthy.id, { updated: dayjs().subtract(2, 'hour').toISOString() })++ await flowBackgroundJobs(app!.log).reapTombstonedFlows()++ expect(await waitForRowToDisappear(healthy.id)).toBeNull()+ expect(await db.findOneBy<Flow>('flow', { id: doomed.id })).not.toBeNull()+ })++ it('is gone to every reader the moment the request returns, not when the job lands', async () => {+ const ctx = await createTestContext(app!)+ const flow = await savePublishedFlow(ctx, { operationStatus: FlowOperationStatus.DELETING })++ const opened = await ctx.get(`/v1/flows/${flow.id}`)+ const listed = await ctx.get('/v1/flows', { projectId: ctx.project.id })++ expect(opened?.statusCode).toBe(StatusCodes.NOT_FOUND)+ expect(listed?.json().data.map((f: Flow) => f.id)).not.toContain(flow.id)+ })++ it('frees the external id so the flow can be recreated while the tombstone lingers', async () => {+ const ctx = await createTestContext(app!)+ const flow = await savePublishedFlow(ctx, { operationStatus: FlowOperationStatus.DELETING })++ const response = await ctx.post('/v1/flows', {+ displayName: 'recreated',+ projectId: ctx.project.id,+ externalId: flow.externalId,+ }, { query: { projectId: ctx.project.id } })++ expect(response?.statusCode).toBe(StatusCodes.CREATED)+ expect(response?.json().externalId).toBe(flow.externalId)+ })++ it('survives concurrent deletes and a reaper pass racing the same flow', async () => {+ const ctx = await createTestContext(app!)+ const flow = await savePublishedFlow(ctx)++ const responses = await Promise.all([+ ctx.delete(`/v1/flows/${flow.id}`),+ ctx.delete(`/v1/flows/${flow.id}`),+ ctx.delete(`/v1/flows/${flow.id}`),+ flowBackgroundJobs(app!.log).reapTombstonedFlows(),+ ])++ for (const response of responses.slice(0, 3)) {+ expect(response).not.toBeUndefined()+ expect([StatusCodes.NO_CONTENT, StatusCodes.NOT_FOUND]).toContain((response as { statusCode: number }).statusCode)+ }+ expect(await waitForRowToDisappear(flow.id)).toBeNull()+ expect(await db.findBy('flow_version', { flowId: flow.id })).toHaveLength(0)+ expect(await db.findBy('trigger_source', { flowId: flow.id })).toHaveLength(0)+ })++ it('is safe to tear the trigger source down twice', async () => {+ const ctx = await createTestContext(app!)+ const { flowId } = await savePollingFlowWithLiveTrigger(ctx, { status: FlowStatus.DISABLED })+ const flowToDelete = await db.findOneByOrFail<Flow>('flow', { id: flowId })++ await flowSideEffects(app!.log).preDelete({ flowToDelete })+ const afterFirst = await db.findBy('trigger_source', { flowId })++ await flowSideEffects(app!.log).preDelete({ flowToDelete })+ const afterSecond = await db.findBy('trigger_source', { flowId })++ expect(afterFirst).toHaveLength(0)+ expect(afterSecond).toHaveLength(0)+ })++ it('reclaims a just-requested deletion rather than waiting on a clock', async () => {+ const ctx = await createTestContext(app!)+ const flow = await savePublishedFlow(ctx, { operationStatus: FlowOperationStatus.DELETING })++ await flowBackgroundJobs(app!.log).reapTombstonedFlows()+ await flowBackgroundJobs(app!.log).reapTombstonedFlows()++ expect(await waitForRowToDisappear(flow.id)).toBeNull()+ })+})