Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(flows): polling triggers no longer skip new items after a republish (#16742)

Hazem Adel635e1661a2ff
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index 82bfc5ca36d..a5d63d3730e 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -21,6 +21,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the
- `concurrency === 1` primes the sandbox to the full container RAM (cgroup-aware) via `primeFullContainerMemory()`.
### Gotchas
+- **A local engine edit doesn't reach sandboxes until the worker restarts, unless `AP_ENVIRONMENT` is dev.** `engineInstaller` copies `dist/packages/engine/main.js` into `cache/v<N>/common/main.js` once per worker process (`installedPaths`) and only re-copies on every install in `DEVELOPMENT`. With a prod-like local env, sandboxes keep running whatever engine was built when the worker first ran a job, and killing sandboxes doesn't help. Restart the worker (touching `packages/server/worker/src/bootstrap.ts` makes `tsx watch` do it), then check the cached `main.js` for your change before testing.
- **Piece bundles download over the worker's own `AP_FRONTEND_URL` (`internalApiUrl`), never the app's `PUBLIC_URL` — keep it that way.** Through 0.92.x the sandbox built `GET /v1/engine/pieces/bundle` from the `PUBLIC_URL` the app pushes over the socket, so a standalone worker ignored its own `AP_FRONTEND_URL=http://app` for this one hop. A `localhost` public URL (the docs' default for compose and `docker run -p 8080:80`) failed every uncached install with `fetch failed` (GIT-1838). A public URL that reached a *different* instance sharing the JWT secret returned 404, so custom-piece upload failed with `PIECE_BUNDLE_NOT_AVAILABLE` while the `PACKAGE_ARCHIVE` row sat intact in Postgres. Tell-tale: the bundle GET never shows up in the app's own request log. Fixed by GIT-1718.
- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads, piece bundle downloads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.
- **The piece workspace pins `linker = "isolated"` in its own `bunfig.toml` — do not delete it, the engine's piece resolver depends on the layout bun chooses.** `pieceInstaller` generates a bun workspace at `<cache>/<version>/common` whose members are `pieces/<pieceName>-<pieceVersion>`, and the engine (`piece-loader.ts` → `resolveInstalledPieceEntry`) resolves a piece **only** at `pieces/<alias>/node_modules/<pieceName>`, which just the isolated linker produces. The workspace used to carry no bunfig and inherited whatever ancestor bunfig bun discovered above the cache mount (`/usr/src/app/bunfig.toml`); when bun instead uses the **hoisted** linker it writes the package as a real directory at the workspace root `node_modules/@activepieces/<name>` and leaves the member with no `node_modules` at all, so every run of that piece fails `PieceNotFoundError: Piece not found for package: <name>-<version>` while `bun install` still exits 0. It was silent and self-perpetuating — `markPiecesAsUsed` writes `ready` and `pieceCheckIfAlreadyInstalled` only looks for *a* `node_modules`, so the piece is reinstalled forever and never resolves. Sep 2026 production: 125/125 folders missing `node_modules` were exactly the 125 packages sitting hoisted at the workspace root; 7,400+ `EXECUTE_FLOW` jobs, 15+ platforms, ~60 official pieces, ~300/hr. Two things had to line up — bun 1.4.0 (Aug 31) changed the layout newly-installed members got, and removing the piece-upgrade Redis gate (#15236, deployed 2026-09-03 08:09 UTC) mass-upgraded every platform's flows to newer piece versions so almost every install was suddenly a *first-time* install. Diagnose on a worker with `for d in <cache>/v*/common/pieces/@activepieces/*/; do [ -d "$d/node_modules" ] || basename $d; done`. Fixed by pinning the linker plus a `LATEST_CACHE_VERSION` bump (v14 → v15) so the fleet rebuilds one clean workspace under the pinned layout rather than carrying a mixed one.
diff --git a/brain/knowledge/flows-execution/triggers.md b/brain/knowledge/flows-execution/triggers.md
index 1dad4fddc83..ccc03ae0bfb 100644
--- a/brain/knowledge/flows-execution/triggers.md
+++ b/brain/knowledge/flows-execution/triggers.md
@@ -21,8 +21,8 @@ Triggers define how and when a flow starts. The module handles registration, eve
### Gotchas
- **Deduplication** (polling): extracts `__DEDUPE_KEY_PROPERTY`, Redis INCR with 30s TTL — first passes, duplicates filtered; the dedupe key is stripped from returned payloads.
-- **Republish preserves the polling checkpoint** (`isRepublish`): republishing a running flow does `onDisable(old) → onEnable(new)`, which used to reset `lastPoll`/`lastItem` to now and silently drop events created in between. `flowService.applyStatusChangeForPublishedFlow` (the one entry point for both a direct `LOCK_AND_PUBLISH` and an EE approval) sets `isRepublish=true` only for a publish of an already-`ENABLED` flow whose trigger is unchanged — same piece, same trigger name, **and deep-equal `settings.input`** (`flowPublishUtils.isSameTrigger`); the flag is threaded through the ON_ENABLE job → `ExecuteTriggerOperation` → trigger context (`context.isRepublish`), and `pollingHelper.onEnable` then keeps the existing checkpoint. A fresh enable, a manual off→on toggle, a trigger swap, and any change to the trigger's props all still reset to now. The props check is not cosmetic: a checkpoint kept across a props change points at a resource that is no longer being polled, and `pollingHelper.poll` treats a `LAST_ITEM` id it cannot find in the fetched page (`findIndex → -1`) the same as "no checkpoint", emitting **every** item. Custom polling triggers that don't use `pollingHelper` can opt in by reading `context.isRepublish`. A new publish path must go through `applyStatusChangeForPublishedFlow` with the pre-publish flow snapshot; before ENG-938 the approval path called `applyStatusChange` itself and every approved republish reset the checkpoint.
-- **A trigger only keeps its checkpoint if it forwards `isRepublish`.** `pollingHelper.onEnable(polling, context)` gets it; an object literal like `{ auth, store, propsValue }` (RSS and roughly half of polling triggers) drops it and resets to now on every republish.
+- **Republish preserves the polling checkpoint** (`isRepublish`): republishing a running flow does `onDisable(old) → onEnable(new)`, which used to reset `lastPoll`/`lastItem` to now and silently drop events created in between. `flowService.isRepublish` decides it once, before anything is disabled: true only when an already-`ENABLED` flow is published to stay `ENABLED` with an unchanged trigger — same piece, **same piece version**, same trigger name, **and deep-equal `settings.input`** (`flowPublishUtils.isSameTrigger`). A piece upgrade counts as a change and starts fresh, because the new version may read its state differently. Both publish paths (direct `LOCK_AND_PUBLISH` and EE approval) pass that one value to the old version's ON_DISABLE job and to the new version's ON_ENABLE job → `ExecuteTriggerOperation` → trigger context (`context.isRepublish`). A fresh enable, a manual off→on toggle, a trigger swap, and any change to the trigger's props all still reset to now. The props check is not cosmetic: a checkpoint kept across a props change points at a resource that is no longer being polled, and `pollingHelper.poll` treats a `LAST_ITEM` id it cannot find in the fetched page (`findIndex → -1`) the same as "no checkpoint", emitting **every** item. A new publish path must call `flowService.isRepublish` with the pre-publish flow snapshot and pass the result to both `triggerSourceService.disable` and `applyStatusChangeForPublishedFlow`; before ENG-938 the approval path skipped it and every approved republish reset the checkpoint.
+- **The engine keeps the checkpoint, not the piece.** On a republish of a `POLLING` trigger, `republishStore.forLifecycleHook` (`packages/server/engine/src/lib/piece-context`) makes the store read-only for both the old version's `onDisable` and the new version's `onEnable`: reads work, writes and deletes do nothing, so the state ends exactly as it was. Per-operation rules were tried and each missed a piece: Twilio clears its checkpoint in `onDisable` with `put(key, null)`, Shopify's `onEnable` deletes `lastItem` and writes new scan keys, ~38 triggers delete their checkpoint in `onDisable`. Pieces don't need to forward the flag. Webhook triggers are excluded because they must remove and re-create their webhook. The cost: a cache that `onEnable` would refresh (Zoho's field list) stays until it expires on its own (6h for Zoho). A plain off toggle or a flow delete sends no flag, so `onDisable` still cleans up there.
- **A missing timestamp permanently kills a TIMEBASED polling trigger** — `pollingHelper.poll` advances the checkpoint with `items.reduce((acc, i) => Math.max(acc, i.epochMilliSeconds), lastPoll)`, and `Math.max(n, NaN)` is `NaN`, so a single item whose date field was never requested (`dayjs(undefined).valueOf()` → `NaN`) writes `NaN` into `lastPoll`; every later poll then filters on `> NaN` → false and the trigger silently never fires again, with no error anywhere. Two guards, both needed: request the date in the API's `fields`/select mask, and drop items with an unusable date before mapping to `epochMilliSeconds`. The filter must check the raw value, not just `.isValid()` — `dayjs(undefined)` is *now* and reports valid.
- **The timestamp you poll on may be client-supplied, and a future one is fatal.** Google Drive sets `modifiedTime` from the *local file mtime* on upload, not the upload moment, so it can be older than `createdTime` — or years ahead if the uploader's clock is fast. A TIMEBASED watermark is `max(epochMilliSeconds)` over the emitted items, so one future-dated row pushes `lastPoll` into the future and the trigger emits nothing until wall-clock catches up. Hold back items timestamped after `Date.now()` — they fire once the clock passes them. Do **not** clamp the watermark instead: a clamped watermark re-emits that same row on every poll forever.
- The **simulate flag** lets a production source and a test source coexist independently.
diff --git a/packages/server/api/src/app/ee/flows/flow-approval/flow-approval-request.service.ts b/packages/server/api/src/app/ee/flows/flow-approval/flow-approval-request.service.ts
index c27ae4c9c87..918eb689b93 100644
--- a/packages/server/api/src/app/ee/flows/flow-approval/flow-approval-request.service.ts
+++ b/packages/server/api/src/app/ee/flows/flow-approval/flow-approval-request.service.ts
@@ -89,6 +89,7 @@ export const flowApprovalRequestService = (log: FastifyBaseLogger) => ({
flowId: flow.id,
versionId: approval.flowVersionId,
})
+ const isRepublish = await flowService(log).isRepublish({ previousFlow: flow, versionToPublish: lockedVersion, newStatus: approval.requestedStatus })
const decidedAt = new Date().toISOString()
const approverId = approverPrincipal.type === PrincipalType.SERVICE ? null : approverPrincipal.id
@@ -115,13 +116,15 @@ export const flowApprovalRequestService = (log: FastifyBaseLogger) => ({
projectId: flow.projectId,
simulate: false,
ignoreError: true,
+ isRepublish,
})
}
await flowExecutionCache(log).invalidate(flow.id)
await flowService(log).applyStatusChangeForPublishedFlow({
- previousFlow: flow,
- publishedVersion: lockedVersion,
+ id: flow.id,
+ projectId: flow.projectId,
newStatus: approval.requestedStatus,
+ isRepublish,
})
rejectedPromiseHandler(telemetry(log).trackProject({
projectId: approval.projectId,
diff --git a/packages/server/api/src/app/flows/flow/flow-publish-utils.ts b/packages/server/api/src/app/flows/flow/flow-publish-utils.ts
index f0285c7b0c5..462b93e6c07 100644
--- a/packages/server/api/src/app/flows/flow/flow-publish-utils.ts
+++ b/packages/server/api/src/app/flows/flow/flow-publish-utils.ts
@@ -5,6 +5,7 @@ function isSameTrigger({ published, toPublish }: IsSameTriggerParams): boolean {
return published.type === FlowTriggerType.PIECE
&& toPublish.type === FlowTriggerType.PIECE
&& published.settings.pieceName === toPublish.settings.pieceName
+ && published.settings.pieceVersion === toPublish.settings.pieceVersion
&& published.settings.triggerName === toPublish.settings.triggerName
&& deepEqual(published.settings.input, toPublish.settings.input)
}
diff --git a/packages/server/api/src/app/flows/flow/flow.service.ts b/packages/server/api/src/app/flows/flow/flow.service.ts
index c21a40a6833..9a03739ff33 100644
--- a/packages/server/api/src/app/flows/flow/flow.service.ts
+++ b/packages/server/api/src/app/flows/flow/flow.service.ts
@@ -376,14 +376,16 @@ export const flowService = (log: FastifyBaseLogger) => ({
})
break
}
- const publishedFlow = await this.updatedPublishedVersionId({
+ const isRepublish = await this.isRepublish({ previousFlow: flow, versionToPublish: flowVersionToPublish, newStatus: requestedStatus })
+ await this.updatedPublishedVersionId({
id,
userId,
projectId,
platformId,
flowVersionToPublish,
+ isRepublish,
})
- await this.applyStatusChangeForPublishedFlow({ previousFlow: flow, publishedVersion: publishedFlow.version, newStatus: requestedStatus })
+ await this.applyStatusChangeForPublishedFlow({ id, projectId, newStatus: requestedStatus, isRepublish })
break
}
@@ -500,6 +502,7 @@ export const flowService = (log: FastifyBaseLogger) => ({
projectId,
platformId,
flowVersionToPublish,
+ isRepublish,
}: UpdatePublishedVersionIdParams): Promise<PopulatedFlow> {
const flowToUpdate = await this.getOneOrThrow({ id, projectId })
@@ -509,6 +512,7 @@ export const flowService = (log: FastifyBaseLogger) => ({
projectId: flowToUpdate.projectId,
simulate: false,
ignoreError: true,
+ isRepublish,
})
}
@@ -563,16 +567,21 @@ export const flowService = (log: FastifyBaseLogger) => ({
})
},
- async applyStatusChangeForPublishedFlow({ previousFlow, publishedVersion, newStatus }: ApplyStatusChangeForPublishedFlowParams): Promise<void> {
- const previousVersionId = previousFlow.status === FlowStatus.ENABLED ? previousFlow.publishedVersionId : null
- const previousVersion = isNil(previousVersionId)
- ? null
- : await flowVersionService(log).getFlowVersionOrThrow({ flowId: previousFlow.id, versionId: previousVersionId })
- const isRepublish = !isNil(previousVersion) && flowPublishUtils.isSameTrigger({
+ async isRepublish({ previousFlow, versionToPublish, newStatus }: IsRepublishParams): Promise<boolean> {
+ const previousVersionId = previousFlow.publishedVersionId
+ const keepsRunning = previousFlow.status === FlowStatus.ENABLED && newStatus === FlowStatus.ENABLED
+ if (!keepsRunning || isNil(previousVersionId)) {
+ return false
+ }
+ const previousVersion = await flowVersionService(log).getFlowVersionOrThrow({ flowId: previousFlow.id, versionId: previousVersionId })
+ return flowPublishUtils.isSameTrigger({
published: previousVersion.trigger,
- toPublish: publishedVersion.trigger,
+ toPublish: versionToPublish.trigger,
})
- await applyStatusChange({ id: previousFlow.id, projectId: previousFlow.projectId, newStatus, isRepublish }, log)
+ },
+
+ async applyStatusChangeForPublishedFlow({ id, projectId, newStatus, isRepublish }: ApplyStatusChangeForPublishedFlowParams): Promise<void> {
+ await applyStatusChange({ id, projectId, newStatus, isRepublish }, log)
},
async delete({ id, projectId, previousFlow, userId, ip, emitEvents = true }: DeleteParams): Promise<void> {
@@ -1008,6 +1017,7 @@ type UpdatePublishedVersionIdParams = {
platformId: PlatformId
projectId: ProjectId
flowVersionToPublish: FlowVersion
+ isRepublish: boolean
}
type SetPublishedVersionParams = {
@@ -1016,10 +1026,17 @@ type SetPublishedVersionParams = {
entityManager: EntityManager
}
-type ApplyStatusChangeForPublishedFlowParams = {
+type IsRepublishParams = {
previousFlow: Flow
- publishedVersion: FlowVersion
+ versionToPublish: FlowVersion
+ newStatus: FlowStatus
+}
+
+type ApplyStatusChangeForPublishedFlowParams = {
+ id: FlowId
+ projectId: ProjectId
newStatus: FlowStatus
+ isRepublish: boolean
}
type DeleteParams = EventEmissionParams & {
diff --git a/packages/server/api/src/app/trigger/trigger-source/flow-trigger-side-effect.ts b/packages/server/api/src/app/trigger/trigger-source/flow-trigger-side-effect.ts
index d763b59aced..26cf3f3b6d6 100644
--- a/packages/server/api/src/app/trigger/trigger-source/flow-trigger-side-effect.ts
+++ b/packages/server/api/src/app/trigger/trigger-source/flow-trigger-side-effect.ts
@@ -75,7 +75,7 @@ export const flowTriggerSideEffect = (log: FastifyBaseLogger) => {
if (environment === ApEnvironment.TESTING) {
return
}
- const { flowId, flowVersionId, projectId, simulate, pieceTrigger } = params
+ const { flowId, flowVersionId, projectId, simulate, pieceTrigger, isRepublish } = params
const platformId = await projectService(log).getPlatformId(projectId)
const { error, data: engineHelperResponse } = await tryCatch(
() => userInteractionWatcher.submitAndWaitForResponse<EngineResponse<ExecuteTriggerResponse<TriggerHookType.ON_DISABLE>>>({
@@ -86,6 +86,7 @@ export const flowTriggerSideEffect = (log: FastifyBaseLogger) => {
test: simulate,
projectId,
platformId,
+ isRepublish,
}, log),
)
if (!isNil(error)) {
diff --git a/packages/server/api/src/app/trigger/trigger-source/trigger-source-service.ts b/packages/server/api/src/app/trigger/trigger-source/trigger-source-service.ts
index 9dd6221c084..0b494b404bf 100644
--- a/packages/server/api/src/app/trigger/trigger-source/trigger-source-service.ts
+++ b/packages/server/api/src/app/trigger/trigger-source/trigger-source-service.ts
@@ -195,6 +195,7 @@ export const triggerSourceService = (log: FastifyBaseLogger) => {
pieceTrigger,
simulate,
ignoreError: params.ignoreError,
+ isRepublish: params.isRepublish,
})
log.info('[triggerSourceService#disable] Disabled flow trigger side effect')
}
@@ -250,6 +251,7 @@ type DisableTriggerParams = {
simulate: boolean
ignoreError: boolean
templateId?: string
+ isRepublish?: boolean
}
type EnableTriggerParams = {
diff --git a/packages/server/api/test/integration/ee/flow-approval/approved-republish-keeps-checkpoint.test.ts b/packages/server/api/test/integration/ee/flow-approval/approved-republish-keeps-checkpoint.test.ts
index 33d835df27d..602f2f734bd 100644
--- a/packages/server/api/test/integration/ee/flow-approval/approved-republish-keeps-checkpoint.test.ts
+++ b/packages/server/api/test/integration/ee/flow-approval/approved-republish-keeps-checkpoint.test.ts
@@ -40,9 +40,9 @@ vi.mock('../../../../src/app/trigger/trigger-source/trigger-source-service', asy
}
return null
},
- disable: async (params: { flowId: string, simulate: boolean }) => {
+ disable: async (params: { flowId: string, simulate: boolean, isRepublish?: boolean }) => {
if (!params.simulate) {
- triggerCalls.push({ call: 'disable', flowId: params.flowId })
+ triggerCalls.push({ call: 'disable', flowId: params.flowId, isRepublish: params.isRepublish })
}
},
}),
@@ -66,14 +66,14 @@ describe('republishing a flow through an approval request', () => {
flowStatus: FlowStatus.ENABLED,
requestedStatus: FlowStatus.ENABLED,
republishedInput: { projectId: 'AP' },
- expected: [{ call: 'disable' }, { call: 'enable', isRepublish: true }],
+ expected: [{ call: 'disable', isRepublish: true }, { call: 'enable', isRepublish: true }],
},
{
name: 'resets the checkpoint when the trigger input changed',
flowStatus: FlowStatus.ENABLED,
requestedStatus: FlowStatus.ENABLED,
republishedInput: { projectId: 'OTHER' },
- expected: [{ call: 'disable' }, { call: 'enable', isRepublish: false }],
+ expected: [{ call: 'disable', isRepublish: false }, { call: 'enable', isRepublish: false }],
},
{
name: 'resets the checkpoint when the flow was disabled',
@@ -87,7 +87,7 @@ describe('republishing a flow through an approval request', () => {
flowStatus: FlowStatus.ENABLED,
requestedStatus: FlowStatus.DISABLED,
republishedInput: { projectId: 'AP' },
- expected: [{ call: 'disable' }],
+ expected: [{ call: 'disable', isRepublish: false }],
},
])('$name', async ({ flowStatus, requestedStatus, republishedInput, expected }) => {
const { statusCode, calls } = await approveRepublish({ flowStatus, requestedStatus, republishedInput })
@@ -98,39 +98,48 @@ describe('republishing a flow through an approval request', () => {
})
describe('republishing a flow directly', () => {
- it('keeps the checkpoint of an enabled flow whose trigger is unchanged', async () => {
+ it.each([
+ {
+ name: 'keeps the checkpoint of an enabled flow whose trigger is unchanged',
+ draftInput: { projectId: 'AP' },
+ requestedStatus: FlowStatus.ENABLED,
+ expected: [{ call: 'disable', isRepublish: true }, { call: 'enable', isRepublish: true }],
+ },
+ {
+ name: 'resets the checkpoint when the trigger input changed',
+ draftInput: { projectId: 'OTHER' },
+ requestedStatus: FlowStatus.ENABLED,
+ expected: [{ call: 'disable', isRepublish: false }, { call: 'enable', isRepublish: false }],
+ },
+ {
+ name: 'clears the old trigger when the publish keeps the flow disabled',
+ draftInput: { projectId: 'AP' },
+ requestedStatus: FlowStatus.DISABLED,
+ expected: [{ call: 'disable', isRepublish: false }],
+ },
+ {
+ name: 'resets the checkpoint when the piece version changed',
+ draftInput: { projectId: 'AP' },
+ draftPieceVersion: '0.4.2',
+ requestedStatus: FlowStatus.ENABLED,
+ expected: [{ call: 'disable', isRepublish: false }, { call: 'enable', isRepublish: false }],
+ },
+ ])('$name', async ({ draftInput, draftPieceVersion, requestedStatus, expected }) => {
const ctx = await createTestContext(app!)
- await db.save('piece_metadata', createMockPieceMetadata({
- name: '@activepieces/piece-jira-cloud',
- version: '0.4.1',
- triggers: {
- new_issue: {
- name: 'new_issue',
- displayName: 'New Issue',
- description: 'New Issue',
- requireAuth: false,
- props: {},
- type: TriggerStrategy.POLLING,
- sampleData: {},
- testStrategy: TriggerTestStrategy.TEST_FUNCTION,
- },
- },
- pieceType: PieceType.OFFICIAL,
- packageType: PackageType.REGISTRY,
- }))
+ await db.save('piece_metadata', jiraPollingPiece({ version: '0.4.1' }))
+ await db.save('piece_metadata', jiraPollingPiece({ version: '0.4.2' }))
const flow = createMockFlow({ projectId: ctx.project.id, status: FlowStatus.ENABLED })
await db.save('flow', flow)
const published = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED, valid: true, trigger: pollingTrigger({ input: { projectId: 'AP' } }) })
await db.save('flow_version', published)
await db.update('flow', flow.id, { publishedVersionId: published.id })
- const draft = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.DRAFT, valid: true, created: dayjs().add(1, 'minute').toISOString(), trigger: pollingTrigger({ input: { projectId: 'AP' } }) })
+ const draft = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.DRAFT, valid: true, created: dayjs().add(1, 'minute').toISOString(), trigger: pollingTrigger({ input: draftInput, pieceVersion: draftPieceVersion }) })
await db.save('flow_version', draft)
- const response = await ctx.post(`/v1/flows/${flow.id}`, { type: FlowOperationType.LOCK_AND_PUBLISH, request: {} })
+ const response = await ctx.post(`/v1/flows/${flow.id}`, { type: FlowOperationType.LOCK_AND_PUBLISH, request: { status: requestedStatus } })
expect(response?.statusCode).toBe(StatusCodes.OK)
- expect(recordedCalls({ flowId: flow.id }))
- .toEqual([{ call: 'disable' }, { call: 'enable', isRepublish: true }])
+ expect(recordedCalls({ flowId: flow.id })).toEqual(expected)
})
})
@@ -177,12 +186,33 @@ function recordedCalls({ flowId }: { flowId: string }): Omit<TriggerCall, 'flowI
return triggerCalls.filter((call) => call.flowId === flowId).map(({ call, isRepublish }) => ({ call, isRepublish }))
}
-function pollingTrigger({ input }: { input: Record<string, unknown> }): FlowVersion['trigger'] {
+function jiraPollingPiece({ version }: { version: string }): ReturnType<typeof createMockPieceMetadata> {
+ return createMockPieceMetadata({
+ name: '@activepieces/piece-jira-cloud',
+ version,
+ triggers: {
+ new_issue: {
+ name: 'new_issue',
+ displayName: 'New Issue',
+ description: 'New Issue',
+ requireAuth: false,
+ props: {},
+ type: TriggerStrategy.POLLING,
+ sampleData: {},
+ testStrategy: TriggerTestStrategy.TEST_FUNCTION,
+ },
+ },
+ pieceType: PieceType.OFFICIAL,
+ packageType: PackageType.REGISTRY,
+ })
+}
+
+function pollingTrigger({ input, pieceVersion = '0.4.1' }: { input: Record<string, unknown>, pieceVersion?: string }): FlowVersion['trigger'] {
return {
type: FlowTriggerType.PIECE,
settings: {
pieceName: '@activepieces/piece-jira-cloud',
- pieceVersion: '0.4.1',
+ pieceVersion,
triggerName: 'new_issue',
input,
propertySettings: {
diff --git a/packages/server/engine/src/lib/helper/trigger-helper.ts b/packages/server/engine/src/lib/helper/trigger-helper.ts
index d87f84a208c..807141b6808 100644
--- a/packages/server/engine/src/lib/helper/trigger-helper.ts
+++ b/packages/server/engine/src/lib/helper/trigger-helper.ts
@@ -6,6 +6,7 @@ import { EngineConstants, ResolvedExecuteTriggerOperation } from '../handler/con
import { FlowExecutorContext } from '../handler/context/flow-execution-context'
import { createFileUploader } from '../piece-context/file-uploader'
import { createFlowsContext } from '../piece-context/flows'
+import { republishStore } from '../piece-context/republish-store'
import { createContextStore } from '../piece-context/store'
import { utils } from '../utils'
import { propsProcessor } from '../variables/props-processor'
@@ -164,11 +165,17 @@ export const triggerHelper = {
}
switch (params.hookType) {
case TriggerHookType.ON_DISABLE: {
- await pieceTrigger.onDisable(context)
+ await pieceTrigger.onDisable({
+ ...context,
+ store: republishStore.forLifecycleHook({ store: context.store, isRepublish: params.isRepublish, triggerType: pieceTrigger.type }),
+ })
return {}
}
case TriggerHookType.ON_ENABLE: {
- await pieceTrigger.onEnable(context)
+ await pieceTrigger.onEnable({
+ ...context,
+ store: republishStore.forLifecycleHook({ store: context.store, isRepublish: params.isRepublish, triggerType: pieceTrigger.type }),
+ })
return {
listeners: appListeners,
scheduleOptions: pieceTrigger.type === TriggerStrategy.POLLING ? scheduleOptions : undefined,
diff --git a/packages/server/engine/src/lib/piece-context/republish-store.ts b/packages/server/engine/src/lib/piece-context/republish-store.ts
new file mode 100644
index 00000000000..f9be5800600
--- /dev/null
+++ b/packages/server/engine/src/lib/piece-context/republish-store.ts
@@ -0,0 +1,25 @@
+import { Store, TriggerStrategy } from '@activepieces/pieces-framework'
+
+export const republishStore = {
+ forLifecycleHook({ store, isRepublish, triggerType }: RepublishStoreParams): Store {
+ const keepsPollingState = isRepublish === true && triggerType === TriggerStrategy.POLLING
+ return keepsPollingState ? readOnly(store) : store
+ },
+}
+
+function readOnly(store: Store): Store {
+ return {
+ get: (key, scope) => store.get(key, scope),
+ async put<T>(_key: string, value: T): Promise<T> {
+ return value
+ },
+ async delete(): Promise<void> {
+ },
+ }
+}
+
+type RepublishStoreParams = {
+ store: Store
+ isRepublish: boolean | undefined
+ triggerType: TriggerStrategy
+}
diff --git a/packages/server/engine/test/helper/trigger-helper-republish.test.ts b/packages/server/engine/test/helper/trigger-helper-republish.test.ts
new file mode 100644
index 00000000000..1a10de1dd32
--- /dev/null
+++ b/packages/server/engine/test/helper/trigger-helper-republish.test.ts
@@ -0,0 +1,200 @@
+import { Store, TriggerStrategy } from '@activepieces/pieces-framework'
+import { ExecuteTriggerResponse, FlowVersionState, TriggerHookType } from '@activepieces/shared'
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+
+const { mockGetPieceAndTriggerOrThrow, storeEntries } = vi.hoisted(() => ({
+ mockGetPieceAndTriggerOrThrow: vi.fn(),
+ storeEntries: new Map<string, unknown>(),
+}))
+vi.mock('../../src/lib/helper/piece-loader', () => ({
+ pieceLoader: {
+ getPieceAndTriggerOrThrow: mockGetPieceAndTriggerOrThrow,
+ },
+}))
+vi.mock('../../src/lib/piece-context/store', () => ({
+ createContextStore: (): Store => ({
+ async put<T>(key: string, value: T): Promise<T> {
+ storeEntries.set(key, value)
+ return value
+ },
+ async get<T>(key: string): Promise<T | null> {
+ return storeEntries.has(key) ? storeEntries.get(key) as T : null
+ },
+ async delete(key: string): Promise<void> {
+ storeEntries.delete(key)
+ },
+ }),
+}))
+
+import { EngineConstants, ResolvedExecuteTriggerOperation } from '../../src/lib/handler/context/engine-constants'
+import { triggerHelper } from '../../src/lib/helper/trigger-helper'
+
+describe('a republish leaves a polling trigger\'s stored state untouched', () => {
+ beforeEach(() => {
+ mockGetPieceAndTriggerOrThrow.mockReset()
+ storeEntries.clear()
+ })
+
+ it('keeps the checkpoint onEnable overwrites', async () => {
+ storeEntries.set('lastPoll', OLD_CHECKPOINT)
+ mockTrigger({ type: TriggerStrategy.POLLING, onEnable: async ({ store }) => {
+ await store.put('lastPoll', NEW_CHECKPOINT)
+ } })
+
+ await executeHook({ hookType: TriggerHookType.ON_ENABLE, isRepublish: true })
+
+ expect(storeEntries.get('lastPoll')).toBe(OLD_CHECKPOINT)
+ })
+
+ it('keeps the checkpoint onDisable deletes', async () => {
+ storeEntries.set('_seenVideoIds', ['video-1'])
+ mockTrigger({ type: TriggerStrategy.POLLING, onDisable: async ({ store }) => {
+ await store.delete('_seenVideoIds')
+ } })
+
+ await executeHook({ hookType: TriggerHookType.ON_DISABLE, isRepublish: true })
+
+ expect(storeEntries.get('_seenVideoIds')).toEqual(['video-1'])
+ })
+
+ it('keeps the checkpoint onDisable clears by writing null', async () => {
+ storeEntries.set('_new_incoming_sms_trigger', { sid: 'message-1' })
+ mockTrigger({
+ type: TriggerStrategy.POLLING,
+ onDisable: async ({ store }) => {
+ await store.put('_new_incoming_sms_trigger', null)
+ },
+ onEnable: async ({ store }) => {
+ const last = await store.get('_new_incoming_sms_trigger')
+ if (last === null) {
+ await store.put('_new_incoming_sms_trigger', { sid: 'message-now' })
+ }
+ },
+ })
+
+ await executeHook({ hookType: TriggerHookType.ON_DISABLE, isRepublish: true })
+ await executeHook({ hookType: TriggerHookType.ON_ENABLE, isRepublish: true })
+
+ expect(storeEntries.get('_new_incoming_sms_trigger')).toEqual({ sid: 'message-1' })
+ })
+
+ it('does not let onEnable replace the checkpoint with new scan state', async () => {
+ storeEntries.set('lastItem', 'checkout-1')
+ mockTrigger({ type: TriggerStrategy.POLLING, onEnable: async ({ store }) => {
+ await store.put('newestCheckoutScanCursor', 'checkout-99')
+ await store.put('newestCheckoutScanEnabledAt', NEW_CHECKPOINT)
+ await store.delete('lastItem')
+ } })
+
+ await executeHook({ hookType: TriggerHookType.ON_ENABLE, isRepublish: true })
+
+ expect(Object.fromEntries(storeEntries)).toEqual({ lastItem: 'checkout-1' })
+ })
+
+ it('still registers the polling schedule', async () => {
+ mockTrigger({ type: TriggerStrategy.POLLING, onEnable: async ({ store, setSchedule }) => {
+ await store.put('lastPoll', NEW_CHECKPOINT)
+ setSchedule({ cronExpression: '*/5 * * * *' })
+ } })
+
+ const response = await executeHook({ hookType: TriggerHookType.ON_ENABLE, isRepublish: true })
+
+ expect(response).toMatchObject({ scheduleOptions: { cronExpression: '*/5 * * * *' } })
+ })
+})
+
+describe('outside a republish the store behaves normally', () => {
+ beforeEach(() => {
+ mockGetPieceAndTriggerOrThrow.mockReset()
+ storeEntries.clear()
+ })
+
+ it('lets onEnable reset the checkpoint on a fresh enable', async () => {
+ storeEntries.set('lastPoll', OLD_CHECKPOINT)
+ mockTrigger({ type: TriggerStrategy.POLLING, onEnable: async ({ store }) => {
+ await store.put('lastPoll', NEW_CHECKPOINT)
+ } })
+
+ await executeHook({ hookType: TriggerHookType.ON_ENABLE, isRepublish: false })
+
+ expect(storeEntries.get('lastPoll')).toBe(NEW_CHECKPOINT)
+ })
+
+ it('lets onDisable clean up when the flow is turned off', async () => {
+ storeEntries.set('_seenVideoIds', ['video-1'])
+ mockTrigger({ type: TriggerStrategy.POLLING, onDisable: async ({ store }) => {
+ await store.delete('_seenVideoIds')
+ } })
+
+ await executeHook({ hookType: TriggerHookType.ON_DISABLE, isRepublish: false })
+
+ expect(storeEntries.has('_seenVideoIds')).toBe(false)
+ })
+
+ it('lets a webhook trigger replace its webhook on a republish', async () => {
+ storeEntries.set('webhookId', 'old-webhook')
+ mockTrigger({
+ type: TriggerStrategy.WEBHOOK,
+ onDisable: async ({ store }) => {
+ await store.delete('webhookId')
+ },
+ onEnable: async ({ store }) => {
+ await store.put('webhookId', 'new-webhook')
+ },
+ })
+
+ await executeHook({ hookType: TriggerHookType.ON_DISABLE, isRepublish: true })
+ await executeHook({ hookType: TriggerHookType.ON_ENABLE, isRepublish: true })
+
+ expect(storeEntries.get('webhookId')).toBe('new-webhook')
+ })
+})
+
+function mockTrigger({ type, onEnable, onDisable }: { type: TriggerStrategy, onEnable?: (context: HookContext) => Promise<void>, onDisable?: (context: HookContext) => Promise<void> }): void {
+ mockGetPieceAndTriggerOrThrow.mockResolvedValue({
+ piece: { auth: undefined },
+ pieceTrigger: { type, props: {}, requireAuth: false, onEnable, onDisable },
+ })
+}
+
+async function executeHook({ hookType, isRepublish }: { hookType: TriggerHookType.ON_ENABLE | TriggerHookType.ON_DISABLE, isRepublish: boolean }): Promise<ExecuteTriggerResponse<TriggerHookType>> {
+ const params = {
+ hookType,
+ test: false,
+ isRepublish,
+ webhookUrl: 'http://127.0.0.1:3000/api/v1/webhooks/flow-1',
+ flowVersion: {
+ id: 'flow-version-1',
+ flowId: 'flow-1',
+ state: FlowVersionState.LOCKED,
+ trigger: {
+ name: 'trigger',
+ type: 'PIECE_TRIGGER',
+ valid: true,
+ displayName: 'Polling trigger',
+ settings: {
+ pieceName: '@activepieces/piece-under-test',
+ pieceVersion: '1.0.0',
+ triggerName: 'new_item',
+ input: {},
+ propertySettings: {},
+ },
+ },
+ },
+ projectId: 'project-1',
+ platformId: 'platform-1',
+ engineToken: 'engine-token',
+ internalApiUrl: 'http://127.0.0.1:3000/api/',
+ publicApiUrl: 'http://127.0.0.1:4200/api/',
+ timeoutInSeconds: 30,
+ } as unknown as ResolvedExecuteTriggerOperation<TriggerHookType>
+ return triggerHelper.executeTrigger({ params, constants: EngineConstants.fromExecuteTriggerInput(params) })
+}
+
+const OLD_CHECKPOINT = 1000
+const NEW_CHECKPOINT = 2000
+
+type HookContext = {
+ store: Store
+ setSchedule: (request: { cronExpression: string }) => void
+}