AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(pieces): FTP New File no longer starts a run for the same file on every poll (#16669)
Hazem Adel044ae67927cb
diff --git a/brain/knowledge/flows-execution/triggers.md b/brain/knowledge/flows-execution/triggers.mdindex 0cd2b0efaa8..bb4db1aaa49 100644--- a/brain/knowledge/flows-execution/triggers.md+++ b/brain/knowledge/flows-execution/triggers.md@@ -28,6 +28,7 @@ Triggers define how and when a flow starts. The module handles registration, eve- **Renewal jobs** re-register expiring webhook pieces via the ON_RENEW hook.- **`*/X` cron is not "every X minutes"** — it means "minutes divisible by X", so it double-fires at :00 and :X for X > 30 and gaps unevenly when X doesn't divide 60. Use `INTERVAL`/`intervalMs` for a rolling interval; reserve cron for wall-clock schedules. This bit the default poll schedule until GIT-1632.- **A `basic-ftp` login that fails leaves the control socket open.** `access()` connects first and *then* logs in, so a refused login rejects with the socket still established, and the caller's client variable was never assigned, so no `finally` closes it. Measured against vsftpd on one refused login: the Node process stayed alive 304s waiting on that handle, versus 3s once `getClient` calls `close()` before rethrowing. Any helper that builds an FTP client must close it on a failed `access()`.+- **`basic-ftp` only fills in `modifiedAt` from `MLSD`.** `parseListUnix` and `parseListDOS` set only the unparsed `rawModifiedAt`, and the library deliberately refuses to parse it because LIST dates carry no reliable timezone or year. A LIST-only server therefore hands you `modifiedAt: undefined`, and `dayjs(undefined)` is *now*, so every file looks freshly modified on every poll and a TIMEBASED trigger re-fires the whole folder forever (ENG-1032; against vsftpd, 6 polls emitted the same file and its subdirectory 12 times). `MDTM` via `client.lastMod` is the reliable fallback: precise UTC for files, but `550` on a directory, and `basic-ftp` rejects concurrent commands on one control connection, so ask only for regular files and ask sequentially. Treat only `550` as "skip this entry" — swallowing every `lastMod` error loses files, because one entry's transient blip is skipped while a sibling's newer timestamp still advances `lastPoll` past it. Check `FEAT` once per poll rather than per file, and when the server offers neither `MLSD` nor `MDTM` keep the old "treat as now" behaviour instead of failing: there is no timestamp to be had, and changing what those flows do is a worse trade than leaving them as they are.- **Trigger health** (`triggerRunStats`): Redis key `trigger_run:{platformId}:{pieceName}:{date}:{status}`, 14-day retention, shown in Platform Admin (Cloud).- **A polling `items` that catches its own connect or auth error and returns `[]` hides the outage.** The worker counts the poll COMPLETED, logs no warning and the flow stays enabled, so a rotated password looks like an empty folder. Let it throw: `pollingHelper.poll` fails before it writes `lastPoll`, the poll counts FAILED with a worker warn, and the next good poll catches up on everything missed. SFTP New File was fixed this way (ENG-796); dozens of community triggers still swallow errors the same way.diff --git a/bun.lock b/bun.lockindex 252e9a54e66..b0bb647b588 100644--- a/bun.lock+++ b/bun.lock@@ -10770,7 +10770,7 @@},"packages/pieces/core/sftp": {"name": "@activepieces/piece-sftp",- "version": "0.5.11",+ "version": "0.5.12","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",diff --git a/packages/pieces/core/sftp/package.json b/packages/pieces/core/sftp/package.jsonindex ef6f855911c..cb67e7409a2 100644--- a/packages/pieces/core/sftp/package.json+++ b/packages/pieces/core/sftp/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/piece-sftp",- "version": "0.5.11",+ "version": "0.5.12","main": "./dist/src/index.js","types": "./dist/src/index.d.ts","dependencies": {diff --git a/packages/pieces/core/sftp/src/lib/triggers/new-modified-file.ts b/packages/pieces/core/sftp/src/lib/triggers/new-modified-file.tsindex 6bd96aedb70..46f80d31309 100644--- a/packages/pieces/core/sftp/src/lib/triggers/new-modified-file.ts+++ b/packages/pieces/core/sftp/src/lib/triggers/new-modified-file.ts@@ -1,16 +1,49 @@-import { PiecePropValueSchema, Property, createTrigger, TriggerStrategy, AppConnectionValueForAuthProperty } from '@activepieces/pieces-framework';+import { Property, createTrigger, TriggerStrategy, AppConnectionValueForAuthProperty, isNil, tryCatch } from '@activepieces/pieces-framework';import { DedupeStrategy, Polling, pollingHelper } from '@activepieces/pieces-common';import { getClient, getProtocolBackwardCompatibility, endClient } from '../common';import { sftpAuth } from '../auth';import dayjs from 'dayjs';import Client from 'ssh2-sftp-client';-import { Client as FTPClient, FileInfo as FTPFileInfo } from 'basic-ftp';+import { Client as FTPClient, FTPError, FileInfo as FTPFileInfo, FileType as FTPFileType } from 'basic-ftp';import { newOrModifiedFileTriggerOutputSchema } from '../output-schemas';-function getModifyTime(file: Client.FileInfo | FTPFileInfo, protocol: string): number {- return protocol === 'sftp' ?- (file as Client.FileInfo).modifyTime :- dayjs((file as FTPFileInfo).modifiedAt).valueOf();+const FTP_FILE_UNAVAILABLE = 550;++async function getModifyTime({ client, file, protocol, directory, serverCanTimestamp }: { client: Client | FTPClient; file: Client.FileInfo | FTPFileInfo; protocol: string; directory: string; serverCanTimestamp: boolean }): Promise<number | null> {+ if (protocol === 'sftp') {+ return (file as Client.FileInfo).modifyTime;+ }++ const ftpFile = file as FTPFileInfo;+ if (!isNil(ftpFile.modifiedAt)) {+ return dayjs(ftpFile.modifiedAt).valueOf();+ }+ if (ftpFile.type === FTPFileType.Directory) {+ return null;+ }+ if (!serverCanTimestamp) {+ return Date.now();+ }++ const { data, error } = await tryCatch(() => (client as FTPClient).lastMod(`${directory.replace(/\/+$/, '')}/${ftpFile.name}`));+ if (isNil(error)) {+ return dayjs(data).valueOf();+ }++ const serverRefusedThisEntry = error instanceof FTPError && error.code === FTP_FILE_UNAVAILABLE;+ if (serverRefusedThisEntry) {+ return null;+ }+ throw error;+}++async function serverCanTimestampEntries({ client, protocol, files }: { client: Client | FTPClient; protocol: string; files: (Client.FileInfo | FTPFileInfo)[] }): Promise<boolean> {+ const listingCarriesTimes = protocol === 'sftp' || files.every(file => !isNil((file as FTPFileInfo).modifiedAt));+ if (listingCarriesTimes) {+ return true;+ }+ const features = await (client as FTPClient).features();+ return features.has('MDTM');}const polling: Polling<AppConnectionValueForAuthProperty<typeof sftpAuth>, { path: string; ignoreHiddenFiles?: boolean }> = {@@ -22,26 +55,27 @@ const polling: Polling<AppConnectionValueForAuthProperty<typeof sftpAuth>, { patclient = await getClient(auth.props);const files = await client.list(propsValue.path);- const filteredFiles = files.filter(file => {- const modTime = getModifyTime(file, protocolBackwardCompatibility);- return dayjs(modTime).valueOf() > lastFetchEpochMS;- });+ const visibleFiles: (Client.FileInfo | FTPFileInfo)[] = propsValue.ignoreHiddenFiles ?+ files.filter(file => !file.name.startsWith('.')) :+ files;- const finalFiles: (Client.FileInfo | FTPFileInfo)[] = propsValue.ignoreHiddenFiles ?- filteredFiles.filter(file => !file.name.startsWith('.')) :- filteredFiles;+ const serverCanTimestamp = await serverCanTimestampEntries({ client, protocol: protocolBackwardCompatibility, files: visibleFiles });- return finalFiles.map(file => {- const modTime = getModifyTime(file, protocolBackwardCompatibility);-- return {+ const newFiles: { data: unknown; epochMilliSeconds: number }[] = [];+ for (const file of visibleFiles) {+ const modifyTime = await getModifyTime({ client, file, protocol: protocolBackwardCompatibility, directory: propsValue.path, serverCanTimestamp });+ if (isNil(modifyTime) || modifyTime <= lastFetchEpochMS) {+ continue;+ }+ newFiles.push({data: {...file,path: `${propsValue.path}/${file.name}`,},- epochMilliSeconds: dayjs(modTime).valueOf(),- };- });+ epochMilliSeconds: modifyTime,+ });+ }+ return newFiles;} finally {if (client) {await endClient(client, auth.props.protocol);diff --git a/packages/pieces/core/sftp/test/new-file-trigger-ftp-timestamps.test.ts b/packages/pieces/core/sftp/test/new-file-trigger-ftp-timestamps.test.tsnew file mode 100644index 00000000000..a434ea0b39a--- /dev/null+++ b/packages/pieces/core/sftp/test/new-file-trigger-ftp-timestamps.test.ts@@ -0,0 +1,167 @@+import { Store } from '@activepieces/pieces-framework';+import { FTPError, FileType } from 'basic-ftp';+import { beforeEach, describe, expect, test, vi } from 'vitest';+import { newOrModifiedFile } from '../src/lib/triggers/new-modified-file';++const { ftpClient } = vi.hoisted(() => ({+ ftpClient: { access: vi.fn(), list: vi.fn(), lastMod: vi.fn(), features: vi.fn(), close: vi.fn() },+}));++vi.mock('ssh2-sftp-client', () => ({ default: vi.fn() }));+vi.mock('basic-ftp', async (importOriginal) => ({+ ...(await importOriginal<typeof import('basic-ftp')>()),+ Client: vi.fn(() => ftpClient),+}));++const MODIFIED_AT = new Date('2026-10-20T14:40:13.000Z');++describe('New File trigger over FTP', () => {+ beforeEach(() => {+ vi.resetAllMocks();+ ftpClient.access.mockResolvedValue(undefined);+ ftpClient.list.mockResolvedValue([]);+ ftpClient.features.mockResolvedValue(new Map([['MDTM', '']]));+ });++ test('emits a file once when the server has no MLSD, instead of re-emitting it every poll', async () => {+ const store = memoryStore({ lastPoll: MODIFIED_AT.getTime() - 60_000 });+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);++ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([+ expect.objectContaining({ name: 'report.csv' }),+ ]);+ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([]);+ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([]);+ });++ test('asks the server for the timestamp with the full path, once per file', async () => {+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);++ await newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })));++ expect(ftpClient.lastMod).toHaveBeenCalledTimes(1);+ expect(ftpClient.lastMod).toHaveBeenCalledWith('/inbox/report.csv');+ });++ test('does not double the separator when the watched path ends in a slash', async () => {+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);++ await newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 }), './'));++ expect(ftpClient.lastMod).toHaveBeenCalledWith('./report.csv');+ });++ test('never asks the server to timestamp a directory, which answers 550', async () => {+ ftpClient.list.mockResolvedValue([listEntry({ name: 'subdir', type: FileType.Directory })]);++ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([]);+ expect(ftpClient.lastMod).not.toHaveBeenCalled();+ });++ test('emits a symlinked file, whose timestamp the server can resolve', async () => {+ ftpClient.list.mockResolvedValue([listEntry({ name: 'linked.csv', type: FileType.SymbolicLink })]);+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);++ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([+ expect.objectContaining({ name: 'linked.csv' }),+ ]);+ expect(ftpClient.lastMod).toHaveBeenCalledWith('/inbox/linked.csv');+ });++ test('skips an entry the server permanently refuses to timestamp', async () => {+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);+ ftpClient.lastMod.mockRejectedValue(new FTPError({ code: 550, message: '550 Could not get file modification time.' }));++ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([]);+ });++ test('fails the poll on a temporary timestamp error instead of losing the file behind a newer one', async () => {+ const store = memoryStore({ lastPoll: 0 });+ ftpClient.list.mockResolvedValue([listEntry({ name: 'blips.csv' }), listEntry({ name: 'fine.csv' })]);+ ftpClient.lastMod+ .mockRejectedValueOnce(new FTPError({ code: 421, message: '421 Service not available.' }))+ .mockResolvedValue(MODIFIED_AT);++ await expect(newOrModifiedFile.run(pollContext(store))).rejects.toThrow('421 Service not available.');+ expect(await store.get('lastPoll')).toBe(0);++ ftpClient.lastMod.mockReset();+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);++ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([+ expect.objectContaining({ name: 'blips.csv' }),+ expect.objectContaining({ name: 'fine.csv' }),+ ]);+ });++ test('keeps the previous behaviour on a server that offers neither MLSD nor MDTM', async () => {+ ftpClient.features.mockResolvedValue(new Map([['PASV', '']]));+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);++ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([+ expect.objectContaining({ name: 'report.csv' }),+ ]);+ expect(ftpClient.lastMod).not.toHaveBeenCalled();+ });++ test('does not ask the server for its features when the listing already carries times', async () => {+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv', modifiedAt: MODIFIED_AT })]);++ await newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })));++ expect(ftpClient.features).not.toHaveBeenCalled();+ });++ test('uses the listing timestamp and skips MDTM when the server supports MLSD', async () => {+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv', modifiedAt: MODIFIED_AT })]);++ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([+ expect.objectContaining({ name: 'report.csv' }),+ ]);+ expect(ftpClient.lastMod).not.toHaveBeenCalled();+ });++ test('emits the file again once it is genuinely modified', async () => {+ const store = memoryStore({ lastPoll: 0 });+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);+ await newOrModifiedFile.run(pollContext(store));++ ftpClient.lastMod.mockResolvedValue(new Date(MODIFIED_AT.getTime() + 1_000));++ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([+ expect.objectContaining({ name: 'report.csv' }),+ ]);+ });+});++function listEntry({ name, type = FileType.File, modifiedAt }: { name: string; type?: FileType; modifiedAt?: Date }) {+ return { name, type, size: 12, rawModifiedAt: 'Oct 20 14:40', modifiedAt };+}++function memoryStore(initial: Record<string, unknown>): Store {+ const entries = new Map(Object.entries(initial));+ return {+ async put<T>(key: string, value: T): Promise<T> {+ entries.set(key, value);+ return value;+ },+ async get<T>(key: string): Promise<T | null> {+ return entries.has(key) ? (entries.get(key) as T) : null;+ },+ async delete(key: string): Promise<void> {+ entries.delete(key);+ },+ };+}++function pollContext(store: Store, path = '/inbox') {+ return {+ auth: { props: { protocol: 'ftp', host: 'files.example.com', port: 21, username: 'user', password: 'secret' } },+ propsValue: { path, ignoreHiddenFiles: false },+ store,+ } as never;+}