Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(pieces): FTP New File no longer starts a run for the same file on every poll (#16669)

Hazem Adel044ae67927cb
diff --git a/brain/knowledge/flows-execution/triggers.md b/brain/knowledge/flows-execution/triggers.md
index 0cd2b0efaa8..bb4db1aaa49 100644
--- a/brain/knowledge/flows-execution/triggers.md
+++ b/brain/knowledge/flows-execution/triggers.md
@@ -28,6 +28,7 @@ Triggers define how and when a flow starts. The module handles registration, eve
- **Renewal jobs** re-register expiring webhook pieces via the ON_RENEW hook.
- **`*/X` cron is not "every X minutes"** — it means "minutes divisible by X", so it double-fires at :00 and :X for X > 30 and gaps unevenly when X doesn't divide 60. Use `INTERVAL`/`intervalMs` for a rolling interval; reserve cron for wall-clock schedules. This bit the default poll schedule until GIT-1632.
- **A `basic-ftp` login that fails leaves the control socket open.** `access()` connects first and *then* logs in, so a refused login rejects with the socket still established, and the caller's client variable was never assigned, so no `finally` closes it. Measured against vsftpd on one refused login: the Node process stayed alive 304s waiting on that handle, versus 3s once `getClient` calls `close()` before rethrowing. Any helper that builds an FTP client must close it on a failed `access()`.
+- **`basic-ftp` only fills in `modifiedAt` from `MLSD`.** `parseListUnix` and `parseListDOS` set only the unparsed `rawModifiedAt`, and the library deliberately refuses to parse it because LIST dates carry no reliable timezone or year. A LIST-only server therefore hands you `modifiedAt: undefined`, and `dayjs(undefined)` is *now*, so every file looks freshly modified on every poll and a TIMEBASED trigger re-fires the whole folder forever (ENG-1032; against vsftpd, 6 polls emitted the same file and its subdirectory 12 times). `MDTM` via `client.lastMod` is the reliable fallback: precise UTC for files, but `550` on a directory, and `basic-ftp` rejects concurrent commands on one control connection, so ask only for regular files and ask sequentially. Treat only `550` as "skip this entry" — swallowing every `lastMod` error loses files, because one entry's transient blip is skipped while a sibling's newer timestamp still advances `lastPoll` past it. Check `FEAT` once per poll rather than per file, and when the server offers neither `MLSD` nor `MDTM` keep the old "treat as now" behaviour instead of failing: there is no timestamp to be had, and changing what those flows do is a worse trade than leaving them as they are.
- **Trigger health** (`triggerRunStats`): Redis key `trigger_run:{platformId}:{pieceName}:{date}:{status}`, 14-day retention, shown in Platform Admin (Cloud).
- **A polling `items` that catches its own connect or auth error and returns `[]` hides the outage.** The worker counts the poll COMPLETED, logs no warning and the flow stays enabled, so a rotated password looks like an empty folder. Let it throw: `pollingHelper.poll` fails before it writes `lastPoll`, the poll counts FAILED with a worker warn, and the next good poll catches up on everything missed. SFTP New File was fixed this way (ENG-796); dozens of community triggers still swallow errors the same way.
diff --git a/bun.lock b/bun.lock
index 252e9a54e66..b0bb647b588 100644
--- a/bun.lock
+++ b/bun.lock
@@ -10770,7 +10770,7 @@
},
"packages/pieces/core/sftp": {
"name": "@activepieces/piece-sftp",
- "version": "0.5.11",
+ "version": "0.5.12",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
diff --git a/packages/pieces/core/sftp/package.json b/packages/pieces/core/sftp/package.json
index ef6f855911c..cb67e7409a2 100644
--- a/packages/pieces/core/sftp/package.json
+++ b/packages/pieces/core/sftp/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/piece-sftp",
- "version": "0.5.11",
+ "version": "0.5.12",
"main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts",
"dependencies": {
diff --git a/packages/pieces/core/sftp/src/lib/triggers/new-modified-file.ts b/packages/pieces/core/sftp/src/lib/triggers/new-modified-file.ts
index 6bd96aedb70..46f80d31309 100644
--- a/packages/pieces/core/sftp/src/lib/triggers/new-modified-file.ts
+++ b/packages/pieces/core/sftp/src/lib/triggers/new-modified-file.ts
@@ -1,16 +1,49 @@
-import { PiecePropValueSchema, Property, createTrigger, TriggerStrategy, AppConnectionValueForAuthProperty } from '@activepieces/pieces-framework';
+import { Property, createTrigger, TriggerStrategy, AppConnectionValueForAuthProperty, isNil, tryCatch } from '@activepieces/pieces-framework';
import { DedupeStrategy, Polling, pollingHelper } from '@activepieces/pieces-common';
import { getClient, getProtocolBackwardCompatibility, endClient } from '../common';
import { sftpAuth } from '../auth';
import dayjs from 'dayjs';
import Client from 'ssh2-sftp-client';
-import { Client as FTPClient, FileInfo as FTPFileInfo } from 'basic-ftp';
+import { Client as FTPClient, FTPError, FileInfo as FTPFileInfo, FileType as FTPFileType } from 'basic-ftp';
import { newOrModifiedFileTriggerOutputSchema } from '../output-schemas';
-function getModifyTime(file: Client.FileInfo | FTPFileInfo, protocol: string): number {
- return protocol === 'sftp' ?
- (file as Client.FileInfo).modifyTime :
- dayjs((file as FTPFileInfo).modifiedAt).valueOf();
+const FTP_FILE_UNAVAILABLE = 550;
+
+async function getModifyTime({ client, file, protocol, directory, serverCanTimestamp }: { client: Client | FTPClient; file: Client.FileInfo | FTPFileInfo; protocol: string; directory: string; serverCanTimestamp: boolean }): Promise<number | null> {
+ if (protocol === 'sftp') {
+ return (file as Client.FileInfo).modifyTime;
+ }
+
+ const ftpFile = file as FTPFileInfo;
+ if (!isNil(ftpFile.modifiedAt)) {
+ return dayjs(ftpFile.modifiedAt).valueOf();
+ }
+ if (ftpFile.type === FTPFileType.Directory) {
+ return null;
+ }
+ if (!serverCanTimestamp) {
+ return Date.now();
+ }
+
+ const { data, error } = await tryCatch(() => (client as FTPClient).lastMod(`${directory.replace(/\/+$/, '')}/${ftpFile.name}`));
+ if (isNil(error)) {
+ return dayjs(data).valueOf();
+ }
+
+ const serverRefusedThisEntry = error instanceof FTPError && error.code === FTP_FILE_UNAVAILABLE;
+ if (serverRefusedThisEntry) {
+ return null;
+ }
+ throw error;
+}
+
+async function serverCanTimestampEntries({ client, protocol, files }: { client: Client | FTPClient; protocol: string; files: (Client.FileInfo | FTPFileInfo)[] }): Promise<boolean> {
+ const listingCarriesTimes = protocol === 'sftp' || files.every(file => !isNil((file as FTPFileInfo).modifiedAt));
+ if (listingCarriesTimes) {
+ return true;
+ }
+ const features = await (client as FTPClient).features();
+ return features.has('MDTM');
}
const polling: Polling<AppConnectionValueForAuthProperty<typeof sftpAuth>, { path: string; ignoreHiddenFiles?: boolean }> = {
@@ -22,26 +55,27 @@ const polling: Polling<AppConnectionValueForAuthProperty<typeof sftpAuth>, { pat
client = await getClient(auth.props);
const files = await client.list(propsValue.path);
- const filteredFiles = files.filter(file => {
- const modTime = getModifyTime(file, protocolBackwardCompatibility);
- return dayjs(modTime).valueOf() > lastFetchEpochMS;
- });
+ const visibleFiles: (Client.FileInfo | FTPFileInfo)[] = propsValue.ignoreHiddenFiles ?
+ files.filter(file => !file.name.startsWith('.')) :
+ files;
- const finalFiles: (Client.FileInfo | FTPFileInfo)[] = propsValue.ignoreHiddenFiles ?
- filteredFiles.filter(file => !file.name.startsWith('.')) :
- filteredFiles;
+ const serverCanTimestamp = await serverCanTimestampEntries({ client, protocol: protocolBackwardCompatibility, files: visibleFiles });
- return finalFiles.map(file => {
- const modTime = getModifyTime(file, protocolBackwardCompatibility);
-
- return {
+ const newFiles: { data: unknown; epochMilliSeconds: number }[] = [];
+ for (const file of visibleFiles) {
+ const modifyTime = await getModifyTime({ client, file, protocol: protocolBackwardCompatibility, directory: propsValue.path, serverCanTimestamp });
+ if (isNil(modifyTime) || modifyTime <= lastFetchEpochMS) {
+ continue;
+ }
+ newFiles.push({
data: {
...file,
path: `${propsValue.path}/${file.name}`,
},
- epochMilliSeconds: dayjs(modTime).valueOf(),
- };
- });
+ epochMilliSeconds: modifyTime,
+ });
+ }
+ return newFiles;
} finally {
if (client) {
await endClient(client, auth.props.protocol);
diff --git a/packages/pieces/core/sftp/test/new-file-trigger-ftp-timestamps.test.ts b/packages/pieces/core/sftp/test/new-file-trigger-ftp-timestamps.test.ts
new file mode 100644
index 00000000000..a434ea0b39a
--- /dev/null
+++ b/packages/pieces/core/sftp/test/new-file-trigger-ftp-timestamps.test.ts
@@ -0,0 +1,167 @@
+import { Store } from '@activepieces/pieces-framework';
+import { FTPError, FileType } from 'basic-ftp';
+import { beforeEach, describe, expect, test, vi } from 'vitest';
+import { newOrModifiedFile } from '../src/lib/triggers/new-modified-file';
+
+const { ftpClient } = vi.hoisted(() => ({
+ ftpClient: { access: vi.fn(), list: vi.fn(), lastMod: vi.fn(), features: vi.fn(), close: vi.fn() },
+}));
+
+vi.mock('ssh2-sftp-client', () => ({ default: vi.fn() }));
+vi.mock('basic-ftp', async (importOriginal) => ({
+ ...(await importOriginal<typeof import('basic-ftp')>()),
+ Client: vi.fn(() => ftpClient),
+}));
+
+const MODIFIED_AT = new Date('2026-10-20T14:40:13.000Z');
+
+describe('New File trigger over FTP', () => {
+ beforeEach(() => {
+ vi.resetAllMocks();
+ ftpClient.access.mockResolvedValue(undefined);
+ ftpClient.list.mockResolvedValue([]);
+ ftpClient.features.mockResolvedValue(new Map([['MDTM', '']]));
+ });
+
+ test('emits a file once when the server has no MLSD, instead of re-emitting it every poll', async () => {
+ const store = memoryStore({ lastPoll: MODIFIED_AT.getTime() - 60_000 });
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);
+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);
+
+ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([
+ expect.objectContaining({ name: 'report.csv' }),
+ ]);
+ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([]);
+ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([]);
+ });
+
+ test('asks the server for the timestamp with the full path, once per file', async () => {
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);
+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);
+
+ await newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })));
+
+ expect(ftpClient.lastMod).toHaveBeenCalledTimes(1);
+ expect(ftpClient.lastMod).toHaveBeenCalledWith('/inbox/report.csv');
+ });
+
+ test('does not double the separator when the watched path ends in a slash', async () => {
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);
+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);
+
+ await newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 }), './'));
+
+ expect(ftpClient.lastMod).toHaveBeenCalledWith('./report.csv');
+ });
+
+ test('never asks the server to timestamp a directory, which answers 550', async () => {
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'subdir', type: FileType.Directory })]);
+
+ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([]);
+ expect(ftpClient.lastMod).not.toHaveBeenCalled();
+ });
+
+ test('emits a symlinked file, whose timestamp the server can resolve', async () => {
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'linked.csv', type: FileType.SymbolicLink })]);
+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);
+
+ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([
+ expect.objectContaining({ name: 'linked.csv' }),
+ ]);
+ expect(ftpClient.lastMod).toHaveBeenCalledWith('/inbox/linked.csv');
+ });
+
+ test('skips an entry the server permanently refuses to timestamp', async () => {
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);
+ ftpClient.lastMod.mockRejectedValue(new FTPError({ code: 550, message: '550 Could not get file modification time.' }));
+
+ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([]);
+ });
+
+ test('fails the poll on a temporary timestamp error instead of losing the file behind a newer one', async () => {
+ const store = memoryStore({ lastPoll: 0 });
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'blips.csv' }), listEntry({ name: 'fine.csv' })]);
+ ftpClient.lastMod
+ .mockRejectedValueOnce(new FTPError({ code: 421, message: '421 Service not available.' }))
+ .mockResolvedValue(MODIFIED_AT);
+
+ await expect(newOrModifiedFile.run(pollContext(store))).rejects.toThrow('421 Service not available.');
+ expect(await store.get('lastPoll')).toBe(0);
+
+ ftpClient.lastMod.mockReset();
+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);
+
+ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([
+ expect.objectContaining({ name: 'blips.csv' }),
+ expect.objectContaining({ name: 'fine.csv' }),
+ ]);
+ });
+
+ test('keeps the previous behaviour on a server that offers neither MLSD nor MDTM', async () => {
+ ftpClient.features.mockResolvedValue(new Map([['PASV', '']]));
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);
+
+ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([
+ expect.objectContaining({ name: 'report.csv' }),
+ ]);
+ expect(ftpClient.lastMod).not.toHaveBeenCalled();
+ });
+
+ test('does not ask the server for its features when the listing already carries times', async () => {
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv', modifiedAt: MODIFIED_AT })]);
+
+ await newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })));
+
+ expect(ftpClient.features).not.toHaveBeenCalled();
+ });
+
+ test('uses the listing timestamp and skips MDTM when the server supports MLSD', async () => {
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv', modifiedAt: MODIFIED_AT })]);
+
+ await expect(newOrModifiedFile.run(pollContext(memoryStore({ lastPoll: 0 })))).resolves.toEqual([
+ expect.objectContaining({ name: 'report.csv' }),
+ ]);
+ expect(ftpClient.lastMod).not.toHaveBeenCalled();
+ });
+
+ test('emits the file again once it is genuinely modified', async () => {
+ const store = memoryStore({ lastPoll: 0 });
+ ftpClient.list.mockResolvedValue([listEntry({ name: 'report.csv' })]);
+ ftpClient.lastMod.mockResolvedValue(MODIFIED_AT);
+ await newOrModifiedFile.run(pollContext(store));
+
+ ftpClient.lastMod.mockResolvedValue(new Date(MODIFIED_AT.getTime() + 1_000));
+
+ await expect(newOrModifiedFile.run(pollContext(store))).resolves.toEqual([
+ expect.objectContaining({ name: 'report.csv' }),
+ ]);
+ });
+});
+
+function listEntry({ name, type = FileType.File, modifiedAt }: { name: string; type?: FileType; modifiedAt?: Date }) {
+ return { name, type, size: 12, rawModifiedAt: 'Oct 20 14:40', modifiedAt };
+}
+
+function memoryStore(initial: Record<string, unknown>): Store {
+ const entries = new Map(Object.entries(initial));
+ return {
+ async put<T>(key: string, value: T): Promise<T> {
+ entries.set(key, value);
+ return value;
+ },
+ async get<T>(key: string): Promise<T | null> {
+ return entries.has(key) ? (entries.get(key) as T) : null;
+ },
+ async delete(key: string): Promise<void> {
+ entries.delete(key);
+ },
+ };
+}
+
+function pollContext(store: Store, path = '/inbox') {
+ return {
+ auth: { props: { protocol: 'ftp', host: 'files.example.com', port: 21, username: 'user', password: 'secret' } },
+ propsValue: { path, ignoreHiddenFiles: false },
+ store,
+ } as never;
+}