AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(files): send upload file names as ASCII-safe headers (#15959)
Amr Elmohamady1849cf07fdce
diff --git a/brain/knowledge/data-storage-observability/file-storage.md b/brain/knowledge/data-storage-observability/file-storage.mdindex 1c343f6136d..3f0cbdf11fa 100644--- a/brain/knowledge/data-storage-observability/file-storage.md+++ b/brain/knowledge/data-storage-observability/file-storage.md@@ -32,6 +32,7 @@ The central service for persisting binary files, backing the execution engine an- User-facing docs: [Large File Streaming](../../../docs/build-pieces/piece-reference/large-file-streaming.mdx).### Gotchas+- **The engine's upload file name travels in a header, so it must be ASCII on the wire.** `x-ap-file-name` used to carry the raw name; Node fetch sends a Latin-1 char like `é` as the single byte `0xE9`, which is invalid UTF-8, and chars above 255 (`–`, emoji, CJK) throw `Cannot convert argument to a ByteString` before the request leaves. Behind Cloudflare the first case is a 403 from the managed rule *Anomaly:Header — Invalid UTF-8 Encoding*, surfacing as `EngineFileUploadError: … 403 Forbidden` (Alan, 2026-09-29, French document names). The engine now sends an ASCII fallback in `x-ap-file-name` plus `encodeURIComponent(name)` in `x-ap-file-name-encoded`; the app prefers the encoded one and falls back to the raw header, so old/new engines and apps interoperate in both directions. Never put user text in a header raw.- **On cloud the real key prefix is doubled — `<bucket>/<bucket>/…` — so ad-hoc CLI work against the bucket silently finds nothing.** Cloud's `AP_S3_ENDPOINT` embeds the bucket as a *path segment* (`https://<account>.r2.cloudflarestorage.com/ap-files-prod`), and `getS3Client` sets `forcePathStyle: true` whenever an endpoint is present, so the SDK appends the bucket again. An object the app stores as `pieces/x.tgz` actually lands at `ap-files-prod/pieces/x.tgz` inside bucket `ap-files-prod`. Symptom when you get it wrong: `aws s3 ls` returns `NoSuchKey` on a *prefix* listing (Aug 2026 — cost three attempts to spot). Strip the trailing `/<bucket>` from the endpoint for the CLI, then prepend the bucket name to the prefix; or better, do bulk work through `s3Helper` so the same client resolves the same paths. Note cloud's object store is **Cloudflare R2** while the piece CDN is a **DigitalOcean Space** — two different systems, easy to conflate.- **The live piece-bundle cache sits *inside* the legacy one — `pieces/v2/` is nested under `pieces/`, so a recursive delete of `pieces/` takes the active cache with it.** `S3_PIECES_PREFIX` in `piece-bundle.ts` is `pieces/v2/`; the bare `pieces/` keys beside it are pre-CDN tarballs left by the older writer. Combined with the doubled prefix above, the real keys are `ap-files-prod/pieces/…` (legacy) and `ap-files-prod/pieces/v2/…` (live). Probe both with `wrangler r2 object get` before any prefix-wide operation — wiping v2 used to be survivable because it refilled lazily, at the cost of a burst of cache misses on every piece. **Both prefixes are now dead storage and safe to sweep:** the `BUNDLE_PIECE` job and the S3 mirror were removed, so `resolve()` no longer reads or writes either prefix and registry pieces redirect straight to the CDN (else npm). The mirror was deleted because it was written from whichever source was preferred *at cache time* and then took precedence over the CDN forever — a bucket populated before the CDN became preferred kept serving the unbundled npm build, which is what fans out one `@activepieces/shared` copy per piece in the engine (see the Workers page).- **`deleteFiles` succeeding does not mean the objects are gone.** `DeleteObjectsCommand` reports per-object failures in `response.Errors` and does **not** throw, and `Quiet: true` only suppresses the success entries — so a request that "worked" can still have left objects behind. `deleteFiles` logs a warn naming the failure codes, which is all its callers (best-effort cleanup) need. Anything whose *correctness* depends on the prefix being empty afterwards would have to surface those keys and retry — but prefer not to need that at all: a reader that must not see the old objects should read from a new key prefix rather than race a delete against writers that may still be running old code.diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.mdindex 9f911b8a737..d252a3a436b 100644--- a/brain/knowledge/execution-runtime/workers.md+++ b/brain/knowledge/execution-runtime/workers.md@@ -20,6 +20,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the- `concurrency === 1` primes the sandbox to the full container RAM (cgroup-aware) via `primeFullContainerMemory()`.### Gotchas+- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.- **The piece workspace pins `linker = "isolated"` in its own `bunfig.toml` — do not delete it, the engine's piece resolver depends on the layout bun chooses.** `pieceInstaller` generates a bun workspace at `<cache>/<version>/common` whose members are `pieces/<pieceName>-<pieceVersion>`, and the engine (`piece-loader.ts` → `resolveInstalledPieceEntry`) resolves a piece **only** at `pieces/<alias>/node_modules/<pieceName>`, which just the isolated linker produces. The workspace used to carry no bunfig and inherited whatever ancestor bunfig bun discovered above the cache mount (`/usr/src/app/bunfig.toml`); when bun instead uses the **hoisted** linker it writes the package as a real directory at the workspace root `node_modules/@activepieces/<name>` and leaves the member with no `node_modules` at all, so every run of that piece fails `PieceNotFoundError: Piece not found for package: <name>-<version>` while `bun install` still exits 0. It was silent and self-perpetuating — `markPiecesAsUsed` writes `ready` and `pieceCheckIfAlreadyInstalled` only looks for *a* `node_modules`, so the piece is reinstalled forever and never resolves. Sep 2026 production: 125/125 folders missing `node_modules` were exactly the 125 packages sitting hoisted at the workspace root; 7,400+ `EXECUTE_FLOW` jobs, 15+ platforms, ~60 official pieces, ~300/hr. Two things had to line up — bun 1.4.0 (Aug 31) changed the layout newly-installed members got, and removing the piece-upgrade Redis gate (#15236, deployed 2026-09-03 08:09 UTC) mass-upgraded every platform's flows to newer piece versions so almost every install was suddenly a *first-time* install. Diagnose on a worker with `for d in <cache>/v*/common/pieces/@activepieces/*/; do [ -d "$d/node_modules" ] || basename $d; done`. Fixed by pinning the linker plus a `LATEST_CACHE_VERSION` bump (v14 → v15) so the fleet rebuilds one clean workspace under the pinned layout rather than carrying a mixed one.- **No PM2 anymore — the container is the supervision unit.** `docker-entrypoint.sh` launches the bootstrap scripts with plain `node --enable-source-maps` (removed the `pm2-runtime` + `/tmp/ecosystem.config.js` machinery). `APP`/`WORKER` `exec` a single node as PID 1; `WORKER_AND_APP` runs both and, if either exits, kills the other and exits non-zero so the orchestrator restarts the whole container. This drops PM2's *in-container* crash/OOM restart: an OOM-kill no longer silently recycles a child every ~4 min behind a `RestartCount: 0` (the 2026-07-26 wedge's supply of retry attempts — see below); the container now dies and is rescheduled instead. The historical incident notes below still describe the old PM2 behavior as it happened.- **Version gate (rolling-deploy safety)**: dispatch requires an exact release match, enforced both sides via `versionsAreCompatible` (fail-closed — `undefined` or `UNKNOWN_VERSION` `'0.0.0'` is treated incompatible). App withholds jobs from a mismatched worker (`poll` returns null); worker pauses polling 10s. Ordinary mismatch self-heals on convergence; a read failure does not (cached for process life) and pages on-call once at startup via `assertReleaseReadable`.diff --git a/packages/server/api/src/app/file/files-controller.ts b/packages/server/api/src/app/file/files-controller.tsindex df44816f9fb..9a3454c44bd 100644--- a/packages/server/api/src/app/file/files-controller.ts+++ b/packages/server/api/src/app/file/files-controller.ts@@ -1,5 +1,6 @@+import type { IncomingHttpHeaders } from 'node:http'import { Readable } from 'node:stream'-import { ActivepiecesError, ApId, assertNotNullOrUndefined, ErrorCode, isNil } from '@activepieces/core-utils'+import { ActivepiecesError, ApId, assertNotNullOrUndefined, ErrorCode, isNil, tryCatchSync } from '@activepieces/core-utils'import { ALL_PRINCIPAL_TYPES, EnginePrincipal, FileCompression, FileTransportQueryParams, FileType, Principal, PrincipalType } from '@activepieces/shared'import contentDisposition from 'content-disposition'import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'@@ -33,7 +34,7 @@ export const filesController: FastifyPluginAsyncZod = async (app) => {const token = (request.query as { token: string }).tokenconst principal = await verifyEnginePrincipal(token, request.log)const fileType = parseFileTypeHeader(request.headers[fileTransportHeaders.TYPE])- const fileName = parseStringHeader(request.headers[fileTransportHeaders.NAME])+ const fileName = parseFileNameHeader(request.headers)const contentEncoding = parseStringHeader(request.headers['content-encoding'])const compression = contentEncoding === 'zstd' ? FileCompression.ZSTD : FileCompression.NONEconst contentLength = Number(request.headers['content-length'] ?? 0)@@ -79,7 +80,7 @@ export const filesController: FastifyPluginAsyncZod = async (app) => {const { fileId } = request.paramsconst principal = await verifyEnginePrincipal(request.query.token, request.log)const fileType = parseFileTypeHeader(request.headers[fileTransportHeaders.TYPE])- const fileName = parseStringHeader(request.headers[fileTransportHeaders.NAME])+ const fileName = parseFileNameHeader(request.headers)const contentEncoding = parseStringHeader(request.headers['content-encoding'])const compression = contentEncoding === 'zstd' ? FileCompression.ZSTD : FileCompression.NONE@@ -231,6 +232,17 @@ function parseFileTypeHeader(value: unknown): FileType {return raw as FileType}+function parseFileNameHeader(headers: IncomingHttpHeaders): string | undefined {+ const encoded = parseStringHeader(headers[fileTransportHeaders.ENCODED_NAME])+ if (!isNil(encoded) && encoded.length > 0) {+ const { data: decoded } = tryCatchSync(() => decodeURIComponent(encoded))+ if (!isNil(decoded)) {+ return decoded+ }+ }+ return parseStringHeader(headers[fileTransportHeaders.NAME])+}+function parseStringHeader(value: unknown): string | undefined {if (typeof value === 'string') {return valuediff --git a/packages/server/api/src/app/file/files-service.ts b/packages/server/api/src/app/file/files-service.tsindex 69b45a45e3a..95cac867916 100644--- a/packages/server/api/src/app/file/files-service.ts+++ b/packages/server/api/src/app/file/files-service.ts@@ -63,6 +63,7 @@ export const fileTransportHeaders = {READ_URL: 'x-ap-file-read-url',TYPE: 'x-ap-file-type',NAME: 'x-ap-file-name',+ ENCODED_NAME: 'x-ap-file-name-encoded',} as constexport const ENGINE_WRITABLE_FILE_TYPES: ReadonlySet<FileType> = new Set([diff --git a/packages/server/api/test/integration/ce/file/files-controller.test.ts b/packages/server/api/test/integration/ce/file/files-controller.test.tsindex f9c270dc2be..cad4c4611ac 100644--- a/packages/server/api/test/integration/ce/file/files-controller.test.ts+++ b/packages/server/api/test/integration/ce/file/files-controller.test.ts@@ -338,6 +338,49 @@ describe('Files Controller', () => {expect(getResponse?.headers['content-disposition']).toBe('attachment; filename="invoice.pdf"')})+ it.each([+ {+ scenario: 'a current engine (ASCII fallback + encoded header)',+ headers: { 'x-ap-file-name': 'Lettre _ Elodie.pdf', 'x-ap-file-name-encoded': encodeURIComponent('Lettre – Élodie.pdf') },+ storedName: 'Lettre – Élodie.pdf',+ },+ {+ scenario: 'an older engine (raw legacy header only)',+ headers: { 'x-ap-file-name': 'invoice.pdf' },+ storedName: 'invoice.pdf',+ },+ {+ scenario: 'a malformed encoded header',+ headers: { 'x-ap-file-name': 'report.pdf', 'x-ap-file-name-encoded': 'report%E0%A4%A.pdf' },+ storedName: 'report.pdf',+ },+ ])('stores the upload name sent by $scenario', async ({ headers, storedName }) => {+ const { mockProject, mockPlatform } = await mockAndSaveBasicSetup()+ const engineToken = await generateMockToken({+ type: PrincipalType.ENGINE,+ id: apId(),+ projectId: mockProject.id,+ platform: { id: mockPlatform.id },+ })+ const fileId = apId()++ const putResponse = await app!.inject({+ method: 'PUT',+ url: `/api/v1/files/${fileId}`,+ query: { token: engineToken },+ headers: {+ 'content-type': 'application/octet-stream',+ 'x-ap-file-type': FileType.FLOW_STEP_FILE,+ ...headers,+ },+ payload: Buffer.from('%PDF-1.4', 'utf-8'),+ })++ expect(putResponse?.statusCode).toBe(StatusCodes.OK)+ const file = await fileService(app!.log).getFileOrThrow({ fileId, projectId: mockProject.id })+ expect(file.fileName).toBe(storedName)+ })+it.each([{ fileName: 'résumé.pdf', expected: 'attachment; filename="résumé.pdf"' },{ fileName: '報告書.json', expected: 'attachment; filename="???.json"; filename*=UTF-8\'\'%E5%A0%B1%E5%91%8A%E6%9B%B8.json' },diff --git a/packages/server/engine/src/lib/api/engine-file-api.ts b/packages/server/engine/src/lib/api/engine-file-api.tsindex feddcc28efe..d0bd67b4817 100644--- a/packages/server/engine/src/lib/api/engine-file-api.ts+++ b/packages/server/engine/src/lib/api/engine-file-api.ts@@ -8,6 +8,7 @@ const zstdDecompress = promisify(zstdDecompressCallback)const READ_URL_HEADER = 'x-ap-file-read-url'const FILE_TYPE_HEADER = 'x-ap-file-type'const FILE_NAME_HEADER = 'x-ap-file-name'+const FILE_NAME_ENCODED_HEADER = 'x-ap-file-name-encoded'export const engineFileApi = {async upload({ engineToken, apiUrl, fileId, type, fileName, compression, data }: UploadParams): Promise<UploadResult> {@@ -130,7 +131,9 @@ function buildPutHeaders({ type, fileName, compression, contentLength }: BuildHeheaders['Content-Length'] = String(contentLength)}if (fileName) {- headers[FILE_NAME_HEADER] = fileName+ const wellFormedName = toWellFormed(fileName)+ headers[FILE_NAME_HEADER] = toAsciiHeaderValue(wellFormedName)+ headers[FILE_NAME_ENCODED_HEADER] = encodeURIComponent(wellFormedName)}if (compression === FileCompression.ZSTD) {headers['Content-Encoding'] = 'zstd'@@ -138,6 +141,14 @@ function buildPutHeaders({ type, fileName, compression, contentLength }: BuildHereturn headers}+function toWellFormed(value: string): string {+ return value.replace(/[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g, '\uFFFD')+}++function toAsciiHeaderValue(value: string): string {+ return value.normalize('NFKD').replace(/[\u0300-\u036f]/g, '').replace(/[^\x20-\x7e]/g, '_')+}+function stripApHeaders(headers: Record<string, string>): Record<string, string> {const result: Record<string, string> = {}for (const [key, value] of Object.entries(headers)) {diff --git a/packages/server/engine/test/engine-file-api.test.ts b/packages/server/engine/test/engine-file-api.test.tsindex e21d0b9dad3..7f2cd21fdd5 100644--- a/packages/server/engine/test/engine-file-api.test.ts+++ b/packages/server/engine/test/engine-file-api.test.ts@@ -1,5 +1,6 @@import { promisify } from 'node:util'import { zstdCompress as zstdCompressCallback } from 'node:zlib'+import { FileType } from '@activepieces/shared'import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'import { engineFileApi } from '../src/lib/api/engine-file-api'@@ -43,3 +44,53 @@ describe('engineFileApi.download zstd auto-decompression', () => {})})})++describe('engineFileApi.upload file-name headers', () => {+ afterEach(() => {+ vi.restoreAllMocks()+ })++ async function uploadAndCaptureHeaders(fileName: string): Promise<Headers> {+ const fetchSpy = vi.spyOn(global, 'fetch').mockResolvedValue(+ new Response(JSON.stringify({ readUrl: 'http://localhost:3000/v1/files/file-1' }), { status: 200 }),+ )+ await engineFileApi.upload({+ ...PARAMS,+ type: FileType.FLOW_STEP_FILE,+ fileName,+ data: Buffer.from('payload'),+ })+ return new Headers(fetchSpy.mock.calls[0][1]?.headers)+ }++ it.each([+ { fileName: 'résumé.pdf', ascii: 'resume.pdf' },+ { fileName: 'Lettre – Élodie.pdf', ascii: 'Lettre _ Elodie.pdf' },+ { fileName: '報告書.json', ascii: '___.json' },+ { fileName: 'evil\r\nX-Injected: 1.json', ascii: 'evil__X-Injected: 1.json' },+ ])('sends $fileName as printable ASCII plus a lossless encoded copy', async ({ fileName, ascii }) => {+ const headers = await uploadAndCaptureHeaders(fileName)++ expect(headers.get('x-ap-file-name')).toBe(ascii)+ expect(headers.get('x-ap-file-name')).toMatch(/^[\x20-\x7e]*$/)+ expect(decodeURIComponent(headers.get('x-ap-file-name-encoded') ?? '')).toBe(fileName)+ })++ it.each([+ { fileName: 'cut\uD83D.pdf', ascii: 'cut_.pdf', decoded: 'cut\uFFFD.pdf' },+ { fileName: '\uDE00lone-low.pdf', ascii: '_lone-low.pdf', decoded: '\uFFFDlone-low.pdf' },+ { fileName: 'whole \uD83D\uDE00.pdf', ascii: 'whole __.pdf', decoded: 'whole \uD83D\uDE00.pdf' },+ ])('replaces an unpaired surrogate instead of aborting the upload ($fileName)', async ({ fileName, ascii, decoded }) => {+ const headers = await uploadAndCaptureHeaders(fileName)++ expect(headers.get('x-ap-file-name')).toBe(ascii)+ expect(decodeURIComponent(headers.get('x-ap-file-name-encoded') ?? '')).toBe(decoded)+ })++ it('keeps a plain ASCII name unchanged in the legacy header', async () => {+ const headers = await uploadAndCaptureHeaders('invoice.pdf')++ expect(headers.get('x-ap-file-name')).toBe('invoice.pdf')+ expect(headers.get('x-ap-file-name-encoded')).toBe('invoice.pdf')+ })+})