Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(files): send upload file names as ASCII-safe headers (#15959)

Amr Elmohamady1849cf07fdce
diff --git a/brain/knowledge/data-storage-observability/file-storage.md b/brain/knowledge/data-storage-observability/file-storage.md
index 1c343f6136d..3f0cbdf11fa 100644
--- a/brain/knowledge/data-storage-observability/file-storage.md
+++ b/brain/knowledge/data-storage-observability/file-storage.md
@@ -32,6 +32,7 @@ The central service for persisting binary files, backing the execution engine an
- User-facing docs: [Large File Streaming](../../../docs/build-pieces/piece-reference/large-file-streaming.mdx).
### Gotchas
+- **The engine's upload file name travels in a header, so it must be ASCII on the wire.** `x-ap-file-name` used to carry the raw name; Node fetch sends a Latin-1 char like `é` as the single byte `0xE9`, which is invalid UTF-8, and chars above 255 (`–`, emoji, CJK) throw `Cannot convert argument to a ByteString` before the request leaves. Behind Cloudflare the first case is a 403 from the managed rule *Anomaly:Header — Invalid UTF-8 Encoding*, surfacing as `EngineFileUploadError: … 403 Forbidden` (Alan, 2026-09-29, French document names). The engine now sends an ASCII fallback in `x-ap-file-name` plus `encodeURIComponent(name)` in `x-ap-file-name-encoded`; the app prefers the encoded one and falls back to the raw header, so old/new engines and apps interoperate in both directions. Never put user text in a header raw.
- **On cloud the real key prefix is doubled — `<bucket>/<bucket>/…` — so ad-hoc CLI work against the bucket silently finds nothing.** Cloud's `AP_S3_ENDPOINT` embeds the bucket as a *path segment* (`https://<account>.r2.cloudflarestorage.com/ap-files-prod`), and `getS3Client` sets `forcePathStyle: true` whenever an endpoint is present, so the SDK appends the bucket again. An object the app stores as `pieces/x.tgz` actually lands at `ap-files-prod/pieces/x.tgz` inside bucket `ap-files-prod`. Symptom when you get it wrong: `aws s3 ls` returns `NoSuchKey` on a *prefix* listing (Aug 2026 — cost three attempts to spot). Strip the trailing `/<bucket>` from the endpoint for the CLI, then prepend the bucket name to the prefix; or better, do bulk work through `s3Helper` so the same client resolves the same paths. Note cloud's object store is **Cloudflare R2** while the piece CDN is a **DigitalOcean Space** — two different systems, easy to conflate.
- **The live piece-bundle cache sits *inside* the legacy one — `pieces/v2/` is nested under `pieces/`, so a recursive delete of `pieces/` takes the active cache with it.** `S3_PIECES_PREFIX` in `piece-bundle.ts` is `pieces/v2/`; the bare `pieces/` keys beside it are pre-CDN tarballs left by the older writer. Combined with the doubled prefix above, the real keys are `ap-files-prod/pieces/…` (legacy) and `ap-files-prod/pieces/v2/…` (live). Probe both with `wrangler r2 object get` before any prefix-wide operation — wiping v2 used to be survivable because it refilled lazily, at the cost of a burst of cache misses on every piece. **Both prefixes are now dead storage and safe to sweep:** the `BUNDLE_PIECE` job and the S3 mirror were removed, so `resolve()` no longer reads or writes either prefix and registry pieces redirect straight to the CDN (else npm). The mirror was deleted because it was written from whichever source was preferred *at cache time* and then took precedence over the CDN forever — a bucket populated before the CDN became preferred kept serving the unbundled npm build, which is what fans out one `@activepieces/shared` copy per piece in the engine (see the Workers page).
- **`deleteFiles` succeeding does not mean the objects are gone.** `DeleteObjectsCommand` reports per-object failures in `response.Errors` and does **not** throw, and `Quiet: true` only suppresses the success entries — so a request that "worked" can still have left objects behind. `deleteFiles` logs a warn naming the failure codes, which is all its callers (best-effort cleanup) need. Anything whose *correctness* depends on the prefix being empty afterwards would have to surface those keys and retry — but prefer not to need that at all: a reader that must not see the old objects should read from a new key prefix rather than race a delete against writers that may still be running old code.
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index 9f911b8a737..d252a3a436b 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -20,6 +20,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the
- `concurrency === 1` primes the sandbox to the full container RAM (cgroup-aware) via `primeFullContainerMemory()`.
### Gotchas
+- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.
- **The piece workspace pins `linker = "isolated"` in its own `bunfig.toml` — do not delete it, the engine's piece resolver depends on the layout bun chooses.** `pieceInstaller` generates a bun workspace at `<cache>/<version>/common` whose members are `pieces/<pieceName>-<pieceVersion>`, and the engine (`piece-loader.ts` → `resolveInstalledPieceEntry`) resolves a piece **only** at `pieces/<alias>/node_modules/<pieceName>`, which just the isolated linker produces. The workspace used to carry no bunfig and inherited whatever ancestor bunfig bun discovered above the cache mount (`/usr/src/app/bunfig.toml`); when bun instead uses the **hoisted** linker it writes the package as a real directory at the workspace root `node_modules/@activepieces/<name>` and leaves the member with no `node_modules` at all, so every run of that piece fails `PieceNotFoundError: Piece not found for package: <name>-<version>` while `bun install` still exits 0. It was silent and self-perpetuating — `markPiecesAsUsed` writes `ready` and `pieceCheckIfAlreadyInstalled` only looks for *a* `node_modules`, so the piece is reinstalled forever and never resolves. Sep 2026 production: 125/125 folders missing `node_modules` were exactly the 125 packages sitting hoisted at the workspace root; 7,400+ `EXECUTE_FLOW` jobs, 15+ platforms, ~60 official pieces, ~300/hr. Two things had to line up — bun 1.4.0 (Aug 31) changed the layout newly-installed members got, and removing the piece-upgrade Redis gate (#15236, deployed 2026-09-03 08:09 UTC) mass-upgraded every platform's flows to newer piece versions so almost every install was suddenly a *first-time* install. Diagnose on a worker with `for d in <cache>/v*/common/pieces/@activepieces/*/; do [ -d "$d/node_modules" ] || basename $d; done`. Fixed by pinning the linker plus a `LATEST_CACHE_VERSION` bump (v14 → v15) so the fleet rebuilds one clean workspace under the pinned layout rather than carrying a mixed one.
- **No PM2 anymore — the container is the supervision unit.** `docker-entrypoint.sh` launches the bootstrap scripts with plain `node --enable-source-maps` (removed the `pm2-runtime` + `/tmp/ecosystem.config.js` machinery). `APP`/`WORKER` `exec` a single node as PID 1; `WORKER_AND_APP` runs both and, if either exits, kills the other and exits non-zero so the orchestrator restarts the whole container. This drops PM2's *in-container* crash/OOM restart: an OOM-kill no longer silently recycles a child every ~4 min behind a `RestartCount: 0` (the 2026-07-26 wedge's supply of retry attempts — see below); the container now dies and is rescheduled instead. The historical incident notes below still describe the old PM2 behavior as it happened.
- **Version gate (rolling-deploy safety)**: dispatch requires an exact release match, enforced both sides via `versionsAreCompatible` (fail-closed — `undefined` or `UNKNOWN_VERSION` `'0.0.0'` is treated incompatible). App withholds jobs from a mismatched worker (`poll` returns null); worker pauses polling 10s. Ordinary mismatch self-heals on convergence; a read failure does not (cached for process life) and pages on-call once at startup via `assertReleaseReadable`.
diff --git a/packages/server/api/src/app/file/files-controller.ts b/packages/server/api/src/app/file/files-controller.ts
index df44816f9fb..9a3454c44bd 100644
--- a/packages/server/api/src/app/file/files-controller.ts
+++ b/packages/server/api/src/app/file/files-controller.ts
@@ -1,5 +1,6 @@
+import type { IncomingHttpHeaders } from 'node:http'
import { Readable } from 'node:stream'
-import { ActivepiecesError, ApId, assertNotNullOrUndefined, ErrorCode, isNil } from '@activepieces/core-utils'
+import { ActivepiecesError, ApId, assertNotNullOrUndefined, ErrorCode, isNil, tryCatchSync } from '@activepieces/core-utils'
import { ALL_PRINCIPAL_TYPES, EnginePrincipal, FileCompression, FileTransportQueryParams, FileType, Principal, PrincipalType } from '@activepieces/shared'
import contentDisposition from 'content-disposition'
import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'
@@ -33,7 +34,7 @@ export const filesController: FastifyPluginAsyncZod = async (app) => {
const token = (request.query as { token: string }).token
const principal = await verifyEnginePrincipal(token, request.log)
const fileType = parseFileTypeHeader(request.headers[fileTransportHeaders.TYPE])
- const fileName = parseStringHeader(request.headers[fileTransportHeaders.NAME])
+ const fileName = parseFileNameHeader(request.headers)
const contentEncoding = parseStringHeader(request.headers['content-encoding'])
const compression = contentEncoding === 'zstd' ? FileCompression.ZSTD : FileCompression.NONE
const contentLength = Number(request.headers['content-length'] ?? 0)
@@ -79,7 +80,7 @@ export const filesController: FastifyPluginAsyncZod = async (app) => {
const { fileId } = request.params
const principal = await verifyEnginePrincipal(request.query.token, request.log)
const fileType = parseFileTypeHeader(request.headers[fileTransportHeaders.TYPE])
- const fileName = parseStringHeader(request.headers[fileTransportHeaders.NAME])
+ const fileName = parseFileNameHeader(request.headers)
const contentEncoding = parseStringHeader(request.headers['content-encoding'])
const compression = contentEncoding === 'zstd' ? FileCompression.ZSTD : FileCompression.NONE
@@ -231,6 +232,17 @@ function parseFileTypeHeader(value: unknown): FileType {
return raw as FileType
}
+function parseFileNameHeader(headers: IncomingHttpHeaders): string | undefined {
+ const encoded = parseStringHeader(headers[fileTransportHeaders.ENCODED_NAME])
+ if (!isNil(encoded) && encoded.length > 0) {
+ const { data: decoded } = tryCatchSync(() => decodeURIComponent(encoded))
+ if (!isNil(decoded)) {
+ return decoded
+ }
+ }
+ return parseStringHeader(headers[fileTransportHeaders.NAME])
+}
+
function parseStringHeader(value: unknown): string | undefined {
if (typeof value === 'string') {
return value
diff --git a/packages/server/api/src/app/file/files-service.ts b/packages/server/api/src/app/file/files-service.ts
index 69b45a45e3a..95cac867916 100644
--- a/packages/server/api/src/app/file/files-service.ts
+++ b/packages/server/api/src/app/file/files-service.ts
@@ -63,6 +63,7 @@ export const fileTransportHeaders = {
READ_URL: 'x-ap-file-read-url',
TYPE: 'x-ap-file-type',
NAME: 'x-ap-file-name',
+ ENCODED_NAME: 'x-ap-file-name-encoded',
} as const
export const ENGINE_WRITABLE_FILE_TYPES: ReadonlySet<FileType> = new Set([
diff --git a/packages/server/api/test/integration/ce/file/files-controller.test.ts b/packages/server/api/test/integration/ce/file/files-controller.test.ts
index f9c270dc2be..cad4c4611ac 100644
--- a/packages/server/api/test/integration/ce/file/files-controller.test.ts
+++ b/packages/server/api/test/integration/ce/file/files-controller.test.ts
@@ -338,6 +338,49 @@ describe('Files Controller', () => {
expect(getResponse?.headers['content-disposition']).toBe('attachment; filename="invoice.pdf"')
})
+ it.each([
+ {
+ scenario: 'a current engine (ASCII fallback + encoded header)',
+ headers: { 'x-ap-file-name': 'Lettre _ Elodie.pdf', 'x-ap-file-name-encoded': encodeURIComponent('Lettre – Élodie.pdf') },
+ storedName: 'Lettre – Élodie.pdf',
+ },
+ {
+ scenario: 'an older engine (raw legacy header only)',
+ headers: { 'x-ap-file-name': 'invoice.pdf' },
+ storedName: 'invoice.pdf',
+ },
+ {
+ scenario: 'a malformed encoded header',
+ headers: { 'x-ap-file-name': 'report.pdf', 'x-ap-file-name-encoded': 'report%E0%A4%A.pdf' },
+ storedName: 'report.pdf',
+ },
+ ])('stores the upload name sent by $scenario', async ({ headers, storedName }) => {
+ const { mockProject, mockPlatform } = await mockAndSaveBasicSetup()
+ const engineToken = await generateMockToken({
+ type: PrincipalType.ENGINE,
+ id: apId(),
+ projectId: mockProject.id,
+ platform: { id: mockPlatform.id },
+ })
+ const fileId = apId()
+
+ const putResponse = await app!.inject({
+ method: 'PUT',
+ url: `/api/v1/files/${fileId}`,
+ query: { token: engineToken },
+ headers: {
+ 'content-type': 'application/octet-stream',
+ 'x-ap-file-type': FileType.FLOW_STEP_FILE,
+ ...headers,
+ },
+ payload: Buffer.from('%PDF-1.4', 'utf-8'),
+ })
+
+ expect(putResponse?.statusCode).toBe(StatusCodes.OK)
+ const file = await fileService(app!.log).getFileOrThrow({ fileId, projectId: mockProject.id })
+ expect(file.fileName).toBe(storedName)
+ })
+
it.each([
{ fileName: 'résumé.pdf', expected: 'attachment; filename="résumé.pdf"' },
{ fileName: '報告書.json', expected: 'attachment; filename="???.json"; filename*=UTF-8\'\'%E5%A0%B1%E5%91%8A%E6%9B%B8.json' },
diff --git a/packages/server/engine/src/lib/api/engine-file-api.ts b/packages/server/engine/src/lib/api/engine-file-api.ts
index feddcc28efe..d0bd67b4817 100644
--- a/packages/server/engine/src/lib/api/engine-file-api.ts
+++ b/packages/server/engine/src/lib/api/engine-file-api.ts
@@ -8,6 +8,7 @@ const zstdDecompress = promisify(zstdDecompressCallback)
const READ_URL_HEADER = 'x-ap-file-read-url'
const FILE_TYPE_HEADER = 'x-ap-file-type'
const FILE_NAME_HEADER = 'x-ap-file-name'
+const FILE_NAME_ENCODED_HEADER = 'x-ap-file-name-encoded'
export const engineFileApi = {
async upload({ engineToken, apiUrl, fileId, type, fileName, compression, data }: UploadParams): Promise<UploadResult> {
@@ -130,7 +131,9 @@ function buildPutHeaders({ type, fileName, compression, contentLength }: BuildHe
headers['Content-Length'] = String(contentLength)
}
if (fileName) {
- headers[FILE_NAME_HEADER] = fileName
+ const wellFormedName = toWellFormed(fileName)
+ headers[FILE_NAME_HEADER] = toAsciiHeaderValue(wellFormedName)
+ headers[FILE_NAME_ENCODED_HEADER] = encodeURIComponent(wellFormedName)
}
if (compression === FileCompression.ZSTD) {
headers['Content-Encoding'] = 'zstd'
@@ -138,6 +141,14 @@ function buildPutHeaders({ type, fileName, compression, contentLength }: BuildHe
return headers
}
+function toWellFormed(value: string): string {
+ return value.replace(/[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g, '\uFFFD')
+}
+
+function toAsciiHeaderValue(value: string): string {
+ return value.normalize('NFKD').replace(/[\u0300-\u036f]/g, '').replace(/[^\x20-\x7e]/g, '_')
+}
+
function stripApHeaders(headers: Record<string, string>): Record<string, string> {
const result: Record<string, string> = {}
for (const [key, value] of Object.entries(headers)) {
diff --git a/packages/server/engine/test/engine-file-api.test.ts b/packages/server/engine/test/engine-file-api.test.ts
index e21d0b9dad3..7f2cd21fdd5 100644
--- a/packages/server/engine/test/engine-file-api.test.ts
+++ b/packages/server/engine/test/engine-file-api.test.ts
@@ -1,5 +1,6 @@
import { promisify } from 'node:util'
import { zstdCompress as zstdCompressCallback } from 'node:zlib'
+import { FileType } from '@activepieces/shared'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { engineFileApi } from '../src/lib/api/engine-file-api'
@@ -43,3 +44,53 @@ describe('engineFileApi.download zstd auto-decompression', () => {
})
})
})
+
+describe('engineFileApi.upload file-name headers', () => {
+ afterEach(() => {
+ vi.restoreAllMocks()
+ })
+
+ async function uploadAndCaptureHeaders(fileName: string): Promise<Headers> {
+ const fetchSpy = vi.spyOn(global, 'fetch').mockResolvedValue(
+ new Response(JSON.stringify({ readUrl: 'http://localhost:3000/v1/files/file-1' }), { status: 200 }),
+ )
+ await engineFileApi.upload({
+ ...PARAMS,
+ type: FileType.FLOW_STEP_FILE,
+ fileName,
+ data: Buffer.from('payload'),
+ })
+ return new Headers(fetchSpy.mock.calls[0][1]?.headers)
+ }
+
+ it.each([
+ { fileName: 'résumé.pdf', ascii: 'resume.pdf' },
+ { fileName: 'Lettre – Élodie.pdf', ascii: 'Lettre _ Elodie.pdf' },
+ { fileName: '報告書.json', ascii: '___.json' },
+ { fileName: 'evil\r\nX-Injected: 1.json', ascii: 'evil__X-Injected: 1.json' },
+ ])('sends $fileName as printable ASCII plus a lossless encoded copy', async ({ fileName, ascii }) => {
+ const headers = await uploadAndCaptureHeaders(fileName)
+
+ expect(headers.get('x-ap-file-name')).toBe(ascii)
+ expect(headers.get('x-ap-file-name')).toMatch(/^[\x20-\x7e]*$/)
+ expect(decodeURIComponent(headers.get('x-ap-file-name-encoded') ?? '')).toBe(fileName)
+ })
+
+ it.each([
+ { fileName: 'cut\uD83D.pdf', ascii: 'cut_.pdf', decoded: 'cut\uFFFD.pdf' },
+ { fileName: '\uDE00lone-low.pdf', ascii: '_lone-low.pdf', decoded: '\uFFFDlone-low.pdf' },
+ { fileName: 'whole \uD83D\uDE00.pdf', ascii: 'whole __.pdf', decoded: 'whole \uD83D\uDE00.pdf' },
+ ])('replaces an unpaired surrogate instead of aborting the upload ($fileName)', async ({ fileName, ascii, decoded }) => {
+ const headers = await uploadAndCaptureHeaders(fileName)
+
+ expect(headers.get('x-ap-file-name')).toBe(ascii)
+ expect(decodeURIComponent(headers.get('x-ap-file-name-encoded') ?? '')).toBe(decoded)
+ })
+
+ it('keeps a plain ASCII name unchanged in the legacy header', async () => {
+ const headers = await uploadAndCaptureHeaders('invoice.pdf')
+
+ expect(headers.get('x-ap-file-name')).toBe('invoice.pdf')
+ expect(headers.get('x-ap-file-name-encoded')).toBe('invoice.pdf')
+ })
+})