Skip to content

⭐️⭐️⭐️ Open-source, self-hosted alternative to Linear and Plane. Project management and issue tracking where teams and AI agents work side by side to plan and ship products.

fix(worker): reject a failed attachment download (#471)

Thomas Gallaway4fa48da0f5d4
diff --git a/apps/worker/AGENTS.md b/apps/worker/AGENTS.md
index a889c50c..84ecb2b6 100644
--- a/apps/worker/AGENTS.md
+++ b/apps/worker/AGENTS.md
@@ -65,8 +65,10 @@ running to completion in one.
`putObject`/`attachmentObjectKey`) is checked before the upload and once
more inside the same advisory lock (`lockAttachmentStorage`) an interactive
upload takes, so the two never both pass a check that only one of them can
- actually fit. A rejected attachment (`AttachmentRejectedError`) is logged
- and skipped, not treated as a failed tick.
+ actually fit. A download that answers anything but 2xx (an error page, a
+ redirect) is refused before its body is read (`attachment-download.ts`). A
+ rejected attachment (`AttachmentRejectedError`) is logged and skipped, not
+ treated as a failed tick.
## Invariants
@@ -90,7 +92,8 @@ running to completion in one.
- **Pure logic stays dependency-free.** `backoff.ts`, `signature.ts`, and
`isRetryableStatus` import nothing from `@repo/db`, so unit tests run without a
database. Keep DB access in `store.ts`. Same split for imports: `canonical.ts`,
- `reader.ts`, `plane-adapter.ts`, and `cross-reference.ts` import nothing from
+ `reader.ts`, `plane-adapter.ts`, `cross-reference.ts`, and
+ `attachment-download.ts` import nothing from
`@repo/db` (its state-category normalization, markdown conversion,
cursor/rate-limit, and cross-reference matching logic are unit-tested
directly); `@repo/db` access stays in `import-store.ts`.
diff --git a/apps/worker/src/__tests__/unit/attachment-download.test.ts b/apps/worker/src/__tests__/unit/attachment-download.test.ts
new file mode 100644
index 00000000..3d6e38d1
--- /dev/null
+++ b/apps/worker/src/__tests__/unit/attachment-download.test.ts
@@ -0,0 +1,53 @@
+import { describe, it, expect } from 'bun:test';
+import type { PinnedRequestInit } from '@repo/net';
+import {
+ downloadAttachment,
+ AttachmentRejectedError,
+ type AttachmentFetch,
+} from '../../attachment-download';
+
+const FILE_URL = 'https://files.example.test/report.pdf';
+
+function respondWith(response: Response): AttachmentFetch {
+ return async () => response;
+}
+
+describe('downloadAttachment', () => {
+ it('returns the body of a 2xx response', async () => {
+ const bytes = await downloadAttachment(FILE_URL, 1024, respondWith(new Response('file bytes')));
+ expect(bytes.toString()).toBe('file bytes');
+ });
+
+ it.each([401, 403, 404, 500, 503])(
+ 'rejects an HTTP %d response instead of returning its body as the file',
+ async (status) => {
+ const error = await downloadAttachment(
+ FILE_URL,
+ 1024,
+ respondWith(new Response('<Error>AccessDenied</Error>', { status })),
+ ).catch((e: unknown) => e);
+ expect(error).toBeInstanceOf(AttachmentRejectedError);
+ expect((error as Error).message).toContain(`HTTP ${status}`);
+ },
+ );
+
+ it('rejects a redirect, which pinnedFetch returns without following', async () => {
+ const redirect = new Response(null, {
+ status: 302,
+ headers: { location: 'https://elsewhere.example.test/report.pdf' },
+ });
+ await expect(downloadAttachment(FILE_URL, 1024, respondWith(redirect))).rejects.toBeInstanceOf(
+ AttachmentRejectedError,
+ );
+ });
+
+ it('downloads with the byte limit and a timeout', async () => {
+ let seen: { url: string; init: PinnedRequestInit | undefined } | null = null;
+ const fetch: AttachmentFetch = async (url, init) => {
+ seen = { url, init };
+ return new Response('x');
+ };
+ await downloadAttachment(FILE_URL, 2048, fetch);
+ expect(seen!).toEqual({ url: FILE_URL, init: { timeoutMs: 30_000, maxBytes: 2048 } });
+ });
+});
diff --git a/apps/worker/src/attachment-download.ts b/apps/worker/src/attachment-download.ts
new file mode 100644
index 00000000..da1659f9
--- /dev/null
+++ b/apps/worker/src/attachment-download.ts
@@ -0,0 +1,24 @@
+import { pinnedFetch } from '@repo/net';
+
+// An attachment that cannot be stored: over this instance's size, type or quota
+// limits, or a download that did not return the file. The Attachments phase logs
+// and skips it instead of failing the tick.
+export class AttachmentRejectedError extends Error {}
+
+export type AttachmentFetch = typeof pinnedFetch;
+
+const DOWNLOAD_TIMEOUT_MS = 30_000;
+
+// pinnedFetch returns any status as-is, a redirect included, so an error page
+// must be refused here or it would be stored as the file.
+export async function downloadAttachment(
+ url: string,
+ maxBytes: number,
+ fetch: AttachmentFetch = pinnedFetch,
+): Promise<Buffer> {
+ const res = await fetch(url, { timeoutMs: DOWNLOAD_TIMEOUT_MS, maxBytes });
+ if (!res.ok) {
+ throw new AttachmentRejectedError(`the download answered HTTP ${res.status}`);
+ }
+ return Buffer.from(await res.arrayBuffer());
+}
diff --git a/apps/worker/src/import-store.ts b/apps/worker/src/import-store.ts
index f7a26ae0..e3a86eec 100644
--- a/apps/worker/src/import-store.ts
+++ b/apps/worker/src/import-store.ts
@@ -30,6 +30,9 @@ import {
} from '@repo/storage';
import type { CanonicalState, CanonicalLabel, CanonicalCycle } from './canonical';
import type { PlaneCredential } from './plane-adapter';
+import { AttachmentRejectedError } from './attachment-download';
+
+export { AttachmentRejectedError };
// All @repo/db access for the Plane import: claiming due import_job rows (the
// same FOR UPDATE SKIP LOCKED + lease pattern as store.ts uses for
@@ -699,12 +702,6 @@ export async function findProjectMemberUserId(
return rows[0]?.userId ?? null;
}
-// Thrown for an attachment that simply does not fit this instance's own
-// configured limits (file size, mime type, or project quota) — the Attachments
-// phase catches this specifically and moves on to the next attachment, rather
-// than treating it as a failed tick the way a network or database error is.
-export class AttachmentRejectedError extends Error {}
-
export interface NewLocalAttachment {
projectId: number;
issueId: number;
diff --git a/apps/worker/src/import-worker.ts b/apps/worker/src/import-worker.ts
index 1b97db24..900e0dd6 100644
--- a/apps/worker/src/import-worker.ts
+++ b/apps/worker/src/import-worker.ts
@@ -1,4 +1,3 @@
-import { pinnedFetch } from '@repo/net';
import { getStorageSettings, mimeAllowed, MB } from '@repo/db';
import { startPollLoop, type WorkerHandle } from './poll-loop';
import { intEnv } from './env';
@@ -7,6 +6,7 @@ import { PlaneReader, PlaneRateLimitedError } from './plane-adapter';
import type { SourceReader } from './reader';
import type { CanonicalComment, CanonicalStateCategory } from './canonical';
import { extractCrossReferences, applyCrossReferenceReplacements } from './cross-reference';
+import { downloadAttachment } from './attachment-download';
import {
claimDueImportJobs,
decryptImportCredential,
@@ -480,11 +480,7 @@ async function runAttachments(job: ClaimedImportJob, reader: SourceReader): Prom
);
}
const url = await reader.resolveAttachmentDownloadUrl(record.sourceId, attachment.sourceId);
- const res = await pinnedFetch(url, {
- timeoutMs: 30_000,
- maxBytes: limits.maxAttachmentMb * MB,
- });
- const bytes = Buffer.from(await res.arrayBuffer());
+ const bytes = await downloadAttachment(url, limits.maxAttachmentMb * MB);
await createLocalAttachmentAndRecord(job.id, attachment.sourceId, {
projectId: job.projectId,
issueId: record.localId,
diff --git a/docs/dev/plane-import.md b/docs/dev/plane-import.md
index 7a1678be..2e8a8f35 100644
--- a/docs/dev/plane-import.md
+++ b/docs/dev/plane-import.md
@@ -92,9 +92,12 @@ and mime type are checked against the instance's own upload settings (`getStorag
checked once before the upload (so an already-over-quota file is never written to the
object store at all) and once more inside the same advisory lock
(`lockAttachmentStorage`) an interactive upload takes, immediately before the insert — the
-two never both pass a check the project can only actually fit one of. A rejected attachment
-(`AttachmentRejectedError`, `import-store.ts`) is logged and skipped, not treated as a
-failed tick that retries. Re-running an import reuses an existing attachment by exact
+two never both pass a check the project can only actually fit one of. A download that
+answers anything but 2xx is refused before its body is read (`downloadAttachment`,
+`attachment-download.ts`): `pinnedFetch` returns error statuses and redirects as-is, and
+their body is not the file. A rejected attachment (`AttachmentRejectedError`) is logged and
+skipped, not treated as a failed tick that retries. Re-running an import reuses an existing
+attachment by exact
filename on the same issue, the same reuse-by-content-match philosophy every other entity
here already has.
diff --git a/docs/importing-from-plane.md b/docs/importing-from-plane.md
index 72f519a7..03e688cd 100644
--- a/docs/importing-from-plane.md
+++ b/docs/importing-from-plane.md
@@ -45,7 +45,8 @@ of them affect data you may not notice is missing until later.
- **Attachment files**, downloaded and attached to the same issue they were on in Plane.
A file is skipped, not imported, when it is larger than this instance's own upload
limit or of a file type this instance doesn't accept — the same limits an ordinary
- attachment upload on this instance is held to. A skipped file is only noted in the
+ attachment upload on this instance is held to — or when Plane's file storage answers
+ the download with an error instead of the file. A skipped file is only noted in the
server log, not shown anywhere in this screen today. Re-running an import reuses a
file already attached with the same name on the same issue, rather than attaching it
a second time.