Skip to content

⭐️⭐️⭐️ Open-source, self-hosted alternative to Linear and Plane. Project management and issue tracking where teams and AI agents work side by side to plan and ship products.

refactor(worker): make the import pipeline source-neutral (#477)

Thomas Gallaway1acb20ccced4
diff --git a/apps/worker/AGENTS.md b/apps/worker/AGENTS.md
index 84ecb2b6..0dd6fc8c 100644
--- a/apps/worker/AGENTS.md
+++ b/apps/worker/AGENTS.md
@@ -31,7 +31,14 @@ running to completion in one.
- `canonical.ts` — the source-independent shape an adapter produces
(`CanonicalIssue`, `CanonicalState`, ...). Plain types, no logic.
-- `reader.ts` — the `SourceReader` port a source adapter implements.
+- `reader.ts` — the `SourceReader` port a source adapter implements, and the
+ `SourceRateLimitedError` an adapter throws when the source's rate limit is reached.
+- `import-sources.ts` — one entry per `import_job.source`: its credential and
+ config types, how to build its reader, and its issue-key prefix for Rewrite. A
+ new source adds an entry here; the phases do not change.
+- `import-retry.ts` — what a failed tick does: a rate limit is waited out and
+ never fails the job by itself, any other error retries with backoff until
+ `MAX_ATTEMPTS` in a row.
- `plane-adapter.ts` — the only implementation today. HTTP against a Plane
instance via `pinnedFetch`, using the credential decrypted from the job row
(base URL, workspace slug, API key — never from env, since this has to work
@@ -92,8 +99,8 @@ running to completion in one.
- **Pure logic stays dependency-free.** `backoff.ts`, `signature.ts`, and
`isRetryableStatus` import nothing from `@repo/db`, so unit tests run without a
database. Keep DB access in `store.ts`. Same split for imports: `canonical.ts`,
- `reader.ts`, `plane-adapter.ts`, `cross-reference.ts`, and
- `attachment-download.ts` import nothing from
+ `reader.ts`, `plane-adapter.ts`, `cross-reference.ts`, `attachment-download.ts`,
+ `import-sources.ts`, and `import-retry.ts` import nothing from
`@repo/db` (its state-category normalization, markdown conversion,
cursor/rate-limit, and cross-reference matching logic are unit-tested
directly); `@repo/db` access stays in `import-store.ts`.
@@ -110,7 +117,7 @@ All via env with defaults (see `src/config.ts`): `WEBHOOK_POLL_INTERVAL_MS`,
`WEBHOOK_CLEANUP_EVERY_TICKS`. Only `DATABASE_URL` is required for webhook
delivery. Notification delivery also needs `APP_ENCRYPTION_KEY` (the same value the
api uses) to read the stored provider credentials, and so does source import (it
-decrypts the stored Plane credential with it too). `IMPORT_POLL_INTERVAL_MS` tunes
+decrypts the stored source credential with it too). `IMPORT_POLL_INTERVAL_MS` tunes
the import worker's poll interval. The Attachments phase needs the same `S3_*`
variables (`S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`,
`S3_REGION`, `S3_FORCE_PATH_STYLE`) the api reads for its own uploads — both
diff --git a/apps/worker/src/__tests__/integration/import-store.test.ts b/apps/worker/src/__tests__/integration/import-store.test.ts
index e2137544..0a327232 100644
--- a/apps/worker/src/__tests__/integration/import-store.test.ts
+++ b/apps/worker/src/__tests__/integration/import-store.test.ts
@@ -16,9 +16,13 @@ import {
type StorageSettings,
} from '@repo/db';
import { eq } from 'drizzle-orm';
+import { encryptSecret } from '@repo/crypto';
import { getObject } from '@repo/storage';
+import { readerForJob, sourceProjectKeyForJob } from '../../import-sources';
+import { PlaneReader } from '../../plane-adapter';
import {
claimDueImportJobs,
+ decryptImportCredential,
createLocalStateAndRecord,
createLocalIssueAndRecord,
createLocalAttachmentAndRecord,
@@ -132,6 +136,45 @@ describe('claimDueImportJobs', () => {
});
});
+describe('decryptImportCredential', () => {
+ it('resumes a Plane job the api created, mid-phase, with the same reader and key', async () => {
+ const { projectId, userId } = await makeProject();
+ const credential = {
+ baseUrl: 'https://plane.example.test',
+ workspaceSlug: 'acme',
+ apiKey: 'plane-api-key',
+ };
+ const encrypted = encryptSecret(JSON.stringify(credential));
+ const [row] = await db
+ .insert(importJob)
+ .values({
+ projectId,
+ createdByUserId: userId,
+ source: 'plane',
+ phase: 'create',
+ config: {
+ planeProjectId: 'project-1',
+ planeProjectKey: 'ROOMS',
+ unmatchedUserPolicy: 'unassigned',
+ },
+ cursor: { lastRecordId: 41 },
+ credentialCiphertext: encrypted.ciphertext,
+ credentialIv: encrypted.iv,
+ credentialAuthTag: encrypted.authTag,
+ nextAttemptAt: new Date(Date.now() - 1000),
+ })
+ .returning({ id: importJob.id });
+
+ const job = (await claimDueImportJobs(1000)).find((j) => j.id === row!.id);
+
+ expect(job).toMatchObject({ source: 'plane', phase: 'create', cursor: { lastRecordId: 41 } });
+ const decrypted = decryptImportCredential(job!);
+ expect(decrypted).toEqual(credential);
+ expect(readerForJob(job!, decrypted)).toBeInstanceOf(PlaneReader);
+ expect(sourceProjectKeyForJob(job!)).toBe('ROOMS');
+ });
+});
+
describe('createLocalStateAndRecord', () => {
it('creates a new column and records the mapping', async () => {
const { projectId, userId } = await makeProject();
diff --git a/apps/worker/src/__tests__/unit/attachment-download.test.ts b/apps/worker/src/__tests__/unit/attachment-download.test.ts
index 3d6e38d1..60ee13d3 100644
--- a/apps/worker/src/__tests__/unit/attachment-download.test.ts
+++ b/apps/worker/src/__tests__/unit/attachment-download.test.ts
@@ -14,7 +14,11 @@ function respondWith(response: Response): AttachmentFetch {
describe('downloadAttachment', () => {
it('returns the body of a 2xx response', async () => {
- const bytes = await downloadAttachment(FILE_URL, 1024, respondWith(new Response('file bytes')));
+ const bytes = await downloadAttachment(
+ { url: FILE_URL },
+ 1024,
+ respondWith(new Response('file bytes')),
+ );
expect(bytes.toString()).toBe('file bytes');
});
@@ -22,7 +26,7 @@ describe('downloadAttachment', () => {
'rejects an HTTP %d response instead of returning its body as the file',
async (status) => {
const error = await downloadAttachment(
- FILE_URL,
+ { url: FILE_URL },
1024,
respondWith(new Response('<Error>AccessDenied</Error>', { status })),
).catch((e: unknown) => e);
@@ -36,9 +40,9 @@ describe('downloadAttachment', () => {
status: 302,
headers: { location: 'https://elsewhere.example.test/report.pdf' },
});
- await expect(downloadAttachment(FILE_URL, 1024, respondWith(redirect))).rejects.toBeInstanceOf(
- AttachmentRejectedError,
- );
+ await expect(
+ downloadAttachment({ url: FILE_URL }, 1024, respondWith(redirect)),
+ ).rejects.toBeInstanceOf(AttachmentRejectedError);
});
it('downloads with the byte limit and a timeout', async () => {
@@ -47,7 +51,25 @@ describe('downloadAttachment', () => {
seen = { url, init };
return new Response('x');
};
- await downloadAttachment(FILE_URL, 2048, fetch);
+ await downloadAttachment({ url: FILE_URL }, 2048, fetch);
expect(seen!).toEqual({ url: FILE_URL, init: { timeoutMs: 30_000, maxBytes: 2048 } });
});
+
+ it('sends the headers the source asked for', async () => {
+ let seen: PinnedRequestInit | undefined;
+ const fetch: AttachmentFetch = async (_url, init) => {
+ seen = init;
+ return new Response('x');
+ };
+ await downloadAttachment(
+ { url: FILE_URL, headers: { Authorization: 'source-token' } },
+ 2048,
+ fetch,
+ );
+ expect(seen).toEqual({
+ headers: { Authorization: 'source-token' },
+ timeoutMs: 30_000,
+ maxBytes: 2048,
+ });
+ });
});
diff --git a/apps/worker/src/__tests__/unit/import-retry.test.ts b/apps/worker/src/__tests__/unit/import-retry.test.ts
new file mode 100644
index 00000000..4556d871
--- /dev/null
+++ b/apps/worker/src/__tests__/unit/import-retry.test.ts
@@ -0,0 +1,41 @@
+import { describe, it, expect } from 'bun:test';
+import { tickErrorOutcome, MAX_ATTEMPTS } from '../../import-retry';
+import { SourceRateLimitedError } from '../../reader';
+
+describe('tickErrorOutcome', () => {
+ it('waits out a rate limit with the exact last_error the Settings page matches', () => {
+ expect(tickErrorOutcome(new SourceRateLimitedError(45_000), 1)).toEqual({
+ action: 'retry',
+ delayMs: 45_000,
+ lastError: 'rate limited',
+ });
+ });
+
+ it('never fails a job for a rate limit, however many attempts ran', () => {
+ expect(tickErrorOutcome(new SourceRateLimitedError(1000), MAX_ATTEMPTS + 5).action).toBe(
+ 'retry',
+ );
+ });
+
+ it('retries any other error with backoff and its message', () => {
+ const outcome = tickErrorOutcome(new Error('Plane request failed: HTTP 502'), 1);
+ expect(outcome).toMatchObject({
+ action: 'retry',
+ lastError: 'Plane request failed: HTTP 502',
+ });
+ const delayMs = (outcome as { delayMs: number }).delayMs;
+ expect(delayMs).toBeGreaterThanOrEqual(30_000);
+ expect(delayMs).toBeLessThanOrEqual(60_000);
+ });
+
+ it('fails the job once the consecutive attempts reach the limit', () => {
+ expect(tickErrorOutcome(new Error('still broken'), MAX_ATTEMPTS)).toEqual({
+ action: 'fail',
+ lastError: 'still broken',
+ });
+ });
+
+ it('records a thrown non-Error as its string form', () => {
+ expect(tickErrorOutcome('boom', MAX_ATTEMPTS)).toEqual({ action: 'fail', lastError: 'boom' });
+ });
+});
diff --git a/apps/worker/src/__tests__/unit/import-sources.test.ts b/apps/worker/src/__tests__/unit/import-sources.test.ts
new file mode 100644
index 00000000..4e5541a9
--- /dev/null
+++ b/apps/worker/src/__tests__/unit/import-sources.test.ts
@@ -0,0 +1,46 @@
+import { describe, it, expect } from 'bun:test';
+import { readerForJob, sourceProjectKeyForJob } from '../../import-sources';
+import { PlaneReader } from '../../plane-adapter';
+
+const planeCredential = {
+ baseUrl: 'https://plane.example.test',
+ workspaceSlug: 'acme',
+ apiKey: 'plane-api-key',
+};
+
+describe('readerForJob', () => {
+ it('builds a Plane reader for a Plane job', () => {
+ const job = { id: 7, source: 'plane', config: { planeProjectId: 'project-1' } };
+ expect(readerForJob(job, planeCredential)).toBeInstanceOf(PlaneReader);
+ });
+
+ it('refuses a Plane job with no source project', () => {
+ const job = { id: 7, source: 'plane', config: {} };
+ expect(() => readerForJob(job, planeCredential)).toThrow(
+ 'import job 7 has no source project configured',
+ );
+ });
+
+ it.each(['jira', 'linear', 'toString', ''])('refuses the unsupported source "%s"', (source) => {
+ const job = { id: 7, source, config: { planeProjectId: 'project-1' } };
+ expect(() => readerForJob(job, planeCredential)).toThrow(
+ `import job 7 has unsupported source "${source}"`,
+ );
+ });
+});
+
+describe('sourceProjectKeyForJob', () => {
+ it("returns a Plane job's project identifier", () => {
+ const job = {
+ id: 7,
+ source: 'plane',
+ config: { planeProjectId: 'p', planeProjectKey: 'ROOMS' },
+ };
+ expect(sourceProjectKeyForJob(job)).toBe('ROOMS');
+ });
+
+ it('returns null for a Plane job created before the Rewrite phase existed', () => {
+ const job = { id: 7, source: 'plane', config: { planeProjectId: 'p' } };
+ expect(sourceProjectKeyForJob(job)).toBeNull();
+ });
+});
diff --git a/apps/worker/src/__tests__/unit/plane-adapter.test.ts b/apps/worker/src/__tests__/unit/plane-adapter.test.ts
index f261718b..10949ac9 100644
--- a/apps/worker/src/__tests__/unit/plane-adapter.test.ts
+++ b/apps/worker/src/__tests__/unit/plane-adapter.test.ts
@@ -1,10 +1,13 @@
import { describe, it, expect } from 'bun:test';
+import type { PinnedRequestInit } from '@repo/net';
import {
+ PlaneReader,
normalizeStateCategory,
nextPageCursor,
rateLimitBackoffMs,
htmlToMarkdown,
} from '../../plane-adapter';
+import { SourceRateLimitedError } from '../../reader';
describe('normalizeStateCategory', () => {
it("maps the British-spelled cancelled to itsaplan's canceled", () => {
@@ -137,3 +140,79 @@ describe('htmlToMarkdown', () => {
expect(htmlToMarkdown('')).toBe('');
});
});
+
+interface SentRequest {
+ url: string;
+ init: PinnedRequestInit | undefined;
+}
+
+function readerAnswering(response: Response, sent: SentRequest[] = []): PlaneReader {
+ const credential = {
+ baseUrl: 'https://plane.example.test/',
+ workspaceSlug: 'acme',
+ apiKey: 'plane-api-key',
+ };
+ return new PlaneReader(credential, 'project-1', async (url, init) => {
+ sent.push({ url, init });
+ return response;
+ });
+}
+
+describe('PlaneReader', () => {
+ it('sends the API key to the project-scoped Plane endpoint', async () => {
+ const sent: SentRequest[] = [];
+ const reader = readerAnswering(
+ Response.json({ results: [], next_cursor: null, next_page_results: false }),
+ sent,
+ );
+ expect(await reader.listStates()).toEqual([]);
+ expect(sent).toEqual([
+ {
+ url: 'https://plane.example.test/api/v1/workspaces/acme/projects/project-1/states/',
+ init: { headers: { 'X-Api-Key': 'plane-api-key' }, timeoutMs: 15_000 },
+ },
+ ]);
+ });
+
+ it('throws the source-neutral rate-limit error on a 429', async () => {
+ const reader = readerAnswering(
+ new Response('', { status: 429, headers: { 'retry-after': '30' } }),
+ );
+ const error = await reader.listStates().catch((e: unknown) => e);
+ expect(error).toBeInstanceOf(SourceRateLimitedError);
+ expect((error as SourceRateLimitedError).retryAfterMs).toBe(30_000);
+ expect((error as Error).message).toBe('Plane rate limit reached');
+ });
+
+ it('throws it when the remaining budget reaches zero on a 200', async () => {
+ const reader = readerAnswering(
+ Response.json(
+ { results: [], next_cursor: null, next_page_results: false },
+ { headers: { 'x-ratelimit-remaining': '0' } },
+ ),
+ );
+ await expect(reader.listStates()).rejects.toBeInstanceOf(SourceRateLimitedError);
+ });
+
+ it('resolves an attachment to the redirect target, with no Plane credential attached', async () => {
+ const sent: SentRequest[] = [];
+ const target = 'https://s3.example.test/bucket/report.pdf?X-Amz-Signature=abc';
+ const reader = readerAnswering(
+ new Response(null, { status: 302, headers: { location: target } }),
+ sent,
+ );
+ expect(await reader.resolveAttachmentDownload('issue-1', 'att-1')).toStrictEqual({
+ url: target,
+ });
+ expect(sent[0]!.url).toBe(
+ 'https://plane.example.test/api/v1/workspaces/acme/projects/project-1/work-items/issue-1/attachments/att-1/',
+ );
+ });
+
+ it('refuses an attachment resolve that did not redirect', async () => {
+ const reader = readerAnswering(new Response('{}', { status: 200 }));
+ await expect(reader.resolveAttachmentDownload('issue-1', 'att-1')).rejects.toThrow(
+ 'Plane attachment resolve did not redirect: GET /work-items/issue-1/attachments/att-1/ -> HTTP 200',
+ );
+ });
+});
diff --git a/apps/worker/src/attachment-download.ts b/apps/worker/src/attachment-download.ts
index da1659f9..2f81f95e 100644
--- a/apps/worker/src/attachment-download.ts
+++ b/apps/worker/src/attachment-download.ts
@@ -1,4 +1,5 @@
import { pinnedFetch } from '@repo/net';
+import type { AttachmentDownload } from './reader';
// An attachment that cannot be stored: over this instance's size, type or quota
// limits, or a download that did not return the file. The Attachments phase logs
@@ -12,11 +13,15 @@ const DOWNLOAD_TIMEOUT_MS = 30_000;
// pinnedFetch returns any status as-is, a redirect included, so an error page
// must be refused here or it would be stored as the file.
export async function downloadAttachment(
- url: string,
+ download: AttachmentDownload,
maxBytes: number,
fetch: AttachmentFetch = pinnedFetch,
): Promise<Buffer> {
- const res = await fetch(url, { timeoutMs: DOWNLOAD_TIMEOUT_MS, maxBytes });
+ const res = await fetch(download.url, {
+ headers: download.headers,
+ timeoutMs: DOWNLOAD_TIMEOUT_MS,
+ maxBytes,
+ });
if (!res.ok) {
throw new AttachmentRejectedError(`the download answered HTTP ${res.status}`);
}
diff --git a/apps/worker/src/import-retry.ts b/apps/worker/src/import-retry.ts
new file mode 100644
index 00000000..296547a4
--- /dev/null
+++ b/apps/worker/src/import-retry.ts
@@ -0,0 +1,21 @@
+import { equalJitterBackoffMs } from './backoff';
+import { SourceRateLimitedError } from './reader';
+
+// import-store.ts resets job.attempts to 0 on every tick that makes progress, so
+// this counts consecutive failed ticks: a long import does not fail itself out.
+export const MAX_ATTEMPTS = 10;
+
+// SettingsImportExportJobRow.tsx (apps/web) matches this exact last_error.
+const RATE_LIMITED = 'rate limited';
+
+export type TickErrorOutcome =
+ { action: 'retry'; delayMs: number; lastError: string } | { action: 'fail'; lastError: string };
+
+export function tickErrorOutcome(error: unknown, attempts: number): TickErrorOutcome {
+ if (error instanceof SourceRateLimitedError) {
+ return { action: 'retry', delayMs: error.retryAfterMs, lastError: RATE_LIMITED };
+ }
+ const message = error instanceof Error ? error.message : String(error);
+ if (attempts >= MAX_ATTEMPTS) return { action: 'fail', lastError: message };
+ return { action: 'retry', delayMs: equalJitterBackoffMs(attempts), lastError: message };
+}
diff --git a/apps/worker/src/import-sources.ts b/apps/worker/src/import-sources.ts
new file mode 100644
index 00000000..9f94147a
--- /dev/null
+++ b/apps/worker/src/import-sources.ts
@@ -0,0 +1,80 @@
+import type { CanonicalStateCategory } from './canonical';
+import { PlaneReader, type PlaneCredential } from './plane-adapter';
+import type { SourceReader } from './reader';
+
+// The import_job.config fields every source shares; the phases read them directly.
+export interface CommonImportConfig {
+ unmatchedUserPolicy?: 'unassigned' | 'skip';
+ stateOverrides?: Record<string, CanonicalStateCategory>;
+}
+
+export interface PlaneImportConfig extends CommonImportConfig {
+ planeProjectId: string;
+ // Absent on a job created before the Rewrite phase existed; Rewrite then leaves text as written.
+ planeProjectKey?: string;
+}
+
+// One entry per import_job.source. A new source adds a member to both maps and an
+// entry to IMPORT_SOURCES.
+export interface ImportCredentials {
+ plane: PlaneCredential;
+}
+
+export interface ImportConfigs {
+ plane: PlaneImportConfig;
+}
+
+export type ImportSourceKind = keyof ImportConfigs;
+export type ImportCredential = ImportCredentials[ImportSourceKind];
+export type ImportConfig = ImportConfigs[ImportSourceKind];
+
+interface ImportSourceDefinition<S extends ImportSourceKind> {
+ buildReader(
+ credential: ImportCredentials[S],
+ config: Partial<ImportConfigs[S]>,
+ jobId: number,
+ ): SourceReader;
+ // The prefix of the source's own issue identifiers ("ROOMS" in "ROOMS-524").
+ projectKey(config: Partial<ImportConfigs[S]>): string | null;
+}
+
+const IMPORT_SOURCES: { [S in ImportSourceKind]: ImportSourceDefinition<S> } = {
+ plane: {
+ buildReader(credential, config, jobId) {
+ if (!config.planeProjectId) {
+ throw new Error(`import job ${jobId} has no source project configured`);
+ }
+ return new PlaneReader(credential, config.planeProjectId);
+ },
+ projectKey: (config) => config.planeProjectKey ?? null,
+ },
+};
+
+export interface ImportJobSource {
+ id: number;
+ source: string;
+ config: Record<string, unknown>;
+}
+
+function sourceOf(job: ImportJobSource): ImportSourceKind {
+ if (!Object.hasOwn(IMPORT_SOURCES, job.source)) {
+ throw new Error(`import job ${job.id} has unsupported source "${job.source}"`);
+ }
+ return job.source as ImportSourceKind;
+}
+
+function definitionOf<S extends ImportSourceKind>(source: S): ImportSourceDefinition<S> {
+ return IMPORT_SOURCES[source];
+}
+
+export function readerForJob(job: ImportJobSource, credential: ImportCredential): SourceReader {
+ return definitionOf(sourceOf(job)).buildReader(
+ credential,
+ job.config as Partial<ImportConfig>,
+ job.id,
+ );
+}
+
+export function sourceProjectKeyForJob(job: ImportJobSource): string | null {
+ return definitionOf(sourceOf(job)).projectKey(job.config as Partial<ImportConfig>);
+}
diff --git a/apps/worker/src/import-store.ts b/apps/worker/src/import-store.ts
index 50d36b73..15748a94 100644
--- a/apps/worker/src/import-store.ts
+++ b/apps/worker/src/import-store.ts
@@ -29,12 +29,12 @@ import {
attachmentObjectKey,
} from '@repo/storage';
import type { CanonicalState, CanonicalLabel, CanonicalCycle } from './canonical';
-import type { PlaneCredential } from './plane-adapter';
+import type { ImportCredential } from './import-sources';
import { AttachmentRejectedError } from './attachment-download';
export { AttachmentRejectedError };
-// All @repo/db access for the Plane import: claiming due import_job rows (the
+// All @repo/db access for a source import: claiming due import_job rows (the
// same FOR UPDATE SKIP LOCKED + lease pattern as store.ts uses for
// webhook_delivery), reading/writing a job's progress, the import_record
// upsert primitive, and creating the local rows an import produces. The
@@ -101,7 +101,7 @@ export async function claimDueImportJobs(limit = 1): Promise<ClaimedImportJob[]>
return rows as unknown as ClaimedImportJob[];
}
-export function decryptImportCredential(job: ClaimedImportJob): PlaneCredential {
+export function decryptImportCredential(job: ClaimedImportJob): ImportCredential {
if (!job.credentialCiphertext || !job.credentialIv || !job.credentialAuthTag) {
throw new Error(`import job ${job.id} has no stored credential`);
}
@@ -110,7 +110,7 @@ export function decryptImportCredential(job: ClaimedImportJob): PlaneCredential
iv: job.credentialIv,
authTag: job.credentialAuthTag,
};
- return JSON.parse(decryptSecret(encrypted)) as PlaneCredential;
+ return JSON.parse(decryptSecret(encrypted)) as ImportCredential;
}
// A successful tick clears the claim lease (nextAttemptAt) and lastError, and
diff --git a/apps/worker/src/import-worker.ts b/apps/worker/src/import-worker.ts
index 1202f9ff..0337fb4a 100644
--- a/apps/worker/src/import-worker.ts
+++ b/apps/worker/src/import-worker.ts
@@ -1,10 +1,10 @@
import { getStorageSettings, mimeAllowed, MB } from '@repo/db';
import { startPollLoop, type WorkerHandle } from './poll-loop';
import { intEnv } from './env';
-import { equalJitterBackoffMs } from './backoff';
-import { PlaneReader, PlaneRateLimitedError } from './plane-adapter';
+import { tickErrorOutcome } from './import-retry';
+import { readerForJob, sourceProjectKeyForJob, type CommonImportConfig } from './import-sources';
import type { SourceReader } from './reader';
-import type { CanonicalComment, CanonicalStateCategory } from './canonical';
+import type { CanonicalComment } from './canonical';
import { extractCrossReferences, applyCrossReferenceReplacements } from './cross-reference';
import { downloadAttachment } from './attachment-download';
import {
@@ -41,7 +41,7 @@ import {
type ClaimedImportJob,
} from './import-store';
-// The poll loop that drives a Plane import through its phases: discover ->
+// The poll loop that drives a source import through its phases: discover ->
// create -> link -> rewrite -> attachments -> done. Follows the same claim-a-lease,
// bounded-chunk-per-tick shape as worker.ts and agent-worker.ts. Each tick
// claims at most one due import_job and advances it by one bounded unit of
@@ -55,11 +55,6 @@ export function startImportWorker(): WorkerHandle {
// additional Plane requests (comments, attachments, relations) against a
// ~60 requests/minute budget, so this stays well under the discover page size.
const ISSUES_PER_TICK = 15;
-// import_store.ts resets job.attempts to 0 on every tick that makes progress
-// (saveImportJobCursor/advanceImportJobPhase), so this counts consecutive
-// failed ticks, not total ticks claimed over the job's life — a multi-tick
-// import doesn't fail itself out just by running long.
-const MAX_ATTEMPTS = 10;
async function tick(): Promise<void> {
const [job] = await claimDueImportJobs();
@@ -91,37 +86,16 @@ async function tick(): Promise<void> {
}
async function handleTickError(job: ClaimedImportJob, error: unknown): Promise<void> {
- if (error instanceof PlaneRateLimitedError) {
- await retryImportJobLater(job.id, error.retryAfterMs, 'rate limited');
+ const outcome = tickErrorOutcome(error, job.attempts);
+ if (outcome.action === 'fail') {
+ await failImportJob(job.id, outcome.lastError);
return;
}
- const message = error instanceof Error ? error.message : String(error);
- if (job.attempts >= MAX_ATTEMPTS) {
- await failImportJob(job.id, message);
- return;
- }
- await retryImportJobLater(job.id, equalJitterBackoffMs(job.attempts), message);
-}
-
-interface PlaneImportConfig {
- planeProjectId: string;
- // The source project's own short identifier (Plane's "identifier", e.g. "ROOMS"),
- // stored at job creation for the Rewrite phase to build "ROOMS-524"-style
- // cross-reference patterns from. Optional: a job created before this existed has
- // none, and Rewrite is a no-op for it rather than a failure.
- planeProjectKey?: string;
- // Set from the mapping review step at job creation. Both are optional and default
- // to itsaplan's automatic mapping when absent.
- unmatchedUserPolicy?: 'unassigned' | 'skip';
- stateOverrides?: Record<string, CanonicalStateCategory>;
+ await retryImportJobLater(job.id, outcome.delayMs, outcome.lastError);
}
function buildReader(job: ClaimedImportJob): SourceReader {
- const credential = decryptImportCredential(job);
- const config = job.config as Partial<PlaneImportConfig>;
- if (!config.planeProjectId)
- throw new Error(`import job ${job.id} has no source project configured`);
- return new PlaneReader(credential, config.planeProjectId);
+ return readerForJob(job, decryptImportCredential(job));
}
// --- Discover: snapshot every source id up front, writing import_record rows
@@ -211,7 +185,7 @@ export async function runCreate(job: ClaimedImportJob, reader: SourceReader): Pr
async function materializeStates(job: ClaimedImportJob, reader: SourceReader): Promise<void> {
const pending = await listUncreatedImportRecords(job.id, 'state', 0, 1000);
if (pending.length === 0) return;
- const overrides = (job.config as Partial<PlaneImportConfig>).stateOverrides ?? {};
+ const overrides = (job.config as CommonImportConfig).stateOverrides ?? {};
const states = new Map((await reader.listStates()).map((s) => [s.sourceId, s]));
for (const record of pending) {
const state = states.get(record.sourceId);
@@ -316,7 +290,7 @@ async function createComments(
issueLocalId: number,
comments: CanonicalComment[],
): Promise<void> {
- const unmatchedUserPolicy = (job.config as Partial<PlaneImportConfig>).unmatchedUserPolicy;
+ const unmatchedUserPolicy = (job.config as CommonImportConfig).unmatchedUserPolicy;
const localIdBySourceId = new Map<string, number>();
for (const comment of comments) {
const existing = await findImportRecord(job.id, 'comment', comment.sourceId);
@@ -417,9 +391,9 @@ async function runRewrite(job: ClaimedImportJob): Promise<void> {
return;
}
- // A job created before this phase existed has no planeProjectKey — its
+ // A job created before this phase existed has no source project key — its
// description/comment text is left exactly as Create wrote it.
- const sourceProjectKey = (job.config as Partial<PlaneImportConfig>).planeProjectKey;
+ const sourceProjectKey = sourceProjectKeyForJob(job);
if (sourceProjectKey) {
const localProjectKey = await getProjectKey(job.projectId);
for (const record of pending) {
@@ -485,8 +459,11 @@ async function runAttachments(job: ClaimedImportJob, reader: SourceReader): Prom
`"${attachment.filename}" is type "${attachment.contentType}", not accepted on this instance`,
);
}
- const url = await reader.resolveAttachmentDownloadUrl(record.sourceId, attachment.sourceId);
- const bytes = await downloadAttachment(url, limits.maxAttachmentMb * MB);
+ const download = await reader.resolveAttachmentDownload(
+ record.sourceId,
+ attachment.sourceId,
+ );
+ const bytes = await downloadAttachment(download, limits.maxAttachmentMb * MB);
await createLocalAttachmentAndRecord(job.id, attachment.sourceId, {
projectId: job.projectId,
issueId: record.localId,
diff --git a/apps/worker/src/plane-adapter.ts b/apps/worker/src/plane-adapter.ts
index 68426785..12e4c448 100644
--- a/apps/worker/src/plane-adapter.ts
+++ b/apps/worker/src/plane-adapter.ts
@@ -1,5 +1,10 @@
import { pinnedFetch } from '@repo/net';
-import type { Page, SourceReader } from './reader';
+import {
+ SourceRateLimitedError,
+ type AttachmentDownload,
+ type Page,
+ type SourceReader,
+} from './reader';
import type {
CanonicalState,
CanonicalStateCategory,
@@ -24,15 +29,6 @@ export interface PlaneCredential {
const PER_PAGE = 100;
-// Thrown when Plane's rate limit is hit (or about to be): x-ratelimit-remaining
-// reaches 0, or a 429 arrives. The worker catches this and reschedules the job
-// instead of treating it as a failed attempt.
-export class PlaneRateLimitedError extends Error {
- constructor(public readonly retryAfterMs: number) {
- super('Plane rate limit reached');
- }
-}
-
// A 404 confirmed live to mean "this Plane version doesn't have this
// endpoint" (custom properties, work item types, ...) rather than a broken
// import, when it happens on one of those. Nothing catches it specially yet —
@@ -217,6 +213,7 @@ export class PlaneReader implements SourceReader {
constructor(
private readonly credential: PlaneCredential,
private readonly projectId: string,
+ private readonly fetch: typeof pinnedFetch = pinnedFetch,
) {}
async listStates(): Promise<CanonicalState[]> {
@@ -326,10 +323,10 @@ export class PlaneReader implements SourceReader {
// Two-hop resolve, confirmed live (see "Attachments" in the notes file): this
// call itself 302s to an S3 URL valid for exactly an hour, needing no Plane
// auth — never resolved ahead of the moment it's actually downloaded.
- async resolveAttachmentDownloadUrl(
+ async resolveAttachmentDownload(
issueSourceId: string,
attachmentSourceId: string,
- ): Promise<string> {
+ ): Promise<AttachmentDownload> {
const path = `/work-items/${issueSourceId}/attachments/${attachmentSourceId}/`;
const res = await this.requestRaw(path);
const location = res.headers.get('location');
@@ -338,7 +335,7 @@ export class PlaneReader implements SourceReader {
`Plane attachment resolve did not redirect: GET ${path} -> HTTP ${res.status}`,
);
}
- return location;
+ return { url: location };
}
// members/ is flat, unpaginated, and carries email directly — exactly what
@@ -372,11 +369,11 @@ export class PlaneReader implements SourceReader {
}
// The rate-limit and 404 checks every Plane call needs, without requiring a
- // 2xx status — resolveAttachmentDownloadUrl's whole point is a 3xx response.
+ // 2xx status — resolveAttachmentDownload's whole point is a 3xx response.
private async requestRaw(path: string): Promise<Response> {
const base = this.credential.baseUrl.replace(/\/$/, '');
const url = `${base}/api/v1/workspaces/${this.credential.workspaceSlug}/projects/${this.projectId}${path}`;
- const res = await pinnedFetch(url, {
+ const res = await this.fetch(url, {
headers: { 'X-Api-Key': this.credential.apiKey },
timeoutMs: 15_000,
});
@@ -386,7 +383,8 @@ export class PlaneReader implements SourceReader {
resetEpochSec: res.headers.get('x-ratelimit-reset'),
retryAfterSec: res.headers.get('retry-after'),
});
- if (backoff !== null) throw new PlaneRateLimitedError(backoff);
+ // x-ratelimit-remaining reaching 0 counts too, not only a 429.
+ if (backoff !== null) throw new SourceRateLimitedError(backoff, 'Plane rate limit reached');
if (res.status === 404) throw new PlaneNotFoundError(path);
return res;
}
diff --git a/apps/worker/src/reader.ts b/apps/worker/src/reader.ts
index 279b0f5f..10635731 100644
--- a/apps/worker/src/reader.ts
+++ b/apps/worker/src/reader.ts
@@ -8,6 +8,25 @@ import type {
CanonicalAttachment,
} from './canonical';
+// Thrown by a source adapter when the source's rate limit is reached. The worker
+// reschedules the job after retryAfterMs: a rate limit is waited out and never
+// fails the job by itself.
+export class SourceRateLimitedError extends Error {
+ constructor(
+ public readonly retryAfterMs: number,
+ message = 'source rate limit reached',
+ ) {
+ super(message);
+ }
+}
+
+// headers is for a source whose file host needs the source's own credential; the
+// Attachments phase sends them with the download and nowhere else.
+export interface AttachmentDownload {
+ url: string;
+ headers?: Record<string, string>;
+}
+
// One page of a cursor-paginated list. cursor is the exact, opaque value the
// source API returned for the next page, or null when there is no next page.
// It must be passed back to the source verbatim — never reconstructed — and a
@@ -40,9 +59,12 @@ export interface SourceReader {
listIssueRelations(issueSourceId: string): Promise<CanonicalRelation[]>;
listIssueComments(issueSourceId: string): Promise<CanonicalComment[]>;
listIssueAttachments(issueSourceId: string): Promise<CanonicalAttachment[]>;
- // Resolves one attachment's download URL, fresh each call: the source's own
+ // Resolves how to download one attachment, fresh each call: the source's own
// resolved URL is a presigned link that expires (Plane's lasts exactly an
// hour), so this is called at the moment of download, never cached or
// resolved ahead of time.
- resolveAttachmentDownloadUrl(issueSourceId: string, attachmentSourceId: string): Promise<string>;
+ resolveAttachmentDownload(
+ issueSourceId: string,
+ attachmentSourceId: string,
+ ): Promise<AttachmentDownload>;
}
diff --git a/docs/dev/plane-import.md b/docs/dev/plane-import.md
index 040c2476..7726f834 100644
--- a/docs/dev/plane-import.md
+++ b/docs/dev/plane-import.md
@@ -12,8 +12,9 @@ returns.
`next_attempt_at`/`last_error` the same shape as `webhook_delivery`, credential columns
encrypted the same way as `integration_credential`) and `import_record` (source id →
local id mapping, the idempotency and resume primitive).
-- `apps/worker/src/{canonical,reader,plane-adapter,import-store,import-worker}.ts` — the
- `SourceReader` port, the only implementation (Plane), and the phase state machine
+- `apps/worker/src/{canonical,reader,import-sources,plane-adapter,import-store,import-worker}.ts`
+ — the `SourceReader` port, the per-source registry the worker dispatches on
+ `import_job.source` through, the only implementation (Plane), and the phase state machine
(discover → create → link → rewrite → attachments → done) that drives a job one bounded chunk per
tick.
- `packages/storage` and `packages/db/src/domains/storage.ts` — object storage and upload
@@ -161,10 +162,12 @@ attachment) are a distinct path, not covered by this — see "Inline images" in
## Rate limiting and resumability, working as designed
`rateLimitBackoffMs` (`plane-adapter.ts`) reads `x-ratelimit-remaining`/`x-ratelimit-reset`/
-a 429 off every response; hitting the limit throws `PlaneRateLimitedError`, and
-`handleTickError` (`import-worker.ts`) reschedules via `retryImportJobLater` rather than
-counting it as a failed attempt. `import_job.last_error` is cleared the moment a claim starts
-a new attempt, as well as on success and on completion (`import-store.ts`'s
+a 429 off every response; hitting the limit throws `SourceRateLimitedError` (`reader.ts`).
+`tickErrorOutcome` (`import-retry.ts`) only decides what a failed tick does — for a rate
+limit, retry after the reset with `last_error` `'rate limited'` — and `handleTickError`
+(`import-worker.ts`) reschedules the job via `retryImportJobLater`, so a rate limit is waited
+out and never fails the job by itself. `import_job.last_error` is cleared the moment a claim
+starts a new attempt, as well as on success and on completion (`import-store.ts`'s
`claimDueImportJobs`/`saveImportJobCursor`/`advanceImportJobPhase`/`completeImportJob`), so a
non-null `lastError` on a still-`pending` job reliably means "currently waiting out a retry,"
never "an attempt is in flight" — which is what the Settings page's warning banner reads.