⭐️⭐️⭐️ Open-source, self-hosted alternative to Linear and Plane. Project management and issue tracking where teams and AI agents work side by side to plan and ship products.
TypeScript agenticagileai-agentsbug-reportingbug-trackerganttissue-trackerjirajira-alternativekanbanlinearlinear-alternativeopen-sourceproduct-managementproject-managementroadmapself-hostedsprint-planningteam-collaborationtrello-alternative
refactor(worker): make the import pipeline source-neutral (#477)
Thomas Gallaway1acb20ccced4
diff --git a/apps/worker/AGENTS.md b/apps/worker/AGENTS.mdindex 84ecb2b6..0dd6fc8c 100644--- a/apps/worker/AGENTS.md+++ b/apps/worker/AGENTS.md@@ -31,7 +31,14 @@ running to completion in one.- `canonical.ts` — the source-independent shape an adapter produces(`CanonicalIssue`, `CanonicalState`, ...). Plain types, no logic.-- `reader.ts` — the `SourceReader` port a source adapter implements.+- `reader.ts` — the `SourceReader` port a source adapter implements, and the+ `SourceRateLimitedError` an adapter throws when the source's rate limit is reached.+- `import-sources.ts` — one entry per `import_job.source`: its credential and+ config types, how to build its reader, and its issue-key prefix for Rewrite. A+ new source adds an entry here; the phases do not change.+- `import-retry.ts` — what a failed tick does: a rate limit is waited out and+ never fails the job by itself, any other error retries with backoff until+ `MAX_ATTEMPTS` in a row.- `plane-adapter.ts` — the only implementation today. HTTP against a Planeinstance via `pinnedFetch`, using the credential decrypted from the job row(base URL, workspace slug, API key — never from env, since this has to work@@ -92,8 +99,8 @@ running to completion in one.- **Pure logic stays dependency-free.** `backoff.ts`, `signature.ts`, and`isRetryableStatus` import nothing from `@repo/db`, so unit tests run without adatabase. Keep DB access in `store.ts`. Same split for imports: `canonical.ts`,- `reader.ts`, `plane-adapter.ts`, `cross-reference.ts`, and- `attachment-download.ts` import nothing from+ `reader.ts`, `plane-adapter.ts`, `cross-reference.ts`, `attachment-download.ts`,+ `import-sources.ts`, and `import-retry.ts` import nothing from`@repo/db` (its state-category normalization, markdown conversion,cursor/rate-limit, and cross-reference matching logic are unit-testeddirectly); `@repo/db` access stays in `import-store.ts`.@@ -110,7 +117,7 @@ All via env with defaults (see `src/config.ts`): `WEBHOOK_POLL_INTERVAL_MS`,`WEBHOOK_CLEANUP_EVERY_TICKS`. Only `DATABASE_URL` is required for webhookdelivery. Notification delivery also needs `APP_ENCRYPTION_KEY` (the same value theapi uses) to read the stored provider credentials, and so does source import (it-decrypts the stored Plane credential with it too). `IMPORT_POLL_INTERVAL_MS` tunes+decrypts the stored source credential with it too). `IMPORT_POLL_INTERVAL_MS` tunesthe import worker's poll interval. The Attachments phase needs the same `S3_*`variables (`S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`,`S3_REGION`, `S3_FORCE_PATH_STYLE`) the api reads for its own uploads — bothdiff --git a/apps/worker/src/__tests__/integration/import-store.test.ts b/apps/worker/src/__tests__/integration/import-store.test.tsindex e2137544..0a327232 100644--- a/apps/worker/src/__tests__/integration/import-store.test.ts+++ b/apps/worker/src/__tests__/integration/import-store.test.ts@@ -16,9 +16,13 @@ import {type StorageSettings,} from '@repo/db';import { eq } from 'drizzle-orm';+import { encryptSecret } from '@repo/crypto';import { getObject } from '@repo/storage';+import { readerForJob, sourceProjectKeyForJob } from '../../import-sources';+import { PlaneReader } from '../../plane-adapter';import {claimDueImportJobs,+ decryptImportCredential,createLocalStateAndRecord,createLocalIssueAndRecord,createLocalAttachmentAndRecord,@@ -132,6 +136,45 @@ describe('claimDueImportJobs', () => {});});+describe('decryptImportCredential', () => {+ it('resumes a Plane job the api created, mid-phase, with the same reader and key', async () => {+ const { projectId, userId } = await makeProject();+ const credential = {+ baseUrl: 'https://plane.example.test',+ workspaceSlug: 'acme',+ apiKey: 'plane-api-key',+ };+ const encrypted = encryptSecret(JSON.stringify(credential));+ const [row] = await db+ .insert(importJob)+ .values({+ projectId,+ createdByUserId: userId,+ source: 'plane',+ phase: 'create',+ config: {+ planeProjectId: 'project-1',+ planeProjectKey: 'ROOMS',+ unmatchedUserPolicy: 'unassigned',+ },+ cursor: { lastRecordId: 41 },+ credentialCiphertext: encrypted.ciphertext,+ credentialIv: encrypted.iv,+ credentialAuthTag: encrypted.authTag,+ nextAttemptAt: new Date(Date.now() - 1000),+ })+ .returning({ id: importJob.id });++ const job = (await claimDueImportJobs(1000)).find((j) => j.id === row!.id);++ expect(job).toMatchObject({ source: 'plane', phase: 'create', cursor: { lastRecordId: 41 } });+ const decrypted = decryptImportCredential(job!);+ expect(decrypted).toEqual(credential);+ expect(readerForJob(job!, decrypted)).toBeInstanceOf(PlaneReader);+ expect(sourceProjectKeyForJob(job!)).toBe('ROOMS');+ });+});+describe('createLocalStateAndRecord', () => {it('creates a new column and records the mapping', async () => {const { projectId, userId } = await makeProject();diff --git a/apps/worker/src/__tests__/unit/attachment-download.test.ts b/apps/worker/src/__tests__/unit/attachment-download.test.tsindex 3d6e38d1..60ee13d3 100644--- a/apps/worker/src/__tests__/unit/attachment-download.test.ts+++ b/apps/worker/src/__tests__/unit/attachment-download.test.ts@@ -14,7 +14,11 @@ function respondWith(response: Response): AttachmentFetch {describe('downloadAttachment', () => {it('returns the body of a 2xx response', async () => {- const bytes = await downloadAttachment(FILE_URL, 1024, respondWith(new Response('file bytes')));+ const bytes = await downloadAttachment(+ { url: FILE_URL },+ 1024,+ respondWith(new Response('file bytes')),+ );expect(bytes.toString()).toBe('file bytes');});@@ -22,7 +26,7 @@ describe('downloadAttachment', () => {'rejects an HTTP %d response instead of returning its body as the file',async (status) => {const error = await downloadAttachment(- FILE_URL,+ { url: FILE_URL },1024,respondWith(new Response('<Error>AccessDenied</Error>', { status })),).catch((e: unknown) => e);@@ -36,9 +40,9 @@ describe('downloadAttachment', () => {status: 302,headers: { location: 'https://elsewhere.example.test/report.pdf' },});- await expect(downloadAttachment(FILE_URL, 1024, respondWith(redirect))).rejects.toBeInstanceOf(- AttachmentRejectedError,- );+ await expect(+ downloadAttachment({ url: FILE_URL }, 1024, respondWith(redirect)),+ ).rejects.toBeInstanceOf(AttachmentRejectedError);});it('downloads with the byte limit and a timeout', async () => {@@ -47,7 +51,25 @@ describe('downloadAttachment', () => {seen = { url, init };return new Response('x');};- await downloadAttachment(FILE_URL, 2048, fetch);+ await downloadAttachment({ url: FILE_URL }, 2048, fetch);expect(seen!).toEqual({ url: FILE_URL, init: { timeoutMs: 30_000, maxBytes: 2048 } });});++ it('sends the headers the source asked for', async () => {+ let seen: PinnedRequestInit | undefined;+ const fetch: AttachmentFetch = async (_url, init) => {+ seen = init;+ return new Response('x');+ };+ await downloadAttachment(+ { url: FILE_URL, headers: { Authorization: 'source-token' } },+ 2048,+ fetch,+ );+ expect(seen).toEqual({+ headers: { Authorization: 'source-token' },+ timeoutMs: 30_000,+ maxBytes: 2048,+ });+ });});diff --git a/apps/worker/src/__tests__/unit/import-retry.test.ts b/apps/worker/src/__tests__/unit/import-retry.test.tsnew file mode 100644index 00000000..4556d871--- /dev/null+++ b/apps/worker/src/__tests__/unit/import-retry.test.ts@@ -0,0 +1,41 @@+import { describe, it, expect } from 'bun:test';+import { tickErrorOutcome, MAX_ATTEMPTS } from '../../import-retry';+import { SourceRateLimitedError } from '../../reader';++describe('tickErrorOutcome', () => {+ it('waits out a rate limit with the exact last_error the Settings page matches', () => {+ expect(tickErrorOutcome(new SourceRateLimitedError(45_000), 1)).toEqual({+ action: 'retry',+ delayMs: 45_000,+ lastError: 'rate limited',+ });+ });++ it('never fails a job for a rate limit, however many attempts ran', () => {+ expect(tickErrorOutcome(new SourceRateLimitedError(1000), MAX_ATTEMPTS + 5).action).toBe(+ 'retry',+ );+ });++ it('retries any other error with backoff and its message', () => {+ const outcome = tickErrorOutcome(new Error('Plane request failed: HTTP 502'), 1);+ expect(outcome).toMatchObject({+ action: 'retry',+ lastError: 'Plane request failed: HTTP 502',+ });+ const delayMs = (outcome as { delayMs: number }).delayMs;+ expect(delayMs).toBeGreaterThanOrEqual(30_000);+ expect(delayMs).toBeLessThanOrEqual(60_000);+ });++ it('fails the job once the consecutive attempts reach the limit', () => {+ expect(tickErrorOutcome(new Error('still broken'), MAX_ATTEMPTS)).toEqual({+ action: 'fail',+ lastError: 'still broken',+ });+ });++ it('records a thrown non-Error as its string form', () => {+ expect(tickErrorOutcome('boom', MAX_ATTEMPTS)).toEqual({ action: 'fail', lastError: 'boom' });+ });+});diff --git a/apps/worker/src/__tests__/unit/import-sources.test.ts b/apps/worker/src/__tests__/unit/import-sources.test.tsnew file mode 100644index 00000000..4e5541a9--- /dev/null+++ b/apps/worker/src/__tests__/unit/import-sources.test.ts@@ -0,0 +1,46 @@+import { describe, it, expect } from 'bun:test';+import { readerForJob, sourceProjectKeyForJob } from '../../import-sources';+import { PlaneReader } from '../../plane-adapter';++const planeCredential = {+ baseUrl: 'https://plane.example.test',+ workspaceSlug: 'acme',+ apiKey: 'plane-api-key',+};++describe('readerForJob', () => {+ it('builds a Plane reader for a Plane job', () => {+ const job = { id: 7, source: 'plane', config: { planeProjectId: 'project-1' } };+ expect(readerForJob(job, planeCredential)).toBeInstanceOf(PlaneReader);+ });++ it('refuses a Plane job with no source project', () => {+ const job = { id: 7, source: 'plane', config: {} };+ expect(() => readerForJob(job, planeCredential)).toThrow(+ 'import job 7 has no source project configured',+ );+ });++ it.each(['jira', 'linear', 'toString', ''])('refuses the unsupported source "%s"', (source) => {+ const job = { id: 7, source, config: { planeProjectId: 'project-1' } };+ expect(() => readerForJob(job, planeCredential)).toThrow(+ `import job 7 has unsupported source "${source}"`,+ );+ });+});++describe('sourceProjectKeyForJob', () => {+ it("returns a Plane job's project identifier", () => {+ const job = {+ id: 7,+ source: 'plane',+ config: { planeProjectId: 'p', planeProjectKey: 'ROOMS' },+ };+ expect(sourceProjectKeyForJob(job)).toBe('ROOMS');+ });++ it('returns null for a Plane job created before the Rewrite phase existed', () => {+ const job = { id: 7, source: 'plane', config: { planeProjectId: 'p' } };+ expect(sourceProjectKeyForJob(job)).toBeNull();+ });+});diff --git a/apps/worker/src/__tests__/unit/plane-adapter.test.ts b/apps/worker/src/__tests__/unit/plane-adapter.test.tsindex f261718b..10949ac9 100644--- a/apps/worker/src/__tests__/unit/plane-adapter.test.ts+++ b/apps/worker/src/__tests__/unit/plane-adapter.test.ts@@ -1,10 +1,13 @@import { describe, it, expect } from 'bun:test';+import type { PinnedRequestInit } from '@repo/net';import {+ PlaneReader,normalizeStateCategory,nextPageCursor,rateLimitBackoffMs,htmlToMarkdown,} from '../../plane-adapter';+import { SourceRateLimitedError } from '../../reader';describe('normalizeStateCategory', () => {it("maps the British-spelled cancelled to itsaplan's canceled", () => {@@ -137,3 +140,79 @@ describe('htmlToMarkdown', () => {expect(htmlToMarkdown('')).toBe('');});});++interface SentRequest {+ url: string;+ init: PinnedRequestInit | undefined;+}++function readerAnswering(response: Response, sent: SentRequest[] = []): PlaneReader {+ const credential = {+ baseUrl: 'https://plane.example.test/',+ workspaceSlug: 'acme',+ apiKey: 'plane-api-key',+ };+ return new PlaneReader(credential, 'project-1', async (url, init) => {+ sent.push({ url, init });+ return response;+ });+}++describe('PlaneReader', () => {+ it('sends the API key to the project-scoped Plane endpoint', async () => {+ const sent: SentRequest[] = [];+ const reader = readerAnswering(+ Response.json({ results: [], next_cursor: null, next_page_results: false }),+ sent,+ );+ expect(await reader.listStates()).toEqual([]);+ expect(sent).toEqual([+ {+ url: 'https://plane.example.test/api/v1/workspaces/acme/projects/project-1/states/',+ init: { headers: { 'X-Api-Key': 'plane-api-key' }, timeoutMs: 15_000 },+ },+ ]);+ });++ it('throws the source-neutral rate-limit error on a 429', async () => {+ const reader = readerAnswering(+ new Response('', { status: 429, headers: { 'retry-after': '30' } }),+ );+ const error = await reader.listStates().catch((e: unknown) => e);+ expect(error).toBeInstanceOf(SourceRateLimitedError);+ expect((error as SourceRateLimitedError).retryAfterMs).toBe(30_000);+ expect((error as Error).message).toBe('Plane rate limit reached');+ });++ it('throws it when the remaining budget reaches zero on a 200', async () => {+ const reader = readerAnswering(+ Response.json(+ { results: [], next_cursor: null, next_page_results: false },+ { headers: { 'x-ratelimit-remaining': '0' } },+ ),+ );+ await expect(reader.listStates()).rejects.toBeInstanceOf(SourceRateLimitedError);+ });++ it('resolves an attachment to the redirect target, with no Plane credential attached', async () => {+ const sent: SentRequest[] = [];+ const target = 'https://s3.example.test/bucket/report.pdf?X-Amz-Signature=abc';+ const reader = readerAnswering(+ new Response(null, { status: 302, headers: { location: target } }),+ sent,+ );+ expect(await reader.resolveAttachmentDownload('issue-1', 'att-1')).toStrictEqual({+ url: target,+ });+ expect(sent[0]!.url).toBe(+ 'https://plane.example.test/api/v1/workspaces/acme/projects/project-1/work-items/issue-1/attachments/att-1/',+ );+ });++ it('refuses an attachment resolve that did not redirect', async () => {+ const reader = readerAnswering(new Response('{}', { status: 200 }));+ await expect(reader.resolveAttachmentDownload('issue-1', 'att-1')).rejects.toThrow(+ 'Plane attachment resolve did not redirect: GET /work-items/issue-1/attachments/att-1/ -> HTTP 200',+ );+ });+});diff --git a/apps/worker/src/attachment-download.ts b/apps/worker/src/attachment-download.tsindex da1659f9..2f81f95e 100644--- a/apps/worker/src/attachment-download.ts+++ b/apps/worker/src/attachment-download.ts@@ -1,4 +1,5 @@import { pinnedFetch } from '@repo/net';+import type { AttachmentDownload } from './reader';// An attachment that cannot be stored: over this instance's size, type or quota// limits, or a download that did not return the file. The Attachments phase logs@@ -12,11 +13,15 @@ const DOWNLOAD_TIMEOUT_MS = 30_000;// pinnedFetch returns any status as-is, a redirect included, so an error page// must be refused here or it would be stored as the file.export async function downloadAttachment(- url: string,+ download: AttachmentDownload,maxBytes: number,fetch: AttachmentFetch = pinnedFetch,): Promise<Buffer> {- const res = await fetch(url, { timeoutMs: DOWNLOAD_TIMEOUT_MS, maxBytes });+ const res = await fetch(download.url, {+ headers: download.headers,+ timeoutMs: DOWNLOAD_TIMEOUT_MS,+ maxBytes,+ });if (!res.ok) {throw new AttachmentRejectedError(`the download answered HTTP ${res.status}`);}diff --git a/apps/worker/src/import-retry.ts b/apps/worker/src/import-retry.tsnew file mode 100644index 00000000..296547a4--- /dev/null+++ b/apps/worker/src/import-retry.ts@@ -0,0 +1,21 @@+import { equalJitterBackoffMs } from './backoff';+import { SourceRateLimitedError } from './reader';++// import-store.ts resets job.attempts to 0 on every tick that makes progress, so+// this counts consecutive failed ticks: a long import does not fail itself out.+export const MAX_ATTEMPTS = 10;++// SettingsImportExportJobRow.tsx (apps/web) matches this exact last_error.+const RATE_LIMITED = 'rate limited';++export type TickErrorOutcome =+ { action: 'retry'; delayMs: number; lastError: string } | { action: 'fail'; lastError: string };++export function tickErrorOutcome(error: unknown, attempts: number): TickErrorOutcome {+ if (error instanceof SourceRateLimitedError) {+ return { action: 'retry', delayMs: error.retryAfterMs, lastError: RATE_LIMITED };+ }+ const message = error instanceof Error ? error.message : String(error);+ if (attempts >= MAX_ATTEMPTS) return { action: 'fail', lastError: message };+ return { action: 'retry', delayMs: equalJitterBackoffMs(attempts), lastError: message };+}diff --git a/apps/worker/src/import-sources.ts b/apps/worker/src/import-sources.tsnew file mode 100644index 00000000..9f94147a--- /dev/null+++ b/apps/worker/src/import-sources.ts@@ -0,0 +1,80 @@+import type { CanonicalStateCategory } from './canonical';+import { PlaneReader, type PlaneCredential } from './plane-adapter';+import type { SourceReader } from './reader';++// The import_job.config fields every source shares; the phases read them directly.+export interface CommonImportConfig {+ unmatchedUserPolicy?: 'unassigned' | 'skip';+ stateOverrides?: Record<string, CanonicalStateCategory>;+}++export interface PlaneImportConfig extends CommonImportConfig {+ planeProjectId: string;+ // Absent on a job created before the Rewrite phase existed; Rewrite then leaves text as written.+ planeProjectKey?: string;+}++// One entry per import_job.source. A new source adds a member to both maps and an+// entry to IMPORT_SOURCES.+export interface ImportCredentials {+ plane: PlaneCredential;+}++export interface ImportConfigs {+ plane: PlaneImportConfig;+}++export type ImportSourceKind = keyof ImportConfigs;+export type ImportCredential = ImportCredentials[ImportSourceKind];+export type ImportConfig = ImportConfigs[ImportSourceKind];++interface ImportSourceDefinition<S extends ImportSourceKind> {+ buildReader(+ credential: ImportCredentials[S],+ config: Partial<ImportConfigs[S]>,+ jobId: number,+ ): SourceReader;+ // The prefix of the source's own issue identifiers ("ROOMS" in "ROOMS-524").+ projectKey(config: Partial<ImportConfigs[S]>): string | null;+}++const IMPORT_SOURCES: { [S in ImportSourceKind]: ImportSourceDefinition<S> } = {+ plane: {+ buildReader(credential, config, jobId) {+ if (!config.planeProjectId) {+ throw new Error(`import job ${jobId} has no source project configured`);+ }+ return new PlaneReader(credential, config.planeProjectId);+ },+ projectKey: (config) => config.planeProjectKey ?? null,+ },+};++export interface ImportJobSource {+ id: number;+ source: string;+ config: Record<string, unknown>;+}++function sourceOf(job: ImportJobSource): ImportSourceKind {+ if (!Object.hasOwn(IMPORT_SOURCES, job.source)) {+ throw new Error(`import job ${job.id} has unsupported source "${job.source}"`);+ }+ return job.source as ImportSourceKind;+}++function definitionOf<S extends ImportSourceKind>(source: S): ImportSourceDefinition<S> {+ return IMPORT_SOURCES[source];+}++export function readerForJob(job: ImportJobSource, credential: ImportCredential): SourceReader {+ return definitionOf(sourceOf(job)).buildReader(+ credential,+ job.config as Partial<ImportConfig>,+ job.id,+ );+}++export function sourceProjectKeyForJob(job: ImportJobSource): string | null {+ return definitionOf(sourceOf(job)).projectKey(job.config as Partial<ImportConfig>);+}diff --git a/apps/worker/src/import-store.ts b/apps/worker/src/import-store.tsindex 50d36b73..15748a94 100644--- a/apps/worker/src/import-store.ts+++ b/apps/worker/src/import-store.ts@@ -29,12 +29,12 @@ import {attachmentObjectKey,} from '@repo/storage';import type { CanonicalState, CanonicalLabel, CanonicalCycle } from './canonical';-import type { PlaneCredential } from './plane-adapter';+import type { ImportCredential } from './import-sources';import { AttachmentRejectedError } from './attachment-download';export { AttachmentRejectedError };-// All @repo/db access for the Plane import: claiming due import_job rows (the+// All @repo/db access for a source import: claiming due import_job rows (the// same FOR UPDATE SKIP LOCKED + lease pattern as store.ts uses for// webhook_delivery), reading/writing a job's progress, the import_record// upsert primitive, and creating the local rows an import produces. The@@ -101,7 +101,7 @@ export async function claimDueImportJobs(limit = 1): Promise<ClaimedImportJob[]>return rows as unknown as ClaimedImportJob[];}-export function decryptImportCredential(job: ClaimedImportJob): PlaneCredential {+export function decryptImportCredential(job: ClaimedImportJob): ImportCredential {if (!job.credentialCiphertext || !job.credentialIv || !job.credentialAuthTag) {throw new Error(`import job ${job.id} has no stored credential`);}@@ -110,7 +110,7 @@ export function decryptImportCredential(job: ClaimedImportJob): PlaneCredentialiv: job.credentialIv,authTag: job.credentialAuthTag,};- return JSON.parse(decryptSecret(encrypted)) as PlaneCredential;+ return JSON.parse(decryptSecret(encrypted)) as ImportCredential;}// A successful tick clears the claim lease (nextAttemptAt) and lastError, anddiff --git a/apps/worker/src/import-worker.ts b/apps/worker/src/import-worker.tsindex 1202f9ff..0337fb4a 100644--- a/apps/worker/src/import-worker.ts+++ b/apps/worker/src/import-worker.ts@@ -1,10 +1,10 @@import { getStorageSettings, mimeAllowed, MB } from '@repo/db';import { startPollLoop, type WorkerHandle } from './poll-loop';import { intEnv } from './env';-import { equalJitterBackoffMs } from './backoff';-import { PlaneReader, PlaneRateLimitedError } from './plane-adapter';+import { tickErrorOutcome } from './import-retry';+import { readerForJob, sourceProjectKeyForJob, type CommonImportConfig } from './import-sources';import type { SourceReader } from './reader';-import type { CanonicalComment, CanonicalStateCategory } from './canonical';+import type { CanonicalComment } from './canonical';import { extractCrossReferences, applyCrossReferenceReplacements } from './cross-reference';import { downloadAttachment } from './attachment-download';import {@@ -41,7 +41,7 @@ import {type ClaimedImportJob,} from './import-store';-// The poll loop that drives a Plane import through its phases: discover ->+// The poll loop that drives a source import through its phases: discover ->// create -> link -> rewrite -> attachments -> done. Follows the same claim-a-lease,// bounded-chunk-per-tick shape as worker.ts and agent-worker.ts. Each tick// claims at most one due import_job and advances it by one bounded unit of@@ -55,11 +55,6 @@ export function startImportWorker(): WorkerHandle {// additional Plane requests (comments, attachments, relations) against a// ~60 requests/minute budget, so this stays well under the discover page size.const ISSUES_PER_TICK = 15;-// import_store.ts resets job.attempts to 0 on every tick that makes progress-// (saveImportJobCursor/advanceImportJobPhase), so this counts consecutive-// failed ticks, not total ticks claimed over the job's life — a multi-tick-// import doesn't fail itself out just by running long.-const MAX_ATTEMPTS = 10;async function tick(): Promise<void> {const [job] = await claimDueImportJobs();@@ -91,37 +86,16 @@ async function tick(): Promise<void> {}async function handleTickError(job: ClaimedImportJob, error: unknown): Promise<void> {- if (error instanceof PlaneRateLimitedError) {- await retryImportJobLater(job.id, error.retryAfterMs, 'rate limited');+ const outcome = tickErrorOutcome(error, job.attempts);+ if (outcome.action === 'fail') {+ await failImportJob(job.id, outcome.lastError);return;}- const message = error instanceof Error ? error.message : String(error);- if (job.attempts >= MAX_ATTEMPTS) {- await failImportJob(job.id, message);- return;- }- await retryImportJobLater(job.id, equalJitterBackoffMs(job.attempts), message);-}--interface PlaneImportConfig {- planeProjectId: string;- // The source project's own short identifier (Plane's "identifier", e.g. "ROOMS"),- // stored at job creation for the Rewrite phase to build "ROOMS-524"-style- // cross-reference patterns from. Optional: a job created before this existed has- // none, and Rewrite is a no-op for it rather than a failure.- planeProjectKey?: string;- // Set from the mapping review step at job creation. Both are optional and default- // to itsaplan's automatic mapping when absent.- unmatchedUserPolicy?: 'unassigned' | 'skip';- stateOverrides?: Record<string, CanonicalStateCategory>;+ await retryImportJobLater(job.id, outcome.delayMs, outcome.lastError);}function buildReader(job: ClaimedImportJob): SourceReader {- const credential = decryptImportCredential(job);- const config = job.config as Partial<PlaneImportConfig>;- if (!config.planeProjectId)- throw new Error(`import job ${job.id} has no source project configured`);- return new PlaneReader(credential, config.planeProjectId);+ return readerForJob(job, decryptImportCredential(job));}// --- Discover: snapshot every source id up front, writing import_record rows@@ -211,7 +185,7 @@ export async function runCreate(job: ClaimedImportJob, reader: SourceReader): Prasync function materializeStates(job: ClaimedImportJob, reader: SourceReader): Promise<void> {const pending = await listUncreatedImportRecords(job.id, 'state', 0, 1000);if (pending.length === 0) return;- const overrides = (job.config as Partial<PlaneImportConfig>).stateOverrides ?? {};+ const overrides = (job.config as CommonImportConfig).stateOverrides ?? {};const states = new Map((await reader.listStates()).map((s) => [s.sourceId, s]));for (const record of pending) {const state = states.get(record.sourceId);@@ -316,7 +290,7 @@ async function createComments(issueLocalId: number,comments: CanonicalComment[],): Promise<void> {- const unmatchedUserPolicy = (job.config as Partial<PlaneImportConfig>).unmatchedUserPolicy;+ const unmatchedUserPolicy = (job.config as CommonImportConfig).unmatchedUserPolicy;const localIdBySourceId = new Map<string, number>();for (const comment of comments) {const existing = await findImportRecord(job.id, 'comment', comment.sourceId);@@ -417,9 +391,9 @@ async function runRewrite(job: ClaimedImportJob): Promise<void> {return;}- // A job created before this phase existed has no planeProjectKey — its+ // A job created before this phase existed has no source project key — its// description/comment text is left exactly as Create wrote it.- const sourceProjectKey = (job.config as Partial<PlaneImportConfig>).planeProjectKey;+ const sourceProjectKey = sourceProjectKeyForJob(job);if (sourceProjectKey) {const localProjectKey = await getProjectKey(job.projectId);for (const record of pending) {@@ -485,8 +459,11 @@ async function runAttachments(job: ClaimedImportJob, reader: SourceReader): Prom`"${attachment.filename}" is type "${attachment.contentType}", not accepted on this instance`,);}- const url = await reader.resolveAttachmentDownloadUrl(record.sourceId, attachment.sourceId);- const bytes = await downloadAttachment(url, limits.maxAttachmentMb * MB);+ const download = await reader.resolveAttachmentDownload(+ record.sourceId,+ attachment.sourceId,+ );+ const bytes = await downloadAttachment(download, limits.maxAttachmentMb * MB);await createLocalAttachmentAndRecord(job.id, attachment.sourceId, {projectId: job.projectId,issueId: record.localId,diff --git a/apps/worker/src/plane-adapter.ts b/apps/worker/src/plane-adapter.tsindex 68426785..12e4c448 100644--- a/apps/worker/src/plane-adapter.ts+++ b/apps/worker/src/plane-adapter.ts@@ -1,5 +1,10 @@import { pinnedFetch } from '@repo/net';-import type { Page, SourceReader } from './reader';+import {+ SourceRateLimitedError,+ type AttachmentDownload,+ type Page,+ type SourceReader,+} from './reader';import type {CanonicalState,CanonicalStateCategory,@@ -24,15 +29,6 @@ export interface PlaneCredential {const PER_PAGE = 100;-// Thrown when Plane's rate limit is hit (or about to be): x-ratelimit-remaining-// reaches 0, or a 429 arrives. The worker catches this and reschedules the job-// instead of treating it as a failed attempt.-export class PlaneRateLimitedError extends Error {- constructor(public readonly retryAfterMs: number) {- super('Plane rate limit reached');- }-}-// A 404 confirmed live to mean "this Plane version doesn't have this// endpoint" (custom properties, work item types, ...) rather than a broken// import, when it happens on one of those. Nothing catches it specially yet —@@ -217,6 +213,7 @@ export class PlaneReader implements SourceReader {constructor(private readonly credential: PlaneCredential,private readonly projectId: string,+ private readonly fetch: typeof pinnedFetch = pinnedFetch,) {}async listStates(): Promise<CanonicalState[]> {@@ -326,10 +323,10 @@ export class PlaneReader implements SourceReader {// Two-hop resolve, confirmed live (see "Attachments" in the notes file): this// call itself 302s to an S3 URL valid for exactly an hour, needing no Plane// auth — never resolved ahead of the moment it's actually downloaded.- async resolveAttachmentDownloadUrl(+ async resolveAttachmentDownload(issueSourceId: string,attachmentSourceId: string,- ): Promise<string> {+ ): Promise<AttachmentDownload> {const path = `/work-items/${issueSourceId}/attachments/${attachmentSourceId}/`;const res = await this.requestRaw(path);const location = res.headers.get('location');@@ -338,7 +335,7 @@ export class PlaneReader implements SourceReader {`Plane attachment resolve did not redirect: GET ${path} -> HTTP ${res.status}`,);}- return location;+ return { url: location };}// members/ is flat, unpaginated, and carries email directly — exactly what@@ -372,11 +369,11 @@ export class PlaneReader implements SourceReader {}// The rate-limit and 404 checks every Plane call needs, without requiring a- // 2xx status — resolveAttachmentDownloadUrl's whole point is a 3xx response.+ // 2xx status — resolveAttachmentDownload's whole point is a 3xx response.private async requestRaw(path: string): Promise<Response> {const base = this.credential.baseUrl.replace(/\/$/, '');const url = `${base}/api/v1/workspaces/${this.credential.workspaceSlug}/projects/${this.projectId}${path}`;- const res = await pinnedFetch(url, {+ const res = await this.fetch(url, {headers: { 'X-Api-Key': this.credential.apiKey },timeoutMs: 15_000,});@@ -386,7 +383,8 @@ export class PlaneReader implements SourceReader {resetEpochSec: res.headers.get('x-ratelimit-reset'),retryAfterSec: res.headers.get('retry-after'),});- if (backoff !== null) throw new PlaneRateLimitedError(backoff);+ // x-ratelimit-remaining reaching 0 counts too, not only a 429.+ if (backoff !== null) throw new SourceRateLimitedError(backoff, 'Plane rate limit reached');if (res.status === 404) throw new PlaneNotFoundError(path);return res;}diff --git a/apps/worker/src/reader.ts b/apps/worker/src/reader.tsindex 279b0f5f..10635731 100644--- a/apps/worker/src/reader.ts+++ b/apps/worker/src/reader.ts@@ -8,6 +8,25 @@ import type {CanonicalAttachment,} from './canonical';+// Thrown by a source adapter when the source's rate limit is reached. The worker+// reschedules the job after retryAfterMs: a rate limit is waited out and never+// fails the job by itself.+export class SourceRateLimitedError extends Error {+ constructor(+ public readonly retryAfterMs: number,+ message = 'source rate limit reached',+ ) {+ super(message);+ }+}++// headers is for a source whose file host needs the source's own credential; the+// Attachments phase sends them with the download and nowhere else.+export interface AttachmentDownload {+ url: string;+ headers?: Record<string, string>;+}+// One page of a cursor-paginated list. cursor is the exact, opaque value the// source API returned for the next page, or null when there is no next page.// It must be passed back to the source verbatim — never reconstructed — and a@@ -40,9 +59,12 @@ export interface SourceReader {listIssueRelations(issueSourceId: string): Promise<CanonicalRelation[]>;listIssueComments(issueSourceId: string): Promise<CanonicalComment[]>;listIssueAttachments(issueSourceId: string): Promise<CanonicalAttachment[]>;- // Resolves one attachment's download URL, fresh each call: the source's own+ // Resolves how to download one attachment, fresh each call: the source's own// resolved URL is a presigned link that expires (Plane's lasts exactly an// hour), so this is called at the moment of download, never cached or// resolved ahead of time.- resolveAttachmentDownloadUrl(issueSourceId: string, attachmentSourceId: string): Promise<string>;+ resolveAttachmentDownload(+ issueSourceId: string,+ attachmentSourceId: string,+ ): Promise<AttachmentDownload>;}diff --git a/docs/dev/plane-import.md b/docs/dev/plane-import.mdindex 040c2476..7726f834 100644--- a/docs/dev/plane-import.md+++ b/docs/dev/plane-import.md@@ -12,8 +12,9 @@ returns.`next_attempt_at`/`last_error` the same shape as `webhook_delivery`, credential columnsencrypted the same way as `integration_credential`) and `import_record` (source id →local id mapping, the idempotency and resume primitive).-- `apps/worker/src/{canonical,reader,plane-adapter,import-store,import-worker}.ts` — the- `SourceReader` port, the only implementation (Plane), and the phase state machine+- `apps/worker/src/{canonical,reader,import-sources,plane-adapter,import-store,import-worker}.ts`+ — the `SourceReader` port, the per-source registry the worker dispatches on+ `import_job.source` through, the only implementation (Plane), and the phase state machine(discover → create → link → rewrite → attachments → done) that drives a job one bounded chunk pertick.- `packages/storage` and `packages/db/src/domains/storage.ts` — object storage and upload@@ -161,10 +162,12 @@ attachment) are a distinct path, not covered by this — see "Inline images" in## Rate limiting and resumability, working as designed`rateLimitBackoffMs` (`plane-adapter.ts`) reads `x-ratelimit-remaining`/`x-ratelimit-reset`/-a 429 off every response; hitting the limit throws `PlaneRateLimitedError`, and-`handleTickError` (`import-worker.ts`) reschedules via `retryImportJobLater` rather than-counting it as a failed attempt. `import_job.last_error` is cleared the moment a claim starts-a new attempt, as well as on success and on completion (`import-store.ts`'s+a 429 off every response; hitting the limit throws `SourceRateLimitedError` (`reader.ts`).+`tickErrorOutcome` (`import-retry.ts`) only decides what a failed tick does — for a rate+limit, retry after the reset with `last_error` `'rate limited'` — and `handleTickError`+(`import-worker.ts`) reschedules the job via `retryImportJobLater`, so a rate limit is waited+out and never fails the job by itself. `import_job.last_error` is cleared the moment a claim+starts a new attempt, as well as on success and on completion (`import-store.ts`'s`claimDueImportJobs`/`saveImportJobCursor`/`advanceImportJobPhase`/`completeImportJob`), so anon-null `lastError` on a still-`pending` job reliably means "currently waiting out a retry,"never "an attempt is in flight" — which is what the Settings page's warning banner reads.