Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: make manuscript issue creation resumable before its final receipt (#10762)
Adam Eivy48d651a475bf
diff --git a/client/src/hooks/useStoryImportIntake.js b/client/src/hooks/useStoryImportIntake.jsindex e0630d4057..129c25564e 100644--- a/client/src/hooks/useStoryImportIntake.js+++ b/client/src/hooks/useStoryImportIntake.js@@ -148,7 +148,7 @@ export default function useStoryImportIntake(onCreated) {const result = await commitImport(payload, { silent: true }).catch((err) => {if (err?.code === 'IMPORTER_PARTIAL_COMMIT_ISSUES' && err?.context?.arcAlreadyPersisted) {patch({ arcAlreadyPersisted: true });- toast.warning('Arc + seasons saved; issues failed and were rolled back. Click again to re-create the issues only — the arc won\'t be re-sent.');+ toast.warning('Arc + seasons saved; the issues did not finish. Click again to create the remaining issues without duplicating any — the arc won\'t be re-sent.');return null;}toast.error(err?.message || 'Import failed');diff --git a/client/src/pages/Importer.jsx b/client/src/pages/Importer.jsxindex 830efc5018..ffded6c8e7 100644--- a/client/src/pages/Importer.jsx+++ b/client/src/pages/Importer.jsx@@ -419,7 +419,7 @@ export default function Importer() {const result = await commitImport(payload, { silent: true }).catch((err) => {if (err?.code === 'IMPORTER_PARTIAL_COMMIT_ISSUES' && err?.context?.arcAlreadyPersisted) {setArcAlreadyPersisted(true);- toast.warning('Arc + seasons saved; issues failed and were rolled back. Retry to re-create the issues only — the arc won\'t be re-sent.');+ toast.warning('Arc + seasons saved; the issues did not finish. Retry to create the remaining issues without duplicating any — the arc won\'t be re-sent.');}throw err;});diff --git a/docs/STORAGE.md b/docs/STORAGE.mdindex 4f0b494545..6ba5d076ed 100644--- a/docs/STORAGE.md+++ b/docs/STORAGE.md@@ -840,7 +840,7 @@ originals and is distinct from space reclamation.### Importer sessions-`data/importer-sessions.json` is `file-primary`, machine-local operational state: for each manuscript imported into a series, how far `POST /api/importer/commit` got (`arc-persisted` after canon/arc/seasons landed, `committed` with the created issue ids). It exists so a reload between the commit and the next step cannot lose the client's own "already committed" markers and re-send the payload (#9943). It is keyed by a derived import id (`imp-` + hash of series id and the normalized manuscript), so a re-analyze of the same text reads the same session back. It is a bounded ledger of what THIS machine's commit did to THIS machine's records, with no foreign keys, cross-record queries, or search, so it stays out of PostgreSQL; each entry is only honored while the issues it recorded still exist. It is deliberately **never federated**: a peer's issues arrive through record sync, and replaying a commit there would describe records that machine did not create. Schema version 1, newest 500 sessions kept, rewritten whole behind a write queue (`services/importerSessions.js`); a payload declaring another version reads as empty rather than half-read, so no migration or `data.reference/` seed is needed. Rsync backups include it.+`data/importer-sessions.json` is `file-primary`, machine-local operational state: for each manuscript imported into a series, how far `POST /api/importer/commit` got (`arc-persisted` after canon/arc/seasons landed, `committed` with the created issue ids). It exists so a reload between the commit and the next step cannot lose the client's own "already committed" markers and re-send the payload (#9943). It is keyed by a derived import id (`imp-` + hash of series id and the normalized manuscript), so a re-analyze of the same text reads the same session back. An `arc-persisted` entry may also carry an additive `plan` (#10762), written before the first issue is created: a stable issue id, arc position and season per proposal, plus a hash binding it to the submitted issue list. The issue writes are idempotent on those planned ids, so a retry after a crash, a failed rollback, or a lost `committed` write keeps the planned issues that exist (with any edits since), recreates only the missing ones, and refuses a different issue list while planned issues survive; when every planned issue exists the entry reads as committed and the retry writes the lost receipt. The `committed` write drops the plan. Entries without a plan (older installs) keep their earlier meaning, and older clients see the same two statuses. It is a bounded ledger of what THIS machine's commit did to THIS machine's records, with no foreign keys, cross-record queries, or search, so it stays out of PostgreSQL; each entry is only honored while the issues it recorded still exist. It is deliberately **never federated**: a peer's issues arrive through record sync, and replaying a commit there would describe records that machine did not create. Schema version 1, newest 500 sessions kept, rewritten whole behind a write queue (`services/importerSessions.js`); a payload declaring another version reads as empty rather than half-read, so no migration or `data.reference/` seed is needed. Rsync backups include it.## Peer administration planning policydiff --git a/server/routes/importer.js b/server/routes/importer.jsindex d269f8a9f2..c382ef5701 100644--- a/server/routes/importer.js+++ b/server/routes/importer.js@@ -15,6 +15,7 @@ import {ERR_VALIDATION,ERR_LOCKED,ERR_PARTIAL_COMMIT_ISSUES,+ ERR_IMPORT_IN_PROGRESS,IMPORTER_SOURCE_CHAR_LIMIT,} from '../services/importer.js';import { ARC_ROLES, ARC_SHAPE_IDS } from '../lib/storyArc.js';@@ -37,9 +38,12 @@ const router = Router();// and crashing on the missing createdIssueIds. 422 makes the helper throw// so the existing error toast surfaces the orchestrator's message// ("universe + series saved, retry to create the remaining issues").+// `ERR_IMPORT_IN_PROGRESS` is a 409: the retry's issue list conflicts with the+// issues an unfinished earlier attempt already created (#10762).const SERVICE_ERROR_STATUS = {[ERR_VALIDATION]: 400,[ERR_LOCKED]: 409,+ [ERR_IMPORT_IN_PROGRESS]: 409,[ERR_PARTIAL_COMMIT_ISSUES]: 422,[universeSvc.ERR_NOT_FOUND]: 404,[seriesSvc.ERR_NOT_FOUND]: 404,diff --git a/server/routes/importer.test.js b/server/routes/importer.test.jsindex 81a5aa284e..df7cca1922 100644--- a/server/routes/importer.test.js+++ b/server/routes/importer.test.js@@ -2,7 +2,9 @@ import { describe, it, expect, vi, beforeEach } from 'vitest';import express from 'express';import { request } from '../lib/testHelper.js';import importerRoutes from './importer.js';-import { ERR_VALIDATION, ERR_LOCKED, ERR_PARTIAL_COMMIT_ISSUES, IMPORTER_SOURCE_CHAR_LIMIT } from '../services/importer.js';+import {+ ERR_VALIDATION, ERR_LOCKED, ERR_PARTIAL_COMMIT_ISSUES, ERR_IMPORT_IN_PROGRESS, IMPORTER_SOURCE_CHAR_LIMIT,+} from '../services/importer.js';import * as universeSvc from '../services/universeBuilder.js';import * as seriesSvc from '../services/pipeline/series.js';import { ARC_ROLES, ARC_SHAPE_IDS } from '../lib/storyArc.js';@@ -206,12 +208,8 @@ describe('POST /api/importer/commit', () => {expect(res.body.code).toBe(ERR_VALIDATION);});- it('returns 409 when the service throws ERR_LOCKED', async () => {- const err = Object.assign(- new Error('Series "X" has a locked arc — commit refused.'),- { code: ERR_LOCKED },- );- importerSvc.commitImport.mockRejectedValue(err);+ it.each([ERR_LOCKED, ERR_IMPORT_IN_PROGRESS])('returns 409 when the service throws %s', async (code) => {+ importerSvc.commitImport.mockRejectedValue(Object.assign(new Error('commit refused'), { code }));const app = buildApp();const res = await request(app).post('/api/importer/commit').send({universeId: 'uni-1',@@ -222,7 +220,7 @@ describe('POST /api/importer/commit', () => {issues: [{ title: 'I1', arcPosition: 1 }],});expect(res.status).toBe(409);- expect(res.body.code).toBe(ERR_LOCKED);+ expect(res.body.code).toBe(code);});it('returns 404 when the service throws the universe not-found code', async () => {diff --git a/server/services/importer.js b/server/services/importer.jsindex f6c262b2ea..d70a3af60d 100644--- a/server/services/importer.js+++ b/server/services/importer.js@@ -10,7 +10,7 @@* future schema change only has to flip the third column.*/-import { randomUUID } from 'crypto';+import { createHash, randomUUID } from 'crypto';import { runStagedLLM } from './stageRunner.js';import { importerEvents, emitImporterProgress } from './importerEvents.js';// Re-export so consumers reach the analyze-phase progress bus through the@@ -30,7 +30,7 @@ import {createSeries,updateSeries,} from './pipeline/series.js';-import { createIssue, deleteIssue, listIssues } from './pipeline/issues.js';+import { deleteIssue, ensureIssueWithId, listIssues, listIssuesForSeries } from './pipeline/issues.js';import { reformatManuscriptText } from './pipeline/manuscriptFix.js';import { sanitizeArc, sanitizeSeasonList, buildSeason, ARC_SHAPE_IDS, ARC_ROLES } from '../lib/storyArc.js';import { COMIC_NUM, COMIC_HEADER_TAIL } from '../lib/comicScriptParser.js';@@ -53,6 +53,10 @@ export const ERR_LOCKED = 'IMPORTER_LOCKED';// already landed. Universe/series are preserved; the partial issue set is// rolled back. Retrying commit is safe (merges are idempotent).export const ERR_PARTIAL_COMMIT_ISSUES = 'IMPORTER_PARTIAL_COMMIT_ISSUES';+// Thrown when a retry sends a DIFFERENT issue list while an earlier attempt's+// planned issues still exist (#10762) — resuming would pair the old issues with+// the wrong proposals, and starting over would duplicate them.+export const ERR_IMPORT_IN_PROGRESS = 'IMPORTER_IMPORT_IN_PROGRESS';// Ordered stages surfaced to the client's live progress checklist during the@@ -708,13 +712,76 @@ export async function classifyImportContent({ source, providerOverride, modelOve* while at least one issue it created still exists: a user who deleted the* imported issues and re-imports the same text is starting over, and treating* that as a replay would silently create nothing.+ *+ * An unfinished session with an issue plan (#10762) is checked against the+ * series: when every planned issue exists, the import finished and only its+ * receipt was lost, so it reads as `committed` with `receiptPending` set.+ * Otherwise `liveIssueIds` names the planned issues that already exist.*/async function resolveImportSession(importId, seriesId) {const session = await getImportSession(importId);if (!session || session.seriesId !== seriesId) return null;- if (session.status !== SESSION_STATUS.COMMITTED) return session;- const live = new Set((await listIssues({ seriesId })).map((issue) => issue.id));- return session.createdIssueIds.some((id) => live.has(id)) ? session : null;+ if (session.status === SESSION_STATUS.ARC_PERSISTED && !session.plan) return session;+ const live = new Set((await listIssuesForSeries(seriesId, { withHistory: false })).map((issue) => issue.id));+ if (session.status === SESSION_STATUS.COMMITTED) {+ return session.createdIssueIds.some((id) => live.has(id)) ? session : null;+ }+ const planIds = session.plan.items.map((item) => item.issueId);+ if (planIds.every((id) => live.has(id))) {+ return {+ ...session,+ status: SESSION_STATUS.COMMITTED,+ createdIssueIds: planIds,+ remappedIssues: session.plan.remappedIssues,+ receiptPending: true,+ };+ }+ return { ...session, liveIssueIds: planIds.filter((id) => live.has(id)) };+}++// Binds an issue plan to the payload it was made for (#10762): the fields that+// decide what each planned issue is. Canon/arc/seasons are left out — a resume+// drops them server-side — and so is `cleanupFormatting`, which only changes+// how the excerpts of the issues still to be created get cleaned.+function hashIssuePayload(issues, contentType) {+ const shape = issues.map((p) => [+ p?.title ?? null, p?.arcPosition ?? null, p?.arcRole ?? null, p?.seasonNumber ?? null,+ p?.proseExcerpt ?? null, p?.logline ?? null, p?.synopsis ?? null,+ ]);+ return createHash('sha256').update(JSON.stringify([contentType ?? null, shape])).digest('hex');+}++/**+ * Promote any import-draft shell once its import's issues all exist (issue+ * #727). analyzeImport creates brand-new universe/series records as import+ * drafts (`ephemeral` + `importDraft`) so an abandoned analyze leaves no+ * syncing orphan; a finished commit turns them into normal records by clearing+ * BOTH flags. Clearing `ephemeral` must go through updateUniverse/updateSeries+ * (they wire the ephemeral→non-ephemeral peer re-subscribe), so the same patch+ * clears `importDraft` too. Gate STRICTLY on `importDraft` — committing an+ * import onto a user's pre-existing private (`ephemeral`-only) record must NOT+ * silently un-privatize it.+ */+async function promoteImportDrafts(universe, series) {+ const promotedUniverse = universe.importDraft === true+ ? await updateUniverse(universe.id, { ephemeral: false, importDraft: false })+ : universe;+ const promotedSeries = series.importDraft === true+ ? await updateSeries(series.id, { ephemeral: false, importDraft: false })+ : series;+ return { universe: promotedUniverse, series: promotedSeries };+}++// The issues exist, so failing to record that must not fail the commit. With+// the issue plan still on disk a retry finishes the receipt without creating+// anything (#10762), so logging is enough.+function recordCommittedReceipt(importId, createdIssueIds, remappedIssues) {+ return recordImportProgress(importId, {+ status: SESSION_STATUS.COMMITTED,+ createdIssueIds,+ remappedIssues,+ plan: null,+ }).catch((err) => console.error(`❌ importer session ${importId} not marked committed (its issue plan stays resumable): ${err.message}`));}const summarizeImportSession = (session) => (session@@ -1111,23 +1178,46 @@ async function commitImportOnce({// the session; additive mode would duplicate, so it does.const session = importId && !replaceMode ? await resolveImportSession(importId, series.id) : null;if (session?.status === SESSION_STATUS.COMMITTED) {- // Answered before the locked-arc gate: nothing is written, so a lock added- // since the commit cannot turn a recorded success into an error.+ // Answered before the locked-arc gate: nothing new is written, so a lock+ // added since the commit cannot turn a recorded success into an error.+ // A receipt lost after the last issue landed (#10762) is completed here.+ let replayedRecords = { universe, series };+ if (session.receiptPending) {+ replayedRecords = await promoteImportDrafts(universe, series);+ await recordCommittedReceipt(importId, session.createdIssueIds, session.remappedIssues);+ }return {- universe,- series,+ ...replayedRecords,createdIssueIds: session.createdIssueIds,remappedIssues: session.remappedIssues,replayed: true,};}+ const payloadHash = hashIssuePayload(issues, contentType);+ // The issue plan an earlier attempt left unfinished (#10762), when this+ // retry carries the same issue list. `resumeLive` holds the planned issues+ // that already exist: they are kept as they are now, never re-created.+ let resumePlan = null;+ const resumeLive = new Set();if (session?.status === SESSION_STATUS.ARC_PERSISTED) {- // The server kept canon/arc/seasons from the attempt whose issues rolled- // back. Resending them would overwrite edits made since, so the server+ // The server kept canon/arc/seasons from the attempt whose issues did not+ // finish. Resending them would overwrite edits made since, so the server// drops them itself rather than trusting the client to have remembered.canonSelections = {};arc = null;seasons = [];+ if (session.plan?.payloadHash === payloadHash) {+ resumePlan = session.plan;+ for (const id of session.liveIssueIds) resumeLive.add(id);+ } else if (session.plan && session.liveIssueIds.length > 0) {+ const k = session.liveIssueIds.length;+ throw codedError(+ `An earlier attempt at this import already created ${k} of its ${session.plan.items.length} issues from a different issue list. Resubmit that issue list to finish it, or delete ${k === 1 ? 'that issue' : 'those issues'} (${session.liveIssueIds.join(', ')}) to start over with this one.`,+ ERR_IMPORT_IN_PROGRESS,+ );+ }+ // Otherwise none of the old plan's issues exist, so nothing is owned and a+ // changed issue list simply plans afresh below.}if (series.locked?.arc === true) {@@ -1216,7 +1306,9 @@ async function commitImportOnce({// off-limits — collisions silently land on disk today since// createIssue doesn't enforce uniqueness.const existingArcPositions = new Set();- if (!replaceMode) {+ // A resumed plan already fixed every position; this import's own surviving+ // issues must not count as collisions with it.+ if (!replaceMode && !resumePlan) {for (const ex of existingIssues) {if (Number.isInteger(ex.arcPosition) && ex.arcPosition >= 1) {existingArcPositions.add(ex.arcPosition);@@ -1239,7 +1331,7 @@ async function commitImportOnce({}const allUsedArcPositions = new Set([...seenArcPositions, ...existingArcPositions]);let nextFreeArcPos = (allUsedArcPositions.size === 0) ? 1 : Math.max(...allUsedArcPositions) + 1;- const issuesWithPositions = issues.map((proposal) => {+ const withAutoPosition = (proposal) => {if (Number.isInteger(proposal.arcPosition) && proposal.arcPosition >= 1) {return proposal;}@@ -1251,7 +1343,12 @@ async function commitImportOnce({}const assigned = nextFreeArcPos++;return { ...proposal, arcPosition: assigned };- });+ };+ // A resume keeps the positions its plan fixed — auto-assigning again would+ // start past this import's own survivors and shift every missing issue.+ const issuesWithPositions = resumePlan+ ? issues.map((proposal, i) => ({ ...proposal, arcPosition: resumePlan.items[i].arcPosition }))+ : issues.map(withAutoPosition);// Same contract for seasons: route Zod enforces `number: int 1..99`,// commitImport mirrors both the floor AND the ceiling so the service is@@ -1285,9 +1382,19 @@ async function commitImportOnce({// Best-effort — `reformatSeededExcerpts` falls back to the verbatim excerpt// per issue, so import never breaks. Returns a fresh array; the create loop// below seeds from it.- const issuesToCreate = cleanupFormatting- ? await reformatSeededExcerpts(issuesWithPositions, { contentType })+ // A resume only reformats the issues it still has to create — the excerpt is+ // dropped from a surviving one so it costs no LLM call (it is never re-written).+ const pendingProposals = resumePlan+ ? issuesWithPositions.map((proposal, i) => (resumeLive.has(resumePlan.items[i].issueId)+ ? { ...proposal, proseExcerpt: undefined }+ : proposal)): issuesWithPositions;+ const issuesToCreate = cleanupFormatting+ ? await reformatSeededExcerpts(pendingProposals, { contentType })+ : pendingProposals;+ // Issues already on the series that this import does not own — the base for+ // the additive issue-count target below.+ const priorIssueCount = existingIssues.filter((ex) => !resumeLive.has(ex.id)).length;// Wipe BEFORE universe + series writes so any delete failure aborts the// commit cleanly — universe canon + series arc are still in their@@ -1402,7 +1509,7 @@ async function commitImportOnce({// Additive: target the POST-import total (issues already on the series plus// the ones being added), not just this batch — adding 3 issues to a series// that already has 2 should target 5, not 3. Only when no target is set yet.- const additiveTotal = existingIssues.length + issueCount;+ const additiveTotal = priorIssueCount + issueCount;if (additiveTotal > 0 && !(series.issueCountTarget > 0)) biblePatch.issueCountTarget = additiveTotal;}@@ -1440,10 +1547,10 @@ async function commitImportOnce({if (arcPremise && !(latest.premise || '').trim()) latestBiblePatch.premise = arcPremise;// Additive: target the POST-import total (issues already on the series// plus the ones being added), not just this batch — adding 3 issues to- // a series that already has 2 should target 5, not 3. `existingIssues`+ // a series that already has 2 should target 5, not 3. `priorIssueCount`// is the up-front count; the issue-create loop below still runs// sequentially against `latest`'s state, so this total stays accurate.- const additiveTotal = existingIssues.length + issueCount;+ const additiveTotal = priorIssueCount + issueCount;if (additiveTotal > 0 && !(latest.issueCountTarget > 0)) latestBiblePatch.issueCountTarget = additiveTotal;return {...(sanitizedArc ? { arc: sanitizedArc } : {}),@@ -1469,53 +1576,73 @@ async function commitImportOnce({const fallbackSeason = sortedSeasons[0] || null;const fallbackSeasonId = fallbackSeason?.id || null;- // Canon, arc and seasons are on disk. Mark that BEFORE the issue loop and let a- // failure here stop the commit: issues created without a way to find them- // again are exactly what a retry would duplicate.+ // Plan every issue's identity BEFORE creating any (#10762): one stable id,+ // position and season per proposal. A replay recognizes these ids instead of+ // minting a second batch, and a planned id that never landed (or was rolled+ // back or deleted since) gets a fresh one here, so a stale tombstone is never+ // resurrected.+ const liveSeasonIds = new Set((updatedSeries.seasons || []).map((s) => s.id));+ const remappedIssues = resumePlan ? [...resumePlan.remappedIssues] : [];+ const planItems = issuesToCreate.map((proposal, i) => {+ const prior = resumePlan?.items[i];+ if (prior) {+ if (resumeLive.has(prior.issueId)) return prior;+ const keepSeason = !prior.seasonId || liveSeasonIds.has(prior.seasonId);+ return { ...prior, issueId: `iss-${randomUUID()}`, seasonId: keepSeason ? prior.seasonId : fallbackSeasonId };+ }+ let seasonId = fallbackSeasonId;+ if (proposal.seasonNumber != null) {+ const matched = seasonByNumber.get(proposal.seasonNumber);+ if (matched) {+ seasonId = matched.id;+ } else {+ // Surface season-remap events so the UI can warn "issue 3 wanted season+ // 5 but landed in S2 — Diaspora." Each entry carries the actual landed+ // season's number + title so the client toast can be specific, not+ // just "first season" (which can lie when seasons are sparsely numbered).+ remappedIssues.push({+ title: proposal.title,+ arcPosition: proposal.arcPosition,+ requestedSeasonNumber: proposal.seasonNumber,+ actualSeasonId: fallbackSeasonId,+ actualSeasonNumber: fallbackSeason?.number ?? null,+ actualSeasonTitle: fallbackSeason?.title ?? null,+ });+ }+ }+ return { issueId: `iss-${randomUUID()}`, arcPosition: proposal.arcPosition, seasonId };+ });++ // Canon, arc and seasons are on disk; record that together with the plan+ // BEFORE the issue loop and let a failure here stop the commit: issues+ // created without a durable record of their ids are exactly what a retry+ // would duplicate.if (importId) {await recordImportProgress(importId, {universeId: updatedUniverse.id,seriesId: updatedSeries.id,status: SESSION_STATUS.ARC_PERSISTED,+ plan: { payloadHash, items: planItems, remappedIssues },});}- const createdIssueIds = [];- // Surface season-remap events so the UI can warn "issue 3 wanted season 5- // but landed in S2 — Diaspora." Each entry carries the actual landed- // season's number + title so the client toast can be specific, not just- // "first season" (which can lie when seasons are sparsely numbered).- const remappedIssues = [];+ // Issues this attempt wrote — the only ones a failure rolls back. Survivors+ // of an earlier attempt belong to the plan and keep any edits made since.+ const createdThisAttempt = [];// Thread C fix — issue-loop with rollback on failure. The universe +- // series are already written above. If createIssue throws mid-loop (e.g.- // transient FS error) we delete every issue created so far and re-throw,- // leaving the universe + series in their updated state but with no partial- // issue set. The universe + series writes are kept because they represent- // user-confirmed data; only the issue set is all-or-nothing from the- // commit's perspective.+ // series are already written above. If an issue write throws mid-loop+ // (e.g. transient FS error) we delete what this attempt created and+ // re-throw, leaving the universe + series in their updated state. The+ // universe + series writes are kept because they represent user-confirmed+ // data. With an importId the plan stays on disk, so the retry recreates+ // only what is missing under fresh planned ids.try {- for (const proposal of issuesToCreate) {- let seasonId = fallbackSeasonId;- if (proposal.seasonNumber != null) {- const matched = seasonByNumber.get(proposal.seasonNumber);- if (matched) {- seasonId = matched.id;- } else {- remappedIssues.push({- title: proposal.title,- arcPosition: proposal.arcPosition,- requestedSeasonNumber: proposal.seasonNumber,- actualSeasonId: fallbackSeasonId,- // Surface the landed season's number + title so the UI can- // render a precise toast ("Issue 'Cold Iron' landed in S2 —- // Diaspora") instead of an inaccurate "first season".- actualSeasonNumber: fallbackSeason?.number ?? null,- actualSeasonTitle: fallbackSeason?.title ?? null,- });- }- }- // Bundle stage seeds into the initial createIssue payload so the+ for (let i = 0; i < issuesToCreate.length; i++) {+ const item = planItems[i];+ if (resumeLive.has(item.issueId)) continue;+ const proposal = issuesToCreate[i];+ // Bundle stage seeds into the initial create payload so the// serialized write tail handles one write per issue instead of// create + updateStage(prose) + updateStage(idea).const stages = {};@@ -1540,33 +1667,45 @@ async function commitImportOnce({// the actual state: input present, generation not yet performed.stages.idea = { status: 'empty', input: ideaSeed };}- const issue = await createIssue({+ const { created } = await ensureIssueWithId(item.issueId, {seriesId: updatedSeries.id,title: proposal.title,- seasonId,- arcPosition: proposal.arcPosition,+ seasonId: item.seasonId,+ arcPosition: item.arcPosition,arcRole: proposal.arcRole,stages,});- createdIssueIds.push(issue.id);+ if (created) createdThisAttempt.push(item.issueId);}} catch (issueErr) {- // Roll back any issues already written so the system isn't left with a- // partial issue set. Rollback failures are logged but don't mask the- // original error — the user gets the real error and can re-commit.- for (const id of createdIssueIds) {- await deleteIssue(id).catch((delErr) =>- console.error(`❌ commitImport rollback: failed to delete issue ${id}: ${delErr.message}`),- );+ // Roll back what this attempt created. A delete that fails leaves its+ // issue in place: it is reported by id rather than claimed as removed,+ // and with an importId the plan still owns it, so the retry keeps it.+ const survivingIssueIds = [];+ for (const id of createdThisAttempt) {+ await deleteIssue(id).catch((delErr) => {+ console.error(`❌ commitImport rollback: failed to delete issue ${id}: ${delErr.message}`);+ survivingIssueIds.push(id);+ });+ }+ const n = issues.length;+ const rolledBack = createdThisAttempt.length - survivingIssueIds.length;+ const plural = (k) => `${k} issue${k === 1 ? '' : 's'}`;+ const notes = [`${plural(rolledBack)} created by this attempt ${rolledBack === 1 ? 'was' : 'were'} rolled back.`];+ if (survivingIssueIds.length > 0) {+ const one = survivingIssueIds.length === 1;+ notes.push(`${plural(survivingIssueIds.length)} could not be removed (${survivingIssueIds.join(', ')}) and ${one ? 'remains' : 'remain'} ${importId+ ? `part of this import — a retry keeps ${one ? 'it' : 'them'} instead of creating a duplicate.`+ : 'in the series.'}`);}+ if (resumeLive.size > 0) notes.push(`${plural(resumeLive.size)} from an earlier attempt ${resumeLive.size === 1 ? 'is' : 'are'} kept.`);// `context.arcAlreadyPersisted` tells the client to drop arc + seasons +// canon from the retry payload — otherwise the retry overwrites any edits// (parallel tab, collaborator) made to the persisted state after the// failure. `skipArcOnRetry` is the imperative form of the same signal.- const n = issues.length;const partial = Object.assign(new Error(- `The universe and series were updated successfully, but ${createdIssueIds.length} of ${n} issue${n === 1 ? '' : 's'} failed and were rolled back — retry to create the remaining issues. (Original error: ${issueErr.message})`,+ `The universe and series were updated successfully, but creating ${n === 1 ? 'the issue' : `the ${n} issues`} failed. ${notes.join(' ')} Retry to create the remaining issues. (Original error: ${issueErr.message})`,),{code: ERR_PARTIAL_COMMIT_ISSUES,@@ -1575,45 +1714,23 @@ async function commitImportOnce({seriesId: updatedSeries.id,arcAlreadyPersisted: true,skipArcOnRetry: true,+ survivingIssueIds,},},);throw partial;}- // Promote any import-draft shell now that the commit fully succeeded (issue- // #727). analyzeImport creates brand-new universe/series records as import- // drafts (`ephemeral` + `importDraft`) so an abandoned analyze leaves no- // syncing orphan; a successful commit turns them into normal records by- // clearing BOTH flags. Clearing `ephemeral` must go through- // updateUniverse/updateSeries (they wire the ephemeral→non-ephemeral peer- // re-subscribe), so the same patch clears `importDraft` too. Gate STRICTLY on- // `importDraft` — committing an import onto a user's pre-existing private- // (`ephemeral`-only) record must NOT silently un-privatize it. Done after the- // issue loop so a rolled-back commit leaves the draft shells GC-eligible.- let promotedUniverse = updatedUniverse;- let promotedSeries = updatedSeries;- if (updatedUniverse.importDraft === true) {- promotedUniverse = await updateUniverse(updatedUniverse.id, { ephemeral: false, importDraft: false });- }- if (updatedSeries.importDraft === true) {- promotedSeries = await updateSeries(updatedSeries.id, { ephemeral: false, importDraft: false });- }+ // Done after the issue loop so a rolled-back commit leaves the draft shells+ // GC-eligible.+ const promoted = await promoteImportDrafts(updatedUniverse, updatedSeries);- // The issues exist now, so a failure to record that must not fail the commit:- // the client would retry and duplicate them. Worst case the next re-analyze- // finds no session and the import is guarded only by the client, as before.- if (importId) {- await recordImportProgress(importId, {- status: SESSION_STATUS.COMMITTED,- createdIssueIds,- remappedIssues,- }).catch((err) => console.error(`❌ importer session ${importId} not marked committed: ${err.message}`));- }+ const createdIssueIds = planItems.map((item) => item.issueId);+ if (importId) await recordCommittedReceipt(importId, createdIssueIds, remappedIssues);return {- universe: promotedUniverse,- series: promotedSeries,+ universe: promoted.universe,+ series: promoted.series,createdIssueIds,remappedIssues,};diff --git a/server/services/importer.test.js b/server/services/importer.test.jsindex 22d88b7144..2d77f89ab2 100644--- a/server/services/importer.test.js+++ b/server/services/importer.test.js@@ -62,33 +62,47 @@ tryReadFile: vi.fn().mockResolvedValue(null),runStagedLLM: (...args) => mockRunStagedLLM(...args),}));-// Mock `./pipeline/issues.js` so a single test can force `createIssue` to-// throw mid-loop and exercise the ERR_PARTIAL_COMMIT_ISSUES rollback path.+// Mock `./pipeline/issues.js` so a single test can force `ensureIssueWithId`+// (the importer's create boundary) to throw mid-loop and exercise the+// ERR_PARTIAL_COMMIT_ISSUES rollback path.// `vi.hoisted` keeps the mock fn ref reachable from both the mock factory// (hoisted before imports) and the beforeEach reset (runs later). Default// behavior passes through to the real module so the dozen other tests that-// expect real createIssue behavior keep working unchanged.+// expect real create behavior keep working unchanged.// `mockDeleteIssue` follows the same pattern so the replaceMode-abort test// can force a delete failure mid-wipe.-const { mockCreateIssue, mockDeleteIssue, realIssuesRef } = vi.hoisted(() => ({- mockCreateIssue: vi.fn(),+const { mockEnsureIssue, mockDeleteIssue, realIssuesRef, sessionFaults } = vi.hoisted(() => ({+ mockEnsureIssue: vi.fn(),mockDeleteIssue: vi.fn(),realIssuesRef: { current: null },+ // Status values whose `recordImportProgress` write rejects — lets a test lose+ // one specific session write (e.g. the final `committed` receipt, #10762).+ sessionFaults: { failStatuses: new Set() },}));vi.mock('./pipeline/issues.js', async () => {const actual = await vi.importActual('./pipeline/issues.js');realIssuesRef.current = actual;return {...actual,- createIssue: (...args) => mockCreateIssue(...args),+ ensureIssueWithId: (...args) => mockEnsureIssue(...args),deleteIssue: (...args) => mockDeleteIssue(...args),};});+vi.mock('./importerSessions.js', async () => {+ const actual = await vi.importActual('./importerSessions.js');+ return {+ ...actual,+ recordImportProgress: (importId, patch) => (sessionFaults.failStatuses.has(patch?.status)+ ? Promise.reject(new Error(`simulated ${patch.status} session write failure`))+ : actual.recordImportProgress(importId, patch)),+ };+});const importerSvc = await import('./importer.js');const universeSvc = await import('./universeBuilder.js');const seriesSvc = await import('./pipeline/series.js');const issuesSvc = await import('./pipeline/issues.js');+const sessionsSvc = await import('./importerSessions.js');// Per-test: wipe every file under tempRoot so each test starts with a clean// data dir. We can't rmSync the dir itself because the universeBuilder@@ -105,11 +119,12 @@ function wipeTempRoot() {beforeEach(() => {wipeTempRoot();mockRunStagedLLM.mockReset();- // Default createIssue + deleteIssue to pass-through to the real module —+ // Default ensureIssueWithId + deleteIssue to pass-through to the real module —// only the rollback and replace-abort tests override these to inject// failures.- mockCreateIssue.mockReset();- mockCreateIssue.mockImplementation((...args) => realIssuesRef.current.createIssue(...args));+ mockEnsureIssue.mockReset();+ mockEnsureIssue.mockImplementation((...args) => realIssuesRef.current.ensureIssueWithId(...args));+ sessionFaults.failStatuses.clear();mockDeleteIssue.mockReset();mockDeleteIssue.mockImplementation((...args) => realIssuesRef.current.deleteIssue(...args));});@@ -1477,10 +1492,10 @@ describe('commitImport', () => {it('rolls back created issues + throws ERR_PARTIAL_COMMIT_ISSUES when createIssue fails mid-loop', async () => {const { uni, ser } = await setupForCommit();let call = 0;- mockCreateIssue.mockImplementation(async (...args) => {+ mockEnsureIssue.mockImplementation(async (...args) => {call++;if (call === 2) throw new Error('simulated mid-loop FS error');- return realIssuesRef.current.createIssue(...args);+ return realIssuesRef.current.ensureIssueWithId(...args);});let caught;@@ -1764,10 +1779,10 @@ describe('commitImport import sessions (#9943)', () => {wireDefaultLLMResponses();const preview = await analyze();let call = 0;- mockCreateIssue.mockImplementation(async (...args) => {+ mockEnsureIssue.mockImplementation(async (...args) => {call++;if (call === 2) throw new Error('simulated mid-loop FS error');- return realIssuesRef.current.createIssue(...args);+ return realIssuesRef.current.ensureIssueWithId(...args);});await expect(importerSvc.commitImport(payloadFor(preview))).rejects.toMatchObject({code: importerSvc.ERR_PARTIAL_COMMIT_ISSUES,@@ -1836,6 +1851,97 @@ describe('commitImport import sessions (#9943)', () => {expect(after.map((i) => i.title)).toEqual(['Replacement']);});+ // #10762: issue identities are planned durably before the first create, so a+ // lost receipt or an interrupted loop resumes instead of minting a second batch.+ it('finishes a commit whose final receipt was lost without creating a second copy', async () => {+ wireDefaultLLMResponses();+ const preview = await analyze();+ // No arcPosition — a retry used to auto-assign the next free slot and land+ // a duplicate at position 2.+ const issues = [{ title: 'Example Chapter', proseExcerpt: 'p1' }];+ sessionFaults.failStatuses.add('committed');+ const first = await importerSvc.commitImport(payloadFor(preview, { issues }));+ sessionFaults.failStatuses.clear();+ expect(first.createdIssueIds).toHaveLength(1);++ const reanalyzed = await analyze();+ expect(reanalyzed.importSession).toEqual({ status: 'committed', createdIssueIds: first.createdIssueIds });+ const retried = await importerSvc.commitImport(payloadFor(reanalyzed, { issues }));++ expect(retried.replayed).toBe(true);+ expect(retried.createdIssueIds).toEqual(first.createdIssueIds);+ const after = await issuesSvc.listIssues({ seriesId: preview.series.id });+ expect(after.map((i) => [i.id, i.arcPosition])).toEqual([[first.createdIssueIds[0], 1]]);+ // The replay wrote the receipt the first attempt lost.+ expect(await sessionsSvc.getImportSession(preview.importId)).toMatchObject({+ status: 'committed', createdIssueIds: first.createdIssueIds, plan: null,+ });+ });++ // The disk state a crash after the first issue leaves: the plan is recorded,+ // one issue exists, and nothing rolled it back.+ const interruptAfterFirstIssue = async (preview, issues) => {+ let call = 0;+ mockEnsureIssue.mockImplementation(async (...args) => {+ call++;+ if (call === 2) throw new Error('simulated crash');+ return realIssuesRef.current.ensureIssueWithId(...args);+ });+ mockDeleteIssue.mockRejectedValue(new Error('simulated delete failure'));+ const err = await importerSvc.commitImport(payloadFor(preview, { issues })).catch((e) => e);+ mockEnsureIssue.mockImplementation((...args) => realIssuesRef.current.ensureIssueWithId(...args));+ mockDeleteIssue.mockImplementation((...args) => realIssuesRef.current.deleteIssue(...args));+ return err;+ };++ it.each([+ ['explicit', [{ title: 'I1', arcPosition: 1, proseExcerpt: 'p1' }, { title: 'I2', arcPosition: 2, proseExcerpt: 'p2' }]],+ ['omitted', [{ title: 'I1', proseExcerpt: 'p1' }, { title: 'I2', proseExcerpt: 'p2' }]],+ ])('resumes an interrupted issue set with %s positions, keeping the survivor and its edits', async (_label, issues) => {+ wireDefaultLLMResponses();+ const preview = await analyze();+ const err = await interruptAfterFirstIssue(preview, issues);+ // The rollback could not remove the survivor, and says so rather than+ // reporting every created issue as rolled back.+ expect(err.code).toBe(importerSvc.ERR_PARTIAL_COMMIT_ISSUES);+ expect(err.context.survivingIssueIds).toHaveLength(1);+ expect(err.message).toMatch(/0 issues created by this attempt were rolled back/);+ expect(err.message).toMatch(/1 issue could not be removed .* remains part of this import/);+ const [survivorId] = err.context.survivingIssueIds;+ await issuesSvc.updateIssue(survivorId, { title: 'I1 (edited)' });++ const reanalyzed = await analyze();+ expect(reanalyzed.importSession.status).toBe('arc-persisted');+ const retried = await importerSvc.commitImport(payloadFor(reanalyzed, { issues }));++ expect(retried.createdIssueIds[0]).toBe(survivorId);+ const after = await issuesSvc.listIssues({ seriesId: preview.series.id });+ expect(after.map((i) => [i.id, i.title, i.arcPosition])).toEqual([+ [survivorId, 'I1 (edited)', 1],+ [retried.createdIssueIds[1], 'I2', 2],+ ]);+ expect((await analyze()).importSession).toEqual({ status: 'committed', createdIssueIds: retried.createdIssueIds });+ });++ it("refuses a changed issue list while an earlier attempt's issues still exist", async () => {+ wireDefaultLLMResponses();+ const preview = await analyze();+ const issues = [{ title: 'I1', arcPosition: 1, proseExcerpt: 'p1' }, { title: 'I2', arcPosition: 2, proseExcerpt: 'p2' }];+ const { context: { survivingIssueIds } } = await interruptAfterFirstIssue(preview, issues);++ await expect(importerSvc.commitImport(payloadFor(preview, {+ issues: [{ title: 'A different split', proseExcerpt: 'x' }],+ }))).rejects.toMatchObject({ code: importerSvc.ERR_IMPORT_IN_PROGRESS });+ expect((await issuesSvc.listIssues({ seriesId: preview.series.id })).map((i) => i.id)).toEqual(survivingIssueIds);++ // Deleting the survivor releases the plan: the new list then imports normally.+ await issuesSvc.deleteIssue(survivingIssueIds[0]);+ const fresh = await importerSvc.commitImport(payloadFor(preview, {+ issues: [{ title: 'A different split', proseExcerpt: 'x' }],+ }));+ expect(fresh.createdIssueIds).toHaveLength(1);+ });+it('commits without an importId exactly as before (older clients)', async () => {const { uni, ser } = await setupForCommit();const body = { universeId: uni.id, seriesId: ser.id, issues: [{ title: 'Solo', proseExcerpt: 'p' }] };diff --git a/server/services/importerSessions.js b/server/services/importerSessions.jsindex 3a31dfbb9b..92954c787a 100644--- a/server/services/importerSessions.js+++ b/server/services/importerSessions.js@@ -32,7 +32,9 @@ const STORE_VERSION = 1;export const IMPORT_SESSION_MAX = 500;export const SESSION_STATUS = Object.freeze({- // Canon + arc + seasons landed; the issue set did not (it rolled back).+ // Canon + arc + seasons landed; the issue set is not finished. With a `plan`,+ // the issues it names are this import's and a retry resumes them (#10762);+ // without one (older installs, or a plan never written) nothing is owned yet.ARC_PERSISTED: 'arc-persisted',// Every issue was created.COMMITTED: 'committed',@@ -58,6 +60,32 @@ export function deriveImportId({ seriesId, source }) {const strArray = (v) => (Array.isArray(v) ? v.filter((x) => typeof x === 'string' && x) : []);+/**+ * The issue-creation plan (#10762), written BEFORE the first issue is created:+ * one stable issue id + resolved arc position + season per proposal, bound to+ * the accepted issue payload by `payloadHash`. Item `i` belongs to proposal `i`.+ * Any malformed item invalidates the whole plan rather than shifting the+ * remaining items onto the wrong proposals.+ */+function sanitizePlan(raw) {+ if (!raw || typeof raw !== 'object' || typeof raw.payloadHash !== 'string' || !raw.payloadHash) return null;+ if (!Array.isArray(raw.items) || raw.items.length === 0) return null;+ const items = [];+ for (const item of raw.items) {+ if (!item || typeof item.issueId !== 'string' || !item.issueId) return null;+ items.push({+ issueId: item.issueId,+ arcPosition: Number.isInteger(item.arcPosition) ? item.arcPosition : null,+ seasonId: typeof item.seasonId === 'string' && item.seasonId ? item.seasonId : null,+ });+ }+ return {+ payloadHash: raw.payloadHash,+ items,+ remappedIssues: Array.isArray(raw.remappedIssues) ? raw.remappedIssues : [],+ };+}+function sanitizeSession(id, raw) {if (!raw || typeof raw !== 'object' || !STATUSES.has(raw.status)) return null;if (typeof raw.seriesId !== 'string' || !raw.seriesId) return null;@@ -68,6 +96,8 @@ function sanitizeSession(id, raw) {status: raw.status,createdIssueIds: strArray(raw.createdIssueIds),remappedIssues: Array.isArray(raw.remappedIssues) ? raw.remappedIssues : [],+ // Only an unfinished session carries a plan; `committed` drops it.+ plan: raw.status === SESSION_STATUS.ARC_PERSISTED ? sanitizePlan(raw.plan) : null,updatedAt: typeof raw.updatedAt === 'string' ? raw.updatedAt : new Date(0).toISOString(),};}@@ -94,7 +124,8 @@ export async function getImportSession(importId) {/*** Upsert a session's progress. `patch` carries `status` and, for a committed- * session, the ids it created. Returns the stored session.+ * session, the ids it created; an `arc-persisted` patch may carry the issue+ * `plan` (pass `plan: null` to drop it). Returns the stored session.*/export async function recordImportProgress(importId, patch) {if (!IMPORT_ID_RE.test(importId || '')) throw new Error(`Invalid import id: ${importId}`);diff --git a/server/services/importerSessions.test.js b/server/services/importerSessions.test.jsindex b13f6a254f..952e9244ce 100644--- a/server/services/importerSessions.test.js+++ b/server/services/importerSessions.test.js@@ -46,6 +46,25 @@ describe('session store', () => {});});+ it('keeps an unfinished import\'s issue plan until the commit lands, then drops it (#10762)', async () => {+ const id = idFor(2);+ const plan = {+ payloadHash: 'hash-1',+ items: [{ issueId: 'iss-a', arcPosition: 1, seasonId: 'sea-1' }, { issueId: 'iss-b', arcPosition: 2, seasonId: null }],+ remappedIssues: [],+ };+ await recordImportProgress(id, { seriesId: 'ser-1', status: 'arc-persisted', plan });+ expect((await getImportSession(id)).plan).toEqual(plan);++ // A malformed item voids the plan rather than mis-pairing items with proposals.+ await recordImportProgress(id, { plan: { ...plan, items: [plan.items[0], { arcPosition: 2 }] } });+ expect((await getImportSession(id)).plan).toBeNull();++ await recordImportProgress(id, { plan });+ await recordImportProgress(id, { status: 'committed', createdIssueIds: ['iss-a', 'iss-b'] });+ expect(await getImportSession(id)).toMatchObject({ status: 'committed', plan: null });+ });+it('keeps only the newest sessions, evicting the oldest first', async () => {const seeded = Object.fromEntries(Array.from({ length: IMPORT_SESSION_MAX }, (_, i) => [idFor(i), {seriesId: 'ser-1', status: 'committed', createdIssueIds: ['iss-1'],diff --git a/server/services/pipeline/issueCrud.js b/server/services/pipeline/issueCrud.jsindex 03bd84f73c..6afb2b5daa 100644--- a/server/services/pipeline/issueCrud.js+++ b/server/services/pipeline/issueCrud.js@@ -148,14 +148,42 @@ export async function getIssue(id, { includeDeleted = false } = {}) {}export function createIssue(input = {}, { preloadedSeries = null } = {}) {+ return queueNewIssue(`iss-${randomUUID()}`, input, { preloadedSeries }).then(({ issue }) => issue);+}++/**+ * Create an issue under an id the caller planned in advance — or, when that id+ * already exists live in the same series, return it untouched. This is the+ * idempotent boundary a resumable batch (the manuscript importer, #10762) needs:+ * replaying a planned create after a crash or a lost receipt recognizes the+ * record it already wrote instead of minting a second identity, and never+ * overwrites edits made to it since. Resolves `{ issue, created }`.+ *+ * A tombstoned id, or one living in another series, is refused (ERR_DUPLICATE)+ * rather than resurrected — the caller re-plans a fresh id for that slot.+ */+export function ensureIssueWithId(id, input = {}) {+ if (!isStr(id) || !ISSUE_ID_RE.test(id)) {+ return Promise.reject(codedError(`ensureIssueWithId: invalid id "${id}" (expected iss-<uuid>)`, ERR_VALIDATION));+ }+ return queueNewIssue(id, input, { reuseExisting: true });+}++// Shared by createIssue (fresh random id) and ensureIssueWithId (planned id).+function queueNewIssue(id, input, { preloadedSeries = null, reuseExisting = false } = {}) {const seriesId = trimTo(input.seriesId, SERIES_ID_MAX);if (!seriesId) return Promise.reject(codedError('seriesId is required', ERR_VALIDATION));const title = trimTo(input.title, TITLE_MAX);if (!title) return Promise.reject(codedError(`title is required (1..${TITLE_MAX} chars)`, ERR_VALIDATION));return queueSeriesIssuesWrite(seriesId, async () => {const state = await readState();+ if (reuseExisting) {+ const existing = state.issues.find((i) => i.id === id);+ if (existing && !existing.deleted && existing.seriesId === seriesId) return { issue: existing, created: false };+ if (existing) throw codedError(`Issue id already used: ${id}`, ERR_DUPLICATE);+ }const next = sanitizeIssue({- id: `iss-${randomUUID()}`,+ id,seriesId,// Placeholder — `renumberInline` below derives the canonical number.number: 0,@@ -180,7 +208,7 @@ export function createIssue(input = {}, { preloadedSeries = null } = {}) {await saveIssuesNow(state.issues.filter((i) => i.seriesId === seriesId));// New issue = series-level change for any active share subscription.emitRecordUpdated('series', next.seriesId);- return next;+ return { issue: next, created: true };});}