Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

refactor: give manuscript import one admission phase instead of retesting replace and resume (#11111)

Adam Eivy1e030449493b
diff --git a/server/services/importer.js b/server/services/importer.js
index 723ff53ff9..39271a723d 100644
--- a/server/services/importer.js
+++ b/server/services/importer.js
@@ -1164,120 +1164,89 @@ async function commitImportOnce(args) {
return seriesCommitQueue(seriesId, () => admitAndCommit(args, { formattedExcerpts }));
}
-// Validates and plans the commit against fresh state, then writes it. Runs
-// inside the series admission, except for the `precheck` pass (a fail-fast
-// before formatting, which writes nothing): that returns `{ replay: true }`
-// for an import already committed, else `{ pendingProposals }` — the excerpts
-// the formatting pass should clean. `formattedExcerpts[i]` (from that pass)
-// replaces proposal i's excerpt when it is still to be created.
-async function admitAndCommit({
- universeId,
- seriesId,
- importId = null,
- canonSelections = {},
- arc = null,
- seasons = [],
- issues = [],
- // Content type drives which stage each issue's verbatim excerpt seeds (see
- // the stage-seed block below). Absent → prose-seed (prior behavior).
- contentType = null,
- // Destructive replace: wipes existing issues, overwrites arc + seasons.
- // Universe canon still merges additively (it's shared across series, so
- // a per-series destructive replace would be too coarse). Default false.
- replaceMode = false,
-} = {}, { precheck = false, formattedExcerpts = null } = {}) {
- // Read for validation (universeId/seriesId linkage, lock checks, the
- // arcPosition/season-number gates below) and for the up-front counts used
- // to size auto-assigned positions and the additive issue-count target.
- // The series admission keeps other importer commits out of this series, but
- // NOT edits made outside the importer, and the universe is shared across
- // series — so this snapshot is not the source of truth the universe/series
- // writes below merge against. Those use the mutator form of
- // `updateUniverse`/`updateSeries`, which re-reads the freshest persisted
- // record inside the write queue (issue #8453).
- const universe = await getUniverse(universeId);
- const series = await getSeries(seriesId);
-
- // Defensive: refuse commit when series.universeId is set AND differs,
- // OR when series.universeId is missing entirely. A legacy series
- // persisted before universeId became standard would otherwise silently
- // be re-homed to whichever universe the caller routed through. createSeries
- // sets universeId today, so a missing value implies hand-edited / pre-link
- // legacy data — flag it so the caller decides rather than silently linking.
- if (series.universeId !== universe.id) {
- if (series.universeId) {
- throw codedError(
- `Series "${series.name}" is linked to a different universe — commit refused to avoid cross-linking.`,
- ERR_VALIDATION,
- );
- }
- throw codedError(
- `Series "${series.name}" has no universeId — commit refused. Link the series to a universe explicitly before importing.`,
- ERR_VALIDATION,
- );
- }
+// Where a commit stands against the earlier attempts at the same import — one
+// reading, made before any write, that every later step switches on (#11111).
+const IMPORT_PHASE = Object.freeze({
+ // An already-committed import seen by the precheck pass: it writes nothing.
+ REPLAY_PRECHECK: 'replay-precheck',
+ // An already-committed import: answer with the recorded ids.
+ REPLAY: 'replay',
+ // An unfinished plan for this same issue list: finish it.
+ RESUME: 'resume',
+ // No usable session: plan everything, positions checked against the series.
+ FRESH: 'fresh',
+ // Destructive replace: wipe the series' issues and rewrite it.
+ REPLACE: 'replace',
+});
- // An earlier attempt at this same import (#9943). Replace mode wipes and
- // rewrites the series, so it is idempotent by construction and never consults
- // the session; additive mode would duplicate, so it does.
- const session = importId && !replaceMode ? await resolveImportSession(importId, series.id) : null;
+/**
+ * The single admission decision for a commit (#11111): which phase it is in,
+ * read from the on-disk import session. Reads only — it writes nothing, so a
+ * commit is refused (or replayed) before any canon or issue write.
+ *
+ * - `arcPersisted`: an earlier attempt already wrote canon/arc/seasons
+ * (`ARC_PERSISTED`), so this commit must not resend them.
+ * - `resumePlan` / `resumeLive`: the unfinished plan and the planned issues
+ * that already exist — kept as they are now, never re-created.
+ */
+async function resolveImportAdmission({ importId, replaceMode, precheck, series, payloadHash }) {
+ const admission = { arcPersisted: false, resumePlan: null, resumeLive: new Set() };
+ // Replace mode wipes and rewrites the series, so it is idempotent by
+ // construction and never consults the session (#9943); additive mode would
+ // duplicate, so it does.
+ if (replaceMode) return { ...admission, phase: IMPORT_PHASE.REPLACE };
+ const session = importId ? await resolveImportSession(importId, series.id) : null;
if (session?.status === SESSION_STATUS.COMMITTED) {
- // The precheck writes nothing; admission answers the replay.
- if (precheck) return { replay: true };
- // Answered before the locked-arc gate: nothing new is written, so a lock
- // added since the commit cannot turn a recorded success into an error.
- // A receipt lost after the last issue landed (#10762) is completed here.
- let replayedRecords = { universe, series };
- if (session.receiptPending) {
- replayedRecords = await promoteImportDrafts(universe, series);
- await recordCommittedReceipt(importId, session.createdIssueIds, session.remappedIssues);
- }
+ if (precheck) return { ...admission, phase: IMPORT_PHASE.REPLAY_PRECHECK };
+ const { createdIssueIds, remappedIssues, receiptPending = false } = session;
+ return { ...admission, phase: IMPORT_PHASE.REPLAY, createdIssueIds, remappedIssues, receiptPending };
+ }
+ if (session?.status !== SESSION_STATUS.ARC_PERSISTED) return { ...admission, phase: IMPORT_PHASE.FRESH };
+ // The issue plan an earlier attempt left unfinished (#10762), when this retry
+ // carries the same issue list.
+ if (session.plan?.payloadHash === payloadHash) {
return {
- ...replayedRecords,
- createdIssueIds: session.createdIssueIds,
- remappedIssues: session.remappedIssues,
- replayed: true,
+ phase: IMPORT_PHASE.RESUME,
+ arcPersisted: true,
+ resumePlan: session.plan,
+ resumeLive: new Set(session.liveIssueIds),
};
}
- const payloadHash = hashIssuePayload(issues, contentType);
- // The issue plan an earlier attempt left unfinished (#10762), when this
- // retry carries the same issue list. `resumeLive` holds the planned issues
- // that already exist: they are kept as they are now, never re-created.
- let resumePlan = null;
- const resumeLive = new Set();
- if (session?.status === SESSION_STATUS.ARC_PERSISTED) {
- // The server kept canon/arc/seasons from the attempt whose issues did not
- // finish. Resending them would overwrite edits made since, so the server
- // drops them itself rather than trusting the client to have remembered.
- canonSelections = {};
- arc = null;
- seasons = [];
- if (session.plan?.payloadHash === payloadHash) {
- resumePlan = session.plan;
- for (const id of session.liveIssueIds) resumeLive.add(id);
- } else if (session.plan && session.liveIssueIds.length > 0) {
- const k = session.liveIssueIds.length;
- throw codedError(
- `An earlier attempt at this import already created ${k} of its ${session.plan.items.length} issues from a different issue list. Resubmit that issue list to finish it, or delete ${k === 1 ? 'that issue' : 'those issues'} (${session.liveIssueIds.join(', ')}) to start over with this one.`,
- ERR_IMPORT_IN_PROGRESS,
- );
- }
- // Otherwise none of the old plan's issues exist, so nothing is owned and a
- // changed issue list simply plans afresh below.
- }
-
- if (series.locked?.arc === true) {
+ if (session.plan && session.liveIssueIds.length > 0) {
+ const k = session.liveIssueIds.length;
throw codedError(
- `Series "${series.name}" has a locked arc — commit refused. Unlock the arc to import.`,
- ERR_LOCKED,
+ `An earlier attempt at this import already created ${k} of its ${session.plan.items.length} issues from a different issue list. Resubmit that issue list to finish it, or delete ${k === 1 ? 'that issue' : 'those issues'} (${session.liveIssueIds.join(', ')}) to start over with this one.`,
+ ERR_IMPORT_IN_PROGRESS,
);
}
+ // None of the old plan's issues exist, so nothing is owned and a changed
+ // issue list simply plans afresh — but canon/arc/seasons are still on disk.
+ return { ...admission, phase: IMPORT_PHASE.FRESH, arcPersisted: true };
+}
- // Thread C fix — fail-fast validation BEFORE any state mutation. Run
- // every issue through all gates that createIssue enforces so that a bad
- // payload is rejected here, before the universe + series are written.
- // `createIssue` also requires `seriesId` (supplied below) and generates
- // its own `id`, so those two fields don't need pre-checking.
+// A replayed commit writes nothing new. A receipt lost after the last issue
+// landed (#10762) is completed here: promote the draft shells, then record it.
+async function replayCommittedImport(importId, admission, records) {
+ let replayedRecords = records;
+ if (admission.receiptPending) {
+ replayedRecords = await promoteImportDrafts(records.universe, records.series);
+ await recordCommittedReceipt(importId, admission.createdIssueIds, admission.remappedIssues);
+ }
+ return {
+ ...replayedRecords,
+ createdIssueIds: admission.createdIssueIds,
+ remappedIssues: admission.remappedIssues,
+ replayed: true,
+ };
+}
+
+// Thread C fix — fail-fast validation BEFORE any state mutation. Run every
+// issue through all gates that createIssue enforces so that a bad payload is
+// rejected here, before the universe + series are written. `createIssue` also
+// requires `seriesId` (supplied later) and generates its own `id`, so those
+// two fields don't need pre-checking. Returns the explicit arcPositions, which
+// the auto-assign pass must not reuse.
+function rejectInvalidIssueProposals(issues) {
for (let i = 0; i < issues.length; i++) {
const proposal = issues[i];
if (!isStr(proposal?.title) || !proposal.title.trim()) {
@@ -1290,7 +1259,7 @@ async function admitAndCommit({
// int 1..ARC_POSITION_MAX at the route layer, but commitImport is
// also called directly from tests + future internal callers; mirror
// both the floor and the ceiling here so the service contract holds
- // regardless of caller. Allow undefined — an auto-assign pass below
+ // regardless of caller. Allow undefined — an auto-assign pass
// picks the next free slot. Reject anything that isn't a valid
// integer in [1, ARC_POSITION_MAX].
if (proposal.arcPosition !== undefined && proposal.arcPosition !== null
@@ -1317,8 +1286,7 @@ async function admitAndCommit({
// Reject duplicate explicit arcPosition values across the incoming
// issues array — downstream sorting collapses ties to insertion order
// and the renumber pass would silently re-key, leaving the user
- // wondering why issue #3 became #4. Auto-assign for omitted positions
- // happens below.
+ // wondering why issue #3 became #4.
const seenArcPositions = new Set();
for (let i = 0; i < issues.length; i++) {
const pos = issues[i]?.arcPosition;
@@ -1332,39 +1300,46 @@ async function admitAndCommit({
seenArcPositions.add(pos);
}
}
- // Auto-assign sequential arcPositions to incoming issues that omit one.
- // Mirrors the seasons auto-assign in the merge path. Issues with an
- // explicit position keep it; gaps get filled by the next free integer.
- // Seed nextFree from the UNION of explicit incoming positions AND any
- // pre-existing series.issues[].arcPosition — re-import on a series
- // that already has issues at [1..3] would otherwise auto-assign
- // starting at 1 again and create duplicate positions. createIssue
- // doesn't enforce arcPosition uniqueness, so the collision would
- // silently land on disk and the renumber pass would arbitrarily
- // order the ties.
- // Cap at ARC_POSITION_MAX (matching the wire schema's `.max(9999)`)
- // so a value the wire wouldn't accept can't reach createIssue via the
- // service auto-assign path.
- const existingIssues = await listIssues({ seriesId: series.id });
- // In replace mode we'll wipe these before creating the new set, so any
- // arcPosition that's currently occupied by an existing issue is fair
- // game. In additive merge mode (default), occupied positions are
- // off-limits — collisions silently land on disk today since
- // createIssue doesn't enforce uniqueness.
+ return seenArcPositions;
+}
+
+/**
+ * Every incoming issue with its arcPosition, refused when a position would
+ * collide with an issue already on the series.
+ *
+ * - `resume` keeps the positions its plan fixed — auto-assigning again would
+ * start past this import's own survivors and shift every missing issue. It
+ * still owes the series unique positions: since the earlier attempt, a new
+ * issue — or one of this import's survivors, moved — may have taken a slot
+ * the plan reserved for a missing issue. Survivors left where the plan put
+ * them never collide: the plan's positions are distinct.
+ * - `fresh` refuses an explicit position an existing issue holds, and seeds
+ * auto-assign from the UNION of explicit incoming and existing positions —
+ * createIssue doesn't enforce uniqueness, so a collision would silently land
+ * on disk and the renumber pass would arbitrarily order the ties.
+ * - `replace` wipes the existing issues first, so their positions are free.
+ *
+ * Auto-assign is capped at ARC_POSITION_MAX (matching the wire schema's
+ * `.max(9999)`) so a value the wire wouldn't accept can't reach createIssue.
+ */
+function assignIssuePositions({ phase, resumePlan, resumeLive }, issues, existingIssues, explicitPositions) {
+ if (phase === IMPORT_PHASE.RESUME) {
+ const taken = new Set(existingIssues.map((ex) => ex.arcPosition).filter(Number.isInteger));
+ resumePlan.items.forEach((item, i) => {
+ if (!resumeLive.has(item.issueId) && taken.has(item.arcPosition)) {
+ throw codedError(
+ `Issue at position ${i + 1} was planned for arcPosition ${item.arcPosition}, which another issue now occupies — commit refused before any state changed. Move or delete that issue, then retry to finish this import.`,
+ ERR_VALIDATION,
+ );
+ }
+ });
+ return issues.map((proposal, i) => ({ ...proposal, arcPosition: resumePlan.items[i].arcPosition }));
+ }
const existingArcPositions = new Set();
- // A resumed plan already fixed every position; this import's own surviving
- // issues must not count as collisions with it.
- if (!replaceMode && !resumePlan) {
+ if (phase === IMPORT_PHASE.FRESH) {
for (const ex of existingIssues) {
- if (Number.isInteger(ex.arcPosition) && ex.arcPosition >= 1) {
- existingArcPositions.add(ex.arcPosition);
- }
+ if (Number.isInteger(ex.arcPosition) && ex.arcPosition >= 1) existingArcPositions.add(ex.arcPosition);
}
- // Symmetry with the auto-assign branch: reject incoming issues whose
- // EXPLICIT arcPosition collides with an arcPosition already used by an
- // existing issue on the series. The auto-assign branch dodges this via
- // the union-set seed; the explicit-position branch would otherwise let
- // createIssue happily land a duplicate.
for (let i = 0; i < issues.length; i++) {
const pos = issues[i]?.arcPosition;
if (Number.isInteger(pos) && pos >= 1 && existingArcPositions.has(pos)) {
@@ -1375,47 +1350,26 @@ async function admitAndCommit({
}
}
}
- // A resume still owes the series unique positions: since the earlier attempt,
- // a new issue — or one of this import's survivors, moved — may have taken a
- // slot the plan reserved for a missing issue. Survivors left where the plan
- // put them never collide: the plan's positions are distinct.
- if (!replaceMode && resumePlan) {
- const taken = new Set(existingIssues.map((ex) => ex.arcPosition).filter(Number.isInteger));
- resumePlan.items.forEach((item, i) => {
- if (!resumeLive.has(item.issueId) && taken.has(item.arcPosition)) {
- throw codedError(
- `Issue at position ${i + 1} was planned for arcPosition ${item.arcPosition}, which another issue now occupies — commit refused before any state changed. Move or delete that issue, then retry to finish this import.`,
- ERR_VALIDATION,
- );
- }
- });
- }
- const allUsedArcPositions = new Set([...seenArcPositions, ...existingArcPositions]);
+ const allUsedArcPositions = new Set([...explicitPositions, ...existingArcPositions]);
let nextFreeArcPos = (allUsedArcPositions.size === 0) ? 1 : Math.max(...allUsedArcPositions) + 1;
- const withAutoPosition = (proposal) => {
- if (Number.isInteger(proposal.arcPosition) && proposal.arcPosition >= 1) {
- return proposal;
- }
+ return issues.map((proposal) => {
+ if (Number.isInteger(proposal.arcPosition) && proposal.arcPosition >= 1) return proposal;
if (nextFreeArcPos > ARC_POSITION_MAX) {
throw codedError(
`Cannot auto-assign arcPosition — next free slot (${nextFreeArcPos}) exceeds the max of ${ARC_POSITION_MAX}. Free up a position or set issue.arcPosition explicitly.`,
ERR_VALIDATION,
);
}
- const assigned = nextFreeArcPos++;
- return { ...proposal, arcPosition: assigned };
- };
- // A resume keeps the positions its plan fixed — auto-assigning again would
- // start past this import's own survivors and shift every missing issue.
- const issuesWithPositions = resumePlan
- ? issues.map((proposal, i) => ({ ...proposal, arcPosition: resumePlan.items[i].arcPosition }))
- : issues.map(withAutoPosition);
-
- // Same contract for seasons: route Zod enforces `number: int 1..99`,
- // commitImport mirrors both the floor AND the ceiling so the service is
- // safe under direct calls. Also reject duplicate season numbers — the
- // merge keys by `number`, so two incoming seasons sharing one would
- // silently collapse into a single entry post sanitizeSeasonList.
+ return { ...proposal, arcPosition: nextFreeArcPos++ };
+ });
+}
+
+// Same contract for seasons: route Zod enforces `number: int 1..99`,
+// commitImport mirrors both the floor AND the ceiling so the service is
+// safe under direct calls. Also reject duplicate season numbers — the
+// merge keys by `number`, so two incoming seasons sharing one would
+// silently collapse into a single entry post sanitizeSeasonList.
+function rejectInvalidSeasons(seasons) {
const seenSeasonNumbers = new Set();
for (let i = 0; i < seasons.length; i++) {
const s = seasons[i];
@@ -1435,207 +1389,191 @@ async function admitAndCommit({
seenSeasonNumbers.add(s.number);
}
}
+}
- // Optional AI cleanup of the seeded excerpts (issue #1335) runs between the
- // precheck pass and admission (see commitImportOnce): AFTER all the cheap
- // fail-fast validation above (so a bad payload never burns LLM calls) but
- // BEFORE any destructive write (the wipe + universe/series writes below).
- // Best-effort — `reformatSeededExcerpts` falls back to the verbatim excerpt
- // per issue, so import never breaks.
- // A resume only reformats the issues it still has to create — the excerpt is
- // dropped from a surviving one so it costs no LLM call (it is never re-written).
- const pendingProposals = resumePlan
- ? issuesWithPositions.map((proposal, i) => (resumeLive.has(resumePlan.items[i].issueId)
- ? { ...proposal, proseExcerpt: undefined }
- : proposal))
- : issuesWithPositions;
- if (precheck) return { pendingProposals };
- // An issue that became pending only since the precheck (a survivor deleted in
- // between) has no cleaned excerpt and keeps its verbatim one.
- const issuesToCreate = formattedExcerpts
- ? pendingProposals.map((proposal, i) => (proposal.proseExcerpt && isStr(formattedExcerpts[i])
- ? { ...proposal, proseExcerpt: formattedExcerpts[i] }
- : proposal))
- : pendingProposals;
- // Issues already on the series that this import does not own — the base for
- // the additive issue-count target below.
- const priorIssueCount = existingIssues.filter((ex) => !resumeLive.has(ex.id)).length;
-
- // Wipe BEFORE universe + series writes so any delete failure aborts the
- // commit cleanly — universe canon + series arc are still in their
- // pre-import state, and no new issues have been created yet. Without this
- // ordering, a swallowed delete failure would leave us writing new issues
- // alongside undeleted old ones; the additive-mode collision check is
- // skipped in replace mode (existingArcPositions stays empty), so reused
- // arcPositions would silently produce duplicates on disk that additive
- // mode explicitly rejects.
- //
- // Abort on the FIRST delete failure (not after looping through all of
- // them) to minimize the destructive surface — deletes already landed are
- // unrecoverable, so the fewer that completed before the throw, the more
- // of the user's pre-import state survives. The error message names the
- // succeeded vs remaining sets explicitly so the user can audit what's
- // gone before retrying. Universe + series writes below are idempotent,
- // so re-running the import after the user fixes the underlying error
- // (and accepts the partially-deleted state) is safe.
- if (replaceMode && existingIssues.length > 0) {
- const deletedIds = [];
- for (const ex of existingIssues) {
- let failed = null;
- await deleteIssue(ex.id).catch((delErr) => {
- console.error(`❌ commitImport replace: failed to delete existing issue ${ex.id}: ${delErr.message}`);
- failed = delErr;
- });
- if (failed) {
- // The "still on disk" set includes BOTH the failed-to-delete issue
- // AND the untouched issues that come after it in the loop — all of
- // them are unaffected on disk and will be wiped on retry. Reporting
- // them as one set (instead of singling out `ex.id` separately) keeps
- // the count honest: total still-on-disk == remainingIds.length.
- const deletedSet = new Set(deletedIds);
- const remainingIds = existingIssues
- .map((e) => e.id)
- .filter((id) => !deletedSet.has(id));
- const deletedMsg = deletedIds.length > 0
- ? ` ${deletedIds.length} issue${deletedIds.length === 1 ? '' : 's'} were already deleted before the failure (${deletedIds.join(', ')}) and cannot be recovered.`
- : '';
- const remainingMsg = ` ${remainingIds.length} issue${remainingIds.length === 1 ? '' : 's'} remain on disk (${remainingIds.join(', ')}, including the failed one); retry will wipe them.`;
- throw codedError(
- `Replace mode aborted on first delete failure — issue ${ex.id} could not be deleted (${failed.message}). No universe, series, or new-issue writes were performed.${deletedMsg}${remainingMsg} Resolve the underlying error and retry.`,
- ERR_VALIDATION,
- );
- }
- deletedIds.push(ex.id);
+// Replace mode's wipe, run BEFORE universe + series writes so any delete
+// failure aborts the commit cleanly — universe canon + series arc are still in
+// their pre-import state, and no new issues have been created yet. Without
+// this ordering, a swallowed delete failure would leave us writing new issues
+// alongside undeleted old ones; replace mode skips the collision check, so
+// reused arcPositions would silently produce duplicates on disk that additive
+// mode explicitly rejects.
+//
+// Abort on the FIRST delete failure (not after looping through all of them) to
+// minimize the destructive surface — deletes already landed are
+// unrecoverable, so the fewer that completed before the throw, the more of the
+// user's pre-import state survives. The error message names the succeeded vs
+// remaining sets explicitly so the user can audit what's gone before
+// retrying. Universe + series writes are idempotent, so re-running the import
+// after the user fixes the underlying error (and accepts the partially-deleted
+// state) is safe.
+async function deleteIssuesForReplace(existingIssues) {
+ const deletedIds = [];
+ for (const ex of existingIssues) {
+ let failed = null;
+ await deleteIssue(ex.id).catch((delErr) => {
+ console.error(`❌ commitImport replace: failed to delete existing issue ${ex.id}: ${delErr.message}`);
+ failed = delErr;
+ });
+ if (failed) {
+ // The "still on disk" set includes BOTH the failed-to-delete issue AND
+ // the untouched issues that come after it in the loop — all of them are
+ // unaffected on disk and will be wiped on retry. Reporting them as one
+ // set keeps the count honest: total still-on-disk == remainingIds.length.
+ const deletedSet = new Set(deletedIds);
+ const remainingIds = existingIssues.map((e) => e.id).filter((id) => !deletedSet.has(id));
+ const deletedMsg = deletedIds.length > 0
+ ? ` ${deletedIds.length} issue${deletedIds.length === 1 ? '' : 's'} were already deleted before the failure (${deletedIds.join(', ')}) and cannot be recovered.`
+ : '';
+ const remainingMsg = ` ${remainingIds.length} issue${remainingIds.length === 1 ? '' : 's'} remain on disk (${remainingIds.join(', ')}, including the failed one); retry will wipe them.`;
+ throw codedError(
+ `Replace mode aborted on first delete failure — issue ${ex.id} could not be deleted (${failed.message}). No universe, series, or new-issue writes were performed.${deletedMsg}${remainingMsg} Resolve the underlying error and retry.`,
+ ERR_VALIDATION,
+ );
}
+ deletedIds.push(ex.id);
}
+}
+
+// Additive bible fill (the logline/premise/issueCountTarget the Arc Canvas
+// sidebar reads): only fields the series doesn't already have, so a re-import
+// never clobbers a hand-edited bible. The issue-count target is the POST-import
+// total (issues already on the series plus the ones being added) — adding 3
+// issues to a series that already has 2 should target 5, not 3. Evaluated only
+// against the freshest persisted record, inside the series write (#11111).
+function fillEmptySeriesBible(latest, { arcLogline, arcPremise, priorIssueCount, issueCount }) {
+ const patch = {};
+ if (arcLogline && !(latest.logline || '').trim()) patch.logline = arcLogline;
+ if (arcPremise && !(latest.premise || '').trim()) patch.premise = arcPremise;
+ const additiveTotal = priorIssueCount + issueCount;
+ if (additiveTotal > 0 && !(latest.issueCountTarget > 0)) patch.issueCountTarget = additiveTotal;
+ return patch;
+}
+
+// Wire field, BIBLE_KIND, and storage field collapsed to one row per kind so a
+// future schema change updates one tuple instead of three call sites.
+const CANON_KIND_MAP = [
+ ['characters', BIBLE_KIND.CHARACTER, 'characters'],
+ ['places', BIBLE_KIND.PLACE, 'places'],
+ ['objects', BIBLE_KIND.OBJECT, 'objects'],
+];
- // Wire field, BIBLE_KIND, and storage field collapsed to one row per kind
- // so a future schema change updates one tuple instead of three call sites.
- const KIND_MAP = [
- ['characters', BIBLE_KIND.CHARACTER, 'characters'],
- ['places', BIBLE_KIND.PLACE, 'places'],
- ['objects', BIBLE_KIND.OBJECT, 'objects'],
- ];
+/**
+ * Writes the import's universe canon and the series arc, seasons and bible.
+ * Universe canon always merges additively (it's shared across series, so a
+ * per-series destructive replace would be too coarse). Replace overwrites the
+ * series; every other phase merges into it.
+ *
+ * Both writes use the mutator form of `updateUniverse`/`updateSeries` where
+ * they merge (issue #8453): the series admission keeps other importer commits
+ * out of this series, but NOT edits made outside the importer, and the
+ * universe is shared across series — so the up-front snapshots can already be
+ * stale. `mergeExtractedBible`/`mergeSeasons` rebuild the whole array from
+ * their base list, so merging against a snapshot would silently drop a
+ * concurrent edit. Merging against `latest` — the freshest persisted record,
+ * read inside the write queue — closes that race.
+ */
+async function applyImportedCanon({ phase, universe, series, canonSelections, arc, seasons, priorIssueCount, issueCount }) {
// Only merge kinds the user actually supplied entries for — calling
// mergeExtractedBible with an empty list still rebuilds the array and
- // re-stamps timestamps, churning the file write for no behavior change.
- // Selection kinds are fixed by `canonSelections` (the AI-extraction output),
- // not by anything on `universe`, so this gate is safe to evaluate against
- // the up-front snapshot even though the write below re-reads fresh state.
- const selectedKindMap = KIND_MAP
+ // re-stamps timestamps, churning the file write for no behavior change. An
+ // arc-only import skips the updateUniverse round-trip entirely.
+ const selectedKindMap = CANON_KIND_MAP
.filter(([selectionKey]) => (canonSelections[selectionKey] || []).length > 0);
- // If the user supplied no canon at all (arc-only import), skip the
- // updateUniverse round-trip entirely.
- //
- // Mutator form (issue #8453): the series admission serializes importer
- // commits into ONE series, but the universe is shared across series and is
- // edited outside the importer too, so the up-front `universe` snapshot can
- // already be stale. Merging against it would silently drop a concurrent
- // canon edit to the SAME array (mergeExtractedBible rebuilds the whole array
- // from its base list). Merging against `latest` — the freshest persisted
- // record, read inside updateUniverse's write queue — closes that race.
const updatedUniverse = selectedKindMap.length > 0
? await updateUniverse(universe.id, (latest) => Object.fromEntries(
selectedKindMap.map(([selectionKey, kind, storageKey]) => [
storageKey,
- mergeExtractedBible(
- latest[storageKey] || [],
- canonSelections[selectionKey],
- kind,
- { source: 'imported' },
- ),
+ mergeExtractedBible(latest[storageKey] || [], canonSelections[selectionKey], kind, { source: 'imported' }),
]),
))
: universe;
+ // Bible top-level fields come from the extracted arc + the count of issues
+ // being created — so a freshly-imported series doesn't show a blank bible.
const sanitizedArc = sanitizeArc(arc);
-
- // Bible top-level fields the Arc Canvas sidebar reads. The importer used to
- // write only series.arc + seasons, leaving logline/premise/issueCountTarget
- // empty — so a freshly-imported series showed a blank bible. Fill them from
- // the extracted arc + the count of issues being created. replaceMode
- // overwrites (user-confirmed wipe); additive only fills fields the series
- // doesn't already have, so a re-import never clobbers a hand-edited bible.
- const issueCount = issuesWithPositions.length;
const arcLogline = sanitizedArc?.logline || '';
const arcPremise = sanitizedArc?.summary || sanitizedArc?.protagonistArc || '';
- const biblePatch = {};
- if (replaceMode) {
- // Replace mode is a user-confirmed full overwrite — write the new arc's
- // bible fields unconditionally, INCLUDING empty strings, so an import whose
- // extracted arc lacks a logline/summary clears the prior work's bible
- // rather than leaving stale text beside the freshly-replaced arc + seasons.
- biblePatch.logline = arcLogline;
- biblePatch.premise = arcPremise;
- biblePatch.issueCountTarget = issueCount;
- } else {
- if (arcLogline && !(series.logline || '').trim()) biblePatch.logline = arcLogline;
- if (arcPremise && !(series.premise || '').trim()) biblePatch.premise = arcPremise;
- // Additive: target the POST-import total (issues already on the series plus
- // the ones being added), not just this batch — adding 3 issues to a series
- // that already has 2 should target 5, not 3. Only when no target is set yet.
- const additiveTotal = priorIssueCount + issueCount;
- if (additiveTotal > 0 && !(series.issueCountTarget > 0)) biblePatch.issueCountTarget = additiveTotal;
- }
- let updatedSeries = series;
- if (replaceMode) {
- // null arc + empty seasons are user-confirmed clears in replace mode
- // (in additive mode the same values mean "preserve").
- updatedSeries = await updateSeries(series.id, {
+ if (phase === IMPORT_PHASE.REPLACE) {
+ // A user-confirmed full overwrite: null arc + empty seasons are clears (in
+ // additive mode the same values mean "preserve"), and the bible fields are
+ // written unconditionally, INCLUDING empty strings, so an import whose arc
+ // lacks a logline/summary clears the prior work's bible rather than leaving
+ // stale text beside the freshly-replaced arc + seasons.
+ const updatedSeries = await updateSeries(series.id, {
arc: sanitizedArc,
seasons: sanitizeSeasonList(seasons),
- ...biblePatch,
- });
- } else if (sanitizedArc || seasons.length > 0 || Object.keys(biblePatch).length > 0) {
- // Mutator form (issue #8453): same race as the universe write above — a
- // tab or job outside the importer may add or edit a season on this series
- // between the up-front read and this write.
- // `mergeSeasons` rebuilds the whole array from its base list, so merging
- // against the stale `series` snapshot captured before that window would
- // silently drop the concurrent edit. Recompute the seasons merge and the
- // empty-field fill-ins against `latest` — the freshest persisted record,
- // read inside updateSeries's write queue.
- updatedSeries = await updateSeries(series.id, (latest) => {
- // Only build + persist a seasons array when the caller actually sent
- // some — otherwise an arc-only re-import on a series that already has
- // seasons rewrites the array byte-for-byte identical (just to bump
- // sanitizeSeasonList's normalization), a wasted disk write.
- const seasonsPatch = seasons.length > 0
- ? { seasons: sanitizeSeasonList(mergeSeasons(
- Array.isArray(latest.seasons) ? latest.seasons : [],
- seasons,
- )) }
- : {};
- const latestBiblePatch = {};
- if (arcLogline && !(latest.logline || '').trim()) latestBiblePatch.logline = arcLogline;
- if (arcPremise && !(latest.premise || '').trim()) latestBiblePatch.premise = arcPremise;
- // Additive: target the POST-import total (issues already on the series
- // plus the ones being added), not just this batch — adding 3 issues to
- // a series that already has 2 should target 5, not 3. `priorIssueCount`
- // is the up-front count; the issue-create loop below still runs
- // sequentially against `latest`'s state, so this total stays accurate.
- const additiveTotal = priorIssueCount + issueCount;
- if (additiveTotal > 0 && !(latest.issueCountTarget > 0)) latestBiblePatch.issueCountTarget = additiveTotal;
- return {
- ...(sanitizedArc ? { arc: sanitizedArc } : {}),
- ...seasonsPatch,
- ...latestBiblePatch,
- };
+ logline: arcLogline,
+ premise: arcPremise,
+ issueCountTarget: issueCount,
});
+ return { universe: updatedUniverse, series: updatedSeries };
+ }
+
+ // Returning null skips the write when there is nothing to change. Seasons are
+ // only rebuilt when the caller sent some — otherwise an arc-only re-import
+ // rewrites the array byte-for-byte identical, a wasted disk write.
+ const updatedSeries = await updateSeries(series.id, (latest) => {
+ const patch = {
+ ...(sanitizedArc ? { arc: sanitizedArc } : {}),
+ ...(seasons.length > 0
+ ? { seasons: sanitizeSeasonList(mergeSeasons(Array.isArray(latest.seasons) ? latest.seasons : [], seasons)) }
+ : {}),
+ ...fillEmptySeriesBible(latest, { arcLogline, arcPremise, priorIssueCount, issueCount }),
+ };
+ return Object.keys(patch).length > 0 ? patch : null;
+ });
+ return { universe: updatedUniverse, series: updatedSeries };
+}
+
+// The stage seeds bundled into an issue's create payload, so the serialized
+// write tail handles one write per issue instead of create +
+// updateStage(prose) + updateStage(idea).
+function buildIssueSeedStages(proposal, contentType) {
+ const stages = {};
+ if (proposal.proseExcerpt) {
+ // Route the verbatim excerpt to the stage that matches the source form
+ // (see CONTENT_TYPE_SEED_STAGE): a script-form type seeds its script stage
+ // ready (comic-script → comicScript, screenplay → teleplay) so the pipeline
+ // renders the user's verbatim script instead of regenerating it;
+ // prose-like types seed stages.prose.
+ stages[seedStageFor(contentType)] = { status: 'ready', output: proposal.proseExcerpt };
+ }
+ const ideaSeed = [
+ proposal.logline && `Logline: ${proposal.logline}`,
+ proposal.synopsis && `Synopsis: ${proposal.synopsis}`,
+ ].filter(Boolean).join('\n\n');
+ if (ideaSeed) {
+ // `idea` is seeded with input only — the user/LLM still needs to run the
+ // idea stage to produce `output`. `isStageReady` requires both
+ // `status in {'ready','edited'}` AND non-empty output, so marking 'ready'
+ // here would mislabel it to status-only consumers while failing readiness
+ // predicates downstream. 'empty' matches the actual state.
+ stages.idea = { status: 'empty', input: ideaSeed };
}
+ return stages;
+}
- // seasonNumber → season-record map; missing seasonNumber falls through
- // to the LOWEST-NUMBERED season (not array-position [0]), or null when no
- // seasons exist. mergeSeasons returns `[...retained, ...incomingBuilt]`
- // — retained existing seasons come first regardless of number, so an
- // import that adds season 1 to a series already holding [2, 3] would
- // otherwise pick season 2 as "first" and surprise the user.
+/**
+ * Plans every issue's identity, records the plan, creates the issues, and
+ * writes the receipt. Canon, arc and seasons are already on disk.
+ */
+async function createPlannedIssues({
+ importId, payloadHash, admission, issuesToCreate, contentType, universe, series,
+}) {
+ const { resumePlan, resumeLive } = admission;
+ // seasonNumber → season-record map; missing seasonNumber falls through to
+ // the LOWEST-NUMBERED season (not array-position [0]), or null when no
+ // seasons exist. mergeSeasons returns `[...retained, ...incomingBuilt]` —
+ // retained existing seasons come first regardless of number, so an import
+ // that adds season 1 to a series already holding [2, 3] would otherwise
+ // pick season 2 as "first" and surprise the user.
const seasonByNumber = new Map();
- for (const s of (updatedSeries.seasons || [])) {
+ for (const s of (series.seasons || [])) {
if (Number.isFinite(s.number)) seasonByNumber.set(s.number, s);
}
- const sortedSeasons = [...(updatedSeries.seasons || [])]
+ const sortedSeasons = [...(series.seasons || [])]
.filter((s) => Number.isFinite(s.number))
.sort((a, b) => a.number - b.number);
const fallbackSeason = sortedSeasons[0] || null;
@@ -1646,7 +1584,7 @@ async function admitAndCommit({
// minting a second batch, and a planned id that never landed (or was rolled
// back or deleted since) gets a fresh one here, so a stale tombstone is never
// resurrected.
- const liveSeasonIds = new Set((updatedSeries.seasons || []).map((s) => s.id));
+ const liveSeasonIds = new Set((series.seasons || []).map((s) => s.id));
const remappedIssues = resumePlan ? [...resumePlan.remappedIssues] : [];
const planItems = issuesToCreate.map((proposal, i) => {
const prior = resumePlan?.items[i];
@@ -1684,8 +1622,8 @@ async function admitAndCommit({
// would duplicate.
if (importId) {
await recordImportProgress(importId, {
- universeId: updatedUniverse.id,
- seriesId: updatedSeries.id,
+ universeId: universe.id,
+ seriesId: series.id,
status: SESSION_STATUS.ARC_PERSISTED,
plan: { payloadHash, items: planItems, remappedIssues },
});
@@ -1695,55 +1633,30 @@ async function admitAndCommit({
// of an earlier attempt belong to the plan and keep any edits made since.
const createdThisAttempt = [];
- // Thread C fix — issue-loop with rollback on failure. The universe +
- // series are already written above. If an issue write throws mid-loop
- // (e.g. transient FS error) we delete what this attempt created and
- // re-throw, leaving the universe + series in their updated state. The
- // universe + series writes are kept because they represent user-confirmed
- // data. With an importId the plan stays on disk, so the retry recreates
- // only what is missing under fresh planned ids.
+ // Thread C fix — issue-loop with rollback on failure. The universe + series
+ // are already written. If an issue write throws mid-loop (e.g. transient FS
+ // error) we delete what this attempt created and re-throw, leaving the
+ // universe + series in their updated state. The universe + series writes are
+ // kept because they represent user-confirmed data. With an importId the plan
+ // stays on disk, so the retry recreates only what is missing under fresh
+ // planned ids.
try {
for (let i = 0; i < issuesToCreate.length; i++) {
const item = planItems[i];
if (resumeLive.has(item.issueId)) continue;
const proposal = issuesToCreate[i];
- // Bundle stage seeds into the initial create payload so the
- // serialized write tail handles one write per issue instead of
- // create + updateStage(prose) + updateStage(idea).
- const stages = {};
- if (proposal.proseExcerpt) {
- // Route the verbatim excerpt to the stage that matches the source form
- // (see CONTENT_TYPE_SEED_STAGE): a script-form type seeds its script
- // stage ready (comic-script → comicScript, screenplay → teleplay) so the
- // pipeline renders the user's verbatim script instead of regenerating
- // it; prose-like types seed stages.prose.
- stages[seedStageFor(contentType)] = { status: 'ready', output: proposal.proseExcerpt };
- }
- const ideaSeed = [
- proposal.logline && `Logline: ${proposal.logline}`,
- proposal.synopsis && `Synopsis: ${proposal.synopsis}`,
- ].filter(Boolean).join('\n\n');
- if (ideaSeed) {
- // `idea` is seeded with input only — the user/LLM still needs to
- // run the idea stage to produce `output`. `isStageReady` requires
- // both `status in {'ready','edited'}` AND non-empty output, so
- // marking 'ready' here would mislabel it to status-only consumers
- // while failing readiness predicates downstream. 'empty' matches
- // the actual state: input present, generation not yet performed.
- stages.idea = { status: 'empty', input: ideaSeed };
- }
// The series admission keeps other imports out, but not an ordinary
// issue create; ensureIssueWithId re-checks the planned position against
// the live issues inside the series write queue and refuses one taken
- // since the read above (#10763), so the race fails here loudly and rolls
- // back instead of landing a duplicate position.
+ // since the inventory read (#10763), so the race fails here loudly and
+ // rolls back instead of landing a duplicate position.
const { created } = await ensureIssueWithId(item.issueId, {
- seriesId: updatedSeries.id,
+ seriesId: series.id,
title: proposal.title,
seasonId: item.seasonId,
arcPosition: item.arcPosition,
arcRole: proposal.arcRole,
- stages,
+ stages: buildIssueSeedStages(proposal, contentType),
});
if (created) createdThisAttempt.push(item.issueId);
}
@@ -1758,7 +1671,7 @@ async function admitAndCommit({
survivingIssueIds.push(id);
});
}
- const n = issues.length;
+ const n = issuesToCreate.length;
const rolledBack = createdThisAttempt.length - survivingIssueIds.length;
const plural = (k) => `${k} issue${k === 1 ? '' : 's'}`;
const notes = [`${plural(rolledBack)} created by this attempt ${rolledBack === 1 ? 'was' : 'were'} rolled back.`];
@@ -1773,27 +1686,26 @@ async function admitAndCommit({
// canon from the retry payload — otherwise the retry overwrites any edits
// (parallel tab, collaborator) made to the persisted state after the
// failure. `skipArcOnRetry` is the imperative form of the same signal.
- const partial = Object.assign(
+ throw Object.assign(
new Error(
`The universe and series were updated successfully, but creating ${n === 1 ? 'the issue' : `the ${n} issues`} failed. ${notes.join(' ')} Retry to create the remaining issues. (Original error: ${issueErr.message})`,
),
{
code: ERR_PARTIAL_COMMIT_ISSUES,
context: {
- universeId: updatedUniverse.id,
- seriesId: updatedSeries.id,
+ universeId: universe.id,
+ seriesId: series.id,
arcAlreadyPersisted: true,
skipArcOnRetry: true,
survivingIssueIds,
},
},
);
- throw partial;
}
// Done after the issue loop so a rolled-back commit leaves the draft shells
// GC-eligible.
- const promoted = await promoteImportDrafts(updatedUniverse, updatedSeries);
+ const promoted = await promoteImportDrafts(universe, series);
const createdIssueIds = planItems.map((item) => item.issueId);
if (importId) await recordCommittedReceipt(importId, createdIssueIds, remappedIssues);
@@ -1805,3 +1717,110 @@ async function admitAndCommit({
remappedIssues,
};
}
+
+// Validates and plans the commit against fresh state, then writes it. Runs
+// inside the series admission, except for the `precheck` pass (a fail-fast
+// before formatting, which writes nothing): that returns `{ replay: true }`
+// for an import already committed, else `{ pendingProposals }` — the excerpts
+// the formatting pass should clean. `formattedExcerpts[i]` (from that pass)
+// replaces proposal i's excerpt when it is still to be created.
+//
+// Payload fields: `canonSelections`, `arc`, `seasons`, `issues`; `contentType`
+// drives which stage each issue's verbatim excerpt seeds (absent →
+// prose-seed); `replaceMode` is the destructive replace — it wipes existing
+// issues and overwrites arc + seasons (default false).
+async function admitAndCommit(payload = {}, { precheck = false, formattedExcerpts = null } = {}) {
+ const { universeId, seriesId, importId = null, issues = [], contentType = null, replaceMode = false } = payload;
+ // Read for validation (universeId/seriesId linkage, lock checks, the
+ // arcPosition/season-number gates) and for the up-front counts used to size
+ // auto-assigned positions and the additive issue-count target. Not the
+ // source of truth the universe/series writes merge against (#8453) — see
+ // applyImportedCanon.
+ const universe = await getUniverse(universeId);
+ const series = await getSeries(seriesId);
+
+ // Defensive: refuse commit when series.universeId is set AND differs,
+ // OR when series.universeId is missing entirely. A legacy series
+ // persisted before universeId became standard would otherwise silently
+ // be re-homed to whichever universe the caller routed through. createSeries
+ // sets universeId today, so a missing value implies hand-edited / pre-link
+ // legacy data — flag it so the caller decides rather than silently linking.
+ if (series.universeId !== universe.id) {
+ if (series.universeId) {
+ throw codedError(
+ `Series "${series.name}" is linked to a different universe — commit refused to avoid cross-linking.`,
+ ERR_VALIDATION,
+ );
+ }
+ throw codedError(
+ `Series "${series.name}" has no universeId — commit refused. Link the series to a universe explicitly before importing.`,
+ ERR_VALIDATION,
+ );
+ }
+
+ const payloadHash = hashIssuePayload(issues, contentType);
+ const admission = await resolveImportAdmission({ importId, replaceMode, precheck, series, payloadHash });
+ // A replay is answered before the locked-arc gate: nothing new is written,
+ // so a lock added since the commit cannot turn a recorded success into an
+ // error.
+ if (admission.phase === IMPORT_PHASE.REPLAY_PRECHECK) return { replay: true };
+ if (admission.phase === IMPORT_PHASE.REPLAY) return replayCommittedImport(importId, admission, { universe, series });
+
+ if (series.locked?.arc === true) {
+ throw codedError(
+ `Series "${series.name}" has a locked arc — commit refused. Unlock the arc to import.`,
+ ERR_LOCKED,
+ );
+ }
+
+ // The server kept canon/arc/seasons from an attempt whose issues did not
+ // finish. Resending them would overwrite edits made since, so the server
+ // drops them itself rather than trusting the client to have remembered.
+ const { canonSelections = {}, arc = null, seasons = [] } = admission.arcPersisted ? {} : payload;
+
+ const explicitPositions = rejectInvalidIssueProposals(issues);
+ const existingIssues = await listIssues({ seriesId: series.id });
+ const issuesWithPositions = assignIssuePositions(admission, issues, existingIssues, explicitPositions);
+ rejectInvalidSeasons(seasons);
+
+ // Optional AI cleanup of the seeded excerpts (issue #1335) runs between the
+ // precheck pass and admission (see commitImportOnce): AFTER all the cheap
+ // fail-fast validation above (so a bad payload never burns LLM calls) but
+ // BEFORE any destructive write. Best-effort — `reformatSeededExcerpts` falls
+ // back to the verbatim excerpt per issue, so import never breaks.
+ // A resume only reformats the issues it still has to create — the excerpt is
+ // dropped from a surviving one so it costs no LLM call (it is never re-written).
+ const { resumePlan, resumeLive } = admission;
+ const pendingProposals = resumePlan
+ ? issuesWithPositions.map((proposal, i) => (resumeLive.has(resumePlan.items[i].issueId)
+ ? { ...proposal, proseExcerpt: undefined }
+ : proposal))
+ : issuesWithPositions;
+ if (precheck) return { pendingProposals };
+ // An issue that became pending only since the precheck (a survivor deleted in
+ // between) has no cleaned excerpt and keeps its verbatim one.
+ const issuesToCreate = formattedExcerpts
+ ? pendingProposals.map((proposal, i) => (proposal.proseExcerpt && isStr(formattedExcerpts[i])
+ ? { ...proposal, proseExcerpt: formattedExcerpts[i] }
+ : proposal))
+ : pendingProposals;
+
+ if (admission.phase === IMPORT_PHASE.REPLACE) await deleteIssuesForReplace(existingIssues);
+
+ const records = await applyImportedCanon({
+ phase: admission.phase,
+ universe,
+ series,
+ canonSelections,
+ arc,
+ seasons,
+ // Issues already on the series that this import does not own — the base
+ // for the additive issue-count target.
+ priorIssueCount: existingIssues.filter((ex) => !resumeLive.has(ex.id)).length,
+ issueCount: issuesWithPositions.length,
+ });
+
+ return createPlannedIssues({
+ importId, payloadHash, admission, issuesToCreate, contentType, ...records,
+ });
+}