Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

Storyboards now sync between your paired machines (#12902)

Adam Eivyc07ed047dd27
diff --git a/client/src/pages/DataManager.jsx b/client/src/pages/DataManager.jsx
index 50268f53fa..e16b0bdb4b 100644
--- a/client/src/pages/DataManager.jsx
+++ b/client/src/pages/DataManager.jsx
@@ -42,6 +42,7 @@ const TOMBSTONE_KIND_PLURAL = {
commissionFeedback: 'commission feedback',
creativeCommission: 'commissions',
deck: 'decks',
+ storyboard: 'storyboards',
};
function kindList(kinds) {
diff --git a/client/src/pages/Instances.jsx b/client/src/pages/Instances.jsx
index ab0a5865ad..b84af4a9d3 100644
--- a/client/src/pages/Instances.jsx
+++ b/client/src/pages/Instances.jsx
@@ -584,13 +584,14 @@ const SYNC_CATEGORY_META = [
{ key: 'creativeCommissions', label: 'Commissions', icon: Target, description: 'Creative Commission briefs (name + intent + genre + generation settings) — the schedule + run history stay machine-local (PostgreSQL)' },
{ key: 'commissionFeedback', label: 'Commission Feedback', icon: Target, description: 'Creative Commission taste reactions that steer the next run — the schedule stays machine-local (PostgreSQL)' },
{ key: 'decks', label: 'Decks', icon: Spade, description: 'Card decks: style guide, samples, per-card prompts + rendered card art — the image/LLM pins stay machine-local (PostgreSQL)' },
+ { key: 'storyboards', label: 'Storyboards', icon: Clapperboard, description: 'Storyboards: beats, panels, panel-maker recipes and gallery-image art — sketch drawings and in-flight renders stay machine-local (PostgreSQL)' },
{ key: 'catalog', label: 'Catalog', icon: Library, description: 'Creative ingredients catalog: orphan ingredients + ref links (PostgreSQL)' }
];
// Snapshot categories — exclude the per-record / delta-based categories that
// have no 60s snapshot checksum: brain + memory (delta), catalog, and creative
// record kinds handled by the per-record peer-push pipeline.
-const NON_SNAPSHOT_KEYS = new Set(['brain', 'memory', 'catalog', 'authors', 'artists', 'albums', 'tracks', 'creativeDirectorProjects', 'moodBoards', 'fableLoom', 'writersRoomWorks', 'writersRoomFolders', 'writersRoomExercises', 'musicVideoProjects', 'commissionFeedback', 'creativeCommissions', 'decks']);
+const NON_SNAPSHOT_KEYS = new Set(['brain', 'memory', 'catalog', 'authors', 'artists', 'albums', 'tracks', 'creativeDirectorProjects', 'moodBoards', 'fableLoom', 'writersRoomWorks', 'writersRoomFolders', 'writersRoomExercises', 'musicVideoProjects', 'commissionFeedback', 'creativeCommissions', 'decks', 'storyboards']);
const SNAPSHOT_CATEGORIES = SYNC_CATEGORY_META.filter(m => !NON_SNAPSHOT_KEYS.has(m.key));
// Video-history rows have no toggle of their own: they are the metadata half of
diff --git a/docs/STORAGE.md b/docs/STORAGE.md
index 79ebe67067..cdcfc0c6c7 100644
--- a/docs/STORAGE.md
+++ b/docs/STORAGE.md
@@ -148,7 +148,7 @@ does not recover nonempty output or general failed saves; those still require th
- `creative_commissions` — Creative Commissions (Autonomous Creation Engine, #2657/#2686): standing recurring creative briefs that fire on a cron cadence and drive the Creative Director directive pipeline unattended. One row per commission, the full sanitized record in `data` JSONB with `name`/`enabled`/`created_at`/`updated_at` mirrored into columns for the scheduler's "arm every enabled commission" query. The brief/identity federates as `creativeCommission` so synced feedback can attach to the same commission; `schedule`, `runs`, `assignment`, `enabled`, and feedback view stay machine-local (the per-reaction `commissionFeedback` records federate separately). The opt-in Digital Twin music-taste configuration is bounded brief metadata; raw taste sources and per-run recipes never cross the wire. The file backend is the `NODE_ENV=test`/`MEMORY_BACKEND=file` escape hatch only. Adapter: `server/services/creativeCommissions/db.js`, selected pg-vs-file by `server/services/creativeCommissions/store.js` (file backend is the `NODE_ENV=test`/`MEMORY_BACKEND=file` escape hatch only).
- `games` — Game studio workspaces (#3177): one row per managed-app asset plan, with the full reusable sprite/music binding set, current compiled-manifest pointer, compile history, and user-requested AI feedback history in `data` JSONB; `app_id`/`name`/`updated_at` are mirrored for list and relationship queries. The record is **machine-local** because managed-app registration, sprite atlases, and music-library bytes are machine-local; there is no peer-sync cursor or tombstone, and deletes are hard deletes. Compiled manifests are immutable, SHA-256-addressed artifacts under `data/games/{id}/manifests/`; their pointers and hashes live in the DB record. Adapter: `server/services/games/db.js`, selected pg-vs-collectionStore by `server/services/games/store.js` (collectionStore is test/unsupported file escape hatch only).
- `fableloom_stories` — FableLoom branching narratives: one row per loom (a branching-narrative story), with episodes, scene-node graphs, and intent transitions in `data` JSONB; `name`/`universe_id`/`series_id`/`updated_at` are mirrored for list and relationship queries (both refs are soft — no FK). `db-primary` because looms relate to universes and series and the index queries by recency. Federates through the opt-in per-record `fableLoom` category: whole-record LWW merges carry soft-delete tombstones, conflict-journal recovery, and hashed manifests for scene images/videos; there is no snapshot cursor or `sync_sequence`. Adapter: `server/services/fableLoom/db.js`, selected pg-vs-collectionStore by `server/services/fableLoom/store.js` (collectionStore is the test/unsupported file escape hatch only). Feature doc: [FableLoom](./features/fableloom.md).
-- `storyboards` — shared Storyboards (Create › Storyboards): one row per board, with beats → panels → art takes in `data` JSONB (a board is read and written whole, so no child tables); `name`/`universe_id`/`updated_at` are mirrored for the list sort and universe filter. `db-primary` because productions (music videos, Creative Director videos, FableLoom, Series Pipeline) link boards by id and later phases query those links. Panels reference catalog records by id only; panel art bytes stay in the gallery (`data/images`) and `data/media-sketches`, so the row holds filenames and sketch keys, never image bytes. Not federated yet: the tombstone pair and `updatedAt` keep a later sync category additive. No seed. Adapter: `server/services/storyboards/db.js`, selected pg-vs-collectionStore by `server/services/storyboards/store.js` (collectionStore is the test/unsupported file escape hatch only). Migration 435 is a ledger stub; boot DDL creates the table.
+- `storyboards` — shared Storyboards (Create › Storyboards): one row per board, with beats → panels → art takes in `data` JSONB (a board is read and written whole, so no child tables); `name`/`universe_id`/`updated_at` are mirrored for the list sort and universe filter. `db-primary` because productions (music videos, Creative Director videos, FableLoom, Series Pipeline) link boards by id and later phases query those links. Panels reference catalog records by id only; panel art bytes stay in the gallery (`data/images`) and `data/media-sketches`, so the row holds filenames and sketch keys, never image bytes. Federated as record kind `storyboard` / sync category `storyboards` (schema version-gated): one whole-board LWW unit on `updatedAt`, deletes travel as the tombstone pair, gallery-image takes ride the push asset manifest. Machine-local limits: a `sketch:` take syncs as its key only (the `data/media-sketches` sidecars do not travel, so a peer shows no drawing for it), and in-flight panel-maker runs ship as idle. No seed. Adapter: `server/services/storyboards/db.js`, selected pg-vs-collectionStore by `server/services/storyboards/store.js` (collectionStore is the test/unsupported file escape hatch only). Migration 435 is a ledger stub; boot DDL creates the table.
- `threejs_models` — generated procedural 3D-model workspaces: one row per model with gallery-image lineage, provider/model attribution, generation/refinement status, and the validated declarative scene spec in `data` JSONB. The referenced image bytes remain under `data/images/`; deterministic Three.js source is derived from the stored spec rather than persisted as a second mutable artifact. The table is local-only in this first slice, with soft-delete columns retained so federation can be added without a record-shape migration.
- `code_animation_jobs` — Code Animation generation history (`db-primary` record with a managed HTML output asset): job status/title and the generation brief/provider details are stored in PostgreSQL so the gallery can list and reopen jobs after navigation or restart. The generated HTML is kept as a managed file at `data/code-animations/<id>.html`, with its derived gallery poster beside it at `data/code-animations/<id>.jpg` (drawn from the page in the jailed preview browser when the job completes, or on first gallery view for older jobs; a missing poster is simply redrawn, so it needs no row of its own); PostgreSQL stores only its job record, not the output bytes. Jobs are machine-local and never federated. The database dump covers records and the normal filesystem backup includes the HTML assets.
- `code_animation_projects`, `code_animation_project_revisions`, `code_animation_project_runs` — opt-in Code Animation Production settings, accepted/candidate source pointers, immutable source/package hashes, portable manifest metadata and bounded import history. PostgreSQL owns metadata; source/asset bytes are write-once files under `data/code-animations/projects/<project-id>/revisions/<revision-id>/`. Settings, reference IDs, execution snapshots, managed paths and run IDs are machine-local and never federate. Budgets independently record iterations, elapsed time, tokens, render time and total retained disk bytes; imports consume disk only and never execute or install dependencies. Every attempted import has its own UUID-owned destination; failed/interrupted staging stays attributable to its run and never replaces accepted source. Boot DDL and migration 420 install empty tables without a seed. The PostgreSQL dump covers all three tables, and normal filesystem backup includes the entire project directory (no new excludes). Brief handoff omits local settings/IDs/paths; revision packages export only the validated portable contract. Source acceptance is explicit and does not assert visual, motion, sound or render evidence. Contained production workers stage each run in its own `data/code-animation-workspaces/<uuid>/` (excluded from backups, removed when the run ends); their operator-chosen tool paths are the machine-local `codeAnimationExecution` slice of `data/settings.json`.
diff --git a/scripts/gc-tombstones-now.js b/scripts/gc-tombstones-now.js
index b55a5ef45f..9a4167c6de 100644
--- a/scripts/gc-tombstones-now.js
+++ b/scripts/gc-tombstones-now.js
@@ -51,6 +51,7 @@ const KIND_PLURAL = {
commissionFeedback: 'commission feedback',
creativeCommission: 'commissions',
deck: 'decks',
+ storyboard: 'storyboards',
};
const KIND_SINGULAR = { universe: 'universe', series: 'series', issue: 'issue', mediaCollection: 'media collection' };
const labelKinds = (kinds) => [...new Set(kinds.map((k) => KIND_PLURAL[k] ?? k))].join(', ');
diff --git a/server/lib/conflictJournal.js b/server/lib/conflictJournal.js
index a54214606e..2cf4188965 100644
--- a/server/lib/conflictJournal.js
+++ b/server/lib/conflictJournal.js
@@ -571,6 +571,9 @@ export const RESTORABLE_FIELDS = Object.freeze({
// `restoreDeck`, not offered as an individually-overlayable field), and the
// install-capability pins never reach a snapshot (stripped from the wire).
deck: ['name', 'description', 'styleNotes', 'influences', 'layoutPrompt', 'samples', 'cardSize', 'universeId'],
+ // A storyboard restores its authored plan: identity fields, the panel-maker
+ // recipe and the beats → panels → takes tree. Id and timestamps stay structural.
+ storyboard: ['name', 'logline', 'aspectRatio', 'artStyle', 'universeId', 'moodBoardId', 'recipe', 'beats'],
// FableLoom restores the authored story graph, playback configuration, and
// story-level render preferences.
// Universe/series ids are structural links and stay on the live record.
diff --git a/server/lib/db/schema/media.js b/server/lib/db/schema/media.js
index ac8dcb0c1a..750b255952 100644
--- a/server/lib/db/schema/media.js
+++ b/server/lib/db/schema/media.js
@@ -127,7 +127,7 @@ export const mediaDdl = [
// Storyboards (Create › Storyboards): one row per board, beats → panels →
// art takes inline in `data` JSONB (a board is read and written whole).
// name/universe_id/updated_at mirror for the list sort and universe filter;
- // the tombstone pair keeps a later peer-sync category additive. Panel art
+ // the tombstone pair carries peer-sync deletes (record kind `storyboard`). Panel art
// bytes stay in data/images and data/media-sketches. Mirrors init-db.sql.
`CREATE TABLE IF NOT EXISTS storyboards (
id TEXT PRIMARY KEY,
diff --git a/server/lib/peerSyncValidation.js b/server/lib/peerSyncValidation.js
index 671cc156bd..14645461c3 100644
--- a/server/lib/peerSyncValidation.js
+++ b/server/lib/peerSyncValidation.js
@@ -16,7 +16,7 @@ import { TASK_STATUS_VALUES, TASK_PRIORITY_VALUES } from './taskParser.js';
// subscriptions target another PortOS instance over Tailnet.
export const peerSubscribeSchema = z.object({
peerId: z.string().trim().min(1).max(120),
- recordKind: z.enum(['universe', 'series', 'mediaCollection', 'author', 'artist', 'album', 'track', 'creativeDirectorProject', 'moodBoard', 'fableLoom', 'writersRoomWork', 'writersRoomFolder', 'writersRoomExercise', 'musicVideoProject', 'commissionFeedback', 'creativeCommission', 'deck']),
+ recordKind: z.enum(['universe', 'series', 'mediaCollection', 'author', 'artist', 'album', 'track', 'creativeDirectorProject', 'moodBoard', 'fableLoom', 'writersRoomWork', 'writersRoomFolder', 'writersRoomExercise', 'musicVideoProject', 'commissionFeedback', 'creativeCommission', 'deck', 'storyboard']),
recordId: z.string().trim().min(1).max(120),
}).strict();
@@ -279,6 +279,12 @@ const deckPushSchema = z.object({
kind: z.literal('deck'),
...peerSyncPushBase,
}).strict();
+// Storyboards push the whole board (beats → panels → takes inline); panel art
+// rides the base `assetManifest` as gallery images.
+const storyboardPushSchema = z.object({
+ kind: z.literal('storyboard'),
+ ...peerSyncPushBase,
+}).strict();
const commissionFeedbackPushSchema = z.object({
kind: z.literal('commissionFeedback'),
...peerSyncPushBase,
@@ -306,13 +312,14 @@ export const peerSyncPushSchema = z.discriminatedUnion('kind', [
commissionFeedbackPushSchema,
creativeCommissionPushSchema,
deckPushSchema,
+ storyboardPushSchema,
]);
// Manual sync action schemas — used by POST /sync-record, /sync-now, /pull-metadata.
export const peerSyncRecordSchema = z.object({
peerId: z.string().trim().min(1).max(120),
- recordKind: z.enum(['universe', 'series', 'mediaCollection', 'author', 'artist', 'album', 'track', 'creativeDirectorProject', 'moodBoard', 'fableLoom', 'writersRoomWork', 'writersRoomFolder', 'writersRoomExercise', 'musicVideoProject', 'commissionFeedback', 'creativeCommission', 'deck']),
+ recordKind: z.enum(['universe', 'series', 'mediaCollection', 'author', 'artist', 'album', 'track', 'creativeDirectorProject', 'moodBoard', 'fableLoom', 'writersRoomWork', 'writersRoomFolder', 'writersRoomExercise', 'musicVideoProject', 'commissionFeedback', 'creativeCommission', 'deck', 'storyboard']),
recordId: z.string().trim().min(1).max(200),
}).strict();
diff --git a/server/lib/schemaVersions.js b/server/lib/schemaVersions.js
index 5b49d4fb2f..53e902f4bf 100644
--- a/server/lib/schemaVersions.js
+++ b/server/lib/schemaVersions.js
@@ -250,6 +250,15 @@ export const PORTOS_SCHEMA_VERSIONS = Object.freeze({
// silently strip a user's face-orientation choice, so only the decks
// category pauses until that peer upgrades.
decks: 2,
+ // v1 = storyboards (PostgreSQL `storyboards`) federated via the per-record
+ // peer-sync push pipeline (record kind `storyboard`, sync category
+ // `storyboards`). A brand-NEW synced record type, so it gets its own
+ // per-category gate: a v1 sender pushing to a ≤v0 (pre-feature) receiver is
+ // sender-ahead on `storyboards` and gets a 409 — only this category pauses.
+ // The FIRST incompatible board/beat/panel/take shape change MUST bump this to
+ // 2 (a v1 peer would round-trip the new shape through an unaware sanitizer
+ // and LWW the strip back onto the newer peer).
+ storyboards: 1,
// v1 = author personas (PostgreSQL `authors` table) federated via the
// per-record peer-sync push pipeline (record kind `author`, sync category
// `authors`). A brand-NEW synced record type like `storyBuilder` below, so it
@@ -1033,6 +1042,7 @@ export const RECORD_KIND_SCHEMA_CATEGORIES = Object.freeze({
commissionFeedback: Object.freeze(['commissionFeedback']),
creativeCommission: Object.freeze(['creativeCommissions']),
deck: Object.freeze(['decks']),
+ storyboard: Object.freeze(['storyboards']),
});
/**
diff --git a/server/lib/syncWire.js b/server/lib/syncWire.js
index 3582448ce0..c30fb15b8c 100644
--- a/server/lib/syncWire.js
+++ b/server/lib/syncWire.js
@@ -408,6 +408,25 @@ export function sanitizeRecordForWire(kind, record) {
.sort((a, b) => a.key.localeCompare(b.key));
return { ...rest, cards: wireCards, ...soft };
}
+ case 'storyboard': {
+ // A storyboard federates as ONE whole-record LWW unit. The only wire-local
+ // field is in-flight panel-maker state: a queued/running node carries a
+ // media jobId that means nothing on another machine and would read as a
+ // run that never finishes there, so it ships as idle with no job ids.
+ // Everything authored (recipe, beats, panels, takes — a `sketch:` take as
+ // its key only) travels as written.
+ const { deleted: _d, deletedAt: _da, beats, ...rest } = record;
+ const settled = (state) => (state?.status === 'running' || state?.status === 'queued'
+ ? { ...state, status: 'idle', jobIds: [], failedJobIds: [] }
+ : state);
+ const wireBeats = (Array.isArray(beats) ? beats : []).map((beat) => ({
+ ...beat,
+ panels: (Array.isArray(beat?.panels) ? beat.panels : []).map((panel) => (panel?.make?.nodes
+ ? { ...panel, make: { ...panel.make, nodes: Object.fromEntries(Object.entries(panel.make.nodes).map(([nodeId, state]) => [nodeId, settled(state)])) } }
+ : panel)),
+ }));
+ return { ...rest, beats: wireBeats, ...sanitizeSoftDeleteFields(record) };
+ }
case 'musicVideoProject': {
// Music Video projects are whole-record LWW, but their image render pin
// and video backend are install-capability choices: a peer may not have
diff --git a/server/services/conflictJournalResolver.js b/server/services/conflictJournalResolver.js
index 64cac009d5..273cd3f315 100644
--- a/server/services/conflictJournalResolver.js
+++ b/server/services/conflictJournalResolver.js
@@ -42,6 +42,7 @@ import { updateWork, restoreFolder, restoreExercise } from './writersRoom/local.
import { restoreCommissionFeedback } from './creativeCommissions/feedbackStore.js';
import { restoreCommission } from './creativeCommissions/store.js';
import { restoreDeck } from './decks.js';
+import { restoreStoryboard } from './storyboards/index.js';
import { restoreLoom } from './fableLoom/index.js';
import { codedError } from '../lib/codedError.js';
@@ -198,6 +199,11 @@ async function applyToRecord(kind, recordId, patch, { replace = false } = {}) {
// null for a deck already hard-pruned by the tombstone sweep (→ ERR_TARGET_GONE).
const restored = await restoreDeck(recordId, patch).catch(translateGone);
if (!restored) throw codedError(`The ${kind} this conflict targets no longer exists — discard the entry.`, ERR_TARGET_GONE);
+ } else if (kind === 'storyboard') {
+ // restoreStoryboard overlays the authored fields, un-tombstones, and bumps
+ // updatedAt so the restore wins the next LWW. Null = already hard-pruned.
+ const restored = await restoreStoryboard(recordId, patch).catch(translateGone);
+ if (!restored) throw makeErr(`The ${kind} this conflict targets no longer exists — discard the entry.`, ERR_TARGET_GONE);
} else if (kind === 'fableLoom') {
// Restore the authored story graph and playback settings through the normal
// loom mutation path so the edit is sanitized, timestamped, and re-pushed.
diff --git a/server/services/instances.js b/server/services/instances.js
index 11dc6f1913..fa3163b173 100644
--- a/server/services/instances.js
+++ b/server/services/instances.js
@@ -332,6 +332,7 @@ const DEFAULT_SYNC_CATEGORIES = {
commissionFeedback: false,
creativeCommissions: false,
decks: false,
+ storyboards: false,
catalog: false
};
@@ -463,6 +464,7 @@ const PER_RECORD_CATEGORY_KINDS = Object.freeze([
['commissionFeedback', 'commissionFeedback'],
['creativeCommissions', 'creativeCommission'],
['decks', 'deck'],
+ ['storyboards', 'storyboard'],
]);
export async function addPeer({ address, port = DEFAULT_PEER_PORT, name, host, auth, transport, protocol = 'http' }) {
diff --git a/server/services/sharing/peerSync.js b/server/services/sharing/peerSync.js
index a7e5c3c3af..3f955be828 100644
--- a/server/services/sharing/peerSync.js
+++ b/server/services/sharing/peerSync.js
@@ -45,6 +45,7 @@ import { listLooms } from '../fableLoom/index.js';
import { listWorksForSync, listFoldersForSync, listExercisesForSync } from '../writersRoom/sync.js';
import { listCommissionFeedbackForSync } from '../creativeCommissions/feedbackStore.js';
import { listDecksForSync } from '../decks.js';
+import { listStoryboardsForSync } from '../storyboards/index.js';
import { listCommissionsForSync } from '../creativeCommissions/store.js';
import { initCursor } from './peerTombstoneCursors.js';
import {
@@ -232,6 +233,8 @@ const RECORD_KIND_LISTERS = {
// clock the wire LWW does, so a deck edited since its last confirmed push
// reads as stale rather than fully mirrored.
deck: () => listDecksForSync(),
+ // Live boards as { id, updatedAt } — coverage keys on the wire LWW clock.
+ storyboard: () => listStoryboardsForSync(),
};
/**
diff --git a/server/services/sharing/peerSync.test.js b/server/services/sharing/peerSync.test.js
index 2254f7c420..d63027bd30 100644
--- a/server/services/sharing/peerSync.test.js
+++ b/server/services/sharing/peerSync.test.js
@@ -203,6 +203,13 @@ vi.mock('../creativeCommissions/store.js', async (importOriginal) => ({
getCommissionForSync: vi.fn(),
mergeCommissionsFromSync: vi.fn().mockResolvedValue({ applied: true, count: 1 }),
}));
+vi.mock('../storyboards/index.js', () => ({
+ listStoryboardsForSync: vi.fn().mockResolvedValue([]),
+ listStoryboardIdsForSync: vi.fn().mockResolvedValue([]),
+ getStoryboardForSync: vi.fn(),
+ mergeStoryboardsFromSync: vi.fn().mockResolvedValue({ applied: true, count: 1 }),
+ pruneTombstonedStoryboards: vi.fn().mockResolvedValue({ pruned: 0 }),
+}));
vi.mock('../decks.js', () => ({
listDecksForSync: vi.fn().mockResolvedValue([]),
listDeckIdsForSync: vi.fn().mockResolvedValue([]),
@@ -315,6 +322,7 @@ import { diffWorkBibleManifest } from '../writersRoom/bibleSync.js';
import { listCommissionFeedbackForSync, getCommissionFeedbackForSync, mergeCommissionFeedbackFromSync } from '../creativeCommissions/feedbackStore.js';
import { listCommissionsForSync, getCommissionForSync, mergeCommissionsFromSync } from '../creativeCommissions/store.js';
import { getDeckForSync, mergeDecksFromSync } from '../decks.js';
+import { getStoryboardForSync, mergeStoryboardsFromSync } from '../storyboards/index.js';
import { peerFetch } from '../../lib/peerHttpClient.js';
import { RESPONSE_TOO_LARGE } from '../../lib/httpClient.js';
import { reconcileMediaAssets } from '../mediaAssetIndex/index.js';
@@ -517,7 +525,7 @@ describe('peerSync', () => {
// caught — this list is canonical and its order can affect iteration
// elsewhere (e.g. syncNow's per-kind backfill). Issues piggyback on series
// subscriptions; direct issue subs are intentionally rejected (Stage 2).
- expect(PEER_SUBSCRIBABLE_KINDS).toEqual(['universe', 'series', 'mediaCollection', 'author', 'artist', 'album', 'track', 'creativeDirectorProject', 'moodBoard', 'fableLoom', 'writersRoomWork', 'writersRoomFolder', 'writersRoomExercise', 'musicVideoProject', 'commissionFeedback', 'creativeCommission', 'deck']);
+ expect(PEER_SUBSCRIBABLE_KINDS).toEqual(['universe', 'series', 'mediaCollection', 'author', 'artist', 'album', 'track', 'creativeDirectorProject', 'moodBoard', 'fableLoom', 'writersRoomWork', 'writersRoomFolder', 'writersRoomExercise', 'musicVideoProject', 'commissionFeedback', 'creativeCommission', 'deck', 'storyboard']);
});
});
@@ -4037,12 +4045,14 @@ describe('peerSync', () => {
expect(result.missingAssets).toEqual(manifest.slice(0, 3));
});
- it('admits track render history and deck images but drops other asset kinds', async () => {
+ it('admits track render history, deck and storyboard images but drops other asset kinds', async () => {
for (const [kind, record, own] of [
['track', { id: 'track-1', audioFilename: 'active.mp3', renders: [{ audioFilename: 'older.mp3' }] },
[{ kind: 'music', filename: 'active.mp3' }, { kind: 'music', filename: 'older.mp3' }]],
['deck', { id: 'deck-1', cards: [{ primaryImageRef: 'card.png', imageRefs: ['render.png'] }], samples: [{ imageRef: 'style.png' }] },
['card.png', 'render.png', 'style.png'].map((filename) => ({ kind: 'image', filename }))],
+ ['storyboard', { id: 'sb-1', beats: [{ panels: [{ takes: [{ image: 'panel.png' }, { sketchKey: 'sketch:abc' }] }] }] },
+ [{ kind: 'image', filename: 'panel.png' }]],
]) {
const result = await applyIncomingPush({ kind, record, sourceInstanceId: 'peer-a',
assetManifest: [...own, { kind: 'audio', filename: 'unrelated.mp3' }] });
@@ -5989,6 +5999,7 @@ describe('media-library federation (#1566)', () => {
commissionFeedback: { get: getCommissionFeedbackForSync, merge: mergeCommissionFeedbackFromSync },
creativeCommission: { get: getCommissionForSync, merge: mergeCommissionsFromSync },
deck: { get: getDeckForSync, merge: mergeDecksFromSync },
+ storyboard: { get: getStoryboardForSync, merge: mergeStoryboardsFromSync },
};
it('every PEER_SUBSCRIBABLE_KINDS entry has a test fixture (guards the parity suite itself against a silently-skipped kind)', () => {
diff --git a/server/services/sharing/peerSyncAssets.js b/server/services/sharing/peerSyncAssets.js
index c06b84088e..74ecb1ad68 100644
--- a/server/services/sharing/peerSyncAssets.js
+++ b/server/services/sharing/peerSyncAssets.js
@@ -757,6 +757,28 @@ export function referenceDeckAssetManifest(deck) {
return entries.filter(Boolean);
}
+/**
+ * Hash every gallery image a storyboard's panel takes reference so a peer pulls
+ * the bytes and sees the board's art. A `sketch:` take has no gallery image
+ * (`image` is null) — its sidecars in data/media-sketches stay machine-local.
+ */
+export async function buildStoryboardAssetManifest(board) {
+ return hashAssetReferences(referenceStoryboardAssetManifest(board));
+}
+
+export function referenceStoryboardAssetManifest(board) {
+ const filenames = new Set();
+ for (const beat of Array.isArray(board?.beats) ? board.beats : []) {
+ for (const panel of Array.isArray(beat?.panels) ? beat.panels : []) {
+ for (const take of Array.isArray(panel?.takes) ? panel.takes : []) {
+ const safe = sanitizeAssetFilename(take?.image);
+ if (safe) filenames.add(safe);
+ }
+ }
+ }
+ return [...filenames].map(imageAssetReference).filter(Boolean);
+}
+
export async function buildAssetManifestForSeries(series, issues, linkedCollection = null) {
return hashAssetReferences(referenceAssetManifestForSeries(series, issues, linkedCollection));
}
diff --git a/server/services/sharing/peerSyncShared.js b/server/services/sharing/peerSyncShared.js
index bed4106c10..b7a19ce89a 100644
--- a/server/services/sharing/peerSyncShared.js
+++ b/server/services/sharing/peerSyncShared.js
@@ -15,7 +15,7 @@ import { EventEmitter } from 'events';
import { PATHS, atomicWrite, readJSONFile, ensureDir } from '../../lib/fileUtils.js';
import { getPeers, resolveEffectiveCategories } from '../instances.js';
-export const PEER_SUBSCRIBABLE_KINDS = Object.freeze(['universe', 'series', 'mediaCollection', 'author', 'artist', 'album', 'track', 'creativeDirectorProject', 'moodBoard', 'fableLoom', 'writersRoomWork', 'writersRoomFolder', 'writersRoomExercise', 'musicVideoProject', 'commissionFeedback', 'creativeCommission', 'deck']);
+export const PEER_SUBSCRIBABLE_KINDS = Object.freeze(['universe', 'series', 'mediaCollection', 'author', 'artist', 'album', 'track', 'creativeDirectorProject', 'moodBoard', 'fableLoom', 'writersRoomWork', 'writersRoomFolder', 'writersRoomExercise', 'musicVideoProject', 'commissionFeedback', 'creativeCommission', 'deck', 'storyboard']);
/**
* The optional top-level keys a NEWER sender may attach to a push envelope
@@ -164,6 +164,7 @@ export const KIND_TO_CATEGORY = Object.freeze({
commissionFeedback: 'commissionFeedback',
creativeCommission: 'creativeCommissions',
deck: 'decks',
+ storyboard: 'storyboards',
});
// The half every outbound predicate shares: the peer exists, the user has not
diff --git a/server/services/sharing/recordKinds.js b/server/services/sharing/recordKinds.js
index 966c255591..548de3c95e 100644
--- a/server/services/sharing/recordKinds.js
+++ b/server/services/sharing/recordKinds.js
@@ -61,6 +61,7 @@ import {
import { getCommissionFeedbackForSync, mergeCommissionFeedbackFromSync } from '../creativeCommissions/feedbackStore.js';
import { getCommissionForSync, mergeCommissionsFromSync } from '../creativeCommissions/store.js';
import { getDeckForSync, mergeDecksFromSync } from '../decks.js';
+import { getStoryboardForSync, mergeStoryboardsFromSync } from '../storyboards/index.js';
import {
buildAssetManifest,
buildAssetManifestForSeries,
@@ -74,6 +75,7 @@ import {
buildBoardAssetManifest,
buildFableLoomAssetManifest,
buildDeckAssetManifest,
+ buildStoryboardAssetManifest,
referenceCollectionAssetManifest,
referenceAuthorAssetManifest,
referenceArtistAssetManifest,
@@ -84,6 +86,7 @@ import {
referenceFableLoomAssetManifest,
referenceBoardAssetManifest,
referenceDeckAssetManifest,
+ referenceStoryboardAssetManifest,
referenceAssetManifestForSeries,
referenceAssetManifestWithCollection,
} from './peerSyncAssets.js';
@@ -225,6 +228,15 @@ export const RECORD_KINDS = Object.freeze({
referencedAssets: referenceDeckAssetManifest,
hasEphemeral: false,
},
+ storyboard: {
+ // The loader surfaces tombstones (a delete must push); the whole board is
+ // one record, so the generic manifest builder reads its takes directly.
+ load: (id) => getStoryboardForSync(id),
+ merge: mergeStoryboardsFromSync,
+ buildAssetManifest: buildStoryboardAssetManifest,
+ referencedAssets: referenceStoryboardAssetManifest,
+ hasEphemeral: false,
+ },
});
// --- Asset-integrity manifest (moved from peerSyncAssets.js, #6843) -------
diff --git a/server/services/sharing/tombstoneGc.js b/server/services/sharing/tombstoneGc.js
index 34bbf60b15..77401cee89 100644
--- a/server/services/sharing/tombstoneGc.js
+++ b/server/services/sharing/tombstoneGc.js
@@ -80,6 +80,7 @@ import {
pruneTombstonedCommissions, listCommissionIdsForSync,
} from '../creativeCommissions/store.js';
import { listDeckIdsForSync, pruneTombstonedDecks } from '../decks.js';
+import { listStoryboardIdsForSync, pruneTombstonedStoryboards } from '../storyboards/index.js';
import { pruneOrphanedBaseHashes } from '../../lib/conflictJournal.js';
import { listPeerSubscriptions, pruneOrphanedPeerSubscriptions } from './peerSync.js';
import { getMinAckAcrossPeers } from './peerTombstoneCursors.js';
@@ -122,6 +123,7 @@ const LIVE_ID_LISTERS = Object.freeze({
commissionFeedback: () => listCommissionFeedbackIdsForSync(),
creativeCommission: () => listCommissionIdsForSync(),
deck: () => listDeckIdsForSync(),
+ storyboard: () => listStoryboardIdsForSync(),
});
// Each peer-subscribable kind's UNCAPPED id source INCLUDING tombstones —
@@ -157,6 +159,7 @@ const ALL_ID_LISTERS = Object.freeze({
commissionFeedback: () => listCommissionFeedbackIdsForSync({ includeDeleted: true }),
creativeCommission: () => listCommissionIdsForSync({ includeDeleted: true }),
deck: () => listDeckIdsForSync({ includeDeleted: true }),
+ storyboard: () => listStoryboardIdsForSync({ includeDeleted: true }),
});
// Build a kind-aware id-membership resolver for ONE sweep: lazily list each
@@ -479,6 +482,7 @@ export async function sweepTombstones({ now = Date.now(), graceMs = GRACE_MS } =
commissionFeedbackCutoff,
creativeCommissionCutoff,
deckCutoff,
+ storyboardCutoff,
] = await Promise.all([
cutoffForKind('universe', { peers, subs, now, graceMs }),
cutoffForKind('series', { peers, subs, now, graceMs }),
@@ -497,10 +501,11 @@ export async function sweepTombstones({ now = Date.now(), graceMs = GRACE_MS } =
cutoffForKind('commissionFeedback', { peers, subs, now, graceMs }),
cutoffForKind('creativeCommission', { peers, subs, now, graceMs }),
cutoffForKind('deck', { peers, subs, now, graceMs }),
+ cutoffForKind('storyboard', { peers, subs, now, graceMs }),
]);
const issueCutoff = seriesCutoff;
const itemRemovalCutoff = collectionItemRemovalCutoff({ mediaCollectionCutoff, universeCutoff, seriesCutoff });
- const [u, s, i, c, cir, a, ar, al, t, cd, mv, mb, fl, wr, wrf, wre, cfb, cc, dk] = await Promise.all([
+ const [u, s, i, c, cir, a, ar, al, t, cd, mv, mb, fl, wr, wrf, wre, cfb, cc, dk, sb] = await Promise.all([
universeCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedUniverses(universeCutoff),
seriesCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedSeries(seriesCutoff),
issueCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedIssues(issueCutoff),
@@ -520,6 +525,7 @@ export async function sweepTombstones({ now = Date.now(), graceMs = GRACE_MS } =
commissionFeedbackCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedCommissionFeedback(commissionFeedbackCutoff),
creativeCommissionCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedCommissions(creativeCommissionCutoff),
deckCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedDecks(deckCutoff),
+ storyboardCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedStoryboards(storyboardCutoff),
]);
// Backstop AFTER the tombstone prunes: the prune paths already evict a freshly
// hard-deleted record's base hash, so this sweep mops up only keys that
@@ -555,9 +561,10 @@ export async function sweepTombstones({ now = Date.now(), graceMs = GRACE_MS } =
commissionFeedback: cfb.pruned,
creativeCommissions: cc.pruned,
decks: dk.pruned,
+ storyboards: sb.pruned,
orphanBaseHashes: orphan.pruned,
orphanSubscriptions: orphanSubs.pruned,
- refused: refusedFromCutoffs({ universeCutoff, seriesCutoff, mediaCollectionCutoff, authorCutoff, artistCutoff, albumCutoff, trackCutoff, creativeDirectorProjectCutoff, musicVideoProjectCutoff, moodBoardCutoff, fableLoomCutoff, writersRoomWorkCutoff, writersRoomFolderCutoff, writersRoomExerciseCutoff, commissionFeedbackCutoff, creativeCommissionCutoff, deckCutoff }),
+ refused: refusedFromCutoffs({ universeCutoff, seriesCutoff, mediaCollectionCutoff, authorCutoff, artistCutoff, albumCutoff, trackCutoff, creativeDirectorProjectCutoff, musicVideoProjectCutoff, moodBoardCutoff, fableLoomCutoff, writersRoomWorkCutoff, writersRoomFolderCutoff, writersRoomExerciseCutoff, commissionFeedbackCutoff, creativeCommissionCutoff, deckCutoff, storyboardCutoff }),
};
}
@@ -568,7 +575,7 @@ export async function sweepTombstones({ now = Date.now(), graceMs = GRACE_MS } =
// snapshot coverage matters), so this hardcodes graceMs:0 internally.
export async function getSweepStatus({ now = Date.now() } = {}) {
const { peers, subs } = await loadState();
- const [universeCutoff, seriesCutoff, mediaCollectionCutoff, authorCutoff, artistCutoff, albumCutoff, trackCutoff, creativeDirectorProjectCutoff, musicVideoProjectCutoff, moodBoardCutoff, fableLoomCutoff, writersRoomWorkCutoff, writersRoomFolderCutoff, writersRoomExerciseCutoff, commissionFeedbackCutoff, creativeCommissionCutoff, deckCutoff] = await Promise.all([
+ const [universeCutoff, seriesCutoff, mediaCollectionCutoff, authorCutoff, artistCutoff, albumCutoff, trackCutoff, creativeDirectorProjectCutoff, musicVideoProjectCutoff, moodBoardCutoff, fableLoomCutoff, writersRoomWorkCutoff, writersRoomFolderCutoff, writersRoomExerciseCutoff, commissionFeedbackCutoff, creativeCommissionCutoff, deckCutoff, storyboardCutoff] = await Promise.all([
cutoffForKind('universe', { peers, subs, now, graceMs: 0 }),
cutoffForKind('series', { peers, subs, now, graceMs: 0 }),
cutoffForKind('mediaCollection', { peers, subs, now, graceMs: 0 }),
@@ -586,9 +593,10 @@ export async function getSweepStatus({ now = Date.now() } = {}) {
cutoffForKind('commissionFeedback', { peers, subs, now, graceMs: 0 }),
cutoffForKind('creativeCommission', { peers, subs, now, graceMs: 0 }),
cutoffForKind('deck', { peers, subs, now, graceMs: 0 }),
+ cutoffForKind('storyboard', { peers, subs, now, graceMs: 0 }),
]);
return {
- refused: refusedFromCutoffs({ universeCutoff, seriesCutoff, mediaCollectionCutoff, authorCutoff, artistCutoff, albumCutoff, trackCutoff, creativeDirectorProjectCutoff, musicVideoProjectCutoff, moodBoardCutoff, fableLoomCutoff, writersRoomWorkCutoff, writersRoomFolderCutoff, writersRoomExerciseCutoff, commissionFeedbackCutoff, creativeCommissionCutoff, deckCutoff }),
+ refused: refusedFromCutoffs({ universeCutoff, seriesCutoff, mediaCollectionCutoff, authorCutoff, artistCutoff, albumCutoff, trackCutoff, creativeDirectorProjectCutoff, musicVideoProjectCutoff, moodBoardCutoff, fableLoomCutoff, writersRoomWorkCutoff, writersRoomFolderCutoff, writersRoomExerciseCutoff, commissionFeedbackCutoff, creativeCommissionCutoff, deckCutoff, storyboardCutoff }),
heldByPausedPeers: pausedPeersHoldingKinds(peers, subs),
};
}
diff --git a/server/services/sharing/tombstoneGc.test.js b/server/services/sharing/tombstoneGc.test.js
index a4e6b1be64..90bcdd948e 100644
--- a/server/services/sharing/tombstoneGc.test.js
+++ b/server/services/sharing/tombstoneGc.test.js
@@ -76,6 +76,10 @@ vi.mock('../creativeCommissions/store.js', () => ({
pruneTombstonedCommissions: vi.fn().mockResolvedValue({ pruned: 0 }),
listCommissionIdsForSync: vi.fn().mockResolvedValue([]),
}));
+vi.mock('../storyboards/index.js', () => ({
+ pruneTombstonedStoryboards: vi.fn().mockResolvedValue({ pruned: 0 }),
+ listStoryboardIdsForSync: vi.fn().mockResolvedValue([]),
+}));
vi.mock('../decks.js', () => ({
pruneTombstonedDecks: vi.fn().mockResolvedValue({ pruned: 0 }),
listDeckIdsForSync: vi.fn().mockResolvedValue([]),
@@ -126,6 +130,7 @@ import { pruneTombstonedWorks, pruneTombstonedFolders, pruneTombstonedExercises
import { pruneTombstonedCommissionFeedback } from '../creativeCommissions/feedbackStore.js';
import { pruneTombstonedCommissions } from '../creativeCommissions/store.js';
import { pruneTombstonedDecks } from '../decks.js';
+import { pruneTombstonedStoryboards } from '../storyboards/index.js';
import { pruneOrphanedBaseHashes } from '../../lib/conflictJournal.js';
import { listPeerSubscriptions, pruneOrphanedPeerSubscriptions } from './peerSync.js';
import { getMinAckAcrossPeers } from './peerTombstoneCursors.js';
@@ -172,7 +177,7 @@ describe('TOMBSTONE_GRACE_MS', () => {
});
describe('sweepTombstones — no peers subscribed', () => {
- it('uses now-GRACE as the cutoff for all 17 subscribable kinds when nobody is subscribed (#6843: every cutoff variable, including the 6 renamed to match their kind id, reaches its prune call)', async () => {
+ it('uses now-GRACE as the cutoff for all 18 subscribable kinds when nobody is subscribed (#6843: every cutoff variable, including the 6 renamed to match their kind id, reaches its prune call)', async () => {
await sweepTombstones({ now: NOW });
const expectedCutoff = NOW - TOMBSTONE_GRACE_MS + 1;
expect(pruneTombstonedUniverses).toHaveBeenCalledWith(expectedCutoff);
@@ -201,6 +206,7 @@ describe('sweepTombstones — no peers subscribed', () => {
expect(pruneTombstonedCommissionFeedback).toHaveBeenCalledWith(expectedCutoff);
expect(pruneTombstonedCommissions).toHaveBeenCalledWith(expectedCutoff);
expect(pruneTombstonedDecks).toHaveBeenCalledWith(expectedCutoff);
+ expect(pruneTombstonedStoryboards).toHaveBeenCalledWith(expectedCutoff);
});
it('refusedFromCutoffs iterates PEER_SUBSCRIBABLE_KINDS (#6843) — refused stays scoped to the 4 kinds whose cutoff can actually be null', async () => {
@@ -243,6 +249,7 @@ describe('sweepTombstones — unreadable peer registry fails closed (#8412)', ()
expect(pruneTombstonedCommissionFeedback).not.toHaveBeenCalled();
expect(pruneTombstonedCommissions).not.toHaveBeenCalled();
expect(pruneTombstonedDecks).not.toHaveBeenCalled();
+ expect(pruneTombstonedStoryboards).not.toHaveBeenCalled();
expect(pruneOrphanedBaseHashes).not.toHaveBeenCalled();
expect(pruneOrphanedPeerSubscriptions).not.toHaveBeenCalled();
});
@@ -733,6 +740,7 @@ describe('sweepTombstones — return shape', () => {
commissionFeedback: 0,
creativeCommissions: 0,
decks: 0,
+ storyboards: 0,
orphanBaseHashes: 0,
orphanSubscriptions: 0,
refused: [],
diff --git a/server/services/storyboards/db.js b/server/services/storyboards/db.js
index a5132e307a..91e7b77878 100644
--- a/server/services/storyboards/db.js
+++ b/server/services/storyboards/db.js
@@ -44,3 +44,28 @@ export async function writeRaw(id, record) {
);
return record;
}
+
+export async function deleteRaw(id) {
+ await query('DELETE FROM storyboards WHERE id = $1', [id]);
+}
+
+/** Every board id (live AND tombstones). */
+export async function listIds() {
+ const { rows } = await query('SELECT id FROM storyboards');
+ return rows.map((row) => row.id);
+}
+
+/** Live board ids only — id-only projection hitting idx_storyboards_live. */
+export async function listLiveIds() {
+ const { rows } = await query('SELECT id FROM storyboards WHERE deleted = FALSE');
+ return rows.map((row) => row.id);
+}
+
+/** Tombstoned board ids deleted before `beforeMs` — the GC candidate scan. */
+export async function listTombstoneIdsBefore(beforeMs) {
+ const { rows } = await query(
+ 'SELECT id FROM storyboards WHERE deleted = TRUE AND deleted_at IS NOT NULL AND deleted_at < $1',
+ [new Date(beforeMs).toISOString()],
+ );
+ return rows.map((row) => row.id);
+}
diff --git a/server/services/storyboards/index.js b/server/services/storyboards/index.js
index cde2322b29..990809a539 100644
--- a/server/services/storyboards/index.js
+++ b/server/services/storyboards/index.js
@@ -6,8 +6,8 @@
* racing a panel add merge against the freshest record and a malformed
* mutation never persists. A tombstoned board reads as missing.
*
- * Not federated yet: the tombstone pair and `updatedAt` stamping keep adding
- * a peer-sync category later additive (same posture as sprite records).
+ * Federated as record kind `storyboard` (sync.js): every write emits a record
+ * event so subscribed peers get the new revision, and a delete is a tombstone.
*/
import { ServerError } from '../../lib/errorHandler.js';
@@ -15,7 +15,11 @@ import { sanitizeStoryboard, sanitizeTake, summarizeStoryboard } from '../../lib
import {
isValidStoryboardId, newStoryboardEntityId, newStoryboardId, STORYBOARD_LIMITS,
} from '../../lib/storyboardVocabulary.js';
+import { autoSubscribeRecordToAllPeers, emitRecordDeleted, emitRecordUpdated } from '../sharing/recordEvents.js';
import { listRaw, queueStoryboardWrite, readRaw, writeRaw } from './store.js';
+import { STORYBOARD_KIND } from './sync.js';
+
+export * from './sync.js';
const notFound = (what = 'Storyboard') => new ServerError(`${what} not found`, { status: 404, code: 'NOT_FOUND' });
const tooMany = (message) => new ServerError(message, { status: 409, code: 'STORYBOARD_LIMIT' });
@@ -56,6 +60,8 @@ export async function createStoryboard(input = {}) {
const now = new Date().toISOString();
const board = sanitizeStoryboard({ ...pickFields(input, BOARD_FIELDS), id: newStoryboardId(), beats: [], createdAt: now, updatedAt: now });
await writeRaw(board.id, board);
+ emitRecordUpdated(STORYBOARD_KIND, board.id);
+ autoSubscribeRecordToAllPeers(STORYBOARD_KIND, board.id).catch(() => {});
console.log(`🎬 Created storyboard ${board.id}`);
return board;
}
@@ -67,7 +73,7 @@ export async function createStoryboard(input = {}) {
*/
export async function mutateStoryboard(id, mutate) {
if (!isValidStoryboardId(id)) throw notFound();
- return queueStoryboardWrite(id, async () => {
+ const outcome = await queueStoryboardWrite(id, async () => {
const current = await readLive(id);
if (!current) throw notFound();
const draft = structuredClone(current);
@@ -76,6 +82,8 @@ export async function mutateStoryboard(id, mutate) {
await writeRaw(id, next);
return { board: next, result };
});
+ (outcome.board.deleted ? emitRecordDeleted : emitRecordUpdated)(STORYBOARD_KIND, id);
+ return outcome;
}
export async function updateStoryboard(id, patch) {
diff --git a/server/services/storyboards/store.js b/server/services/storyboards/store.js
index 80d8924b92..caa16555f2 100644
--- a/server/services/storyboards/store.js
+++ b/server/services/storyboards/store.js
@@ -32,6 +32,14 @@ function makeFileBackend() {
readRaw: (id) => collection.loadOneRaw(id),
listRaw: () => collection.loadAll(),
writeRaw: (id, record) => collection.saveOneNow(id, record),
+ deleteRaw: (id) => collection.deleteOneNow(id),
+ // `deleted` lives inside each record on the file layout, so the live and
+ // tombstone id scans read them all (dev/test only; Postgres projects ids).
+ listLiveIds: async () => (await collection.loadAll()).filter((r) => r && r.deleted !== true).map((r) => r.id),
+ listIds: async () => (await collection.loadAll()).filter((r) => r && typeof r.id === 'string').map((r) => r.id),
+ listTombstoneIdsBefore: async (beforeMs) => (await collection.loadAll())
+ .filter((r) => r?.deleted && typeof r.id === 'string' && Date.parse(r.deletedAt || '') < beforeMs)
+ .map((r) => r.id),
};
}
@@ -40,7 +48,16 @@ const facade = createPgFileFacade({
makePg: () => resolvePgBackend({
requirement: 'Storyboards require PostgreSQL — run `npm run setup:db` (dev/test only: set MEMORY_BACKEND=file for the unsupported file backend)',
loadDb: () => import('./db.js'),
- makePg: (db) => ({ name: 'postgres', readRaw: db.readRaw, listRaw: db.listRaw, writeRaw: db.writeRaw }),
+ makePg: (db) => ({
+ name: 'postgres',
+ readRaw: db.readRaw,
+ listRaw: db.listRaw,
+ writeRaw: db.writeRaw,
+ deleteRaw: db.deleteRaw,
+ listIds: db.listIds,
+ listLiveIds: db.listLiveIds,
+ listTombstoneIdsBefore: db.listTombstoneIdsBefore,
+ }),
}),
});
@@ -58,6 +75,20 @@ export const writeRaw = async (id, record) => {
return (await facade.getBackend()).writeRaw(id, record);
};
+export const deleteRaw = async (id) => {
+ assertId(id);
+ return (await facade.getBackend()).deleteRaw(id);
+};
+
+/** Every board id (live AND tombstones) — the tombstone sweep's id source. */
+export const listIds = async () => (await facade.getBackend()).listIds();
+
+/** Live board ids only — id-only projection, no JSONB bodies. */
+export const listLiveIds = async () => (await facade.getBackend()).listLiveIds();
+
+/** Tombstoned board ids deleted before `beforeMs` — the GC candidate scan. */
+export const listTombstoneIdsBefore = async (beforeMs) => (await facade.getBackend()).listTombstoneIdsBefore(beforeMs);
+
export const queueStoryboardWrite = (id, fn) => queueRecordWrite(id, fn);
export function _resetStoryboardBackend() {
diff --git a/server/services/storyboards/sync.js b/server/services/storyboards/sync.js
new file mode 100644
index 0000000000..ffa8c022fb
--- /dev/null
+++ b/server/services/storyboards/sync.js
@@ -0,0 +1,133 @@
+/**
+ * Storyboard federation — the peer-sync face of the storyboards service.
+ *
+ * A board syncs as ONE whole-record LWW unit on `updatedAt` (record kind and
+ * category `storyboard`/`storyboards`); a delete travels as the tombstone pair
+ * and the shared GC sweep hard-removes it once every peer has had the grace
+ * window. Panel art takes ride the push asset manifest as gallery images
+ * (`buildStoryboardAssetManifest`); a `sketch:` take syncs as its reference
+ * only — the stroke/PNG sidecars in data/media-sketches are machine-local.
+ *
+ * Imports the store directly (never ./index.js) so index.js can re-export this
+ * module without a cycle.
+ */
+
+import { compareNewerWins } from '../../lib/lwwTimestamp.js';
+import {
+ contentHashForRecord,
+ deleteSyncBaseHash,
+ flushBaseHashes,
+ maybeJournalBeforeOverwrite,
+ setSyncBaseHash,
+ withBaseHashFlushBatch,
+} from '../../lib/conflictJournal.js';
+import { sanitizeStoryboard } from '../../lib/storyboardRecord.js';
+import { isValidStoryboardId } from '../../lib/storyboardVocabulary.js';
+import { emitRecordUpdated } from '../sharing/recordEvents.js';
+import {
+ deleteRaw, listIds, listLiveIds, listRaw, listTombstoneIdsBefore, queueStoryboardWrite, readRaw, writeRaw,
+} from './store.js';
+
+export const STORYBOARD_KIND = 'storyboard';
+
+// The authored fields a conflict-journal restore overlays (mirrored in
+// RESTORABLE_FIELDS.storyboard); identity and timestamps stay structural.
+const RESTORABLE_FIELDS = ['name', 'logline', 'aspectRatio', 'artStyle', 'universeId', 'moodBoardId', 'recipe', 'beats'];
+
+/** One board's sanitized record with its tombstone surfaced, or null when unknown. */
+export async function getStoryboardForSync(id) {
+ if (!isValidStoryboardId(id)) return null;
+ return sanitizeStoryboard(await readRaw(id));
+}
+
+/** Every LIVE board as `{ id, updatedAt }` — the full-sync coverage compare set. */
+export async function listStoryboardsForSync() {
+ return (await listRaw())
+ .map(sanitizeStoryboard)
+ .filter((board) => board && !board.deleted)
+ .map(({ id, updatedAt }) => ({ id, updatedAt }));
+}
+
+/** Board ids — live only by default, or all (incl. tombstones) for the sweep. */
+export async function listStoryboardIdsForSync({ includeDeleted = false } = {}) {
+ return includeDeleted ? listIds() : listLiveIds();
+}
+
+/**
+ * Merge boards received from a peer: sanitize, then whole-record LWW on
+ * `updatedAt`, serialized per id against the REST writers. Journals the local
+ * version a remote win is about to overwrite and re-stamps the conflict-journal
+ * base hash for the record it wrote.
+ */
+export async function mergeStoryboardsFromSync(remoteRecords, { source = { via: 'sync', peerId: null } } = {}) {
+ if (!Array.isArray(remoteRecords)) return { applied: false, count: 0 };
+ const byId = new Map();
+ for (const raw of remoteRecords) {
+ const remote = sanitizeStoryboard(raw);
+ if (remote && !byId.has(remote.id)) byId.set(remote.id, remote);
+ }
+ let changed = 0;
+ for (const remote of byId.values()) {
+ const applied = await queueStoryboardWrite(remote.id, async () => {
+ const local = sanitizeStoryboard(await readRaw(remote.id));
+ if (local && !compareNewerWins(remote.updatedAt, local.updatedAt)) return false;
+ // Re-receiving a tombstone already held would restart its GC grace period.
+ if (local?.deleted === true && remote.deleted === true) return false;
+ if (local) {
+ await maybeJournalBeforeOverwrite({ kind: STORYBOARD_KIND, id: remote.id, local, remote, source });
+ }
+ await writeRaw(remote.id, remote);
+ await setSyncBaseHash(STORYBOARD_KIND, remote.id, contentHashForRecord(STORYBOARD_KIND, remote));
+ return true;
+ });
+ if (applied) changed += 1;
+ }
+ await flushBaseHashes();
+ if (changed > 0) console.log(`🎬 Storyboard sync: merged ${changed} board(s)`);
+ return { applied: changed > 0, count: changed };
+}
+
+/** Hard-prune tombstones deleted before the absolute epoch-ms `cutoffEpochMs`. */
+export async function pruneTombstonedStoryboards(cutoffEpochMs) {
+ if (!Number.isFinite(cutoffEpochMs)) return { pruned: 0 };
+ const candidateIds = await listTombstoneIdsBefore(cutoffEpochMs);
+ let pruned = 0;
+ await withBaseHashFlushBatch(async () => {
+ for (const id of candidateIds) {
+ // Re-check inside the per-id queue: a merge that rewrote the tombstone
+ // with a fresher deletedAt restarted its grace period.
+ const removed = await queueStoryboardWrite(id, async () => {
+ const current = sanitizeStoryboard(await readRaw(id));
+ const deletedAtMs = Date.parse(current?.deletedAt || '');
+ if (!current?.deleted || !Number.isFinite(deletedAtMs) || deletedAtMs >= cutoffEpochMs) return false;
+ await deleteRaw(id);
+ await deleteSyncBaseHash(STORYBOARD_KIND, id);
+ return true;
+ });
+ if (removed) pruned += 1;
+ }
+ });
+ if (pruned > 0) console.log(`🎬 Storyboard tombstone GC: pruned ${pruned} board(s)`);
+ return { pruned };
+}
+
+/**
+ * Restore a board from a conflict-journal snapshot: overlay the authored
+ * fields, un-tombstone, and bump `updatedAt` so the restore wins the next LWW
+ * and re-pushes. Null when the board was already hard-pruned.
+ */
+export async function restoreStoryboard(id, patch = {}) {
+ if (!isValidStoryboardId(id)) return null;
+ const restored = await queueStoryboardWrite(id, async () => {
+ const current = sanitizeStoryboard(await readRaw(id));
+ if (!current) return null;
+ const overlay = Object.fromEntries(RESTORABLE_FIELDS.filter((key) => key in patch).map((key) => [key, patch[key]]));
+ const next = sanitizeStoryboard({
+ ...current, ...overlay, deleted: false, deletedAt: null, updatedAt: new Date().toISOString(),
+ });
+ await writeRaw(id, next);
+ return next;
+ });
+ if (restored) emitRecordUpdated(STORYBOARD_KIND, id);
+ return restored;
+}
diff --git a/server/services/storyboards/sync.test.js b/server/services/storyboards/sync.test.js
new file mode 100644
index 0000000000..2f00a5a6d0
--- /dev/null
+++ b/server/services/storyboards/sync.test.js
@@ -0,0 +1,123 @@
+import { afterAll, describe, expect, it, vi } from 'vitest';
+import { cleanupTempDataRoots, lazyTempDataRoot, makePathsProxy } from '../../lib/mockPathsDataRoot.js';
+
+vi.mock('../../lib/fileUtils.js', async (importOriginal) =>
+ makePathsProxy(await importOriginal(), { dataRoot: () => lazyTempDataRoot('portos-storyboards-sync-') }));
+vi.mock('../sharing/recordEvents.js', () => ({
+ emitRecordUpdated: vi.fn(),
+ emitRecordDeleted: vi.fn(),
+ autoSubscribeRecordToAllPeers: vi.fn(() => Promise.resolve()),
+}));
+
+const { flushBaseHashes, __resetBaseHashCacheForTests } = await import('../../lib/conflictJournal.js');
+const { sanitizeRecordForWire } = await import('../../lib/syncWire.js');
+const { buildStoryboardAssetManifest } = await import('../sharing/peerSyncAssets.js');
+const { PORTOS_SCHEMA_VERSIONS } = await import('../../lib/schemaVersions.js');
+const events = await import('../sharing/recordEvents.js');
+const svc = await import('./index.js');
+
+afterAll(async () => {
+ await flushBaseHashes();
+ __resetBaseHashCacheForTests();
+ cleanupTempDataRoots();
+});
+
+const later = (iso, ms) => new Date(Date.parse(iso) + ms).toISOString();
+const asWire = (board, patch = {}) => ({ ...sanitizeRecordForWire('storyboard', board), ...patch });
+
+async function boardWithArt() {
+ const board = await svc.createStoryboard({ name: 'Wire Board' });
+ const { beatId } = await svc.addBeat(board.id, { title: 'Opening' });
+ const { panelId } = await svc.addPanel(board.id, beatId, { action: 'Wide of the gym doors' });
+ await svc.addPanelTake(board.id, panelId, { source: 'upload', image: 'example.png' });
+ await svc.addPanelTake(board.id, panelId, { source: 'sketch', sketchKey: 'sketch:abc-123' });
+ return { id: board.id, panelId, beatId };
+}
+
+describe('storyboard federation', () => {
+ it('declares its own version-gated sync category', () => {
+ expect(PORTOS_SCHEMA_VERSIONS.storyboards).toBe(1);
+ });
+
+ it('emits a record event on create, edit and delete so subscribed peers get each revision', async () => {
+ events.emitRecordUpdated.mockClear();
+ events.emitRecordDeleted.mockClear();
+ const { id } = await boardWithArt();
+ expect(events.autoSubscribeRecordToAllPeers).toHaveBeenCalledWith('storyboard', id);
+ expect(events.emitRecordUpdated).toHaveBeenCalledWith('storyboard', id);
+ await svc.deleteStoryboard(id);
+ expect(events.emitRecordDeleted).toHaveBeenCalledWith('storyboard', id);
+ });
+
+ it('merges a newer remote board whole and leaves an older one alone (LWW on updatedAt)', async () => {
+ const { id } = await boardWithArt();
+ const local = await svc.getStoryboard(id);
+ const stale = asWire(local, { name: 'Stale Peer Edit', updatedAt: later(local.updatedAt, -60_000) });
+ expect(await svc.mergeStoryboardsFromSync([stale])).toEqual({ applied: false, count: 0 });
+ expect((await svc.getStoryboard(id)).name).toBe('Wire Board');
+
+ const fresh = asWire(local, { name: 'Renamed On Peer', updatedAt: later(local.updatedAt, 60_000) });
+ expect(await svc.mergeStoryboardsFromSync([fresh])).toEqual({ applied: true, count: 1 });
+ expect((await svc.getStoryboard(id)).name).toBe('Renamed On Peer');
+ });
+
+ it('creates a board that only exists remotely; a repeated tombstone does not restart its grace period', async () => {
+ const { id } = await boardWithArt();
+ const wire = asWire(await svc.getStoryboard(id));
+ await svc.deleteStoryboard(id);
+ // The local tombstone is newer than the pushed live copy → the live copy loses.
+ expect(await svc.mergeStoryboardsFromSync([wire])).toEqual({ applied: false, count: 0 });
+ await expect(svc.getStoryboard(id)).rejects.toMatchObject({ status: 404 });
+
+ const coldId = 'sb-00000000-0000-4000-8000-000000000001';
+ const cold = { ...wire, id: coldId };
+ expect(await svc.mergeStoryboardsFromSync([cold])).toEqual({ applied: true, count: 1 });
+ expect((await svc.getStoryboard(coldId)).name).toBe('Wire Board');
+ const tomb = { ...cold, deleted: true, deletedAt: later(cold.updatedAt, 1000), updatedAt: later(cold.updatedAt, 1000) };
+ expect(await svc.mergeStoryboardsFromSync([tomb])).toEqual({ applied: true, count: 1 });
+ expect(await svc.mergeStoryboardsFromSync([tomb])).toEqual({ applied: false, count: 0 });
+ expect(await svc.listStoryboards()).not.toContainEqual(expect.objectContaining({ id: coldId }));
+ });
+
+ it('ships in-flight panel-maker state as idle and keeps sketch takes as references', async () => {
+ const { id, panelId } = await boardWithArt();
+ await svc.mutateStoryboard(id, (draft) => {
+ const panel = draft.beats[0].panels.find((p) => p.id === panelId);
+ panel.make = { nodes: { 'node-image': { status: 'running', jobIds: ['job-1'], runId: 'run-1' } } };
+ });
+ const wire = sanitizeRecordForWire('storyboard', await svc.getStoryboard(id));
+ const state = wire.beats[0].panels[0].make.nodes['node-image'];
+ expect(state).toMatchObject({ status: 'idle', jobIds: [], runId: 'run-1' });
+ expect(wire.beats[0].panels[0].takes.map((t) => t.sketchKey ?? t.image)).toEqual(['example.png', 'sketch:abc-123']);
+ // The local record is untouched.
+ expect((await svc.getStoryboard(id)).beats[0].panels[0].make.nodes['node-image'].status).toBe('running');
+ });
+
+ it('advertises each gallery take image in the asset manifest and none for a sketch take', async () => {
+ const { id } = await boardWithArt();
+ const { referenceStoryboardAssetManifest } = await import('../sharing/peerSyncAssets.js');
+ expect(referenceStoryboardAssetManifest(await svc.getStoryboard(id))).toEqual([
+ expect.objectContaining({ kind: 'image', filename: 'example.png' }),
+ ]);
+ // No bytes on disk in this test → the hashed manifest drops the unreadable entry.
+ expect(await buildStoryboardAssetManifest(await svc.getStoryboard(id))).toEqual([]);
+ });
+
+ it('prunes a tombstone only past the grace cutoff and restore brings a board back', async () => {
+ const { id } = await boardWithArt();
+ await svc.deleteStoryboard(id);
+ expect(await svc.listStoryboardIdsForSync({ includeDeleted: true })).toContain(id);
+ expect(await svc.listStoryboardIdsForSync()).not.toContain(id);
+ expect(await svc.pruneTombstonedStoryboards(Date.now() - 86_400_000)).toEqual({ pruned: 0 });
+ expect(await svc.listStoryboardIdsForSync({ includeDeleted: true })).toContain(id);
+
+ const restored = await svc.restoreStoryboard(id, { name: 'Restored Name', beats: [] });
+ expect(restored).toMatchObject({ name: 'Restored Name', deleted: false, beats: [] });
+ expect(await svc.listStoryboardsForSync()).toContainEqual(expect.objectContaining({ id }));
+
+ await svc.deleteStoryboard(id);
+ expect((await svc.pruneTombstonedStoryboards(Date.now() + 86_400_000)).pruned).toBeGreaterThanOrEqual(1);
+ expect(await svc.listStoryboardIdsForSync({ includeDeleted: true })).not.toContain(id);
+ expect(await svc.restoreStoryboard(id, {})).toBeNull();
+ });
+});