Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: defer media recovery until completion hooks register (#12054)

Adam Eivyf2f8a7cfb274
diff --git a/docs/STORAGE.md b/docs/STORAGE.md
index 054fb88964..f293c1a8ea 100644
--- a/docs/STORAGE.md
+++ b/docs/STORAGE.md
@@ -320,7 +320,7 @@ goal data together; interrupted intents remain retryable after restore.
**Examples.**
- Upload staging — `data/uploads/*`. Ephemeral; do not put in DB except as job references.
-- Media job queue — runtime queue state can stay file-backed short term, but **job history and artifact lineage are `db-primary`** and should move to the DB. Admission is durable: `enqueueJob` resolves only once a snapshot containing the job is written, a failed write refuses the job (`503 MEDIA_QUEUE_PERSIST_FAILED`) before any dispatch, and graceful shutdown first stops dispatch (`quiesceMediaJobQueue()`, before its first await — waiting jobs stay queued for the next boot instead of starting during teardown and restoring as interrupted; Run Now answers `503 MEDIA_QUEUE_SHUTTING_DOWN`), then runs the bounded `flushMediaJobQueue()` before exit. If boot finds the snapshot unreadable, persistence stays latched off to preserve that file, and new admissions are refused until the snapshot is repaired and the server restarts. Waiting work is bounded: at 250 queued jobs across all lanes (`MAX_PENDING_MEDIA_JOBS`, separate from each lane's running concurrency) a new submission is refused with a retryable `429 MEDIA_QUEUE_FULL` before any state changes; a restored snapshot above the bound is kept whole and only blocks new admissions until it drains. Local video failure holds live in the existing `data/media-jobs.json` envelope as `videoHolds`, beside unchanged `jobs`. They are machine-local dispatch state, never federated and never sent in capability/status payloads to peers; the local queue API exposes them on retained jobs and through `GET /api/media-jobs/holds`, so status and resume remain available when no retained jobs remain. Three matching terminal causes hold one catalog model/runtime until explicit resume. If hold metadata is damaged, boot restores unrelated work but holds all local video, including new submissions, behind an explicit session-only recovery action; the original snapshot stays preserved until repaired and restarted. Migration 344 adds an empty hold list to legacy envelopes without changing jobs; fresh installs use the empty seed. The envelope retains its existing filesystem backup coverage, with no new store, search index, sync cursor, or tombstone. A completion that cannot enter backup publication admission retains an optional `durabilityPending` receipt in its existing job row, while the published status stays `running`. Boot restores that receipt for admitted completion and hook delivery instead of rerunning the provider or marking it interrupted. The receipt is machine-local recovery state and excluded from API/peer projections. Legacy rows without a receipt keep their existing recovery behavior; no existing data needs transformation.
+- Media job queue — runtime queue state can stay file-backed short term, but **job history and artifact lineage are `db-primary`** and should move to the DB. Admission is durable: `enqueueJob` resolves only once a snapshot containing the job is written, a failed write refuses the job (`503 MEDIA_QUEUE_PERSIST_FAILED`) before any dispatch, and graceful shutdown first stops dispatch (`quiesceMediaJobQueue()`, before its first await — waiting jobs stay queued for the next boot instead of starting during teardown and restoring as interrupted; Run Now answers `503 MEDIA_QUEUE_SHUTTING_DOWN`), then runs the bounded `flushMediaJobQueue()` before exit. If boot finds the snapshot unreadable, persistence stays latched off to preserve that file, and new admissions are refused until the snapshot is repaired and the server restarts. Waiting work is bounded: at 250 queued jobs across all lanes (`MAX_PENDING_MEDIA_JOBS`, separate from each lane's running concurrency) a new submission is refused with a retryable `429 MEDIA_QUEUE_FULL` before any state changes; a restored snapshot above the bound is kept whole and only blocks new admissions until it drains. Local video failure holds live in the existing `data/media-jobs.json` envelope as `videoHolds`, beside unchanged `jobs`. They are machine-local dispatch state, never federated and never sent in capability/status payloads to peers; the local queue API exposes them on retained jobs and through `GET /api/media-jobs/holds`, so status and resume remain available when no retained jobs remain. Three matching terminal causes hold one catalog model/runtime until explicit resume. If hold metadata is damaged, boot restores unrelated work but holds all local video, including new submissions, behind an explicit session-only recovery action; the original snapshot stays preserved until repaired and restarted. Migration 344 adds an empty hold list to legacy envelopes without changing jobs; fresh installs use the empty seed. The envelope retains its existing filesystem backup coverage, with no new store, search index, sync cursor, or tombstone. A completion that cannot enter backup publication admission retains an optional `durabilityPending` receipt in its existing job row, while the published status stays `running`. Boot restores that receipt and opens admitted completion recovery only after domain completion hooks are registered, without rerunning the provider or marking it interrupted. The receipt is machine-local recovery state and excluded from API/peer projections. Legacy rows without a receipt keep their existing recovery behavior; no existing data needs transformation.
- Browser CDP profile / downloads — `data/browser-profile/`, `data/browser-downloads/` — cache, non-overridable backup excludes.
- Brain parity audit results — `data/brain_parity_reports.json` (`server/services/brainParity.js`, #4519). The last record-level brain-parity report per peer, keyed by peer `instanceId` (the same key `data/instances_sync_cursors.json` uses). `ephemeral-file` because it is a point-in-time *observation* about two installs, fully regenerable by re-running the audit, and stale the moment either side syncs — the durable state it describes lives in the brain stores. **Intentionally machine-local — never federated**: it is this install's view of a peer, and each peer computes its own. No sync cursor, no tombstone, no migration (an absent file reads as "nothing audited yet"). Per-type record lists are capped at 25 ids with a `truncated` flag so a badly diverged install can't grow the file without bound.
- Persistent-mind rollups and decision journal — `data/cos/persistent-mind-rollups.json` (`server/services/persistentMindContext.js`) and `data/cos/persistent-mind-journal.json` (`server/services/persistentMindJournal.js`). The rollups are sealed prose summaries of ranges that have left raw retention; the journal is the typed, individually addressable record of what the mind decided, committed to, is still asking, and is watching, each entry citing the message sequences it was drawn from. `file-primary` rather than `ephemeral-file`: both outlive the raw events they were derived from, so neither is regenerable, and a corrupt store fails CLOSED rather than resetting to `[]`. **Intentionally machine-local — never federated**: both quote one human private conversation, so the [machine-local privacy ADR](./decisions/2026-08-08-privacy-records-machine-local.md) keeps them off the federation layer entirely — no `PORTOS_SCHEMA_VERSIONS` entry, no sync cursor, no tombstone. No migration and no `data.reference/` seed: an absent file is the correct empty state. Retirement is a STATUS TRANSITION, never a deletion — a superseded entry stays readable with a pointer to what replaced it. Bounded (100 rollups; 500 journal entries, shedding settled history before anything still active), and backed up with the rest of `data/cos/`. The `history` cleanup scope clears the journal along with the messages it cites; a context-only clear does not.
diff --git a/server/services/README.md b/server/services/README.md
index a475e870ee..32307663bd 100644
--- a/server/services/README.md
+++ b/server/services/README.md
@@ -14,7 +14,7 @@ image-generation paths below are useful entry points when tracing render jobs.
| `imageGen/prepareParams.js` | Shared image request preparation and the canonical local image model selector. Use `selectLocalImageModelFromSettings()` for model identity and `resolveLocalImageModel()` when the caller must also enforce local runtime, hardware, and edit-image requirements. |
| `musicVideo/renderCancellation.js` | Shared capture-abort and ffmpeg kill selection for full/excerpt renders; records explicit cancellation before signaling and classifies nonzero closes without releasing ownership. |
| `musicVideo/sharingCopy.js` | Source-hashed private sharing MP4s: queued 720p H.264/AAC export, strict decimal 100 MB cap, supervised cancellation, verified download and owned-file cleanup. |
-| `mediaJobQueue/index.js` | Owns admission, lanes, durability and terminal effects; `videoHolds.js` returns local-video cohort preparation results and owns circuit breakers, `recoverySettlement.js` names the transient recovery evidence and the maintenance-permit release decision, while `videoGen/modelSelection.js` owns model-selection policy. |
+| `mediaJobQueue/index.js` | Owns admission, lanes, durability and terminal effects; boot passes `deferCompletionRecovery` to `initMediaJobQueue()` and calls `resumeMediaJobCompletions()` after completion hooks are registered; `videoHolds.js` returns local-video cohort preparation results and owns circuit breakers, `recoverySettlement.js` names the transient recovery evidence and the maintenance-permit release decision, while `videoGen/modelSelection.js` owns model-selection policy. |
| `imageGen/index.js` | Mode-aware image backend execution dispatcher. Preparation uses `lib/imageCleanDefaults.js` for cleaner policy and resolves local model identity before enqueueing. |
| `superColliderRuntime.js` | Managed SuperCollider runtime I/O (#9412): Docker probe, recipe-hash-stamped image build, `runSuperColliderContainer` (contained `sclang` run with forced container removal), the synthetic render probe and its cached evidence, `getSuperColliderStatus` and idempotent `setupSuperColliderRuntime`. Never runs at boot. |
| `superColliderRender.js` | Contained SuperCollider renders (#9413): media-queue `supercollider` jobs that snapshot source into a job-private dir, run the trusted NRT wrapper via `runSuperColliderContainer` (output quota, timeout, cancel → container force-removed), validate the decoded WAV and publish a 24 h preview + provenance sidecar (`readSuperColliderPreview`). Sweeps orphaned containers/scratch before each render. |
diff --git a/server/services/bootstrap.js b/server/services/bootstrap.js
index 63d8170d55..b74c53889d 100644
--- a/server/services/bootstrap.js
+++ b/server/services/bootstrap.js
@@ -141,7 +141,7 @@ import { initSyncLog } from './brainSyncLog.js';
import { backfillOriginInstanceId, brainCollectionStores } from './brainStorage.js';
import { relayUnloggedRecords } from './brainReconcile.js';
import { initSyncOrchestrator } from './syncOrchestrator.js';
-import { initMediaJobQueue, flushMediaJobQueue, quiesceMediaJobQueue } from './mediaJobQueue/index.js';
+import { initMediaJobQueue, resumeMediaJobCompletions, flushMediaJobQueue, quiesceMediaJobQueue } from './mediaJobQueue/index.js';
import { initSpriteLocalAnimationHook } from './sprites/localAnimationJobHook.js';
import { initLoraTraining } from './loraTraining/index.js';
import { initSharing } from './sharing/index.js';
@@ -964,6 +964,7 @@ export const runBootSequence = ({ io, httpServer, localHttpServer, httpsEnabled,
// no-op it: the reconcile needs a database the hermetic root does not have.
initLoraTraining,
initMediaJobDependentHooks,
+ resumeMediaJobCompletions,
// Sharing: attach chokidar watchers to every registered share bucket so
// incoming manifests from peers are picked up live. Backlog processing
diff --git a/server/services/bootstrapSequence.js b/server/services/bootstrapSequence.js
index 872d9f40d6..9449bd57d6 100644
--- a/server/services/bootstrapSequence.js
+++ b/server/services/bootstrapSequence.js
@@ -380,6 +380,7 @@ export const runPostRouteSequence = ({
initMediaJobQueue,
initLoraTraining,
initMediaJobDependentHooks,
+ resumeMediaJobCompletions,
initSharing,
recoverCreativeDirectorProjects,
runDatabasePhase: databasePhase,
@@ -396,9 +397,13 @@ export const runPostRouteSequence = ({
bestEffort(recoverStuckClassifications(), logFailure('Brain recovery failed'));
return bestEffort(recoverInterruptedRepoClones(), logFailure('Brain clone recovery failed'));
})
- .then(() => initMediaJobQueue())
+ .then(() => initMediaJobQueue({ deferCompletionRecovery: true }))
.then(() => bestEffort(initLoraTraining(), logFailure('loraTraining init failed')))
.then(() => initMediaJobDependentHooks())
+ .then(() => {
+ // Recovery may wait behind a backup cut; arm it without blocking boot.
+ resumeMediaJobCompletions();
+ })
.then(() => {
// Sharing attaches chokidar watchers + drains the offline manifest
// backlog. Fire-and-forget — a failed bucket shouldn't block boot.
diff --git a/server/services/bootstrapSequence.test.js b/server/services/bootstrapSequence.test.js
index c1fbaf6ad4..53a73aa7f0 100644
--- a/server/services/bootstrapSequence.test.js
+++ b/server/services/bootstrapSequence.test.js
@@ -663,6 +663,7 @@ describe('runPostRouteSequence — post-route boot order', () => {
initMediaJobQueue: step('initMediaJobQueue', () => Promise.resolve()),
initLoraTraining: step('initLoraTraining', () => Promise.resolve()),
initMediaJobDependentHooks: step('initMediaJobDependentHooks'),
+ resumeMediaJobCompletions: step('resumeMediaJobCompletions'),
initSharing: step('initSharing', () => Promise.resolve()),
recoverCreativeDirectorProjects: step('recoverCreativeDirectorProjects', () => Promise.resolve()),
runDatabasePhase: step('runDatabasePhase', () => Promise.resolve({ dbReady: true })),
@@ -687,6 +688,7 @@ describe('runPostRouteSequence — post-route boot order', () => {
'initMediaJobQueue',
'initLoraTraining',
'initMediaJobDependentHooks',
+ 'resumeMediaJobCompletions',
'initSharing',
'recoverCreativeDirectorProjects',
'runDatabasePhase',
@@ -707,6 +709,7 @@ describe('runPostRouteSequence — post-route boot order', () => {
'initSyncLog',
'initMediaJobQueue',
'initMediaJobDependentHooks',
+ 'resumeMediaJobCompletions',
'runDatabasePhase',
'loadSeriesCoverBackfill',
'backfillSeriesCoverImages',
diff --git a/server/services/mediaJobQueue/index.js b/server/services/mediaJobQueue/index.js
index ad8f52afcc..46ce5d7a4c 100644
--- a/server/services/mediaJobQueue/index.js
+++ b/server/services/mediaJobQueue/index.js
@@ -653,7 +653,7 @@ function captureQueueSnapshot() {
// other write (progress, enqueue, the shutdown flush, or one landing after a
// completion's own retries gave up) keeps it `running` on disk and leaves it
// to an admitted recovery commit, never publishing it in the middle of a cut.
- const completionAdmitted = holdsBackupAssetPublication();
+ const completionAdmitted = !completionRecoveryDeferred && holdsBackupAssetPublication();
const canWriteOutcome = (job) => completionAdmitted || job.durabilityPending?.status !== 'completed';
const hasDeferredCompletion = [...pendingTransitions.keys()].some((job) => !canWriteOutcome(job));
// Strip non-serializable bits.
@@ -682,8 +682,20 @@ async function acknowledgeWrittenTransitions(transitions) {
// hooks join the lease from the fan-out. Never awaited inside the persist
// chain, so a cut draining an admitted write cannot deadlock on it.
let completionRecovery = null;
+let completionRecoveryDeferred = false;
+
+// Boot restores the queue before wiring domain hooks. Open recovery only after
+// those listeners exist; keeping the receipt private also protects completions
+// from a newly dispatched job during that startup window.
+export function resumeMediaJobCompletions() {
+ completionRecoveryDeferred = false;
+ if ([...pendingTransitions.keys()].some(job => job.durabilityPending?.status === 'completed')) {
+ recoverCompletionsUnderAdmission();
+ }
+}
+
function recoverCompletionsUnderAdmission() {
- if (completionRecovery) return;
+ if (completionRecovery || completionRecoveryDeferred) return;
// Tracked as a terminal operation so the shutdown flush drains it (and times
// out, reporting failure, if an open cut still holds the commit).
completionRecovery = trackTerminalOperation(withBackupAssetPublication(() => persist())
@@ -717,8 +729,9 @@ async function announceTerminalTransition(job, onDurable = () => {}) {
runOutsideBackupAssetPublication(() => mediaJobEvents.emit(job.status, job));
}
-export async function initMediaJobQueue() {
+export async function initMediaJobQueue({ deferCompletionRecovery = false } = {}) {
if (initPromise) return initPromise;
+ completionRecoveryDeferred = deferCompletionRecovery;
initPromise = (async () => {
// Once-at-boot: subsequent persist() calls assume the data dir exists.
await ensureDir(PATHS.data);
@@ -2060,6 +2073,7 @@ export function __resetForTests() {
// jobs file would otherwise leave every later test's queue unable to persist.
persistBlocked = false;
completionRecovery = null;
+ completionRecoveryDeferred = false;
}
// Test-only deterministic settle hook. EventEmitter terminal handlers and
diff --git a/server/services/mediaJobQueue/index.test.js b/server/services/mediaJobQueue/index.test.js
index 9825716987..f2e556cd48 100644
--- a/server/services/mediaJobQueue/index.test.js
+++ b/server/services/mediaJobQueue/index.test.js
@@ -3775,6 +3775,46 @@ describe('backup snapshot admission for media completion', () => {
? [expect.objectContaining({ resource: jobId, unsettled: true })] : []);
});
+ it('waits for boot hook registration before recovering a receipt with admission already open', async () => {
+ const jobId = '00000000-0000-4000-8000-000000000092';
+ const result = { filename: `${jobId}.mp4` };
+ writeFileSync(join(tempDataDir, 'media-jobs.json'), JSON.stringify({ jobs: [{
+ id: jobId, kind: 'video', status: 'running', params: { prompt: 'synthetic' },
+ durabilityPending: { status: 'completed', result, completedAt: new Date().toISOString() },
+ }] }));
+ mediaJobQueue.mediaJobEvents.off('completed', onCompleted);
+ const completed = vi.fn();
+ const { runPostRouteSequence } = await import('../bootstrapSequence.js');
+ const noWork = async () => {};
+ const onFatal = vi.fn();
+ let beforeHooks;
+ await runPostRouteSequence({
+ startBackgroundServices: noWork, ensureSelf: noWork, initSyncLog: noWork,
+ recoverStuckClassifications: noWork, recoverInterruptedRepoClones: noWork,
+ initMediaJobQueue: mediaJobQueue.initMediaJobQueue,
+ initLoraTraining: async () => {
+ await mediaJobQueue.flushMediaJobQueue();
+ beforeHooks = { job: persistedJobs()[0], rowCount: rows.size };
+ },
+ initMediaJobDependentHooks: () => {
+ mediaJobQueue.mediaJobEvents.on('completed', onCompleted);
+ mediaJobQueue.mediaJobEvents.on('completed', completed);
+ },
+ resumeMediaJobCompletions: mediaJobQueue.resumeMediaJobCompletions,
+ initSharing: noWork, recoverCreativeDirectorProjects: noWork,
+ runDatabasePhase: noWork, loadSeriesCoverBackfill: async () => noWork,
+ startListening: noWork, onFatal,
+ });
+ await flush();
+ await waitFor(() => rows.has(jobId));
+ expect(onFatal).not.toHaveBeenCalled();
+ expect(beforeHooks).toMatchObject({ job: { status: 'running', durabilityPending: { result } }, rowCount: 0 });
+ expect(completed).toHaveBeenCalledTimes(1);
+ expect(mediaJobQueue.getJob(jobId)).toMatchObject({ status: 'completed', result });
+ expect(stubs.generateVideo).not.toHaveBeenCalled();
+ mediaJobQueue.mediaJobEvents.off('completed', completed);
+ });
+
it('drains a completion already publishing its queue row and hook rows before the cut', async () => {
let finishWrite;
const writeGate = new Promise((resolve) => { finishWrite = resolve; });