Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: recover finished media jobs after backup admission times out (#12054)
Adam Eivy51af9437b416
diff --git a/docs/STORAGE.md b/docs/STORAGE.mdindex 27dbc97dd8..46fa95867c 100644--- a/docs/STORAGE.md+++ b/docs/STORAGE.md@@ -320,7 +320,7 @@ goal data together; interrupted intents remain retryable after restore.**Examples.**- Upload staging — `data/uploads/*`. Ephemeral; do not put in DB except as job references.-- Media job queue — runtime queue state can stay file-backed short term, but **job history and artifact lineage are `db-primary`** and should move to the DB. Admission is durable: `enqueueJob` resolves only once a snapshot containing the job is written, a failed write refuses the job (`503 MEDIA_QUEUE_PERSIST_FAILED`) before any dispatch, and graceful shutdown first stops dispatch (`quiesceMediaJobQueue()`, before its first await — waiting jobs stay queued for the next boot instead of starting during teardown and restoring as interrupted; Run Now answers `503 MEDIA_QUEUE_SHUTTING_DOWN`), then runs the bounded `flushMediaJobQueue()` before exit. The one exception is a boot that found the snapshot unreadable: persistence stays latched off to preserve that file, and the queue keeps accepting work in memory only. Waiting work is bounded: at 250 queued jobs across all lanes (`MAX_PENDING_MEDIA_JOBS`, separate from each lane's running concurrency) a new submission is refused with a retryable `429 MEDIA_QUEUE_FULL` before any state changes; a restored snapshot above the bound is kept whole and only blocks new admissions until it drains. Local video failure holds live in the existing `data/media-jobs.json` envelope as `videoHolds`, beside unchanged `jobs`. They are machine-local dispatch state, never federated and never sent in capability/status payloads to peers; the local queue API exposes them on retained jobs and through `GET /api/media-jobs/holds`, so status and resume remain available when no retained jobs remain. Three matching terminal causes hold one catalog model/runtime until explicit resume. If hold metadata is damaged, boot restores unrelated work but holds all local video, including new submissions, behind an explicit session-only recovery action; the original snapshot stays preserved until repaired and restarted. Migration 344 adds an empty hold list to legacy envelopes without changing jobs; fresh installs use the empty seed. The envelope retains its existing filesystem backup coverage, with no new store, search index, sync cursor, or tombstone.+- Media job queue — runtime queue state can stay file-backed short term, but **job history and artifact lineage are `db-primary`** and should move to the DB. Admission is durable: `enqueueJob` resolves only once a snapshot containing the job is written, a failed write refuses the job (`503 MEDIA_QUEUE_PERSIST_FAILED`) before any dispatch, and graceful shutdown first stops dispatch (`quiesceMediaJobQueue()`, before its first await — waiting jobs stay queued for the next boot instead of starting during teardown and restoring as interrupted; Run Now answers `503 MEDIA_QUEUE_SHUTTING_DOWN`), then runs the bounded `flushMediaJobQueue()` before exit. The one exception is a boot that found the snapshot unreadable: persistence stays latched off to preserve that file, and the queue keeps accepting work in memory only. Waiting work is bounded: at 250 queued jobs across all lanes (`MAX_PENDING_MEDIA_JOBS`, separate from each lane's running concurrency) a new submission is refused with a retryable `429 MEDIA_QUEUE_FULL` before any state changes; a restored snapshot above the bound is kept whole and only blocks new admissions until it drains. Local video failure holds live in the existing `data/media-jobs.json` envelope as `videoHolds`, beside unchanged `jobs`. They are machine-local dispatch state, never federated and never sent in capability/status payloads to peers; the local queue API exposes them on retained jobs and through `GET /api/media-jobs/holds`, so status and resume remain available when no retained jobs remain. Three matching terminal causes hold one catalog model/runtime until explicit resume. If hold metadata is damaged, boot restores unrelated work but holds all local video, including new submissions, behind an explicit session-only recovery action; the original snapshot stays preserved until repaired and restarted. Migration 344 adds an empty hold list to legacy envelopes without changing jobs; fresh installs use the empty seed. The envelope retains its existing filesystem backup coverage, with no new store, search index, sync cursor, or tombstone. A completion that cannot enter backup publication admission retains an optional `durabilityPending` receipt in its existing job row, while the published status stays `running`. Boot restores that receipt for admitted completion and hook delivery instead of rerunning the provider or marking it interrupted. The receipt is machine-local recovery state and excluded from API/peer projections. Legacy rows without a receipt keep their existing recovery behavior; no existing data needs transformation.- Browser CDP profile / downloads — `data/browser-profile/`, `data/browser-downloads/` — cache, non-overridable backup excludes.- Brain parity audit results — `data/brain_parity_reports.json` (`server/services/brainParity.js`, #4519). The last record-level brain-parity report per peer, keyed by peer `instanceId` (the same key `data/instances_sync_cursors.json` uses). `ephemeral-file` because it is a point-in-time *observation* about two installs, fully regenerable by re-running the audit, and stale the moment either side syncs — the durable state it describes lives in the brain stores. **Intentionally machine-local — never federated**: it is this install's view of a peer, and each peer computes its own. No sync cursor, no tombstone, no migration (an absent file reads as "nothing audited yet"). Per-type record lists are capped at 25 ids with a `truncated` flag so a badly diverged install can't grow the file without bound.- Persistent-mind rollups and decision journal — `data/cos/persistent-mind-rollups.json` (`server/services/persistentMindContext.js`) and `data/cos/persistent-mind-journal.json` (`server/services/persistentMindJournal.js`). The rollups are sealed prose summaries of ranges that have left raw retention; the journal is the typed, individually addressable record of what the mind decided, committed to, is still asking, and is watching, each entry citing the message sequences it was drawn from. `file-primary` rather than `ephemeral-file`: both outlive the raw events they were derived from, so neither is regenerable, and a corrupt store fails CLOSED rather than resetting to `[]`. **Intentionally machine-local — never federated**: both quote one human private conversation, so the [machine-local privacy ADR](./decisions/2026-08-08-privacy-records-machine-local.md) keeps them off the federation layer entirely — no `PORTOS_SCHEMA_VERSIONS` entry, no sync cursor, no tombstone. No migration and no `data.reference/` seed: an absent file is the correct empty state. Retirement is a STATUS TRANSITION, never a deletion — a superseded entry stays readable with a pointer to what replaced it. Bounded (100 rollups; 500 journal entries, shedding settled history before anything still active), and backed up with the rest of `data/cos/`. The `history` cleanup scope clears the journal along with the messages it cites; a context-only clear does not.diff --git a/server/services/mediaJobQueue/index.js b/server/services/mediaJobQueue/index.jsindex f7a91546e6..ad8f52afcc 100644--- a/server/services/mediaJobQueue/index.js+++ b/server/services/mediaJobQueue/index.js@@ -657,8 +657,9 @@ function captureQueueSnapshot() {const canWriteOutcome = (job) => completionAdmitted || job.durabilityPending?.status !== 'completed';const hasDeferredCompletion = [...pendingTransitions.keys()].some((job) => !canWriteOutcome(job));// Strip non-serializable bits.- const serializable = live.map((job) => (canWriteOutcome(job) ? { ...job, ...job.durabilityPending } : job)).map(({ id, kind, owner, status, queuedAt, startedAt, completedAt, params, result, error, position, progress, statusMsg, etaMs }) =>- ({ id, kind, owner, status, queuedAt, startedAt, completedAt, params, result, error, position, progress, statusMsg, etaMs }),+ const serializable = live.map((job) => (canWriteOutcome(job) ? { ...job, ...job.durabilityPending } : job)).map(({ id, kind, owner, status, queuedAt, startedAt, completedAt, params, result, error, position, progress, statusMsg, etaMs, durabilityPending }) =>+ ({ id, kind, owner, status, queuedAt, startedAt, completedAt, params, result, error, position, progress, statusMsg, etaMs,+ ...(status !== 'completed' && durabilityPending?.status === 'completed' ? { durabilityPending } : {}) }),);const transitions = [...pendingTransitions].filter(([job]) => canWriteOutcome(job));return { data: { jobs: serializable, videoHolds: videoHolds.snapshot() }, transitions, hasDeferredCompletion };@@ -690,6 +691,32 @@ function recoverCompletionsUnderAdmission() {.finally(() => { completionRecovery = null; }));}+async function announceTerminalTransition(job, onDurable = () => {}) {+ if (job.params?.videoProduction) {+ const tag = job.params.videoProduction;+ const { settleVideoAttempt } = await import('../creativeDirector/videoExecution.js');+ await settleVideoAttempt(tag.projectId, tag.attemptId, { jobId: job.id, status: job.params.videoProduction.submissionUncertain || (job.status === 'failed' && /timeout|timed out|interrupted/i.test(job.error || '')) ? 'uncertain' : job.status })+ .catch(error => console.error(`❌ Video receipt could not settle: ${error.message}`));+ }+ // Wake the lane finalizer without polling. Some providers emit their+ // terminal event just before their kickoff promise resolves; the promise+ // retains that signal until runJob reaches the await below.+ onDurable();+ const logPrefix = job.status === 'completed' ? '✅' : job.status === 'canceled' ? '🛑' : '❌';+ const logSuffix = job.status === 'failed' ? `: ${job.error}` : job.status === 'canceled' ? ' (was running)' : '';+ const log = job.status === 'failed' ? console.error : console.log;+ log(`${logPrefix} media-job [${job.id.slice(0, 8)}] ${job.status}${logSuffix}`);+ const ssePayload =+ job.status === 'completed' ? { type: 'complete', result: job.result }+ : job.status === 'canceled' ? { type: 'canceled', reason: job.error }+ : { type: 'error', error: job.error };+ broadcastSse(ensureSseEntry(job.id), ssePayload);+ closeJobAfterDelay(sseJobs, job.id);+ // Hooks that publish files or rows acquire their own admission; see+ // runOutsideBackupAssetPublication.+ runOutsideBackupAssetPublication(() => mediaJobEvents.emit(job.status, job));+}+export async function initMediaJobQueue() {if (initPromise) return initPromise;initPromise = (async () => {@@ -744,6 +771,22 @@ export async function initMediaJobQueue() {// Reconcile the original idempotency key while preserving its peer slot.const j = isRemoteMediaJob(persisted) && persisted.params?.remotePeerReservation === true&& ['failed', 'canceled'].includes(persisted.status) ? { ...persisted, status: 'running' } : persisted;+ if (j.status === 'running' && j.durabilityPending?.status === 'completed') {+ // This optional receipt is private queue state, not a published result.+ // Legacy snapshots have no receipt and retain their existing recovery.+ const receipt = j.durabilityPending;+ const permit = maintenance.recoverOwned('media', j.id);+ archive.push(j);+ stageTerminalTransition(j, target => Object.assign(target, receipt), () => permit.run(async () => {+ videoHolds.recordTerminal(j);+ videoHolds.updateQueued(queue);+ await announceTerminalTransition(j);+ // A receipt proves completion, not remote executor teardown or input+ // cleanup. finish deliberately retains any uncertain ownership.+ await permit.finish();+ }));+ continue;+ }if (j.status === 'running' && maintenance.held()) {maintenance.recoverOwned('media', j.id);if (j.kind === 'training' && j.params?.runId && await jobHasSurvivingTrainer(j.params.runId)) {@@ -936,14 +979,16 @@ function startLaneJob(job, { lane }) {? (job.params?.mode || 'cloud'): lane === 'remote' ? `remote ${job.kind}` : job.kind;let finalized = false;+ let terminalCounted = false;let archiveWrite;function finalizeLane() {+ if (!terminalCounted && ['completed', 'failed', 'canceled'].includes(job.status)) {+ terminalCounted = true;+ videoHolds.recordTerminal(job);+ videoHolds.updateQueued(queue);+ }if (finalized) return archiveWrite;finalized = true;- // Count once, after every terminal path (including pre-dispatch throws),- // and before releasing the lane. Boot reconciliation never comes here.- videoHolds.recordTerminal(job);- videoHolds.updateQueued(queue);if (lane === 'cloud') {const idx = cloudRunning.indexOf(job);if (idx >= 0) cloudRunning.splice(idx, 1);@@ -974,7 +1019,7 @@ function startLaneJob(job, { lane }) {broadcastSse(ensureSseEntry(job.id), { type: 'started', kind: job.kind });mediaJobEvents.emit('started', job);console.log(`▶️ media-job [${job.id.slice(0, 8)}] ${label} started`);- const lifecycle = runJob(job, permit);+ const lifecycle = runJob(job, permit, finalizeLane);markRunning();lifecycleEvidence = await lifecycle;} catch (err) {@@ -1275,22 +1320,24 @@ function makeGenDispatcher(emitter, job, handlers) {// The dispatch window closes once the provider has been invoked (not settled —// a provider kickoff may legitimately stay pending) or the job failed before// reaching it. The finally covers a throw anywhere before that point.-async function runJob(job, permit) {+async function runJob(job, permit, finalizeLane) {let markDispatched;trackOperation(dispatchOperations, new Promise((resolve) => { markDispatched = resolve; }));try {- return await runJobLifecycle(job, markDispatched, permit);+ return await runJobLifecycle(job, markDispatched, permit, finalizeLane);} finally {markDispatched();}}-async function runJobLifecycle(job, markDispatched, permit) {+async function runJobLifecycle(job, markDispatched, permit, finalizeLane) {const sseEntry = ensureSseEntry(job.id);let resolveTerminalState;const terminalState = new Promise((resolve) => { resolveTerminalState = resolve; });let resolveProviderSettlement;const providerSettlement = new Promise(resolve => { resolveProviderSettlement = resolve; });+ let resolveDeferredCompletion;+ const deferredCompletion = new Promise(resolve => { resolveDeferredCompletion = resolve; });let watchdogNeedsProviderSettlement = false;// Transient: returned to the lane finalizer, never persisted or projected.const recovery = createRecoveryEvidence();@@ -1386,7 +1433,24 @@ async function runJobLifecycle(job, markDispatched, permit) {emitter.off?.('activity', onActivity);emitter.off?.('progress', onActivity);await drainProgressPersist();- await admitCompletion(state, () => publishTerminalTransition(state, apply, remoteInputsDisposable));+ let publicationStarted = false;+ try {+ await admitCompletion(state, () => {+ publicationStarted = true;+ return publishTerminalTransition(state, apply, remoteInputsDisposable);+ });+ } catch (error) {+ if (publicationStarted || state !== 'completed') throw error;+ console.warn(`⏸️ media-job [${job.id.slice(0, 8)}] completion retained for publication recovery: ${error.message}`);+ // The provider finished, but admission never opened. Persist a pending+ // receipt without publishing completion, so recovery can finish it even+ // after a restart. Do not replay a callback that already started.+ try {+ await publishTerminalTransition(state, apply, remoteInputsDisposable);+ } finally {+ resolveDeferredCompletion();+ }+ }}async function publishTerminalTransition(state, apply, remoteInputsDisposable) {@@ -1411,29 +1475,7 @@ async function runJobLifecycle(job, markDispatched, permit) {// A shared persist can acknowledge another job: restore THAT job's// maintenance context, and never let cleanup failure lose this callback.await releaseRemoteAudioInputs(remoteInputsDisposable);- if (job.params?.videoProduction) {- const tag = job.params.videoProduction;- const { settleVideoAttempt } = await import('../creativeDirector/videoExecution.js');- await settleVideoAttempt(tag.projectId, tag.attemptId, { jobId: job.id, status: job.params.videoProduction.submissionUncertain || (state === 'failed' && /timeout|timed out|interrupted/i.test(job.error || '')) ? 'uncertain' : state })- .catch(error => console.error(`❌ Video receipt could not settle: ${error.message}`));- }- // Wake the lane finalizer without polling. Some providers emit their- // terminal event just before their kickoff promise resolves; the promise- // retains that signal until runJob reaches the await below.- resolveTerminalState();- const logPrefix = state === 'completed' ? '✅' : state === 'canceled' ? '🛑' : '❌';- const logSuffix = state === 'failed' ? `: ${job.error}` : state === 'canceled' ? ' (was running)' : '';- const log = state === 'failed' ? console.error : console.log;- log(`${logPrefix} media-job [${job.id.slice(0, 8)}] ${state}${logSuffix}`);- const ssePayload =- state === 'completed' ? { type: 'complete', result: job.result }- : state === 'canceled' ? { type: 'canceled', reason: job.error }- : { type: 'error', error: job.error };- broadcastSse(sseEntry, ssePayload);- closeJobAfterDelay(sseJobs, job.id);- // Hooks that publish files or rows acquire their own admission; see- // runOutsideBackupAssetPublication.- runOutsideBackupAssetPublication(() => mediaJobEvents.emit(state, job));+ await announceTerminalTransition(job, resolveTerminalState);});await persistTerminalTransition(job);}@@ -1679,6 +1721,10 @@ async function runJobLifecycle(job, markDispatched, permit) {// The gen modules emit completed/failed asynchronously after kickoff. Await// the explicit terminal signal rather than polling every 100ms; this makes// lane release immediate and gives tests a deterministic lifecycle to drain.+ // Only release the execution lane after the provider invocation has settled.+ // Keep maintenance ownership until the admitted completion is durable.+ await Promise.race([terminalState, deferredCompletion]);+ if (job.durabilityPending) finalizeLane();await terminalState;if (watchdogNeedsProviderSettlement) await providerSettlement;// A watchdog can win the terminal row before the real provider responds.diff --git a/server/services/mediaJobQueue/index.test.js b/server/services/mediaJobQueue/index.test.jsindex eee2c5b000..20f706b2a0 100644--- a/server/services/mediaJobQueue/index.test.js+++ b/server/services/mediaJobQueue/index.test.js@@ -3679,6 +3679,86 @@ describe('backup snapshot admission for media completion', () => {expect(capture(jobId)).toEqual({ files: [`${jobId}.mp4`], queueStatus: 'completed', rows: new Map([[jobId, `${jobId}.mp4`]]) });});+ it('releases a finished provider lane after admission times out and recovers its pending result', async () => {+ vi.useFakeTimers();+ const completed = vi.fn();+ mediaJobQueue.mediaJobEvents.on('completed', completed);+ let release;+ let finishProvider;+ stubs.generateVideo.mockImplementationOnce(() => new Promise(resolve => { finishProvider = resolve; }));+ try {+ const { jobId } = await mediaJobQueue.enqueueJob({ kind: 'video', params: { prompt: 'synthetic' } });+ await vi.waitFor(() => expect(stubs.generateVideo).toHaveBeenCalledTimes(1));+ await flush();+ release = await boundary.acquireBackupSnapshotCut();+ renderFinishes(jobId);+ await vi.advanceTimersByTimeAsync(0);+ // Jump to the final notice interval instead of executing an hour of worker ticks.+ vi.setSystemTime(Date.now() + 59 * 60_000);+ await vi.advanceTimersByTimeAsync(60_000);+ const pending = { status: 'completed', result: { filename: `${jobId}.mp4` } };+ await vi.waitFor(() => expect(mediaJobQueue.getJob(jobId)).toMatchObject({ status: 'running', durabilityPending: pending }));+ expect(mediaJobQueue.getRunningJob().id).toBe(jobId);+ finishProvider({});+ await vi.waitFor(() => expect(mediaJobQueue.getRunningJob()).toBeNull());+ expect(persistedJobs().find(j => j.id === jobId)).toMatchObject({ status: 'running', durabilityPending: pending });+ expect(completed).not.toHaveBeenCalled();+ expect(rows.size).toBe(0);++ const next = await mediaJobQueue.enqueueJob({ kind: 'video', params: { prompt: 'next synthetic' } });+ await vi.waitFor(() => expect(stubs.generateVideo).toHaveBeenCalledTimes(2));+ expect(mediaJobQueue.getRunningJob().id).toBe(next.jobId);+ release();+ release = null;+ await flush();+ await vi.waitFor(() => expect(rows.get(jobId)).toBe(`${jobId}.mp4`));+ expect(completed).toHaveBeenCalledTimes(1);+ expect(mediaJobQueue.getJob(jobId)).toMatchObject({ status: 'completed', result: pending.result });+ expect(mediaJobQueue.getJob(jobId).durabilityPending).toBeUndefined();+ renderFinishes(next.jobId);+ await flush();+ await expect(mediaJobQueue.flushMediaJobQueue()).resolves.toEqual({ ok: true });+ const { maintenance } = await import('../../lib/maintenanceAdmission.js');+ expect(maintenance.status().blockers).toEqual([]);+ } finally {+ finishProvider?.({});+ release?.();+ await flush();+ vi.clearAllTimers();+ vi.useRealTimers();+ mediaJobQueue.mediaJobEvents.off('completed', completed);+ }+ });++ it('restores a pending completion receipt without replaying its provider or publishing during a cut', async () => {+ const jobId = '00000000-0000-4000-8000-000000000091';+ const result = { filename: `${jobId}.mp4` };+ writeFileSync(join(tempDataDir, 'media-jobs.json'), JSON.stringify({ jobs: [{+ id: jobId, kind: 'video', status: 'running', params: { prompt: 'synthetic' },+ durabilityPending: { status: 'completed', result, progress: 1, statusMsg: 'Completed', completedAt: new Date().toISOString() },+ }] }));+ const release = await boundary.acquireBackupSnapshotCut();+ try {+ await mediaJobQueue.initMediaJobQueue();+ expect(mediaJobQueue.getJob(jobId)).toMatchObject({ status: 'running', durabilityPending: { status: 'completed', result } });+ expect(mediaJobQueue.getRunningJob()).toBeNull();+ expect(persistedJobs()[0]).toMatchObject({ status: 'running', durabilityPending: { result } });+ expect(rows.size).toBe(0);+ expect(stubs.generateVideo).not.toHaveBeenCalled();+ } finally {+ release();+ }+ await flush();+ await waitFor(() => rows.has(jobId));+ expect(mediaJobQueue.getJob(jobId)).toMatchObject({ status: 'completed', result });+ expect(mediaJobQueue.getJob(jobId).error).toBeUndefined();+ expect(persistedJobs()[0]).toMatchObject({ status: 'completed', result });+ expect(persistedJobs()[0].durabilityPending).toBeUndefined();+ expect(stubs.generateVideo).not.toHaveBeenCalled();+ const { maintenance } = await import('../../lib/maintenanceAdmission.js');+ expect(maintenance.status().blockers).toEqual([]);+ });+it('drains a completion already publishing its queue row and hook rows before the cut', async () => {let finishWrite;const writeGate = new Promise((resolve) => { finishWrite = resolve; });@@ -3745,6 +3825,7 @@ describe('backup snapshot admission for media completion', () => {const release = await boundary.acquireBackupSnapshotCut({ timeoutMs: 50 });// Storage is writable again and an unrelated queue write lands during the cut.await mediaJobQueue.enqueueJob({ kind: 'video', params: { prompt: 'unrelated' } });+ mediaJobQueue.quiesceMediaJobQueue();await new Promise((resolve) => setTimeout(resolve, 150));expect(mediaJobQueue.getJob(recovered).status).toBe('running');expect(capture(recovered)).toEqual({ files: [`${recovered}.mp4`], queueStatus: 'running', rows: new Map() });@@ -3762,6 +3843,7 @@ describe('backup snapshot admission for media completion', () => {await settle();const release = await boundary.acquireBackupSnapshotCut({ timeoutMs: 50 });await mediaJobQueue.enqueueJob({ kind: 'video', params: { prompt: 'unrelated' } });+ mediaJobQueue.quiesceMediaJobQueue();const flushed = await mediaJobQueue.flushMediaJobQueue({ timeoutMs: 200 });expect(flushed).toMatchObject({ ok: false, timedOut: true });release();