Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
feat: hold queued renders for a backend whose quota says when it resets
Stephen Hilderbrandcea48cde9bd3
diff --git a/server/lib/README.md b/server/lib/README.mdindex 3120005b55..de041e1457 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -46,6 +46,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `generationModes.js` | Shared immutable image/video render-backend alphabets (`IMAGE_GEN_MODE[S]`, `QUEUEABLE_IMAGE_MODES`, `VIDEO_GEN_MODE[S]`, and cloud subsets) plus `mediaJobExecutionLane({ kind, mode, remote })` / `MEDIA_JOB_EXECUTION_LANES` — THE media-queue lane rule the scheduler and the public job projection share, kept below both validation and generation services so schemas never import orchestration modules. || `mediaJobPreparation.js` | `createMediaJobPreparation()` tracks per-job cancellation before cloud providers install a render handle; `run`, `cancel`, `take`, and `ids` cover preparation, handoff, and bulk cancellation. || `imageGenCapabilities.js` | Per-backend image-gen capability literals, a dependency-free leaf beside `generationModes.js`: `EDIT_INCAPABLE_IMAGE_MODES`/`isEditCapableMode` and the derived `I2I_CAPABLE_MODES`, the shipped defaults (`CODEX_IMAGEGEN_DEFAULT_MODEL`/`_EFFORT`, `AGY_IMAGEGEN_DEFAULT_MODEL`, `AGY_IMAGEGEN_IMAGE_MODEL`), the `AGY_`/`GROK_ASPECT_RATIOS` alphabets, and `IMAGE_GEN_PROVIDER_CAPABILITIES` (`supportsModelOverride`, `maxInputImages`, `promptRequiredWithInputImage`) with its `supportsCloudModelOverride`/`maxInputImages`/`cloudPromptRequired` readers. The `services/imageGen/*` modules re-export these, and `client/src/lib/imageGenModes.js` imports them instead of copying them (#6590). |+| `imageQuotaSignal.js` | `parseQuotaDeferral(text, { now })` — read a provider quota exhaustion that states its own reset out of a failed render's error text, returning `{ resumeAt, resetsInMs }` or `null`. Requires BOTH an exhaustion marker and a parseable Go-style duration (`2h26m1s`), because the text embeds the agent's own prose about the page being rendered. Plus `parseGoDuration`, `formatDeferral`, and `MAX_QUOTA_DEFERRAL_MS`. Consumed by `services/mediaJobQueue` to hold queued renders for a cooling backend instead of burning them against the same wall. || `falImageModels.js` | Curated fal.ai image model catalog, a dependency-free leaf the client imports: `FAL_IMAGE_FAMILIES` (text + `/edit` endpoint per family, per-model reference cap, sizing alphabet, pricing), `FAL_IMAGE_DEFAULT_MODEL`, `FAL_IMAGE_MODEL_IDS` (the only ids a fal `cloudModel` may name), `falImageFamily`/`isFalImageModel`, `falImagePriceLabel`, `buildFalImageRequest` (routes to `/edit` when input images are present; maps width/height to aspect-ratio + resolution tier or pixel `image_size`) and `estimateFalImageCostUsd` (with `FAL_MEGAPIXEL`, fal.ai's binary megapixel). || `imageRuntimeRemedies.js` | Local image-runtime vocabulary, a dependency-free leaf beside `imageGenCapabilities.js`: `IMAGE_RUNTIME_READINESS` (ready/unavailable/unknown), `IMAGE_RUNTIME` + `IMAGE_RUNTIME_LABELS` (the shared torch venv vs. the configurable mflux interpreter), `IMAGE_RUNTIME_REMEDY` (the one action that fixes each unavailable state), and `IMAGE_RUNTIME_UNAVAILABLE_CODE`. `services/imageGen/localRuntime.js` builds the diagnosis from these, `services/imageGen/local.js` tags its pre-flight render refusal with the same remedy, and `client/src/lib/imageGenModes.js` re-exports them so no component hand-copies a remedy kind. || `renderModeLadder.js` | `resolveRenderPins` and `resolveRenderTargetPins` share backend-scoped model candidate inheritance across browser and server. Image-gen mode resolution ladder (#6815), a dependency-free leaf beside `generationModes.js`/`renderTargets.js` so the client imports it directly instead of hand-copying it: `isModeUsable(settings, mode)` (queueable + enabled gate, no per-provider spec lookup), `pickUsableMode(settings, candidates)` (first usable candidate, falling through disabled pins to the cloud-then-local tail), `renderTargetDefaults(settings, target)` (the normalized `settings.renderDefaults[target]` pin), and `imageModeCandidates(settings, target, record)` (the shared `[recordPin, targetPin, installDefault]` candidate order, declared once so a UI's own "what will Auto resolve to" display can't drift from the resolver). `services/imageGen/cloudProviderConfig.js` re-exports the first three for its existing callers. |diff --git a/server/lib/imageQuotaSignal.js b/server/lib/imageQuotaSignal.jsnew file mode 100644index 0000000000..00c0883a6b--- /dev/null+++ b/server/lib/imageQuotaSignal.js@@ -0,0 +1,104 @@+/**+ * Detect a provider quota exhaustion that states its own reset time, so the+ * render queue can wait for that instant instead of burning every queued job+ * against the same wall.+ *+ * The signal this reads is narration, not a structured field. Antigravity+ * relays `generate_image`'s failure through whatever prose the driving agent+ * chose that turn, so the same outage was recorded three different ways in one+ * afternoon ("The `generate_image` tool failed due to a rate-limit / quota+ * exhaustion error: - **Error:** `429 RESOURCE_EXHAUSTED: …`", a `>` blockquote+ * form, and a bare fenced form). Only two things were stable across all of+ * them: an exhaustion marker, and Google's Go-style duration+ * (`Your quota will reset after 2h26m1s.`).+ *+ * So the contract is deliberately conservative — BOTH an exhaustion marker and+ * a parseable duration must be present. A render prompt is attacker-adjacent+ * input here: the failure text embeds the agent's own words about a comic page,+ * and a page of dialogue could easily contain "quota" or "429" on its own.+ * Requiring the pair, and bounding the result, keeps a story about a rationing+ * office from parking the queue for a day.+ */++/** Longest deferral worth honoring; past this, failing is more honest. */+export const MAX_QUOTA_DEFERRAL_MS = 24 * 60 * 60 * 1000;++/** Below this a "reset" is noise — the queue would re-dispatch instantly. */+const MIN_QUOTA_DEFERRAL_MS = 1000;++// An explicit exhaustion marker. `429` alone is NOT enough: it is a plausible+// number in prose. Each alternative names the condition.+const EXHAUSTION_MARKERS = [+ /RESOURCE_EXHAUSTED/,+ /\bexhausted your capacity\b/i,+ /\bquota (?:reached|exceeded|exhausted)\b/i,+ /\b429\b[^\n]{0,40}\btoo many requests\b/i,+ /\brate[- ]?limit(?:ed|ing)?\b[^\n]{0,60}\bquota\b/i,+];++// Go-style duration as Google renders it: `2h26m1s`, `45m10s`, `30s`, `1h`.+// Anchored to reset wording so an unrelated duration in the prose (a comic+// panel's "3h20m later") cannot be mistaken for a quota instant.+const RESET_PHRASES = [+ /\bquota will reset (?:after|in)\s+([0-9hms]+)/i,+ /\bresets?\s+(?:after|in)\s+([0-9hms]+)/i,+];++const DURATION_PART = /(\d+)\s*([hms])/gi;++/**+ * Parse a Go-style compound duration into milliseconds.+ * Returns null when the string carries no recognizable unit.+ */+export function parseGoDuration(text) {+ if (typeof text !== 'string' || !text.trim()) return null;+ // Reject anything beyond digits and h/m/s so a bare number or a stray token+ // cannot be read as a duration.+ if (!/^(?:\d+\s*[hms])+$/i.test(text.trim())) return null;+ let total = 0;+ let matched = false;+ for (const [, value, unit] of text.matchAll(DURATION_PART)) {+ const n = Number(value);+ if (!Number.isFinite(n)) return null;+ matched = true;+ if (unit.toLowerCase() === 'h') total += n * 3_600_000;+ else if (unit.toLowerCase() === 'm') total += n * 60_000;+ else total += n * 1000;+ }+ return matched ? total : null;+}++/**+ * Read a deferral out of a failed render's error text.+ *+ * @param {string} text — the failure message the backend recorded.+ * @param {{now?: number}} [options]+ * @returns {{resumeAt: number, resetsInMs: number}|null} null when the text is+ * not a quota exhaustion, or states no usable reset. Null means "fail+ * normally" — never "retry at some guessed time".+ */+export function parseQuotaDeferral(text, { now = Date.now() } = {}) {+ if (typeof text !== 'string' || !text) return null;+ if (!EXHAUSTION_MARKERS.some((pattern) => pattern.test(text))) return null;+ for (const phrase of RESET_PHRASES) {+ const match = phrase.exec(text);+ if (!match) continue;+ const resetsInMs = parseGoDuration(match[1]);+ if (resetsInMs == null) continue;+ if (resetsInMs < MIN_QUOTA_DEFERRAL_MS || resetsInMs > MAX_QUOTA_DEFERRAL_MS) continue;+ return { resumeAt: now + resetsInMs, resetsInMs };+ }+ return null;+}++/** Compact "2h26m" for a status line; seconds only when that is all there is. */+export function formatDeferral(ms) {+ if (!Number.isFinite(ms) || ms <= 0) return 'now';+ const totalSeconds = Math.round(ms / 1000);+ const hours = Math.floor(totalSeconds / 3600);+ const minutes = Math.floor((totalSeconds % 3600) / 60);+ const seconds = totalSeconds % 60;+ if (hours) return minutes ? `${hours}h${minutes}m` : `${hours}h`;+ if (minutes) return `${minutes}m`;+ return `${seconds}s`;+}diff --git a/server/lib/imageQuotaSignal.test.js b/server/lib/imageQuotaSignal.test.jsnew file mode 100644index 0000000000..6d3d54b2e7--- /dev/null+++ b/server/lib/imageQuotaSignal.test.js@@ -0,0 +1,86 @@+import { describe, it, expect } from 'vitest';+import {+ parseQuotaDeferral, parseGoDuration, formatDeferral, MAX_QUOTA_DEFERRAL_MS,+} from './imageQuotaSignal.js';++// Verbatim shapes recorded by this install on 2026-10-17. Antigravity relays+// generate_image's failure through whatever prose the driving agent chose, so+// the same outage produced three different wrappings within one afternoon. They+// are fixtures precisely because the wrapping is NOT stable — a parser written+// against one of them silently stops working on the next.+const NARRATED = 'Agy did not produce an image at the directed path. Agy said: "I have delegated the image generation request to the `image-generator` subagent and am waiting for it to finish generating the comic page. The `generate_image` tool failed due to a rate-limit / quota exhaustion error: - **Error:** `429 RESOURCE_EXHAUSTED: "You have exhausted your capacity on this model. Your quota will reset after 2h26m1s."`"';+const BLOCKQUOTED = 'Agy did not produce an image at the directed path. Agy said: "Generating the comic page image via the `image-generator` subagent. The `generate_image` tool failed due to a rate-limit / quota exhaustion error: > `429 RESOURCE_EXHAUSTED: "You have exhausted your capacity on this model. Your quota will reset after 4h12m35s."`"';+const FENCED = 'Agy did not produce an image at the directed path. Agy said: "``` image failed to generate: failed to generate content: 429 Too Many Requests, body: {"error": {"code": 429, "message": "You have exhausted your capacity on this model. Your quota will reset after 2h29m35s.", "status": "RESOURCE_EXHAUSTED"}} ```"';++describe('parseQuotaDeferral', () => {+ it.each([+ ['narrated bullet form', NARRATED, 2 * 3600 + 26 * 60 + 1],+ ['blockquote form', BLOCKQUOTED, 4 * 3600 + 12 * 60 + 35],+ ['fenced JSON body form', FENCED, 2 * 3600 + 29 * 60 + 35],+ ])('reads the reset out of the %s', (_label, text, expectedSeconds) => {+ expect(parseQuotaDeferral(text, { now: 0 })).toEqual({+ resumeAt: expectedSeconds * 1000,+ resetsInMs: expectedSeconds * 1000,+ });+ });++ // The failure text embeds the agent's own words about the page being drawn,+ // so a comic script is effectively untrusted input to this parser. Parking+ // the whole render queue for hours because a character mentioned a quota+ // would be a worse bug than the one this fixes.+ it.each([+ ['prose mentioning a quota office and a duration', 'KITTA: "The quota office resets in 3h20m, they said." She did not believe it.'],+ ['an exhaustion marker with no stated reset', 'Agy said: "429 RESOURCE_EXHAUSTED: capacity gone." No reset was given.'],+ ['a reset with no exhaustion marker', 'The render pipeline resets in 45m for maintenance.'],+ ['a bare number where a duration belongs', 'RESOURCE_EXHAUSTED. Your quota will reset after 12.'],+ ['empty input', ''],+ ['a non-string', null],+ ])('returns null for %s', (_label, text) => {+ expect(parseQuotaDeferral(text, { now: 0 })).toBeNull();+ });++ it('refuses a deferral longer than the cap rather than parking for a day', () => {+ const text = 'RESOURCE_EXHAUSTED: Your quota will reset after 48h0m0s.';+ expect(parseQuotaDeferral(text, { now: 0 })).toBeNull();+ const withinCap = 'RESOURCE_EXHAUSTED: Your quota will reset after 23h0m0s.';+ expect(parseQuotaDeferral(withinCap, { now: 0 })?.resetsInMs).toBe(23 * 3600 * 1000);+ expect(MAX_QUOTA_DEFERRAL_MS).toBe(24 * 3600 * 1000);+ });++ it('refuses a sub-second reset that would re-dispatch into the same wall', () => {+ expect(parseQuotaDeferral('RESOURCE_EXHAUSTED: Your quota will reset after 0s.', { now: 0 })).toBeNull();+ });++ it('anchors resumeAt to the supplied clock', () => {+ const now = 1_700_000_000_000;+ expect(parseQuotaDeferral(NARRATED, { now }).resumeAt).toBe(now + (2 * 3600 + 26 * 60 + 1) * 1000);+ });+});++describe('parseGoDuration', () => {+ it.each([+ ['2h26m1s', 8_761_000],+ ['45m10s', 2_710_000],+ ['30s', 30_000],+ ['1h', 3_600_000],+ ])('parses %s', (text, expected) => {+ expect(parseGoDuration(text)).toBe(expected);+ });++ it.each(['12', 'soon', '', '2 hours', '2h30'])('rejects %s', (text) => {+ expect(parseGoDuration(text)).toBeNull();+ });+});++describe('formatDeferral', () => {+ it.each([+ [8_761_000, '2h26m'],+ [3_600_000, '1h'],+ [2_710_000, '45m'],+ [30_000, '30s'],+ [0, 'now'],+ [-5, 'now'],+ ])('renders %i as %s', (ms, expected) => {+ expect(formatDeferral(ms)).toBe(expected);+ });+});diff --git a/server/lib/index.js b/server/lib/index.jsindex cb31813d37..e56d5b09f4 100644--- a/server/lib/index.js+++ b/server/lib/index.js@@ -612,6 +612,7 @@ export * from './renderTargets.js';export * from './renderTiming.js';export * from './generationModes.js';export * from './imageGenCapabilities.js';+export * from './imageQuotaSignal.js';export * from './falImageModels.js';export * from './imageRuntimeRemedies.js';export * from './renderModeLadder.js';diff --git a/server/services/mediaJobQueue/index.js b/server/services/mediaJobQueue/index.jsindex f7a91546e6..a3aaf71b3e 100644--- a/server/services/mediaJobQueue/index.js+++ b/server/services/mediaJobQueue/index.js@@ -48,6 +48,7 @@ import { trainingEvents } from '../loraTraining/events.js';import { audioGenEvents } from '../audioGen/events.js';import { getSettings } from '../settings.js';import { IMAGE_GEN_MODE, CLOUD_IMAGE_GEN_MODES } from '../imageGen/modes.js';+import { parseQuotaDeferral, formatDeferral } from '../../lib/imageQuotaSignal.js';import { VIDEO_GEN_MODE, CLOUD_VIDEO_GEN_MODES, mediaJobExecutionLane } from '../../lib/generationModes.js';import { REMOTE_MEDIA_MODULES, isRemoteMediaJob } from './remoteMediaJob.js';import { createVideoHolds } from './videoHolds.js';@@ -640,6 +641,84 @@ function trimArchiveForSnapshot() {}// Capture the rows and eligible acknowledgements together, before storage yields.+// Per-backend quota cooldowns: mode → epoch ms the provider said it resets.+//+// A subscription image backend answers an exhausted quota with the instant it+// recovers ("Your quota will reset after 2h26m1s"). Without this, every job+// queued behind the one that hit the wall dispatches straight into the same+// 429 and fails within seconds — on 2026-10-17 a single comic run burned ten+// renders that way, each one a failed slot the user had to notice and re-drive+// by hand.+//+// Keyed by MODE, not by job: the quota belongs to the provider account, so one+// job's rejection is evidence about every queued job routed to that backend.+// Queue-level rather than per-job because `captureQueueSnapshot` serializes a+// fixed field whitelist, so a new per-job field would silently not survive a+// restart; this rides alongside `videoHolds` in the same snapshot instead.+const quotaCooldowns = new Map();++/** Remaining cooldown for a job's backend, 0 when it is clear to dispatch. */+function quotaCooldownRemaining(job, now = Date.now()) {+ const mode = job?.params?.mode;+ if (!mode) return 0;+ const resumeAt = quotaCooldowns.get(mode);+ if (!resumeAt) return 0;+ if (resumeAt <= now) {+ // Expired — drop it so a later failure starts a fresh window rather than+ // extending a stale one, and so the map cannot grow without bound.+ quotaCooldowns.delete(mode);+ return 0;+ }+ return resumeAt - now;+}++/**+ * Record a cooldown when a failure carries the provider's own reset time.+ * Called from the failed handler for its EVIDENCE only — the job still fails+ * through the normal terminal path, so no lifecycle invariant changes here.+ */+function recordQuotaCooldown(job, errorText) {+ const mode = job?.params?.mode;+ if (!mode) return;+ const deferral = parseQuotaDeferral(errorText);+ if (!deferral) return;+ const existing = quotaCooldowns.get(mode) || 0;+ // Keep the later instant: a second rejection during the same outage reports a+ // shorter remaining wait, and taking it would release the queue early.+ if (deferral.resumeAt <= existing) return;+ quotaCooldowns.set(mode, deferral.resumeAt);+ console.warn(`⏳ media-job: ${mode} quota exhausted — holding queued ${mode} renders for ${formatDeferral(deferral.resetsInMs)}`);+}++/** Snapshot shape: [[mode, resumeAt], ...], dropping anything already expired. */+function snapshotQuotaCooldowns(now = Date.now()) {+ return [...quotaCooldowns.entries()].filter(([, resumeAt]) => resumeAt > now);+}++/**+ * Restore cooldowns across a restart. Returns false on a malformed snapshot so+ * the caller can latch persistence the same way a malformed hold does — but+ * never throws: an unreadable cooldown must not keep the server from booting.+ */+function restoreQuotaCooldowns(raw, now = Date.now()) {+ quotaCooldowns.clear();+ if (raw === undefined || raw === null) return true;+ if (!Array.isArray(raw)) return false;+ let ok = true;+ for (const entry of raw) {+ if (!Array.isArray(entry) || entry.length !== 2) { ok = false; continue; }+ const [mode, resumeAt] = entry;+ if (typeof mode !== 'string' || !mode || !Number.isFinite(resumeAt)) { ok = false; continue; }+ if (resumeAt > now) quotaCooldowns.set(mode, resumeAt);+ }+ return ok;+}++/** Test-only reset so a suite cannot leak a cooldown into the next case. */+export function __resetQuotaCooldownsForTest() {+ quotaCooldowns.clear();+}+function captureQueueSnapshot() {const live = [...(running ? [running] : []),@@ -661,7 +740,7 @@ function captureQueueSnapshot() {({ id, kind, owner, status, queuedAt, startedAt, completedAt, params, result, error, position, progress, statusMsg, etaMs }),);const transitions = [...pendingTransitions].filter(([job]) => canWriteOutcome(job));- return { data: { jobs: serializable, videoHolds: videoHolds.snapshot() }, transitions, hasDeferredCompletion };+ return { data: { jobs: serializable, videoHolds: videoHolds.snapshot(), quotaCooldowns: snapshotQuotaCooldowns() }, transitions, hasDeferredCompletion };}async function acknowledgeWrittenTransitions(transitions) {@@ -713,6 +792,12 @@ export async function initMediaJobQueue() {// indication that repairing or deleting one file restores it.console.error(`❌ media-job queue: ${JOBS_FILE} is present but unreadable — starting empty and NOT persisting so it is preserved; repair or delete it and restart to re-enable persistence`);}+ if (!restoreQuotaCooldowns(data?.quotaCooldowns)) {+ // Same latch as a malformed hold: preserve the file rather than+ // overwriting it, but boot normally with no cooldown in force.+ persistBlocked = true;+ console.error('❌ media-job queue: invalid quota cooldowns — snapshot preserved and cooldowns cleared; repair the file and restart');+ }const holdsReadable = videoHolds.restore(data?.videoHolds);if (!holdsReadable) {// A malformed hold must neither kill server boot nor silently release a@@ -1074,6 +1159,19 @@ async function drainLoop() {};for (const job of candidates) {if (job.status !== 'queued' || job.hold || job.admitting) continue;+ // The provider told us when it recovers; dispatching before then just+ // converts a queued job into a failed one. Hold rather than fail, and say+ // when it resumes — `statusMsg` is in the persisted field whitelist, so+ // the Render Queue shows the wait across a restart too.+ const cooling = quotaCooldownRemaining(job);+ if (cooling > 0) {+ const waiting = `Waiting for ${job.params.mode} quota — resumes in ${formatDeferral(cooling)}`;+ if (job.statusMsg !== waiting) {+ job.statusMsg = waiting;+ persist().catch((e) => console.error(`❌ media-job [${job.id.slice(0, 8)}] mediaJobQueue persist on quota hold failed: ${e.message}`));+ }+ continue;+ }const lane = jobLane(job);if (slots[lane] <= 0) continue;// One GPU job per peer initially. Different peers and this Mac remain@@ -1485,6 +1583,10 @@ async function runJobLifecycle(job, markDispatched, permit) {trackTerminalOperation(terminate('failed', (j) => {j.error = payload.error || 'unknown error';+ // Evidence only — this job still fails through the normal terminal+ // path. What it buys is every job queued BEHIND it: they hold for the+ // provider's stated reset instead of dispatching into the same 429.+ recordQuotaCooldown(j, j.error);videoHolds.captureFailure(j, j.error, payload);}, payload.remoteInputsDisposable === true).catch((e) => console.error(`❌ media-job [${job.id.slice(0, 8)}] terminal handler failed: ${e.message}`)),diff --git a/server/services/mediaJobQueue/index.test.js b/server/services/mediaJobQueue/index.test.jsindex eee2c5b000..620576be9d 100644--- a/server/services/mediaJobQueue/index.test.js+++ b/server/services/mediaJobQueue/index.test.js@@ -3877,3 +3877,78 @@ describe('mediaJobQueue per-kind runtime wiring', () => {}});});++describe('provider quota cooldowns', () => {+ // Verbatim from this install on 2026-10-17 (redacted of nothing — it carries+ // no private state). The duration is the only machine-readable part.+ const QUOTA_ERROR = 'Agy did not produce an image at the directed path. Agy said: "The `generate_image` tool failed due to a rate-limit / quota exhaustion error: `429 RESOURCE_EXHAUSTED: "You have exhausted your capacity on this model. Your quota will reset after 2h26m1s."`"';++ // No local beforeEach: the file-level one already gives each test a fresh+ // temp data dir and a fresh module instance, so the cooldown map starts+ // empty. Re-importing here would reset modules mid-setup.++ it('holds a later render for the same backend instead of dispatching it into the same 429', async () => {+ stubs.generateImageCodex.mockImplementation(() => new Promise(() => {}));+ const first = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 1' } });+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 1);++ imageGenEvents.emit('failed', { generationId: first.jobId, error: QUOTA_ERROR });+ await waitFor(() => mediaJobQueue.getJob(first.jobId)?.status === 'failed');++ const second = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 2' } });+ await waitFor(() => (mediaJobQueue.getJob(second.jobId)?.statusMsg || '').startsWith('Waiting for codex quota'));++ // The point of the whole change: it is still queued, not burned.+ expect(mediaJobQueue.getJob(second.jobId).status).toBe('queued');+ expect(stubs.generateImageCodex).toHaveBeenCalledTimes(1);+ expect(mediaJobQueue.getJob(second.jobId).statusMsg).toMatch(/resumes in 2h2[56]m/);+ });++ it('leaves a different backend free to run during another backend\'s cooldown', async () => {+ stubs.generateImageCodex.mockImplementation(() => new Promise(() => {}));+ const first = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 1' } });+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 1);+ imageGenEvents.emit('failed', { generationId: first.jobId, error: QUOTA_ERROR });+ await waitFor(() => mediaJobQueue.getJob(first.jobId)?.status === 'failed');++ stubs.generateImage.mockImplementation(() => new Promise(() => {}));+ const local = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'local', prompt: 'page 2' } });+ await waitFor(() => stubs.generateImage.mock.calls.length === 1);+ expect(mediaJobQueue.getJob(local.jobId).status).toBe('running');+ });++ it('does not hold anything for an ordinary failure that states no reset', async () => {+ // Resolve rather than hang: the lane is released by the gen call settling,+ // so a hanging stub would keep the cloud slot and mask the assertion.+ stubs.generateImageCodex.mockResolvedValue({ jobId: 'codex-job' });+ const first = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 1' } });+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 1);+ imageGenEvents.emit('failed', { generationId: first.jobId, error: 'Agy generation failed: Exit code 1' });+ await waitFor(() => mediaJobQueue.getJob(first.jobId)?.status === 'failed');+ await flush();++ const second = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 2' } });+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 2);+ expect(mediaJobQueue.getJob(second.jobId).status).toBe('running');+ });++ it('keeps the cooldown across a restart so a reboot does not release the queue early', async () => {+ stubs.generateImageCodex.mockResolvedValue({ jobId: 'codex-job' });+ const first = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 1' } });+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 1);+ imageGenEvents.emit('failed', { generationId: first.jobId, error: QUOTA_ERROR });+ await waitFor(() => mediaJobQueue.getJob(first.jobId)?.status === 'failed');+ await flush();++ const persisted = JSON.parse(readFileSync(join(tempDataDir, 'media-jobs.json'), 'utf8'));+ expect(persisted.quotaCooldowns).toEqual([['codex', expect.any(Number)]]);++ stubs.generateImageCodex.mockClear();+ await importFresh();+ // The restart itself: init is what re-reads the snapshot.+ await mediaJobQueue.initMediaJobQueue();+ const second = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 2' } });+ await waitFor(() => (mediaJobQueue.getJob(second.jobId)?.statusMsg || '').startsWith('Waiting for codex quota'));+ expect(stubs.generateImageCodex).not.toHaveBeenCalled();+ });+});