Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

feat: hold queued renders for a backend whose quota says when it resets

Stephen Hilderbrandcea48cde9bd3
diff --git a/server/lib/README.md b/server/lib/README.md
index 3120005b55..de041e1457 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -46,6 +46,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `generationModes.js` | Shared immutable image/video render-backend alphabets (`IMAGE_GEN_MODE[S]`, `QUEUEABLE_IMAGE_MODES`, `VIDEO_GEN_MODE[S]`, and cloud subsets) plus `mediaJobExecutionLane({ kind, mode, remote })` / `MEDIA_JOB_EXECUTION_LANES` — THE media-queue lane rule the scheduler and the public job projection share, kept below both validation and generation services so schemas never import orchestration modules. |
| `mediaJobPreparation.js` | `createMediaJobPreparation()` tracks per-job cancellation before cloud providers install a render handle; `run`, `cancel`, `take`, and `ids` cover preparation, handoff, and bulk cancellation. |
| `imageGenCapabilities.js` | Per-backend image-gen capability literals, a dependency-free leaf beside `generationModes.js`: `EDIT_INCAPABLE_IMAGE_MODES`/`isEditCapableMode` and the derived `I2I_CAPABLE_MODES`, the shipped defaults (`CODEX_IMAGEGEN_DEFAULT_MODEL`/`_EFFORT`, `AGY_IMAGEGEN_DEFAULT_MODEL`, `AGY_IMAGEGEN_IMAGE_MODEL`), the `AGY_`/`GROK_ASPECT_RATIOS` alphabets, and `IMAGE_GEN_PROVIDER_CAPABILITIES` (`supportsModelOverride`, `maxInputImages`, `promptRequiredWithInputImage`) with its `supportsCloudModelOverride`/`maxInputImages`/`cloudPromptRequired` readers. The `services/imageGen/*` modules re-export these, and `client/src/lib/imageGenModes.js` imports them instead of copying them (#6590). |
+| `imageQuotaSignal.js` | `parseQuotaDeferral(text, { now })` — read a provider quota exhaustion that states its own reset out of a failed render's error text, returning `{ resumeAt, resetsInMs }` or `null`. Requires BOTH an exhaustion marker and a parseable Go-style duration (`2h26m1s`), because the text embeds the agent's own prose about the page being rendered. Plus `parseGoDuration`, `formatDeferral`, and `MAX_QUOTA_DEFERRAL_MS`. Consumed by `services/mediaJobQueue` to hold queued renders for a cooling backend instead of burning them against the same wall. |
| `falImageModels.js` | Curated fal.ai image model catalog, a dependency-free leaf the client imports: `FAL_IMAGE_FAMILIES` (text + `/edit` endpoint per family, per-model reference cap, sizing alphabet, pricing), `FAL_IMAGE_DEFAULT_MODEL`, `FAL_IMAGE_MODEL_IDS` (the only ids a fal `cloudModel` may name), `falImageFamily`/`isFalImageModel`, `falImagePriceLabel`, `buildFalImageRequest` (routes to `/edit` when input images are present; maps width/height to aspect-ratio + resolution tier or pixel `image_size`) and `estimateFalImageCostUsd` (with `FAL_MEGAPIXEL`, fal.ai's binary megapixel). |
| `imageRuntimeRemedies.js` | Local image-runtime vocabulary, a dependency-free leaf beside `imageGenCapabilities.js`: `IMAGE_RUNTIME_READINESS` (ready/unavailable/unknown), `IMAGE_RUNTIME` + `IMAGE_RUNTIME_LABELS` (the shared torch venv vs. the configurable mflux interpreter), `IMAGE_RUNTIME_REMEDY` (the one action that fixes each unavailable state), and `IMAGE_RUNTIME_UNAVAILABLE_CODE`. `services/imageGen/localRuntime.js` builds the diagnosis from these, `services/imageGen/local.js` tags its pre-flight render refusal with the same remedy, and `client/src/lib/imageGenModes.js` re-exports them so no component hand-copies a remedy kind. |
| `renderModeLadder.js` | `resolveRenderPins` and `resolveRenderTargetPins` share backend-scoped model candidate inheritance across browser and server. Image-gen mode resolution ladder (#6815), a dependency-free leaf beside `generationModes.js`/`renderTargets.js` so the client imports it directly instead of hand-copying it: `isModeUsable(settings, mode)` (queueable + enabled gate, no per-provider spec lookup), `pickUsableMode(settings, candidates)` (first usable candidate, falling through disabled pins to the cloud-then-local tail), `renderTargetDefaults(settings, target)` (the normalized `settings.renderDefaults[target]` pin), and `imageModeCandidates(settings, target, record)` (the shared `[recordPin, targetPin, installDefault]` candidate order, declared once so a UI's own "what will Auto resolve to" display can't drift from the resolver). `services/imageGen/cloudProviderConfig.js` re-exports the first three for its existing callers. |
diff --git a/server/lib/imageQuotaSignal.js b/server/lib/imageQuotaSignal.js
new file mode 100644
index 0000000000..00c0883a6b
--- /dev/null
+++ b/server/lib/imageQuotaSignal.js
@@ -0,0 +1,104 @@
+/**
+ * Detect a provider quota exhaustion that states its own reset time, so the
+ * render queue can wait for that instant instead of burning every queued job
+ * against the same wall.
+ *
+ * The signal this reads is narration, not a structured field. Antigravity
+ * relays `generate_image`'s failure through whatever prose the driving agent
+ * chose that turn, so the same outage was recorded three different ways in one
+ * afternoon ("The `generate_image` tool failed due to a rate-limit / quota
+ * exhaustion error: - **Error:** `429 RESOURCE_EXHAUSTED: …`", a `>` blockquote
+ * form, and a bare fenced form). Only two things were stable across all of
+ * them: an exhaustion marker, and Google's Go-style duration
+ * (`Your quota will reset after 2h26m1s.`).
+ *
+ * So the contract is deliberately conservative — BOTH an exhaustion marker and
+ * a parseable duration must be present. A render prompt is attacker-adjacent
+ * input here: the failure text embeds the agent's own words about a comic page,
+ * and a page of dialogue could easily contain "quota" or "429" on its own.
+ * Requiring the pair, and bounding the result, keeps a story about a rationing
+ * office from parking the queue for a day.
+ */
+
+/** Longest deferral worth honoring; past this, failing is more honest. */
+export const MAX_QUOTA_DEFERRAL_MS = 24 * 60 * 60 * 1000;
+
+/** Below this a "reset" is noise — the queue would re-dispatch instantly. */
+const MIN_QUOTA_DEFERRAL_MS = 1000;
+
+// An explicit exhaustion marker. `429` alone is NOT enough: it is a plausible
+// number in prose. Each alternative names the condition.
+const EXHAUSTION_MARKERS = [
+ /RESOURCE_EXHAUSTED/,
+ /\bexhausted your capacity\b/i,
+ /\bquota (?:reached|exceeded|exhausted)\b/i,
+ /\b429\b[^\n]{0,40}\btoo many requests\b/i,
+ /\brate[- ]?limit(?:ed|ing)?\b[^\n]{0,60}\bquota\b/i,
+];
+
+// Go-style duration as Google renders it: `2h26m1s`, `45m10s`, `30s`, `1h`.
+// Anchored to reset wording so an unrelated duration in the prose (a comic
+// panel's "3h20m later") cannot be mistaken for a quota instant.
+const RESET_PHRASES = [
+ /\bquota will reset (?:after|in)\s+([0-9hms]+)/i,
+ /\bresets?\s+(?:after|in)\s+([0-9hms]+)/i,
+];
+
+const DURATION_PART = /(\d+)\s*([hms])/gi;
+
+/**
+ * Parse a Go-style compound duration into milliseconds.
+ * Returns null when the string carries no recognizable unit.
+ */
+export function parseGoDuration(text) {
+ if (typeof text !== 'string' || !text.trim()) return null;
+ // Reject anything beyond digits and h/m/s so a bare number or a stray token
+ // cannot be read as a duration.
+ if (!/^(?:\d+\s*[hms])+$/i.test(text.trim())) return null;
+ let total = 0;
+ let matched = false;
+ for (const [, value, unit] of text.matchAll(DURATION_PART)) {
+ const n = Number(value);
+ if (!Number.isFinite(n)) return null;
+ matched = true;
+ if (unit.toLowerCase() === 'h') total += n * 3_600_000;
+ else if (unit.toLowerCase() === 'm') total += n * 60_000;
+ else total += n * 1000;
+ }
+ return matched ? total : null;
+}
+
+/**
+ * Read a deferral out of a failed render's error text.
+ *
+ * @param {string} text — the failure message the backend recorded.
+ * @param {{now?: number}} [options]
+ * @returns {{resumeAt: number, resetsInMs: number}|null} null when the text is
+ * not a quota exhaustion, or states no usable reset. Null means "fail
+ * normally" — never "retry at some guessed time".
+ */
+export function parseQuotaDeferral(text, { now = Date.now() } = {}) {
+ if (typeof text !== 'string' || !text) return null;
+ if (!EXHAUSTION_MARKERS.some((pattern) => pattern.test(text))) return null;
+ for (const phrase of RESET_PHRASES) {
+ const match = phrase.exec(text);
+ if (!match) continue;
+ const resetsInMs = parseGoDuration(match[1]);
+ if (resetsInMs == null) continue;
+ if (resetsInMs < MIN_QUOTA_DEFERRAL_MS || resetsInMs > MAX_QUOTA_DEFERRAL_MS) continue;
+ return { resumeAt: now + resetsInMs, resetsInMs };
+ }
+ return null;
+}
+
+/** Compact "2h26m" for a status line; seconds only when that is all there is. */
+export function formatDeferral(ms) {
+ if (!Number.isFinite(ms) || ms <= 0) return 'now';
+ const totalSeconds = Math.round(ms / 1000);
+ const hours = Math.floor(totalSeconds / 3600);
+ const minutes = Math.floor((totalSeconds % 3600) / 60);
+ const seconds = totalSeconds % 60;
+ if (hours) return minutes ? `${hours}h${minutes}m` : `${hours}h`;
+ if (minutes) return `${minutes}m`;
+ return `${seconds}s`;
+}
diff --git a/server/lib/imageQuotaSignal.test.js b/server/lib/imageQuotaSignal.test.js
new file mode 100644
index 0000000000..6d3d54b2e7
--- /dev/null
+++ b/server/lib/imageQuotaSignal.test.js
@@ -0,0 +1,86 @@
+import { describe, it, expect } from 'vitest';
+import {
+ parseQuotaDeferral, parseGoDuration, formatDeferral, MAX_QUOTA_DEFERRAL_MS,
+} from './imageQuotaSignal.js';
+
+// Verbatim shapes recorded by this install on 2026-10-17. Antigravity relays
+// generate_image's failure through whatever prose the driving agent chose, so
+// the same outage produced three different wrappings within one afternoon. They
+// are fixtures precisely because the wrapping is NOT stable — a parser written
+// against one of them silently stops working on the next.
+const NARRATED = 'Agy did not produce an image at the directed path. Agy said: "I have delegated the image generation request to the `image-generator` subagent and am waiting for it to finish generating the comic page. The `generate_image` tool failed due to a rate-limit / quota exhaustion error: - **Error:** `429 RESOURCE_EXHAUSTED: "You have exhausted your capacity on this model. Your quota will reset after 2h26m1s."`"';
+const BLOCKQUOTED = 'Agy did not produce an image at the directed path. Agy said: "Generating the comic page image via the `image-generator` subagent. The `generate_image` tool failed due to a rate-limit / quota exhaustion error: > `429 RESOURCE_EXHAUSTED: "You have exhausted your capacity on this model. Your quota will reset after 4h12m35s."`"';
+const FENCED = 'Agy did not produce an image at the directed path. Agy said: "``` image failed to generate: failed to generate content: 429 Too Many Requests, body: {"error": {"code": 429, "message": "You have exhausted your capacity on this model. Your quota will reset after 2h29m35s.", "status": "RESOURCE_EXHAUSTED"}} ```"';
+
+describe('parseQuotaDeferral', () => {
+ it.each([
+ ['narrated bullet form', NARRATED, 2 * 3600 + 26 * 60 + 1],
+ ['blockquote form', BLOCKQUOTED, 4 * 3600 + 12 * 60 + 35],
+ ['fenced JSON body form', FENCED, 2 * 3600 + 29 * 60 + 35],
+ ])('reads the reset out of the %s', (_label, text, expectedSeconds) => {
+ expect(parseQuotaDeferral(text, { now: 0 })).toEqual({
+ resumeAt: expectedSeconds * 1000,
+ resetsInMs: expectedSeconds * 1000,
+ });
+ });
+
+ // The failure text embeds the agent's own words about the page being drawn,
+ // so a comic script is effectively untrusted input to this parser. Parking
+ // the whole render queue for hours because a character mentioned a quota
+ // would be a worse bug than the one this fixes.
+ it.each([
+ ['prose mentioning a quota office and a duration', 'KITTA: "The quota office resets in 3h20m, they said." She did not believe it.'],
+ ['an exhaustion marker with no stated reset', 'Agy said: "429 RESOURCE_EXHAUSTED: capacity gone." No reset was given.'],
+ ['a reset with no exhaustion marker', 'The render pipeline resets in 45m for maintenance.'],
+ ['a bare number where a duration belongs', 'RESOURCE_EXHAUSTED. Your quota will reset after 12.'],
+ ['empty input', ''],
+ ['a non-string', null],
+ ])('returns null for %s', (_label, text) => {
+ expect(parseQuotaDeferral(text, { now: 0 })).toBeNull();
+ });
+
+ it('refuses a deferral longer than the cap rather than parking for a day', () => {
+ const text = 'RESOURCE_EXHAUSTED: Your quota will reset after 48h0m0s.';
+ expect(parseQuotaDeferral(text, { now: 0 })).toBeNull();
+ const withinCap = 'RESOURCE_EXHAUSTED: Your quota will reset after 23h0m0s.';
+ expect(parseQuotaDeferral(withinCap, { now: 0 })?.resetsInMs).toBe(23 * 3600 * 1000);
+ expect(MAX_QUOTA_DEFERRAL_MS).toBe(24 * 3600 * 1000);
+ });
+
+ it('refuses a sub-second reset that would re-dispatch into the same wall', () => {
+ expect(parseQuotaDeferral('RESOURCE_EXHAUSTED: Your quota will reset after 0s.', { now: 0 })).toBeNull();
+ });
+
+ it('anchors resumeAt to the supplied clock', () => {
+ const now = 1_700_000_000_000;
+ expect(parseQuotaDeferral(NARRATED, { now }).resumeAt).toBe(now + (2 * 3600 + 26 * 60 + 1) * 1000);
+ });
+});
+
+describe('parseGoDuration', () => {
+ it.each([
+ ['2h26m1s', 8_761_000],
+ ['45m10s', 2_710_000],
+ ['30s', 30_000],
+ ['1h', 3_600_000],
+ ])('parses %s', (text, expected) => {
+ expect(parseGoDuration(text)).toBe(expected);
+ });
+
+ it.each(['12', 'soon', '', '2 hours', '2h30'])('rejects %s', (text) => {
+ expect(parseGoDuration(text)).toBeNull();
+ });
+});
+
+describe('formatDeferral', () => {
+ it.each([
+ [8_761_000, '2h26m'],
+ [3_600_000, '1h'],
+ [2_710_000, '45m'],
+ [30_000, '30s'],
+ [0, 'now'],
+ [-5, 'now'],
+ ])('renders %i as %s', (ms, expected) => {
+ expect(formatDeferral(ms)).toBe(expected);
+ });
+});
diff --git a/server/lib/index.js b/server/lib/index.js
index cb31813d37..e56d5b09f4 100644
--- a/server/lib/index.js
+++ b/server/lib/index.js
@@ -612,6 +612,7 @@ export * from './renderTargets.js';
export * from './renderTiming.js';
export * from './generationModes.js';
export * from './imageGenCapabilities.js';
+export * from './imageQuotaSignal.js';
export * from './falImageModels.js';
export * from './imageRuntimeRemedies.js';
export * from './renderModeLadder.js';
diff --git a/server/services/mediaJobQueue/index.js b/server/services/mediaJobQueue/index.js
index f7a91546e6..a3aaf71b3e 100644
--- a/server/services/mediaJobQueue/index.js
+++ b/server/services/mediaJobQueue/index.js
@@ -48,6 +48,7 @@ import { trainingEvents } from '../loraTraining/events.js';
import { audioGenEvents } from '../audioGen/events.js';
import { getSettings } from '../settings.js';
import { IMAGE_GEN_MODE, CLOUD_IMAGE_GEN_MODES } from '../imageGen/modes.js';
+import { parseQuotaDeferral, formatDeferral } from '../../lib/imageQuotaSignal.js';
import { VIDEO_GEN_MODE, CLOUD_VIDEO_GEN_MODES, mediaJobExecutionLane } from '../../lib/generationModes.js';
import { REMOTE_MEDIA_MODULES, isRemoteMediaJob } from './remoteMediaJob.js';
import { createVideoHolds } from './videoHolds.js';
@@ -640,6 +641,84 @@ function trimArchiveForSnapshot() {
}
// Capture the rows and eligible acknowledgements together, before storage yields.
+// Per-backend quota cooldowns: mode → epoch ms the provider said it resets.
+//
+// A subscription image backend answers an exhausted quota with the instant it
+// recovers ("Your quota will reset after 2h26m1s"). Without this, every job
+// queued behind the one that hit the wall dispatches straight into the same
+// 429 and fails within seconds — on 2026-10-17 a single comic run burned ten
+// renders that way, each one a failed slot the user had to notice and re-drive
+// by hand.
+//
+// Keyed by MODE, not by job: the quota belongs to the provider account, so one
+// job's rejection is evidence about every queued job routed to that backend.
+// Queue-level rather than per-job because `captureQueueSnapshot` serializes a
+// fixed field whitelist, so a new per-job field would silently not survive a
+// restart; this rides alongside `videoHolds` in the same snapshot instead.
+const quotaCooldowns = new Map();
+
+/** Remaining cooldown for a job's backend, 0 when it is clear to dispatch. */
+function quotaCooldownRemaining(job, now = Date.now()) {
+ const mode = job?.params?.mode;
+ if (!mode) return 0;
+ const resumeAt = quotaCooldowns.get(mode);
+ if (!resumeAt) return 0;
+ if (resumeAt <= now) {
+ // Expired — drop it so a later failure starts a fresh window rather than
+ // extending a stale one, and so the map cannot grow without bound.
+ quotaCooldowns.delete(mode);
+ return 0;
+ }
+ return resumeAt - now;
+}
+
+/**
+ * Record a cooldown when a failure carries the provider's own reset time.
+ * Called from the failed handler for its EVIDENCE only — the job still fails
+ * through the normal terminal path, so no lifecycle invariant changes here.
+ */
+function recordQuotaCooldown(job, errorText) {
+ const mode = job?.params?.mode;
+ if (!mode) return;
+ const deferral = parseQuotaDeferral(errorText);
+ if (!deferral) return;
+ const existing = quotaCooldowns.get(mode) || 0;
+ // Keep the later instant: a second rejection during the same outage reports a
+ // shorter remaining wait, and taking it would release the queue early.
+ if (deferral.resumeAt <= existing) return;
+ quotaCooldowns.set(mode, deferral.resumeAt);
+ console.warn(`⏳ media-job: ${mode} quota exhausted — holding queued ${mode} renders for ${formatDeferral(deferral.resetsInMs)}`);
+}
+
+/** Snapshot shape: [[mode, resumeAt], ...], dropping anything already expired. */
+function snapshotQuotaCooldowns(now = Date.now()) {
+ return [...quotaCooldowns.entries()].filter(([, resumeAt]) => resumeAt > now);
+}
+
+/**
+ * Restore cooldowns across a restart. Returns false on a malformed snapshot so
+ * the caller can latch persistence the same way a malformed hold does — but
+ * never throws: an unreadable cooldown must not keep the server from booting.
+ */
+function restoreQuotaCooldowns(raw, now = Date.now()) {
+ quotaCooldowns.clear();
+ if (raw === undefined || raw === null) return true;
+ if (!Array.isArray(raw)) return false;
+ let ok = true;
+ for (const entry of raw) {
+ if (!Array.isArray(entry) || entry.length !== 2) { ok = false; continue; }
+ const [mode, resumeAt] = entry;
+ if (typeof mode !== 'string' || !mode || !Number.isFinite(resumeAt)) { ok = false; continue; }
+ if (resumeAt > now) quotaCooldowns.set(mode, resumeAt);
+ }
+ return ok;
+}
+
+/** Test-only reset so a suite cannot leak a cooldown into the next case. */
+export function __resetQuotaCooldownsForTest() {
+ quotaCooldowns.clear();
+}
+
function captureQueueSnapshot() {
const live = [
...(running ? [running] : []),
@@ -661,7 +740,7 @@ function captureQueueSnapshot() {
({ id, kind, owner, status, queuedAt, startedAt, completedAt, params, result, error, position, progress, statusMsg, etaMs }),
);
const transitions = [...pendingTransitions].filter(([job]) => canWriteOutcome(job));
- return { data: { jobs: serializable, videoHolds: videoHolds.snapshot() }, transitions, hasDeferredCompletion };
+ return { data: { jobs: serializable, videoHolds: videoHolds.snapshot(), quotaCooldowns: snapshotQuotaCooldowns() }, transitions, hasDeferredCompletion };
}
async function acknowledgeWrittenTransitions(transitions) {
@@ -713,6 +792,12 @@ export async function initMediaJobQueue() {
// indication that repairing or deleting one file restores it.
console.error(`❌ media-job queue: ${JOBS_FILE} is present but unreadable — starting empty and NOT persisting so it is preserved; repair or delete it and restart to re-enable persistence`);
}
+ if (!restoreQuotaCooldowns(data?.quotaCooldowns)) {
+ // Same latch as a malformed hold: preserve the file rather than
+ // overwriting it, but boot normally with no cooldown in force.
+ persistBlocked = true;
+ console.error('❌ media-job queue: invalid quota cooldowns — snapshot preserved and cooldowns cleared; repair the file and restart');
+ }
const holdsReadable = videoHolds.restore(data?.videoHolds);
if (!holdsReadable) {
// A malformed hold must neither kill server boot nor silently release a
@@ -1074,6 +1159,19 @@ async function drainLoop() {
};
for (const job of candidates) {
if (job.status !== 'queued' || job.hold || job.admitting) continue;
+ // The provider told us when it recovers; dispatching before then just
+ // converts a queued job into a failed one. Hold rather than fail, and say
+ // when it resumes — `statusMsg` is in the persisted field whitelist, so
+ // the Render Queue shows the wait across a restart too.
+ const cooling = quotaCooldownRemaining(job);
+ if (cooling > 0) {
+ const waiting = `Waiting for ${job.params.mode} quota — resumes in ${formatDeferral(cooling)}`;
+ if (job.statusMsg !== waiting) {
+ job.statusMsg = waiting;
+ persist().catch((e) => console.error(`❌ media-job [${job.id.slice(0, 8)}] mediaJobQueue persist on quota hold failed: ${e.message}`));
+ }
+ continue;
+ }
const lane = jobLane(job);
if (slots[lane] <= 0) continue;
// One GPU job per peer initially. Different peers and this Mac remain
@@ -1485,6 +1583,10 @@ async function runJobLifecycle(job, markDispatched, permit) {
trackTerminalOperation(
terminate('failed', (j) => {
j.error = payload.error || 'unknown error';
+ // Evidence only — this job still fails through the normal terminal
+ // path. What it buys is every job queued BEHIND it: they hold for the
+ // provider's stated reset instead of dispatching into the same 429.
+ recordQuotaCooldown(j, j.error);
videoHolds.captureFailure(j, j.error, payload);
}, payload.remoteInputsDisposable === true)
.catch((e) => console.error(`❌ media-job [${job.id.slice(0, 8)}] terminal handler failed: ${e.message}`)),
diff --git a/server/services/mediaJobQueue/index.test.js b/server/services/mediaJobQueue/index.test.js
index eee2c5b000..620576be9d 100644
--- a/server/services/mediaJobQueue/index.test.js
+++ b/server/services/mediaJobQueue/index.test.js
@@ -3877,3 +3877,78 @@ describe('mediaJobQueue per-kind runtime wiring', () => {
}
});
});
+
+describe('provider quota cooldowns', () => {
+ // Verbatim from this install on 2026-10-17 (redacted of nothing — it carries
+ // no private state). The duration is the only machine-readable part.
+ const QUOTA_ERROR = 'Agy did not produce an image at the directed path. Agy said: "The `generate_image` tool failed due to a rate-limit / quota exhaustion error: `429 RESOURCE_EXHAUSTED: "You have exhausted your capacity on this model. Your quota will reset after 2h26m1s."`"';
+
+ // No local beforeEach: the file-level one already gives each test a fresh
+ // temp data dir and a fresh module instance, so the cooldown map starts
+ // empty. Re-importing here would reset modules mid-setup.
+
+ it('holds a later render for the same backend instead of dispatching it into the same 429', async () => {
+ stubs.generateImageCodex.mockImplementation(() => new Promise(() => {}));
+ const first = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 1' } });
+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 1);
+
+ imageGenEvents.emit('failed', { generationId: first.jobId, error: QUOTA_ERROR });
+ await waitFor(() => mediaJobQueue.getJob(first.jobId)?.status === 'failed');
+
+ const second = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 2' } });
+ await waitFor(() => (mediaJobQueue.getJob(second.jobId)?.statusMsg || '').startsWith('Waiting for codex quota'));
+
+ // The point of the whole change: it is still queued, not burned.
+ expect(mediaJobQueue.getJob(second.jobId).status).toBe('queued');
+ expect(stubs.generateImageCodex).toHaveBeenCalledTimes(1);
+ expect(mediaJobQueue.getJob(second.jobId).statusMsg).toMatch(/resumes in 2h2[56]m/);
+ });
+
+ it('leaves a different backend free to run during another backend\'s cooldown', async () => {
+ stubs.generateImageCodex.mockImplementation(() => new Promise(() => {}));
+ const first = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 1' } });
+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 1);
+ imageGenEvents.emit('failed', { generationId: first.jobId, error: QUOTA_ERROR });
+ await waitFor(() => mediaJobQueue.getJob(first.jobId)?.status === 'failed');
+
+ stubs.generateImage.mockImplementation(() => new Promise(() => {}));
+ const local = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'local', prompt: 'page 2' } });
+ await waitFor(() => stubs.generateImage.mock.calls.length === 1);
+ expect(mediaJobQueue.getJob(local.jobId).status).toBe('running');
+ });
+
+ it('does not hold anything for an ordinary failure that states no reset', async () => {
+ // Resolve rather than hang: the lane is released by the gen call settling,
+ // so a hanging stub would keep the cloud slot and mask the assertion.
+ stubs.generateImageCodex.mockResolvedValue({ jobId: 'codex-job' });
+ const first = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 1' } });
+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 1);
+ imageGenEvents.emit('failed', { generationId: first.jobId, error: 'Agy generation failed: Exit code 1' });
+ await waitFor(() => mediaJobQueue.getJob(first.jobId)?.status === 'failed');
+ await flush();
+
+ const second = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 2' } });
+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 2);
+ expect(mediaJobQueue.getJob(second.jobId).status).toBe('running');
+ });
+
+ it('keeps the cooldown across a restart so a reboot does not release the queue early', async () => {
+ stubs.generateImageCodex.mockResolvedValue({ jobId: 'codex-job' });
+ const first = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 1' } });
+ await waitFor(() => stubs.generateImageCodex.mock.calls.length === 1);
+ imageGenEvents.emit('failed', { generationId: first.jobId, error: QUOTA_ERROR });
+ await waitFor(() => mediaJobQueue.getJob(first.jobId)?.status === 'failed');
+ await flush();
+
+ const persisted = JSON.parse(readFileSync(join(tempDataDir, 'media-jobs.json'), 'utf8'));
+ expect(persisted.quotaCooldowns).toEqual([['codex', expect.any(Number)]]);
+
+ stubs.generateImageCodex.mockClear();
+ await importFresh();
+ // The restart itself: init is what re-reads the snapshot.
+ await mediaJobQueue.initMediaJobQueue();
+ const second = await mediaJobQueue.enqueueJob({ kind: 'image', params: { mode: 'codex', prompt: 'page 2' } });
+ await waitFor(() => (mediaJobQueue.getJob(second.jobId)?.statusMsg || '').startsWith('Waiting for codex quota'));
+ expect(stubs.generateImageCodex).not.toHaveBeenCalled();
+ });
+});