Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Continue scene video generation when duration metadata is unavailable

Codex release-state testdcc79db88f0a
diff --git a/docs/providers-and-models.md b/docs/providers-and-models.md
index df3779f40..23f614a28 100644
--- a/docs/providers-and-models.md
+++ b/docs/providers-and-models.md
@@ -325,8 +325,15 @@ read once per call that sets any of them. From a schema, WebBrain takes the inpu
body has, its listed clip lengths (sent as listed, such as `6`, `"6"` or
`"8s"`) or the whole seconds of its length range (sent as numbers), and its
listed aspect ratios and resolutions (matched without regard to case and sent as
-listed, such as `768P`). Without a schema, a single clip's options are sent as
-given and scenes are refused, since the clip lengths are unknown. A local
+listed, such as `768P`). When fal.ai's catalog has no readable clip lengths,
+WebBrain also tries its public OpenAPI endpoint without credentials. If clip
+lengths remain unavailable, scenes use the schema's stated duration default,
+or an 8-second fallback. A one-minute request can therefore use eight distinct
+8-second scenes, aiming for 64 seconds; the output is measured separately.
+Fallback lengths are attempts, not advertised model limits. String duration
+inputs keep their declared encoding; a schema with no duration field uses the
+provider's default without adding that field. Real submission errors still
+stop the request, while missing metadata alone does not. A local
ComfyUI workflow sets its own length and shape, so these options are refused
there. A longer video is passed as `scenes`: WebBrain makes one clip per
scene at a length the model offers (a scene longer than its longest clip is
@@ -349,8 +356,9 @@ the tool result says why (`joinError`), which traces show. The model never sees
the video, so the tool result gives it the video's measured length
(`durationSeconds`) and `clipCount` when available. Plans that knowingly fall
short of `estimated_duration` within the 120-second cap are rejected before
-submission; missing clip-length metadata must not silently fall back to a
-default short clip. Results retain the original estimate (`estimatedSeconds`),
+submission; missing clip-length metadata uses short scene clips covering the
+full target. Results retain any assumed clip length (`fallbackClipSeconds`),
+the original estimate (`estimatedSeconds`),
accepted plan (`plannedSeconds`), and `durationStatus`: `short`, `enough`,
or `unverified`. A measured shortfall keeps the generated asset and reports
incomplete delivery; hosted clips without a measured length stay unverified.
diff --git a/package.json b/package.json
index a9a20503f..828642c6f 100644
--- a/package.json
+++ b/package.json
@@ -87,7 +87,7 @@
"test:cms:browser": "node --test test/cms-browser-smoke.mjs",
"test:compass-quota": "node --test test/compass-quota.mjs",
"test:compass-quota:ui": "node test/compass-quota-ui.mjs",
- "test:generative-media": "node --test test/generative-media.mjs test/generative-media-availability.mjs test/generative-media-video.mjs",
+ "test:generative-media": "node --test test/generative-media.mjs test/generative-media-availability.mjs test/generative-media-video.mjs test/generative-media-fal-schema.mjs",
"test:generative-media:ui": "node test/generative-media-ui.mjs",
"test:generative-media:extension": "node test/generative-media-extension.mjs",
"test:feedback": "node --test test/feedback-traces.mjs",
diff --git a/src/chrome/skills/video-scenes.md b/src/chrome/skills/video-scenes.md
index abde7593a..f870aebad 100644
--- a/src/chrome/skills/video-scenes.md
+++ b/src/chrome/skills/video-scenes.md
@@ -10,7 +10,7 @@
Use this skill before calling `generate_image` with `scenes`. The video model makes each scene as a separate clip from its text alone, without seeing the other clips; WebBrain then joins the clips into one video.
-Keep the user's full requested length in `estimated_duration`. A 60-second video can use four 15-second scenes when the model supports that clip length. The per-clip limit does not limit the whole video, and no separate joining tool is needed. If WebBrain cannot read supported clip lengths, report that capability lookup problem without replacing the requested video with one short default clip.
+Keep the user's full requested length in `estimated_duration`. A 60-second video can use four 15-second scenes when the model supports that clip length. The per-clip limit does not limit the whole video, and no separate joining tool is needed. When clip lengths are unavailable, use the provider's stated default or eight 8-second scenes for about 64 seconds. Continue generation and report the measured result; missing metadata alone is not a blocker. If the tool asks for shorter scenes, split them into distinct prompts while keeping the full target.
## The shared prompt
diff --git a/src/chrome/src/agent/generative-media.js b/src/chrome/src/agent/generative-media.js
index 7aa131900..b728e3a61 100644
--- a/src/chrome/src/agent/generative-media.js
+++ b/src/chrome/src/agent/generative-media.js
@@ -10,6 +10,7 @@ const MAX_INLINE_BYTES = 20 * 1024 * 1024;
export const MAX_VIDEO_SECONDS = 120;
export const MAX_VIDEO_CLIPS = 24;
const VIDEO_CLIP_CONCURRENCY = 3;
+const FALLBACK_VIDEO_CLIP_SECONDS = 8;
const VIDEO_ARGUMENTS = ['duration', 'aspect_ratio', 'resolution', 'scenes', 'estimated_duration'];
const INLINE_MEDIA = /^data:(image\/(?:png|jpeg|webp|gif)|audio\/(?:mpeg|wav|ogg)|video\/mp4);base64,[A-Za-z0-9+/=\s]+$/;
@@ -313,7 +314,8 @@ export function videoInputCapabilities(openapi) {
// a listed value (6, "6", "6.0" or "8s"; "auto" is not a length), or a
// second within a range.
const lengths = new Map();
- for (const value of forms(input.get('duration')).flatMap(form => formValues(form) ?? formSeconds(form))) {
+ const durationForms = forms(input.get('duration'));
+ for (const value of durationForms.flatMap(form => formValues(form) ?? formSeconds(form))) {
const match = /^(\d+)(?:\.0+)?\s*s?$/i.exec(String(value).trim());
const seconds = match ? Number(match[1]) : 0;
if (seconds > 0 && !lengths.has(seconds)) lengths.set(seconds, value);
@@ -321,6 +323,30 @@ export function videoInputCapabilities(openapi) {
if (lengths.size) {
entry.supported_durations = [...lengths.keys()].sort((a, b) => a - b);
entry.encodeDuration = seconds => lengths.get(seconds);
+ } else if (input.has('duration')) {
+ // A published default is useful even without a bounded list of lengths.
+ // Preserve that exact value, but do not infer units for other durations.
+ let defaultValue;
+ let defaultSeconds;
+ for (const form of durationForms) {
+ for (const schema of form) {
+ const value = schema.default;
+ const match = /^(\d+)(?:\.0+)?\s*s?$/i.exec(String(value).trim());
+ const seconds = match ? Number(match[1]) : 0;
+ if (Number.isSafeInteger(seconds) && seconds > 0 && formAllows(form, value)) {
+ defaultValue = value;
+ defaultSeconds = seconds;
+ break;
+ }
+ }
+ if (defaultSeconds !== undefined) break;
+ }
+ if (defaultSeconds !== undefined) entry.default_duration = defaultSeconds;
+ const admitsType = (form, allowed) => form.every(schema => schema.type === undefined
+ || [].concat(schema.type).some(type => allowed.includes(type)));
+ const stringOnly = durationForms.some(form => admitsType(form, ['string']))
+ && !durationForms.some(form => admitsType(form, ['integer', 'number']));
+ entry.encodeDuration = seconds => seconds === defaultSeconds ? defaultValue : stringOnly ? String(seconds) : seconds;
}
const aspectRatios = listed('aspect_ratio');
if (aspectRatios) entry.supported_aspect_ratios = aspectRatios;
@@ -331,7 +357,7 @@ export function videoInputCapabilities(openapi) {
// A clip's length as the model takes it, ready to merge into its request.
function clipLength(entry, clip) {
- if (clip.duration === undefined) return {};
+ if (clip.duration === undefined || (entry?.inputs && !entry.inputs.includes('duration'))) return {};
return { duration: entry?.encodeDuration ? entry.encodeDuration(clip.duration) : clip.duration };
}
@@ -342,25 +368,26 @@ function clipLength(entry, clip) {
* the model would make the same clip twice. Refuses more than `maxSeconds` in
* total or `maxClips`.
*/
-export function planVideoClips(scenes, durations, { maxSeconds = MAX_VIDEO_SECONDS, maxClips = MAX_VIDEO_CLIPS } = {}) {
- if (!durations?.length) {
- throw new Error('This video model does not list the clip lengths it supports. WebBrain can join scene clips, but cannot determine their supported lengths from this model\'s schema. No clips were generated; use a video model with published clip lengths to make the requested video.');
- }
- const longest = durations[durations.length - 1];
+export function planVideoClips(scenes, durations, { maxSeconds = MAX_VIDEO_SECONDS, maxClips = MAX_VIDEO_CLIPS, fallbackSeconds = FALLBACK_VIDEO_CLIP_SECONDS } = {}) {
+ const lengths = durations?.length ? durations : [fallbackSeconds];
+ const longest = lengths[lengths.length - 1];
const clips = scenes.map((scene, index) => {
const prompt = String(scene?.prompt || '').trim();
if (!prompt) throw new Error(`Scene ${index + 1} needs a prompt.`);
if (scene.duration > longest) {
+ if (!durations?.length) {
+ throw new Error(`Scene ${index + 1} is ${scene.duration} s. Published clip lengths are unavailable, so WebBrain is using ${longest} s as a fallback, not a confirmed model limit. Retry with distinct scenes of up to ${longest} s that cover the full requested duration; WebBrain will generate and join them.`);
+ }
throw new Error(`Scene ${index + 1} is ${scene.duration} s, and this model makes clips of up to ${longest} s. Split it into scenes of up to ${longest} s, each with its own prompt for its part, so no clip repeats another.`);
}
- return { prompt, duration: durations.find(value => value >= scene.duration) ?? longest };
+ return { prompt, duration: lengths.find(value => value >= scene.duration) ?? longest };
});
const total = clips.reduce((sum, clip) => sum + clip.duration, 0);
if (clips.length > maxClips) {
throw new Error(`These scenes need ${clips.length} clips and WebBrain makes at most ${maxClips}. Use fewer, longer scenes.`);
}
if (total > maxSeconds) {
- throw new Error(`These scenes come to ${total} s with this model's clip lengths (${durations.join(', ')} s), and WebBrain makes videos up to ${maxSeconds} s. Shorten or merge scenes.`);
+ throw new Error(`These scenes come to ${total} s with the ${durations?.length ? "model's clip lengths" : 'fallback clip length'} (${lengths.join(', ')} s), and WebBrain makes videos up to ${maxSeconds} s. Shorten or merge scenes.`);
}
return { clips, total };
}
@@ -369,6 +396,8 @@ export function planVideoClips(scenes, durations, { maxSeconds = MAX_VIDEO_SECON
// clips of its scenes.
function planVideo(prompt, video, entry) {
const durations = clipDurations(entry);
+ const fallbackSeconds = entry?.default_duration || FALLBACK_VIDEO_CLIP_SECONDS;
+ const fallback = durations ? {} : { fallbackClipSeconds: fallbackSeconds };
const shape = videoShape(entry, video);
const scenes = Array.isArray(video?.scenes) ? video.scenes : null;
if (scenes?.length) {
@@ -376,12 +405,13 @@ function planVideo(prompt, video, entry) {
if (video.estimated_duration === undefined) {
throw new Error('With scenes, give estimated_duration: how many seconds the requested video needs before fitting it to the 120 s maximum.');
}
- const { clips, total } = planVideoClips(scenes, durations);
+ const { clips, total } = planVideoClips(scenes, durations, { fallbackSeconds });
if (video.estimated_duration <= MAX_VIDEO_SECONDS && total < video.estimated_duration) {
throw new Error(`These scenes provide ${total} s, but the requested video needs ${video.estimated_duration} s. Add or lengthen scenes to cover the requested duration before generating.`);
}
return {
scenes: true,
+ ...fallback,
total,
estimated: video.estimated_duration,
shape,
@@ -390,18 +420,18 @@ function planVideo(prompt, video, entry) {
clips: clips.map(clip => ({ ...clip, prompt: `${prompt}\n\n${clip.prompt}` })),
};
}
- const longest = durations?.[durations.length - 1];
+ const longest = durations?.[durations.length - 1] ?? fallbackSeconds;
const wanted = video?.estimated_duration === undefined
? video?.duration : Math.max(video.estimated_duration, video.duration ?? 0);
if (wanted !== undefined && longest && wanted > longest) {
+ if (!durations) {
+ throw new Error(`A ${formatSeconds(wanted)} video needs multiple clips with WebBrain's ${longest} s fallback. Published clip lengths are unavailable. Retry with distinct scenes of up to ${longest} s covering the full requested duration, and keep estimated_duration: ${video?.estimated_duration ?? wanted}. WebBrain will generate and join them.`);
+ }
throw new Error(`A ${formatSeconds(wanted)} video needs more than one clip: this model makes clips of up to ${longest} s. Describe it as scenes of up to ${longest} s each, ${MAX_VIDEO_SECONDS} s in total, with estimated_duration.`);
}
if (video?.duration !== undefined && video?.estimated_duration !== undefined && video.duration < video.estimated_duration) {
throw new Error(`The requested video needs ${video.estimated_duration} s, but duration specifies only ${video.duration} s. Increase the clip duration or describe the full video as scenes before generating.`);
}
- if (video?.estimated_duration !== undefined && video?.duration === undefined && !durations) {
- throw new Error(`WebBrain cannot determine a supported clip length for the requested ${video.estimated_duration} s video from this model's schema. No clip was generated; use a video model with published clip lengths or supply a supported duration explicitly.`);
- }
let duration;
if (video?.duration !== undefined) {
if (entry?.inputs && !entry.inputs.includes('duration')) throw new Error('This video model does not take duration.');
@@ -409,10 +439,10 @@ function planVideo(prompt, video, entry) {
throw new Error(`This video model does not make ${video.duration} s clips. Supported: ${durations.join(', ')} s.`);
}
duration = video.duration;
- } else if (wanted !== undefined && durations) {
- duration = durations.find(value => value >= wanted) ?? longest;
+ } else if (wanted !== undefined) {
+ duration = durations?.find(value => value >= wanted) ?? longest;
}
- return { scenes: false, shape, clips: [{ prompt, ...(duration !== undefined ? { duration } : {}) }] };
+ return { scenes: false, shape, ...(duration !== undefined ? fallback : {}), clips: [{ prompt, ...(duration !== undefined ? { duration } : {}) }] };
}
function videoRequestBody(model, clip, shape) {
@@ -515,10 +545,12 @@ async function mapWithConcurrency(items, limit, signal, task) {
*/
async function runVideo({ model, entry, text, video, signal, onProgress, confirmVideoLength, single, clip }) {
const plan = planVideo(text, video, entry);
+ const fallback = plan.fallbackClipSeconds ? { fallbackClipSeconds: plan.fallbackClipSeconds } : {};
if (!plan.scenes) return {
...(await single(plan.clips[0], plan.shape)), model,
...(video?.estimated_duration !== undefined ? { estimatedSeconds: video.estimated_duration } : {}),
...(plan.clips[0].duration !== undefined ? { totalSeconds: plan.clips[0].duration } : {}),
+ ...fallback,
};
// The user decides on the card whether a shorter version is worth
@@ -551,7 +583,7 @@ async function runVideo({ model, entry, text, video, signal, onProgress, confirm
}
}
});
- return { clips, model, ...lengthDetails };
+ return { clips, model, ...lengthDetails, ...fallback };
}
async function runOpenRouter({ text, media, config, video, fetchImpl, signal, timeoutMs, pollMs, onProgress, confirmVideoLength }) {
@@ -589,23 +621,53 @@ async function runOpenRouter({ text, media, config, video, fetchImpl, signal, ti
async function modelSchema(url, headers, label, fetchImpl, signal, timeoutMs) {
const operation = operationSignal(signal, Math.min(timeoutMs, 30000));
try {
+ operation.signal.throwIfAborted();
const response = await fetchImpl(url, { headers, signal: operation.signal, redirect: 'error', credentials: 'omit' });
+ operation.signal.throwIfAborted();
return response.status === 404 ? null : await jsonResponse(response, label);
} catch (error) { throw operation.signal.aborted ? operation.signal.reason : error; }
finally { operation.dispose(); }
}
-// What a fal.ai video model accepts ({} when fal.ai publishes no schema for it).
+// Metadata outages do not prevent the caller from using its clip fallback.
+// User cancellation and authenticated API authorization failures still stop
+// the run. A denial from the public schema host is only a metadata failure.
+async function optionalModelSchema(url, headers, label, fetchImpl, signal, timeoutMs) {
+ try { return await modelSchema(url, headers, label, fetchImpl, signal, timeoutMs); }
+ catch (error) {
+ signal?.throwIfAborted?.();
+ const authenticated = headers?.Authorization || headers?.['X-API-Key'];
+ if (error?.name === 'AbortError' || (authenticated && [401, 403].includes(error?.status))) throw error;
+ return null;
+ }
+}
+
+// The catalog can return an expansion_failed object instead of an OpenAPI
+// document. The model page's public Schema endpoint is a second source; it
+// never receives the API key. Both reads share the metadata lookup budget.
async function falVideoEntry(model, apiKey, fetchImpl, signal, timeoutMs) {
+ const deadline = Date.now() + Math.min(timeoutMs, 30000);
+ const read = async (url, headers) => {
+ signal?.throwIfAborted?.();
+ const remaining = deadline - Date.now();
+ if (remaining <= 0) return null;
+ return optionalModelSchema(url, headers, 'fal.ai model schema', fetchImpl, signal, remaining);
+ };
const query = new URLSearchParams({ endpoint_id: model, expand: 'openapi-3.0' });
- const listing = await modelSchema(`https://api.fal.ai/v1/models?${query}`, { Authorization: `Key ${apiKey}` }, 'fal.ai model schema', fetchImpl, signal, timeoutMs);
+ const listing = await read(`https://api.fal.ai/v1/models?${query}`, { Authorization: `Key ${apiKey}` });
const openapi = (Array.isArray(listing?.models) ? listing.models : []).find(item => item?.endpoint_id === model)?.openapi;
- return videoInputCapabilities(openapi) || {};
+ const entry = videoInputCapabilities(openapi);
+ if (clipDurations(entry)) return entry;
+ const publicQuery = new URLSearchParams({ endpoint_id: model });
+ const fallback = videoInputCapabilities(await read(`https://fal.ai/api/openapi/queue/openapi.json?${publicQuery}`));
+ // An incomplete fallback must not discard constraints or a default from a
+ // usable catalog document. The caller decides how to handle unknown lengths.
+ return clipDurations(fallback) ? fallback : entry || fallback || {};
}
// What a Comfy Router video model accepts ({} when Comfy Router publishes no schema for it).
async function comfyRouterVideoEntry(model, apiKey, fetchImpl, signal, timeoutMs) {
- const openapi = await modelSchema(`https://api.comfy.org/v2/models/${model}/openapi.json`, { 'X-API-Key': apiKey }, 'Comfy Router model schema', fetchImpl, signal, timeoutMs);
+ const openapi = await optionalModelSchema(`https://api.comfy.org/v2/models/${model}/openapi.json`, { 'X-API-Key': apiKey }, 'Comfy Router model schema', fetchImpl, signal, timeoutMs);
return videoInputCapabilities(openapi) || {};
}
@@ -765,6 +827,7 @@ function videoDeliveryDetails(result, asset) {
return {
estimatedSeconds: roundSeconds(result.estimatedSeconds) || plannedSeconds,
plannedSeconds,
+ ...(result.fallbackClipSeconds ? { fallbackClipSeconds: result.fallbackClipSeconds } : {}),
durationStatus,
...(durationStatus === 'short' ? {
notice: `Generated ${actual} s of the planned ${plannedSeconds} s. Keep the generated asset, but report the duration shortfall; the requested video is incomplete. Do not claim it is complete.`,
diff --git a/src/chrome/src/agent/tools.js b/src/chrome/src/agent/tools.js
index f8f573f72..8bb830a47 100644
--- a/src/chrome/src/agent/tools.js
+++ b/src/chrome/src/agent/tools.js
@@ -37,7 +37,7 @@ const DONE_REQUIRED_WITH_OUTCOME = ['summary', 'outcome'];
export const SYSTEM_PROMPT_GENERATIVE_MEDIA = `GENERATIVE MEDIA:
- generate_image: Create an image, or with media "video" a video, directly from a text prompt through the user's configured image and video models. When the user asks to GENERATE media ("generate an image of a red apple", "make a logo", "create a video clip"), call this tool — do NOT navigate to third-party image sites (Midjourney, DALL·E, Bing Images, etc.). Not available in Ask mode.
- Videos: set media to "video" and estimate how long the requested video should run. One that needs more than one clip (an explainer, a story) is one call with scenes, at most 120 s in total, and estimated_duration set to the full length it needs; WebBrain joins the clips into one video and asks the user first when that estimate is longer than 120 s. You never see the result: report its length and clip count from the tool result, not its contents.
-- A model's clip limit applies to each scene. Joining is built into generate_image; no separate joining tool is needed. For a one-minute request keep estimated_duration:60 and plan scenes covering 60 seconds, for example four 15-second scenes when supported. If a scene exceeds the reported clip limit, split it into distinct scenes and retry. Missing clip-length metadata is a capability lookup problem: do not claim the model cannot make scenes or substitute a shorter default clip. Report any durationStatus "short" or joined:false result as incomplete, preserving the generated assets.`;
+- A model's clip limit applies to each scene. Joining is built into generate_image; no separate joining tool is needed. For a one-minute request keep estimated_duration:60 and plan scenes covering 60 seconds, for example four 15-second scenes when supported. If clip lengths are unavailable, use the provider's default or eight 8-second scenes (about 64 seconds), and continue generation. If the tool requests shorter scenes, split them into distinct prompts and retry while keeping the full target. A fallback length is an attempt, not a confirmed model limit; report actual provider rejections rather than claiming missing metadata prevents videos. Report any durationStatus "short" or joined:false result as incomplete, preserving the generated assets.`;
export const SYSTEM_PROMPT_BATCHING = `BATCHING:
- Use browser_batch for predictable steps on the current page with targets you already observed, such as filling known fields or opening a menu and choosing a known item. A navigation followed by a page read can also share one batch.
diff --git a/src/chrome/src/run-ui-journal.js b/src/chrome/src/run-ui-journal.js
index 2b41254e3..a25bd48d5 100644
--- a/src/chrome/src/run-ui-journal.js
+++ b/src/chrome/src/run-ui-journal.js
@@ -101,7 +101,7 @@ function compactRunUiToolResult(result, name) {
if (typeof source[key] === 'string') compacted[key] = source[key];
}
if (source.inlineMedia === true) compacted.inlineMedia = true;
- for (const key of ['durationSeconds', 'clipCount', 'estimatedSeconds', 'plannedSeconds']) {
+ for (const key of ['durationSeconds', 'clipCount', 'estimatedSeconds', 'plannedSeconds', 'fallbackClipSeconds']) {
if (Number.isFinite(source[key])) compacted[key] = source[key];
}
if (['short', 'enough', 'unverified'].includes(source.durationStatus)) compacted.durationStatus = source.durationStatus;
diff --git a/src/firefox/skills/video-scenes.md b/src/firefox/skills/video-scenes.md
index abde7593a..f870aebad 100644
--- a/src/firefox/skills/video-scenes.md
+++ b/src/firefox/skills/video-scenes.md
@@ -10,7 +10,7 @@
Use this skill before calling `generate_image` with `scenes`. The video model makes each scene as a separate clip from its text alone, without seeing the other clips; WebBrain then joins the clips into one video.
-Keep the user's full requested length in `estimated_duration`. A 60-second video can use four 15-second scenes when the model supports that clip length. The per-clip limit does not limit the whole video, and no separate joining tool is needed. If WebBrain cannot read supported clip lengths, report that capability lookup problem without replacing the requested video with one short default clip.
+Keep the user's full requested length in `estimated_duration`. A 60-second video can use four 15-second scenes when the model supports that clip length. The per-clip limit does not limit the whole video, and no separate joining tool is needed. When clip lengths are unavailable, use the provider's stated default or eight 8-second scenes for about 64 seconds. Continue generation and report the measured result; missing metadata alone is not a blocker. If the tool asks for shorter scenes, split them into distinct prompts while keeping the full target.
## The shared prompt
diff --git a/src/firefox/src/agent/generative-media.js b/src/firefox/src/agent/generative-media.js
index 7aa131900..b728e3a61 100644
--- a/src/firefox/src/agent/generative-media.js
+++ b/src/firefox/src/agent/generative-media.js
@@ -10,6 +10,7 @@ const MAX_INLINE_BYTES = 20 * 1024 * 1024;
export const MAX_VIDEO_SECONDS = 120;
export const MAX_VIDEO_CLIPS = 24;
const VIDEO_CLIP_CONCURRENCY = 3;
+const FALLBACK_VIDEO_CLIP_SECONDS = 8;
const VIDEO_ARGUMENTS = ['duration', 'aspect_ratio', 'resolution', 'scenes', 'estimated_duration'];
const INLINE_MEDIA = /^data:(image\/(?:png|jpeg|webp|gif)|audio\/(?:mpeg|wav|ogg)|video\/mp4);base64,[A-Za-z0-9+/=\s]+$/;
@@ -313,7 +314,8 @@ export function videoInputCapabilities(openapi) {
// a listed value (6, "6", "6.0" or "8s"; "auto" is not a length), or a
// second within a range.
const lengths = new Map();
- for (const value of forms(input.get('duration')).flatMap(form => formValues(form) ?? formSeconds(form))) {
+ const durationForms = forms(input.get('duration'));
+ for (const value of durationForms.flatMap(form => formValues(form) ?? formSeconds(form))) {
const match = /^(\d+)(?:\.0+)?\s*s?$/i.exec(String(value).trim());
const seconds = match ? Number(match[1]) : 0;
if (seconds > 0 && !lengths.has(seconds)) lengths.set(seconds, value);
@@ -321,6 +323,30 @@ export function videoInputCapabilities(openapi) {
if (lengths.size) {
entry.supported_durations = [...lengths.keys()].sort((a, b) => a - b);
entry.encodeDuration = seconds => lengths.get(seconds);
+ } else if (input.has('duration')) {
+ // A published default is useful even without a bounded list of lengths.
+ // Preserve that exact value, but do not infer units for other durations.
+ let defaultValue;
+ let defaultSeconds;
+ for (const form of durationForms) {
+ for (const schema of form) {
+ const value = schema.default;
+ const match = /^(\d+)(?:\.0+)?\s*s?$/i.exec(String(value).trim());
+ const seconds = match ? Number(match[1]) : 0;
+ if (Number.isSafeInteger(seconds) && seconds > 0 && formAllows(form, value)) {
+ defaultValue = value;
+ defaultSeconds = seconds;
+ break;
+ }
+ }
+ if (defaultSeconds !== undefined) break;
+ }
+ if (defaultSeconds !== undefined) entry.default_duration = defaultSeconds;
+ const admitsType = (form, allowed) => form.every(schema => schema.type === undefined
+ || [].concat(schema.type).some(type => allowed.includes(type)));
+ const stringOnly = durationForms.some(form => admitsType(form, ['string']))
+ && !durationForms.some(form => admitsType(form, ['integer', 'number']));
+ entry.encodeDuration = seconds => seconds === defaultSeconds ? defaultValue : stringOnly ? String(seconds) : seconds;
}
const aspectRatios = listed('aspect_ratio');
if (aspectRatios) entry.supported_aspect_ratios = aspectRatios;
@@ -331,7 +357,7 @@ export function videoInputCapabilities(openapi) {
// A clip's length as the model takes it, ready to merge into its request.
function clipLength(entry, clip) {
- if (clip.duration === undefined) return {};
+ if (clip.duration === undefined || (entry?.inputs && !entry.inputs.includes('duration'))) return {};
return { duration: entry?.encodeDuration ? entry.encodeDuration(clip.duration) : clip.duration };
}
@@ -342,25 +368,26 @@ function clipLength(entry, clip) {
* the model would make the same clip twice. Refuses more than `maxSeconds` in
* total or `maxClips`.
*/
-export function planVideoClips(scenes, durations, { maxSeconds = MAX_VIDEO_SECONDS, maxClips = MAX_VIDEO_CLIPS } = {}) {
- if (!durations?.length) {
- throw new Error('This video model does not list the clip lengths it supports. WebBrain can join scene clips, but cannot determine their supported lengths from this model\'s schema. No clips were generated; use a video model with published clip lengths to make the requested video.');
- }
- const longest = durations[durations.length - 1];
+export function planVideoClips(scenes, durations, { maxSeconds = MAX_VIDEO_SECONDS, maxClips = MAX_VIDEO_CLIPS, fallbackSeconds = FALLBACK_VIDEO_CLIP_SECONDS } = {}) {
+ const lengths = durations?.length ? durations : [fallbackSeconds];
+ const longest = lengths[lengths.length - 1];
const clips = scenes.map((scene, index) => {
const prompt = String(scene?.prompt || '').trim();
if (!prompt) throw new Error(`Scene ${index + 1} needs a prompt.`);
if (scene.duration > longest) {
+ if (!durations?.length) {
+ throw new Error(`Scene ${index + 1} is ${scene.duration} s. Published clip lengths are unavailable, so WebBrain is using ${longest} s as a fallback, not a confirmed model limit. Retry with distinct scenes of up to ${longest} s that cover the full requested duration; WebBrain will generate and join them.`);
+ }
throw new Error(`Scene ${index + 1} is ${scene.duration} s, and this model makes clips of up to ${longest} s. Split it into scenes of up to ${longest} s, each with its own prompt for its part, so no clip repeats another.`);
}
- return { prompt, duration: durations.find(value => value >= scene.duration) ?? longest };
+ return { prompt, duration: lengths.find(value => value >= scene.duration) ?? longest };
});
const total = clips.reduce((sum, clip) => sum + clip.duration, 0);
if (clips.length > maxClips) {
throw new Error(`These scenes need ${clips.length} clips and WebBrain makes at most ${maxClips}. Use fewer, longer scenes.`);
}
if (total > maxSeconds) {
- throw new Error(`These scenes come to ${total} s with this model's clip lengths (${durations.join(', ')} s), and WebBrain makes videos up to ${maxSeconds} s. Shorten or merge scenes.`);
+ throw new Error(`These scenes come to ${total} s with the ${durations?.length ? "model's clip lengths" : 'fallback clip length'} (${lengths.join(', ')} s), and WebBrain makes videos up to ${maxSeconds} s. Shorten or merge scenes.`);
}
return { clips, total };
}
@@ -369,6 +396,8 @@ export function planVideoClips(scenes, durations, { maxSeconds = MAX_VIDEO_SECON
// clips of its scenes.
function planVideo(prompt, video, entry) {
const durations = clipDurations(entry);
+ const fallbackSeconds = entry?.default_duration || FALLBACK_VIDEO_CLIP_SECONDS;
+ const fallback = durations ? {} : { fallbackClipSeconds: fallbackSeconds };
const shape = videoShape(entry, video);
const scenes = Array.isArray(video?.scenes) ? video.scenes : null;
if (scenes?.length) {
@@ -376,12 +405,13 @@ function planVideo(prompt, video, entry) {
if (video.estimated_duration === undefined) {
throw new Error('With scenes, give estimated_duration: how many seconds the requested video needs before fitting it to the 120 s maximum.');
}
- const { clips, total } = planVideoClips(scenes, durations);
+ const { clips, total } = planVideoClips(scenes, durations, { fallbackSeconds });
if (video.estimated_duration <= MAX_VIDEO_SECONDS && total < video.estimated_duration) {
throw new Error(`These scenes provide ${total} s, but the requested video needs ${video.estimated_duration} s. Add or lengthen scenes to cover the requested duration before generating.`);
}
return {
scenes: true,
+ ...fallback,
total,
estimated: video.estimated_duration,
shape,
@@ -390,18 +420,18 @@ function planVideo(prompt, video, entry) {
clips: clips.map(clip => ({ ...clip, prompt: `${prompt}\n\n${clip.prompt}` })),
};
}
- const longest = durations?.[durations.length - 1];
+ const longest = durations?.[durations.length - 1] ?? fallbackSeconds;
const wanted = video?.estimated_duration === undefined
? video?.duration : Math.max(video.estimated_duration, video.duration ?? 0);
if (wanted !== undefined && longest && wanted > longest) {
+ if (!durations) {
+ throw new Error(`A ${formatSeconds(wanted)} video needs multiple clips with WebBrain's ${longest} s fallback. Published clip lengths are unavailable. Retry with distinct scenes of up to ${longest} s covering the full requested duration, and keep estimated_duration: ${video?.estimated_duration ?? wanted}. WebBrain will generate and join them.`);
+ }
throw new Error(`A ${formatSeconds(wanted)} video needs more than one clip: this model makes clips of up to ${longest} s. Describe it as scenes of up to ${longest} s each, ${MAX_VIDEO_SECONDS} s in total, with estimated_duration.`);
}
if (video?.duration !== undefined && video?.estimated_duration !== undefined && video.duration < video.estimated_duration) {
throw new Error(`The requested video needs ${video.estimated_duration} s, but duration specifies only ${video.duration} s. Increase the clip duration or describe the full video as scenes before generating.`);
}
- if (video?.estimated_duration !== undefined && video?.duration === undefined && !durations) {
- throw new Error(`WebBrain cannot determine a supported clip length for the requested ${video.estimated_duration} s video from this model's schema. No clip was generated; use a video model with published clip lengths or supply a supported duration explicitly.`);
- }
let duration;
if (video?.duration !== undefined) {
if (entry?.inputs && !entry.inputs.includes('duration')) throw new Error('This video model does not take duration.');
@@ -409,10 +439,10 @@ function planVideo(prompt, video, entry) {
throw new Error(`This video model does not make ${video.duration} s clips. Supported: ${durations.join(', ')} s.`);
}
duration = video.duration;
- } else if (wanted !== undefined && durations) {
- duration = durations.find(value => value >= wanted) ?? longest;
+ } else if (wanted !== undefined) {
+ duration = durations?.find(value => value >= wanted) ?? longest;
}
- return { scenes: false, shape, clips: [{ prompt, ...(duration !== undefined ? { duration } : {}) }] };
+ return { scenes: false, shape, ...(duration !== undefined ? fallback : {}), clips: [{ prompt, ...(duration !== undefined ? { duration } : {}) }] };
}
function videoRequestBody(model, clip, shape) {
@@ -515,10 +545,12 @@ async function mapWithConcurrency(items, limit, signal, task) {
*/
async function runVideo({ model, entry, text, video, signal, onProgress, confirmVideoLength, single, clip }) {
const plan = planVideo(text, video, entry);
+ const fallback = plan.fallbackClipSeconds ? { fallbackClipSeconds: plan.fallbackClipSeconds } : {};
if (!plan.scenes) return {
...(await single(plan.clips[0], plan.shape)), model,
...(video?.estimated_duration !== undefined ? { estimatedSeconds: video.estimated_duration } : {}),
...(plan.clips[0].duration !== undefined ? { totalSeconds: plan.clips[0].duration } : {}),
+ ...fallback,
};
// The user decides on the card whether a shorter version is worth
@@ -551,7 +583,7 @@ async function runVideo({ model, entry, text, video, signal, onProgress, confirm
}
}
});
- return { clips, model, ...lengthDetails };
+ return { clips, model, ...lengthDetails, ...fallback };
}
async function runOpenRouter({ text, media, config, video, fetchImpl, signal, timeoutMs, pollMs, onProgress, confirmVideoLength }) {
@@ -589,23 +621,53 @@ async function runOpenRouter({ text, media, config, video, fetchImpl, signal, ti
async function modelSchema(url, headers, label, fetchImpl, signal, timeoutMs) {
const operation = operationSignal(signal, Math.min(timeoutMs, 30000));
try {
+ operation.signal.throwIfAborted();
const response = await fetchImpl(url, { headers, signal: operation.signal, redirect: 'error', credentials: 'omit' });
+ operation.signal.throwIfAborted();
return response.status === 404 ? null : await jsonResponse(response, label);
} catch (error) { throw operation.signal.aborted ? operation.signal.reason : error; }
finally { operation.dispose(); }
}
-// What a fal.ai video model accepts ({} when fal.ai publishes no schema for it).
+// Metadata outages do not prevent the caller from using its clip fallback.
+// User cancellation and authenticated API authorization failures still stop
+// the run. A denial from the public schema host is only a metadata failure.
+async function optionalModelSchema(url, headers, label, fetchImpl, signal, timeoutMs) {
+ try { return await modelSchema(url, headers, label, fetchImpl, signal, timeoutMs); }
+ catch (error) {
+ signal?.throwIfAborted?.();
+ const authenticated = headers?.Authorization || headers?.['X-API-Key'];
+ if (error?.name === 'AbortError' || (authenticated && [401, 403].includes(error?.status))) throw error;
+ return null;
+ }
+}
+
+// The catalog can return an expansion_failed object instead of an OpenAPI
+// document. The model page's public Schema endpoint is a second source; it
+// never receives the API key. Both reads share the metadata lookup budget.
async function falVideoEntry(model, apiKey, fetchImpl, signal, timeoutMs) {
+ const deadline = Date.now() + Math.min(timeoutMs, 30000);
+ const read = async (url, headers) => {
+ signal?.throwIfAborted?.();
+ const remaining = deadline - Date.now();
+ if (remaining <= 0) return null;
+ return optionalModelSchema(url, headers, 'fal.ai model schema', fetchImpl, signal, remaining);
+ };
const query = new URLSearchParams({ endpoint_id: model, expand: 'openapi-3.0' });
- const listing = await modelSchema(`https://api.fal.ai/v1/models?${query}`, { Authorization: `Key ${apiKey}` }, 'fal.ai model schema', fetchImpl, signal, timeoutMs);
+ const listing = await read(`https://api.fal.ai/v1/models?${query}`, { Authorization: `Key ${apiKey}` });
const openapi = (Array.isArray(listing?.models) ? listing.models : []).find(item => item?.endpoint_id === model)?.openapi;
- return videoInputCapabilities(openapi) || {};
+ const entry = videoInputCapabilities(openapi);
+ if (clipDurations(entry)) return entry;
+ const publicQuery = new URLSearchParams({ endpoint_id: model });
+ const fallback = videoInputCapabilities(await read(`https://fal.ai/api/openapi/queue/openapi.json?${publicQuery}`));
+ // An incomplete fallback must not discard constraints or a default from a
+ // usable catalog document. The caller decides how to handle unknown lengths.
+ return clipDurations(fallback) ? fallback : entry || fallback || {};
}
// What a Comfy Router video model accepts ({} when Comfy Router publishes no schema for it).
async function comfyRouterVideoEntry(model, apiKey, fetchImpl, signal, timeoutMs) {
- const openapi = await modelSchema(`https://api.comfy.org/v2/models/${model}/openapi.json`, { 'X-API-Key': apiKey }, 'Comfy Router model schema', fetchImpl, signal, timeoutMs);
+ const openapi = await optionalModelSchema(`https://api.comfy.org/v2/models/${model}/openapi.json`, { 'X-API-Key': apiKey }, 'Comfy Router model schema', fetchImpl, signal, timeoutMs);
return videoInputCapabilities(openapi) || {};
}
@@ -765,6 +827,7 @@ function videoDeliveryDetails(result, asset) {
return {
estimatedSeconds: roundSeconds(result.estimatedSeconds) || plannedSeconds,
plannedSeconds,
+ ...(result.fallbackClipSeconds ? { fallbackClipSeconds: result.fallbackClipSeconds } : {}),
durationStatus,
...(durationStatus === 'short' ? {
notice: `Generated ${actual} s of the planned ${plannedSeconds} s. Keep the generated asset, but report the duration shortfall; the requested video is incomplete. Do not claim it is complete.`,
diff --git a/src/firefox/src/agent/tools.js b/src/firefox/src/agent/tools.js
index 35a78cec0..4a077af90 100644
--- a/src/firefox/src/agent/tools.js
+++ b/src/firefox/src/agent/tools.js
@@ -37,7 +37,7 @@ const DONE_REQUIRED_WITH_OUTCOME = ['summary', 'outcome'];
export const SYSTEM_PROMPT_GENERATIVE_MEDIA = `GENERATIVE MEDIA:
- generate_image: Create an image, or with media "video" a video, directly from a text prompt through the user's configured image and video models. When the user asks to GENERATE media ("generate an image of a red apple", "make a logo", "create a video clip"), call this tool — do NOT navigate to third-party image sites (Midjourney, DALL·E, Bing Images, etc.). Not available in Ask mode.
- Videos: set media to "video" and estimate how long the requested video should run. One that needs more than one clip (an explainer, a story) is one call with scenes, at most 120 s in total, and estimated_duration set to the full length it needs; WebBrain joins the clips into one video and asks the user first when that estimate is longer than 120 s. You never see the result: report its length and clip count from the tool result, not its contents.
-- A model's clip limit applies to each scene. Joining is built into generate_image; no separate joining tool is needed. For a one-minute request keep estimated_duration:60 and plan scenes covering 60 seconds, for example four 15-second scenes when supported. If a scene exceeds the reported clip limit, split it into distinct scenes and retry. Missing clip-length metadata is a capability lookup problem: do not claim the model cannot make scenes or substitute a shorter default clip. Report any durationStatus "short" or joined:false result as incomplete, preserving the generated assets.`;
+- A model's clip limit applies to each scene. Joining is built into generate_image; no separate joining tool is needed. For a one-minute request keep estimated_duration:60 and plan scenes covering 60 seconds, for example four 15-second scenes when supported. If clip lengths are unavailable, use the provider's default or eight 8-second scenes (about 64 seconds), and continue generation. If the tool requests shorter scenes, split them into distinct prompts and retry while keeping the full target. A fallback length is an attempt, not a confirmed model limit; report actual provider rejections rather than claiming missing metadata prevents videos. Report any durationStatus "short" or joined:false result as incomplete, preserving the generated assets.`;
export const SYSTEM_PROMPT_BATCHING = `BATCHING:
- Use browser_batch for predictable steps on the current page with targets you already observed, such as filling known fields or opening a menu and choosing a known item. A navigation followed by a page read can also share one batch.
diff --git a/src/firefox/src/run-ui-journal.js b/src/firefox/src/run-ui-journal.js
index 2b41254e3..a25bd48d5 100644
--- a/src/firefox/src/run-ui-journal.js
+++ b/src/firefox/src/run-ui-journal.js
@@ -101,7 +101,7 @@ function compactRunUiToolResult(result, name) {
if (typeof source[key] === 'string') compacted[key] = source[key];
}
if (source.inlineMedia === true) compacted.inlineMedia = true;
- for (const key of ['durationSeconds', 'clipCount', 'estimatedSeconds', 'plannedSeconds']) {
+ for (const key of ['durationSeconds', 'clipCount', 'estimatedSeconds', 'plannedSeconds', 'fallbackClipSeconds']) {
if (Number.isFinite(source[key])) compacted[key] = source[key];
}
if (['short', 'enough', 'unverified'].includes(source.durationStatus)) compacted.durationStatus = source.durationStatus;
diff --git a/test/generative-media-fal-schema.mjs b/test/generative-media-fal-schema.mjs
new file mode 100644
index 000000000..75e98d72b
--- /dev/null
+++ b/test/generative-media-fal-schema.mjs
@@ -0,0 +1,216 @@
+import assert from 'node:assert/strict';
+import test from 'node:test';
+
+const model = 'minimax/h3/text-to-video';
+const config = { provider: 'fal', apiKey: 'private-fal-key', videoModel: model };
+const json = (body, status = 200) => new Response(JSON.stringify(body), {
+ status, headers: { 'Content-Type': 'application/json' },
+});
+const schema = properties => ({
+ openapi: '3.0.4',
+ paths: { '/model': { post: { requestBody: { content: {
+ 'application/json': { schema: { $ref: '#/components/schemas/Input' } },
+ } } } } },
+ components: { schemas: { Input: { type: 'object', properties: { prompt: { type: 'string' }, ...properties } } } },
+});
+const bounded = schema({ duration: { type: 'integer', minimum: 5, maximum: 15 } });
+const listing = openapi => json({ models: [{ endpoint_id: model, openapi }] });
+
+// Only metadata and a simulated single-clip queue are available. No real
+// request or generated media download can escape this injected fetch.
+function api({ catalog = () => listing(bounded), direct = () => json(bounded) } = {}) {
+ const calls = { catalog: 0, direct: 0, submits: [] };
+ const fetchImpl = async (url, init = {}) => {
+ const parsed = new URL(url);
+ assert.equal(init.credentials, 'omit');
+ assert.equal(init.redirect, 'error');
+ if (parsed.origin === 'https://api.fal.ai') {
+ assert.equal(parsed.pathname, '/v1/models');
+ assert.equal(parsed.searchParams.get('endpoint_id'), model);
+ assert.equal(parsed.searchParams.get('expand'), 'openapi-3.0');
+ assert.equal(init.headers.Authorization, `Key ${config.apiKey}`);
+ calls.catalog += 1;
+ return catalog(init);
+ }
+ if (parsed.origin === 'https://fal.ai') {
+ assert.equal(parsed.pathname, '/api/openapi/queue/openapi.json');
+ assert.deepEqual([...parsed.searchParams], [['endpoint_id', model]]);
+ assert.equal(init.headers, undefined, 'the public schema never receives credentials');
+ calls.direct += 1;
+ return direct(init);
+ }
+ assert.equal(parsed.origin, 'https://queue.fal.run', 'never follow a schema reference or returned host');
+ assert.equal(init.headers.Authorization, `Key ${config.apiKey}`);
+ if (init.method === 'POST') {
+ calls.submits.push(JSON.parse(init.body));
+ return json({ request_id: 'one',
+ status_url: `https://queue.fal.run/${model}/requests/one/status`,
+ response_url: `https://queue.fal.run/${model}/requests/one`,
+ });
+ }
+ if (parsed.pathname.endsWith('/status')) return json({ status: 'COMPLETED' });
+ return json({ video: { url: 'https://v3b.fal.media/generated.mp4' } });
+ };
+ return { calls, fetchImpl };
+}
+
+const comfyConfig = { provider: 'comfyrouter', apiKey: 'private-comfy-key', videoModel: 'minimax/hailuo-02' };
+function comfyApi(metadata) {
+ const calls = { schemas: 0, submits: [] };
+ const base = `https://api.comfy.org/v2/models/${comfyConfig.videoModel}`;
+ const fetchImpl = async (url, init = {}) => {
+ assert.equal(init.credentials, 'omit');
+ assert.equal(init.redirect, 'error');
+ assert.equal(init.headers['X-API-Key'], comfyConfig.apiKey);
+ if (url === `${base}/openapi.json`) { calls.schemas += 1; return metadata(init); }
+ assert.ok(url.startsWith(`${base}/requests`), 'Comfy metadata has no external fallback');
+ if (init.method === 'POST') {
+ calls.submits.push(JSON.parse(init.body));
+ return json({ request_id: 'one' });
+ }
+ if (url.endsWith('/status')) return json({ status: 'COMPLETED' });
+ return json({ video: { url: 'https://media.example/generated.mp4' } });
+ };
+ return { calls, fetchImpl };
+}
+
+for (const build of ['chrome', 'firefox']) {
+ const { runMediaGeneration } = await import(`../src/${build}/src/agent/generative-media.js`);
+ const run = (mock, extra = {}) => runMediaGeneration({
+ prompt: 'A leaf moves across a grid.', video: { duration: 8 }, config,
+ fetchImpl: mock.fetchImpl, pollIntervalMs: 1, ...extra,
+ });
+
+ test(`${build}: bounded fal catalog avoids a redundant schema request`, async () => {
+ const mock = api({ direct: () => assert.fail('public schema is unnecessary') });
+ await run(mock);
+ assert.deepEqual([mock.calls.catalog, mock.calls.direct, mock.calls.submits[0].duration], [1, 0, 8]);
+ });
+
+ test(`${build}: fal public schema recovers missing, failed and unbounded catalog expansion`, async () => {
+ const cases = [
+ () => json({ models: [] }),
+ () => listing({ error: { code: 'expansion_failed', message: 'Schema lookup failed' } }),
+ () => listing(schema({ duration: { type: 'integer', default: 5 } })),
+ () => json({ error: { message: 'Not found' } }, 404),
+ () => json({ error: { message: 'Temporarily unavailable' } }, 503),
+ () => { throw new TypeError('Network read failed'); },
+ () => new Response('{broken', { headers: { 'Content-Type': 'application/json' } }),
+ () => listing(schema({ duration: { $ref: 'https://untrusted.example/Duration' } })),
+ ];
+ for (const catalog of cases) {
+ const mock = api({ catalog, direct: () => json(schema({ duration: { type: 'string', enum: ['5', '15'] } })) });
+ await run(mock, { video: { duration: 15 } });
+ assert.equal(mock.calls.direct, 1);
+ assert.equal(mock.calls.submits[0].duration, '15', 'use the public schema duration encoding');
+ }
+ });
+
+ test(`${build}: an incomplete public schema retains catalog constraints`, async () => {
+ const catalog = () => listing(schema({
+ duration: { type: 'integer', default: 8 },
+ resolution: { type: 'string', enum: ['720p'] },
+ }));
+ for (const direct of [
+ () => json(schema({ duration: { type: 'integer' } })),
+ () => json({ error: { message: 'Not found' } }, 404),
+ () => json({ error: { message: 'Unavailable' } }, 503),
+ () => { throw new TypeError('Public metadata unavailable'); },
+ ]) {
+ const mock = api({ catalog, direct });
+ await assert.rejects(run(mock, { video: { duration: 8, resolution: '1080p' } }), /Supported: 720p/);
+ assert.equal(mock.calls.direct, 1);
+ assert.equal(mock.calls.submits.length, 0);
+ }
+ const defaultOnly = api({
+ catalog: () => listing(schema({ duration: { type: 'string', default: '5' } })),
+ direct: () => json({}, 404),
+ });
+ await run(defaultOnly, { video: { estimated_duration: 5 } });
+ assert.equal(defaultOnly.calls.submits[0].duration, '5', 'retain the catalog default and its encoding');
+ });
+
+ test(`${build}: absent fal metadata does not prevent an explicit clip duration`, async () => {
+ const mock = api({ catalog: () => json({}, 404), direct: () => json({}, 404) });
+ await run(mock);
+ assert.equal(mock.calls.direct, 1);
+ assert.equal(mock.calls.submits[0].duration, 8);
+ });
+
+ test(`${build}: fal schema authorization failures remain errors`, async () => {
+ for (const status of [401, 403]) {
+ const mock = api({ catalog: () => json({ error: { message: 'Access denied' } }, status) });
+ await assert.rejects(run(mock), error => error.status === status);
+ assert.equal(mock.calls.submits.length, 0);
+ assert.equal(mock.calls.direct, 0);
+ }
+ });
+
+ test(`${build}: public fal schema access denial permits the default clip fallback`, async () => {
+ for (const status of [401, 403]) {
+ const mock = api({
+ catalog: () => listing({ error: { code: 'expansion_failed', message: 'Schema lookup failed' } }),
+ direct: () => json({ error: { message: 'Public metadata access denied' } }, status),
+ });
+ const result = await run(mock, { video: { estimated_duration: 8 } });
+ assert.equal(mock.calls.direct, 1);
+ assert.equal(mock.calls.submits[0].duration, 8);
+ assert.equal(result.fallbackClipSeconds, 8);
+ }
+ });
+
+ test(`${build}: Stop during either fal schema source prevents generation`, async () => {
+ for (const source of ['catalog', 'direct']) {
+ const controller = new AbortController();
+ const reason = new DOMException('Stopped by user', 'AbortError');
+ const cancelled = () => { controller.abort(reason); return json(bounded); };
+ const mock = api(source === 'catalog' ? { catalog: cancelled }
+ : { catalog: () => listing(null), direct: cancelled });
+ await assert.rejects(run(mock, { signal: controller.signal }), error => error === reason);
+ assert.equal(mock.calls.submits.length, 0);
+ assert.equal(mock.calls.direct, source === 'catalog' ? 0 : 1);
+ }
+ const controller = new AbortController();
+ controller.abort();
+ const mock = api();
+ await assert.rejects(run(mock, { signal: controller.signal }), { name: 'AbortError' });
+ assert.deepEqual(mock.calls, { catalog: 0, direct: 0, submits: [] });
+ });
+
+ test(`${build}: a fal metadata timeout exhausts one lookup budget and permits explicit duration`, async () => {
+ const mock = api({ catalog: init => new Promise((_, reject) => {
+ init.signal.addEventListener('abort', () => reject(init.signal.reason), { once: true });
+ }) });
+ await run(mock, { timeoutMs: 50 });
+ assert.equal(mock.calls.direct, 0, 'do not start a fresh metadata deadline');
+ assert.equal(mock.calls.submits[0].duration, 8);
+ });
+
+ test(`${build}: Comfy Router metadata failures allow the default clip fallback`, async () => {
+ for (const metadata of [
+ () => json({}, 404),
+ () => json({ error: { message: 'Unavailable' } }, 503),
+ () => { throw new TypeError('Network read failed'); },
+ () => new Response('{broken'),
+ ]) {
+ const mock = comfyApi(metadata);
+ const result = await run(mock, { config: comfyConfig, video: { estimated_duration: 8 } });
+ assert.equal(mock.calls.schemas, 1);
+ assert.equal(mock.calls.submits[0].duration, 8);
+ assert.equal(result.fallbackClipSeconds, 8);
+ }
+ });
+
+ test(`${build}: Comfy Router metadata authorization and cancellation still stop generation`, async () => {
+ for (const status of [401, 403]) {
+ const mock = comfyApi(() => json({ error: { message: 'Access denied' } }, status));
+ await assert.rejects(run(mock, { config: comfyConfig }), error => error.status === status);
+ assert.equal(mock.calls.submits.length, 0);
+ }
+ const controller = new AbortController();
+ const reason = new DOMException('Stopped by user', 'AbortError');
+ const mock = comfyApi(() => { controller.abort(reason); return json(bounded); });
+ await assert.rejects(run(mock, { config: comfyConfig, signal: controller.signal }), error => error === reason);
+ assert.equal(mock.calls.submits.length, 0);
+ });
+}
diff --git a/test/generative-media-video.mjs b/test/generative-media-video.mjs
index 4a90a9494..6907a0517 100644
--- a/test/generative-media-video.mjs
+++ b/test/generative-media-video.mjs
@@ -156,6 +156,12 @@ const falConfig = (model, settings = {}) => ({ provider: 'fal', apiKey: 'fal-key
function falApi({ model, schema, clip = () => fixtures['h264-aac-1'], fail = () => false }) {
const calls = { schemas: 0, submits: [], downloads: [], cancels: [] };
const fetchImpl = async (url, init = {}) => {
+ if (url.startsWith('https://fal.ai/api/openapi/queue/openapi.json?')) {
+ assert.equal(init.headers?.Authorization, undefined);
+ assert.equal(init.credentials, 'omit');
+ assert.equal(new URL(url).searchParams.get('endpoint_id'), model);
+ return json({ error: 'No public schema in this fixture' }, 404);
+ }
if (url.startsWith('https://v3b.fal.media/')) {
// A hosted clip is downloaded without the key.
assert.equal(init.headers, undefined);
@@ -542,12 +548,10 @@ for (const build of ['chrome', 'firefox']) {
assert.equal(media.planVideoClips(scenes(8, 15), grok.supported_durations).total, 120);
assert.throws(() => media.planVideoClips(scenes(9, 15), grok.supported_durations), /up to 120 s/);
assert.throws(() => media.planVideoClips(scenes(25, 4), grok.supported_durations), /at most 24/);
- assert.throws(() => media.planVideoClips(scenes(2, 5), null), error => {
- assert.match(error.message, /does not list the clip lengths/);
- assert.match(error.message, /WebBrain can join scene clips/);
- assert.doesNotMatch(error.message, /Generate one clip instead/);
- return true;
+ assert.deepEqual(media.planVideoClips(scenes(2, 5), null), {
+ clips: scenes(2, 8), total: 16,
});
+ assert.throws(() => media.planVideoClips(scenes(4, 15), null), /8 s as a fallback, not a confirmed model limit.*Retry with distinct scenes/);
assert.throws(() => media.planVideoClips([{ prompt: ' ', duration: 5 }], [5]), /Scene 1 needs a prompt/);
});
@@ -609,7 +613,7 @@ for (const build of ['chrome', 'firefox']) {
: videoApi({ entry: { ...grok, supported_durations: undefined } });
await assert.rejects(media.runMediaGeneration({ prompt: 'Explainer', video: { estimated_duration: 60 },
config: provider === 'fal' ? falConfig(h3Max) : config, fetchImpl: api.fetchImpl, pollIntervalMs: 1 }),
- /cannot determine a supported clip length for the requested 60 s video/);
+ /1:00 video needs multiple clips with WebBrain's 8 s fallback.*keep estimated_duration: 60/);
assert.equal(api.calls.submits.length, 0, `${provider}: an unknown length does not use a default clip`);
}
});
@@ -784,11 +788,6 @@ for (const build of ['chrome', 'firefox']) {
[h3Max, { resolution: '4K' }, /does not support resolution "4K"\. Supported: 480P, 768P, 1080P/],
[pro, { duration: 6 }, /This video model does not take duration/],
[pro, { aspect_ratio: '16:9' }, /This video model does not take aspect_ratio/],
- [pro, { scenes: scenes(2, 5), estimated_duration: 10 }, /does not list the clip lengths it supports/],
- // fal.ai publishes no schema for some models, and knows none for an unknown ID.
- [{ id: 'fal-ai/kling-video/v2.1/standard/text-to-video', schema: null }, { scenes: scenes(2, 5), estimated_duration: 10 }, /does not list the clip lengths/],
- [{ id: 'fal-ai/unknown', schema: 404 }, { scenes: scenes(2, 5), estimated_duration: 10 }, /does not list the clip lengths/],
- [{ id: 'fal-ai/busy', schema: 503 }, { duration: 5 }, /fal\.ai model schema failed \(HTTP 503\)/],
]) {
const api = falApi({ model: model.id, schema: model.schema });
await assert.rejects(media.runMediaGeneration({ prompt: 'Story', video, config: falConfig(model), fetchImpl: api.fetchImpl, pollIntervalMs: 1 }),
@@ -802,7 +801,7 @@ for (const build of ['chrome', 'firefox']) {
assert.deepEqual(unknown.calls.submits, [{ prompt: 'Story', duration: 5, aspect_ratio: '1:1' }]);
// Comfy Router: no schema for the model, or a read it refuses.
const comfyConfig = { provider: 'comfyrouter', apiKey: 'comfy-key', model: 'bfl/flux-2-pro', videoModel: 'minimax/hailuo-02' };
- for (const [schema, pattern] of [[404, /does not list the clip lengths/], [401, /Comfy Router model schema failed \(HTTP 401\)/]]) {
+ for (const [schema, pattern] of [[401, /Comfy Router model schema failed \(HTTP 401\)/]]) {
const api = comfyRouterApi({ model: 'minimax/hailuo-02', schema });
await assert.rejects(media.runMediaGeneration({ prompt: 'Story', video: { scenes: scenes(2, 5), estimated_duration: 10 }, config: comfyConfig,
fetchImpl: api.fetchImpl, pollIntervalMs: 1 }), pattern);
@@ -810,6 +809,47 @@ for (const build of ['chrome', 'firefox']) {
}
});
+ test(`${build}: missing clip metadata uses eight-second scenes and joins the full requested video`, async () => {
+ const eightSeconds = join.joinMp4Clips(Array(8).fill(fixtures['h264-aac-1'])).bytes;
+ for (const provider of ['fal', 'comfyrouter', 'openrouter']) {
+ const api = provider === 'fal' ? falApi({ model: h3Max.id, schema: null, clip: () => eightSeconds })
+ : provider === 'comfyrouter' ? comfyRouterApi({ model: rayTwo.id, schema: 404, clip: () => eightSeconds })
+ : videoApi({ entry: { ...grok, supported_durations: undefined }, clip: () => eightSeconds });
+ const selected = provider === 'fal' ? falConfig(h3Max)
+ : provider === 'comfyrouter' ? { provider, apiKey: 'comfy-key', videoModel: rayTwo.id } : config;
+ const result = await generate({ prompt: 'Consistent cast', scenes: scenes(8, 8), estimated_duration: 60 },
+ { config: selected, fetchImpl: api.fetchImpl, pollIntervalMs: 1 });
+ assert.equal(result.success, true, `${provider}: ${result.error}`);
+ assert.deepEqual(api.calls.submits.map(body => body.duration), Array(8).fill(8));
+ assert.deepEqual([result.estimatedSeconds, result.plannedSeconds, result.durationSeconds, result.fallbackClipSeconds],
+ [60, 64, 64, 8], provider);
+ assert.equal(result.durationStatus, 'enough');
+ assert.equal(join.mp4DurationSeconds(await storedBytes(result.mediaId)), 64);
+ const replay = journal.compactRunUiData('tool_result', { name: 'generate_image', result });
+ assert.equal(replay.result.fallbackClipSeconds, 8);
+ }
+ });
+
+ test(`${build}: fallback scenes respect published defaults and omit unsupported duration fields`, async () => {
+ for (const [duration, expected] of [[{ type: 'string', default: '6s' }, '6s'], [{ type: 'string' }, '8'], [null, undefined]]) {
+ const model = { id: 'provider/video', schema: falSchema(duration ? { duration } : {}) };
+ const api = falApi({ model: model.id, schema: model.schema });
+ const seconds = duration?.default ? 6 : 8;
+ const result = await generate({ prompt: 'Story', scenes: scenes(2, seconds), estimated_duration: seconds * 2 },
+ { config: falConfig(model), fetchImpl: api.fetchImpl, pollIntervalMs: 1 });
+ assert.equal(result.success, true, result.error);
+ assert.deepEqual(api.calls.submits.map(body => body.duration), [expected, expected]);
+ assert.equal(result.fallbackClipSeconds, seconds);
+ }
+ const rejected = falApi({ model: h3Max.id, schema: null, fail: () => 422 });
+ const failed = await generate({ prompt: 'Story', scenes: scenes(8, 8), estimated_duration: 60 },
+ { config: falConfig(h3Max), fetchImpl: rejected.fetchImpl, pollIntervalMs: 1 });
+ assert.equal(failed.success, false);
+ assert.match(failed.error, /422/);
+ assert.ok(rejected.calls.submits.length <= 3, 'a validation rejection is not retried');
+ assert.deepEqual(rejected.calls.downloads, []);
+ });
+
test(`${build}: Comfy Router scenes take each clip's length over the saved model input`, async () => {
const api = comfyRouterApi({ model: rayTwo.id, schema: rayTwo.schema, clip: index => fixtures[index ? 'h264-aac-2' : 'h264-aac-1'] });
const config = { provider: 'comfyrouter', apiKey: 'comfy-key', model: 'bfl/flux-2-pro', videoModel: rayTwo.id,
diff --git a/test/generative-media.mjs b/test/generative-media.mjs
index 82fff806d..b4cc03cbc 100644
--- a/test/generative-media.mjs
+++ b/test/generative-media.mjs
@@ -221,6 +221,41 @@ for (const build of ['chrome', 'firefox']) {
assert.equal(media.videoInputCapabilities(null), null);
});
+ test(`${build}: unbounded video duration schemas preserve valid defaults and input types`, () => {
+ const entryFor = duration => media.videoInputCapabilities({
+ openapi: '3.0.4', paths: { '/model': { post: { requestBody: { content: { 'application/json': { schema: {
+ type: 'object', properties: { prompt: { type: 'string' }, duration },
+ } } } } } } },
+ });
+ for (const [duration, expected, encoded] of [
+ [{ type: 'integer', default: 5 }, 5, 5],
+ [{ type: 'number', default: 6 }, 6, 6],
+ [{ type: 'string', default: '5s' }, 5, '5s'],
+ [{ type: 'string', default: '6.0' }, 6, '6.0'],
+ [{ allOf: [{ type: 'integer', default: 6 }, { minimum: 5, multipleOf: 2 }] }, 6, 6],
+ ]) {
+ const entry = entryFor(duration);
+ assert.equal(entry.supported_durations, undefined);
+ assert.equal(entry.default_duration, expected);
+ assert.equal(entry.encodeDuration(expected), encoded);
+ }
+ for (const duration of [
+ { type: 'integer', default: 0 }, { type: 'number', default: -5 }, { type: 'number', default: 5.5 },
+ { type: 'integer', default: '5' }, { type: 'string', default: 5 }, { type: 'string', default: 'auto' },
+ { type: 'integer', default: 5, minimum: 6 }, { type: 'integer', default: 5, multipleOf: 2 },
+ { type: 'integer', default: Number.MAX_SAFE_INTEGER + 1 },
+ ]) assert.equal(entryFor(duration).default_duration, undefined, JSON.stringify(duration));
+ assert.equal(entryFor({ type: 'string' }).encodeDuration(8), '8');
+ assert.equal(entryFor({ type: ['string', 'null'] }).encodeDuration(8), '8');
+ assert.equal(entryFor({ anyOf: [{ type: 'string' }, { type: 'null' }] }).encodeDuration(8), '8');
+ assert.equal(entryFor({ type: 'integer' }).encodeDuration(8), 8);
+ assert.equal(entryFor({ anyOf: [{ type: 'string' }, { type: 'number' }] }).encodeDuration(8), 8);
+ assert.equal(entryFor({ type: 'string', default: '5s' }).encodeDuration(8), '8', 'no units are invented');
+ const bounded = entryFor({ type: 'integer', minimum: 5, maximum: 6, default: 5 });
+ assert.deepEqual(bounded.supported_durations, [5, 6]);
+ assert.equal(bounded.default_duration, undefined, 'bounded durations remain authoritative');
+ });
+
test(`${build}: OpenRouter Image API returns inline media without a chat request`, async () => {
const calls = [];
const result = await media.runMediaGeneration({ prompt: 'red apple', config: { provider: 'openrouter', apiKey: 'or-key', model: 'google/gemini-2.5-flash-image' }, fetchImpl: async (url, init) => {