Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

Drop fal LTX-2.3; OmniHuman is the fal music-driven model

Claude7cbee8a716e7
diff --git a/client/src/components/musicVideo/SceneCard.test.jsx b/client/src/components/musicVideo/SceneCard.test.jsx
index 2e54b6e68f..2c99944d4c 100644
--- a/client/src/components/musicVideo/SceneCard.test.jsx
+++ b/client/src/components/musicVideo/SceneCard.test.jsx
@@ -62,7 +62,7 @@ describe('music-driven performance', () => {
expect(onSave).toHaveBeenLastCalledWith('s1', { shotMode: 'performance', performanceStyle: 'music' });
expect(shot.value).toBe('music');
// The shot's own 3.5s, not lip-sync's 5s minimum window.
- expect(screen.getByTestId('performance-plan').textContent).toMatch(/^Music-driven via fal\.ai LTX-2\.3 audio-to-video \(fal-ai\/ltx-2\.3-22b\/audio-to-video\) · song 0:10–0:13 · est\. \$0\.30 \(3\.5s at 1080p\)/);
+ expect(screen.getByTestId('performance-plan').textContent).toMatch(/^Music-driven via fal\.ai OmniHuman v1\.5 \(audio-driven person\) \(fal-ai\/bytedance\/omnihuman\/v1\.5\) · song 0:10–0:13 · est\. \$0\.56 \(3\.5s at 1080p\)/);
fireEvent.change(shot, { target: { value: 'performance' } });
expect(onSave).toHaveBeenLastCalledWith('s1', { shotMode: 'performance', performanceStyle: null });
expect(screen.getByTestId('performance-plan').textContent).toMatch(/^Lip-sync via/);
diff --git a/client/src/components/musicVideo/VideoRenderSettings.jsx b/client/src/components/musicVideo/VideoRenderSettings.jsx
index e15dd27580..b99c4e1cc7 100644
--- a/client/src/components/musicVideo/VideoRenderSettings.jsx
+++ b/client/src/components/musicVideo/VideoRenderSettings.jsx
@@ -244,21 +244,26 @@ export default function VideoRenderSettings({ videoSettings, generating }) {
/>
Transcript-guided lip-sync
</label>
- <label htmlFor="mv-fal-music-model" className="sr-only">fal.ai music-driven model</label>
- <select
- id="mv-fal-music-model"
- value={settings.falMusicModelId || ''}
- onChange={(e) => change({ falMusicModelId: e.target.value || null, falMusicResolution: null })}
- disabled={locked}
- title="Audio-to-video model for music-driven performance shots: the scene's song slice drives the whole frame's motion, not just a mouth"
- className={`${falSelectCls} sm:max-w-[280px]`}
- >
- {musicModels.map((lane) => (
- <option key={lane.modelId} value={lane.modelId === FAL_AUDIO_VIDEO_MODEL ? '' : lane.modelId}>
- {`Music-driven · ${getFalVideoModel(lane.modelId).label} · ${describeFalVideoRate(lane.modelId, lane.defaultResolution)}${lane.modelId === FAL_AUDIO_VIDEO_MODEL ? ' (default)' : ''}`}
- </option>
- ))}
- </select>
+ {/* The model picker appears once there is a choice; the resolution picker always does. */}
+ {musicModels.length > 1 && (
+ <>
+ <label htmlFor="mv-fal-music-model" className="sr-only">fal.ai music-driven model</label>
+ <select
+ id="mv-fal-music-model"
+ value={settings.falMusicModelId || ''}
+ onChange={(e) => change({ falMusicModelId: e.target.value || null, falMusicResolution: null })}
+ disabled={locked}
+ title="Audio-to-video model for music-driven performance shots: the scene's song slice drives the whole frame's motion, not just a mouth"
+ className={`${falSelectCls} sm:max-w-[280px]`}
+ >
+ {musicModels.map((lane) => (
+ <option key={lane.modelId} value={lane.modelId === FAL_AUDIO_VIDEO_MODEL ? '' : lane.modelId}>
+ {`Music-driven · ${getFalVideoModel(lane.modelId).label} · ${describeFalVideoRate(lane.modelId, lane.defaultResolution)}${lane.modelId === FAL_AUDIO_VIDEO_MODEL ? ' (default)' : ''}`}
+ </option>
+ ))}
+ </select>
+ </>
+ )}
<label htmlFor="mv-fal-music-resolution" className="sr-only">fal.ai music-driven resolution</label>
<select
id="mv-fal-music-resolution"
diff --git a/server/lib/README.md b/server/lib/README.md
index 3db6272fef..cceb194705 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -321,11 +321,11 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `cursor.js` | Cursor Agent (`cursor-agent`) provider helpers — the `CURSOR_COMMAND` binary constant, the `isCursorCommand` predicate, and the `ensureCursorHeadlessArgs` (`--print --force`, folding any pinned reasoning effort into `--model`) / `ensureCursorTuiArgs` (`--force`) argv builders. `--force` is load-bearing beyond approvals: it also clears cursor's workspace-trust gate, which otherwise EXITS a headless run before any work happens. The prompt rides raw stdin (like claude/codex, unlike grok/kimi), and cursor needs no configured-default sentinel — its `auto` router is a real model id passed straight to `--model`. |
| `grok.js` | xAI Grok Build (`grok`) provider helpers — id/endpoint constants (`GROK_API_ID`/`GROK_CLI_ID`/`GROK_TUI_ID`/`GROK_API_ENDPOINT`), `isGrokCommand`/`isGrokCliProvider`/`isGrokTuiProvider` predicates, `ensureGrokHeadlessArgs`/`ensureGrokTuiArgs` argv builders (grok reads its prompt from `--prompt-file /dev/stdin`, not raw stdin; model selection uses the `GROK_CONFIGURED_DEFAULT` sentinel in `providerModels.js` so PortOS omits `--model` like Antigravity), and `prepareGrokPromptFile` (Windows temp-file delivery fallback; `cleanup({ throwOnError: true })` reports cleanup failure to an owning lifecycle). |
| `filmStyleGrammars.js` | Curated film style grammar catalog (`FILM_STYLE_GRAMMARS`, eight code-rendered media with rendering, colour, type, motion, camera, sound and native-move guidance), `getFilmStyleGrammar`, `summarizeFilmStyleGrammar` (picker projection) and `knownFilmStyleIdSchema` (request-field id check) and `renderFilmStyleGrammarPrompt(id, { parts })` — one prompt section capped at `FILM_STYLE_PROMPT_MAX_CHARS`; unknown id or bad parts is a 400. |
-| `falVideoModels.js` | Curated fal.ai VIDEO model catalog — `FAL_VIDEO_MODELS` (Hailuo-02 standard t2v/i2v default, MiniMax H3 Max/Max Turbo i2v, H3 reference-to-video, H3 lip-sync, LTX-2.3 and OmniHuman v1.5 audio-to-video (`FAL_AUDIO_VIDEO_MODELS`, kind `a2v`: the supplied audio drives the whole frame's motion), Seedance 2.5 t2v/i2v/reference, Kling v3 Pro i2v, Veo 3.1 Fast i2v) with per-model duration rules (enum/range/audio), resolutions, aspect ratios, optional inputs and list prices verified on `FAL_VIDEO_PRICING_VERIFIED_AT`; `getFalVideoModel()`, `coerceFalVideoSeconds()` (round UP to a covering length), `resolveFalVideoResolution()`, `buildFalVideoRequest()` (the exact queue body per model; null for an uncurated id, which keeps the legacy body), `estimateFalVideoCostUsd()` (null = unknown), `describeFalVideoRate()`. Dependency-free; the client re-exports it. |
+| `falVideoModels.js` | Curated fal.ai VIDEO model catalog — `FAL_VIDEO_MODELS` (Hailuo-02 standard t2v/i2v default, MiniMax H3 Max/Max Turbo i2v, H3 reference-to-video, H3 lip-sync, OmniHuman v1.5 audio-to-video (`FAL_AUDIO_VIDEO_MODELS`, kind `a2v`: the supplied audio drives the figure's whole motion; LTX audio-to-video runs locally, never on fal), Seedance 2.5 t2v/i2v/reference, Kling v3 Pro i2v, Veo 3.1 Fast i2v) with per-model duration rules (enum/range/audio), resolutions, aspect ratios, optional inputs and list prices verified on `FAL_VIDEO_PRICING_VERIFIED_AT`; `getFalVideoModel()`, `coerceFalVideoSeconds()` (round UP to a covering length), `resolveFalVideoResolution()`, `buildFalVideoRequest()` (the exact queue body per model; null for an uncurated id, which keeps the legacy body), `estimateFalVideoCostUsd()` (null = unknown), `describeFalVideoRate()`. Dependency-free; the client re-exports it. |
| `grokVideoClip.js` | Clip lengths grok's `image_to_video` actually delivers — `GROK_VIDEO_DURATIONS` (`[6, 10]`, measured in #3022: a 2s/3s request returns the same 6.04s clip), `GROK_VIDEO_DEFAULT_DURATION`, `resolveGrokDuration()` (validate an already-grok-shaped request, else default), and `nearestGrokDuration()` (round a length authored against another backend's continuous contract UP to the shortest clip that covers it — for translating a CD/commission `targetDurationSeconds`). Dependency-free so `validation.js` / `routes/videoGen.js` can derive their schemas from the same list the services gate on. |
| `fontMetadata.js` | `fontContainer(buf)` (sfnt / woff2 / null) and `fontFamilyName(buf)` — a font file's family name read from its `name` table (TrueType, OpenType, and WOFF2's brotli stream) with no font library; null for anything unreadable (#10345). |
| `musicVideoCoverOverlay.js` | Music Video cover lettering (#10345): `COVER_DESIGN_OPTIONS` / `DEFAULT_COVER_DESIGN`, `normalizeCoverDesign(design, { fonts })` (accepts every stored design; a `font:<id>` typeface only when that uploaded font is listed), `coverOverlaySvg({ title, tag, design, widths, fonts })` and the `coverFontSpec` / `fontAttrs` / `titleLinesFor` / `tagTextFor` helpers the width probe reads. Pure; the server render (`coverArtCompose.js`) and the client's live Lettering preview run this one layout. |
-| `musicVideoShotTiming.js` | Music Video shot modes (#8977): `MUSIC_VIDEO_SHOT_MODES` (`cutaway`/`performance`), `isPerformanceScene`, `MUSIC_VIDEO_PERFORMANCE_STYLES` (`lipsync`/`music`) + `isLipSyncScene`, `selectedPerformanceInstruction(scene)` (the selected take's shot instruction, performance scenes only), `SOURCE_AUDIO_LIPSYNC` (verified source-audio lip-sync lanes — only fal.ai MiniMax H3 lip-sync, 5–30s per take, default 1080P), `SOURCE_AUDIO_MUSIC` (music-driven lanes — fal.ai audio-to-video models, picked by `videoSettings.falMusicModelId`, default LTX-2.3 at 1080p), `falSceneTake()` (the fal model/length/resolution a scene take requests and its estimated cost — shared by the board, the submit path and the autopilot spend cap) + `falTakeRequestFields(take)` (that take as `/api/video-gen` fields), `performanceCapability(backend, { scene, videoSettings })` (lip-sync lane, or the music-driven one for a music-driven scene), `performanceBlockedReason(backend)` (Grok is cutaway-only), `planPerformanceWindow()` (provider-bounded audio window + clip-relative edit in/out points; refuses over-long shots instead of relying on provider truncation), `planPerformanceRepair()` (complete measured suffix evidence, safe word-gap boundary, source/clip timebases and next spend), `clipRelativeCues()`, `grokCoverage(spanSec)` (covering 6/10s request, flags spans needing a split), `approximateMotionCues()`, `shotSplitLimit(scene, backend)` (per-take limit: lip-sync window or Grok's longest clip), `planShotSplit()` (contiguous pieces cut at lyric pauses → line/phrase boundaries → beats; one extra piece at most to avoid cutting a sung line). Dependency-free; the client re-exports it. |
+| `musicVideoShotTiming.js` | Music Video shot modes (#8977): `MUSIC_VIDEO_SHOT_MODES` (`cutaway`/`performance`), `isPerformanceScene`, `MUSIC_VIDEO_PERFORMANCE_STYLES` (`lipsync`/`music`) + `isLipSyncScene`, `selectedPerformanceInstruction(scene)` (the selected take's shot instruction, performance scenes only), `SOURCE_AUDIO_LIPSYNC` (verified source-audio lip-sync lanes — only fal.ai MiniMax H3 lip-sync, 5–30s per take, default 1080P), `SOURCE_AUDIO_MUSIC` (music-driven lanes — fal.ai audio-to-video models, picked by `videoSettings.falMusicModelId`, default OmniHuman v1.5 at 1080p), `falSceneTake()` (the fal model/length/resolution a scene take requests and its estimated cost — shared by the board, the submit path and the autopilot spend cap) + `falTakeRequestFields(take)` (that take as `/api/video-gen` fields), `performanceCapability(backend, { scene, videoSettings })` (lip-sync lane, or the music-driven one for a music-driven scene), `performanceBlockedReason(backend)` (Grok is cutaway-only), `planPerformanceWindow()` (provider-bounded audio window + clip-relative edit in/out points; refuses over-long shots instead of relying on provider truncation), `planPerformanceRepair()` (complete measured suffix evidence, safe word-gap boundary, source/clip timebases and next spend), `clipRelativeCues()`, `grokCoverage(spanSec)` (covering 6/10s request, flags spans needing a split), `approximateMotionCues()`, `shotSplitLimit(scene, backend)` (per-take limit: lip-sync window or Grok's longest clip), `planShotSplit()` (contiguous pieces cut at lyric pauses → line/phrase boundaries → beats; one extra piece at most to avoid cutting a sung line). Dependency-free; the client re-exports it. |
| `reactorStartingFrame.js` | Fits a reactor.inc `starting_frame` to the fast-h3 session canvas before it is uploaded. `readImageSize(path)` → `{width,height}` or `null` (never a guessed dimension), and `prepareReactorStartingFrame(sourceImagePath, requestedAspect, outputPath)` → `{ aspect, canvas, framePath, fittedPath }`: an absent `requestedAspect` DERIVES the canvas from the frame's own shape (`nearestReactorAspect`), then centre cover-crops the frame to that canvas's exact pixel size so reactor's own "fitted to the session canvas" step is a no-op. `fittedPath` is the temp file the caller deletes; `null` means the original was uploaded unchanged. A frame sharp cannot decode (or a resize that fails) passes through untouched on the default canvas rather than failing a render PortOS merely could not measure — reactor decodes more containers than sharp does. |
| `reactorVideoClip.js` | The reactor.inc `fast-h3` clip contract, dependency-free so the route schema, the FableLoom prompt compiler and the VideoGen picker all read one copy — `REACTOR_MAX_PROMPT_LENGTH` (800; the API rejects a longer prompt rather than truncating it), `REACTOR_MIN_CLIP_SECONDS` / `REACTOR_MAX_CLIP_SECONDS` (5.167–14.375, i.e. 124–345 frames at the `REACTOR_CLIP_FPS` 24 it delivers), `REACTOR_MAX_CLIP_ID_LENGTH`, and the picker list `REACTOR_CLIP_LENGTHS` + `REACTOR_DEFAULT_CLIP_LENGTH` + `reactorClipLengthLabel()` (the two exact endpoints plus every whole second between them, since the accepted range is continuous but starts and ends off a round number). Also the CANVAS contract `set_canvas` accepts: `REACTOR_CANVASES` (`16:9` 1344×768, `4:3` 1024×768, `1:1` 768×768, `9:16` 768×1344 — every one a 768px short edge, so the aspect string is the whole choice), `REACTOR_ASPECTS`, `REACTOR_DEFAULT_ASPECT`, `reactorCanvas(aspect)` and `nearestReactorAspect(width, height)` (closest canvas in log-ratio space; anything unmeasurable answers the default rather than deriving one). Mirrored to `client/src/lib/reactorVideoClip.js`, parity pinned by `reactorVideoClip.test.js`. |
| `kimi.js` | Moonshot AI Kimi Code (`kimi`) provider helpers — id constants (`KIMI_CLI_ID`/`KIMI_TUI_ID`), `isKimiCommand`/`isKimiCliProvider`/`isKimiTuiProvider` predicates, `ensureKimiHeadlessArgs` (model flag only — non-interactive mode is implicit in `--prompt`, and kimi has no `--print`/`--afk` and refuses approval flags alongside `--prompt`) / `ensureKimiTuiArgs` (`--yolo`) argv builders (model selection uses the `KIMI_CONFIGURED_DEFAULT` sentinel in `providerModels.js` so PortOS omits `--model` like Grok/Antigravity), and `prepareKimiPrompt` (delivers the prompt as the `--prompt <value>` argv, not stdin). |
diff --git a/server/lib/falVideoModels.js b/server/lib/falVideoModels.js
index 0307ac7c68..b938784200 100644
--- a/server/lib/falVideoModels.js
+++ b/server/lib/falVideoModels.js
@@ -39,7 +39,7 @@ export const FAL_VIDEO_KINDS = Object.freeze(['t2v', 'i2v', 'ref2v', 'lipsync',
export const FAL_DEFAULT_TEXT_VIDEO_MODEL = 'fal-ai/minimax/hailuo-02/standard/text-to-video';
export const FAL_DEFAULT_IMAGE_VIDEO_MODEL = 'fal-ai/minimax/hailuo-02/standard/image-to-video';
export const FAL_LIPSYNC_VIDEO_MODEL = 'minimax/h3-max/lip-sync/image-to-video';
-export const FAL_AUDIO_VIDEO_MODEL = 'fal-ai/ltx-2.3-22b/audio-to-video';
+export const FAL_AUDIO_VIDEO_MODEL = 'fal-ai/bytedance/omnihuman/v1.5';
const H3_RESOLUTIONS = Object.freeze(['480P', '768P', '1080P']);
const SEEDANCE_RESOLUTIONS = Object.freeze(['480p', '720p', '1080p']);
@@ -52,9 +52,6 @@ const H3_MAX_RATES = Object.freeze({ '480P': 0.05, '768P': 0.08, '1080P': 0.16 }
// per-second price for a 16:9 output; those published figures are used (they
// run ~2% above the nominal-pixel formula, i.e. the provider pads the canvas).
const SEEDANCE_PER_SEC_16X9 = Object.freeze({ '480p': 0.2205, '720p': 0.473, '1080p': 1.164 });
-// LTX-2.3 takes an explicit {width, height} canvas; PortOS names it by its
-// short side and keeps the requested aspect (see `ltxCanvas`).
-const LTX_SHORT_SIDES = Object.freeze({ '480p': 480, '720p': 720, '1080p': 1080 });
const seedance = (id, kind, label, extra) => ({
id, kind, label, family: 'seedance-2.5',
@@ -154,31 +151,10 @@ export const FAL_VIDEO_MODELS = Object.freeze([
seedance('bytedance/seedance-2.5/text-to-video', 't2v', 'Seedance 2.5 (text)', {
aspectRatio: { param: 'aspect_ratio', options: WIDE_ASPECTS, auto: 'auto' },
}),
- {
- // LTX-2.3 22B is a joint audio-video model: the supplied audio conditions
- // every frame, so motion lands on the music (and a singer's mouth on the
- // vocal) rather than being only a mouth layered over a still pose.
- id: FAL_AUDIO_VIDEO_MODEL, kind: 'a2v', label: 'LTX-2.3 audio-to-video', family: 'ltx-2.3',
- prompt: 'required',
- image: { param: 'image_url', required: false },
- endImage: { param: 'end_image_url' },
- // num_frames is 9–481 at 24fps, i.e. up to ~20s; with match_audio_length
- // the frame count follows the audio.
- audioInput: { param: 'audio_url', minSec: 1, maxSec: 20 },
- duration: { kind: 'audio' },
- resolution: { param: 'video_size', options: Object.keys(LTX_SHORT_SIDES), default: '720p', wire: 'canvas', fps: 24 },
- aspectRatio: null,
- negativePrompt: 'native',
- seed: 'seed',
- // audio_strength 1 keeps the supplied audio unmodified in the output.
- fixed: { match_audio_length: true, fps: 24, audio_strength: 1 },
- // "$0.001605 per megapixel of generated video data (width × height × frames)".
- pricing: { perMegapixel: 0.001605 },
- },
{
// Audio-driven human figure: expression, gesture and body movement follow
// the audio, not just the mouth. Expects a person in the frame.
- id: 'fal-ai/bytedance/omnihuman/v1.5', kind: 'a2v', label: 'OmniHuman v1.5 (audio-driven person)', family: 'omnihuman',
+ id: FAL_AUDIO_VIDEO_MODEL, kind: 'a2v', label: 'OmniHuman v1.5 (audio-driven person)', family: 'omnihuman',
prompt: 'optional',
image: { param: 'image_url', required: true },
// "Audio must be under 30s long for 1080p generation and under 60s long for 720p".
@@ -282,24 +258,6 @@ const wireDuration = (rule, sec) => (
const ratioOf = (r) => { const [w, h] = String(r).split(':').map(Number); return w / h; };
-const round32 = (n) => Math.max(32, Math.round(n / 32) * 32);
-
-/**
- * The explicit canvas a `wire: 'canvas'` model renders at `resolution`: the
- * named short side, the long side from the requested width/height (16:9 when
- * none is given), both on the 32-pixel grid the model works in.
- */
-function ltxCanvas(resolution, { width, height } = {}) {
- const short = LTX_SHORT_SIDES[resolution];
- if (!short) return null;
- const w = Number(width);
- const h = Number(height);
- const portrait = w > 0 && h > 0 && h > w;
- const aspect = w > 0 && h > 0 ? Math.max(w, h) / Math.min(w, h) : 16 / 9;
- const long = round32(short * aspect);
- return portrait ? { width: short, height: long } : { width: long, height: short };
-}
-
function pickAspectRatio(rule, { aspectRatio, width, height }) {
if (!rule) return null;
if (rule.options.includes(aspectRatio)) return aspectRatio;
@@ -345,15 +303,6 @@ export function estimateFalVideoCostUsd({ modelId, seconds, resolution, width, h
}
const res = resolveFalVideoResolution(model, resolution);
let perSec;
- if (model.pricing.perMegapixel) {
- const canvas = ltxCanvas(res, { width, height });
- if (!canvas) return null;
- const fps = model.resolution.fps;
- // The frames the audio needs, rounded UP to the 8n+1 counts LTX renders,
- // so the estimate never undershoots the bill.
- const frames = Math.ceil(Math.ceil(billedSec * fps) / 8) * 8 + 1;
- return round4((canvas.width * canvas.height * frames / 1e6) * model.pricing.perMegapixel);
- }
if (model.pricing.videoTokensPerSec16x9) {
perSec = model.pricing.videoTokensPerSec16x9[res] ?? null;
const w = Number(width);
@@ -373,12 +322,6 @@ export function describeFalVideoRate(modelId, resolution) {
const model = getFalVideoModel(modelId);
if (!model) return null;
const res = resolveFalVideoResolution(model, resolution);
- if (model.pricing.perMegapixel) {
- const canvas = ltxCanvas(res);
- if (!canvas) return null;
- const perSec = (canvas.width * canvas.height * model.resolution.fps / 1e6) * model.pricing.perMegapixel;
- return `≈$${perSec.toFixed(4)}/s at ${res} (16:9)`;
- }
if (model.pricing.videoTokensPerSec16x9) {
const perSec = model.pricing.videoTokensPerSec16x9[res];
return perSec == null ? null : `≈$${perSec.toFixed(4)}/s at ${res} (16:9)`;
@@ -457,7 +400,7 @@ export function buildFalVideoRequest({
const coercedSec = coerceFalVideoSeconds(model, seconds);
if (coercedSec != null) body.duration = wireDuration(model.duration, coercedSec);
const res = resolveFalVideoResolution(model, resolution);
- if (res) body[model.resolution.param] = model.resolution.wire === 'canvas' ? ltxCanvas(res, { width, height }) : res;
+ if (res) body[model.resolution.param] = res;
const aspect = pickAspectRatio(model.aspectRatio, { aspectRatio, width, height });
if (aspect) body[model.aspectRatio.param] = aspect;
const wantsAudio = generateAudio === true && !!model.audio;
diff --git a/server/lib/falVideoModels.test.js b/server/lib/falVideoModels.test.js
index b9ef23cf49..83960d72b0 100644
--- a/server/lib/falVideoModels.test.js
+++ b/server/lib/falVideoModels.test.js
@@ -77,26 +77,14 @@ describe('buildFalVideoRequest — per-family wire bodies', () => {
expect(req.estimatedCostUsd).toBe(3.84);
});
- it('LTX-2.3 audio-to-video: prompt + frame + audio on an explicit canvas that keeps the shot aspect; length follows the audio', () => {
- const req = buildFalVideoRequest({
- modelId: FAL_AUDIO_VIDEO_MODEL, prompt: 'drummer', negativePrompt: 'blur', imageUrl: 'i', audioUrl: 'a', audioSec: 3.735,
- width: 1920, height: 1080, resolution: '1080P',
- });
- expect(req.body).toEqual({
- prompt: 'drummer', negative_prompt: 'blur', image_url: 'i', audio_url: 'a', video_size: { width: 1920, height: 1080 },
- match_audio_length: true, fps: 24, audio_strength: 1,
- });
- expect(req.seconds).toBeNull();
- // 1920×1080 × 97 frames (3.735s at 24fps, up to LTX's 8n+1 count) at $0.001605/MP.
- expect(req.estimatedCostUsd).toBe(0.3228);
- expect(buildFalVideoRequest({ modelId: FAL_AUDIO_VIDEO_MODEL, prompt: 'x', audioUrl: 'a', width: 1080, height: 1920 }).body.video_size)
- .toEqual({ width: 720, height: 1280 });
- });
-
it('OmniHuman v1.5: frame + audio, an optional prompt and its own resolution enum', () => {
- expect(buildFalVideoRequest({ modelId: 'fal-ai/bytedance/omnihuman/v1.5', imageUrl: 'i', audioUrl: 'a', resolution: '720P' }).body)
- .toEqual({ image_url: 'i', audio_url: 'a', resolution: '720p' });
- expect(estimateFalVideoCostUsd({ modelId: 'fal-ai/bytedance/omnihuman/v1.5', seconds: 4 })).toBe(0.64);
+ expect(FAL_AUDIO_VIDEO_MODEL).toBe('fal-ai/bytedance/omnihuman/v1.5');
+ const req = buildFalVideoRequest({ modelId: FAL_AUDIO_VIDEO_MODEL, prompt: 'drummer', imageUrl: 'i', audioUrl: 'a', audioSec: 3.735, resolution: '720P' });
+ expect(req.body).toEqual({ prompt: 'drummer', image_url: 'i', audio_url: 'a', resolution: '720p' });
+ expect(req.seconds).toBeNull();
+ expect(estimateFalVideoCostUsd({ modelId: FAL_AUDIO_VIDEO_MODEL, seconds: 4 })).toBe(0.64);
+ // LTX audio-to-video runs locally; fal never offers it.
+ expect(FAL_VIDEO_MODELS.some((m) => /ltx/i.test(m.id))).toBe(false);
});
it('refuses a request the model cannot honor before anything is paid for; an uncurated id is not built', () => {
diff --git a/server/lib/musicVideoShotTiming.js b/server/lib/musicVideoShotTiming.js
index 95d483eae0..af8d856ecc 100644
--- a/server/lib/musicVideoShotTiming.js
+++ b/server/lib/musicVideoShotTiming.js
@@ -132,8 +132,10 @@ export const SOURCE_AUDIO_LIPSYNC = Object.freeze({
});
/**
- * Default output resolution of a music-driven take. Both audio-to-video
- * models offer 1080p, and LTX-2.3 bills by pixel at a few cents a second.
+ * Default output resolution of a music-driven take (OmniHuman's own default).
+ * LTX audio-to-video is deliberately not a fal lane: PortOS runs LTX-2.3 /
+ * LTX-2.5 locally, where Make › Supplied song audio conditions a cutaway on
+ * its own song slice (submitJob.js `_suppliedAudio`).
*/
const MUSIC_VIDEO_MUSIC_DEFAULT_RESOLUTION = '1080p';
@@ -158,7 +160,7 @@ const musicCapability = (model) => {
/**
* Verified music-driven lanes, keyed by backend then model id. The project's
* `videoSettings.falMusicModelId` picks one; absent (or a model this install
- * does not curate) = `FAL_AUDIO_VIDEO_MODEL` (LTX-2.3 audio-to-video).
+ * does not curate) = `FAL_AUDIO_VIDEO_MODEL` (OmniHuman v1.5).
*/
export const SOURCE_AUDIO_MUSIC = Object.freeze({
fal: Object.freeze(Object.fromEntries(FAL_AUDIO_VIDEO_MODELS.map((model) => [model.id, musicCapability(model)]))),
@@ -188,7 +190,7 @@ export function performanceBlockedReason(backend) {
return 'Grok video is cutaway-only: it cannot synchronize a singer to the song. Pin fal.ai for performance shots, or switch this scene to Cutaway.';
}
if (backend === 'local') {
- return 'Local video models cannot lip-sync to the song. Pin fal.ai for performance shots, or switch this scene to Cutaway.';
+ return 'Local video models cannot lip-sync to the song. Pin fal.ai for performance shots, or switch this scene to Cutaway — with Supplied song audio and a local LTX model, the scene\'s song slice still drives its motion.';
}
return 'Performance shots need a source-audio lip-sync provider — pin fal.ai as this project\'s video backend.';
}
diff --git a/server/lib/musicVideoShotTiming.test.js b/server/lib/musicVideoShotTiming.test.js
index 37ae9f7ef9..9506fee87d 100644
--- a/server/lib/musicVideoShotTiming.test.js
+++ b/server/lib/musicVideoShotTiming.test.js
@@ -95,8 +95,7 @@ describe('music-driven performance takes', () => {
it('render on the audio-to-video model for the shot itself, without lip-sync\'s 5s padding', () => {
const take = falSceneTake({ scene: drummer, videoSettings: {}, songDurationSec: 240 });
- expect(take).toMatchObject({ performance: true, modelId: 'fal-ai/ltx-2.3-22b/audio-to-video', seconds: 3.735, resolution: '1080p' });
- expect(take.costUsd).toBeGreaterThan(0);
+ expect(take).toMatchObject({ performance: true, modelId: 'fal-ai/bytedance/omnihuman/v1.5', seconds: 3.735, resolution: '1080p', costUsd: 0.5976 });
expect(falTakeRequestFields(take)).toEqual({ falResolution: '1080p' });
// The same shot as lip-sync pads to the route's 5s minimum.
expect(falSceneTake({ scene: { ...drummer, performanceStyle: null }, songDurationSec: 240 }).seconds).toBeCloseTo(5.05, 6);
@@ -105,10 +104,10 @@ describe('music-driven performance takes', () => {
it('follow the project pin (model and resolution) and fall back to the default for an uncurated pin', () => {
const pinned = falSceneTake({ scene: drummer, videoSettings: { falMusicModelId: 'fal-ai/bytedance/omnihuman/v1.5', falMusicResolution: '720p' }, songDurationSec: 240 });
expect(pinned).toMatchObject({ modelId: 'fal-ai/bytedance/omnihuman/v1.5', resolution: '720p', costUsd: 0.5976 });
- expect(falSceneTake({ scene: drummer, videoSettings: { falMusicModelId: 'example/newer-model' }, songDurationSec: 240 }).modelId)
- .toBe('fal-ai/ltx-2.3-22b/audio-to-video');
- expect(shotSplitLimit(drummer, 'fal')).toBeCloseTo(20 - PERFORMANCE_WINDOW_MARGIN_SEC, 6);
- expect(shotSplitLimit(drummer, 'fal', { falMusicModelId: 'fal-ai/bytedance/omnihuman/v1.5' })).toBeCloseTo(30 - PERFORMANCE_WINDOW_MARGIN_SEC, 6);
+ // A pin this install no longer curates (fal LTX-2.3, removed) falls back to the default.
+ expect(falSceneTake({ scene: drummer, videoSettings: { falMusicModelId: 'fal-ai/ltx-2.3-22b/audio-to-video' }, songDurationSec: 240 }).modelId)
+ .toBe('fal-ai/bytedance/omnihuman/v1.5');
+ expect(shotSplitLimit(drummer, 'fal')).toBeCloseTo(30 - PERFORMANCE_WINDOW_MARGIN_SEC, 6);
});
});
diff --git a/server/lib/musicVideoValidation.js b/server/lib/musicVideoValidation.js
index 7bb66d441c..08ae9ff873 100644
--- a/server/lib/musicVideoValidation.js
+++ b/server/lib/musicVideoValidation.js
@@ -156,7 +156,7 @@ export const musicVideoVideoSettingsSchema = z.object({
// Audio-to-video model and resolution for music-driven performance takes
// (scene.performanceStyle 'music'). Loosely validated like falModelId: a peer
// on a newer catalog may pin a model this install does not curate, which then
- // falls back to the default (LTX-2.3 audio-to-video).
+ // falls back to the default (OmniHuman v1.5).
falMusicModelId: z.string().min(1).max(200).nullable().optional(),
falMusicResolution: z.string().min(1).max(16).nullable().optional(),
generationMode: z.enum(['image', 'audioReactive', 'suppliedAudio']).optional(),
diff --git a/server/services/musicVideo/performanceShot.test.js b/server/services/musicVideo/performanceShot.test.js
index cc20b791db..524958cba9 100644
--- a/server/services/musicVideo/performanceShot.test.js
+++ b/server/services/musicVideo/performanceShot.test.js
@@ -311,9 +311,9 @@ describe.skipIf(!ffmpeg)('music-video performance shot through the fal lip-sync
expect(si.audioWindow).toEqual({ startSec: 20, endSec: 21.5, durationSec: 1.5 });
expect(si.edit).toEqual({ inSec: 0, outSec: 1.5, targetSec: 1.5 });
expect(si).toMatchObject({ performanceStyle: 'music', transcription: false,
- capability: { style: 'music', modelId: 'fal-ai/ltx-2.3-22b/audio-to-video', resolution: '1080p' } });
+ capability: { style: 'music', modelId: 'fal-ai/bytedance/omnihuman/v1.5', resolution: '1080p' } });
expect(si.audio.conditioning).toMatchObject({ source: 'master', filename: 'song.wav', selection: 'music-driven' });
- expect(prepared).toMatchObject({ modelId: 'fal-ai/ltx-2.3-22b/audio-to-video', resolution: '1080p', enableTranscription: false });
+ expect(prepared).toMatchObject({ modelId: 'fal-ai/bytedance/omnihuman/v1.5', resolution: '1080p', enableTranscription: false });
const samples = pcm(await readFile(prepared.audioFilePath));
expect(samples.readInt16LE(0)).toBe(30000);
@@ -340,29 +340,18 @@ describe.skipIf(!ffmpeg)('music-video performance shot through the fal lip-sync
});
expect(await waitForTerminal(job.jobId)).toMatchObject({ type: 'completed' });
expect(posts).toHaveLength(1);
- expect(posts[0].url).toBe('https://queue.fal.run/fal-ai/ltx-2.3-22b/audio-to-video');
+ expect(posts[0].url).toBe('https://queue.fal.run/fal-ai/bytedance/omnihuman/v1.5');
expect(posts[0].body).toMatchObject({
- prompt: 'drummer hits on the beat', video_size: { width: 1920, height: 1080 }, match_audio_length: true, fps: 24, audio_strength: 1,
+ prompt: 'drummer hits on the beat', resolution: '1080p',
});
expect(posts[0].body.image_url).toMatch(/^data:image\//);
expect(posts[0].body.audio_url).toMatch(/^data:audio\/wav;base64,/);
const { loadHistory } = await import('../videoGen/history.js');
- // 1920×1080 × 41 frames (1.5s at 24fps, up to LTX's 8n+1 count) at $0.001605/MP.
+ // 1.5s at $0.16/s.
expect((await loadHistory()).find((h) => h.id === job.jobId)).toMatchObject({
- modelId: 'fal:fal-ai/ltx-2.3-22b/audio-to-video', resolution: '1080p', estimatedCostUsd: 0.1365, audioDriven: true,
+ modelId: 'fal:fal-ai/bytedance/omnihuman/v1.5', resolution: '1080p', estimatedCostUsd: 0.24, audioDriven: true,
});
- // With no requested size the canvas follows the start frame's shape (a portrait plate stays portrait).
- execFileSync(ffmpeg, ['-v', 'error', '-f', 'lavfi', '-i', 'color=gray:s=540x960', '-frames:v', '1', '-y', join(PATHS.images, 'portrait.png')]);
- const again = await preparePerformanceShot({
- musicVideo: { projectId: 'mv-1', sceneId: 'mvs-1' }, backend: 'fal', sourceImagePath: join(PATHS.images, 'portrait.png'), mode: 'image',
- });
- const portraitJob = await fal.generateVideo({
- apiKey: 'test-key', modelId: again.modelId, resolution: again.resolution, prompt: 'drummer hits on the beat',
- sourceImagePath: join(PATHS.images, 'portrait.png'), audioFilePath: again.audioFilePath, shotInstruction: again.shotInstruction,
- });
- expect(await waitForTerminal(portraitJob.jobId)).toMatchObject({ type: 'completed' });
- expect(posts[1].body.video_size).toEqual({ width: 1080, height: 1920 });
}, 60_000);
it('uses the full span of a long shot in one take, at the project\'s lip-sync resolution pin', async () => {
diff --git a/server/services/videoGen/fal.js b/server/services/videoGen/fal.js
index 5ba979669d..3e94cb185e 100644
--- a/server/services/videoGen/fal.js
+++ b/server/services/videoGen/fal.js
@@ -189,15 +189,8 @@ export async function generateVideo({
const audioSec = audioFilePath
? (shotInstruction?.audioWindow?.durationSec || await probeVideoDuration(audioFilePath).catch(() => null))
: null;
- // A model that takes an explicit canvas (LTX-2.3) keeps the shot's shape:
- // with no requested size, read it off the start frame rather than assume 16:9.
- let canvas = { width, height };
- if (catalogModel?.resolution?.wire === 'canvas' && !(Number(width) > 0 && Number(height) > 0) && sourceImagePath) {
- const { readImageSize } = await import('../../lib/reactorStartingFrame.js');
- canvas = (await readImageSize(sourceImagePath)) || canvas;
- }
const requestSpec = {
- modelId, prompt, negativePrompt, seconds: duration, audioSec, aspectRatio, width: canvas.width, height: canvas.height, resolution,
+ modelId, prompt, negativePrompt, seconds: duration, audioSec, aspectRatio, width, height, resolution,
generateAudio: generateAudio === true, enableTranscription: lipSync?.enableTranscription === true,
};
// Dry-run the curated body with placeholder media before anything is read or