Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
Lip-sync performance shots locally on LTX-2.5 audio-to-video
Claude222431aaf462
diff --git a/client/src/lib/musicVideoSceneAttention.test.js b/client/src/lib/musicVideoSceneAttention.test.jsindex cbb5a706c0..6c3583c32d 100644--- a/client/src/lib/musicVideoSceneAttention.test.js+++ b/client/src/lib/musicVideoSceneAttention.test.js@@ -35,7 +35,7 @@ describe('sceneAttention', () => {it('flags a blocked performance, pending candidates, failures and an unverified plate', () => {const performance = { ...ready, shotMode: 'performance' };- expect(sceneAttention(performance, { lipSyncBackend: 'local' })).toContain('perf-blocked');+ expect(sceneAttention(performance, { lipSyncBackend: 'grok' })).toContain('perf-blocked');expect(sceneAttention({ ...ready, takes: [{ takeId: 't1', kind: 'image', assetId: 'g.png', status: 'candidate' }] })).toEqual(['candidates-pending']);expect(sceneAttention(ready, { failed: { video: { s: true } } })).toEqual(['last-failure']);const withIntent = { ...ready, direction: { actionContract: { purpose: 'Singer turns toward the door' } },diff --git a/docs/features/product-surfaces.md b/docs/features/product-surfaces.mdindex 16a5837569..2df49c0de3 100644--- a/docs/features/product-surfaces.md+++ b/docs/features/product-surfaces.md@@ -110,7 +110,7 @@ An end-to-end creative production suite for authors, worldbuilders, filmmakers,| **SongBook** | `/songbook` | Interactive chord charts, guitar tablature, lyric repertoire, and auto-scrolling practice sheet music. | — || **Creative Director** | `/creative-director` | Multi-scene episodic video production orchestrator: turn briefs into treatments, storyboards, render plans, and completed episodes. | — || **Creative Commissions** | `/creative-commission` | Standing briefs and scheduled/nightly autonomous media generation aligning with personal taste profiles and feedback. A universe and/or mood board can be picked as the style source: each run loads its style tags and reference images as the Creative Director's art-direction base, attaches those images to vision scene evaluation, and conditions local image renders on them. | — |-| **Music Video Studio** | `/music-video` | A project is six deep-linked steps (`/music-video/:projectId/:stage` — Song `setup`, Look `cast-sets`, Storyboard `board`, Make `produce`, Final render `review`, Publish `publish`; an old `compose` link opens Make), listed down the left with a state word and one fact each (a bottom icon bar on a phone). Each step opens with what "done" means and an always-visible checklist, and ends with its own approval (art on Look, the timed storyboard on Storyboard; an animated proof on Make is optional and never holds the final render). Approvals never require typed notes. A sticky header gives one status line ("Step 3 of 6: Storyboard needs you"), the one next action, an Autopilot button, paid-generation spend against the cap and **Project settings** — a drawer holding the render style and services, advanced audio tools, the autopilot brief and runs, and every development file, the external handoff and the making-of export; a **Needs attention** banner under that status line lists any state a half-finished workflow left on the server — an open revision, an interrupted Cast & Sets stage, an auto-review waiting for its revised sections, a final render nobody is watching — with its Resume / Continue / Cancel / Reattach inline, derived from the saved record so it survives a reload; a preview player is docked beside every step — the final render, the live composition document, each finished draft excerpt and, before anything is rendered, a storyboard animatic (the song under each shot's frame or card), picked from the dock (`?play=`) — and scene cards seek it. Audio-reactive video creation, beat-detection synchronization, scene choreography, tempo alignment, and audience-driven pre-production treatments that direct each shot. Scenes are cutaways or lip-synced performance shots; a performance shot renders only on a verified source-audio provider (fal.ai MiniMax H3 lip-sync, 5–14.8s song slices placed at an exact edit in-point, cut from an optional full-length vocal stem when one is attached — uploaded, or separated from the song with a locally installed demucs) — Grok video is cutaway-only. The autopilot kickoff analyzes the song, imports the linked track's lyric sheet (its `[Verse]`/`[Chorus]` headers and stage directions kept as lyric markers that name the analysis sections and steer shot framing), separates the vocal, aligns the known lyrics to the vocal stem with local MMS_FA CTC (downloaded on first use, a garbage token between lines absorbs instrumental gaps; standalone alignment asks for Demucs htdemucs_ft consent before creating a missing stem), builds a Cast & Sets check-in (one creative-direction call that interprets the song, then a canonical character sheet, expression and looks sheets, one empty plate per set and in-set tests on the image queue, each later image conditioned on the character sheet; assembled into a self-contained sheet saved as a development artifact), stops for the director's review unless the brief auto-approves it, then plans against the approved references. Development artifacts (check-in sheets, animatics, treatments, storyboards; HTML/Markdown/MP4/PNG/JPG) are versioned per project, reviewed with notes and approve/request-changes, and viewed in a sandboxed viewer at `/music-video/:projectId/:stage/dev/:artifactId` (the stage-less `/music-video/:projectId/dev/:artifactId` still opens it). Draft excerpts are reviewed and selectively revised by hand, or by an opt-in auto-review run bounded by director-set review and paid-generation limits (a frame-only look never passes motion or audio sync). A run started from the board generates its revised sections on the server too (#10014), routed from the project's saved image and video pins, so it completes with no tab open; a section that cannot be generated unattended (a synchronous image backend, a clip lane only the board can submit) parks the run as needing the director, with the reason named. A server-owned production run (`/api/music-video/:id/production-runs`, #9066) plans an empty board, generates missing frames and clips only on a director-allowed image/video provider+model pool, hands the continuous draft to auto-review and dispatches its revisions — within explicit generation, review and (priced routes only) dollar limits, without the browser open, and never resuming on its own after a restart; the song is the sole audio unless a sound-design bed is explicitly chosen to mix under it. A footage or composed render can intercut (#9290): shots are re-cut on the song's section energy (every 2 beats in the loudest quarter, up to every 8 in the quietest) and on sung word onsets, alternating the host shot — a lip-synced take stays in sync — with unshown footage from a cutaway in the same section, so the edit gets faster without generating more clips; an autopilot production run defaults to it unless the director chose one cut per shot. Lyric lines sung two or more times are hooks: the treatment's text cues give them a word-by-word build (#9291) — each word lands on its sung onset in heavy caps, the newest in the accent color. Draft excerpts can be re-framed as vertical or square social cuts with faded audio edges, from suggested hook windows (#9280). The **Autonomous** entry point (`docs/features/music-video-autonomous.md`) starts from one prompt instead of a chosen track: it writes the brief, lyrics and a mood board, makes the song in Suno through the PortOS Browser, and runs the production — with optional approval checkpoints, and a `music-video-autopilot` scheduled task that turns unused Brain ideas into videos. The Publish stage (#9281) builds a release kit from the final render: an X-sized 1080p, a 720p preview, a hook teaser, thumbnail candidates, lyric captions (SRT) and YouTube chapters. It also drafts every platform's copy in one director-triggered provider call from their own making-of notes, and every field stays editable. Posting (#9282) fills each platform's post (YouTube, Suno, an X thread, LinkedIn, Facebook, Shorts, TikTok, Instagram Reels, Reddit, Stacker News, Substack, DistroKid) in a new PortOS Browser tab with the director's signed-in sessions, shows a screenshot and summary of the filled draft, and posts only when the director presses Post for that draft; the posted links are kept on the project. Reddit defaults to a native video post in r/aivideo — title and flair, no body — through the new composer's own Video button (#9307). Platforms are opt-in (#9287): a "Where you post" list turns each one on (all start off) and optionally names the account posted as — X and Reddit drafts filled while signed in as someone else are refused. Copy is drafted and posts are offered only for enabled platforms. Each post (including one recorded by hand) can be rated good, mixed or poor with notes; ratings roll up per platform across projects and are passed to later copy drafts. | — |+| **Music Video Studio** | `/music-video` | A project is six deep-linked steps (`/music-video/:projectId/:stage` — Song `setup`, Look `cast-sets`, Storyboard `board`, Make `produce`, Final render `review`, Publish `publish`; an old `compose` link opens Make), listed down the left with a state word and one fact each (a bottom icon bar on a phone). Each step opens with what "done" means and an always-visible checklist, and ends with its own approval (art on Look, the timed storyboard on Storyboard; an animated proof on Make is optional and never holds the final render). Approvals never require typed notes. A sticky header gives one status line ("Step 3 of 6: Storyboard needs you"), the one next action, an Autopilot button, paid-generation spend against the cap and **Project settings** — a drawer holding the render style and services, advanced audio tools, the autopilot brief and runs, and every development file, the external handoff and the making-of export; a **Needs attention** banner under that status line lists any state a half-finished workflow left on the server — an open revision, an interrupted Cast & Sets stage, an auto-review waiting for its revised sections, a final render nobody is watching — with its Resume / Continue / Cancel / Reattach inline, derived from the saved record so it survives a reload; a preview player is docked beside every step — the final render, the live composition document, each finished draft excerpt and, before anything is rendered, a storyboard animatic (the song under each shot's frame or card), picked from the dock (`?play=`) — and scene cards seek it. Audio-reactive video creation, beat-detection synchronization, scene choreography, tempo alignment, and audience-driven pre-production treatments that direct each shot. Scenes are cutaways or lip-synced performance shots; a performance shot renders only on a verified source-audio provider (fal.ai MiniMax H3 lip-sync, 5–14.8s song slices placed at an exact edit in-point, cut from an optional full-length vocal stem when one is attached — uploaded, or separated from the song with a locally installed demucs; or free on local LTX-2.5 / LTX-2.3 dgrauet audio-to-video, up to 20s per take, with the song slice held fixed through both denoise passes) — Grok video is cutaway-only. The autopilot kickoff analyzes the song, imports the linked track's lyric sheet (its `[Verse]`/`[Chorus]` headers and stage directions kept as lyric markers that name the analysis sections and steer shot framing), separates the vocal, aligns the known lyrics to the vocal stem with local MMS_FA CTC (downloaded on first use, a garbage token between lines absorbs instrumental gaps; standalone alignment asks for Demucs htdemucs_ft consent before creating a missing stem), builds a Cast & Sets check-in (one creative-direction call that interprets the song, then a canonical character sheet, expression and looks sheets, one empty plate per set and in-set tests on the image queue, each later image conditioned on the character sheet; assembled into a self-contained sheet saved as a development artifact), stops for the director's review unless the brief auto-approves it, then plans against the approved references. Development artifacts (check-in sheets, animatics, treatments, storyboards; HTML/Markdown/MP4/PNG/JPG) are versioned per project, reviewed with notes and approve/request-changes, and viewed in a sandboxed viewer at `/music-video/:projectId/:stage/dev/:artifactId` (the stage-less `/music-video/:projectId/dev/:artifactId` still opens it). Draft excerpts are reviewed and selectively revised by hand, or by an opt-in auto-review run bounded by director-set review and paid-generation limits (a frame-only look never passes motion or audio sync). A run started from the board generates its revised sections on the server too (#10014), routed from the project's saved image and video pins, so it completes with no tab open; a section that cannot be generated unattended (a synchronous image backend, a clip lane only the board can submit) parks the run as needing the director, with the reason named. A server-owned production run (`/api/music-video/:id/production-runs`, #9066) plans an empty board, generates missing frames and clips only on a director-allowed image/video provider+model pool, hands the continuous draft to auto-review and dispatches its revisions — within explicit generation, review and (priced routes only) dollar limits, without the browser open, and never resuming on its own after a restart; the song is the sole audio unless a sound-design bed is explicitly chosen to mix under it. A footage or composed render can intercut (#9290): shots are re-cut on the song's section energy (every 2 beats in the loudest quarter, up to every 8 in the quietest) and on sung word onsets, alternating the host shot — a lip-synced take stays in sync — with unshown footage from a cutaway in the same section, so the edit gets faster without generating more clips; an autopilot production run defaults to it unless the director chose one cut per shot. Lyric lines sung two or more times are hooks: the treatment's text cues give them a word-by-word build (#9291) — each word lands on its sung onset in heavy caps, the newest in the accent color. Draft excerpts can be re-framed as vertical or square social cuts with faded audio edges, from suggested hook windows (#9280). The **Autonomous** entry point (`docs/features/music-video-autonomous.md`) starts from one prompt instead of a chosen track: it writes the brief, lyrics and a mood board, makes the song in Suno through the PortOS Browser, and runs the production — with optional approval checkpoints, and a `music-video-autopilot` scheduled task that turns unused Brain ideas into videos. The Publish stage (#9281) builds a release kit from the final render: an X-sized 1080p, a 720p preview, a hook teaser, thumbnail candidates, lyric captions (SRT) and YouTube chapters. It also drafts every platform's copy in one director-triggered provider call from their own making-of notes, and every field stays editable. Posting (#9282) fills each platform's post (YouTube, Suno, an X thread, LinkedIn, Facebook, Shorts, TikTok, Instagram Reels, Reddit, Stacker News, Substack, DistroKid) in a new PortOS Browser tab with the director's signed-in sessions, shows a screenshot and summary of the filled draft, and posts only when the director presses Post for that draft; the posted links are kept on the project. Reddit defaults to a native video post in r/aivideo — title and flair, no body — through the new composer's own Video button (#9307). Platforms are opt-in (#9287): a "Where you post" list turns each one on (all start off) and optionally names the account posted as — X and Reddit drafts filled while signed in as someone else are refused. Copy is drafted and posts are offered only for enabled platforms. Each post (including one recorded by hand) can be rated good, mixed or poor with notes; ratings roll up per platform across projects and are passed to later copy drafts. | — || **Sharing & Conflict Resolution** | `/sharing`, `/sharing/duplicates`, `/sharing/conflicts` | Collaborative asset exchange via cloud storage buckets (Google Drive, Dropbox, iCloud, Syncthing) with duplicate reconciliation and conflict recovery journals. | — |---diff --git a/scripts/generate_ltx2.py b/scripts/generate_ltx2.pyindex f1e5e1b282..91fe1503e3 100644--- a/scripts/generate_ltx2.py+++ b/scripts/generate_ltx2.py@@ -319,6 +319,51 @@ def _install_guided_denoise_teacache_patches() -> None:_install_guided_denoise_teacache_patches()+_A2V_FREEZE_STAGE2_AUDIO = False+_A2V_FREEZE_PATCH_OK = False+++def _install_a2v_stage2_audio_freeze() -> None:+ """Hold the supplied audio fixed through A2V's Stage-2 refine, as in Stage 1.++ `A2VidPipelineTwoStage` freezes the encoded song in Stage 1 (denoise_mask 0)+ but re-noises it at Stage 2's start sigma and denoises it with the video, so+ the refine pass attends to a drifting audio latent and the mouth loosens+ off the real vocal. The pipeline discards that refined audio anyway (it+ muxes the original file), so freezing it costs nothing. Stage 2 is the+ module's only `denoise_loop` call; the patch swaps its audio state for the+ clean latent with a zero mask while `run_a2v` has the flag set.+ """+ global _A2V_FREEZE_PATCH_OK+ try:+ import mlx.core as mx+ import ltx_pipelines_mlx.a2vid_two_stage as a2v_mod+ from ltx_core_mlx.conditioning.types.latent_cond import LatentState+ except Exception:+ return+ original_denoise_loop = getattr(a2v_mod, "denoise_loop", None)+ if original_denoise_loop is None:+ return++ def denoise_loop_with_frozen_audio(*args, **kwargs):+ audio_state = kwargs.get("audio_state")+ if _A2V_FREEZE_STAGE2_AUDIO and audio_state is not None:+ kwargs["audio_state"] = LatentState(+ latent=audio_state.clean_latent,+ clean_latent=audio_state.clean_latent,+ denoise_mask=mx.zeros_like(audio_state.denoise_mask),+ positions=audio_state.positions,+ attention_mask=audio_state.attention_mask,+ )+ return original_denoise_loop(*args, **kwargs)++ a2v_mod.denoise_loop = denoise_loop_with_frozen_audio+ _A2V_FREEZE_PATCH_OK = True+++_install_a2v_stage2_audio_freeze()++def _teacache_config(enabled: bool, patch_ok: bool, num_steps: int,thresh: float | None = None) -> dict:return {@@ -1829,7 +1874,7 @@ def run_a2v(args: argparse.Namespace) -> str:--image is optional: when provided, conditions the FIRST frame the sameway ImageToVideoPipeline does, so motion + audio sync to a chosen still."""- global _A2V_TC_CONFIG+ global _A2V_TC_CONFIG, _A2V_FREEZE_STAGE2_AUDIOAudioToVideoPipeline = _resolve_pipeline("A2VidPipelineTwoStage", "AudioToVideoPipeline")if not args.audio:raise SystemExit("--audio is required for a2v mode")@@ -1853,6 +1898,9 @@ def run_a2v(args: argparse.Namespace) -> str:audio_path, audio_start, padded_audio = a2v_audio_covering_clip(args.audio, args.audio_start, args.num_frames, args.fps, str(Path(args.output).parent))restore_image_rate = _patch_a2v_image_conditioning_rate(args.fps)+ _A2V_FREEZE_STAGE2_AUDIO = _A2V_FREEZE_PATCH_OK+ if _A2V_FREEZE_PATCH_OK:+ emit_status("Holding the supplied audio fixed through the refine pass")try:return _run_with_ltx_stepwise_preview(pipe, args, lambda: pipe.generate_and_save(prompt=args.prompt,@@ -1872,6 +1920,7 @@ def run_a2v(args: argparse.Namespace) -> str:finally:restore_image_rate()_A2V_TC_CONFIG = None+ _A2V_FREEZE_STAGE2_AUDIO = Falseif padded_audio:Path(padded_audio).unlink(missing_ok=True)diff --git a/scripts/generate_ltx2.test.js b/scripts/generate_ltx2.test.jsindex 22a4d1ad58..3babcdb65f 100644--- a/scripts/generate_ltx2.test.js+++ b/scripts/generate_ltx2.test.js@@ -569,6 +569,36 @@ describe.skipIf(!pyBin)('generate_ltx2.py', () => {].join('\n')}`);expect(output.trim()).toBe('/tmp/previews');});++ // The Stage-2 refine must hear the song the clip is lip-synced to, not a+ // re-noised copy of it: only while run_a2v sets the flag.+ it('holds the A2V audio latent fixed through the Stage-2 denoise loop', () => {+ const output = runPython([+ 'import sys, types',+ 'from dataclasses import dataclass',+ 'def mod(name, **attrs):',+ ' m = types.ModuleType(name); m.__path__ = []; m.__dict__.update(attrs); sys.modules[name] = m; return m',+ '@dataclass',+ 'class LatentState:',+ ' latent: object',+ ' clean_latent: object',+ ' denoise_mask: object',+ ' positions: object = None',+ ' attention_mask: object = None',+ 'mod("mlx"); mod("mlx.core", zeros_like=lambda a: ["zero"] * len(a))',+ 'mod("ltx_core_mlx"); mod("ltx_core_mlx.conditioning"); mod("ltx_core_mlx.conditioning.types")',+ 'mod("ltx_core_mlx.conditioning.types.latent_cond", LatentState=LatentState)',+ 'mod("ltx_pipelines_mlx")',+ 'a2v = mod("ltx_pipelines_mlx.a2vid_two_stage", denoise_loop=lambda **kw: kw["audio_state"])',+ importRunner,+ 'noised = LatentState(latent="noised", clean_latent="song", denoise_mask=[1, 1], positions="pos")',+ 'print(runner._A2V_FREEZE_PATCH_OK, a2v.denoise_loop(audio_state=noised).latent)',+ 'runner._A2V_FREEZE_STAGE2_AUDIO = True',+ 'held = a2v.denoise_loop(audio_state=noised)',+ 'print(held.latent, held.clean_latent, held.denoise_mask, held.positions)',+ ].join('\n'));+ expect(output.trim().split(/\r?\n/)).toEqual(['True noised', "song song ['zero', 'zero'] pos"]);+ });});// Speed profiles (#4875). The Node side decides the schedule declaratively;diff --git a/server/lib/musicVideoMediumPlan.test.js b/server/lib/musicVideoMediumPlan.test.jsindex 101b5dfaed..81c255e9ba 100644--- a/server/lib/musicVideoMediumPlan.test.js+++ b/server/lib/musicVideoMediumPlan.test.js@@ -41,7 +41,7 @@ describe('footage tools on a code-first project (#10450)', () => {});it('does not demand a Performance shot mode when the backend has no lip-sync lane', () => {- const p = project(['video:local'], 'local');+ const p = project(['video:grok'], 'grok');p.treatment = { revision: 1, appliedRevision: 1, shotDirections: planMusicVideoMedia(p, directions).map((d) => ({ ...d, mediumRationale: 'x' })) };expect(codeFirstProductionAssets(p).conflicts).toEqual([]);});diff --git a/server/lib/musicVideoShotTiming.js b/server/lib/musicVideoShotTiming.jsindex be670e235b..5e6d7faade 100644--- a/server/lib/musicVideoShotTiming.js+++ b/server/lib/musicVideoShotTiming.js@@ -121,6 +121,21 @@ export const MUSIC_VIDEO_LIPSYNC_DEFAULT_RESOLUTION = '1080P';* one lip-sync lane fits the other. Takes past 15s are billed at 1.2×; the* cost label says so, and a director who wants to avoid it splits shorter.*/+function localAudioLane(style) {+ return Object.freeze({+ provider: 'local',+ style,+ label: style === 'music' ? 'Local LTX-2.5 audio-to-video' : 'Local LTX-2.5 lip-sync',+ modelId: null,+ minAudioSec: 1,+ maxAudioSec: 20,+ transcription: false,+ resolutions: Object.freeze([]),+ defaultResolution: null,+ costLabel: 'Free, renders on this machine',+ });+}+export const SOURCE_AUDIO_LIPSYNC = Object.freeze({fal: Object.freeze({provider: 'fal',@@ -136,6 +151,12 @@ export const SOURCE_AUDIO_LIPSYNC = Object.freeze({// take's own estimate beside it (falSceneTake).costLabel: `${describeFalVideoRate(FAL_LIPSYNC_VIDEO_MODEL, MUSIC_VIDEO_LIPSYNC_DEFAULT_RESOLUTION)} — billed to your fal.ai account`,}),+ // LTX audio-to-video on this machine: the song slice is the audio latent the+ // picture is denoised against, held fixed through both passes+ // (scripts/generate_ltx2.py `_install_a2v_stage2_audio_freeze`). The model is+ // the project's local LTX pick, checked at submission; 20s keeps one take+ // inside what LTX-2.5 renders well and within a Mac's memory.+ local: localAudioLane('lipsync'),});/**@@ -169,6 +190,8 @@ const musicCapability = (model) => {* `videoSettings.falMusicModelId` picks one; absent (or a model this install* does not curate) = `FAL_AUDIO_VIDEO_MODEL` (OmniHuman v1.5).*/+const LOCAL_MUSIC_LANE = localAudioLane('music');+export const SOURCE_AUDIO_MUSIC = Object.freeze({fal: Object.freeze(Object.fromEntries(FAL_AUDIO_VIDEO_MODELS.map((model) => [model.id, musicCapability(model)]))),});@@ -181,6 +204,7 @@ export const SOURCE_AUDIO_MUSIC = Object.freeze({export function performanceCapability(backend, { scene = null, videoSettings = null } = {}) {if (!Object.prototype.hasOwnProperty.call(SOURCE_AUDIO_LIPSYNC, backend || '')) return null;if (scene?.performanceStyle !== 'music') return SOURCE_AUDIO_LIPSYNC[backend];+ if (backend === 'local') return LOCAL_MUSIC_LANE;const lanes = SOURCE_AUDIO_MUSIC[backend];const pinned = videoSettings?.falMusicModelId;return (pinned && Object.prototype.hasOwnProperty.call(lanes, pinned) ? lanes[pinned] : null) || lanes[FAL_AUDIO_VIDEO_MODEL];@@ -196,9 +220,6 @@ export function performanceBlockedReason(backend) {if (backend === 'grok') {return 'Grok video is cutaway-only: it cannot synchronize a singer to the song. Pin fal.ai for performance shots, or switch this scene to Cutaway.';}- if (backend === 'local') {- return 'Local video models cannot lip-sync to the song. Pin fal.ai for performance shots, or switch this scene to Cutaway — with Supplied song audio and a local LTX model, the scene\'s song slice still drives its motion.';- }return 'Performance shots need a source-audio lip-sync provider — pin fal.ai as this project\'s video backend.';}diff --git a/server/lib/musicVideoShotTiming.test.js b/server/lib/musicVideoShotTiming.test.jsindex d18df1343d..1c8c897f0c 100644--- a/server/lib/musicVideoShotTiming.test.js+++ b/server/lib/musicVideoShotTiming.test.js@@ -1,6 +1,7 @@import { describe, it, expect } from 'vitest';import {- falSceneTake, falTakeRequestFields, planShotSplit, shotSplitLimit, SOURCE_AUDIO_LIPSYNC, PERFORMANCE_WINDOW_MARGIN_SEC,+ falSceneTake, falTakeRequestFields, performanceCapability, planPerformanceWindow, planShotSplit, shotSplitLimit,+ SOURCE_AUDIO_LIPSYNC, PERFORMANCE_WINDOW_MARGIN_SEC,} from './musicVideoShotTiming.js';// The split planner is an algorithm with a real input matrix (#8977): which@@ -87,6 +88,22 @@ describe('shotSplitLimit', () => {});});+// Local LTX lip-sync (the song slice is the a2v audio): no provider minimum to+// pad to, and a shorter per-take ceiling than fal's.+describe('local LTX lip-sync lane', () => {+ it('plans the shot itself as the window, for lip-sync and music-driven scenes alike', () => {+ for (const performanceStyle of [null, 'music']) {+ const capability = performanceCapability('local', { scene: { shotMode: 'performance', performanceStyle } });+ expect(capability).toMatchObject({ provider: 'local', modelId: null, style: performanceStyle === 'music' ? 'music' : 'lipsync' });+ expect(planPerformanceWindow({ startSec: 2, endSec: 4.5, songDurationSec: 60, capability }))+ .toMatchObject({ ok: true, windowStartSec: 2, windowEndSec: 4.5, editInSec: 0 });+ }+ expect(planPerformanceWindow({ startSec: 0, endSec: 25, songDurationSec: 60, capability: performanceCapability('local') }))+ .toMatchObject({ ok: false, code: 'MUSIC_VIDEO_PERFORMANCE_TOO_LONG' });+ expect(shotSplitLimit({ shotMode: 'performance' }, 'local')).toBeCloseTo(20 - PERFORMANCE_WINDOW_MARGIN_SEC, 6);+ });+});+// Each failure variant is a distinct admission regression: evidence, word// gap, provider minimum, or timebase mismatch must never resubmit a prefix.describe('music-driven performance takes', () => {diff --git a/server/services/musicVideo/autonomousService.js b/server/services/musicVideo/autonomousService.jsindex 65d7f04466..049b70b86c 100644--- a/server/services/musicVideo/autonomousService.js+++ b/server/services/musicVideo/autonomousService.js@@ -1446,7 +1446,7 @@ const emptyStages = () => Object.fromEntries(AUTONOMOUS_STAGE_IDS.map((id) => [i* The project video settings the run's own video tool implies, so the plan and* production agree on what can render: its backend and local model, and Supplied* song audio when that local model takes a shot's song window. With no video tool- * it is local video, which plans no lip-sync shots.+ * it is local video, whose lip-sync shots render as LTX audio-to-video.*/async function briefVideoSettings(tools, models) {const route = autonomousPool(tools, models).find((entry) => entry.kind === 'video');diff --git a/server/services/musicVideo/autonomousService.test.js b/server/services/musicVideo/autonomousService.test.jsindex 21c41e4830..8d21641956 100644--- a/server/services/musicVideo/autonomousService.test.js+++ b/server/services/musicVideo/autonomousService.test.js@@ -1526,9 +1526,9 @@ describe('orchestrated mode (brief.orchestrator)', () => {await service.resumeAutonomousVideo('mv-auto', { addTools: ['video:local'] });expect(runOf().brief).toMatchObject({ tools: ['image:local', 'video:local'], stopBeforeVideo: false });expect(store.get('mv-auto').automation.tools).toEqual(['image:local', 'video:local']);- // The added local video tool sets the project's renderer, and a performance shot it cannot lip-sync plays as a cutaway.+ // The added local video tool sets the project's renderer, and a performance shot keeps lip-syncing on local LTX.expect(store.get('mv-auto').videoSettings).toMatchObject({ backend: 'local', generationMode: 'suppliedAudio' });- expect(store.get('mv-auto').scenes[0].shotMode).toBe('cutaway');+ expect(store.get('mv-auto').scenes[0].shotMode).toBe('performance');expect(doubles.resumeProduction).toHaveBeenCalledWith('mv-auto', 'mvpr-1', expect.objectContaining({pool: expect.arrayContaining([expect.objectContaining({ kind: 'video', mode: 'local' })]) }));expect(doubles.resumeProduction.mock.calls[0][2].stopBefore).toBeUndefined();diff --git a/server/services/musicVideo/performanceShot.js b/server/services/musicVideo/performanceShot.jsindex 49a6918ddc..d983a71e72 100644--- a/server/services/musicVideo/performanceShot.js+++ b/server/services/musicVideo/performanceShot.js@@ -287,8 +287,9 @@ export async function preparePerformanceShot({ musicVideo, backend, sourceImagePif (!capability) throw refuse(performanceBlockedReason(backend), 'MUSIC_VIDEO_PERFORMANCE_UNSUPPORTED');const musicDriven = capability.style === 'music';// The lip-sync route is image + source audio; any other semantic mode would- // ask it for something it does not do.- if (mode && mode !== 'image') {+ // ask it for something it does not do. Local LTX renders it as a2v, which+ // the audio-reactive local lane already asks for.+ if (mode && mode !== 'image' && !(mode === 'a2v' && capability.provider === 'local')) {throw refuse(`A performance shot renders from its reference frame and the song — mode '${mode}' is not supported.`, 'MUSIC_VIDEO_PERFORMANCE_MODE_UNSUPPORTED');}diff --git a/server/services/musicVideo/performanceShot.test.js b/server/services/musicVideo/performanceShot.test.jsindex d8555a90a9..2dbca45702 100644--- a/server/services/musicVideo/performanceShot.test.js+++ b/server/services/musicVideo/performanceShot.test.js@@ -390,10 +390,10 @@ describe.skipIf(!ffmpeg)('music-video performance shot through the fal lip-syncit('refuses a performance shot on a lane without source-audio conditioning, before slicing anything', async () => {getProject.mockResolvedValue(project());- for (const backend of ['grok', 'local']) {+ for (const backend of ['grok', 'reactor']) {await expect(preparePerformanceShot({musicVideo: { projectId: 'mv-1', sceneId: 'mvs-1' }, backend, sourceImagePath: '/x/frame.png',- })).rejects.toMatchObject({ status: 400, code: 'MUSIC_VIDEO_PERFORMANCE_UNSUPPORTED', message: expect.stringMatching(backend === 'grok' ? /cutaway-only/ : /cannot lip-sync/) });+ })).rejects.toMatchObject({ status: 400, code: 'MUSIC_VIDEO_PERFORMANCE_UNSUPPORTED', message: expect.stringMatching(backend === 'grok' ? /cutaway-only/ : /pin fal\.ai/) });}const staged = existsSync(PATHS.uploads) ? (await readdir(PATHS.uploads)).filter((f) => f.startsWith('mv-performance-')) : [];expect(staged).toEqual([]);diff --git a/server/services/musicVideo/planner.test.js b/server/services/musicVideo/planner.test.jsindex 2e70ff3aa7..0e577e9b89 100644--- a/server/services/musicVideo/planner.test.js+++ b/server/services/musicVideo/planner.test.js@@ -671,7 +671,7 @@ describe('directed planner', () => {const duration = (list) => list.reduce((sum, s) => sum + s.endSec - s.startSec, 0);const performances = planned.scenes.filter((s) => s.shotMode === 'performance');expect(duration(performances)).toBeLessThanOrEqual(duration(planned.scenes.filter((s) => s.visualLayer === 'footage')) * 0.4);- expect(performances.length > 0).toBe(backend === 'fal');+ expect(performances.length > 0).toBe(backend === 'fal' || backend === 'local');}});diff --git a/server/services/musicVideo/productionPool.js b/server/services/musicVideo/productionPool.jsindex ef50cf60e4..ca6227a7b7 100644--- a/server/services/musicVideo/productionPool.js+++ b/server/services/musicVideo/productionPool.js@@ -36,6 +36,7 @@ import { resolveVideoSupportedModes } from '../../lib/videoModeProfiles.js';import { isHardwareCompatible } from '../../lib/systemCapabilities.js';import { RUNNER_FAMILIES } from '../../lib/runners.js';import { poolHasRoute, routeKey } from './production.js';+import { supportsSourceAudioWindow } from '../federatedMedia/sourceAudio.js';async function localImageModel(route, env) {if (route.model) return (env.imageModels || []).find((model) => model.id === route.model);@@ -208,6 +209,14 @@ async function routeIncapableReason(route, requirement, env) {}}if (requirement.kind === 'video' && requirement.performance) {+ if (route.mode === 'local') {+ // Local lip-sync is LTX audio-to-video, which only some local models run.+ const requested = typeof route.model === 'string' ? route.model.trim() : '';+ const { model, modelId } = await env.resolveVideoModel(requested);+ if (!supportsSourceAudioWindow(model)) {+ return `Local video model "${model?.name || requested || modelId || '(none)'}" cannot lip-sync to the song; choose an LTX-2.5 (or LTX-2.3 dgrauet) model`;+ }+ }return performanceBlockedReason(route.mode);}return null;diff --git a/server/services/videoGen/submitJob.js b/server/services/videoGen/submitJob.jsindex 25ab287d94..91b2152c20 100644--- a/server/services/videoGen/submitJob.js+++ b/server/services/videoGen/submitJob.js@@ -126,16 +126,22 @@ const submitValidatedVideoGenJob = async (body, uploads) => {cleanupStaged: async () => { if (audio) await discardSourceAudioWindow(audio.audioFilePath); } };} else {if (body._suppliedAudio) {- const [{ getProject }, { resolveVideoModelSelection }, { sourceAudioVideoRequest, supportsSourceAudioWindow }] = await Promise.all([+ const [{ getProject }, { resolveVideoModelSelection }, { sourceAudioVideoRequest, supportsSourceAudioWindow }, { isPerformanceScene }] = await Promise.all([import('../musicVideo/projects.js'), import('./modelSelection.js'), import('../federatedMedia/sourceAudio.js'),+ import('../../lib/musicVideoShotTiming.js'),]);const project = await getProject(body.musicVideo.projectId);const scene = project?.scenes?.find((entry) => entry.sceneId === body.musicVideo.sceneId);- const { model, modelId } = await resolveVideoModelSelection(body.modelId);- if (!supportsSourceAudioWindow(model)) {- throw new ServerError('Supplied song windows currently require an LTX MLX model with audio conditioning', { status: 400, code: 'MUSIC_VIDEO_AUDIO_UNSUPPORTED' });+ // A performance shot cuts its own song window (performanceShot.js), so+ // it enters as an image render and becomes a2v once that window exists.+ if (isPerformanceScene(scene)) body = { ...body, mode: 'image', _suppliedAudio: false };+ else {+ const { model, modelId } = await resolveVideoModelSelection(body.modelId);+ if (!supportsSourceAudioWindow(model)) {+ throw new ServerError('Supplied song windows currently require an LTX MLX model with audio conditioning', { status: 400, code: 'MUSIC_VIDEO_AUDIO_UNSUPPORTED' });+ }+ body = sourceAudioVideoRequest(scene, model, { ...body, modelId });}- body = sourceAudioVideoRequest(scene, model, { ...body, modelId });}prepared = await prepareVideoGenParams({ body, uploads, localOnlyParamKeys: VIDEO_GEN_LOCAL_ONLY_FIELD_NAMES });}@@ -224,7 +230,34 @@ const submitValidatedVideoGenJob = async (body, uploads) => {throw error;}): null;- const performanceParams = performance ? {+ // Local LTX lip-sync: the performance window is the a2v audio, on a frame+ // canvas that covers it (the helper pads the tail; the edit points trim it).+ let localPerformanceParams = null;+ if (performance && backend === 'local') {+ const { sourceAudioVideoRequest, supportsSourceAudioWindow } = await import('../federatedMedia/sourceAudio.js');+ const fail = async (message, code) => {+ await unlink(performance.audioFilePath).catch(() => {});+ await cleanupStaged();+ throw new ServerError(message, { status: 400, code });+ };+ if (!supportsSourceAudioWindow(prepared.effectiveModel)) {+ await fail(`${prepared.effectiveModel?.name || 'This local model'} cannot lip-sync; choose an LTX-2.5 (or LTX-2.3 dgrauet) model for performance shots`, 'MUSIC_VIDEO_PERFORMANCE_UNSUPPORTED');+ }+ const { audioWindow } = performance.shotInstruction;+ const { numFrames } = await Promise.resolve()+ .then(() => sourceAudioVideoRequest(audioWindow, prepared.effectiveModel, { fps: body.fps || 24 }))+ .catch((error) => fail(error.message, error.code || 'MUSIC_VIDEO_AUDIO_WINDOW_INVALID'));+ localPerformanceParams = {+ mode: 'a2v',+ numFrames,+ audioFilePath: performance.audioFilePath,+ audioStartSec: 0,+ disableAudio: false,+ uploadedTempPaths: [...(prepared.uploadedTempPaths || []), performance.audioFilePath],+ shotInstruction: performance.shotInstruction,+ };+ }+ const performanceParams = performance && !localPerformanceParams ? {modelId: performance.modelId,// The output length follows the submitted audio window; a clip-length pin// would contradict it, so none is sent.@@ -420,7 +453,10 @@ const submitValidatedVideoGenJob = async (body, uploads) => {...(isDefaultSpeedProfile(body.speedProfileId) ? {} : { speedProfileId: body.speedProfileId }),...(isFullDecode(body.draftDecode) ? {} : { draftDecode: body.draftDecode }),...(isDefaultVideoStreamingMode(body.streamingMode) ? {} : { streamingMode: body.streamingMode }),- disableAudio: isTruthyMeta(body.disableAudio),+ // A Music Video cutaway plays under the song, so its own soundtrack is never+ // heard: local clips render silent unless the song conditions them (a2v),+ // whatever the caller sent. Lip-sync shots set this back below.+ disableAudio: isTruthyMeta(body.disableAudio) || (Boolean(body.musicVideo) && mode !== 'a2v'),// Absent means "use the settings.videoGen.displaySleep default" — only// forward it when the form actually sent an explicit choice....(body.displaySleep !== undefined@@ -451,6 +487,7 @@ const submitValidatedVideoGenJob = async (body, uploads) => {...(body.musicVideo ? { musicVideo: body.musicVideo } : {}),...(body.fableLoom ? { fableLoom: body.fableLoom } : {}),...(body.visualConditioning ? { visualConditioning: body.visualConditioning } : {}),+ ...localPerformanceParams,});return {jobId,diff --git a/server/services/videoGen/submitJob.test.js b/server/services/videoGen/submitJob.test.jsindex a9049762ae..773cef8662 100644--- a/server/services/videoGen/submitJob.test.js+++ b/server/services/videoGen/submitJob.test.js@@ -270,6 +270,51 @@ describe('submitVideoGenJob', () => {expect(mocks.enqueueJob).not.toHaveBeenCalled();});+ describe('on local LTX', () => {+ const ltx = { id: 'ltx25-example', name: 'LTX-2.5 Example', runtime: 'ltx25', frameStride: 8, maxNumFrames: 1017 };+ const slice = '/example/uploads/mv-performance-2.wav';+ const shotInstruction = {+ version: 2, shotMode: 'performance', audioWindow: { startSec: 2.003, endSec: 6.203, durationSec: 4.2 },+ edit: { inSec: 0, outSec: 4.2, targetSec: 4.2 },+ };+ beforeEach(() => {+ mocks.preparePerformanceShot.mockResolvedValue({ audioFilePath: slice, shotInstruction, modelId: null, resolution: null, enableTranscription: false });+ });++ it('renders the song window as LTX audio-to-video on a canvas that covers it', async () => {+ mocks.prepareVideoGenParams.mockResolvedValue(localPrepared({+ effectiveModel: ltx, mode: 'image', sourceImagePath: '/example/images/frame.png', uploadedTempPaths: ['/example/uploads/frame.png'],+ }));+ await submitVideoGenJob({ prompt: 'singer', backend: 'local', modelId: ltx.id, mode: 'image', disableAudio: true, fps: 24, musicVideo }, {});+ const { params } = mocks.enqueueJob.mock.calls[0][0];+ // 4.2 s at 24 fps needs 102 frames; the LTX 8n+1 grid rounds that to 105.+ expect(params).toMatchObject({+ mode: 'a2v', numFrames: 105, audioFilePath: slice, audioStartSec: 0, disableAudio: false, shotInstruction,+ uploadedTempPaths: ['/example/uploads/frame.png', slice],+ });+ expect(params).not.toHaveProperty('lipSync');+ });++ it('renders a local cutaway silent unless the song conditions it', async () => {+ mocks.preparePerformanceShot.mockResolvedValue(null);+ mocks.prepareVideoGenParams.mockResolvedValue(localPrepared({ effectiveModel: ltx, mode: 'image' }));+ await submitVideoGenJob({ prompt: 'skyline', backend: 'local', mode: 'image', musicVideo }, {});+ mocks.prepareVideoGenParams.mockResolvedValue(localPrepared({ effectiveModel: ltx, mode: 'a2v' }));+ await submitVideoGenJob({ prompt: 'skyline', backend: 'local', mode: 'a2v', musicVideo }, {});+ await submitVideoGenJob({ prompt: 'skyline', backend: 'local', mode: 'a2v' }, {});+ expect(mocks.enqueueJob.mock.calls.map(([job]) => job.params.disableAudio)).toEqual([true, false, false]);+ });++ it('refuses a local model without audio-to-video and releases the slice and staged inputs', async () => {+ const prepared = localPrepared({ effectiveModel: { id: 'wan-example', name: 'Example Wan', runtime: 'wan22' }, mode: 'image' });+ mocks.prepareVideoGenParams.mockResolvedValue(prepared);+ await expect(submitVideoGenJob({ prompt: 'singer', backend: 'local', mode: 'image', musicVideo }, {}))+ .rejects.toMatchObject({ status: 400, code: 'MUSIC_VIDEO_PERFORMANCE_UNSUPPORTED', message: expect.stringMatching(/Example Wan cannot lip-sync/) });+ expect(prepared.cleanupStaged).toHaveBeenCalledTimes(1);+ expect(mocks.enqueueJob).not.toHaveBeenCalled();+ });+ });+it('refuses a kickoff tagged for a revision that has since closed, as the last step before the queue write (#9011)', async () => {const closed = Object.assign(new Error('This revision is already canceled'), { status: 409, code: 'REVISION_CLOSED' });mocks.assertRevisionOpen.mockRejectedValueOnce(closed);