Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
Music Video: silent video models get plain physical motion prompts (#11242)
[._.]/43a982f441c5
diff --git a/server/lib/README.md b/server/lib/README.mdindex 685d6bb197..46be386310 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -847,7 +847,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `loopInterval.js` | Loop interval grammar (`MIN_INTERVAL_MS`, `parseInterval`, `isValidLoopInterval`) shared by `services/loops.js` and `createLoopSchema` so the route rejects what the service would. || `musicVideoAspect.js` | `musicVideoAspect(project)` (treatment brief aspect, default 16:9), `MUSIC_VIDEO_FRAME_GEN_SIZES` and `musicVideoFrameGenSize(project)`: the size every Music Video reference frame is requested at, so a backend never picks portrait for a 16:9 video. `musicVideoAtAspect(project, aspect)` re-frames the project for one render (a vertical social cut) without touching the record. Shared by `productionDispatch.js`, `documentRender.js` and the client board. |-| `musicVideoMotion.js` | `MOTION_CONTINUITY_CLAUSE`: the clause every Music Video motion (i2v) prompt ends with, pinning the reference frame's lighting, time of day and palette. Shared by `handoff.js` and the client board. |+| `musicVideoMotion.js` | `MOTION_CONTINUITY_CLAUSE`: the clause every Music Video motion (i2v) prompt ends with, pinning the reference frame's lighting, time of day and palette. Shared by `handoff.js` and the client board. `silentMotionPrompt(prompt, { styleParts })`: the same prompt as plain physical motion (no lyric/beat cues, song times or look words) for a take that hears no audio; applied at video submit. || `musicVideoConditioning.js` | `musicVideoStyleImages`, `musicVideoStylePrompt`, `musicVideoStyleBasis`, and `musicVideoStyleReferenceCapacity` supply bounded moodboard conditioning and text fallback; `musicVideoConditioningReferences(project, scene)` selects the approved character sheet, the scene's mapped set plate and the sheets of supporting cast the scene's prompt names (capped at 4), with capped project-wide fallback; shared by board and production. || `musicVideoMediumPlan.js` | `normalizeMusicVideoProductionPolicy`, `planMusicVideoMedia`, `summarizeMusicVideoMediumPlan`, and `codeFirstProductionAssets`: pinned shot media, union-of-intervals video allowance, and selected-asset preflight shared by planning and Board review. `musicVideoBriefTools`, `musicVideoBriefAllowsVideo` and `musicVideoToolPolicyConflict` read the brief's selected tools against the policy (the setup warning and the video-dispatch refusal). |diff --git a/server/lib/musicVideoMotion.js b/server/lib/musicVideoMotion.jsindex 115c5eb530..b720b4835b 100644--- a/server/lib/musicVideoMotion.js+++ b/server/lib/musicVideoMotion.js@@ -7,3 +7,53 @@* mid-shot in a real run), so every motion prompt ends with this clause.*/export const MOTION_CONTINUITY_CLAUSE = "hold the reference frame's lighting, time of day and palette for the whole shot";++// A song reference a silent i2v model cannot hear: a quoted lyric cue ("on+// 'Sleep'"), a musical event ("on the downbeat", "in time with the snare") or+// a song timestamp ("at 1:42").+// A quoted word is a lyric cue only after a timing preposition ("on 'Sleep'") or when named as one ("before the line 'Go'").+const LYRIC_CUE = /\s*\b(?:(?:on|at|as|when|until)\s+(?:the\s+)?(?:(?:lyric|word|line)\s+)?|(?:with|to|before|after)\s+the\s+(?:lyric|word|line)\s+)["“'‘][^"”'’\n]{1,60}["”'’]/gi;+// Only unambiguous musical words: "bridge", "hook", "drop" and "kick" are also sets and props.+const MUSIC_CUE = /\s*\b(?:on|to|with|in time with|in sync with|synced to|timed to|matching)\s+(?:the|each|every|a)?\s*(?:next\s+|first\s+|final\s+|last\s+)?(?:beats?|downbeats?|backbeats?|snare(?: hits?)?|kick drums?|drum hits?|bass drop|chorus|verse|music|song|rhythm|tempo|lyrics?|vocals?|vocal line|singing)\b/gi;+const SONG_TIMESTAMP = /\s*\b(?:at|around|by|until|from)\s+\d{1,2}:\d{2}(?:\.\d+)?\b/g;+// A look word an i2v model acts out instead of reading as style (falling film for "film grain").+const LOOK_WORD = /\b(?:film grain|grain|grainy|35 ?mm|16 ?mm|super ?8|kodak|eastman|fuji|panavision|anamorphic|lens(?:es)?|bokeh|depth of field|color (?:grade|grading|timing)|colour grad(?:e|ing)|halation|saturation|microcontrast|highlight rolloff|photochemical|film stock|release print|vhs|analog cinematography|vignette|light leaks?|chromatic aberration)\b/i;+// A clause naming a subject is action ("she wipes the camera lens"), never a look clause.+const SUBJECT_WORD = /\b(?:she|he|they|we|i|you|her|his|their|its|it)\b/i;+const isLookClause = (clause) => LOOK_WORD.test(clause) && !SUBJECT_WORD.test(clause) && clause.trim().split(/\s+/).length <= 8;+// Generated intent lines that time the shot to the song.+const SONG_INTENT_LINE = /^(?:Lyric|Word) \d+(?:\.\d+)?s–\d+(?:\.\d+)?s: .*$|^Musical onsets: .*$/;+// Generated shot-intent and camera lines, kept as written.+const CONTRACT_LINE = /^(?:(?:Action|Reaction) \d+(?:\.\d+)?s–|(?:Purpose|Starting emotion|Ending emotion|Active speaker|Camera|Camera movement|Must preserve|Acceptance|Speed|Framing|End):)/;++const tidy = (text) => text.replace(/\s+([,.;:])/g, '$1').replace(/([,;])\s*(?=[,;.])/g, '').replace(/[ \t]{2,}/g, ' ').replace(/^[,;\s]+|[,;\s]+$/gm, '');++/**+ * A motion prompt rewritten for a video model that sees only its reference+ * frame and hears nothing (local LTX without supplied audio, Grok, Hailuo,+ * Kling and other image-to-video lanes): plain physical motion. Song timing+ * and lyric cues are removed, as are the look clauses (`styleParts` verbatim,+ * then any short subjectless clause naming film stock, grain, lens or grade),+ * since the frame already carries the look and these models act style words+ * out literally. The shot-intent block keeps its clip-relative actions but+ * drops its lyric, word and onset lines. An audio-driven or lip-sync take keeps+ * its prompt as authored.+ */+export function silentMotionPrompt(prompt, { styleParts = [] } = {}) {+ let text = String(prompt || '');+ for (const part of styleParts) {+ const trimmed = typeof part === 'string' ? part.trim() : '';+ if (trimmed) text = text.split(trimmed).join('');+ }+ return text.split('\n')+ .filter((line) => !SONG_INTENT_LINE.test(line.trim()))+ .map((line) => {+ if (CONTRACT_LINE.test(line.trim())) return line.replace(/; land the move exactly on the downbeat/, '');+ const stripped = tidy(line.replace(LYRIC_CUE, '').replace(MUSIC_CUE, '').replace(SONG_TIMESTAMP, '')+ .split(/(?<=[,;.!?])/).filter((clause) => !isLookClause(clause)).join(''));+ // Stripping never empties an authored line of its action.+ return stripped.trim() || !line.trim() ? stripped : line;+ })+ .filter((line, index, lines) => line.trim() || (index > 0 && lines[index - 1].trim()))+ .join('\n').trim();+}diff --git a/server/lib/musicVideoMotion.test.js b/server/lib/musicVideoMotion.test.jsnew file mode 100644index 0000000000..880f7a0272--- /dev/null+++ b/server/lib/musicVideoMotion.test.js@@ -0,0 +1,26 @@+import { describe, expect, it } from 'vitest';+import { silentMotionPrompt } from './musicVideoMotion.js';++// Regression: a silent i2v model acted out song cues and look words, and stripping must never erase the action itself.+describe('silentMotionPrompt', () => {+ it.each([+ ['Her head drops onto the pillow on "Sleep", film grain, 35mm Kodak', 'Her head drops onto the pillow'],+ ['She spins on the downbeat and the camera pushes in at 1:42.', 'She spins and the camera pushes in.'],+ ['Shot on 35mm film. She turns toward the window and smiles.', 'She turns toward the window and smiles.'],+ ['She wipes the camera lens and steps back.', 'She wipes the camera lens and steps back.'],+ ['Wind ripples the grain field as she runs through it.', 'Wind ripples the grain field as she runs through it.'],+ ['He walks to the bridge and hangs his coat on a hook.', 'He walks to the bridge and hangs his coat on a hook.'],+ ["She turns to 'face' the crowd.", "She turns to 'face' the crowd."],+ ['Camera pushes in on the downbeat as she turns.', 'Camera pushes in as she turns.'],+ ['Shallow depth of field', 'Shallow depth of field'],+ ])('%s', (input, expected) => {+ expect(silentMotionPrompt(input)).toBe(expected);+ });++ it('drops song-timed intent lines and the project look but keeps the generated contract and camera lines', () => {+ const prompt = ['She runs, grainy 35mm look', 'Camera movement: Push in — slow push', 'Speed: Slow; land the move exactly on the downbeat.',+ 'Action 0.000s–1.000s: Singer — Offers a hand', 'Word 0.500s–1.500s: Stay', 'Musical onsets: 0.750s'].join('\n');+ expect(silentMotionPrompt(prompt, { styleParts: ['grainy 35mm look'] })).toBe(+ 'She runs\nCamera movement: Push in — slow push\nSpeed: Slow.\nAction 0.000s–1.000s: Singer — Offers a hand');+ });+});diff --git a/server/services/musicVideo/orchestratorReview.js b/server/services/musicVideo/orchestratorReview.jsindex 0ab598bd21..a051be0fbf 100644--- a/server/services/musicVideo/orchestratorReview.js+++ b/server/services/musicVideo/orchestratorReview.js@@ -100,7 +100,7 @@ ${block('SHOTS', JSON.stringify(shots, null, 2))}Judge: does the sequence tell the concept's story across the song, does each shot read clearly, do shots vary in framing and energy with the music, and do lyric lines land on shots that show them? ${RULES}${incomplete.length ? `These shots have blank fields that must be filled: ${incomplete.join(', ')}. Fill them in "fill".\n` : ''}Return JSON only:{"verdict":"approve|revise","score":1-10,"notes":"why","changes":[{"sceneId":"a shot id","text":"what must change in that shot"}],"fill":[{"sceneId":"a shot id","action":"","staging":"","camera":"","transition":""}]}-List changes only when verdict is revise; "fill" only for blank fields.`;+List changes only when verdict is revise; "fill" only for blank fields. Write "action" as plain physical motion (who moves, how, the end state): the video model usually hears nothing and acts out every word, so no lyric cues, beats, song times or look words.`;}/**diff --git a/server/services/musicVideo/planner.js b/server/services/musicVideo/planner.jsindex 181694717f..2ba5b6ac27 100644--- a/server/services/musicVideo/planner.js+++ b/server/services/musicVideo/planner.js@@ -148,7 +148,7 @@ function directedPrompts(shot, generated = {}, sets = []) {].filter(Boolean).join(' ');const motion = generated.prompt || (shot.visualLayer === 'card' ? 'Choreograph the graphic scene around the supplied musical anchors; give subject, props and camera a readable action and payoff. Use a hold only when it serves the chosen energy target.': shot.shotMode === 'performance' ? `Perform to the source audio: ${shot.lyricText}`- : `Stage a readable subject action and camera response for ${shot.lyricText || shot.visualIntent || shot.sectionLabel || 'this instrumental passage'}, timed to the supplied musical anchors and chosen energy target.`);+ : `Stage a readable subject action and camera response for ${shot.visualIntent || shot.sectionLabel || 'this instrumental passage'}, at the chosen energy target.`);return { framePrompt: frame.slice(0, SCENE_TEXT_MAX), prompt: motion.slice(0, SCENE_TEXT_MAX) };}@@ -229,11 +229,11 @@ ${shotLines}For EACH shot above, propose a timed composition for its assigned medium (code-authored worlds, selected stills or footage):- "framePrompt": the opening reference still — subject, setting, lighting, composition. Keep it concrete and visual.-- "prompt": the motion for that shot — subject action, staging and mood — building on the frame. Higher-energy sections read more kinetic; calmer sections more lingering.+- "prompt": the motion for that shot as plain physical action — who moves, how, where the camera goes and the end state — building on the frame. Higher-energy sections read more kinetic; calmer sections more lingering. The video model sees only the frame and usually hears nothing, and it acts out every word: no song times, beats, lyric cues ("on 'Sleep'") or song references, and no look words (film stock, grain, lens, grade, palette), which the frame already carries.- "camera": the shot's camera move from the vocabulary below, with its speed, end framing, whether it lands on the downbeat, and (for a static camera) the reason.- Shots in the same section are one edited sequence: keep subject and setting continuous, but vary framing (wide / medium / close), angle, or action from shot to shot so consecutive shots cut rather than repeat.- The OPENING HOOK shot must grab attention immediately.-- In each motion prompt, name absolute start/end times and the supplied musical anchors for subject action, prop transformation, camera framing/movement and any permitted graphic typography. Follow the saved energy target; this proposal still needs human review. Repeated choruses must develop the action, scale or staging instead of replaying the same pose. Include anticipation, payoff and recovery; motivated holds and long takes are valid. Do not replace choreography with continuous camera drift, geometry presence, subtitles alone or an arbitrary fast-cut quota.+- Use each shot's absolute times and musical anchors to decide what happens and when the camera lands (its "onBeat"), but keep them out of the motion prompt text. Follow the saved energy target; this proposal still needs human review. Repeated choruses must develop the action, scale or staging instead of replaying the same pose. Include anticipation, payoff and recovery; motivated holds and long takes are valid. Do not replace choreography with continuous camera drift, geometry presence, subtitles alone or an arbitrary fast-cut quota.${direction ? `- "subject": who the frame shows. "lead" for the protagonist, "none" when no person is in frame (an empty set, a title card, an object insert), or a short description of whoever else is in frame (for example "a desk clerk"). Each frame opens with this subject, and only the lead wears the assigned look, so set it on every shot; a performance is always "lead".${direction.sets?.length ? '- "set": only when a shot belongs in a different listed set than its assigned one, that set\'s exact name; otherwise leave it out.\n' : ''}` : ''}${revising ? '- These shots already exist. Revise each one\'s current frame and motion to address the unresolved review feedback, keeping what the feedback does not ask to change.\n' : ''}- If an energy target or required audio timing is absent, identify it as a director decision to review rather than inventing analysis or claiming the plan is approved.${hasLyrics ? '- Let the lyric lines inform the imagery and emotion of their shot (interpret, do not illustrate word-for-word). Never render the lyrics as on-screen text in footage; assigned card layers use their explicit card text.\n' : ''}${hasDelivery ? '- Follow the delivery directions: spoken or whispered lines play as intimate close-ups; shouts land as hard-hitting cuts or impacts; a silence or stop is a held, frozen or cut-to-black beat.\n' : ''}- Keep the assigned layer and shot mode: performance uses source-audio lip-sync and a frontal medium close-up with the mouth visible; cutaways show narrative action without lip-sync; cards are graphic beats.\n- Honor any director intent given for a shot. Instrumental shots carry no singing or lip-sync.diff --git a/server/services/musicVideo/planner.test.js b/server/services/musicVideo/planner.test.jsindex 2e70ff3aa7..d7947214a4 100644--- a/server/services/musicVideo/planner.test.js+++ b/server/services/musicVideo/planner.test.js@@ -657,7 +657,9 @@ describe('directed planner', () => {expect(scenes[0].framePrompt).toMatch(/Signal keeper.*Signal tower.*Ochre coat/);expect(scenes[4].framePrompt).toMatch(/Echo hall.*Silver jacket/);expect(scenes.every((s) => !s.framePrompt.includes('greenhouse'))).toBe(true);- expect(scenes[0].prompt).toContain('We follow the signal 0');+ // The fallback motion never quotes the lyric: a silent video model acts it out.+ expect(scenes[0].prompt).toMatch(/^Stage a readable subject action/);+ expect(scenes[0].prompt).not.toContain('We follow the signal');expect(planned.composition).toMatchObject({ mode: 'composed', style: { color: '#abcdef' } });});diff --git a/server/services/videoGen/submitJob.js b/server/services/videoGen/submitJob.jsindex 25ab287d94..f51b92b622 100644--- a/server/services/videoGen/submitJob.js+++ b/server/services/videoGen/submitJob.js@@ -42,6 +42,15 @@ const musicVideoProjectAspect = async (projectId) => {]);return musicVideoAspect(await getProject(projectId));};+/** A Music Video motion prompt for a take that hears no audio, with the project's look clauses removed. */+async function silentMusicVideoMotionPrompt(prompt, project) {+ const [{ silentMotionPrompt }, { musicVideoCreativeContext }, { visualDirection }] = await Promise.all([+ import('../../lib/musicVideoMotion.js'), import('../../lib/musicVideoCreativeContext.js'), import('../musicVideo/handoff.js'),+ ]);+ return silentMotionPrompt(prompt, { styleParts: [project?.concept?.style, musicVideoCreativeContext(project?.concept, { moodBoard: false }),+ visualDirection(project?.visualSpec)].filter(Boolean) });+}+const submitValidatedVideoGenJob = async (body, uploads) => {let fableLoomRenderSettings = null;if (body.fableLoom) {@@ -266,7 +275,11 @@ const submitValidatedVideoGenJob = async (body, uploads) => {}const scene = project?.scenes?.find((entry) => entry.sceneId === body.musicVideo.sceneId);assertShotActionContract(scene);+ // A take that hears nothing (no lip-sync or supplied song audio) gets plain physical+ // motion: the model would act out song cues and look words instead of reading them.+ const hearsAudio = !!(performance || prepared.audio || params.audioConditioning || params.audioFilePath);params.prompt = withShotActionPrompt(params.prompt, project, scene, { offsetSec: performance?.shotInstruction?.edit?.inSec || 0 });+ if (!hearsAudio) params.prompt = await silentMusicVideoMotionPrompt(params.prompt, project);const interval = performance?.shotInstruction?.songInterval || prepared.audio?.songInterval;if (params.audioConditioning && Math.round(scene?.startSec * 48000) !== params.audioConditioning.startSample) {throw new ServerError('Scene timing changed while source audio was prepared', { status: 409, code: 'MUSIC_VIDEO_SHOT_TIMING_CHANGED' });diff --git a/server/services/videoGen/submitJob.test.js b/server/services/videoGen/submitJob.test.jsindex a9049762ae..7f6fa947f4 100644--- a/server/services/videoGen/submitJob.test.js+++ b/server/services/videoGen/submitJob.test.js@@ -354,16 +354,24 @@ describe('shot intent at the video submission boundary', () => {mocks.getMusicVideoProject.mockResolvedValue(project());});- it('queues current action/reaction, actual words/onsets and immutable intent even when the client omitted it', async () => {+ it('queues current action/reaction and immutable intent even when the client omitted it', async () => {await submitVideoGenJob({ prompt: 'Legacy free text', musicVideo: { projectId: 'p', sceneId: 's' } }, {});const { params } = mocks.enqueueJob.mock.calls[0][0];expect(params.prompt).toContain('Legacy free text');expect(params.prompt).toContain('Reaction 1.000s–2.000s: Listener');- expect(params.prompt).toContain('Word 0.500s–1.500s: Stay');- expect(params.prompt).toContain('Musical onsets: 0.750s');expect(params.shotInstruction.actionContract).toEqual(contract);});+ // Regression: a model that hears no audio acted out song cues and look words ("film falling onto the bed").+ it('sends a take that hears no audio plain physical motion, without song cues or look words', async () => {+ mocks.getMusicVideoProject.mockResolvedValue({ ...project(), concept: { style: 'grainy 35mm Kodak look' } });+ await submitVideoGenJob({ prompt: 'Her head drops onto the pillow on "Sleep", grainy 35mm Kodak look, camera holds', musicVideo: { projectId: 'p', sceneId: 's' } }, {});+ const { params } = mocks.enqueueJob.mock.calls[0][0];+ expect(params.prompt).toContain('Her head drops onto the pillow, camera holds');+ expect(params.prompt).toContain('Action 0.000s–1.000s: Singer — Offers a hand');+ expect(params.prompt).not.toMatch(/Sleep|Kodak|grain|Word \d|Musical onsets/);+ });+it('shifts contract and words into a padded performance window exactly once', async () => {mocks.prepareVideoGenParams.mockResolvedValue({ backend: 'fal', cleanupStaged: vi.fn(), sourceImagePath: '/example/frame.png' });mocks.preparePerformanceShot.mockResolvedValue({ audioFilePath: '/example/audio.wav', modelId: 'lip-sync', resolution: '1080P',@@ -372,6 +380,7 @@ describe('shot intent at the video submission boundary', () => {const { params } = mocks.enqueueJob.mock.calls[0][0];expect(params.prompt).toContain('Action 1.500s–2.500s: Singer');expect(params.prompt).toContain('Word 2.000s–3.000s: Stay');+ expect(params.prompt).toContain('Musical onsets: 2.250s');expect(params.prompt).not.toContain('stale intent');expect(params.prompt).toContain('Keep moving');expect(params.shotInstruction.edit.inSec).toBe(1.5);