AI media harness for planning, generating, and editing locally
TypeScript
Let agents assign music to video references
Bittid7fee7a7530f
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.mdindex a299a53..49e7691 100644--- a/RELEASE_NOTES.md+++ b/RELEASE_NOTES.md@@ -10,7 +10,7 @@ Changes since 0.3.2.- **Video library:** Organize Videos → Scenes → Shots, play scenes together, split songs into scenes, and share prompts and timed references. Videos can override project Look and resolution settings, and dependent scenes can be queued together. Cancelling a generation also cancels its downstream dependents while preserving earlier generations.- **Scene takes:** Keep up to four takes plus a draft, restore settings, choose the primary take, and upscale saved takes by 1.5×, 2× or 4×. Non-primary takes can be removed during generation.- **Single Shot:** Continue with Context and Previous video anchors, optional generated endings, improved overlap joining and consistent Tone. Context guidance now matches the original anchor to reduce appearance degradation across continuations. Use Bridge for new connections instead of Long Shot.-- **Motions and agents:** Create editable JavaScript animations for the timeline or references. Agents can inspect media frames, generate music and sync Motions to audio.+- **Motions and agents:** Create editable JavaScript animations for the timeline or references. Agents can inspect media frames, generate music, assign generated songs as Video audio references and sync Motions to audio.- **Image editing:** Combine or subtract lasso selections, queue batches of up to 100 images, and edit bounded regions of larger images while preserving their original resolution.- **Live previews:** Watch MiniMax H3 video and image generation in the editor. Cancelling a video generation saves its latest available preview as the Take's video. Optional TAEH3 preview weights are available in Settings.- **Generation effects:** LCS color steering, tone, sharpness and color stabilization, with step ranges and optional regions. LCS Drift adds configurable highlight compression and learned drift correction using an imported scene calibration, including on LAN workers.diff --git a/src-tauri/assets/agent-music.md b/src-tauri/assets/agent-music.mdindex 1b3dc40..4449cf2 100644--- a/src-tauri/assets/agent-music.md+++ b/src-tauri/assets/agent-music.md@@ -17,6 +17,11 @@ Inspect generator.list/get and choose an existing music template. Supported modeGeneration runs in the normal queue and resumes the agent with a completion/failure/cancellation receipt. A completed source produces a NEW song; use the receipt's song and assetId, not the source ID, for follow-up operations. Duplicate requests within the same turn reuse receipts. Never retry failed generation automatically. Results are audio assets usable by clip.add on a timeline track. Do not claim completion until the receipt confirms it; needsSave means persistence failed.+Assign generated music to a Video+---------------------------------+Use {"op":"video.music.set","video":"existing-video-id","song":"generated-song-id"} as a project JSONL command followed by a commit line, or in generation.prepare. Resolve the Video from project JSON videos and the song with music.list/get. The song must have a saved audio asset at least two seconds long. After music.generate, use the receipt's song ID, which may be a new variation.+This replaces only the Video's Music reference (musicSource) with a link to the song's existing audio asset. It preserves the song, scene settings, other references and timeline, and does not generate or regenerate video. Each scene receives the audio segment at its cumulative time offset when subsequently generated. Existing scene generation checks still apply: the song must cover each scene's time range, and music guidance needs a visual reference and a compatible References/Singularity generator. This is audio guidance for generation; use clip.add with the song's assetId to place the actual song on a timeline audio track.+Album cover generation----------------------When the user requests album artwork, use {"kind":"generation","summary":"Generate the song's album cover","command":{"op":"music.cover.generate","song":"existing-song-id"}}. No template argument: this uses the dedicated album-cover generator selected in Settings → Generators → Icons, with the same defaults as the song context-menu action. The song must exist after optional prepare commands; its audio does not need to be generated first. Artwork is a square 1024×1024 image derived from its title, style and lyrics. To update those inputs, use music.set in prepare.diff --git a/src-tauri/assets/agent-system.md b/src-tauri/assets/agent-system.mdindex 7948826..ab04558 100644--- a/src-tauri/assets/agent-system.md+++ b/src-tauri/assets/agent-system.md@@ -41,6 +41,8 @@ image.draft.add creates and selects a new independent draft. id, name and promptWhen the user asks to generate an image, use {"kind":"generation","summary":"Generate Poster.","command":{"op":"image.generate","image":"poster-draft","template":"existing-generator-template-id"}}. Resolve a usable prompt-mode generator with generator.list/get. The app uses the named draft's saved composition, references, dimensions, seed and steps and fills that image-bar entry with the saved output. It waits and resumes with a generation receipt exactly like scene.generate, with the same cancellation and duplicate-request protections. Do not claim the output has been visually inspected.To prepare and generate in the same user request, a generation response may include an optional prepare array of project commands (no commit line). Example: {"kind":"generation","summary":"Prepare and generate Poster.","prepare":[{"op":"image.draft.add","id":"poster-draft","name":"Poster","prompt":"A red balloon above a snowy forest"}],"command":{"op":"image.generate","image":"poster-draft","template":"existing-generator-template-id"}}. Preparation is validated transactionally and saved before generation starts. If rendering fails, the prepared drafts remain available for manual generation. Only include prepare on the first request for that output; repeated generation requests reuse their earlier receipt without applying preparation again. Do not generate when the user asks only for preparation.+Assign a song as a Video's audio reference with {"op":"video.music.set","video":"existing-video-id","song":"generated-song-id"} followed by a commit line. This sets the Video's Music reference from an existing saved song; see the Music API reference for timing and generation requirements.+Generate album covers---------------------For requested song artwork, use {"kind":"generation","summary":"Generate album cover","command":{"op":"music.cover.generate","song":"existing-song-id"}}. The app uses the dedicated album-cover generator from Settings and the song's title, style and lyrics, waits for saving and returns a receipt. No template argument is accepted. Optional prepare commands can create or edit the song first. See the Music API reference for output and cancellation behavior.diff --git a/src-tauri/src/project/commands/batch.rs b/src-tauri/src/project/commands/batch.rsindex f2424b8..3bd698d 100644--- a/src-tauri/src/project/commands/batch.rs+++ b/src-tauri/src/project/commands/batch.rs@@ -80,7 +80,7 @@ fn apply_command(effects: &mut Effects,) -> Result<(), String> {match command {- ProjectCommand::MusicAdd { .. } | ProjectCommand::MusicSet { .. } | ProjectCommand::MotionSync { .. } => super::music::apply(project, command, timestamp),+ ProjectCommand::MusicAdd { .. } | ProjectCommand::MusicSet { .. } | ProjectCommand::MotionSync { .. } | ProjectCommand::VideoMusicSet { .. } => super::music::apply(project, command, timestamp),ProjectCommand::MotionAdd { .. } | ProjectCommand::MotionSet { .. } => super::motions::apply(project, command, timestamp),ProjectCommand::ImageDraftAdd { .. } | ProjectCommand::ImageDraftSelect { .. } => {super::image_drafts::apply(project, command, timestamp)diff --git a/src-tauri/src/project/commands/music.rs b/src-tauri/src/project/commands/music.rsindex 0597ccf..0c63176 100644--- a/src-tauri/src/project/commands/music.rs+++ b/src-tauri/src/project/commands/music.rs@@ -3,6 +3,16 @@ use crate::project::{music::MusicSong, ProjectConfig};pub(super) fn apply(project: &mut ProjectConfig, command: &ProjectCommand, timestamp: &str) -> Result<(), String> {match command {+ ProjectCommand::VideoMusicSet { video, song } => {+ let song = project.music_songs.iter().find(|item| item.id == *song).ok_or("Song not found. Use music.list for IDs.")?;+ let audio = project.assets.iter().find(|asset| Some(&asset.id) == song.asset_id.as_ref() && asset.kind == "audio")+ .ok_or("Song has no generated audio. Generate it before assigning it to a video.")?;+ if !audio.relative_path.as_ref().is_some_and(|path| !path.trim().is_empty()) || audio.duration_ms.unwrap_or(0) < 2000 {+ return Err("The song needs saved audio at least two seconds long.".into());+ }+ let target = project.videos.iter_mut().find(|item| item.id == *video).ok_or("Video not found. Use an existing id from videos in the project JSON.")?;+ target.music_source = Some(crate::project::video::VideoSource { kind: "asset".into(), id: audio.id.clone() });+ }ProjectCommand::MusicAdd { id, title, style, settings, folder_id, steps, seed } => {if id.trim().is_empty() || project.music_songs.iter().any(|s| s.id == *id) { return Err("Song requires a unique id.".into()); }project.music_songs.push(MusicSong { id: id.clone(), title: title.clone(), style: style.clone(), settings: settings.clone(),@@ -65,6 +75,69 @@ mod tests {fn run(p: &ProjectConfig, commands: Value) -> Result<ProjectConfig, String> {super::super::execute_commands_at(p, &serde_json::from_value::<Vec<ProjectCommand>>(commands).unwrap(), &p.updated_at)}++ fn video_music_fixture() -> ProjectConfig {+ let p: ProjectConfig = serde_json::from_str(include_str!("../../../../fixtures/project-v1-complete.json")).unwrap();+ let mut p = run(&p, json!([{"op":"music.add","id":"song","title":"Intro","style":"Jazz"}])).unwrap();+ p.music_songs[0].asset_id = Some("song-audio".into());+ p.music_songs[0].status = "completed".into();+ p.assets.push(serde_json::from_value(json!({"id":"song-audio","kind":"audio","name":"Intro","relativePath":"media/song.wav","mimeType":"audio/wav","durationMs":30000,"createdAt":p.created_at})).unwrap());+ p.videos = serde_json::from_value(json!([+ {"id":"film","title":"Film","sceneIds":[p.generation_jobs[0].id],"prompt":"Dance","createdAt":p.created_at,+ "musicSource":{"kind":"reference","id":"previous"},"videoSource":{"kind":"reference","id":"visual"}},+ {"id":"other","title":"Other","sceneIds":[],"createdAt":p.created_at}+ ])).unwrap();+ crate::project::validation::validate_and_normalize_config(p).unwrap()+ }++ #[test]+ fn video_music_command_links_saved_audio_and_preserves_other_project_data() {+ let p = video_music_fixture();+ let batch = super::super::parse_jsonl_commands("{\"op\":\"video.music.set\",\"video\":\"film\",\"song\":\"song\"}\n{\"op\":\"commit\",\"summary\":\"Assigned song\"}").unwrap();+ let assigned = super::super::execute_commands_at(&p, &batch.commands, &p.updated_at).unwrap();+ let mut expected = serde_json::to_value(&p).unwrap();+ expected["videos"][0]["musicSource"] = json!({"kind":"asset","id":"song-audio"});+ assert_eq!(serde_json::to_value(&assigned).unwrap(), expected);+ // Reapplying the same assignment cannot create duplicate assets or references.+ let repeated = super::super::execute_commands_at(&assigned, &batch.commands, &p.updated_at).unwrap();+ assert_eq!(serde_json::to_value(&repeated).unwrap(), expected);+ let restored: ProjectConfig = serde_json::from_value(expected).unwrap();+ assert_eq!(restored.videos[0].music_source.as_ref().unwrap().id, "song-audio");+ }++ #[test]+ fn video_music_command_rejects_missing_audio_and_invalid_targets_atomically() {+ let p = video_music_fixture();+ let command = json!([{"op":"video.music.set","video":"film","song":"song"}]);+ for (video, song) in [("missing", "song"), ("film", "missing")] {+ assert!(run(&p, json!([{"op":"video.music.set","video":video,"song":song}])).is_err());+ }+ let audio_index = p.assets.iter().position(|a| a.id == "song-audio").unwrap();+ for scenario in 0..5 {+ let mut invalid = p.clone();+ match scenario {+ 0 => invalid.music_songs[0].asset_id = None,+ 1 => invalid.assets[audio_index].kind = "image".into(),+ 2 => invalid.assets[audio_index].relative_path = None,+ 3 => invalid.assets[audio_index].duration_ms = Some(1000),+ _ => invalid.assets[audio_index].duration_ms = None,+ }+ assert!(run(&invalid, command.clone()).is_err());+ }+ assert!(run(&p, json!([+ {"op":"video.music.set","video":"film","song":"song"},+ {"op":"video.music.set","video":"missing","song":"song"}+ ])).is_err());+ assert_eq!(p.videos[0].music_source.as_ref().unwrap().id, "previous");+ for invalid in [+ json!({"op":"video.music.set","video":"film"}),+ json!({"op":"video.music.set","video":"film","song":"song","relativePath":"invented.wav"}),+ json!({"op":"video.music.set","video":"film","song":"song","assetId":"unrelated"}),+ ] {+ assert!(serde_json::from_value::<ProjectCommand>(invalid).is_err());+ }+ }+#[test]fn album_covers_survive_project_round_trips_and_validate_paths() {let p: ProjectConfig = serde_json::from_str(include_str!("../../../../fixtures/project-v1-complete.json")).unwrap();diff --git a/src-tauri/src/project/commands/types.rs b/src-tauri/src/project/commands/types.rsindex 1b69eb2..3e3a027 100644--- a/src-tauri/src/project/commands/types.rs+++ b/src-tauri/src/project/commands/types.rs@@ -12,6 +12,8 @@ pub const MAX_COMMANDS_PER_TURN: usize = 100;#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]#[serde(tag = "op", deny_unknown_fields)]pub enum ProjectCommand {+ #[serde(rename = "video.music.set")]+ VideoMusicSet { video: String, song: String },#[serde(rename = "music.add")]MusicAdd { id: String, title: String, style: String, #[serde(default)] settings: slopus_core::slopfab::music::MusicSettings,#[serde(default, rename = "folderId")] folder_id: Option<String>, #[serde(default)] steps: Option<i32>, #[serde(default)] seed: Option<i64> },@@ -333,6 +335,7 @@ pub(super) struct CommitLine {}pub(super) fn command_name(command: &ProjectCommand) -> &'static str {match command {+ ProjectCommand::VideoMusicSet { .. } => "video.music.set",ProjectCommand::MusicAdd { .. } => "music.add",ProjectCommand::MusicSet { .. } => "music.set",ProjectCommand::MotionSync { .. } => "motion.sync",diff --git a/src/lib/runtime.ts b/src/lib/runtime.tsindex ca0655a..423e357 100644--- a/src/lib/runtime.ts+++ b/src/lib/runtime.ts@@ -50,6 +50,7 @@ export interface SlopfabStatus {export interface RuntimeStatus { providers: ProviderStatus[]; slopfab: SlopfabStatus }export type ProjectCommand =+ | { op: "video.music.set"; video: string; song: string }| { op: "music.add"; id: string; title: string; style: string; settings?: Partial<import("./music").MusicSettings>; folderId?: string; steps?: number; seed?: number }| { op: "music.set"; id: string; title?: string; style?: string; settings?: Partial<import("./music").MusicSettings>; folderId?: string | null; steps?: number; seed?: number }| { op: "motion.add"; id: string; name: string; motion: import("./motion").Motion }@@ -375,7 +376,7 @@ function executeDemoCommands(config: ProjectConfig, commands: ProjectCommand[]):};for (const command of commands) {switch (command.op) {- case "music.add": case "music.set": case "motion.add": case "motion.set": case "motion.sync":+ case "video.music.set": case "music.add": case "music.set": case "motion.add": case "motion.set": case "motion.sync":throw new Error("Music and Motion agent commands require the desktop app.");case "image.draft.add": case "image.draft.select": {Object.assign(next, applyImageDraftCommand(next, command, now));