Skip to content

AI media harness for planning, generating, and editing locally

TypeScript

Add music generation workspace

Bittif360fd1e85d6
diff --git a/README.md b/README.md
index 905ff03..c506a1c 100644
--- a/README.md
+++ b/README.md
@@ -18,7 +18,7 @@
---
-Slopus combines an AI assistant, local image and video generation, reusable references and a timeline editor. SlopFab powers generation without Python; WebCodecs and WebGPU handle media processing without bundled FFmpeg.
+Slopus combines an AI assistant, local image, video and music generation, reusable references and a timeline editor. SlopFab powers generation without Python; WebCodecs and WebGPU handle media processing without bundled FFmpeg.
![Slopus application screenshot](marketing/screenshot1.png)
@@ -27,12 +27,13 @@ Slopus combines an AI assistant, local image and video generation, reusable refe
- **Agent.** Plan scenes, generate media, inspect frames and edit projects with Claude Code, Codex, OpenRouter or a local OpenAI-compatible model. Toggle the pane with Ctrl+Shift+A.
- **Video.** Generate Prompt, First & last frame and other [scene types](docs/scene-types.md), with shot timing, dialogue, camera and sound controls. Keep up to four takes and an editable Draft; use **Make Primary** to choose the timeline video.
- **Images.** Generate with MiniMax H3, MOSS Image or Fibo 2.0. Use whole-image, box, lasso and Zoom edits where supported, with drafts and versions in the image bar.
+- **Music.** Generate songs with YuE2 or MiniMax Music3, edit lyrics and style, browse grid or list views, and play, reference or add songs to the timeline.
- **References.** Reuse descriptions, images, trimmed video clips and [H3 RefMods](docs/refmods.md) across scenes.
- **Generators.** Manage model weights, supported LoRAs, attention and generation effects. Share [generator templates](docs/generator-weights.md#sharing-generator-templates) as `.slop` bundles or run jobs on LAN workers.
- **Timeline.** Arrange video and audio with trimming, splitting, snapping, waveforms, transitions, chroma key and GPU effects.
- **Motions.** Ask the agent to create JavaScript animations, edit elements and parameters, and render them on the timeline or as reference videos.
- **Export.** Save MP4 video with H.264, VP9 or AV1 where encoders are available, or export images as JPG or PNG. Exports run in the background.
-- **Project editing.** Timeline, Video, Image, References and Export views share Save, Undo and Redo. Projects and media are stored locally; agent requests go to the selected provider. API keys stay outside project files.
+- **Project editing.** Timeline, Video, Image, Music, References and Export views share Save, Undo and Redo. Projects and media are stored locally; agent requests go to the selected provider. API keys stay outside project files.
## Setup
@@ -70,3 +71,5 @@ Tauri 2, Rust, React, TypeScript, WebCodecs, WebGPU, `mp4box.js`, `mp4-muxer` an
MiniMax H3 model weights are licensed separately under the **MiniMax H3 COMMUNITY LICENSE AGREEMENT**.
Fibo 2.0 is released under CC BY-NC 4.0 for non-commercial use. Commercial use is subject to an agreement with BRIA.
+
+[YuE2](https://huggingface.co/Comfy-Org/YuE2) uses CC BY-NC 4.0; [MiniMax Music3](https://huggingface.co/Comfy-Org/MiniMax-Music-3) uses Apache 2.0.
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index c4d46cd..cf09e8d 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -4,6 +4,7 @@ Changes since 0.3.2.
## What's new
+- **Music.** Generate with YuE2 or MiniMax Music3, browse songs in grid or list view, use bottom playback controls, and create references or timeline audio.
- **Motions.** Create JavaScript animations with the agent, edit elements and parameters, and export them on the timeline or as reference videos.
- **Scene history.** Keep up to four takes and an editable Draft, restore take settings, choose the primary timeline video, and duplicate scenes.
- **Agent frame inspection.** Agents can view timeline frames, scenes, takes, references, generated images and Motions at a requested time.
@@ -21,4 +22,4 @@ Changes since 0.3.2.
- Image edits preserve source images, prompts and settings; queued results get separate entries, and whole-image edits disable feathering.
- Imported images now expose steps, seed and resolution; new images use generator step defaults, and renamed image roots retain their names.
- Generation effects reuse VAE fingerprint checks to avoid repeated calibration work.
-- Updated SlopFab to API 1.36 and LAN protocol 17. Update workers alongside the app.
+- Updated SlopFab to API 1.38 and LAN protocol 18. Update workers alongside the app.
diff --git a/docs/slopfab-compatibility.md b/docs/slopfab-compatibility.md
index b4504f6..8aeba40 100644
--- a/docs/slopfab-compatibility.md
+++ b/docs/slopfab-compatibility.md
@@ -1,5 +1,16 @@
# Slopfab compatibility
+## Music (API 1.38)
+
+Bundled Windows and Linux runtimes support YuE2 and MiniMax Music3 on CUDA and
+Vulkan, with FFmpeg disabled. LAN protocol 18 carries music settings and
+audio-only results. Update workers alongside the app.
+
+Music requests use `slopfab_request_set_music_settings` with style in the prompt,
+model-specific controls and portable Vulkan arithmetic. They bypass H3 video
+settings. Stereo samples are saved as 24-bit PCM WAV in `media/generated` and
+appear in Music and timeline media. Song settings persist in `musicSongs`.
+
## Generation effects (API 1.31)
Scene and image root inspectors provide **Effects > Add effect** for native
@@ -45,7 +56,7 @@ with model weights were not evaluated in this integration.
## MOSS image generation (API 1.30+)
-The bundled Windows and Linux libraries expose SlopFab API 1.36.0, including
+The bundled Windows and Linux libraries expose SlopFab API 1.38.0, including
the MOSS model-family setter introduced in 1.30. Both builds disable FFmpeg.
MOSS uses CUDA/Flash attention, the image VAE slot, and a separate Qwen3.5
tokenizer file. It returns one RGB frame through the existing image save path.
diff --git a/src-tauri/core/src/models.rs b/src-tauri/core/src/models.rs
index 3b3dc31..d31cb8f 100644
--- a/src-tauri/core/src/models.rs
+++ b/src-tauri/core/src/models.rs
@@ -8,6 +8,7 @@ pub struct GeneratorFeatures {
pub label: String,
pub image_generation: bool,
pub video_generation: bool,
+ pub music_generation: bool,
pub image_editing: bool,
pub image_references: bool,
pub image_outpainting: bool,
diff --git a/src-tauri/core/src/rendered.rs b/src-tauri/core/src/rendered.rs
index 3794891..003143c 100644
--- a/src-tauri/core/src/rendered.rs
+++ b/src-tauri/core/src/rendered.rs
@@ -237,7 +237,8 @@ pub fn from_source(
.and_then(|value| value.checked_mul(height as usize))
.and_then(|value| value.checked_mul(channels as usize))
.ok_or_else(|| "The render's dimensions do not fit in memory.".to_string())?;
- if frame_count == 0 || width == 0 || height == 0 || channels != 3 {
+ let audio_only = frame_count == 0 && width == 0 && height == 0 && video_float_count == 0 && !audio.is_empty();
+ if !audio_only && (frame_count == 0 || width == 0 || height == 0 || channels != 3) {
return Err(format!(
"The render returned {frame_count} frames of {width}x{height}x{channels}."
));
@@ -416,6 +417,18 @@ mod tests {
use super::*;
use std::sync::{atomic::{AtomicBool, Ordering}, Arc};
+ #[test]
+ fn audio_only_results_keep_samples_without_requesting_video_frames() {
+ let dropped = Arc::new(AtomicBool::new(false));
+ let result = from_source("music", Box::new(TrackedStill { dropped: dropped.clone(), fail: true }),
+ vec![0.25, -0.25], 0, 0, 0, 0, 0, 0.0, 2, 48000).unwrap();
+ assert!(result.frame(0).unwrap().is_none());
+ assert_eq!(result.audio_bytes(), [0.25_f32.to_le_bytes(), (-0.25_f32).to_le_bytes()].concat());
+ assert_eq!(result.summary().audio_samples, 2);
+ drop(result);
+ assert!(dropped.load(Ordering::Acquire));
+ }
+
struct TrackedStill {
dropped: Arc<AtomicBool>,
fail: bool,
diff --git a/src-tauri/core/src/slopfab/config.rs b/src-tauri/core/src/slopfab/config.rs
index 140a213..125e765 100644
--- a/src-tauri/core/src/slopfab/config.rs
+++ b/src-tauri/core/src/slopfab/config.rs
@@ -123,6 +123,7 @@ impl Configuration {
}
pub(super) fn validate_inputs(&self, request: &GenerationRequest) -> Result<(), String> {
+ if request.music.is_some() { return super::music::validate(request, self); }
let features = crate::models::generator_features(&self.model_type);
if let Some(negative) = &request.negative_prompt {
if negative.contains('\0') { return Err("Negative prompt cannot contain null characters.".into()); }
diff --git a/src-tauri/core/src/slopfab/ffi/mod.rs b/src-tauri/core/src/slopfab/ffi/mod.rs
index 3a768f2..8ddfb54 100644
--- a/src-tauri/core/src/slopfab/ffi/mod.rs
+++ b/src-tauri/core/src/slopfab/ffi/mod.rs
@@ -237,6 +237,12 @@ impl Api {
self.check(r)?;
self.inner.set_image_guidance(r.pointer.as_ptr(), scale, negative_prompt)
}
+ pub fn set_music_settings(&self, r: &RequestHandle, settings: &str) -> Result<(), String> {
+ self.check(r)?; self.inner.set_music_settings(r.pointer.as_ptr(), settings)
+ }
+ pub fn set_vulkan_portable(&self, r: &RequestHandle) -> Result<(), String> {
+ self.check(r)?; self.inner.set_vulkan_portable(r.pointer.as_ptr())
+ }
pub fn set_model_family(&self, r: &RequestHandle, family: &str) -> Result<(), String> {
self.check(r)?;
self.inner.set_model_family(r.pointer.as_ptr(), family)
diff --git a/src-tauri/core/src/slopfab/ffi/raw/mod.rs b/src-tauri/core/src/slopfab/ffi/raw/mod.rs
index e400f62..36d261a 100644
--- a/src-tauri/core/src/slopfab/ffi/raw/mod.rs
+++ b/src-tauri/core/src/slopfab/ffi/raw/mod.rs
@@ -80,6 +80,8 @@ type ExportRefmod = unsafe extern "C" fn(*const Request, *const c_char, *const c
type SetImageGuidance = unsafe extern "C" fn(*mut Request, f32, *const c_char, *const i32, i32) -> i32;
pub struct Api {
+ set_music_settings: Option<unsafe extern "C" fn(*mut Request, *const c_char) -> i32>,
+ set_vulkan_arithmetic: Option<unsafe extern "C" fn(*mut Request, i32) -> i32>,
set_image_guidance: Option<SetImageGuidance>,
_library: Library,
capi_version: unsafe extern "C" fn() -> u32,
@@ -247,6 +249,8 @@ impl Api {
unsafe extern "C" fn(*mut Request, u64) -> i32
),
set_image_guidance: library.get::<SetImageGuidance>(b"slopfab_request_set_image_guidance\0").ok().map(|symbol| *symbol),
+ set_music_settings: library.get::<unsafe extern "C" fn(*mut Request, *const c_char) -> i32>(b"slopfab_request_set_music_settings\0").ok().map(|symbol| *symbol),
+ set_vulkan_arithmetic: library.get::<unsafe extern "C" fn(*mut Request, i32) -> i32>(b"slopfab_request_set_vulkan_arithmetic\0").ok().map(|symbol| *symbol),
set_model_family: library
.get::<unsafe extern "C" fn(*mut Request, *const c_char) -> i32>(b"slopfab_request_set_model_family\0")
.ok().map(|symbol| *symbol),
@@ -601,6 +605,15 @@ impl Api {
let skip_layers = [22_i32];
self.error(unsafe { set(r, scale, negative.as_ptr(), skip_layers.as_ptr(), 1) })
}
+ pub fn set_music_settings(&self, r: *mut Request, settings: &str) -> Result<(), String> {
+ let set = self.set_music_settings.ok_or("Music requires SlopFab API 1.38 or newer. Update the app and workers.")?;
+ let json = CString::new(settings).map_err(|e| e.to_string())?;
+ self.error(unsafe { set(r, json.as_ptr()) })
+ }
+ pub fn set_vulkan_portable(&self, r: *mut Request) -> Result<(), String> {
+ let set = self.set_vulkan_arithmetic.ok_or("Music requires SlopFab API 1.38 or newer.")?;
+ self.error(unsafe { set(r, 1) })
+ }
pub fn set_model_family(&self, r: *mut Request, family: &str) -> Result<(), String> {
let set = self.set_model_family.ok_or("This model requires a newer SlopFab runtime. Update the machine running generation.")?;
let value = CString::new(family).map_err(|_| "Invalid model family.")?;
diff --git a/src-tauri/core/src/slopfab/generation.rs b/src-tauri/core/src/slopfab/generation.rs
index 62221c7..64f8764 100644
--- a/src-tauri/core/src/slopfab/generation.rs
+++ b/src-tauri/core/src/slopfab/generation.rs
@@ -174,7 +174,7 @@ fn run_single_generation(
}
fn collect_output(item: &QueueItem, output: ffi::Output, source: Box<dyn rendered::FrameSource>, timing_profile: String) -> Result<(OutputMetadata, rendered::RenderedVideo), String> {
- let (_, frames) = scene_frame_window(&item.request, output.frames)?;
+ let frames = if item.request.music.is_some() { 0 } else { scene_frame_window(&item.request, output.frames)?.1 };
// The VAE decoded both sides of the boundary together. Keep that complete
// output, including the source-side pixels and soundtrack near the join.
let audio = output.audio;
@@ -191,12 +191,14 @@ fn collect_output(item: &QueueItem, output: ffi::Output, source: Box<dyn rendere
seconds_total: output.seconds_total,
steps_computed: output.steps_computed,
steps_skipped: output.steps_skipped,
- duration_seconds: if output.fps > 0.0 {
+ duration_seconds: if item.request.music.is_some() && output.audio_channels > 0 && output.audio_sample_rate > 0 {
+ audio.len() as f64 / output.audio_channels as f64 / output.audio_sample_rate as f64
+ } else if output.fps > 0.0 {
frames as f64 / output.fps
} else {
0.0
},
- boundary: "Latents are saved in the project. Decoded buffers remain owned by slopfab; the webview encodes the full joined video and audio on demand.",
+ boundary: if item.request.music.is_some() { "Music samples are ready to save as WAV." } else { "Latents are saved in the project. Decoded buffers remain owned by slopfab; the webview encodes the full joined video and audio on demand." },
};
let pictures = rendered::from_source(
&item.request.job_id,
diff --git a/src-tauri/core/src/slopfab/generation_tests.rs b/src-tauri/core/src/slopfab/generation_tests.rs
index 2814201..d0694e7 100644
--- a/src-tauri/core/src/slopfab/generation_tests.rs
+++ b/src-tauri/core/src/slopfab/generation_tests.rs
@@ -2,6 +2,32 @@ use super::*;
use crate::slopfab::{config::Configuration, events::NativeEvent, references::ReferenceVideos, types::{GenerationRequest, ProgressEvent}};
use std::sync::{Arc, Mutex, atomic::AtomicBool};
+#[test]
+#[ignore = "requires local music weights and a GPU; set SLOPUS_MUSIC_WEIGHTS"]
+fn music_gpu_smoke() {
+ use crate::settings::{ProviderOption, ProviderSetting};
+ let root = std::path::PathBuf::from(std::env::var("SLOPUS_MUSIC_WEIGHTS").unwrap());
+ for family in ["yue2", "minimax_music3"] {
+ let paths = if family == "yue2" { vec![("transformer", "yue2_3b_int8_convrot.safetensors")] } else {
+ vec![("transformer", "minimax_music3_dit_int8_convrot.safetensors"), ("textEncoder", "text_encoder/minimax_music3_text_encoder_pruned_int8_convrot.safetensors"), ("audioVae", "minimax_music3_dav.safetensors")]
+ };
+ let mut options = std::collections::BTreeMap::from([("modelType".into(), ProviderOption::String(family.into()))]);
+ for (key, file) in paths { options.insert(key.into(), ProviderOption::String(root.join(file).to_string_lossy().into_owned())); }
+ let settings = std::collections::BTreeMap::from([("slopfab".into(), ProviderSetting { enabled: true, model: None, options })]);
+ let request = GenerationRequest { job_id: format!("smoke-{family}"), prompt: "Quiet solo piano, instrumental".into(), steps: 2, seed: 42,
+ music: Some(crate::slopfab::music::MusicSettings { duration: 2.0, planning: if family == "yue2" { "off" } else { "full" }.into(), ..Default::default() }), ..Default::default() };
+ let item = QueueItem { request, configuration: Configuration::from_settings(&settings), references: ReferenceVideos::default(),
+ events: Arc::new(|_| {}), cancel: Arc::new(AtomicBool::new(false)) };
+ let (metadata, output) = run_generation(&item).unwrap();
+ assert_eq!(metadata.frames, 0);
+ assert_eq!(metadata.audio_channels, 2);
+ println!("{family}: {} seconds, {} Hz stereo", metadata.duration_seconds, metadata.audio_sample_rate);
+ assert!((1.9..=2.1).contains(&metadata.duration_seconds));
+ assert!(output.frame(0).unwrap().is_none());
+ assert!(!output.audio_bytes().is_empty());
+ }
+}
+
struct Pixels(u8);
impl rendered::FrameSource for Pixels {
fn frame_rgba(&self, _: u32, _: u32, _: u32) -> Result<Vec<u8>, String> { Ok(vec![self.0, 20, 30, 255]) }
diff --git a/src-tauri/core/src/slopfab/h3.rs b/src-tauri/core/src/slopfab/h3.rs
index f89dbcf..a93df59 100644
--- a/src-tauri/core/src/slopfab/h3.rs
+++ b/src-tauri/core/src/slopfab/h3.rs
@@ -17,6 +17,7 @@ pub(super) fn configure_request(
purpose: RequestPurpose,
references: &ReferenceVideos,
) -> Result<(), String> {
+ if request.music.is_some() { return super::music::configure(api, handle, request, configuration, platform); }
configuration.validate_inputs(request)?;
if crate::models::generator_features(&configuration.model_type).cuda_only {
if platform == ComputePlatform::Vulkan {
diff --git a/src-tauri/core/src/slopfab/mod.rs b/src-tauri/core/src/slopfab/mod.rs
index 5426fc9..a91e2d5 100644
--- a/src-tauri/core/src/slopfab/mod.rs
+++ b/src-tauri/core/src/slopfab/mod.rs
@@ -18,6 +18,7 @@ mod generation;
mod lcs;
mod lcs_fingerprint;
mod h3;
+pub mod music;
mod icons;
mod planning;
mod platform;
diff --git a/src-tauri/core/src/slopfab/music.rs b/src-tauri/core/src/slopfab/music.rs
new file mode 100644
index 0000000..13a3f0e
--- /dev/null
+++ b/src-tauri/core/src/slopfab/music.rs
@@ -0,0 +1,86 @@
+//! Audio-only music requests. Do not apply H3 video or sampling defaults.
+use super::{config::Configuration, ffi::{Api, RequestHandle}, platform::ComputePlatform, types::GenerationRequest};
+use serde::{Deserialize, Serialize};
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(default, deny_unknown_fields)]
+pub struct MusicSettings {
+ pub lyrics: String,
+ pub duration: f64,
+ pub planning: String,
+ pub abc: String,
+ pub abc_max_tokens: i32,
+ pub temperature: f64,
+ pub top_p: f64,
+ pub top_k: i32,
+ pub repetition_penalty: f64,
+ pub semantic_guidance: f64,
+ pub flow_guidance: f64,
+}
+impl Default for MusicSettings {
+ fn default() -> Self { Self { lyrics: String::new(), duration: 30.0, planning: "full".into(), abc: String::new(),
+ abc_max_tokens: 4096, temperature: 1.0, top_p: 0.95, top_k: 0, repetition_penalty: 1.2,
+ semantic_guidance: 0.0, flow_guidance: 0.0 } }
+}
+impl MusicSettings {
+ pub fn validate(&self) -> Result<(), String> {
+ if !self.duration.is_finite() || !(2.0..=300.0).contains(&self.duration)
+ || !matches!(self.planning.as_str(), "full" | "melody" | "off")
+ || !(32..=4096).contains(&self.abc_max_tokens) || !(0..=1024).contains(&self.top_k)
+ || !self.temperature.is_finite() || !(0.0..=10.0).contains(&self.temperature)
+ || !self.top_p.is_finite() || self.top_p <= 0.0 || self.top_p > 1.0
+ || !self.repetition_penalty.is_finite() || !(1.0..=10.0).contains(&self.repetition_penalty)
+ || !self.semantic_guidance.is_finite() || !(0.0..=10.0).contains(&self.semantic_guidance)
+ || (self.semantic_guidance > 0.0 && self.semantic_guidance < 1.0)
+ || !self.flow_guidance.is_finite() || !(0.0..=10.0).contains(&self.flow_guidance)
+ || self.lyrics.chars().count() > 30000 || self.abc.chars().count() > 30000
+ || self.lyrics.contains('\0') || self.abc.contains('\0') {
+ return Err("Invalid music settings. Use a duration from 2 to 300 seconds and supported sampling ranges.".into());
+ }
+ if self.planning == "off" && !self.abc.is_empty() { return Err("ABC requires full or melody planning.".into()); }
+ Ok(())
+ }
+}
+
+pub(super) fn validate(request: &GenerationRequest, config: &Configuration) -> Result<(), String> {
+ let settings = request.music.as_ref().ok_or("Music settings are missing.")?;
+ settings.validate()?;
+ if !matches!(config.model_type.as_str(), "yue2" | "minimax_music3") { return Err("Choose a music generator.".into()); }
+ if request.prompt.trim().is_empty() || request.prompt.contains('\0') || request.prompt.chars().count() > 10000
+ || !(2..=100).contains(&request.steps) || request.seed < -1 { return Err("Music needs a style prompt, 2–100 steps and a valid seed.".into()); }
+ if request.still_image || request.image_edit.is_some() || request.reference_count() != 0 || !request.refmods.is_empty()
+ || request.latent_upscale || request.latent_bridge.is_some() || request.continuation_path.is_some()
+ || request.continuation_relative_path.is_some() || request.video_transition.is_some() || request.save_latents_path.is_some()
+ || request.audio_steps.is_some() || request.video_sigma_shift.is_some() || request.audio_sigma_shift.is_some()
+ || request.guidance_scale.is_some() || request.negative_prompt.is_some() || request.generation_effects.is_some()
+ || config.animate || config.motion_cache || config.prompt_embedding.is_some()
+ || !config.loras.as_ref().map_err(Clone::clone)?.is_empty() {
+ return Err("Music does not support image/video controls, references, LoRAs or generation effects.".into());
+ }
+ let defaults = MusicSettings::default();
+ if config.model_type == "minimax_music3" && (settings.planning != defaults.planning || !settings.abc.is_empty()
+ || settings.abc_max_tokens != defaults.abc_max_tokens || settings.temperature != defaults.temperature
+ || settings.top_p != defaults.top_p || settings.repetition_penalty != defaults.repetition_penalty) {
+ return Err("MiniMax Music3 does not support YuE2 planning and sampling controls.".into());
+ }
+ if config.model_type == "yue2" && settings.flow_guidance != 0.0 { return Err("Flow guidance is available for MiniMax Music3 only.".into()); }
+ Ok(())
+}
+
+pub(super) fn configure(api: &Api, handle: &RequestHandle, request: &GenerationRequest, config: &Configuration, platform: ComputePlatform) -> Result<(), String> {
+ validate(request, config)?;
+ api.set_model_family(handle, &config.model_type)?;
+ api.set_prompt(handle, &request.prompt)?;
+ api.set_music_settings(handle, &serde_json::to_string(request.music.as_ref().unwrap()).map_err(|e| e.to_string())?)?;
+ api.set_steps(handle, request.steps)?;
+ api.set_seed(handle, super::planning::generation_seed(request.seed)?)?;
+ api.set_inference_backend(handle, platform.backend())?;
+ if platform == ComputePlatform::Vulkan { api.set_vulkan_portable(handle)?; }
+ api.set_attention(handle, "flash2")?;
+ for (id, _, path) in &config.models {
+ if (*id == 0 || config.model_type == "minimax_music3" && (*id == 1 || *id == 4)) && path.is_some() {
+ api.set_model(handle, *id, path.as_ref().unwrap())?;
+ }
+ }
+ Ok(())
+}
diff --git a/src-tauri/core/src/slopfab/planning.rs b/src-tauri/core/src/slopfab/planning.rs
index 81ae8f6..4e534f0 100644
--- a/src-tauri/core/src/slopfab/planning.rs
+++ b/src-tauri/core/src/slopfab/planning.rs
@@ -40,12 +40,12 @@ pub fn resolve_plan(
)?;
let plan = api.resolve(&handle)?;
let description = api.describe(&handle)?;
- let (_, frames) = scene_frame_window(request, plan.aligned_frames)?;
+ let frames = if request.music.is_some() { 0 } else { scene_frame_window(request, plan.aligned_frames)?.1 };
Ok(ResolvedPlan {
canvas_width: if request.latent_upscale { request.canvas_width } else { plan.canvas_width },
canvas_height: if request.latent_upscale { request.canvas_height } else { plan.canvas_height },
aligned_frames: frames,
- duration_seconds: plan.duration_seconds * frames as f64 / plan.aligned_frames as f64,
+ duration_seconds: if request.music.is_some() { plan.duration_seconds } else { plan.duration_seconds * frames as f64 / plan.aligned_frames as f64 },
model_evaluations: plan.num_model_evaluations * request.image_edit.as_ref().map_or(1, |edit| edit.edits.len() as i32),
sequence_rows_without_text: plan.sequence_rows_without_text,
latent_frames: plan.latent_frames,
@@ -57,6 +57,7 @@ pub fn resolve_plan(
}
pub(super) fn validate_generation_controls(request: &GenerationRequest) -> Result<(), String> {
+ if let Some(music) = &request.music { return music.validate(); }
if let Some(scale) = request.guidance_scale { crate::models::validate_cfg_scale(f64::from(scale))?; }
if let Some(effects) = &request.generation_effects { effects.validate()?; }
super::continuation::validate(request)?;
diff --git a/src-tauri/core/src/slopfab/status.rs b/src-tauri/core/src/slopfab/status.rs
index aece519..c9f68ef 100644
--- a/src-tauri/core/src/slopfab/status.rs
+++ b/src-tauri/core/src/slopfab/status.rs
@@ -6,7 +6,11 @@ fn model_statuses(configuration: &Configuration, available: impl Fn(&Path) -> bo
let mut models = configuration
.models
.iter()
- .filter(|(id, _, _)| crate::models::generator_features(&configuration.model_type).video_generation || *id != 4)
+ .filter(|(id, _, _)| match configuration.model_type.as_str() {
+ "yue2" => *id == 0,
+ "minimax_music3" => matches!(id, 0 | 1 | 4),
+ _ => crate::models::generator_features(&configuration.model_type).video_generation || *id != 4,
+ })
.filter(|(id, _, _)| !configuration.animate || (*id != 1 && *id != 2))
.map(|(_, id, path)| ModelStatus {
id,
diff --git a/src-tauri/core/src/slopfab/tests.rs b/src-tauri/core/src/slopfab/tests.rs
index bbf00a5..a918042 100644
--- a/src-tauri/core/src/slopfab/tests.rs
+++ b/src-tauri/core/src/slopfab/tests.rs
@@ -5,6 +5,31 @@ use super::{
use crate::settings::{ProviderOption, ProviderSetting};
use std::collections::BTreeMap;
+#[test]
+fn music_plans_on_cuda_and_vulkan_without_video_controls() {
+ let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../lib/slopfab").join(super::DLL_FILE_NAME);
+ let api = ffi::Api::load(&path).unwrap();
+ for family in ["yue2", "minimax_music3"] {
+ let mut configuration = Configuration::from_settings(&BTreeMap::new());
+ configuration.model_type = family.into();
+ let mut request = GenerationRequest { prompt: "Quiet piano".into(), steps: 2, seed: 42,
+ music: Some(music::MusicSettings { duration: 2.0, ..Default::default() }), ..Default::default() };
+ for backend in [ComputePlatform::Cuda13, ComputePlatform::Vulkan] {
+ let handle = RequestHandle::new(&api).unwrap();
+ configure_request(&api, &handle, &request, &configuration, backend, RequestPurpose::Plan, &ReferenceVideos::default()).unwrap();
+ let plan = api.resolve(&handle).unwrap();
+ assert!(plan.duration_seconds >= 2.0);
+ assert_eq!(plan.canvas_width, 0);
+ assert_eq!(plan.canvas_height, 0);
+ }
+ request.reference_paths.push("reference.wav".into());
+ assert!(configuration.validate_inputs(&request).unwrap_err().contains("references"));
+ request.reference_paths.clear();
+ request.music.as_mut().unwrap().duration = f64::NAN;
+ assert!(configuration.validate_inputs(&request).is_err());
+ }
+}
+
#[test]
fn moss_capabilities_reject_video_edits_and_references() {
let mut configuration = Configuration::from_settings(&BTreeMap::new());
diff --git a/src-tauri/core/src/slopfab/types.rs b/src-tauri/core/src/slopfab/types.rs
index 7fe874e..c237cf6 100644
--- a/src-tauri/core/src/slopfab/types.rs
+++ b/src-tauri/core/src/slopfab/types.rs
@@ -54,6 +54,8 @@ pub struct LatentBridgeRequest {
#[derive(Debug, Clone, Default, Deserialize, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct GenerationRequest {
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub music: Option<super::music::MusicSettings>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub negative_prompt: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
diff --git a/src-tauri/core/src/worker/protocol.rs b/src-tauri/core/src/worker/protocol.rs
index f08e48c..c59eaca 100644
--- a/src-tauri/core/src/worker/protocol.rs
+++ b/src-tauri/core/src/worker/protocol.rs
@@ -23,7 +23,7 @@ pub const SERVICE_TYPE: &str = "_slopus-worker._tcp.local.";
// v6 added custom upscale weights.
// v5 added overlap locking.
// v4 added SeedVR2 export streams. v3 added inverted image masks.
-pub const PROTOCOL_VERSION: u32 = 17;
+pub const PROTOCOL_VERSION: u32 = 18;
/// Streaming SeedVR2 restoration; model paths are resolved on the worker.
#[derive(Debug, Clone, Serialize, Deserialize)]
diff --git a/src-tauri/core/src/worker/server.rs b/src-tauri/core/src/worker/server.rs
index 669d8a8..ee80540 100644
--- a/src-tauri/core/src/worker/server.rs
+++ b/src-tauri/core/src/worker/server.rs
@@ -661,7 +661,7 @@ impl Worker {
}
self.prune_jobs();
let (mut request, settings) = self.materialize(start.job)?;
- let latents = if start.save_latents && !request.still_image {
+ let latents = if start.save_latents && !request.still_image && request.music.is_none() {
let directory = self.data.join("jobs").join(id);
let _ = fs::remove_dir_all(&directory);
fs::create_dir_all(&directory).map_err(|error| error.to_string())?;
diff --git a/src-tauri/src/app.rs b/src-tauri/src/app.rs
index 7e3dc1e..a2216e9 100644
--- a/src-tauri/src/app.rs
+++ b/src-tauri/src/app.rs
@@ -179,6 +179,7 @@ pub fn run() {
.plugin(tauri_plugin_process::init())
.invoke_handler(tauri::generate_handler![
commands::artifacts::save_generated_image,
+ commands::artifacts::save_generated_music,
commands::character_sheet::save_character_sheet_view,
commands::character_sheet::combine_character_sheet,
commands::extend_image::prepare_extend_image,
diff --git a/src-tauri/src/commands/artifacts.rs b/src-tauri/src/commands/artifacts.rs
index 68bcdd6..db6a915 100644
--- a/src-tauri/src/commands/artifacts.rs
+++ b/src-tauri/src/commands/artifacts.rs
@@ -146,6 +146,62 @@ pub(crate) fn generated_summary(job_id: String) -> Option<rendered::RenderedSumm
rendered::summary(&job_id)
}
+/// Save audio-only generations as PCM WAV without a media subprocess.
+#[tauri::command]
+pub(crate) async fn save_generated_music(folder_path: String, job_id: String) -> Result<serde_json::Value, String> {
+ tauri::async_runtime::spawn_blocking(move || {
+ let summary = rendered::summary(&job_id).ok_or("Generated music is no longer available.")?;
+ let samples = rendered::audio(&job_id).ok_or("Generated music has no audio.")?;
+ let wav = music_wav(&samples, summary.audio_channels, summary.audio_sample_rate)?;
+ let root = ProjectRoot::open(&folder_path)?;
+ let stem = generated_file_stem(&job_id)?;
+ let relative_path = format!("media/generated/{stem}.wav");
+ crate::storage::atomic::write_atomically(&root.directory("media/generated")?.join(format!("{stem}.wav")), &wav)?;
+ Ok(serde_json::json!({ "relativePath": relative_path,
+ "durationMs": (samples.len() as f64 / 4.0 / f64::from(summary.audio_channels) / f64::from(summary.audio_sample_rate) * 1000.0).round() as u64 }))
+ }).await.map_err(|e| e.to_string())?
+}
+
+fn music_wav(samples: &[u8], channels: u32, rate: u32) -> Result<Vec<u8>, String> {
+ if !(1..=2).contains(&channels) || !(8000..=192000).contains(&rate) || samples.is_empty()
+ || samples.len() % (4 * channels as usize) != 0 { return Err("Invalid generated audio format.".into()); }
+ let size = u32::try_from(samples.len() / 4 * 3).map_err(|_| "Generated audio is too large.")?;
+ let riff_size = size.checked_add(36).ok_or("Generated audio is too large.")?;
+ let mut wav = Vec::with_capacity(size as usize + 44);
+ wav.extend_from_slice(b"RIFF"); wav.extend_from_slice(&riff_size.to_le_bytes()); wav.extend_from_slice(b"WAVEfmt ");
+ wav.extend_from_slice(&16_u32.to_le_bytes()); wav.extend_from_slice(&1_u16.to_le_bytes());
+ wav.extend_from_slice(&(channels as u16).to_le_bytes()); wav.extend_from_slice(&rate.to_le_bytes());
+ wav.extend_from_slice(&(rate * channels * 3).to_le_bytes()); wav.extend_from_slice(&((channels * 3) as u16).to_le_bytes());
+ wav.extend_from_slice(&24_u16.to_le_bytes()); wav.extend_from_slice(b"data"); wav.extend_from_slice(&size.to_le_bytes());
+ for bytes in samples.chunks_exact(4) {
+ let sample = f32::from_le_bytes(bytes.try_into().unwrap());
+ if !sample.is_finite() { return Err("Generated audio contains non-finite samples.".into()); }
+ let pcm = (sample.clamp(-1.0, 1.0) * 8_388_607.0).round() as i32;
+ wav.extend_from_slice(&pcm.to_le_bytes()[..3]);
+ }
+ Ok(wav)
+}
+
+#[cfg(test)]
+mod music_tests {
+ use super::music_wav;
+ #[test]
+ fn music_wav_encodes_stereo_pcm_and_rejects_invalid_samples() {
+ let samples: Vec<u8> = [0_f32, 1.0, -1.0, 0.5].into_iter().flat_map(f32::to_le_bytes).collect();
+ let wav = music_wav(&samples, 2, 48000).unwrap();
+ assert_eq!(&wav[..4], b"RIFF");
+ assert_eq!(&wav[8..16], b"WAVEfmt ");
+ assert_eq!(u16::from_le_bytes(wav[22..24].try_into().unwrap()), 2);
+ assert_eq!(u32::from_le_bytes(wav[24..28].try_into().unwrap()), 48000);
+ assert_eq!(u16::from_le_bytes(wav[34..36].try_into().unwrap()), 24);
+ assert_eq!(&wav[44..50], &[0, 0, 0, 255, 255, 127]);
+ assert_eq!(wav.len(), 56);
+ assert!(music_wav(&[0; 3], 2, 48000).is_err());
+ assert!(music_wav(&f32::NAN.to_le_bytes(), 1, 48000).is_err());
+ assert!(music_wav(&samples, 0, 48000).is_err());
+ }
+}
+
/// Save only a true 256x256 still-image result, directly from the DLL's frame
/// buffer. Icon artwork is stored separately from conditioning attachments.
#[tauri::command]
diff --git a/src-tauri/src/commands/generation.rs b/src-tauri/src/commands/generation.rs
index 278f707..0da080f 100644
--- a/src-tauri/src/commands/generation.rs
+++ b/src-tauri/src/commands/generation.rs
@@ -249,7 +249,7 @@ pub(crate) async fn enqueue_slopfab_generation(
blocking(move || {
prepare_continuation_path(&mut request, Some(&folder_path))?;
prepare_image_edit_path(&mut request, Some(&folder_path))?;
- if !request.still_image {
+ if !request.still_image && request.music.is_none() {
request.save_latents_path = Some(generated_latent_destination(&folder_path, &request.job_id)?);
}
let job_id = request.job_id.clone();
diff --git a/src-tauri/src/project/mod.rs b/src-tauri/src/project/mod.rs
index 462d892..1324ff1 100644
--- a/src-tauri/src/project/mod.rs
+++ b/src-tauri/src/project/mod.rs
@@ -17,3 +17,4 @@ pub(crate) mod commands;
pub(crate) mod reference_tokens;
pub(crate) mod motion;
+pub(crate) mod music;
diff --git a/src-tauri/src/project/model.rs b/src-tauri/src/project/model.rs
index 0489d04..c610c36 100644
--- a/src-tauri/src/project/model.rs
+++ b/src-tauri/src/project/model.rs
@@ -18,6 +18,8 @@ pub(crate) enum GenerationType {
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub(crate) struct ProjectConfig {
+ #[serde(default, skip_serializing_if = "Vec::is_empty")]
+ pub(crate) music_songs: Vec<super::music::MusicSong>,
pub(crate) schema_version: u32,
#[serde(default)]
pub(crate) generation_type: GenerationType,
diff --git a/src-tauri/src/project/music.rs b/src-tauri/src/project/music.rs
new file mode 100644
index 0000000..2d0e459
--- /dev/null
+++ b/src-tauri/src/project/music.rs
@@ -0,0 +1,31 @@
+use serde::{Deserialize, Serialize};
+use slopus_core::slopfab::music::MusicSettings;
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub(crate) struct MusicSong {
+ pub id: String,
+ pub title: String,
+ #[serde(default)] pub style: String,
+ pub generator_id: String,
+ pub settings: MusicSettings,
+ pub steps: i32,
+ pub seed: i64,
+ pub status: String,
+ pub asset_id: Option<String>,
+ pub error: Option<String>,
+ pub created_at: String,
+}
+
+pub(crate) fn validate(config: &super::ProjectConfig) -> Result<(), String> {
+ let mut ids = std::collections::HashSet::new();
+ for song in &config.music_songs {
+ if song.id.is_empty() || !ids.insert(&song.id) || song.title.trim().is_empty() || song.title.chars().count() > 200
+ || song.style.chars().count() > 10000 || !(2..=100).contains(&song.steps) || song.seed < -1
+ || !matches!(song.status.as_str(), "draft" | "queued" | "generating" | "completed" | "failed" | "cancelled") {
+ return Err("Invalid music song.".into());
+ }
+ song.settings.validate()?;
+ }
+ Ok(())
+}
diff --git a/src-tauri/src/project/validation/mod.rs b/src-tauri/src/project/validation/mod.rs
index 22e325b..33bfd65 100644
--- a/src-tauri/src/project/validation/mod.rs
+++ b/src-tauri/src/project/validation/mod.rs
@@ -21,6 +21,7 @@ pub(crate) fn validate_and_normalize_config(
pub(crate) fn validate_checked(
mut config: ProjectConfig,
) -> Result<ProjectConfig, issue::ValidationIssue> {
+ super::music::validate(&config).map_err(|message| issue::ValidationIssue::new("project.invalid", Some(config.id.clone()), "musicSongs", message))?;
if let Some(scene) = &config.image_scene {
scene.validate().map_err(|message| {
issue::ValidationIssue::new(
diff --git a/src-tauri/src/tests/schema.rs b/src-tauri/src/tests/schema.rs
index 218831a..48dc651 100644
--- a/src-tauri/src/tests/schema.rs
+++ b/src-tauri/src/tests/schema.rs
@@ -1,5 +1,22 @@
use super::*;
+#[test]
+fn music_settings_survive_project_validation_and_reopen() {
+ let mut config = created_fixture();
+ config.music_songs.push(serde_json::from_value(serde_json::json!({
+ "id": "song-1", "title": "Evening", "style": "Quiet jazz", "generatorId": "yue2",
+ "settings": { "lyrics": "[Verse]\nHome again", "duration": 30 }, "steps": 32, "seed": -1,
+ "status": "draft", "createdAt": "2026-10-25T00:00:00Z"
+ })).unwrap());
+ let saved = validate_and_normalize_config(config).unwrap();
+ let reopened: ProjectConfig = serde_json::from_slice(&serde_json::to_vec(&saved).unwrap()).unwrap();
+ assert_eq!(reopened.music_songs, saved.music_songs);
+ assert_eq!(reopened.music_songs[0].settings.lyrics, "[Verse]\nHome again");
+ let mut invalid = reopened;
+ invalid.music_songs[0].steps = 1000;
+ assert!(validate_and_normalize_config(invalid).is_err());
+}
+
#[test]
fn rust_never_emits_a_null_the_frontend_schema_refuses() {
// The cross-layer direction that went unchecked for nine rounds: Rust
diff --git a/src/App.tsx b/src/App.tsx
index f14e791..98eb7c7 100644
--- a/src/App.tsx
+++ b/src/App.tsx
@@ -230,6 +230,8 @@ function App() {
if (item.kind === "image") {
const imageId = [item.imageDraftId, item.id, item.imageAssetId].find((id) => record.config.assets.some((asset) => asset.kind === "image" && asset.id === id));
navigation = { view: "editor", imageId: imageId ?? undefined };
+ } else if (item.kind === "music") {
+ navigation = { view: "music", songId: item.sceneId };
} else if (item.kind === "refmod" || item.kind === "reference-icons") {
navigation = { view: "references", referenceId: item.sceneId };
} else navigation = { view: "generator", sceneId: item.sceneId };
diff --git a/src/components/ImageProject.integration.test.tsx b/src/components/ImageProject.integration.test.tsx
index e6734ff..a651a56 100644
--- a/src/components/ImageProject.integration.test.tsx
+++ b/src/components/ImageProject.integration.test.tsx
@@ -312,11 +312,11 @@ it("creates one project without a separate image or video type", async () => {
expect(submit.mock.calls[0][0]).not.toHaveProperty("generationType");
});
-it("opens a legacy image project on Image with all five tabs and saves composition edits", async () => {
+it("opens a legacy image project on Image with all six tabs and saves composition edits", async () => {
const save = vi.fn(async () => undefined);
render(<ProjectWorkspace project={record()} onBack={vi.fn()} onSave={save} />);
const navigation = within(screen.getByRole("tablist", { name: "Project views" }));
- expect(navigation.getAllByRole("tab").map((tab) => tab.textContent)).toEqual(["Timeline", "Video", "Image", "References", "Export"]);
+ expect(navigation.getAllByRole("tab").map((tab) => tab.textContent)).toEqual(["Timeline", "Video", "Image", "Music", "References", "Export"]);
expect(navigation.getByRole("tab", { name: "Image" })).toHaveAttribute("aria-selected", "true");
fireEvent.contextMenu(screen.getByRole("button", { name: "Image" }));
fireEvent.click(screen.getByRole("menuitem", { name: "New Text" }));
diff --git a/src/components/ProjectWorkspace.test.ts b/src/components/ProjectWorkspace.test.ts
index 524893d..7ab00e7 100644
--- a/src/components/ProjectWorkspace.test.ts
+++ b/src/components/ProjectWorkspace.test.ts
@@ -128,7 +128,7 @@ describe("project workspace timecode", () => {
// The views are tabs in the title bar; the agent is not one of them.
const tabs = screen.getByRole("tablist", { name: "Project views" });
- expect(within(tabs).getAllByRole("tab").map((tab) => tab.textContent?.trim())).toEqual(["Timeline", "Video", "Image", "References", "Export"]);
+ expect(within(tabs).getAllByRole("tab").map((tab) => tab.textContent?.trim())).toEqual(["Timeline", "Video", "Image", "Music", "References", "Export"]);
expect(within(tabs).getByRole("tab", { name: "Timeline" })).toHaveAttribute("aria-selected", "true");
const panel = document.getElementById(within(tabs).getByRole("tab", { name: "Timeline" }).getAttribute("aria-controls")!);
expect(panel).toHaveAttribute("role", "tabpanel");
diff --git a/src/components/ProjectWorkspace.tsx b/src/components/ProjectWorkspace.tsx
index fc6a90e..cd73002 100644
--- a/src/components/ProjectWorkspace.tsx
+++ b/src/components/ProjectWorkspace.tsx
@@ -1,4 +1,6 @@
import { Agent16, Agent16Filled, Back16, ChevronDown12, Download16, Download16Filled, Film16, Film16Filled, Image16, Image16Filled, Redo16, References16, References16Filled, Save16, Sparkle16, Sparkle16Filled, Undo16 } from "./ui/icons";
+import { MusicView } from "./workspace/MusicView";
+import { Audio16 } from "./ui/icons";
import { ImageEditor } from "./workspace/ImageEditor";
import { useEffect, useId, useMemo, useRef, useState, useSyncExternalStore, type ReactNode } from "react";
import type { GenerationJob, ProjectConfig, ProjectRecord } from "../lib/project";
@@ -24,9 +26,9 @@ import { outputDimensions, videoDurationMs } from "../lib/export";
/** "agent" is not a view any more — the agent is a docked pane — but opening
* a project "on the agent" still means something: the pane opens with it. */
-export type ProjectView = "timeline" | "generator" | "references" | "agent" | "export" | "editor";
+export type ProjectView = "music" | "timeline" | "generator" | "references" | "agent" | "export" | "editor";
type ShownView = Exclude<ProjectView, "agent">;
-export interface WorkspaceNavigation { view: ShownView; sceneId?: string; imageId?: string; referenceId?: string; exportKind?: "video" | "image" }
+export interface WorkspaceNavigation { view: ShownView; sceneId?: string; imageId?: string; songId?: string; referenceId?: string; exportKind?: "video" | "image" }
/** The scene badge includes native generation and the final encoding step. */
export const isGenerationOngoing = (job: GenerationJob) =>
@@ -67,7 +69,7 @@ const writeAgentPane = (open: boolean) => {
};
const VIEW_LABELS: Record<ShownView, string> = {
- timeline: "Timeline", generator: "Video", references: "References", export: "Export", editor: "Image",
+ music: "Music", timeline: "Timeline", generator: "Video", references: "References", export: "Export", editor: "Image",
};
/* The project window.
@@ -144,6 +146,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
const [savingProjectSettings, setSavingProjectSettings] = useState(false);
const [selectedGenerationJobId, setSelectedGenerationJobId] = useState<string | undefined>(project.config.generationJobs[0]?.id);
const [imageNavigation, setImageNavigation] = useState<{ id: string }>();
+ const [musicNavigation, setMusicNavigation] = useState<{ id: string }>();
const [referenceNavigation, setReferenceNavigation] = useState<{ id: string }>();
useEffect(() => {
if (!navigation) return;
@@ -151,6 +154,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
if (navigation.view === "export") setMediaMode(navigation.exportKind ?? "video");
if (navigation.sceneId) setSelectedGenerationJobId(navigation.sceneId);
if (navigation.imageId) setImageNavigation({ id: navigation.imageId });
+ if (navigation.songId) setMusicNavigation({ id: navigation.songId });
if (navigation.referenceId) setReferenceNavigation({ id: navigation.referenceId });
onNavigated?.();
}, [navigation]);
@@ -194,7 +198,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
const lengthMs = useMemo(() => videoDurationMs(config), [config]);
const sequenceFormat = `${frameWidth}×${frameHeight} · ${config.settings.frameRate} fps · ${formatSequenceLength(lengthMs)}`;
- const views: ShownView[] = ["timeline", "generator", "editor", "references", "export"];
+ const views: ShownView[] = ["timeline", "generator", "editor", "music", "references", "export"];
const running = config.generationJobs.filter(isGenerationOngoing).length;
const toggleAgent = () => {
const open = !agentOpen;
@@ -208,7 +212,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
useShortcut("Ctrl+S", () => { if (dirty && !saving) void save(); }, { enabled: idle, allowInInput: true });
useShortcut("Ctrl+Z", () => { session.undo(); }, { enabled: idle });
useShortcut(["Ctrl+Y", "Ctrl+Shift+Z"], () => { session.redo(); }, { enabled: idle });
- useShortcut(["Ctrl+1", "Ctrl+2", "Ctrl+3", "Ctrl+4", "Ctrl+5"], (event) => {
+ useShortcut(["Ctrl+1", "Ctrl+2", "Ctrl+3", "Ctrl+4", "Ctrl+5", "Ctrl+6"], (event) => {
const next = views[Number(event.code?.replace("Digit", "") || event.key) - 1];
if (!next) return false;
setView(next);
@@ -217,7 +221,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
useShortcut("Ctrl+Shift+A", toggleAgent, { enabled: idle, allowInInput: true });
useShortcut("Alt+ArrowLeft", goBack, { enabled: idle });
- const viewIcon = (id: ShownView, selected: boolean) => id === "timeline" ? (selected ? <Film16Filled /> : <Film16 />)
+ const viewIcon = (id: ShownView, selected: boolean) => id === "music" ? <Audio16 /> : id === "timeline" ? (selected ? <Film16Filled /> : <Film16 />)
: id === "generator" ? (selected ? <Sparkle16Filled /> : <Sparkle16 />)
: id === "references" ? (selected ? <References16Filled /> : <References16 />)
: id === "export" ? (selected ? <Download16Filled /> : <Download16 />)
@@ -286,6 +290,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
<ProjectStatusSlot.Provider value={statusSlot}>
<div className={`project-body${agentOpen ? " project-body--agent" : ""}`} style={agentPane.style}>
<div id={`${panelId}-view`} role="tabpanel" aria-label={VIEW_LABELS[view]} className={`project-content project-content--${view}`}>
+ {view === "music" && <MusicView openSong={musicNavigation} onSongOpened={() => setMusicNavigation(undefined)} config={config} folderPath={project.folderPath} onChange={changeConfig} onGenerate={(id, template) => queue.enqueueMusic(session, id, template)} onCancel={(id) => void queue.cancel(id)} workItems={projectItems.filter((item) => item.kind === "music")} />}
{view === "editor" && <ImageEditor openImage={imageNavigation} onImageOpened={() => setImageNavigation(undefined)} config={config} folderPath={project.folderPath} onChange={changeConfig} onGenerate={(template, count) => queue.enqueueImage(session, template, count)} onGenerateCharacterSheet={(template, sourceId, options) => queue.enqueueCharacterSheet(session, template, sourceId, options)} onGenerateExtend={(template, sourceId, options) => queue.enqueueExtendImage(session, template, sourceId, options)} onCancel={(id) => queue.cancel(id)} workItems={projectItems.filter((item) => item.kind === "image")} onOpenGenerator={(jobId) => { setSelectedGenerationJobId(jobId); setView("generator"); }} onOpenReferences={() => setView("references")} />}
{view === "timeline" && <TimelineView config={config} openSceneId={selectedGenerationJobId} folderPath={project.folderPath} generationCompletionTimes={generationCompletionTimes} onChange={changeConfig} onMeasured={recordMeasurement} onOpenGenerator={(jobId) => { setSelectedGenerationJobId(jobId); setView("generator"); }} />}
{view === "generator" && <GeneratorView onGenerate={(submissions) => queue.enqueue(session, submissions)} onCancelGeneration={(ids) => queue.cancelScenes(session, ids)} cancellingJobIds={cancellingJobIds} config={config} folderPath={project.folderPath} generationCompletionTimes={generationCompletionTimes} runtime={runtime?.slopfab ?? null} onRuntimeChange={onGeneratorRuntimeChange} onChange={changeConfig} selectedJobId={selectedGenerationJobId} onSelectedJobChange={setSelectedGenerationJobId} onOpenTimeline={() => setView("timeline")} />}
@@ -303,7 +308,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
<aside ref={agentPaneRef} id={`${panelId}-agent`} className="project-agent-pane" aria-label="Agent" hidden={!agentOpen}>
<AgentDock
mode={mediaMode}
- context={view === "editor" ? "this image composition" : view === "timeline" ? "the edit" : view === "generator" ? "this generation queue" : view === "references" ? "project references" : "this export"}
+ context={view === "music" ? "the music library" : view === "editor" ? "this image composition" : view === "timeline" ? "the edit" : view === "generator" ? "this generation queue" : view === "references" ? "project references" : "this export"}
record={{ ...project, config }}
providers={runtime?.providers ?? CHECKING_PROVIDERS}
onClose={() => { setAgentOpen(false); agentToggle.current?.focus(); }}
diff --git a/src/components/SettingsView.test.tsx b/src/components/SettingsView.test.tsx
index d7d7d5e..8c6b7c5 100644
--- a/src/components/SettingsView.test.tsx
+++ b/src/components/SettingsView.test.tsx
@@ -77,7 +77,7 @@ describe("the settings screen", () => {
saveLoras([{ id: "future-adapter", name: "MOSS adapter", modelType: "moss", path: "D:/moss-adapter.safetensors" }]);
open(null);
const models = screen.getByRole("list", { name: "Generator models" });
- expect(within(models).getAllByRole("button").map((button) => button.getAttribute("aria-label"))).toEqual(["MiniMax H3", "MOSS", "Fibo 2.0", "Other"]);
+ expect(within(models).getAllByRole("button").map((button) => button.getAttribute("aria-label"))).toEqual(["MiniMax H3", "MOSS", "Fibo 2.0", "YuE2", "MiniMax Music3", "Other"]);
expect(screen.queryByRole("button", { name: "Edit Default generator" })).toBeNull();
fireEvent.click(within(models).getByRole("button", { name: "MiniMax H3" }));
expect(screen.getByRole("button", { name: "Edit Default generator" })).toBeTruthy();
diff --git a/src/components/SettingsView.tsx b/src/components/SettingsView.tsx
index b5b6880..388b152 100644
--- a/src/components/SettingsView.tsx
+++ b/src/components/SettingsView.tsx
@@ -8,7 +8,7 @@ import { useAgentPane } from "../lib/useAgentPane";
import { useShortcut } from "../lib/commands";
import { executeGeneratorCommands, type GeneratorCommand } from "../lib/agentGenerators";
import {
- Add20, Agent16, Agent16Filled, Agent20, Back16, Check14, Check16, Check20, ChevronRight20, Delete16, Diagnostics16,
+ Add20, Agent16, Agent16Filled, Agent20, Audio22, Back16, Check14, Check16, Check20, ChevronRight20, Delete16, Diagnostics16,
Diagnostics16Filled, Download16, Download20, Error16, Error20, Flash20, Folder20, Gauge20, Gpu20, Image20, ModelFile20, More16,
Options20, Palette16, Palette16Filled, Palette20, Refresh16, Refresh16Filled, Rename20, Reset16, Sparkle20, Spinner16,
Spinner20, Steps20, TextFile20, Video16, Video16Filled, Video20, Worker16, Worker16Filled,
@@ -579,7 +579,7 @@ export function SettingsView({ onClose, updates, initialTab = "engine", titleBar
<ComboBox aria-label="Generator model" value={selectedTemplate.modelType} options={[
...GENERATOR_MODELS,
...(!GENERATOR_MODELS.some(({ value }) => value === selectedTemplate.modelType) ? [{ value: selectedTemplate.modelType, label: selectedTemplate.modelType }] : []),
- ]} onChange={(modelType) => updateTemplate({ modelType, editingModel: generatorModel(modelType).imageEditing, ...(modelType === MOSS_MODEL_TYPE || modelType === FIBO_MODEL_TYPE ? { mode: "prompt", attention: "flash2", defaultSteps: modelType === FIBO_MODEL_TYPE ? 4 : 50, motionCache: false, loras: [], audioSigmaShift: undefined, ...(modelType === FIBO_MODEL_TYPE ? { videoSigmaShift: undefined } : {}) } : {}) })} />
+ ]} onChange={(modelType) => updateTemplate({ modelType, editingModel: generatorModel(modelType).imageEditing, ...(modelType === MOSS_MODEL_TYPE || modelType === FIBO_MODEL_TYPE || generatorModel(modelType).musicGeneration ? { mode: "prompt", attention: "flash2", defaultSteps: modelType === "yue2" ? 32 : modelType === "minimax_music3" ? 30 : modelType === FIBO_MODEL_TYPE ? 4 : 50, motionCache: false, loras: [], audioSigmaShift: undefined, ...(modelType === FIBO_MODEL_TYPE || generatorModel(modelType).musicGeneration ? { videoSigmaShift: undefined } : {}) } : {}) })} />
</SettingsCard>
{generatorModel(selectedTemplate.modelType).videoGeneration && <SettingsCard icon={<Video20 />} header="Mode">
<ComboBox aria-label="Generator mode" value={selectedTemplate.mode ?? "prompt"} onChange={(value) => updateTemplate({ mode: value === "animate" ? "animate" : "prompt" })}
@@ -615,7 +615,7 @@ export function SettingsView({ onClose, updates, initialTab = "engine", titleBar
<SettingsGroup heading="Performance">
<SettingsCard icon={<Gauge20 />} header="Attention" description={selectedTemplate.attention === "sol" ? "Sol attention requires a CUDA GPU on the machine running generation." : undefined}>
<ComboBox aria-label="Generator attention" value={selectedTemplate.attention} onChange={(value) => updateTemplate({ attention: value as AttentionMode })}
- options={generatorModel(selectedTemplate.modelType).cudaOnly ? [{ value: "flash2", label: "Flash attention" }] : [{ value: "exact", label: "Exact attention" }, { value: "flash2", label: "Flash attention" }, { value: "sage2", label: "Sage attention" }, { value: "sol", label: "Sol attention" }]} />
+ options={generatorModel(selectedTemplate.modelType).cudaOnly || generatorModel(selectedTemplate.modelType).musicGeneration ? [{ value: "flash2", label: "Flash attention" }] : [{ value: "exact", label: "Exact attention" }, { value: "flash2", label: "Flash attention" }, { value: "sage2", label: "Sage attention" }, { value: "sol", label: "Sol attention" }]} />
</SettingsCard>
{generatorModel(selectedTemplate.modelType).motionCache && <SettingsCard icon={<Flash20 />} header="Enable MotionCache" description={<span id="motion-cache-help">{selectedTemplate.mode === "animate" ? "Unavailable in Animate mode" : "Reuses similar denoising results to reduce computation. May affect detail and motion."}</span>}>
<ToggleSwitch aria-label="Enable MotionCache" aria-describedby="motion-cache-help"
@@ -648,7 +648,7 @@ export function SettingsView({ onClose, updates, initialTab = "engine", titleBar
<ReferenceIconSetting templates={templateSettings} />
<div className="settings-card-list" role="list" aria-label="Generator models">
{[...new Set([...GENERATOR_MODELS.map(({ value }) => value), ...templateSettings.templates.map(({ modelType }) => modelType)]), "other"].map((model) =>
- <OpenableCard key={model} icon={model === "other" ? <ModelFile20 /> : !generatorModel(model).videoGeneration ? <Image20 /> : <Video20 />} header={modelPageLabel(model)}
+ <OpenableCard key={model} icon={model === "other" ? <ModelFile20 /> : generatorModel(model).musicGeneration ? <Audio22 /> : !generatorModel(model).videoGeneration ? <Image20 /> : <Video20 />} header={modelPageLabel(model)}
openLabel={modelPageLabel(model)} openRef={`model:${model}`} description={model === "other" ? "Other weights" : "Generators and LoRAs"} onOpen={() => setModelPage(model)} />)}
</div>
</> : modelPage === "other" ? <OtherWeightsSettings desktop={desktop} /> : <>
diff --git a/src/components/WorkQueuePanel.tsx b/src/components/WorkQueuePanel.tsx
index a27074f..32fa816 100644
--- a/src/components/WorkQueuePanel.tsx
+++ b/src/components/WorkQueuePanel.tsx
@@ -14,7 +14,7 @@ import { Flyout, ProgressBar, ProgressRing } from "./ui";
it runs, a 3px bar. The render settings a row was queued with sit in its
tooltip rather than on a third line. */
-const settingsLine = (item: WorkItem) => item.kind === "refmod" ? "Safetensors refmod" : `${item.settings.canvasWidth} × ${item.settings.canvasHeight} · ${item.kind === "reference-icons" ? "JPG icons" : item.kind === "image" ? "JPG image" : `${(item.settings.frames / GENERATION_FRAME_RATE).toFixed(1)}s`} · ${item.settings.steps} steps${item.settings.audioSteps !== undefined ? ` · ${item.settings.audioSteps} audio steps` : ""} · ${item.settings.seed === -1 ? "Random seed" : `Seed ${item.settings.seed}`}`;
+const settingsLine = (item: WorkItem) => item.kind === "music" ? `Music · ${item.settings.steps} steps · Seed ${item.settings.seed}` : item.kind === "refmod" ? "Safetensors refmod" : `${item.settings.canvasWidth} × ${item.settings.canvasHeight} · ${item.kind === "reference-icons" ? "JPG icons" : item.kind === "image" ? "JPG image" : `${(item.settings.frames / GENERATION_FRAME_RATE).toFixed(1)}s`} · ${item.settings.steps} steps${item.settings.audioSteps !== undefined ? ` · ${item.settings.audioSteps} audio steps` : ""} · ${item.settings.seed === -1 ? "Random seed" : `Seed ${item.settings.seed}`}`;
function StateGlyph({ item }: { item: WorkItem }) {
if (item.status === "completed") return <Check16 />;
diff --git a/src/components/workspace/MusicView.test.tsx b/src/components/workspace/MusicView.test.tsx
new file mode 100644
index 0000000..cfa8713
--- /dev/null
+++ b/src/components/workspace/MusicView.test.tsx
@@ -0,0 +1,59 @@
+// @vitest-environment jsdom
+import "@testing-library/jest-dom/vitest";
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { useState } from "react";
+import { MusicView } from "./MusicView";
+import { createProjectConfig, type ProjectConfig } from "../../lib/project";
+import { newSong } from "../../lib/music";
+import { readMediaFileUrl } from "../../lib/persistence";
+import { loadGeneratorTemplateSettings, saveGeneratorTemplateSettings } from "../../lib/settings";
+
+vi.mock("../../lib/persistence", async (original) => ({ ...await original<typeof import("../../lib/persistence")>(), isTauri: () => true, readMediaFileUrl: vi.fn(async () => "blob:song") }));
+let latest: ProjectConfig;
+const generate = vi.fn();
+const createConfig = () => createProjectConfig({ name: "Music", prompt: "", aspectRatio: "16:9", resolution: "416p", targetDurationSeconds: 30 });
+function Harness({ initial = createConfig() }: { initial?: ProjectConfig }) {
+ const [config, setConfig] = useState(initial); latest = config;
+ return <MusicView config={config} folderPath="C:/Music" onChange={setConfig} onGenerate={generate} onCancel={() => {}} workItems={[]} />;
+}
+beforeEach(() => {
+ vi.clearAllMocks(); localStorage.clear();
+ const settings = loadGeneratorTemplateSettings();
+ saveGeneratorTemplateSettings({ ...settings, templates: settings.templates.map((template) => template.id === "yue2" ? { ...template, paths: { ...template.paths, transformer: "C:/weights/yue2.safetensors" }, sources: {} } : template) });
+ vi.spyOn(HTMLMediaElement.prototype, "pause").mockImplementation(() => {});
+ vi.spyOn(HTMLMediaElement.prototype, "play").mockResolvedValue();
+ vi.stubGlobal("URL", Object.assign(URL, { revokeObjectURL: vi.fn() }));
+});
+afterEach(() => { cleanup(); vi.restoreAllMocks(); vi.unstubAllGlobals(); });
+
+it("adds songs, edits lyrics and style, switches layout and queues the selected song", () => {
+ render(<Harness />);
+ fireEvent.click(screen.getAllByRole("button", { name: "New song" })[0]);
+ fireEvent.change(screen.getByLabelText("Song title"), { target: { value: "Night drive" } });
+ fireEvent.change(screen.getByLabelText("Music style"), { target: { value: "Synth pop" } });
+ fireEvent.change(screen.getByLabelText("Music lyrics"), { target: { value: "[Verse]\nCity lights" } });
+ expect(latest.musicSongs![0]).toMatchObject({ title: "Night drive", style: "Synth pop", settings: { lyrics: "[Verse]\nCity lights" } });
+ fireEvent.click(screen.getByRole("button", { name: "List view" }));
+ expect(screen.getByRole("list", { name: "Songs" })).toHaveClass("music-songs--list");
+ fireEvent.click(screen.getByRole("button", { name: "Generate music" }));
+ expect(generate).toHaveBeenCalledWith(latest.musicSongs![0].id, expect.objectContaining({ modelType: "yue2" }));
+});
+
+it("plays generated audio and creates a reference for the context-clicked song", async () => {
+ const first = { ...newSong(), title: "First", status: "completed" as const, assetId: "song-a" };
+ const second = { ...newSong(), title: "Second", status: "completed" as const, assetId: "song-b" };
+ const config = { ...createConfig(), musicSongs: [first, second], assets: [first, second].map((song) => ({ id: song.assetId, kind: "audio" as const, name: song.title, relativePath: `media/${song.assetId}.wav`, durationMs: 30000, mimeType: "audio/wav", createdAt: song.createdAt })) };
+ const { container } = render(<Harness initial={config} />);
+ fireEvent.click(screen.getByRole("button", { name: "Play First" }));
+ await waitFor(() => expect(container.querySelector("audio")).toHaveAttribute("src", "blob:song"));
+ fireEvent.canPlay(container.querySelector("audio")!);
+ expect(HTMLMediaElement.prototype.play).toHaveBeenCalled();
+ fireEvent.play(container.querySelector("audio")!);
+ expect(screen.getByRole("button", { name: "Pause music" })).toBeEnabled();
+ fireEvent.click(screen.getByRole("button", { name: "Select Second" }));
+ expect(readMediaFileUrl).toHaveBeenCalledTimes(1);
+ fireEvent.contextMenu(screen.getByRole("button", { name: "Select First" }));
+ fireEvent.click(screen.getByRole("menuitem", { name: "Create Reference" }));
+ expect(latest.references.at(-1)).toMatchObject({ kind: "audio", name: "First", relativePath: "media/song-a.wav", audio: { startSeconds: 0, durationSeconds: 15 } });
+});
diff --git a/src/components/workspace/MusicView.tsx b/src/components/workspace/MusicView.tsx
new file mode 100644
index 0000000..349b0f4
--- /dev/null
+++ b/src/components/workspace/MusicView.tsx
@@ -0,0 +1,197 @@
+import { useEffect, useRef, useState } from "react";
+import type { ProjectAsset, ProjectConfig } from "../../lib/project";
+import { isTauri, readMediaFileUrl } from "../../lib/persistence";
+import { loadGeneratorTemplateSettings, subscribeGeneratorTemplates, templateSupportsMusic, templateUsable, generatorPathFields, type GeneratorTemplate } from "../../lib/settings";
+import { musicSettingsForModel, newSong, songReference, updateSong, type MusicSettings, type MusicSong } from "../../lib/music";
+import { isWorkActive, type WorkItem } from "../../lib/workQueue";
+import { ComboBox, PropRow, PropSection, Slider, Splitter, useContextMenu, usePaneSize } from "../ui";
+import { Add16, Audio16, GoToEnd16, GoToStart16, GridView16, ListView16, Pause16, Play16, References16, Speaker14 } from "../ui/icons";
+import { generatorOptions } from "../generatorOptions";
+import { SliderNumberInput, GENERATION_STEPS_RANGE } from "./SliderNumberInput";
+import type { ConfigUpdate } from "./TimelineView";
+
+const time = (seconds: number) => `${Math.floor(seconds / 60)}:${Math.floor(seconds % 60).toString().padStart(2, "0")}`;
+
+export function MusicView({ config, folderPath, onChange, onGenerate, onCancel, workItems, openSong, onSongOpened }: {
+ openSong?: { id: string }; onSongOpened?: () => void;
+ config: ProjectConfig; folderPath: string; onChange: (update: ConfigUpdate, key?: string) => void;
+ onGenerate: (songId: string, template: GeneratorTemplate) => void; onCancel: (id: string) => void;
+ workItems: readonly WorkItem[];
+}) {
+ const [templates, setTemplates] = useState(() => loadGeneratorTemplateSettings().templates.filter(templateSupportsMusic));
+ useEffect(() => subscribeGeneratorTemplates(() => setTemplates(loadGeneratorTemplateSettings().templates.filter(templateSupportsMusic))), []);
+ const [selectedId, setSelectedId] = useState(config.musicSongs?.[0]?.id);
+ useEffect(() => { if (openSong) { setSelectedId(openSong.id); onSongOpened?.(); } }, [openSong]);
+ const selected = (config.musicSongs ?? []).find((song) => song.id === selectedId) ?? config.musicSongs?.[0];
+ const [newGeneratorId, setNewGeneratorId] = useState(templates[0]?.id ?? "");
+ const template = templates.find((item) => item.id === (selected?.generatorId ?? newGeneratorId)) ?? templates[0];
+ const [view, setView] = useState<"grid" | "list">(() => localStorage.getItem("slopus.music.view") === "list" ? "list" : "grid");
+ const inspector = usePaneSize("music.inspector", 340, { min: 280, max: 560 });
+ const menu = useContextMenu();
+ const [error, setError] = useState<string | null>(null);
+ const [notice, setNotice] = useState<string | null>(null);
+ const activeWork = selected && workItems.find((item) => item.sceneId === selected.id && isWorkActive(item));
+ const busy = Boolean(activeWork);
+ const ready = template && templateUsable(template) && generatorPathFields(template).every((field) => !field.required || template.paths[field.id].trim() || template.sources?.[field.id]?.length);
+ const edit = (patch: Partial<MusicSong>) => {
+ if (selected && !busy) onChange((current) => updateSong(current, selected.id, patch), `music:${selected.id}`);
+ };
+ const setting = (patch: Partial<MusicSettings>) => { if (selected) edit({ settings: { ...selected.settings, ...patch } }); };
+ const add = () => {
+ const song = newSong(template?.id, Math.min(100, template?.defaultSteps ?? 32));
+ onChange((current) => ({ ...current, musicSongs: [...(current.musicSongs ?? []), song] }));
+ setSelectedId(song.id); setError(null);
+ };
+ const chooseGenerator = (id: string) => {
+ const next = templates.find((item) => item.id === id);
+ setNewGeneratorId(id);
+ if (next && selected) edit({ generatorId: id, steps: Math.min(100, next.defaultSteps), settings: musicSettingsForModel(selected.settings, next.modelType) });
+ };
+ const generate = () => {
+ if (!selected || !template) return;
+ setError(null);
+ try { onGenerate(selected.id, template); } catch (reason) { setError(String(reason)); }
+ };
+
+ const audio = useRef<HTMLAudioElement>(null);
+ const [playerId, setPlayerId] = useState<string | null>(null);
+ const playerAsset = config.assets.find((asset) => asset.id === playerId);
+ const [url, setUrl] = useState<string | null>(null);
+ const [playing, setPlaying] = useState(false);
+ const [position, setPosition] = useState(0);
+ const [length, setLength] = useState(0);
+ const [volume, setVolume] = useState(0.8);
+ const playWhenReady = useRef(false);
+ const [playbackError, setPlaybackError] = useState<string | null>(null);
+ const playable = (config.musicSongs ?? []).flatMap((song) => {
+ const asset = config.assets.find((item) => item.id === song.assetId && item.kind === "audio" && item.relativePath);
+ return asset ? [{ song, asset }] : [];
+ });
+ useEffect(() => {
+ let disposed = false, owned: string | null = null;
+ audio.current?.pause(); setPlaying(false); setPosition(0); setLength(0); setUrl(null); setPlaybackError(null);
+ if (playerAsset) void readMediaFileUrl(folderPath, playerAsset, "audio/wav").then((value) => {
+ if (disposed) { if (value) URL.revokeObjectURL(value); return; }
+ owned = value; setUrl(value);
+ if (!value) { playWhenReady.current = false; setPlaybackError("This song's audio file is unavailable."); }
+ }).catch((reason) => { if (!disposed) { playWhenReady.current = false; setPlaybackError(String(reason)); } });
+ return () => { disposed = true; if (owned) URL.revokeObjectURL(owned); };
+ }, [folderPath, playerAsset?.id, playerAsset?.relativePath]);
+ useEffect(() => { if (audio.current) audio.current.volume = volume; }, [volume, url]);
+ const play = async () => {
+ try { await audio.current?.play(); } catch (reason) { setPlaybackError(`Could not play song: ${String(reason)}`); setPlaying(false); }
+ };
+ const playAsset = (asset: ProjectAsset) => {
+ setPlaybackError(null);
+ if (asset.id === playerId && url) void play();
+ else { playWhenReady.current = true; setPlayerId(asset.id); }
+ };
+ const move = (direction: number) => {
+ if (!playable.length) return;
+ const index = playable.findIndex(({ asset }) => asset.id === playerId);
+ playAsset(playable[(Math.max(0, index) + direction + playable.length) % playable.length].asset);
+ };
+ const toggle = () => {
+ if (playing) { audio.current?.pause(); return; }
+ const asset = playerAsset ?? playable.find(({ song }) => song.id === selected?.id)?.asset ?? playable[0]?.asset;
+ if (asset) playAsset(asset);
+ };
+ const context = (event: React.MouseEvent<HTMLElement> | React.KeyboardEvent<HTMLElement>, song: MusicSong) => {
+ const asset = config.assets.find((item) => item.id === song.assetId);
+ menu.open(event, [{ label: "Create Reference", icon: <References16 />, disabled: !asset?.relativePath || (asset.durationMs ?? 0) < 2000,
+ onSelect: () => {
+ try {
+ const reference = songReference(song, asset!);
+ onChange((current) => ({ ...current, references: [...current.references, reference] }));
+ setNotice(`Added ${song.title} to References.`);
+ } catch (reason) { setError(String(reason)); }
+ } }], { "aria-label": `${song.title} actions` });
+ };
+
+ return <div className="music-view" style={inspector.style}>
+ <div className="music-toolbar" role="toolbar" aria-label="Music tools">
+ <ComboBox aria-label="Music generator" value={template?.id ?? ""} options={generatorOptions(templates)} onChange={chooseGenerator} disabled={busy} />
+ <button type="button" className="secondary-button" onClick={add}><Add16 />New song</button>
+ <div className="music-toolbar__views" role="group" aria-label="Song view">
+ {(["grid", "list"] as const).map((mode) => <button type="button" className="icon-button" key={mode} aria-label={`${mode === "grid" ? "Grid" : "List"} view`} aria-pressed={view === mode}
+ onClick={() => { setView(mode); localStorage.setItem("slopus.music.view", mode); }}>{mode === "grid" ? <GridView16 /> : <ListView16 />}</button>)}
+ </div>
+ </div>
+ <div className="music-body">
+ <div className="music-library">
+ <div className="music-library__heading"><h2>Your music</h2><span>{(config.musicSongs?.length ?? 0)} songs</span></div>
+ {notice && <p role="status">{notice}</p>}
+ {error && <p role="alert">{error}</p>}
+ {!(config.musicSongs?.length ?? 0) ? <div className="music-empty"><Audio16 /><h3>Create your first song</h3><p>Add a song, describe its style and write lyrics, or leave lyrics empty for an instrumental.</p><button type="button" className="primary-button" onClick={add}><Add16 />New song</button></div> :
+ <div className={`music-songs music-songs--${view}`} role="list" aria-label="Songs">
+ {(config.musicSongs ?? []).map((song, index) => {
+ const asset = config.assets.find((item) => item.id === song.assetId);
+ const work = workItems.find((item) => item.sceneId === song.id && isWorkActive(item));
+ return <div className={`music-song${selected?.id === song.id ? " music-song--selected" : ""}`} role="listitem" key={song.id}
+ onContextMenu={(event) => context(event, song)}>
+ <button type="button" className="music-song__select" aria-label={`Select ${song.title}`} aria-pressed={selected?.id === song.id}
+ onClick={() => setSelectedId(song.id)} onDoubleClick={() => { if (asset) playAsset(asset); }}
+ onKeyDown={(event) => { if (event.key === "ContextMenu" || event.shiftKey && event.key === "F10") context(event, song); }}>
+ <span className="music-song__art" aria-hidden="true"><Audio16 /><span>{String(index + 1).padStart(2, "0")}</span></span>
+ <span className="music-song__info"><strong>{song.title}</strong><span>{song.style || "Add a style to get started"}</span></span>
+ <span className="music-song__status">{work ? `${work.detail} · ${Math.round(work.progress * 100)}%` : song.status === "completed" && asset ? time((asset.durationMs ?? 0) / 1000) : song.status}</span>
+ </button>
+ <button type="button" className="icon-button music-song__play" aria-label={`Play ${song.title}`} disabled={!asset?.relativePath} onClick={() => asset && playAsset(asset)}><Play16 /></button>
+ </div>;
+ })}
+ </div>}
+ </div>
+ <Splitter {...inspector.splitterProps} reverse aria-label="Resize music inspector" aria-controls="music-inspector" />
+ <aside id="music-inspector" className="music-inspector" aria-label="Song settings">
+ {selected ? <>
+ <div className="music-inspector__fields">
+ <PropSection title="Song">
+ <PropRow label="Title"><input className="text-field" aria-label="Song title" value={selected.title} maxLength={200} disabled={busy} onChange={(event) => edit({ title: event.target.value || "New song" })} /></PropRow>
+ <label className="music-prompt">Style<textarea className="text-field" aria-label="Music style" value={selected.style} maxLength={10000} rows={3} disabled={busy} placeholder="Genre, mood, instruments and vocals…" onChange={(event) => edit({ style: event.target.value })} /></label>
+ <label className="music-prompt">Lyrics<textarea className="text-field music-lyrics" aria-label="Music lyrics" value={selected.settings.lyrics} maxLength={30000} rows={9} disabled={busy} placeholder="[Verse]… Leave empty for an instrumental." onChange={(event) => setting({ lyrics: event.target.value })} /></label>
+ </PropSection>
+ <PropSection title="Generation" persistKey="music.generation">
+ <PropRow label="Duration"><SliderNumberInput aria-label="Song duration" minimum={2} maximum={300} step={1} suffix="s" value={selected.settings.duration} disabled={busy} onCommit={(duration) => setting({ duration })} /></PropRow>
+ <PropRow label="Steps"><SliderNumberInput aria-label="Music steps" {...GENERATION_STEPS_RANGE} value={selected.steps} disabled={busy} onCommit={(steps) => edit({ steps })} /></PropRow>
+ <PropRow label="Seed" tooltip="Use -1 for a random seed."><input className="text-field" aria-label="Music seed" type="number" min={-1} max={Number.MAX_SAFE_INTEGER} step={1} value={selected.seed} disabled={busy} onChange={(event) => { const seed = event.target.valueAsNumber; if (Number.isSafeInteger(seed) && seed >= -1) edit({ seed }); }} /></PropRow>
+ </PropSection>
+ <PropSection title="Advanced" persistKey="music.advanced" defaultOpen={false}>
+ {template?.modelType === "yue2" && <PropRow label="Planning"><ComboBox aria-label="Music planning" value={selected.settings.planning} disabled={busy} options={[{ value: "full", label: "Full (melody and chords)" }, { value: "melody", label: "Melody" }, { value: "off", label: "Off" }]} onChange={(planning) => setting({ planning: planning as MusicSettings["planning"], ...(planning === "off" ? { abc: "" } : {}) })} /></PropRow>}
+ {([
+ ["top_k", "Top K", 0, 1024, 1], ["semantic_guidance", "Guidance", 0, 5, 1],
+ ...(template?.modelType === "yue2" ? [["temperature", "Temperature", 0, 2, 0.05], ["top_p", "Top P", 0.05, 1, 0.05], ["repetition_penalty", "Repetition penalty", 1, 2, 0.05]] : [["flow_guidance", "Flow guidance", 0, 5, 0.1]]),
+ ] as [keyof MusicSettings, string, number, number, number][]).map(([key, label, minimum, maximum, step]) => <PropRow key={key} label={label} tooltip={key === "top_k" || key.includes("guidance") ? "Zero uses the model default." : undefined}>
+ <SliderNumberInput aria-label={label} minimum={minimum} maximum={maximum} step={step} integer={key === "top_k"} value={Number(selected.settings[key])} disabled={busy} onCommit={(value) => setting({ [key]: key === "semantic_guidance" && value > 0 && value < 1 ? 1 : value })} /></PropRow>)}
+ {template?.modelType === "yue2" && selected.settings.planning !== "off" && <>
+ <PropRow label="Planning tokens"><SliderNumberInput aria-label="Planning tokens" minimum={32} maximum={4096} step={32} integer value={selected.settings.abc_max_tokens} disabled={busy} onCommit={(abc_max_tokens) => setting({ abc_max_tokens })} /></PropRow>
+ <label className="music-prompt">ABC (optional)<textarea className="text-field" aria-label="Music ABC" value={selected.settings.abc} maxLength={30000} rows={3} disabled={busy} onChange={(event) => setting({ abc: event.target.value })} /></label>
+ </>}
+ </PropSection>
+ </div>
+ <div className="music-inspector__actions">
+ {selected.error && <p role="alert">{selected.error}</p>}
+ {!ready && <p>Download or select music model weights in Settings → Generators.</p>}
+ {!isTauri() && <p>Generate music in the desktop app.</p>}
+ {activeWork ? <><progress value={activeWork.progress} max={1} aria-label="Music generation progress" /><button type="button" className="secondary-button inspector-action" disabled={activeWork.cancelling || activeWork.status === "encoding"} onClick={() => onCancel(activeWork.id)}>{activeWork.cancelling ? "Cancelling…" : "Cancel generation"}</button></> :
+ <button type="button" className="primary-button inspector-action" disabled={!ready || !isTauri() || !selected.style.trim()} onClick={generate}>Generate music</button>}
+ </div>
+ </> : <p className="music-inspector__empty">Add a song to edit its lyrics, style and generation settings.</p>}
+ </aside>
+ </div>
+ <div className="music-player" aria-label="Music player">
+ <div className="music-player__now"><Audio16 /><div><strong>{playerAsset?.name ?? "No song playing"}</strong><span>{playing ? "Playing" : playerAsset ? "Paused" : "Choose a generated song"}</span></div></div>
+ <div className="music-player__transport">
+ <div className="music-player__buttons"><button type="button" className="icon-button" aria-label="Previous song" disabled={!playable.length} onClick={() => move(-1)}><GoToStart16 /></button><button type="button" className="music-player__toggle primary-button" aria-label={playing ? "Pause music" : "Play music"} disabled={!playable.length && !playerAsset} onClick={toggle}>{playing ? <Pause16 /> : <Play16 />}</button><button type="button" className="icon-button" aria-label="Next song" disabled={!playable.length} onClick={() => move(1)}><GoToEnd16 /></button></div>
+ <div className="music-player__seek"><span>{time(position)}</span><Slider aria-label="Song playback position" min={0} max={length || 1} step={0.1} value={position} disabled={!url || !length} onChange={(value) => { if (audio.current) audio.current.currentTime = value; setPosition(value); }} /><span>{time(length)}</span></div>
+ </div>
+ <div className="music-player__volume"><Speaker14 /><Slider aria-label="Music volume" min={0} max={1} step={0.01} value={volume} onChange={setVolume} /></div>
+ {playbackError && <p className="music-player__error" role="alert">{playbackError}</p>}
+ <audio ref={audio} src={url ?? undefined} preload="metadata" onPlay={() => setPlaying(true)} onPause={() => setPlaying(false)}
+ onLoadedMetadata={(event) => { const duration = event.currentTarget.duration; setLength(Number.isFinite(duration) ? duration : 0); }}
+ onCanPlay={() => { if (playWhenReady.current) { playWhenReady.current = false; void play(); } }}
+ onTimeUpdate={(event) => setPosition(event.currentTarget.currentTime)} onEnded={() => { setPlaying(false); const index = playable.findIndex(({ asset }) => asset.id === playerId); if (index >= 0 && index + 1 < playable.length) playAsset(playable[index + 1].asset); }}
+ onError={() => { if (url) { playWhenReady.current = false; setPlaying(false); setPlaybackError("This song could not be played."); } }} />
+ </div>
+ {menu.element}
+ </div>;
+}
diff --git a/src/lib/generatorModels.json b/src/lib/generatorModels.json
index 38add48..9c8541c 100644
--- a/src/lib/generatorModels.json
+++ b/src/lib/generatorModels.json
@@ -20,7 +20,8 @@
"cudaOnly": false,
"imageCompositionCreation": false,
"negativePrompt": false,
- "negativePromptDisabledReason": null
+ "negativePromptDisabledReason": null,
+ "musicGeneration": false
},
"moss": {
"label": "MOSS Image",
@@ -45,7 +46,8 @@
"cudaOnly": true,
"imageCompositionCreation": true,
"negativePrompt": true,
- "negativePromptDisabledReason": null
+ "negativePromptDisabledReason": null,
+ "musicGeneration": false
},
"fibo2": {
"label": "Fibo 2.0",
@@ -68,6 +70,55 @@
"cudaOnly": true,
"imageCompositionCreation": true,
"negativePrompt": false,
- "negativePromptDisabledReason": "Fibo 2.0 Turbo uses only the positive prompt. Slopfab does not support negative prompts for this model."
+ "negativePromptDisabledReason": "Fibo 2.0 Turbo uses only the positive prompt. Slopfab does not support negative prompts for this model.",
+ "musicGeneration": false
+ },
+ "yue2": {
+ "label": "YuE2",
+ "description": "Music generation on CUDA or Vulkan",
+ "imageGeneration": false,
+ "videoGeneration": false,
+ "imageEditing": false,
+ "imageReferences": false,
+ "loras": false,
+ "motionCache": false,
+ "generationEffects": false,
+ "cfg": null,
+ "imageOutpainting": false,
+ "characterSheets": false,
+ "requiresImageInput": false,
+ "editImageReferences": false,
+ "maxImageReferences": 0,
+ "refmods": false,
+ "sigmaShifts": false,
+ "cudaOnly": false,
+ "imageCompositionCreation": false,
+ "negativePrompt": false,
+ "negativePromptDisabledReason": null,
+ "musicGeneration": true
+ },
+ "minimax_music3": {
+ "label": "MiniMax Music3",
+ "description": "Music generation on CUDA or Vulkan",
+ "imageGeneration": false,
+ "videoGeneration": false,
+ "imageEditing": false,
+ "imageReferences": false,
+ "loras": false,
+ "motionCache": false,
+ "generationEffects": false,
+ "cfg": null,
+ "imageOutpainting": false,
+ "characterSheets": false,
+ "requiresImageInput": false,
+ "editImageReferences": false,
+ "maxImageReferences": 0,
+ "refmods": false,
+ "sigmaShifts": false,
+ "cudaOnly": false,
+ "imageCompositionCreation": false,
+ "negativePrompt": false,
+ "negativePromptDisabledReason": null,
+ "musicGeneration": true
}
}
diff --git a/src/lib/music.test.ts b/src/lib/music.test.ts
new file mode 100644
index 0000000..db7c79d
--- /dev/null
+++ b/src/lib/music.test.ts
@@ -0,0 +1,22 @@
+import { describe, expect, it } from "vitest";
+import { createProjectConfig, parseProjectConfig, type ProjectAsset } from "./project";
+import { musicSettingsForModel, musicSettingsSchema, newSong, songReference } from "./music";
+
+describe("music projects", () => {
+ it("round-trips songs and preserves older projects without songs", () => {
+ const config = createProjectConfig({ name: "Music", prompt: "", aspectRatio: "16:9", resolution: "416p", targetDurationSeconds: 30 });
+ expect(parseProjectConfig(JSON.parse(JSON.stringify(config))).musicSongs).toBeUndefined();
+ const song = { ...newSong(), style: "Jazz", settings: { ...musicSettingsSchema.parse({}), lyrics: "[Verse]\nHome again" } };
+ expect(parseProjectConfig(JSON.parse(JSON.stringify({ ...config, musicSongs: [song] })))).toMatchObject({ musicSongs: [song] });
+ });
+ it("removes incompatible controls when switching generators", () => {
+ const source = musicSettingsSchema.parse({ lyrics: "Sing", duration: 12, planning: "off", temperature: 0.7, top_p: 0.8, top_k: 80, semantic_guidance: 2, flow_guidance: 1.8 });
+ expect(musicSettingsForModel(source, "minimax_music3")).toMatchObject({ lyrics: "Sing", duration: 12, planning: "full", temperature: 1, top_p: 0.95, top_k: 80, semantic_guidance: 2, flow_guidance: 1.8 });
+ expect(musicSettingsForModel(source, "yue2").flow_guidance).toBe(0);
+ });
+ it("creates an audio reference trimmed to the supported length", () => {
+ const asset: ProjectAsset = { id: "audio", kind: "audio", name: "Song", relativePath: "media/generated/song.wav", durationMs: 30000, mimeType: "audio/wav", createdAt: new Date().toISOString() };
+ expect(songReference(newSong(), asset)).toMatchObject({ kind: "audio", relativePath: asset.relativePath, intendedUse: ["audio"], audio: { startSeconds: 0, durationSeconds: 15 } });
+ expect(() => songReference(newSong(), { ...asset, durationMs: 1000 })).toThrow(/two seconds/);
+ });
+});
diff --git a/src/lib/music.ts b/src/lib/music.ts
new file mode 100644
index 0000000..3a8ee2f
--- /dev/null
+++ b/src/lib/music.ts
@@ -0,0 +1,51 @@
+import { z } from "zod";
+import type { ProjectAsset, ProjectConfig, ProjectReference } from "./project";
+
+export const musicSettingsSchema = z.object({
+ lyrics: z.string().max(30000).default(""),
+ duration: z.number().finite().min(2).max(300).default(30),
+ planning: z.enum(["full", "melody", "off"]).default("full"),
+ abc: z.string().max(30000).default(""),
+ abc_max_tokens: z.number().int().min(32).max(4096).default(4096),
+ temperature: z.number().finite().min(0).max(10).default(1),
+ top_p: z.number().finite().gt(0).max(1).default(0.95),
+ top_k: z.number().int().min(0).max(1024).default(0),
+ repetition_penalty: z.number().finite().min(1).max(10).default(1.2),
+ semantic_guidance: z.number().finite().min(0).max(10).default(0),
+ flow_guidance: z.number().finite().min(0).max(10).default(0),
+}).refine((value) => value.semantic_guidance === 0 || value.semantic_guidance >= 1, "Semantic guidance must be zero (automatic) or at least one.")
+ .refine((value) => value.planning !== "off" || !value.abc, "ABC requires full or melody planning.");
+export const musicSongSchema = z.object({
+ id: z.string().min(1), title: z.string().min(1).max(200),
+ style: z.string().max(10000).default(""),
+ generatorId: z.string().default("yue2"),
+ settings: musicSettingsSchema.default({}),
+ steps: z.number().int().min(2).max(100).default(32),
+ seed: z.number().int().min(-1).max(Number.MAX_SAFE_INTEGER).default(-1),
+ status: z.enum(["draft", "queued", "generating", "completed", "failed", "cancelled"]).default("draft"),
+ assetId: z.string().nullish(), error: z.string().nullish(),
+ createdAt: z.string(),
+});
+export type MusicSong = z.infer<typeof musicSongSchema>;
+export type MusicSettings = z.infer<typeof musicSettingsSchema>;
+
+export function newSong(generatorId = "yue2", steps = 32): MusicSong {
+ return musicSongSchema.parse({ id: `song-${crypto.randomUUID()}`, title: "New song", generatorId, steps, createdAt: new Date().toISOString() });
+}
+
+export function musicSettingsForModel(settings: MusicSettings, modelType: string): MusicSettings {
+ const defaults = musicSettingsSchema.parse({});
+ return modelType === "minimax_music3" ? { ...defaults, lyrics: settings.lyrics, duration: settings.duration,
+ top_k: settings.top_k, semantic_guidance: settings.semantic_guidance, flow_guidance: settings.flow_guidance } : { ...settings, flow_guidance: 0 };
+}
+
+export function songReference(song: MusicSong, asset: ProjectAsset): ProjectReference {
+ if (asset.kind !== "audio" || !asset.relativePath || !asset.durationMs || asset.durationMs < 2000) throw new Error("Generate a song at least two seconds long before creating a reference.");
+ return { id: `ref-${crypto.randomUUID()}`, kind: "audio", name: song.title, description: song.style,
+ relativePath: asset.relativePath, intendedUse: ["audio"], createdAt: new Date().toISOString(),
+ audio: { startSeconds: 0, durationSeconds: Math.min(15, asset.durationMs / 1000) } };
+}
+
+export function updateSong(config: ProjectConfig, id: string, patch: Partial<MusicSong>): ProjectConfig {
+ return { ...config, musicSongs: (config.musicSongs ?? []).map((song) => song.id === id ? { ...song, ...patch } : song) };
+}
diff --git a/src/lib/project.ts b/src/lib/project.ts
index c1d2445..2fc25b8 100644
--- a/src/lib/project.ts
+++ b/src/lib/project.ts
@@ -3,6 +3,7 @@ import { generationEffectsSchema, activeGenerationEffects, type GenerationEffect
import { MIN_SIGMA_SHIFT, MAX_SIGMA_SHIFT } from "./sampling";
import { z } from "zod";
import { backdropDirection, backdropShotDirection, backdropSummary, type BackdropColor } from "./backdrop";
+import { musicSongSchema } from "./music";
import { createImageScene, imageSceneReferenceIds, imageSceneSchema } from "./imageScene";
import { effectEnabledSchema, effectSchemas, isEffectOn } from "./effectSettings";
import { normalizeShotTagSelection, shotTagClauses, SHOT_TAG_ID_PATTERN, type ShotTagClauses, type ShotTagSelection } from "./shot-tags";
@@ -693,6 +694,7 @@ export const projectConfigSchema = z.object({
schemaVersion: z.literal(CURRENT_SCHEMA_VERSION),
generationType: generationTypeSchema.default("video"),
imageScene: imageSceneSchema.nullish(),
+ musicSongs: z.array(musicSongSchema).optional(),
imageSettings: z.object({
resolution: resolutionSchema,
aspectRatio: aspectRatioSchema,
diff --git a/src/lib/runtime.ts b/src/lib/runtime.ts
index ba6c43e..14d7222 100644
--- a/src/lib/runtime.ts
+++ b/src/lib/runtime.ts
@@ -96,6 +96,7 @@ export interface AgentTurnResponse {
}
export interface SlopfabGenerationRequest {
+ music?: import("./music").MusicSettings;
jobId: string;
videoTransition?: "extend" | "bridge";
latentBridge?: import("./project").SceneGenerationInput["latentBridge"];
diff --git a/src/lib/settings.test.ts b/src/lib/settings.test.ts
index 308f46f..7e2bef4 100644
--- a/src/lib/settings.test.ts
+++ b/src/lib/settings.test.ts
@@ -4,7 +4,7 @@ import { beforeEach, describe, expect, it } from "vitest";
import { TURBO_LORA, VIGGLE_ANIMATE_LORA, loadLoras, saveLoras } from "./loras";
import {
agentEndpointProviderSettings,
- selectedImageGeneratorTemplate, createGeneratorTemplate, mossImageTemplate, fiboImageTemplate, fiboBaseTemplate, templateSupportsEditing, templateSupportsOutpainting, defaultGeneratorTemplate, generatorPathFields, templateSupportsVideo,
+ selectedImageGeneratorTemplate, createGeneratorTemplate, musicTemplates, mossImageTemplate, fiboImageTemplate, fiboBaseTemplate, templateSupportsEditing, templateSupportsOutpainting, defaultGeneratorTemplate, generatorPathFields, templateSupportsVideo,
engineProviderSetting,
loadInferenceBackend,
saveInferenceBackend,
@@ -47,7 +47,7 @@ describe("generator templates", () => {
const custom = createGeneratorTemplate("Custom");
saveGeneratorTemplateSettings({ templates: [custom], defaultTemplateId: custom.id, catalogVersion: 13 });
const migrated = loadGeneratorTemplateSettings();
- expect(migrated.templates).toEqual([expect.objectContaining(custom), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(migrated.templates).toEqual([expect.objectContaining(custom), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
expect(migrated.defaultTemplateId).toBe(custom.id);
expect(loadGeneratorTemplateSettings()).toEqual(migrated);
saveGeneratorTemplateSettings({ ...migrated, templates: [custom] });
@@ -75,7 +75,7 @@ describe("generator templates", () => {
const custom = createGeneratorTemplate("Custom");
saveGeneratorTemplateSettings({ templates: [custom], defaultTemplateId: custom.id, catalogVersion: 9 });
const migrated = loadGeneratorTemplateSettings();
- expect(migrated.templates).toEqual([expect.objectContaining(custom), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(migrated.templates).toEqual([expect.objectContaining(custom), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
expect(migrated.defaultTemplateId).toBe(custom.id);
expect(loadGeneratorTemplateSettings()).toEqual(migrated);
saveGeneratorTemplateSettings({ ...migrated, templates: [custom] });
@@ -99,7 +99,7 @@ describe("generator templates", () => {
it("assigns MiniMax H3 to current and legacy generators and preserves future model types", () => {
expect(createGeneratorTemplate().modelType).toBe("minimax-h3");
- expect(loadGeneratorTemplateSettings().templates.filter((template) => ! ["moss-image", "fibo2-edit-turbo", "fibo2-base"].includes(template.id)).every((template) => template.modelType === "minimax-h3")).toBe(true);
+ expect(loadGeneratorTemplateSettings().templates.filter((template) => ! ["moss-image", "fibo2-edit-turbo", "fibo2-base", "yue2", "minimax_music3"].includes(template.id)).every((template) => template.modelType === "minimax-h3")).toBe(true);
for (const modelType of [undefined, null, "", 123]) {
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({
templates: [{ id: "legacy", name: "Renamed generator", modelType }], defaultTemplateId: "legacy", catalogVersion: 9,
@@ -130,7 +130,7 @@ describe("generator templates", () => {
localStorage.setItem("slopus.engine-paths.v1", JSON.stringify({ transformer: "D:\\Models\\main.safetensors" }));
const settings = loadGeneratorTemplateSettings();
expect(settings.defaultTemplateId).toBe("default");
- expect(settings.templates).toHaveLength(9);
+ expect(settings.templates).toHaveLength(11);
expect(settings.templates[1].name).toBe("First/Last Frame");
expect(settings.templates[0]).toMatchObject({
id: "default",
@@ -198,7 +198,7 @@ describe("generator templates", () => {
paths: { ...original.paths, transformer },
sources: { ...original.sources, transformer: [{ ...original.sources!.transformer![0], url: transformer }, { url: "https://huggingface.co/koongrizzly/MiniMax_H3_int4_W4A8_ConvRot_Pruned/resolve/main/diffusion_models/minimax_h3_ref2va_hybrid_b20-49_pruned_w4a8_mixed.safetensors", gpuModel: "", minVramGb: 0 }] },
});
- expect(loadGeneratorTemplateSettings().templates.map(({ name }) => name)).toEqual(["Default", "First/Last Frame", "References", "FastH3 Omni", "Singularity", "Animate", "MOSS Image", "Fibo 2.0 Edit Turbo", "Fibo 2.0 Base"]);
+ expect(loadGeneratorTemplateSettings().templates.map(({ name }) => name)).toEqual(["Default", "First/Last Frame", "References", "FastH3 Omni", "Singularity", "Animate", "MOSS Image", "Fibo 2.0 Edit Turbo", "Fibo 2.0 Base", "YuE2", "MiniMax Music3"]);
});
it.each(["Minimax H3 Original", "My custom generator"])("migrates the saved %s template without changing its configuration", (name) => {
@@ -215,7 +215,7 @@ describe("generator templates", () => {
expect(settings.templates[2]).toEqual(minimaxFastTemplate());
expect(settings.templates[3]).toEqual(minimaxSingularityTemplate());
expect(settings.defaultTemplateId).toBe(template.id);
- expect(settings.catalogVersion).toBe(15);
+ expect(settings.catalogVersion).toBe(16);
expect(loadGeneratorTemplateSettings()).toEqual(settings);
saveGeneratorTemplateSettings({ ...settings, templates: [settings.templates[0]] });
expect(loadGeneratorTemplateSettings().templates).toHaveLength(1);
@@ -225,7 +225,7 @@ describe("generator templates", () => {
const references = minimaxReferencesTemplate();
references.paths.transformer = "D:/Models/custom.safetensors";
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [references], catalogVersion: 3 }));
- expect(loadGeneratorTemplateSettings().templates).toEqual([references, minimaxFastTemplate(), minimaxSingularityTemplate(), viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(loadGeneratorTemplateSettings().templates).toEqual([references, minimaxFastTemplate(), minimaxSingularityTemplate(), viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
});
it("adds Singularity to the previous catalog once while preserving saved templates", () => {
@@ -234,9 +234,9 @@ describe("generator templates", () => {
custom.paths.transformer = "D:/Models/custom.safetensors";
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [custom], defaultTemplateId: custom.id, catalogVersion: 4 }));
const settings = loadGeneratorTemplateSettings();
- expect(settings.templates).toEqual([expect.objectContaining(custom), minimaxSingularityTemplate(), viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(settings.templates).toEqual([expect.objectContaining(custom), minimaxSingularityTemplate(), viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
expect(settings.defaultTemplateId).toBe(custom.id);
- expect(settings.catalogVersion).toBe(15);
+ expect(settings.catalogVersion).toBe(16);
expect(loadGeneratorTemplateSettings()).toEqual(settings);
saveGeneratorTemplateSettings({ ...settings, templates: [custom] });
expect(loadGeneratorTemplateSettings().templates).toEqual([expect.objectContaining(custom)]);
@@ -247,7 +247,7 @@ describe("generator templates", () => {
template.defaultSteps = 8;
template.paths.transformer = "D:/Models/singularity.safetensors";
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], catalogVersion: 4 }));
- expect(loadGeneratorTemplateSettings().templates).toEqual([template, viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(loadGeneratorTemplateSettings().templates).toEqual([template, viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
});
it("adds Turbo to saved Singularity once and preserves other adapters and settings", () => {
@@ -257,7 +257,7 @@ describe("generator templates", () => {
template.loras = [{ loraId: "custom", enabled: false, strength: 0.5 }];
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], defaultTemplateId: template.id, catalogVersion: 5 }));
const settings = loadGeneratorTemplateSettings();
- expect(settings.templates).toEqual([{ ...template, loras: [...template.loras, { loraId: TURBO_LORA.id, enabled: true, strength: 1 }] }, viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(settings.templates).toEqual([{ ...template, loras: [...template.loras, { loraId: TURBO_LORA.id, enabled: true, strength: 1 }] }, viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
expect(settings.defaultTemplateId).toBe("");
expect(loadGeneratorTemplateSettings()).toEqual(settings);
saveGeneratorTemplateSettings({ ...settings, templates: [template] });
@@ -268,7 +268,7 @@ describe("generator templates", () => {
const template = minimaxSingularityTemplate();
template.loras = [{ loraId: TURBO_LORA.id, enabled: false, strength: 0.3 }];
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], catalogVersion: 5 }));
- expect(loadGeneratorTemplateSettings().templates).toEqual([template, viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(loadGeneratorTemplateSettings().templates).toEqual([template, viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
});
it("adds FastH3 Omni with eight steps and its model URLs", () => {
@@ -300,8 +300,8 @@ describe("generator templates", () => {
}
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], catalogVersion: 8 }));
const settings = loadGeneratorTemplateSettings();
- expect(settings.templates).toEqual([minimaxFastTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
- expect(settings.catalogVersion).toBe(15);
+ expect(settings.templates).toEqual([minimaxFastTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
+ expect(settings.catalogVersion).toBe(16);
expect(templateNeedsDownload(settings.templates[0])).toBe(true);
expect(loadGeneratorTemplateSettings()).toEqual(settings);
});
@@ -316,8 +316,8 @@ describe("generator templates", () => {
const otherPaths = { ...template.paths };
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], catalogVersion: 10 }));
const settings = loadGeneratorTemplateSettings();
- expect(settings.catalogVersion).toBe(15);
- expect(settings.templates).toHaveLength(3);
+ expect(settings.catalogVersion).toBe(16);
+ expect(settings.templates).toHaveLength(5);
expect(settings.templates[0]).toMatchObject({ id: template.id, name: "FastH3 Omni", defaultSteps: 8,
paths: { ...otherPaths, transformer: minimaxFastTemplate().paths.transformer },
sources: { transformer: [{ url: minimaxFastTemplate().paths.transformer, gpuModel: "RTX 5090", minVramGb: 24 }] } });
@@ -344,7 +344,7 @@ describe("generator templates", () => {
expect(migrated.paths.transformer).toBe(selection === "url" || selection === "downloaded" ? expectedUrl : previousPath);
expect(migrated.name).toBe(template.name);
expect(migrated.defaultSteps).toBe(12);
- expect(settings.templates.slice(1)).toEqual([references, fast, fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(settings.templates.slice(1)).toEqual([references, fast, fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
expect(loadGeneratorTemplateSettings()).toEqual(settings);
});
@@ -372,7 +372,7 @@ describe("generator templates", () => {
expect(migrated.sources!.transformer).toContainEqual({ url: "https://example.com/custom.safetensors", gpuModel: "Custom GPU", minVramGb: 8 });
expect(migrated.paths.transformer).toBe(selection === "custom" ? "D:/custom.safetensors" : selection === "cached omni" ? "D:/weights/omni.safetensors" : omniUrl);
expect(settings.defaultTemplateId).toBe(selection === "custom" || selection === "cached omni" ? template.id : "");
- expect(settings.templates.slice(1)).toEqual([original, references, fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(settings.templates.slice(1)).toEqual([original, references, fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
expect(loadGeneratorTemplateSettings()).toEqual(settings);
});
@@ -383,11 +383,11 @@ describe("generator templates", () => {
template.paths.transformer = "D:/custom/model.safetensors";
template.sources!.transformer = [{ url: "https://example.com/custom.safetensors", downloadedPath: template.paths.transformer, gpuModel: "", minVramGb: 0 }];
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], catalogVersion: 10 }));
- expect(loadGeneratorTemplateSettings().templates).toEqual([template, fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(loadGeneratorTemplateSettings().templates).toEqual([template, fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
const other = createGeneratorTemplate("Custom");
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [other], defaultTemplateId: other.id, catalogVersion: 10 }));
const settings = loadGeneratorTemplateSettings();
- expect(settings.templates).toEqual([expect.objectContaining(other), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(settings.templates).toEqual([expect.objectContaining(other), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
expect(settings.defaultTemplateId).toBe(other.id);
});
@@ -399,10 +399,10 @@ describe("generator templates", () => {
template.sources = { ...template.sources, videoVae: [{ url: template.paths.videoVae, gpuModel: "", minVramGb: 0 }] };
delete template.sources.transformer;
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], catalogVersion: 8 }));
- expect(loadGeneratorTemplateSettings().templates).toEqual([template, mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(loadGeneratorTemplateSettings().templates).toEqual([template, mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
const original = minimaxOriginalTemplate();
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [original], catalogVersion: 8 }));
- expect(loadGeneratorTemplateSettings().templates).toEqual([original, mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()]);
+ expect(loadGeneratorTemplateSettings().templates).toEqual([original, mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()]);
});
it.each(["textEncoder", "transformer"] as const)("upgrades saved Minimax %s variants once and preserves downloaded weights", (field) => {
@@ -412,7 +412,7 @@ describe("generator templates", () => {
template.paths[field] = downloadedPath;
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], catalogVersion: 1 }));
const settings = loadGeneratorTemplateSettings();
- expect(settings.catalogVersion).toBe(15);
+ expect(settings.catalogVersion).toBe(16);
expect(settings.templates[0].paths[field]).toBe(downloadedPath);
expect(settings.templates[0].sources![field]).toEqual(minimaxOriginalTemplate().sources![field]!.map((source, index) =>
index === 0 ? { ...source, downloadedPath } : source));
@@ -427,7 +427,7 @@ describe("generator templates", () => {
expect(loadGeneratorTemplateSettings().templates[0].sources![field]).toEqual(template.sources![field]);
const custom = createGeneratorTemplate();
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [custom], catalogVersion: 1 }));
- expect(loadGeneratorTemplateSettings().templates.map(({ id }) => id)).toEqual([custom.id, "minimax-h3-references", "minimax-h3-fast", "minimax-h3-singularity", "viggle-animate", "moss-image", "fibo2-edit-turbo", "fibo2-base"]);
+ expect(loadGeneratorTemplateSettings().templates.map(({ id }) => id)).toEqual([custom.id, "minimax-h3-references", "minimax-h3-fast", "minimax-h3-singularity", "viggle-animate", "moss-image", "fibo2-edit-turbo", "fibo2-base", "yue2", "minimax_music3"]);
});
it.each(["url", "downloaded", "original", "custom", "cached original"])("removes the incompatible encoder from saved templates using %s", (selection) => {
@@ -447,7 +447,7 @@ describe("generator templates", () => {
localStorage.setItem("slopus.generator-templates.v1", JSON.stringify({ templates: [template], defaultTemplateId: template.id, catalogVersion: 2 }));
const settings = loadGeneratorTemplateSettings();
const migrated = settings.templates[0];
- expect(settings.catalogVersion).toBe(15);
+ expect(settings.catalogVersion).toBe(16);
expect(migrated.sources!.textEncoder).toEqual([
{ url: originalUrl, gpuModel: "", minVramGb: 0, ...(selection === "cached original" ? { downloadedPath: cachedOriginal } : {}) },
custom,
@@ -528,7 +528,7 @@ it("adds Base once and persists per-template editing through runtime options", (
edit.defaultSteps = 8;
saveGeneratorTemplateSettings({ templates: [edit], defaultTemplateId: "", catalogVersion: 14 });
const migrated = loadGeneratorTemplateSettings();
- expect(migrated.templates).toEqual([edit, fiboBaseTemplate()]);
+ expect(migrated.templates).toEqual([edit, fiboBaseTemplate(), ...musicTemplates()]);
expect(loadGeneratorTemplateSettings()).toEqual(migrated);
const base = migrated.templates[1];
expect(base.name).toBe("Fibo 2.0 Base");
diff --git a/src/lib/settings.ts b/src/lib/settings.ts
index 9644857..8de1f0e 100644
--- a/src/lib/settings.ts
+++ b/src/lib/settings.ts
@@ -75,6 +75,7 @@ function migratedStorageItem(key: string, legacyKey: string): string | null {
export const MINIMAX_H3_MODEL_TYPE = "minimax-h3";
export const MOSS_MODEL_TYPE = "moss";
export const FIBO_MODEL_TYPE = "fibo2";
+export const templateSupportsMusic = (template: Pick<GeneratorTemplate, "modelType">): boolean => generatorModel(template.modelType).musicGeneration;
export const templateSupportsVideo = (template: Pick<GeneratorTemplate, "modelType">): boolean => generatorModel(template.modelType).videoGeneration;
export const templateSupportsImages = (template: Pick<GeneratorTemplate, "modelType" | "mode">): boolean => generatorModel(template.modelType).imageGeneration && template.mode !== "animate";
@@ -119,6 +120,9 @@ export interface WeightSource {
}
export function generatorPathFields(template: Pick<GeneratorTemplate, "mode"> & { modelType?: string }): EnginePathField[] {
+ if (template.modelType === "yue2" || template.modelType === "minimax_music3") return ENGINE_PATH_FIELDS.filter(({ id }) =>
+ id === "transformer" || template.modelType === "minimax_music3" && (id === "textEncoder" || id === "audioVae")).map((field) =>
+ field.id === "transformer" ? { ...field, hint: "The main model that generates music from your style and lyrics." } : field);
if (template.modelType === MOSS_MODEL_TYPE || template.modelType === FIBO_MODEL_TYPE) return ENGINE_PATH_FIELDS.filter(({ id }) => id !== "audioVae").map((field) => {
if (field.id === "transformer") return { ...field, hint: "The main model that turns your description into an image." };
if (field.id === "videoVae") return { ...field, label: "Image VAE weights", hint: "Decodes the model's internal picture into an image." };
@@ -238,12 +242,25 @@ export function createGeneratorTemplate(name = "New template", modelType = MINIM
name,
modelType,
editingModel: generatorModel(modelType).imageEditing,
- defaultSteps: modelType === MOSS_MODEL_TYPE ? 50 : modelType === FIBO_MODEL_TYPE ? 4 : DEFAULT_GENERATION_STEPS,
- attention: generatorModel(modelType).cudaOnly ? "flash2" : "sage2",
+ defaultSteps: modelType === "yue2" ? 32 : modelType === "minimax_music3" ? 30 : modelType === MOSS_MODEL_TYPE ? 50 : modelType === FIBO_MODEL_TYPE ? 4 : DEFAULT_GENERATION_STEPS,
+ attention: generatorModel(modelType).cudaOnly || generatorModel(modelType).musicGeneration ? "flash2" : "sage2",
paths: copyPaths(EMPTY_ENGINE_SETTINGS),
};
}
+export function musicTemplates(): GeneratorTemplate[] {
+ const root = "https://huggingface.co/Comfy-Org/MiniMax-Music-3/resolve/main/";
+ return (["yue2", "minimax_music3"] as const).map((modelType) => {
+ const paths = { ...EMPTY_ENGINE_SETTINGS, ...(modelType === "yue2" ? {
+ transformer: "https://huggingface.co/Comfy-Org/YuE2/resolve/main/checkpoints/yue2_3b_int8_convrot.safetensors",
+ } : { transformer: root + "diffusion_models/minimax_music3_dit_int8_convrot.safetensors",
+ textEncoder: root + "text_encoders/minimax_music3_text_encoder_pruned_int8_convrot.safetensors",
+ audioVae: root + "vae/minimax_music3_dav.safetensors" }) };
+ return { ...createGeneratorTemplate(generatorModel(modelType).label, modelType), id: modelType, paths,
+ sources: Object.fromEntries(Object.entries(paths).filter(([, url]) => url).map(([id, url]) => [id, [{ url, gpuModel: "", minVramGb: 0 }]])) };
+ });
+}
+
export function mossImageTemplate(): GeneratorTemplate {
const root = "https://huggingface.co/Comfy-Org/MOSS-Image/resolve/main/";
const paths: EngineSettings = {
@@ -347,9 +364,9 @@ export function viggleAnimateTemplate(): GeneratorTemplate {
}
const initialTemplateSettings = (paths = EMPTY_ENGINE_SETTINGS): GeneratorTemplateSettings => ({
- templates: [{ id: "default", name: "Default", modelType: MINIMAX_H3_MODEL_TYPE, defaultSteps: DEFAULT_GENERATION_STEPS, attention: "sage2", paths: copyPaths(paths) }, minimaxOriginalTemplate(), minimaxReferencesTemplate(), minimaxFastTemplate(), minimaxSingularityTemplate(), viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate()],
+ templates: [{ id: "default", name: "Default", modelType: MINIMAX_H3_MODEL_TYPE, defaultSteps: DEFAULT_GENERATION_STEPS, attention: "sage2", paths: copyPaths(paths) }, minimaxOriginalTemplate(), minimaxReferencesTemplate(), minimaxFastTemplate(), minimaxSingularityTemplate(), viggleAnimateTemplate(), mossImageTemplate(), fiboImageTemplate(), fiboBaseTemplate(), ...musicTemplates()],
defaultTemplateId: "default",
- catalogVersion: 15,
+ catalogVersion: 16,
});
const normalizeTemplateSettings = (value: unknown): GeneratorTemplateSettings | null => {
@@ -370,7 +387,7 @@ const normalizeTemplateSettings = (value: unknown): GeneratorTemplateSettings |
&& candidate.defaultSteps >= 2
&& candidate.defaultSteps <= MAX_GENERATION_STEPS
? candidate.defaultSteps
- : modelType === FIBO_MODEL_TYPE ? 4 : DEFAULT_GENERATION_STEPS;
+ : modelType === "yue2" ? 32 : modelType === "minimax_music3" ? 30 : modelType === FIBO_MODEL_TYPE ? 4 : DEFAULT_GENERATION_STEPS;
const attention = candidate.attention === "exact" || candidate.attention === "flash2" || candidate.attention === "sol" ? candidate.attention : "sage2";
const paths = pathsFrom(candidate.paths);
const sources: GeneratorTemplate["sources"] = {};
@@ -412,7 +429,7 @@ const normalizeTemplateSettings = (value: unknown): GeneratorTemplateSettings |
templates,
defaultTemplateId: templates.find((template) => template.id === requestedDefault && templateUsable(template))?.id
?? templates.find(templateUsable)?.id ?? "",
- catalogVersion: typeof record.catalogVersion === "number" && [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15].includes(record.catalogVersion) ? record.catalogVersion : 0,
+ catalogVersion: typeof record.catalogVersion === "number" && [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16].includes(record.catalogVersion) ? record.catalogVersion : 0,
};
};
@@ -606,6 +623,11 @@ export function loadGeneratorTemplateSettings(): GeneratorTemplateSettings {
normalized = normalizeTemplateSettings({ ...normalized, catalogVersion: 15 })!;
localStorage.setItem(GENERATOR_TEMPLATES_KEY, JSON.stringify(normalized));
}
+ if ((normalized.catalogVersion ?? 0) < 16) {
+ for (const template of musicTemplates()) if (!normalized.templates.some(({ id }) => id === template.id)) normalized.templates.push(template);
+ normalized = normalizeTemplateSettings({ ...normalized, catalogVersion: 16 })!;
+ localStorage.setItem(GENERATOR_TEMPLATES_KEY, JSON.stringify(normalized));
+ }
return normalized;
}
}
@@ -668,7 +690,7 @@ export function engineProviderSetting(settings: EngineSettings, base?: ProviderS
// A LAN worker receives URLs for weights it should download itself.
const remote = remoteWorkerSelected();
const options: ProviderSetting["options"] = { ...(base?.options ?? {}), attention, inferenceBackend: loadInferenceBackend() };
- const imageOnly = !generatorModel(shifts.modelType).videoGeneration;
+ const pathFields = generatorPathFields({ modelType: shifts.modelType, mode });
options.modelType = shifts.modelType ?? MINIMAX_H3_MODEL_TYPE;
options.editingModel = templateSupportsEditing({ modelType: options.modelType, editingModel: shifts.editingModel, mode });
// Always replace project-carried adapter paths with this machine's selection.
@@ -687,10 +709,10 @@ export function engineProviderSetting(settings: EngineSettings, base?: ProviderS
delete options.workerWeightSources;
delete options.latentUpscaler;
const latentUpscaler = latentUpscalerPath();
- if (latentUpscaler) options.latentUpscaler = latentUpscaler;
+ if (latentUpscaler && !generatorModel(shifts.modelType).musicGeneration) options.latentUpscaler = latentUpscaler;
if (remote && sources) {
options.workerWeightSources = JSON.stringify(Object.fromEntries(Object.entries(sources)
- .filter(([role, entries]) => entries?.length && !(imageOnly && role === "audioVae") && !(mode === "animate" && (role === "textEncoder" || role === "tokenizer")))
+ .filter(([role, entries]) => entries?.length && pathFields.some(({ id }) => id === role))
.map(([role, entries]) => [role, entries!.map(({ url, gpuModel, minVramGb }) => ({ url, gpuModel, minVramGb }))])));
}
if (motionCache && mode !== "animate") options.motionCache = true;
@@ -713,7 +735,7 @@ export function engineProviderSetting(settings: EngineSettings, base?: ProviderS
delete options.motionCache;
}
for (const field of ENGINE_PATH_FIELDS) {
- if ((imageOnly && field.id === "audioVae") || (mode === "animate" && (field.id === "textEncoder" || field.id === "tokenizer"))) {
+ if (!pathFields.some(({ id }) => id === field.id)) {
delete options[field.id];
continue;
}
diff --git a/src/lib/workQueue.test.ts b/src/lib/workQueue.test.ts
index 069ee99..b6a77f7 100644
--- a/src/lib/workQueue.test.ts
+++ b/src/lib/workQueue.test.ts
@@ -20,6 +20,8 @@ import { compileImagePrompt } from "./imagePrompt";
import { addImageNode, createImageEditScene } from "./imageScene";
import { createEmptyImage, imageGenerationSnapshot, makeImagePrimary, restoreGeneratedImage } from "./imageHistory";
import { sceneGenerationRequest } from "./sceneGeneration";
+import { newSong } from "./music";
+import { musicTemplates } from "./settings";
import { bridgeGapFrames, currentLongShotArchives } from "./longShot";
vi.mock("@tauri-apps/api/core", () => ({ invoke: vi.fn() }));
@@ -63,6 +65,42 @@ const finish = async (queue: WorkQueue, id: string) => {
await waitFor(() => expect(queue.getSnapshot().find((item) => item.id === id)?.status).toBe("completed"));
};
+it("queues an immutable music request and saves its audio into timeline media", async () => {
+ const { queue, first, saved } = setup();
+ const song = { ...newSong(), title: "Evening", style: "Soft jazz", seed: 42 };
+ first.update((config) => ({ ...config, musicSongs: [song] }));
+ const template = musicTemplates()[0];
+ template.paths.transformer = "C:/weights/yue2.safetensors";
+ template.sources = {};
+ vi.mocked(invoke).mockResolvedValue({ relativePath: "media/generated/song.wav", durationMs: 30000 });
+ queue.enqueueMusic(first, song.id, template);
+ queue.enqueueMusic(first, song.id, template);
+ first.update((config) => ({ ...config, musicSongs: config.musicSongs!.map((item) => ({ ...item, style: "Changed later" })) }));
+ await waitFor(() => expect(enqueueSlopfabGeneration).toHaveBeenCalledOnce());
+ const request = vi.mocked(enqueueSlopfabGeneration).mock.calls[0][0];
+ expect(request).toMatchObject({ prompt: "Soft jazz", seed: 42, steps: 32, music: { duration: 30 }, referencePaths: [] });
+ expect(request).not.toHaveProperty("saveLatentsPath");
+ const id = queue.getSnapshot()[0].id;
+ await finish(queue, id);
+ expect(invoke).toHaveBeenCalledWith("save_generated_music", { folderPath: first.record.folderPath, jobId: id });
+ expect(first.getSnapshot().config.musicSongs![0]).toMatchObject({ status: "completed", assetId: id, style: "Changed later" });
+ expect(first.getSnapshot().config.assets).toContainEqual(expect.objectContaining({ id, kind: "audio", name: "Evening", relativePath: "media/generated/song.wav", durationMs: 30000 }));
+ expect(saved).toHaveBeenCalled();
+ expect(releaseRendered).toHaveBeenCalledWith(id);
+ expect(saveGeneratedScene).not.toHaveBeenCalled();
+});
+
+it("cancels a queued song without altering another song", async () => {
+ const { queue, first } = setup();
+ const songs = [newSong(), newSong()].map((song) => ({ ...song, style: "Jazz" }));
+ first.update((config) => ({ ...config, musicSongs: songs }));
+ for (const song of songs) queue.enqueueMusic(first, song.id, musicTemplates()[0]);
+ const id = queue.getSnapshot()[1].id;
+ await queue.cancel(id);
+ expect(first.getSnapshot().config.musicSongs![1].status).toBe("cancelled");
+ expect(first.getSnapshot().config.musicSongs![0].status).not.toBe("cancelled");
+});
+
it("releases submitted snapshots after each generation while keeping completed results available", async () => {
const { queue, first } = setup();
const pending = (queue as unknown as { work: Map<string, unknown> }).work;
diff --git a/src/lib/workQueue.ts b/src/lib/workQueue.ts
index 43b9ac5..26ff922 100644
--- a/src/lib/workQueue.ts
+++ b/src/lib/workQueue.ts
@@ -1,4 +1,6 @@
import { completeSceneGeneration, sceneInput } from "./sceneHistory";
+import { musicSongSchema, musicSettingsForModel, updateSong, type MusicSong } from "./music";
+import { templateSupportsMusic } from "./settings";
import { imageCfgScale } from "./generatorModels";
import { activeGenerationEffects } from "./generationEffects";
import { imageSettings } from "./imageSettings";
@@ -46,7 +48,7 @@ function imageGenerationSeed(seed: number): number {
export interface WorkItem {
imageAssetId?: string | null;
imageDraftId?: string;
- kind?: "reference-icons" | "image" | "refmod";
+ kind?: "reference-icons" | "image" | "refmod" | "music";
id: string;
projectKey: string;
folderPath: string;
@@ -64,6 +66,7 @@ export interface WorkItem {
settings: { frames: number; steps: number; audioSteps?: number; videoSigmaShift?: number; audioSigmaShift?: number; latentUpscale?: boolean; seed: number; canvasWidth: number; canvasHeight: number };
}
interface PendingWork {
+ music?: MusicSong;
sceneInput?: SceneInput;
extend?: { sourceId: string; name: string; options: ExtendOptions };
characterSheet?: {
@@ -150,7 +153,8 @@ export class WorkQueue {
if (!session) {
// The queue belongs to this app run. A persisted in-flight status from a
// previous run has no worker behind it and must be restartable.
- const config = isTauri() ? { ...record.config, generationJobs: record.config.generationJobs.map((job) =>
+ const config = isTauri() ? { ...record.config, musicSongs: (record.config.musicSongs ?? []).map((song) =>
+ ["queued", "generating"].includes(song.status) ? { ...song, status: "failed" as const, error: "Generation was interrupted when Slopus closed. Generate again to restart." } : song), generationJobs: record.config.generationJobs.map((job) =>
["queued", "generating", "ready"].includes(job.status) ? { ...job, status: "failed" as const, stage: "failed" as const, error: "Generation was interrupted when Slopus closed. Generate again to restart." } : job) } : record.config;
session = new ProjectSession({ ...record, config }, this.writer);
this.projects.set(key, session);
@@ -275,6 +279,30 @@ export class WorkQueue {
});
}
+ enqueueMusic(session: ProjectSession, songId: string, template: GeneratorTemplate) {
+ if (!isTauri()) throw new Error("Music generation requires the desktop app and model weights.");
+ if (!templateSupportsMusic(template)) throw new Error("Choose a music generator.");
+ const current = session.getSnapshot().config;
+ const song = musicSongSchema.parse((current.musicSongs ?? []).find((item) => item.id === songId));
+ if (!song.style.trim()) throw new Error("Describe the song's style before generating.");
+ const projectKey = projectQueueKey(session.record);
+ if (this.items.some((item) => item.projectKey === projectKey && item.kind === "music" && item.sceneId === songId && isWorkActive(item))) return;
+ const id = `music-${crypto.randomUUID()}`;
+ const captured = { ...structuredClone(song), generatorId: template.id, seed: imageGenerationSeed(song.seed) };
+ const config = structuredClone({ ...current, providerSettings: { ...current.providerSettings,
+ slopfab: engineProviderSetting(template.paths, current.providerSettings.slopfab, "flash2", [], "prompt", [], false, template.sources, { modelType: template.modelType }) } });
+ const request: SlopfabGenerationRequest = { jobId: id, prompt: song.style, steps: song.steps, seed: captured.seed,
+ music: musicSettingsForModel(song.settings, template.modelType), frames: 0, canvasWidth: 0, canvasHeight: 0, referencePaths: [] };
+ let finish!: () => void;
+ const done = new Promise<void>((resolve) => { finish = resolve; });
+ this.work.set(id, { id, music: captured, session, sceneId: songId, request, config, snapshot: JSON.stringify(captured), cancelled: false, submitted: false, done, finish });
+ session.update((value) => updateSong(value, songId, { status: "queued", error: null, generatorId: template.id }));
+ this.items = [...this.items, { id, kind: "music", projectKey, folderPath: session.record.folderPath, projectName: current.name,
+ sceneId: songId, title: song.title, submittedAt: new Date().toISOString(), status: "queued", progress: 0,
+ detail: "Waiting to generate music", error: null, completionAt: null, cancelling: false, needsSave: false,
+ settings: { frames: 0, steps: song.steps, seed: captured.seed, canvasWidth: 0, canvasHeight: 0 } }];
+ this.publish(); void this.pump();
+ }
enqueueImage(session: ProjectSession, template: GeneratorTemplate, count = 1) {
if (!Number.isInteger(count) || count < 1 || count > MAX_IMAGE_BATCH_COUNT) throw new Error(`Count must be a whole number from 1 to ${MAX_IMAGE_BATCH_COUNT}.`);
if (!isTauri()) throw new Error("Image generation requires the desktop app and downloaded model weights.");
@@ -453,6 +481,12 @@ export class WorkQueue {
this.publish(); this.icons.yieldToVideo(); void this.pump();
}
private updateScene(work: PendingWork, patch: Partial<GenerationJob>, dirty = true) {
+ if (work.music) {
+ const id = work.sceneId;
+ const status = patch.status === "ready" ? "generating" : patch.status;
+ work.session.update((current) => updateSong(current, id, { ...(status ? { status } : {}), ...(patch.error !== undefined ? { error: patch.error } : {}) }), dirty);
+ return;
+ }
if (work.image || work.refmod) return;
// The undo replay log retains this callback. Do not retain the submitted
// project snapshot with every progress update.
@@ -523,14 +557,14 @@ export class WorkQueue {
work.snapshot = sceneGenerationSnapshot(captured, work.request);
this.updateScene(work, { generationSnapshot: work.snapshot });
}
- if (!work.image) await purgeTimelineThumbnails(next.folderPath, work.sceneId).catch(() => undefined);
+ if (!work.image && !work.music) await purgeTimelineThumbnails(next.folderPath, work.sceneId).catch(() => undefined);
if (work.cancelled) continue;
await this.prepareReferences(work, next.folderPath);
if (work.cancelled) continue;
const plan = await resolveSlopfabPlan(work.request, work.config, next.folderPath);
if (work.cancelled) continue;
work.outputFrames = plan.alignedFrames;
- this.patch(work.id, { status: "generating", detail: `Generating ${work.image ? "image" : `${plan.alignedFrames} frames`} at ${plan.canvasWidth} × ${plan.canvasHeight}` });
+ this.patch(work.id, { status: "generating", detail: work.music ? "Generating music" : `Generating ${work.image ? "image" : `${plan.alignedFrames} frames`} at ${plan.canvasWidth} × ${plan.canvasHeight}` });
this.updateScene(work, { status: "generating", stage: "preparing" }, false);
await enqueueSlopfabGeneration(work.request, work.config, next.folderPath);
work.submitted = true;
@@ -591,7 +625,7 @@ export class WorkQueue {
try { await cancelSlopfabGeneration(work.request.jobId); }
catch (reason) {
work.cancelled = false;
- this.patch(work.id, { cancelling: false, detail: work.image ? "Generating image" : "Generating video", error: `Could not cancel: ${describeDiagnosticError(reason)}` });
+ this.patch(work.id, { cancelling: false, detail: work.music ? "Generating music" : work.image ? "Generating image" : "Generating video", error: `Could not cancel: ${describeDiagnosticError(reason)}` });
}
}
cancelScenes(session: ProjectSession, sceneIds: string[]) {
@@ -642,7 +676,7 @@ export class WorkQueue {
return;
}
const progress = Math.max(item.progress, event.totalSteps > 0 ? Math.min(0.88, 0.12 + Math.max(0, event.step) / event.totalSteps * 0.76) : event.stage === "delivering" ? 0.88 : 0.08);
- this.patch(work.id, { status: "generating", progress, completionAt: this.timing.update(event), detail: item.cancelling ? "Cancelling generation" : event.stage === "upscaling" ? "Upscaling video latents" : work.request.imageEdit ? `Applying image edits (${Math.min(work.request.imageEdit.edits.length, Math.floor(event.step / Math.max(1, event.totalSteps) * work.request.imageEdit.edits.length) + 1)} of ${work.request.imageEdit.edits.length})` : event.stage === "starting" || event.stage === "transformerLoad" ? "Loading MiniMax H3" : work.image ? "Generating image" : "Generating video" });
+ this.patch(work.id, { status: "generating", progress, completionAt: this.timing.update(event), detail: item.cancelling ? "Cancelling generation" : event.stage === "upscaling" ? "Upscaling video latents" : work.request.imageEdit ? `Applying image edits (${Math.min(work.request.imageEdit.edits.length, Math.floor(event.step / Math.max(1, event.totalSteps) * work.request.imageEdit.edits.length) + 1)} of ${work.request.imageEdit.edits.length})` : work.music ? "Generating music" : event.stage === "starting" || event.stage === "transformerLoad" ? "Loading MiniMax H3" : work.image ? "Generating image" : "Generating video" });
this.updateScene(work, { status: "generating", stage: event.stage === "starting" || event.stage === "transformerLoad" ? "preparing" : "generating", progress }, false);
}
private event(event: JobEvent) {
@@ -727,6 +761,7 @@ export class WorkQueue {
}
}
private async encode(work: PendingWork) {
+ if (work.music) { await this.saveMusic(work); return; }
if (work.image) { await this.saveImage(work); return; }
this.patch(work.id, { status: "encoding", progress: 0.9, detail: "Saving video", completionAt: null });
this.updateScene(work, { status: "ready", stage: "encoding", progress: 0.9, error: null }, false);
@@ -770,9 +805,25 @@ export class WorkQueue {
const item = this.items.find((candidate) => candidate.id === id);
const session = item && this.projects.get(item.projectKey);
if (!item?.needsSave || !session) return;
- try { await session.save(); this.patch(id, { status: "completed", needsSave: false, error: null, detail: item.kind === "image" ? "Image saved" : "Video saved" }); }
+ try { await session.save(); this.patch(id, { status: "completed", needsSave: false, error: null, detail: item.kind === "music" ? "Music saved" : item.kind === "image" ? "Image saved" : "Video saved" }); }
catch (reason) { this.patch(id, { error: describeDiagnosticError(reason) }); }
}
+ private async saveMusic(work: PendingWork) {
+ this.patch(work.id, { status: "encoding", progress: 0.9, detail: "Saving music", completionAt: null });
+ try {
+ const saved = await invoke<{ relativePath: string; durationMs: number }>("save_generated_music", { folderPath: work.session.record.folderPath, jobId: work.id });
+ const song = work.music!, id = work.id;
+ const asset: ProjectAsset = { id, kind: "audio", name: song.title, ...saved, mimeType: "audio/wav", hasAudio: true, createdAt: new Date().toISOString() };
+ work.session.update((current) => ({ ...updateSong(current, song.id, { status: "completed", assetId: id, error: null }), assets: [...current.assets.filter((item) => item.id !== id), asset] }));
+ try {
+ await work.session.save();
+ this.patch(id, { status: "completed", progress: 1, detail: "Music saved" });
+ } catch (reason) {
+ this.patch(id, { status: "failed", progress: 1, needsSave: true, detail: "Music created; project save failed", error: describeDiagnosticError(reason) });
+ }
+ } catch (reason) { this.fail(work, reason); }
+ finally { await releaseRendered(work.id); work.finish(); }
+ }
private async saveImage(work: PendingWork) {
this.patch(work.id, { status: "encoding", progress: 0.9, detail: "Saving image", completionAt: null });
try {
diff --git a/src/styles/index.css b/src/styles/index.css
index ce0f7e1..446a12e 100644
--- a/src/styles/index.css
+++ b/src/styles/index.css
@@ -24,6 +24,7 @@
@import "./workspace.css";
@import "./agent-dock.css";
@import "./generator.css";
+@import "./music.css";
@import "./timeline.css";
@import "./references.css";
@import "./export.css";
diff --git a/src/styles/music.css b/src/styles/music.css
new file mode 100644
index 0000000..d3a9a4b
--- /dev/null
+++ b/src/styles/music.css
@@ -0,0 +1,55 @@
+.music-view { display: grid; grid-template-rows: auto minmax(0, 1fr) auto; height: 100%; min-height: 0; min-width: 0; }
+.music-toolbar { display: flex; align-items: center; gap: var(--space-2); padding: var(--space-2) var(--space-3); border-bottom: 1px solid var(--line); }
+.music-toolbar > .ui-combo { width: 240px; max-width: 50%; }
+.music-toolbar__views { display: flex; gap: var(--space-1); margin-left: auto; }
+.music-toolbar__views [aria-pressed="true"] { background: var(--fill-subtle-hover); color: var(--accent); }
+.music-body { display: grid; grid-template-columns: minmax(180px, 1fr) auto minmax(280px, var(--pane-music-inspector)); min-height: 0; }
+.music-library { min-width: 0; overflow: auto; padding: var(--space-4); }
+.music-library__heading { display: flex; align-items: center; justify-content: space-between; gap: var(--space-2); margin-bottom: var(--space-4); }
+.music-library__heading h2 { margin: 0; font-size: var(--text-lg); }
+.music-library__heading > span, .music-song__info > span, .music-song__status, .music-player__now span { color: var(--text-secondary); font-size: var(--text-xs); }
+.music-songs { display: grid; gap: var(--space-3); }
+.music-songs--grid { grid-template-columns: repeat(auto-fill, minmax(170px, 1fr)); }
+.music-song { position: relative; min-width: 0; border: 1px solid var(--line); border-radius: var(--radius-lg); background: var(--surface-pane); }
+.music-song--selected { border-color: var(--accent); }
+.music-song__select { display: flex; flex-direction: column; gap: var(--space-2); width: 100%; height: 100%; padding: var(--space-3); border: 0; border-radius: inherit; text-align: left; color: var(--text-primary); background: transparent; }
+.music-song__select:hover { background: var(--fill-subtle-hover); }
+.music-song__art { display: flex; align-items: center; justify-content: center; gap: var(--space-3); width: 100%; aspect-ratio: 1; border-radius: var(--radius-sm); background: var(--fill-subtle-hover); color: var(--accent); }
+.music-song__art svg { width: 48px; height: 48px; }
+.music-song__art > span { font-size: var(--text-lg); font-variant-numeric: tabular-nums; }
+.music-song__info { display: flex; flex-direction: column; gap: var(--space-1); min-width: 0; width: 100%; }
+.music-song__info > * { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.music-song__status { padding-right: 28px; text-transform: capitalize; }
+.music-song__play { position: absolute; right: var(--space-2); bottom: var(--space-2); }
+.music-songs--list { gap: var(--space-1); }
+.music-songs--list .music-song__select { flex-direction: row; align-items: center; padding-right: 48px; }
+.music-songs--list .music-song__art { width: 44px; height: 44px; aspect-ratio: auto; flex: none; }
+.music-songs--list .music-song__art svg { width: 20px; height: 20px; }
+.music-songs--list .music-song__art > span { display: none; }
+.music-songs--list .music-song__info { flex: 1; }
+.music-songs--list .music-song__status { padding: 0; max-width: 30%; }
+.music-songs--list .music-song__play { top: 50%; bottom: auto; transform: translateY(-50%); }
+.music-empty { display: flex; flex-direction: column; align-items: center; justify-content: center; min-height: 260px; padding: var(--space-4); text-align: center; }
+.music-empty > svg { width: 48px; height: 48px; color: var(--text-secondary); }
+.music-empty p { max-width: 340px; color: var(--text-secondary); }
+.music-inspector { display: flex; flex-direction: column; min-width: 0; min-height: 0; background: var(--surface-pane); }
+.music-inspector__fields { overflow-y: auto; min-height: 0; flex: 1; }
+.music-prompt { display: flex; flex-direction: column; gap: var(--space-1); color: var(--text-secondary); margin-top: var(--space-2); }
+.music-prompt textarea { width: 100%; resize: vertical; }
+.music-inspector__actions { display: grid; gap: var(--space-2); padding: var(--space-3); border-top: 1px solid var(--line); }
+.music-inspector__actions p, .music-inspector__empty { color: var(--text-secondary); font-size: var(--text-xs); }
+.music-inspector__empty { padding: var(--space-3); }
+.music-inspector__actions progress { width: 100%; }
+.music-player { display: grid; grid-template-columns: minmax(100px, 1fr) minmax(200px, 2fr) minmax(90px, 1fr); align-items: center; gap: var(--space-4); padding: var(--space-3) var(--space-4); border-top: 1px solid var(--line); background: var(--surface-pane); }
+.music-player__now { display: flex; align-items: center; gap: var(--space-3); min-width: 0; }
+.music-player__now > div { display: grid; gap: var(--space-1); min-width: 0; }
+.music-player__now strong { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.music-player__transport { display: grid; gap: var(--space-2); }
+.music-player__buttons { display: flex; justify-content: center; align-items: center; gap: var(--space-3); }
+.music-player__toggle { min-width: 40px; }
+.music-player__seek { display: flex; align-items: center; gap: var(--space-2); font-size: var(--text-xs); font-variant-numeric: tabular-nums; }
+.music-player__seek .ui-slider { flex: 1; min-width: 0; }
+.music-player__volume { display: flex; align-items: center; justify-self: end; gap: var(--space-2); width: min(130px, 100%); }
+.music-player__volume .ui-slider { width: 100%; min-width: 0; }
+.music-player__error { grid-column: 1 / -1; margin: 0; color: var(--text-secondary); }
+.project-content--music { overflow: hidden; }