Skip to content

AI media harness for planning, generating, and editing locally

TypeScript

Support video references in generation

Bittid568eae57c5f
diff --git a/docs/video-references.md b/docs/video-references.md
new file mode 100644
index 0000000..f9cfbd6
--- /dev/null
+++ b/docs/video-references.md
@@ -0,0 +1,32 @@
+# Video references
+
+In **References**, choose **Import video**, or **Add video** on an existing
+reference. MP4, M4V, and MOV containers are accepted; the computer's WebCodecs
+decoder must support the video codec. Video files remain at their original
+locations. The project stores their paths and clip settings.
+
+Set the clip's start and duration, and choose whether to include its sound.
+Each clip must last 2–15 seconds. A generation can use up to three video clips
+totaling 15 seconds, alongside up to nine images. Cite the reference in a shot
+just like an image reference. Use CUDA and a Ref2VA transformer, such as the
+**References** generator. Soundtracks also require audio VAE encoder weights.
+
+The prompt compiler numbers pictures and videos independently, in the order
+the native request receives each kind. Video trim settings and soundtrack
+choices are included in generation snapshots so changing them marks the
+result as needing regeneration.
+
+The work queue reads and demuxes the selected source using mp4box.js, decodes
+the interval with WebCodecs, and uploads frames through binary Tauri IPC.
+Frames are limited to the model's 24 fps cadence and a 768-pixel longest edge.
+Only a small decoder batch is retained in the webview. The current Slopfab
+API accepts host pixels, so this boundary requires readback; it does not
+provide GPU texture interop. Slopfab copies pixels into its reference handle
+and retains immutable snapshots when requests attach them. Optional mono or
+stereo sound is decoded by the webview, trimmed, and sent as interleaved
+float32 PCM. No FFmpeg is used.
+
+Prepared native handles exist only during queued work. Completion, failure,
+and cancellation release them; handles and decoded media are never written
+into project JSON. Old DLLs fail with an explicit update message when their
+video-reference symbols are absent.
diff --git a/src-tauri/src/agent.rs b/src-tauri/src/agent.rs
index 75e92b3..8c26158 100644
--- a/src-tauri/src/agent.rs
+++ b/src-tauri/src/agent.rs
@@ -71,6 +71,7 @@ Example:
When creating or rewriting scenes, plan reusable visual references before writing the shots:
- Inventory every recurring visible character, location, product, important prop, vehicle, creature, or other identity whose look must remain consistent. Reuse a matching project reference when one already exists; otherwise add a top-level text reference before adding the scenes.
- A new ref.add command uses a unique stable id, a clear name, complete text when the user supplied enough detail, and the appropriate use value (such as "character", "animal", "location", or "product"). Slopus creates it as a text reference and supplies its timestamp. Commands cannot invent paths or image entries; only existing project JSON may describe real files.
+- Existing video references can guide movement, camera work, or appearance. Cite their reference IDs with the same @[ref:<reference-id>] tokens; Slopus supplies the <Video N> labels and the saved clip/soundtrack settings. A scene may use up to three video clips totaling 15 seconds. Do not invent video paths or add raw <Video N> labels to shot action.
- A character reference must establish the character's stable identity in enough physical detail to reproduce them: apparent age, build, face, hair, distinguishing features, clothing, footwear, accessories, and the colours/materials of the outfit when relevant. Keep momentary action, pose, expression, and camera direction in the shot instead.
- A location reference establishes persistent architecture, layout, materials, palette, fixtures, and lighting anchors. A product or prop reference establishes persistent shape, proportions, materials, colours, markings, and branding supplied by the user. Do not fabricate brand details.
- Every shot that visibly contains one of these subjects must cite the same reference in its action with the exact token @[ref:<reference-id>]. Slopus derives the scene's referenceIds from these tokens. Reuse the same id across shots and scenes; do not re-describe or rename the subject independently in each shot.
@@ -1865,6 +1866,7 @@ mod tests {
relative_path: None,
source_path: None,
images: Vec::new(),
+ video: None,
intended_use: vec!["character".into()],
subcategory: None,
icon_relative_path: None,
diff --git a/src-tauri/src/agent_commands.rs b/src-tauri/src/agent_commands.rs
index 6fb1ad6..265419c 100644
--- a/src-tauri/src/agent_commands.rs
+++ b/src-tauri/src/agent_commands.rs
@@ -453,6 +453,7 @@ fn apply_command(
relative_path: None,
source_path: None,
images: Vec::new(),
+ video: None,
intended_use: intended_use.clone(),
subcategory: None,
icon_relative_path: None,
diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs
index 82960e6..30b200d 100644
--- a/src-tauri/src/lib.rs
+++ b/src-tauri/src/lib.rs
@@ -210,6 +210,8 @@ struct ReusableReference {
/// readable for projects made when an image was a separate reference kind.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
images: Vec<ReferenceImage>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ video: Option<ReferenceVideoOptions>,
#[serde(default)]
intended_use: Vec<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -230,6 +232,14 @@ struct ReferenceImage {
source_path: Option<String>,
}
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+struct ReferenceVideoOptions {
+ start_seconds: f64,
+ duration_seconds: f64,
+ include_audio: bool,
+}
+
/// One shot inside a scene. Mirrors `sceneShotSchema` in src/lib/project.ts.
///
/// `startSeconds` and `action` are REQUIRED on the zod side and are plain
@@ -810,6 +820,12 @@ fn validate_and_normalize_config(mut config: ProjectConfig) -> Result<ProjectCon
));
}
let mut image_ids = BTreeSet::new();
+ if let Some(video) = &reference.video {
+ if reference.kind != "video" || !video.start_seconds.is_finite() || video.start_seconds < 0.0
+ || !video.duration_seconds.is_finite() || !(2.0..=15.0).contains(&video.duration_seconds) {
+ return Err(format!("Reference '{}' needs a nonnegative clip start and a duration of 2 to 15 seconds.", reference.id));
+ }
+ }
for image in &mut reference.images {
if image.id.trim().is_empty() || image.name.trim().is_empty() {
return Err(format!(
@@ -2134,6 +2150,7 @@ fn create_project_at_with_references(
relative_path: imported.relative_path,
source_path: imported.source_path,
images: Vec::new(),
+ video: None,
intended_use: vec!["style".into()],
subcategory: None,
icon_relative_path: None,
@@ -2311,6 +2328,52 @@ fn cancel_agent_turn(state: tauri::State<'_, agent::AgentRuntime>, request_id: S
state.cancel(&request_id)
}
+#[tauri::command]
+fn choose_reference_video(app: AppHandle, folder_path: String) -> Result<Option<String>, String> {
+ let root = project_root(&folder_path)?;
+ let picked = app.dialog().file().set_title("Add a video reference")
+ .add_filter("MP4 video", &["mp4", "m4v", "mov"]).blocking_pick_file();
+ let Some(picked) = picked else { return Ok(None); };
+ let path = picked.into_path().map_err(|error| format!("Could not access selected video: {error}"))?;
+ if !path.is_file() { return Err("The selected video does not exist.".into()); }
+ picked_external_source_path(&path, &root).map(Some)
+}
+
+#[tauri::command]
+fn create_reference_video(duration_seconds: f64, config: ProjectConfig) -> Result<String, String> {
+ let config = validate_and_normalize_config(config)?;
+ slopfab::create_reference_video(duration_seconds, &config.provider_settings)
+}
+
+fn reference_video_header<'a>(request: &'a tauri::ipc::Request<'_>, name: &str) -> Result<&'a str, String> {
+ request.headers().get(name).and_then(|value| value.to_str().ok())
+ .ok_or_else(|| format!("Missing or invalid {name} header."))
+}
+
+#[tauri::command]
+fn append_reference_video(request: tauri::ipc::Request<'_>) -> Result<(), String> {
+ let tauri::ipc::InvokeBody::Raw(bytes) = request.body() else { return Err("Reference frames require a binary body.".into()); };
+ let id = reference_video_header(&request, "x-reference-id")?;
+ let width = reference_video_header(&request, "x-reference-width")?.parse().map_err(|_| "Invalid frame width.")?;
+ let height = reference_video_header(&request, "x-reference-height")?.parse().map_err(|_| "Invalid frame height.")?;
+ let timestamp = reference_video_header(&request, "x-reference-time")?.parse().map_err(|_| "Invalid frame timestamp.")?;
+ slopfab::append_reference_video(id, bytes, width, height, timestamp)
+}
+
+#[tauri::command]
+fn set_reference_video_audio(request: tauri::ipc::Request<'_>) -> Result<(), String> {
+ let tauri::ipc::InvokeBody::Raw(bytes) = request.body() else { return Err("Reference audio requires a binary body.".into()); };
+ let id = reference_video_header(&request, "x-reference-id")?;
+ let channels = reference_video_header(&request, "x-reference-channels")?.parse().map_err(|_| "Invalid audio channels.")?;
+ let rate = reference_video_header(&request, "x-reference-rate")?.parse().map_err(|_| "Invalid sample rate.")?;
+ slopfab::set_reference_video_audio(id, bytes, channels, rate)
+}
+
+#[tauri::command]
+fn release_reference_videos(ids: Vec<String>) -> Result<(), String> {
+ slopfab::release_reference_videos(&ids)
+}
+
#[tauri::command]
fn resolve_slopfab_plan(
request: slopfab::GenerationRequest,
@@ -2319,7 +2382,7 @@ fn resolve_slopfab_plan(
let context = serde_json::json!({
"jobId": request.job_id, "frames": request.frames, "steps": request.steps,
"canvasWidth": request.canvas_width, "canvasHeight": request.canvas_height,
- "referenceCount": request.reference_paths.len(), "inputChars": request.prompt.chars().count(),
+ "referenceCount": request.reference_count(), "inputChars": request.prompt.chars().count(),
});
diagnostics::info(
"slopfab",
@@ -2858,6 +2921,11 @@ pub fn run() {
choose_initial_reference_images,
choose_reference_image,
choose_reference_images,
+ choose_reference_video,
+ create_reference_video,
+ append_reference_video,
+ set_reference_video_audio,
+ release_reference_videos,
import_media_files,
read_project_file,
read_external_media_file,
@@ -4275,6 +4343,7 @@ mod tests {
relative_path: None,
source_path: None,
images: Vec::new(),
+ video: None,
intended_use: Vec::new(),
subcategory: None,
icon_relative_path: None,
@@ -4286,6 +4355,24 @@ mod tests {
);
}
+ #[test]
+ fn video_reference_clip_settings_survive_save_and_reopen() {
+ let mut config = fixture();
+ let reference = &mut config.references[0];
+ reference.kind = "video".into();
+ reference.relative_path = Some("media/reference.mp4".into());
+ reference.source_path = None;
+ reference.video = Some(ReferenceVideoOptions { start_seconds: 3.5, duration_seconds: 4.0, include_audio: false });
+ let config = validate_and_normalize_config(config).unwrap();
+ let value = serde_json::to_value(&config).unwrap();
+ assert_eq!(value["references"][0]["video"]["startSeconds"], 3.5);
+ let reopened: ProjectConfig = serde_json::from_value(value).unwrap();
+ assert_eq!(reopened.references[0].video, config.references[0].video);
+ let mut invalid = reopened;
+ invalid.references[0].video.as_mut().unwrap().duration_seconds = 16.0;
+ assert!(validate_and_normalize_config(invalid).is_err());
+ }
+
#[test]
fn scene_frame_settings_survive_save_and_reopen_and_validate_images() {
let mut config = fixture();
diff --git a/src-tauri/src/slopfab.rs b/src-tauri/src/slopfab.rs
index d082d5a..edf0cc1 100644
--- a/src-tauri/src/slopfab.rs
+++ b/src-tauri/src/slopfab.rs
@@ -125,6 +125,55 @@ pub struct GenerationRequest {
pub canvas_height: i32,
#[serde(default)]
pub reference_paths: Vec<String>,
+ #[serde(default)]
+ pub reference_video_ids: Vec<String>,
+}
+
+impl GenerationRequest {
+ pub fn reference_count(&self) -> usize {
+ self.reference_paths.len() + self.reference_video_ids.len()
+ }
+}
+
+// A video owns its DLL and input buffers. Commands serialize mutations; the
+// C API retains immutable snapshots when a plan/generation attaches a video.
+static REFERENCE_VIDEOS: std::sync::LazyLock<Mutex<HashMap<String, ffi::ReferenceVideoHandle>>> =
+ std::sync::LazyLock::new(Default::default);
+static NEXT_REFERENCE_VIDEO: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1);
+
+pub fn create_reference_video(duration: f64, settings: &BTreeMap<String, ProviderSetting>) -> Result<String, String> {
+ let configuration = Configuration::from_settings(settings);
+ let api = ffi::Api::load(&configuration.dll_path)?;
+ api.version()?;
+ if configuration.platform(detect_platform(&api)) == ComputePlatform::Vulkan {
+ return Err("Video references require CUDA. Select CUDA in Generator settings and use a Ref2VA transformer.".into());
+ }
+ let video = ffi::ReferenceVideoHandle::new(api, duration, configuration.dll_path)?;
+ let mut videos = REFERENCE_VIDEOS.lock().map_err(|_| "Reference video lock failed.")?;
+ if videos.len() >= 3 { return Err("At most three video references can be prepared at once.".into()); }
+ let id = NEXT_REFERENCE_VIDEO.fetch_add(1, Ordering::Relaxed).to_string();
+ videos.insert(id.clone(), video);
+ Ok(id)
+}
+
+pub fn append_reference_video(id: &str, bytes: &[u8], width: i32, height: i32, timestamp: f64) -> Result<(), String> {
+ REFERENCE_VIDEOS.lock().map_err(|_| "Reference video lock failed.")?
+ .get_mut(id).ok_or("The prepared reference video no longer exists.")?
+ .append(bytes, width, height, timestamp)
+}
+
+pub fn set_reference_video_audio(id: &str, bytes: &[u8], channels: i32, sample_rate: i32) -> Result<(), String> {
+ if bytes.len() % 4 != 0 { return Err("Reference audio must contain complete float32 samples.".into()); }
+ let samples: Vec<f32> = bytes.chunks_exact(4).map(|chunk| f32::from_le_bytes(chunk.try_into().unwrap())).collect();
+ REFERENCE_VIDEOS.lock().map_err(|_| "Reference video lock failed.")?
+ .get_mut(id).ok_or("The prepared reference video no longer exists.")?
+ .set_audio(&samples, channels, sample_rate)
+}
+
+pub fn release_reference_videos(ids: &[String]) -> Result<(), String> {
+ let mut videos = REFERENCE_VIDEOS.lock().map_err(|_| "Reference video lock failed.")?;
+ for id in ids { videos.remove(id); }
+ Ok(())
}
#[derive(Debug, Clone, Serialize)]
@@ -216,7 +265,7 @@ impl Default for SlopfabRuntime {
"steps": item.request.steps,
"canvasWidth": item.request.canvas_width,
"canvasHeight": item.request.canvas_height,
- "referenceCount": item.request.reference_paths.len(),
+ "referenceCount": item.request.reference_count(),
"randomSeed": item.request.seed == -1,
}));
let result = run_generation(&item);
@@ -311,7 +360,7 @@ impl SlopfabRuntime {
"steps": request.steps,
"canvasWidth": request.canvas_width,
"canvasHeight": request.canvas_height,
- "referenceCount": request.reference_paths.len(),
+ "referenceCount": request.reference_count(),
"inputChars": request.prompt.chars().count(),
}),
);
@@ -567,6 +616,12 @@ pub fn resolve_plan(
}
fn validate_generation_controls(request: &GenerationRequest) -> Result<(), String> {
+ if request.reference_paths.len() > 9 || request.reference_video_ids.len() > 3 || request.reference_count() > 12 {
+ return Err("Use at most nine images and three videos per generation.".into());
+ }
+ if request.still_image && !request.reference_video_ids.is_empty() {
+ return Err("Video references cannot be used for still-image generation.".into());
+ }
if request.steps < 2 {
return Err("Generation step count must be at least 2.".into());
}
@@ -619,8 +674,18 @@ fn configure_request(
api.set_model(handle, *id, path)?;
}
}
- for path in &request.reference_paths {
- api.add_reference(handle, Path::new(path))?;
+ }
+ for path in &request.reference_paths {
+ api.add_reference(handle, Path::new(path))?;
+ }
+ if !request.reference_video_ids.is_empty() {
+ if platform == ComputePlatform::Vulkan {
+ return Err("Video references require CUDA and a Ref2VA transformer.".into());
+ }
+ let videos = REFERENCE_VIDEOS.lock().map_err(|_| "Reference video lock failed.")?;
+ for id in &request.reference_video_ids {
+ videos.get(id).ok_or("The prepared reference video no longer exists. Please retry generation.")?
+ .attach(handle, &configuration.dll_path)?;
}
}
Ok(())
@@ -687,7 +752,7 @@ fn run_generation(item: &QueueItem) -> Result<(OutputMetadata, rendered::Rendere
frames: item.request.frames,
canvas_width: item.request.canvas_width,
canvas_height: item.request.canvas_height,
- reference_count: item.request.reference_paths.len(),
+ reference_count: item.request.reference_count(),
timing_profile: timing_profile.clone(),
planned_steps: (item.request.steps - 1).max(0),
});
@@ -760,7 +825,7 @@ fn run_generation(item: &QueueItem) -> Result<(OutputMetadata, rendered::Rendere
frames: output.frames, width: output.width, height: output.height, fps: output.fps,
audio_channels: output.audio_channels, audio_sample_rate: output.audio_sample_rate,
audio_samples: output.audio_float_count,
- reference_count: item.request.reference_paths.len(),
+ reference_count: item.request.reference_count(),
timing_profile,
seconds_conditioning: output.seconds_conditioning,
seconds_denoise: output.seconds_denoise,
@@ -851,6 +916,7 @@ fn write_reference_icon(
canvas_width: crate::reference_icons::RENDER_SIZE as i32,
canvas_height: crate::reference_icons::RENDER_SIZE as i32,
reference_paths: Vec::new(),
+ reference_video_ids: Vec::new(),
};
let handle = RequestHandle::new(api)?;
configure_request(api, handle.0, &request, configuration, platform, true)?;
@@ -1012,6 +1078,62 @@ mod ffi {
};
pub enum Request {}
+ enum ReferenceVideo {}
+ type VideoCreate = unsafe extern "C" fn(f64, *mut *mut ReferenceVideo) -> i32;
+ type VideoDestroy = unsafe extern "C" fn(*mut ReferenceVideo);
+ type VideoAppend = unsafe extern "C" fn(*mut ReferenceVideo, *const u8, usize, i32, i32, usize, f64) -> i32;
+ type VideoAudio = unsafe extern "C" fn(*mut ReferenceVideo, *const f32, usize, i32, i32, f64) -> i32;
+ type VideoAttach = unsafe extern "C" fn(*mut Request, *const ReferenceVideo) -> i32;
+
+ pub struct ReferenceVideoHandle {
+ api: Api,
+ handle: *mut ReferenceVideo,
+ destroy: VideoDestroy,
+ append_rgba: VideoAppend,
+ audio: VideoAudio,
+ attach_video: VideoAttach,
+ dll_path: std::path::PathBuf,
+ frames: usize,
+ }
+
+ // The registry mutex serializes access. The DLL handle and all copied input
+ // buffers outlive every call, and attached snapshots own their references.
+ unsafe impl Send for ReferenceVideoHandle {}
+
+ impl ReferenceVideoHandle {
+ pub fn new(api: Api, duration: f64, dll_path: std::path::PathBuf) -> Result<Self, String> {
+ unsafe {
+ let missing = |error| format!("This slopfab.dll does not support video references. Update the runtime: {error}");
+ let create = *api._library.get::<VideoCreate>(b"slopfab_reference_video_create\0").map_err(missing)?;
+ let destroy = *api._library.get::<VideoDestroy>(b"slopfab_reference_video_destroy\0").map_err(missing)?;
+ let append_rgba = *api._library.get::<VideoAppend>(b"slopfab_reference_video_append_rgba8\0").map_err(missing)?;
+ let audio = *api._library.get::<VideoAudio>(b"slopfab_reference_video_set_audio_f32\0").map_err(missing)?;
+ let attach_video = *api._library.get::<VideoAttach>(b"slopfab_request_add_reference_video\0").map_err(missing)?;
+ let mut handle = std::ptr::null_mut();
+ api.error(create(duration, &mut handle))?;
+ if handle.is_null() { return Err("slopfab returned an empty reference video handle.".into()); }
+ Ok(Self { api, handle, destroy, append_rgba, audio, attach_video, dll_path, frames: 0 })
+ }
+ }
+ pub fn append(&mut self, bytes: &[u8], width: i32, height: i32, timestamp: f64) -> Result<(), String> {
+ if width <= 0 || height <= 0 || width > 1024 || height > 1024 || self.frames >= 361 {
+ return Err("Reference video preparation exceeds the frame or dimension limit.".into());
+ }
+ self.api.error(unsafe { (self.append_rgba)(self.handle, bytes.as_ptr(), bytes.len(), width, height, width as usize * 4, timestamp) })?;
+ self.frames += 1;
+ Ok(())
+ }
+ pub fn set_audio(&mut self, samples: &[f32], channels: i32, sample_rate: i32) -> Result<(), String> {
+ self.api.error(unsafe { (self.audio)(self.handle, samples.as_ptr(), samples.len(), channels, sample_rate, 0.0) })
+ }
+ pub fn attach(&self, request: *mut Request, dll_path: &Path) -> Result<(), String> {
+ if dll_path != self.dll_path { return Err("Reference video and generation must use the same slopfab runtime.".into()); }
+ self.api.error(unsafe { (self.attach_video)(request, self.handle) })
+ }
+ }
+ impl Drop for ReferenceVideoHandle {
+ fn drop(&mut self) { unsafe { (self.destroy)(self.handle) }; }
+ }
pub fn has_cuda_device() -> bool {
// CUDA's driver API uses the system calling convention on Windows.
@@ -1510,6 +1632,30 @@ mod ffi {
#[cfg(test)]
mod tests {
use super::*;
+ #[test]
+ fn video_reference_c_api_copies_inputs_and_retains_attached_snapshot() {
+ let path = default_dll_path();
+ let api = ffi::Api::load(&path).unwrap();
+ let request = RequestHandle::new(&api).unwrap();
+ api.set_prompt(request.0, "A scene using <Video 1>.").unwrap();
+ api.set_frames(request.0, 120).unwrap();
+ api.set_resolution(request.0, 736, 416).unwrap();
+ let mut video = ffi::ReferenceVideoHandle::new(ffi::Api::load(&path).unwrap(), 2.0, path.clone()).unwrap();
+ let pixels = vec![127_u8; 64 * 64 * 4];
+ video.append(&pixels, 64, 64, 0.0).unwrap();
+ assert!(video.append(&pixels, 64, 64, 0.0).is_err());
+ assert!(video.append(&pixels[..10], 64, 64, 0.5).is_err());
+ video.append(&pixels, 64, 64, 1.0).unwrap();
+ video.set_audio(&vec![0.0; 32_000 * 2], 1, 32_000).unwrap();
+ video.attach(request.0, &path).unwrap();
+ drop(video);
+ drop(pixels);
+ // Plan resolution reads the retained video/audio snapshot without a
+ // neural model or a CUDA allocation, even after the host releases it.
+ let plan = api.resolve(request.0).unwrap();
+ assert!(plan.aligned_frames > 0);
+ }
+
#[test]
fn absent_dll_is_an_explicit_disabled_state() {
let mut settings = BTreeMap::new();
diff --git a/src/components/workspace/GeneratorView.test.tsx b/src/components/workspace/GeneratorView.test.tsx
index f0cd3e4..3655e9d 100644
--- a/src/components/workspace/GeneratorView.test.tsx
+++ b/src/components/workspace/GeneratorView.test.tsx
@@ -160,6 +160,21 @@ describe("Generator scene controls", () => {
expect(screen.queryByRole("dialog", { name: "Debug Prompt" })).not.toBeInTheDocument();
});
+ it("sends a cited video with its trim and soundtrack choice in the generation snapshot", async () => {
+ const initial = project();
+ initial.references = [{ id: "motion", kind: "video", name: "Motion", description: "", sourcePath: "C:/motion.mp4", intendedUse: [], createdAt: initial.createdAt,
+ video: { startSeconds: 2, durationSeconds: 3, includeAudio: false } }];
+ initial.generationJobs[0].referenceIds = ["motion"];
+ initial.generationJobs[0].shots![0].action = "Follow @[ref:motion].";
+ const state = setup(parseProjectConfig(initial));
+ fireEvent.click(within(screen.getByRole("region", { name: "First scene" })).getByRole("button", { name: "Generate" }));
+ await waitFor(() => expect(state.latest().generationJobs[0].generationSnapshot).toEqual(expect.any(String)));
+ const snapshot = JSON.parse(state.latest().generationJobs[0].generationSnapshot!);
+ expect(snapshot.referenceVideos).toEqual([{ name: "Motion", sourcePath: "C:/motion.mp4", startSeconds: 2, durationSeconds: 3, includeAudio: false }]);
+ expect(snapshot.referencePaths).toEqual([]);
+ expect(snapshot.prompt).toContain("<Video 1>");
+ });
+
it("offers no inferred Look and sends a selected image as the scene's first frame", async () => {
const initial = project();
initial.references = [{
diff --git a/src/components/workspace/GeneratorView.tsx b/src/components/workspace/GeneratorView.tsx
index e5f92f0..7136ef6 100644
--- a/src/components/workspace/GeneratorView.tsx
+++ b/src/components/workspace/GeneratorView.tsx
@@ -1,4 +1,5 @@
import type { GenerationSubmission } from "../../lib/workQueue";
+import { usableVideoReferences } from "../../lib/project";
import { ChevronDown, Plus, Square, Trash2, WandSparkles } from "lucide-react";
import { invoke } from "@tauri-apps/api/core";
import { useEffect, useMemo, useRef, useState, useSyncExternalStore } from "react";
@@ -306,6 +307,12 @@ export function GeneratorView({ config, folderPath, runtime = null, generationCo
referencePaths: usableReferenceImages(bound)
.map((image) => projectItemPath(folderPath, image) ?? "")
.filter((path) => path.length > 0),
+ referenceVideos: usableVideoReferences(bound).map((reference) => ({
+ name: reference.name, relativePath: reference.relativePath, sourcePath: reference.sourcePath,
+ startSeconds: reference.video?.startSeconds ?? 0,
+ durationSeconds: reference.video?.durationSeconds ?? 2,
+ includeAudio: reference.video?.includeAudio ?? true,
+ })),
};
};
diff --git a/src/components/workspace/ReferenceVideo.tsx b/src/components/workspace/ReferenceVideo.tsx
new file mode 100644
index 0000000..0e64d8c
--- /dev/null
+++ b/src/components/workspace/ReferenceVideo.tsx
@@ -0,0 +1,25 @@
+import { useEffect, useState } from "react";
+import { readMediaFileUrl } from "../../lib/persistence";
+import type { ProjectReference } from "../../lib/project";
+
+export function ReferenceVideo({ folderPath, reference }: { folderPath: string; reference: ProjectReference }) {
+ const [url, setUrl] = useState<string | null>(null);
+ const [error, setError] = useState<string | null>(null);
+ useEffect(() => {
+ let disposed = false;
+ let owned: string | null = null;
+ setUrl(null); setError(null);
+ void readMediaFileUrl(folderPath, reference, "video/mp4").then((value) => {
+ if (disposed) { if (value) URL.revokeObjectURL(value); return; }
+ owned = value; setUrl(value);
+ }).catch((reason: unknown) => { if (!disposed) setError(String(reason)); });
+ return () => { disposed = true; if (owned) URL.revokeObjectURL(owned); };
+ }, [folderPath, reference.sourcePath, reference.relativePath]);
+ if (error) return <p role="alert">{error}</p>;
+ if (!url) return <p>Loading video preview…</p>;
+ return <video className="reference-video-preview" key={`${url}:${reference.video?.startSeconds ?? 0}`}
+ src={url} controls preload="metadata" aria-label={`${reference.name} video preview`}
+ muted={reference.video?.includeAudio === false}
+ onLoadedMetadata={(event) => { event.currentTarget.currentTime = reference.video?.startSeconds ?? 0; }}
+ onError={() => setError("This computer cannot preview this video codec.")} />;
+}
diff --git a/src/components/workspace/ReferencesView.tsx b/src/components/workspace/ReferencesView.tsx
index 0b37c74..4000fac 100644
--- a/src/components/workspace/ReferencesView.tsx
+++ b/src/components/workspace/ReferencesView.tsx
@@ -1,6 +1,6 @@
-import { BookOpen, ChevronRight, FileText, ImagePlus, Plus, RefreshCw, Search, Trash2, Users, X } from "lucide-react";
+import { BookOpen, ChevronRight, FileText, ImagePlus, Plus, RefreshCw, Search, Trash2, Users, Video, X } from "lucide-react";
import { invoke } from "@tauri-apps/api/core";
-import { useEffect, useMemo, useState, useSyncExternalStore } from "react";
+import { useEffect, useMemo, useRef, useState, useSyncExternalStore } from "react";
import { isReferenceDescribed, referenceImages, type ProjectConfig, type ProjectReference, type ProjectReferenceImage } from "../../lib/project";
import {
composeLocationPrompt,
@@ -17,6 +17,8 @@ import {
import { LOCATION_SETTING_GROUPS } from "../../lib/expanded-reference-options";
import { isTauri } from "../../lib/persistence";
import { ReferenceImage } from "./ReferenceImage";
+import { ReferenceVideo } from "./ReferenceVideo";
+import { inspectReferenceVideo } from "../../lib/referenceVideo";
import { DebugPromptDialog } from "./DebugPromptDialog";
import { referenceIconPrompt } from "../../lib/referenceIcons";
import { loadDebugOptionsEnabled, subscribeDebugOptions } from "../../lib/settings";
@@ -26,10 +28,25 @@ import { loadDebugOptionsEnabled, subscribeDebugOptions } from "../../lib/settin
* reference, and neither the card nor the inspector does anything with it. */
const referenceKindLabel = (reference: ProjectReference) => {
const count = referenceImages(reference).length;
- if (count === 0) return reference.kind === "video" ? "Video clip" : reference.kind === "audio" ? "Sound" : "Text definition";
+ if (reference.kind === "video") return `Video clip${count ? ` + ${count} image${count === 1 ? "" : "s"}` : ""}`;
+ if (count === 0) return reference.kind === "audio" ? "Sound" : "Text definition";
return `${count} image${count === 1 ? "" : "s"}${isReferenceDescribed(reference) ? " + text" : ""}`;
};
+function ClipSecondsInput({ label, value, min, max = Number.MAX_SAFE_INTEGER, onChange }: {
+ label: string; value: number; min: number; max?: number; onChange: (value: number) => void;
+}) {
+ const [draft, setDraft] = useState(String(value));
+ useEffect(() => setDraft(String(value)), [value]);
+ return <label>{label}<input type="number" min={min} max={max} step={0.1} value={draft}
+ onChange={(event) => setDraft(event.target.value)} onKeyDown={(event) => { if (event.key === "Enter") event.currentTarget.blur(); }}
+ onBlur={() => {
+ const parsed = Number(draft);
+ const next = draft.trim() && Number.isFinite(parsed) ? Math.max(min, Math.min(max, parsed)) : value;
+ setDraft(String(next)); onChange(next);
+ }} /></label>;
+}
+
export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon, pendingIconIds = new Set<string>(), onOpenGenerator }: {
config: ProjectConfig;
folderPath: string;
@@ -39,10 +56,13 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
onOpenGenerator?: (jobId: string) => void;
}) {
const [selectedId, setSelectedId] = useState<string | undefined>(config.references[0]?.id);
+ const configRef = useRef(config);
+ configRef.current = config;
const debugEnabled = useSyncExternalStore(subscribeDebugOptions, loadDebugOptionsEnabled);
const [showDebugIconPrompt, setShowDebugIconPrompt] = useState(false);
useEffect(() => setShowDebugIconPrompt(false), [selectedId, debugEnabled]);
const [importError, setImportError] = useState<string | null>(null);
+ const [importingVideo, setImportingVideo] = useState(false);
const [iconError, setIconError] = useState<string | null>(null);
const [presetDialog, setPresetDialog] = useState(false);
const [pickerType, setPickerType] = useState<PresetReferenceType>("character");
@@ -58,7 +78,10 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
: undefined, [selected]);
const selectedPresetIcon = selectedImages.length === 0 ? selectedPreset?.icon : undefined;
const jobs = useMemo(() => config.generationJobs.filter((job) => job.referenceIds.includes(selectedId ?? "")), [config.generationJobs, selectedId]);
- const update = (id: string, patch: Partial<ProjectReference>) => onChange({ ...config, references: config.references.map((ref) => ref.id === id ? { ...ref, ...patch } : ref) });
+ const update = (id: string, patch: Partial<ProjectReference>) => {
+ const current = configRef.current;
+ onChange({ ...current, references: current.references.map((ref) => ref.id === id ? { ...ref, ...patch } : ref) });
+ };
// Starts empty on purpose. Seeding it with the instruction text meant an
// unedited definition shipped "Describe the traits…" to the model as if the
// user had written it. The guidance lives in the field's placeholder instead.
@@ -106,6 +129,29 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
})) });
setSelectedId(config.references.find((ref) => ref.id !== selected.id)?.id);
};
+ const addVideo = async (attach = false) => {
+ setImportError(null);
+ if (!isTauri()) { setImportError("Video import is available in the desktop app."); return; }
+ setImportingVideo(true);
+ try {
+ const sourcePath = await invoke<string | null>("choose_reference_video", { folderPath });
+ if (!sourcePath) return;
+ const options = await inspectReferenceVideo(folderPath, sourcePath);
+ const video = { startSeconds: 0, ...options };
+ const current = configRef.current;
+ if (attach && selected) {
+ const target = current.references.find((reference) => reference.id === selected.id);
+ if (!target) throw new Error("The reference was removed while the video was importing.");
+ update(target.id, { kind: "video", relativePath: null, sourcePath, video, images: referenceImages(target) });
+ } else {
+ const id = `ref-${crypto.randomUUID()}`;
+ const name = sourcePath.split(/[\\/]/).pop()!.replace(/\.[^.]+$/, "");
+ onChange({ ...current, references: [{ id, kind: "video", name, description: "", sourcePath, video, intendedUse: [], createdAt: new Date().toISOString() }, ...current.references] });
+ setSelectedId(id);
+ }
+ } catch (reason) { setImportError(reason instanceof Error ? reason.message : String(reason)); }
+ finally { setImportingVideo(false); }
+ };
const createEmptyReference = () => {
addTextReference(`New ${referenceTypeLabel(pickerType).toLocaleLowerCase()}`, "", pickerType, pickerSubcategory === "all" ? "" : pickerSubcategory);
setPresetDialog(false);
@@ -132,6 +178,7 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
<h1>References</h1>
<p>{config.references.length === 1 ? "1 reference" : `${config.references.length} references`}</p>
</div>
+ <button className="secondary-button" disabled={importingVideo} onClick={() => void addVideo()}><Video size={16} /> {importingVideo ? "Importing video…" : "Import video"}</button>
</header>
<section className="reference-library" aria-labelledby="reference-library-heading">
{/* Both regions used to start at h3, so the outline jumped h1 → h3. */}
@@ -148,15 +195,15 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
? <span className="reference-art reference-art--photo"><ReferenceImage folderPath={folderPath} relativePath={ref.iconRelativePath} alt={`${ref.name} icon`} /></span>
: presetIcon
? <span className="reference-art reference-art--photo"><img src={presetIcon} alt="" /></span>
- : <span className="reference-copy-art"><FileText size={26} /></span>}
+ : <span className="reference-copy-art">{ref.kind === "video" ? <Video size={26} /> : <FileText size={26} />}</span>}
<span className="reference-card__body"><span><b>{ref.name}</b>{(images.length > 0 || ref.kind === "video" || ref.kind === "audio") && <small>{referenceKindLabel(ref)}</small>}</span>{isReferenceDescribed(ref)
? <p>{ref.description}</p>
- : <p className="reference-card__incomplete">{images.length > 0
+ : <p className="reference-card__incomplete">{ref.kind === "video" ? "The clip is sent as a reference. Add a prompt to describe what to keep." : images.length > 0
? "Not described yet — the picture is sent, but nothing tells the engine what to keep."
: "Not described yet — it won’t be used until you add a definition."}</p>}<span className="use-tags"><i>{referenceTypeLabel(referenceType(ref))}</i></span></span>
</button>;
})}
- <button className="reference-add-card" onClick={openNewReference}><span><Plus size={22} /></span><b>Add a reference</b><small>Import images or write a definition</small></button>
+ <button className="reference-add-card" onClick={openNewReference}><span><Plus size={22} /></span><b>Add a reference</b><small>Add images, video, or a definition</small></button>
</div>
</section>
</main>
@@ -174,7 +221,18 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
</> : <h2>Reference details</h2>}</header>
<div className="reference-inspector__scroll">
{selected ? <>
- <div className={`reference-detail-art${selectedImages.length > 0 || selected.iconRelativePath || selectedPresetIcon ? " reference-detail-art--photo" : " reference-detail-art--text"}${selectedPresetIcon || selected.iconRelativePath ? " reference-detail-art--preset" : ""}`}>
+ {selected.kind === "video" && <section className="reference-video" aria-label="Video reference">
+ <ReferenceVideo folderPath={folderPath} reference={selected} />
+ <p>Use 2–15 seconds per clip, up to 3 clips totaling 15 seconds. Requires CUDA and a Ref2VA generator.</p>
+ <ClipSecondsInput key={`${selected.id}-start`} label="Clip start (seconds)" min={0} value={selected.video?.startSeconds ?? 0}
+ onChange={(startSeconds) => update(selected.id, { video: { durationSeconds: 2, includeAudio: true, ...selected.video, startSeconds } })} />
+ <ClipSecondsInput key={`${selected.id}-duration`} label="Clip duration (seconds)" min={2} max={15} value={selected.video?.durationSeconds ?? 2}
+ onChange={(durationSeconds) => update(selected.id, { video: { startSeconds: 0, includeAudio: true, ...selected.video, durationSeconds } })} />
+ <label><input type="checkbox" checked={selected.video?.includeAudio ?? true}
+ onChange={(event) => update(selected.id, { video: { startSeconds: 0, durationSeconds: 2, ...selected.video, includeAudio: event.target.checked } })} /> Include sound</label>
+ <button className="secondary-button" onClick={() => update(selected.id, { kind: "text", sourcePath: null, relativePath: null, video: undefined })}><Trash2 size={16} /> Remove video</button>
+ </section>}
+ {(selected.kind !== "video" || selectedImages.length > 0) && <div className={`reference-detail-art${selectedImages.length > 0 || selected.iconRelativePath || selectedPresetIcon ? " reference-detail-art--photo" : " reference-detail-art--text"}${selectedPresetIcon || selected.iconRelativePath ? " reference-detail-art--preset" : ""}`}>
{selectedImages.length > 0
? <div className="reference-detail-images">{selectedImages.map((image) => <ReferenceImage key={image.id} folderPath={folderPath} relativePath={image.relativePath} sourcePath={image.sourcePath} alt={image.name} />)}</div>
: selected.iconRelativePath
@@ -198,11 +256,12 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
catch (reason) { setIconError(reason instanceof Error ? reason.message : String(reason)); }
}}
><RefreshCw size={14} aria-hidden="true" /></button>}
- </div>
+ </div>}
<div className="reference-fields">
<label><span>Prompt</span><textarea value={selected.description} placeholder="Describe what should stay consistent — the traits, materials, colours, or wardrobe Slopus should preserve across shots." onChange={(event) => update(selected.id, { description: event.target.value, content: event.target.value || null, subcategory: selectedSubcategory })} /></label>
<div className="reference-fields__actions">
<button className="secondary-button" onClick={() => void addImages()}><ImagePlus size={16} /> Add images</button>
+ <button className="secondary-button" disabled={importingVideo} onClick={() => void addVideo(true)}><Video size={16} /> {selected.kind === "video" ? "Replace video" : "Add video"}</button>
</div>
</div>
{selectedLocation && <section className="reference-location-settings" aria-label="Location settings">
@@ -272,7 +331,7 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
</section>
</div>
</div></div>}
- {importError && <div className="toast" role="alert"><strong>Couldn’t add image</strong><span>{importError}</span><button onClick={() => setImportError(null)}>Dismiss</button></div>}
+ {importError && <div className="toast" role="alert"><strong>Couldn’t add reference</strong><span>{importError}</span><button onClick={() => setImportError(null)}>Dismiss</button></div>}
{iconError && <div className="toast" role="alert"><strong>Couldn’t regenerate icon</strong><span>{iconError}</span><button onClick={() => setIconError(null)}>Dismiss</button></div>}
</div>;
}
diff --git a/src/components/workspace/ReferencesView.video.test.tsx b/src/components/workspace/ReferencesView.video.test.tsx
new file mode 100644
index 0000000..a81c90a
--- /dev/null
+++ b/src/components/workspace/ReferencesView.video.test.tsx
@@ -0,0 +1,41 @@
+// @vitest-environment jsdom
+import "@testing-library/jest-dom/vitest";
+import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { invoke } from "@tauri-apps/api/core";
+import { useState } from "react";
+import { afterEach, expect, it, vi } from "vitest";
+import { createProjectConfig, parseProjectConfig } from "../../lib/project";
+import { inspectReferenceVideo } from "../../lib/referenceVideo";
+import { ReferencesView } from "./ReferencesView";
+
+vi.mock("@tauri-apps/api/core", () => ({ invoke: vi.fn() }));
+vi.mock("../../lib/referenceVideo", () => ({ inspectReferenceVideo: vi.fn() }));
+vi.mock("./ReferenceVideo", () => ({ ReferenceVideo: () => <div>Video preview</div> }));
+afterEach(() => { cleanup(); vi.resetAllMocks(); delete (window as unknown as Record<string, unknown>).__TAURI_INTERNALS__; });
+
+it("imports a video, edits its saved trim and soundtrack, then removes its attachment", async () => {
+ Object.assign(window, { __TAURI_INTERNALS__: {} });
+ vi.mocked(invoke).mockResolvedValue("C:/motion.mp4");
+ vi.mocked(inspectReferenceVideo).mockResolvedValue({ durationSeconds: 10, includeAudio: true });
+ let latest = createProjectConfig({ name: "Video", prompt: "", aspectRatio: "16:9", resolution: "1080p", targetDurationSeconds: 20 });
+ function Harness() {
+ const [config, setConfig] = useState(latest); latest = config;
+ return <ReferencesView folderPath="C:/project" config={config} onChange={setConfig} />;
+ }
+ render(<Harness />);
+ fireEvent.click(screen.getByRole("button", { name: "Import video" }));
+ await screen.findByText("Video preview");
+ expect(invoke).toHaveBeenCalledWith("choose_reference_video", { folderPath: "C:/project" });
+ fireEvent.change(screen.getByLabelText("Clip start (seconds)"), { target: { value: "2" } });
+ fireEvent.blur(screen.getByLabelText("Clip start (seconds)"));
+ fireEvent.change(screen.getByLabelText("Clip duration (seconds)"), { target: { value: "4" } });
+ fireEvent.blur(screen.getByLabelText("Clip duration (seconds)"));
+ fireEvent.click(screen.getByLabelText("Include sound"));
+ expect(parseProjectConfig(JSON.parse(JSON.stringify(latest))).references[0]).toMatchObject({
+ kind: "video", sourcePath: "C:/motion.mp4", video: { startSeconds: 2, durationSeconds: 4, includeAudio: false },
+ });
+ fireEvent.click(screen.getByRole("button", { name: "Remove video" }));
+ await waitFor(() => expect(latest.references[0].kind).toBe("text"));
+ expect(latest.references[0].video).toBeUndefined();
+ expect(latest.references[0].sourcePath).toBeNull();
+});
diff --git a/src/lib/exportPipeline.ts b/src/lib/exportPipeline.ts
index 1555e76..c74cc0f 100644
--- a/src/lib/exportPipeline.ts
+++ b/src/lib/exportPipeline.ts
@@ -432,6 +432,8 @@ export async function probeCompositor(): Promise<CompositorProbe> {
export interface DemuxedSource {
config: VideoDecoderConfig;
+ durationSeconds?: number;
+ hasAudio?: boolean;
samples: Array<{ data: Uint8Array; timestampUs: number; durationUs: number; key: boolean }>;
}
@@ -523,6 +525,8 @@ export async function demux(bytes: ArrayBuffer, name: string): Promise<DemuxedSo
});
return {
+ durationSeconds: track.duration / track.timescale,
+ hasAudio: state.movie.audioTracks.length > 0,
config: {
codec: track.codec,
codedWidth: track.video?.width ?? track.track_width,
diff --git a/src/lib/project.ts b/src/lib/project.ts
index 4b0190e..e07d3a8 100644
--- a/src/lib/project.ts
+++ b/src/lib/project.ts
@@ -265,6 +265,11 @@ export const projectReferenceSchema = z.object({
// written definition can carry any number of images, and an image-only
// reference is simply a definition whose description is still blank.
images: z.array(projectReferenceImageSchema).optional(),
+ video: z.object({
+ startSeconds: z.number().finite().min(0),
+ durationSeconds: z.number().finite().min(2).max(15),
+ includeAudio: z.boolean(),
+ }).optional(),
intendedUse: z.array(z.enum(["character", "animal", "product", "location", "style", "audio"])).default([]),
subcategory: z.string().optional(),
// Generated library artwork is not an image conditioning attachment.
@@ -272,6 +277,9 @@ export const projectReferenceSchema = z.object({
createdAt: isoDateSchema,
}).superRefine((reference, context) => {
checkOneLocation(reference, context, `Reference '${reference.id}'`);
+ if (reference.video && reference.kind !== "video") {
+ context.addIssue({ code: z.ZodIssueCode.custom, path: ["video"], message: "Clip settings require a video reference." });
+ }
const imageIds = new Set<string>();
for (const image of reference.images ?? []) {
if (imageIds.has(image.id)) {
@@ -631,7 +639,7 @@ export function isReferenceDescribed(reference: ProjectReference): boolean {
* An image qualifies on its file alone: it can be sent to the engine and cited
* as <Picture N> with nothing written about it. */
export function isReferenceUsable(reference: ProjectReference): boolean {
- return referenceImages(reference).length > 0 || isReferenceDescribed(reference);
+ return referenceImages(reference).length > 0 || (reference.kind === "video" && Boolean(reference.sourcePath || reference.relativePath)) || isReferenceDescribed(reference);
}
/** Reference guide §2.1: `<Subject N>` is VISIBLE content. A reference tagged
@@ -646,7 +654,7 @@ export function isReferenceUsable(reference: ProjectReference): boolean {
* meets references saved before that. A reference carrying any other tag, or
* no tag at all, is visible content as before. */
export function isVisualReference(reference: ProjectReference): boolean {
- return reference.intendedUse.length === 0 || reference.intendedUse.some((use) => use !== "audio");
+ return reference.kind === "video" || reference.intendedUse.length === 0 || reference.intendedUse.some((use) => use !== "audio");
}
/* ---------------------------------------------------------------------------
@@ -719,8 +727,8 @@ const DEFAULT_MUSIC = "N/A";
const referenceLabel = (reference: ProjectReference, index: number) => `<Subject ${index + 1}>`;
-/** Image references are the only ones that can be sent to the engine as assets,
- * so they carry the <Picture N> numbering. That numbering MUST match the order
+/** Image references carry the <Picture N> numbering, independently of videos.
+ * That numbering MUST match the order
* of `reference_paths` in the generation request: slopfab.rs iterates the array
* and calls add_reference sequentially, so index 0 is <Picture 1>.
*
@@ -741,6 +749,11 @@ export function usableReferenceImages(references: ProjectReference[]): ProjectRe
return usableImageReferences(references).flatMap(referenceImages);
}
+/** Video numbering is independent of pictures and follows native insertion order. */
+export function usableVideoReferences(references: ProjectReference[]): ProjectReference[] {
+ return references.filter((reference) => reference.kind === "video" && Boolean(reference.sourcePath || reference.relativePath)).filter(isVisualReference);
+}
+
/** Opening and closing pictures come first, then bound references in project
* order. Frame anchors use their first image, even on a multi-image reference;
* the same anchor selected for both ends is sent only once. */
@@ -854,6 +867,7 @@ export interface SceneGenerationInput {
canvasWidth: number;
canvasHeight: number;
referencePaths: readonly string[];
+ referenceVideos?: readonly { name: string; relativePath?: string | null; sourcePath?: string | null; startSeconds: number; durationSeconds?: number; includeAudio: boolean }[];
previousSceneId?: string;
}
@@ -874,6 +888,7 @@ export function sceneGenerationSnapshot(job: GenerationJob, input: SceneGenerati
canvasWidth: input.canvasWidth,
canvasHeight: input.canvasHeight,
referencePaths: input.referencePaths,
+ ...(input.referenceVideos?.length ? { referenceVideos: input.referenceVideos } : {}),
...(input.previousSceneId ? { previousSceneId: input.previousSceneId } : {}),
shots,
});
@@ -1042,6 +1057,7 @@ export function compileScenePromptSegments(scene: ScenePrompt, references: Proje
const imageReferenceIds = usableImageReferences(references).flatMap((reference) =>
referenceImages(reference).map(() => reference.id));
const pictureNumbers = new Map<string, number[]>();
+ const videoNumbers = new Map(usableVideoReferences(references).map((reference, index) => [reference.id, index + 1]));
imageReferenceIds.forEach((referenceId, index) => pictureNumbers.set(referenceId, [...(pictureNumbers.get(referenceId) ?? []), index + 1]));
const listOf = (values: string[]): string => values.length === 1
? values[0]
@@ -1051,9 +1067,8 @@ export function compileScenePromptSegments(scene: ScenePrompt, references: Proje
// cited INSIDE its <Subject N> definition. A scene's explicit first frame is
// the exception: it describes composition, so it is emitted as a standalone
// <Picture N> entry instead of pretending to be a subject.
- // No <Audio N> is emitted either: §2.4 defines it as an actual audio asset,
- // and every reference in this app is text or image. An "audio" intendedUse
- // is a note about desired sound, not a signal to copy.
+ // Video citations follow their own native insertion order. An "audio"
+ // intendedUse remains a legacy text tag, not a standalone audio attachment.
// The identity slot carries ONLY what the user wrote. `name` is a library
// label, not a description: for an imported image it is the source file's
// stem, so using it here shipped "<Subject 1> is IMG_4821, shown in
@@ -1065,10 +1080,11 @@ export function compileScenePromptSegments(scene: ScenePrompt, references: Proje
const definitions = usable.flatMap((reference, index): PromptSegment[] => {
const detail = referenceDefinition(reference).replace(/\s+/g, " ");
const pictures = pictureNumbers.get(reference.id) ?? [];
+ const video = videoNumbers.get(reference.id);
const label = referenceLabel(reference, index);
const lead = index === 0 ? "" : "\n";
- if (pictures.length === 0) return [frame(`${lead}${label}: `), own(detail), ...addedStop(detail)];
- const pictureList = listOf(pictures.map((picture) => `<Picture ${picture}>`));
+ if (pictures.length === 0 && !video) return [frame(`${lead}${label}: `), own(detail), ...addedStop(detail)];
+ const pictureList = listOf([...pictures.map((picture) => `<Picture ${picture}>`), ...(video ? [`<Video ${video}>`] : [])]);
return [
frame(`${lead}${label} is the content shown in ${pictureList}.`),
...(detail ? [frame(" "), own(detail), ...addedStop(detail)] : []),
diff --git a/src/lib/project.video.test.ts b/src/lib/project.video.test.ts
new file mode 100644
index 0000000..730a7a9
--- /dev/null
+++ b/src/lib/project.video.test.ts
@@ -0,0 +1,36 @@
+import { expect, it } from "vitest";
+import { compileMiniMaxH3Prompt, createDraftGenerationJob, isReferenceUsable, projectReferenceSchema, sceneGenerationSnapshot, usableReferenceImages, usableVideoReferences, type ProjectReference } from "./project";
+
+const reference = (id: string, kind: ProjectReference["kind"], description = ""): ProjectReference => ({
+ id, kind, name: id, description, intendedUse: [], createdAt: "2026-09-12T00:00:00.000Z",
+ ...(kind !== "text" ? { sourcePath: `C:/references/${id}.${kind === "video" ? "mp4" : "png"}` } : {}),
+});
+
+it("keeps image and video numbering independent and stable in a mixed prompt", () => {
+ const references = [reference("motion", "video"), reference("hero", "image"), reference("style", "text", "Ink drawing"), reference("camera", "video")];
+ references[0].images = [{ id: "still", name: "Still", sourcePath: "C:/still.png" }];
+ expect(isReferenceUsable(references[0])).toBe(true);
+ expect(usableVideoReferences(references).map((item) => item.id)).toEqual(["motion", "camera"]);
+ expect(usableReferenceImages(references).map((item) => item.sourcePath)).toEqual(["C:/still.png", "C:/references/hero.png"]);
+ const prompt = compileMiniMaxH3Prompt("Follow the movement.", references);
+ expect(prompt).toContain("<Subject 1> is the content shown in <Picture 1> and <Video 1>.");
+ expect(prompt).toContain("<Subject 2> is the content shown in <Picture 2>.");
+ expect(prompt).toContain("<Subject 4> is the content shown in <Video 2>.");
+});
+
+it("round trips clip options and rejects invalid persisted ranges", () => {
+ const video = { ...reference("motion", "video"), video: { startSeconds: 3, durationSeconds: 5, includeAudio: false } };
+ expect(projectReferenceSchema.parse(JSON.parse(JSON.stringify(video))).video).toEqual(video.video);
+ expect(projectReferenceSchema.safeParse({ ...video, video: { ...video.video, durationSeconds: 16 } }).success).toBe(false);
+ expect(projectReferenceSchema.safeParse({ ...video, video: { ...video.video, startSeconds: -1 } }).success).toBe(false);
+});
+
+it("records video paths and trims in generation snapshots without temporary handles", () => {
+ const request = { prompt: "Motion", frames: 120, steps: 12, seed: 1, canvasWidth: 736, canvasHeight: 416, referencePaths: [],
+ referenceVideos: [{ name: "Motion", sourcePath: "C:/motion.mp4", startSeconds: 3, durationSeconds: 5, includeAudio: false }], referenceVideoIds: ["temporary"] };
+ const job = createDraftGenerationJob("Motion");
+ const snapshot = JSON.parse(sceneGenerationSnapshot(job, request));
+ expect(snapshot.referenceVideos).toEqual(request.referenceVideos);
+ expect(snapshot.referenceVideoIds).toBeUndefined();
+ expect(sceneGenerationSnapshot(job, { ...request, referenceVideos: [{ ...request.referenceVideos[0], startSeconds: 4 }] })).not.toBe(JSON.stringify(snapshot));
+});
diff --git a/src/lib/referenceVideo.test.ts b/src/lib/referenceVideo.test.ts
new file mode 100644
index 0000000..73be448
--- /dev/null
+++ b/src/lib/referenceVideo.test.ts
@@ -0,0 +1,90 @@
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import { invoke } from "@tauri-apps/api/core";
+import { demux, type DemuxedSource } from "./exportPipeline";
+import { prepareReferenceVideos, referenceSoundtrack, referenceVideoRange, uploadReferenceVideoFrames } from "./referenceVideo";
+import { createProjectConfig } from "./project";
+
+vi.mock("@tauri-apps/api/core", () => ({ invoke: vi.fn() }));
+vi.mock("./persistence", () => ({ readMediaFileBytes: vi.fn(async () => new ArrayBuffer(0)) }));
+vi.mock("./exportPipeline", () => ({ demux: vi.fn() }));
+beforeEach(() => vi.mocked(invoke).mockResolvedValue("video-1"));
+afterEach(() => { vi.unstubAllGlobals(); vi.resetAllMocks(); });
+
+const input = { name: "Movement", sourcePath: "C:/movement.mp4", startSeconds: 1, durationSeconds: 2, includeAudio: false };
+const source: DemuxedSource = { config: { codec: "avc1" }, durationSeconds: 4, hasAudio: false,
+ samples: [0, 500_000, 1_000_000, 1_500_000, 2_000_000, 2_500_000, 3_000_000].map((timestampUs) => ({
+ timestampUs, durationUs: 500_000, key: timestampUs === 0, data: new Uint8Array(1),
+ })) };
+
+function decoderMocks() {
+ const frames: Array<{ close: ReturnType<typeof vi.fn> }> = [];
+ const make = (timestamp: number) => {
+ const frame = { timestamp, displayWidth: 1920, displayHeight: 1080, close: vi.fn(), clone: () => make(timestamp) };
+ frames.push(frame); return frame;
+ };
+ const close = vi.fn();
+ vi.stubGlobal("EncodedVideoChunk", class { constructor(public init: EncodedVideoChunkInit) {} });
+ vi.stubGlobal("VideoDecoder", class {
+ static async isConfigSupported(config: VideoDecoderConfig) { return { supported: true, config }; }
+ state = "configured"; decodeQueueSize = 0;
+ constructor(private init: VideoDecoderInit) {}
+ configure() {}
+ decode(chunk: { init: EncodedVideoChunkInit }) { this.init.output(make(chunk.init.timestamp) as unknown as VideoFrame); }
+ async flush() {}
+ close = close;
+ });
+ vi.stubGlobal("OffscreenCanvas", class {
+ constructor(public width: number, public height: number) {}
+ getContext() { return { drawImage: vi.fn(), getImageData: () => ({ data: new Uint8ClampedArray(this.width * this.height * 4) }) }; }
+ });
+ return { frames, close };
+}
+
+it("validates trim bounds and uses at most 15 seconds for an unspecified clip", () => {
+ expect(referenceVideoRange(input, 4)).toEqual({ start: 1, duration: 2 });
+ expect(referenceVideoRange({ ...input, durationSeconds: undefined }, 30).duration).toBe(15);
+ expect(() => referenceVideoRange({ ...input, durationSeconds: 1 }, 4)).toThrow("2 to 15");
+ expect(() => referenceVideoRange(input, 2)).toThrow("beyond");
+ expect(() => referenceVideoRange({ ...input, startSeconds: NaN }, 4)).toThrow();
+});
+
+it("uploads the trimmed interval at clip-relative times and closes every frame", async () => {
+ const { frames, close } = decoderMocks();
+ await uploadReferenceVideoFrames(source, "video-1", 1, 2, () => false);
+ const calls = vi.mocked(invoke).mock.calls.filter(([command]) => command === "append_reference_video");
+ expect(calls.map((call) => (call[2]?.headers as Record<string, string>)["x-reference-time"])).toEqual(["0", "0.5", "1", "1.5"]);
+ expect(calls[0][2]?.headers).toMatchObject({ "x-reference-width": "768", "x-reference-height": "432" });
+ expect(calls.every((call) => call[1] instanceof Uint8Array)).toBe(true);
+ expect(frames.every((frame) => frame.close.mock.calls.length === 1)).toBe(true);
+ expect(close).toHaveBeenCalledOnce();
+});
+
+it("releases native inputs and decoder frames after an upload failure", async () => {
+ const { frames, close } = decoderMocks();
+ vi.mocked(demux).mockResolvedValue(source);
+ vi.mocked(invoke).mockImplementation(async (command) => {
+ if (command === "append_reference_video") throw new Error("upload failed");
+ return "video-1";
+ });
+ const config = createProjectConfig({ name: "Test", prompt: "", aspectRatio: "16:9", resolution: "1080p", targetDurationSeconds: 20 });
+ const request = { jobId: "job", prompt: "", frames: 120, steps: 12, seed: 1, canvasWidth: 736, canvasHeight: 416, referencePaths: [], referenceVideos: [input] };
+ await expect(prepareReferenceVideos("C:/project", request, config, () => false, () => undefined)).rejects.toThrow("upload failed");
+ expect(invoke).toHaveBeenCalledWith("release_reference_videos", { ids: ["video-1"] });
+ expect(frames.every((frame) => frame.close.mock.calls.length === 1)).toBe(true);
+ expect(close).toHaveBeenCalledOnce();
+});
+
+it("stops decoding promptly when preparation is cancelled", async () => {
+ const { frames, close } = decoderMocks();
+ let cancelled = false;
+ vi.mocked(invoke).mockImplementation(async () => { cancelled = true; });
+ await expect(uploadReferenceVideoFrames(source, "video-1", 1, 2, () => cancelled)).rejects.toThrow("cancelled");
+ expect(frames.every((frame) => frame.close.mock.calls.length === 1)).toBe(true);
+ expect(close).toHaveBeenCalledOnce();
+});
+
+it("trims and interleaves stereo sound without exceeding the video duration", () => {
+ const audio = { sampleRate: 2, numberOfChannels: 2, length: 8,
+ getChannelData: (channel: number) => new Float32Array(channel ? [0, 0, -0.5, -2, 0.5, 0, 0, 0] : [0, 0, 0.5, 2, -0.5, 0, 0, 0]) } as AudioBuffer;
+ expect(Array.from(referenceSoundtrack(audio, 1, 2))).toEqual([0.5, -0.5, 1, -1, -0.5, 0.5, 0, 0]);
+});
diff --git a/src/lib/referenceVideo.ts b/src/lib/referenceVideo.ts
new file mode 100644
index 0000000..a0611e2
--- /dev/null
+++ b/src/lib/referenceVideo.ts
@@ -0,0 +1,175 @@
+import { invoke } from "@tauri-apps/api/core";
+import { demux, type DemuxedSource } from "./exportPipeline";
+import { readMediaFileBytes } from "./persistence";
+import type { ProjectConfig } from "./project";
+import type { SlopfabGenerationRequest } from "./runtime";
+
+type VideoInput = NonNullable<SlopfabGenerationRequest["referenceVideos"]>[number];
+const cancelledError = () => new Error("Video reference preparation cancelled.");
+
+export function referenceVideoRange(input: VideoInput, sourceDuration: number) {
+ const start = input.startSeconds;
+ const duration = input.durationSeconds ?? Math.min(15, sourceDuration - start);
+ if (!Number.isFinite(start) || start < 0 || !Number.isFinite(duration) || duration < 2 || duration > 15) {
+ throw new Error(`${input.name}: choose a clip lasting 2 to 15 seconds with a nonnegative start.`);
+ }
+ if (!Number.isFinite(sourceDuration) || start + duration > sourceDuration + 0.001) {
+ throw new Error(`${input.name}: the selected clip extends beyond the source video.`);
+ }
+ return { start, duration };
+}
+
+export function sourceVideoDuration(source: DemuxedSource): number {
+ const first = source.samples.reduce((value, sample) => Math.min(value, sample.timestampUs), Infinity);
+ const last = source.samples.reduce((value, sample) => Math.max(value, sample.timestampUs + sample.durationUs), -Infinity);
+ return source.durationSeconds ?? (last - first) / 1_000_000;
+}
+
+export async function inspectReferenceVideo(folderPath: string, sourcePath: string) {
+ const source = await demux(await readMediaFileBytes(folderPath, { sourcePath }), "Video reference");
+ const duration = sourceVideoDuration(source);
+ referenceVideoRange({ name: "Video reference", startSeconds: 0, includeAudio: true }, duration);
+ return { durationSeconds: Math.min(15, duration), includeAudio: Boolean(source.hasAudio) };
+}
+
+/** Decode only the selected interval. At most a small decoder batch and one
+ * retained frame are alive; binary IPC avoids JSON/base64 copies of pixels.
+ * Slopfab consumes host RGB, so readback is confined to this ingestion step. */
+export async function uploadReferenceVideoFrames(source: DemuxedSource, id: string, start: number, duration: number, cancelled: () => boolean) {
+ if (typeof VideoDecoder === "undefined") throw new Error("This webview does not support WebCodecs video decoding.");
+ const support = await VideoDecoder.isConfigSupported({ ...source.config, hardwareAcceleration: "prefer-hardware" });
+ if (!support.supported) throw new Error(`This computer cannot decode reference codec ${source.config.codec}.`);
+ const origin = source.samples.reduce((value, sample) => Math.min(value, sample.timestampUs), Infinity);
+ const beginUs = origin + Math.round(start * 1_000_000);
+ const endUs = beginUs + Math.round(duration * 1_000_000);
+ let first = 0;
+ for (let index = 0; index < source.samples.length; index++) {
+ if (source.samples[index].key && source.samples[index].timestampUs <= beginUs) first = index;
+ }
+ // Include the rest of the last GOP for reordered/B frames at the trim edge.
+ let last = source.samples.length;
+ for (let index = first + 1; index < source.samples.length; index++) {
+ if (source.samples[index].key && source.samples[index].timestampUs >= endUs) { last = index; break; }
+ }
+ const state: { previous: VideoFrame | null; error: unknown; pending: number; uploaded: number; slot: number; canvas: OffscreenCanvas | null } =
+ { previous: null, error: null, pending: 0, uploaded: 0, slot: -1, canvas: null };
+ let chain = Promise.resolve();
+ const check = () => { if (cancelled()) throw cancelledError(); if (state.error) throw state.error; };
+ const upload = async (frame: VideoFrame, seconds: number) => {
+ check();
+ const slot = Math.round(seconds * 24);
+ if (slot <= state.slot || slot >= Math.round(duration * 24)) return;
+ if (!state.canvas) {
+ const aspect = frame.displayWidth / frame.displayHeight;
+ if (aspect < 0.25 || aspect > 4) throw new Error("Video reference aspect ratio must be between 1:4 and 4:1.");
+ const scale = Math.min(1, 768 / Math.max(frame.displayWidth, frame.displayHeight));
+ state.canvas = new OffscreenCanvas(Math.max(1, Math.round(frame.displayWidth * scale)), Math.max(1, Math.round(frame.displayHeight * scale)));
+ }
+ const canvas = state.canvas;
+ const context = canvas.getContext("2d", { willReadFrequently: true });
+ if (!context) throw new Error("Could not create the video reference pixel buffer.");
+ context.drawImage(frame, 0, 0, canvas.width, canvas.height);
+ const pixels = context.getImageData(0, 0, canvas.width, canvas.height).data;
+ await invoke("append_reference_video", new Uint8Array(pixels.buffer), { headers: {
+ "x-reference-id": id, "x-reference-width": String(canvas.width), "x-reference-height": String(canvas.height),
+ "x-reference-time": String(state.uploaded === 0 ? 0 : seconds),
+ } });
+ state.slot = slot;
+ state.uploaded += 1;
+ };
+ const consume = async (frame: VideoFrame) => {
+ check();
+ if (frame.timestamp <= beginUs) {
+ state.previous?.close();
+ state.previous = frame.clone();
+ return;
+ }
+ if (!state.uploaded && state.previous) {
+ await upload(state.previous, 0);
+ state.previous.close(); state.previous = null;
+ }
+ if (frame.timestamp < endUs) await upload(frame, state.uploaded ? (frame.timestamp - beginUs) / 1_000_000 : 0);
+ };
+ const decoder = new VideoDecoder({
+ output: (frame) => {
+ state.pending += 1;
+ chain = chain.then(() => consume(frame)).catch((error: unknown) => { state.error ??= error; })
+ .finally(() => { frame.close(); state.pending -= 1; });
+ },
+ error: (error) => { state.error = error; },
+ });
+ try {
+ decoder.configure(support.config ?? source.config);
+ for (let index = first; index < last; index++) {
+ check();
+ const sample = source.samples[index];
+ decoder.decode(new EncodedVideoChunk({ type: sample.key ? "key" : "delta", timestamp: sample.timestampUs, duration: sample.durationUs, data: sample.data }));
+ while (decoder.decodeQueueSize > 8 || state.pending > 8) {
+ await new Promise((resolve) => setTimeout(resolve, 0)); check();
+ }
+ }
+ await decoder.flush();
+ await chain;
+ check();
+ if (!state.uploaded && state.previous) await upload(state.previous, 0);
+ if (!state.uploaded) throw new Error("The selected reference interval contains no decodable frames.");
+ } finally {
+ if (decoder.state !== "closed") decoder.close();
+ await chain;
+ state.previous?.close();
+ }
+}
+
+export function referenceSoundtrack(buffer: AudioBuffer, start: number, duration: number): Float32Array {
+ if (buffer.numberOfChannels < 1 || buffer.numberOfChannels > 2) throw new Error("Video reference soundtracks must be mono or stereo. Turn off Include sound for this clip or use a mono/stereo source.");
+ const offset = Math.round(start * buffer.sampleRate);
+ const length = Math.min(Math.floor(duration * buffer.sampleRate), buffer.length - offset);
+ if (length <= 0) throw new Error("The selected video interval contains no soundtrack samples. Turn off Include sound for this clip.");
+ const pcm = new Float32Array(length * buffer.numberOfChannels);
+ for (let channel = 0; channel < buffer.numberOfChannels; channel++) {
+ const samples = buffer.getChannelData(channel);
+ for (let index = 0; index < length; index++) pcm[index * buffer.numberOfChannels + channel] = Math.max(-1, Math.min(1, samples[offset + index]));
+ }
+ return pcm;
+}
+
+export async function releaseReferenceVideos(ids: string[]) {
+ if (ids.length) await invoke("release_reference_videos", { ids });
+}
+
+export async function prepareReferenceVideos(folderPath: string, request: SlopfabGenerationRequest, config: ProjectConfig, cancelled: () => boolean, report: (detail: string) => void): Promise<string[]> {
+ const inputs = request.referenceVideos ?? [];
+ if (inputs.length > 3 || request.referencePaths.length > 9) throw new Error("Use at most nine images and three video references per generation.");
+ const ids: string[] = [];
+ let totalDuration = 0;
+ try {
+ for (const input of inputs) {
+ if (cancelled()) throw cancelledError();
+ report(`Preparing video reference: ${input.name}`);
+ if (!/\.(mp4|m4v|mov)$/i.test(input.sourcePath ?? input.relativePath ?? "")) throw new Error(`${input.name}: video references currently require an MP4 or MOV file.`);
+ const bytes = await readMediaFileBytes(folderPath, input);
+ const source = await demux(bytes, input.name);
+ const { start, duration } = referenceVideoRange(input, sourceVideoDuration(source));
+ totalDuration += duration;
+ if (totalDuration > 15 + 1e-9) throw new Error("Video references must total no more than 15 seconds. Shorten the clips in References.");
+ if (cancelled()) throw cancelledError();
+ const id = await invoke<string>("create_reference_video", { durationSeconds: duration, config });
+ ids.push(id);
+ await uploadReferenceVideoFrames(source, id, start, duration, cancelled);
+ if (input.includeAudio && source.hasAudio) {
+ report(`Preparing soundtrack: ${input.name}`);
+ const context = new OfflineAudioContext(2, 1, 48_000);
+ const decoded = await context.decodeAudioData(bytes.slice(0));
+ if (cancelled()) throw cancelledError();
+ const pcm = referenceSoundtrack(decoded, start, duration);
+ await invoke("set_reference_video_audio", new Uint8Array(pcm.buffer), { headers: {
+ "x-reference-id": id, "x-reference-channels": String(decoded.numberOfChannels), "x-reference-rate": String(decoded.sampleRate),
+ } });
+ }
+ }
+ return ids;
+ } catch (error) {
+ await releaseReferenceVideos(ids);
+ throw error;
+ }
+}
diff --git a/src/lib/runtime.ts b/src/lib/runtime.ts
index e96ad25..96546f7 100644
--- a/src/lib/runtime.ts
+++ b/src/lib/runtime.ts
@@ -88,6 +88,9 @@ export interface SlopfabGenerationRequest {
canvasWidth: number;
canvasHeight: number;
referencePaths: string[];
+ referenceVideos?: Array<{ name: string; relativePath?: string | null; sourcePath?: string | null; startSeconds: number; durationSeconds?: number; includeAudio: boolean }>;
+ /** Temporary native handles, populated by the work queue and never saved. */
+ referenceVideoIds?: string[];
/** Resolved by the app queue before native submission. */
previousSceneId?: string;
}
diff --git a/src/lib/workQueue.test.ts b/src/lib/workQueue.test.ts
index f86a722..8e5cebe 100644
--- a/src/lib/workQueue.test.ts
+++ b/src/lib/workQueue.test.ts
@@ -10,6 +10,7 @@ import { cancelSlopfabGeneration, enqueueSlopfabGeneration, resolveSlopfabPlan }
import { saveEngineSettings, EMPTY_ENGINE_SETTINGS } from "./settings";
import { WorkQueue, type GenerationSubmission } from "./workQueue";
import { saveSceneLastFrame } from "./sceneLastFrame";
+import { prepareReferenceVideos, releaseReferenceVideos } from "./referenceVideo";
import { downloadTemplateWeights, getWeightDownloadState } from "./weightDownloads";
vi.mock("@tauri-apps/api/core", () => ({ invoke: vi.fn() }));
@@ -17,6 +18,7 @@ vi.mock("@tauri-apps/api/event", () => ({ listen: vi.fn() }));
vi.mock("./generatedVideo", () => ({ saveGeneratedScene: vi.fn(), releaseRendered: vi.fn(async () => true) }));
vi.mock("./timelineThumbnails", () => ({ purgeTimelineThumbnails: vi.fn(async () => undefined) }));
vi.mock("./sceneLastFrame", () => ({ saveSceneLastFrame: vi.fn(async () => "cache/scene-last/work-new.png") }));
+vi.mock("./referenceVideo", () => ({ prepareReferenceVideos: vi.fn(async () => ["video-1"]), releaseReferenceVideos: vi.fn(async () => undefined) }));
vi.mock("./runtime", () => ({ cancelSlopfabGeneration: vi.fn(), enqueueSlopfabGeneration: vi.fn(), resolveSlopfabPlan: vi.fn() }));
const handlers = new Map<string, (event: { payload: unknown }) => void>();
const stops: (() => void)[] = [];
@@ -51,6 +53,30 @@ const finish = async (queue: WorkQueue, id: string) => {
};
describe("application work queue", () => {
+ it("prepares video inputs for both planning and generation and releases them after completion", async () => {
+ const { queue, first } = setup();
+ const next = submission(first);
+ next.request.referenceVideos = [{ name: "Motion", sourcePath: "C:/motion.mp4", startSeconds: 1, durationSeconds: 2, includeAudio: true }];
+ queue.enqueue(first, [next]);
+ await waitFor(() => expect(enqueueSlopfabGeneration).toHaveBeenCalledOnce());
+ expect(prepareReferenceVideos).toHaveBeenCalledOnce();
+ expect(vi.mocked(resolveSlopfabPlan).mock.calls[0][0].referenceVideoIds).toEqual(["video-1"]);
+ expect(vi.mocked(enqueueSlopfabGeneration).mock.calls[0][0].referenceVideoIds).toEqual(["video-1"]);
+ await finish(queue, queue.getSnapshot()[0].id);
+ await waitFor(() => expect(releaseReferenceVideos).toHaveBeenCalledWith(["video-1"]));
+ });
+
+ it("releases prepared video inputs when planning fails", async () => {
+ const { queue, first } = setup();
+ const next = submission(first);
+ next.request.referenceVideos = [{ name: "Motion", sourcePath: "C:/motion.mp4", startSeconds: 0, durationSeconds: 2, includeAudio: false }];
+ vi.mocked(resolveSlopfabPlan).mockRejectedValueOnce(new Error("Reference limit exceeded"));
+ queue.enqueue(first, [next]);
+ await waitFor(() => expect(queue.getSnapshot()[0].status).toBe("failed"));
+ expect(releaseReferenceVideos).toHaveBeenCalledWith(["video-1"]);
+ expect(enqueueSlopfabGeneration).not.toHaveBeenCalled();
+ });
+
it("resolves a linked first frame after the previous scene finishes encoding and saving", async () => {
const { queue, first } = setup();
first.update((config) => ({ ...config, generationJobs: [config.generationJobs[0], { ...config.generationJobs[0], id: "linked", usePreviousSceneLastFrame: true }] }));
diff --git a/src/lib/workQueue.ts b/src/lib/workQueue.ts
index e106aa0..dac793e 100644
--- a/src/lib/workQueue.ts
+++ b/src/lib/workQueue.ts
@@ -10,6 +10,7 @@ import { cancelSlopfabGeneration, enqueueSlopfabGeneration, resolveSlopfabPlan,
import { withEngineSettings } from "./settings";
import { purgeTimelineThumbnails } from "./timelineThumbnails";
import { ReferenceIconWork } from "./referenceIconWork";
+import { prepareReferenceVideos, releaseReferenceVideos } from "./referenceVideo";
export type WorkStatus = "queued" | "preparing" | "generating" | "encoding" | "completed" | "failed" | "cancelled";
export interface GenerationSubmission { job: GenerationJob; request: SlopfabGenerationRequest; snapshot: string }
@@ -215,6 +216,11 @@ export class WorkQueue {
}
await purgeTimelineThumbnails(next.folderPath, work.sceneId).catch(() => undefined);
if (work.cancelled) continue;
+ if (work.request.referenceVideos?.length) {
+ work.request.referenceVideoIds = await prepareReferenceVideos(next.folderPath, work.request, work.config,
+ () => work.cancelled, (detail) => this.patch(work.id, { detail }));
+ }
+ if (work.cancelled) continue;
const plan = await resolveSlopfabPlan(work.request, work.config);
if (work.cancelled) continue;
this.patch(work.id, { status: "generating", detail: `Generating ${plan.alignedFrames} frames at ${plan.canvasWidth} × ${plan.canvasHeight}` });
@@ -224,6 +230,11 @@ export class WorkQueue {
if (work.cancelled) await this.requestNativeCancellation(work);
await work.done;
} catch (reason) { this.fail(work, reason); }
+ finally {
+ await releaseReferenceVideos(work.request.referenceVideoIds ?? []).catch((reason) =>
+ writeDiagnostic("error", "work-queue", "references.release_failed", describeDiagnosticError(reason), { workId: work.id }));
+ delete work.request.referenceVideoIds;
+ }
}
} finally { this.pumping = false; }
}
diff --git a/src/styles/references.css b/src/styles/references.css
index b6ee16b..0ce28c5 100644
--- a/src/styles/references.css
+++ b/src/styles/references.css
@@ -578,6 +578,37 @@
font: inherit;
}
+.reference-video {
+ display: grid;
+ gap: var(--space-3);
+ padding: var(--space-4);
+}
+
+.reference-video-preview {
+ display: block;
+ width: 100%;
+ max-height: 280px;
+ background: var(--field-bg);
+ border-radius: var(--radius-sm);
+}
+
+.reference-video p {
+ color: var(--text-muted);
+ font-size: var(--text-xs);
+}
+
+.reference-video label {
+ display: flex;
+ align-items: center;
+ gap: var(--space-2);
+}
+
+.reference-video input[type="number"] {
+ min-width: 0;
+ width: 80px;
+ margin-left: auto;
+}
+
.reference-fields__actions {
display: flex;
flex-wrap: wrap;