Skip to content

AI media harness for planning, generating, and editing locally

TypeScript

Add refmod references and unified file picker

Bitti6bdea84aedd1
diff --git a/docs/refmods.md b/docs/refmods.md
new file mode 100644
index 0000000..cc5a769
--- /dev/null
+++ b/docs/refmods.md
@@ -0,0 +1,29 @@
+# Reference files and refmods
+
+In References, select a reference and choose **Add file**. The same picker accepts
+PNG, JPEG, WebP, MP4/M4V/MOV video, and standalone H3 refmod `.safetensors` files.
+Multiple images and refmods can be attached together with one video. Adding a
+video replaces that reference's existing video; other attachments remain.
+Images are copied into the project. Videos and refmods stay at their original
+locations, preserving access to any same-stem refmod metadata `.json` sidecar.
+
+Refmods contain pre-encoded image, video, or audio reference latents. They are
+separate from LoRA weight adapters. Each refmod has **Strength** (0–1, default 1)
+and **Copies** (1–10, default 1). Strength 0 disables it. More copies increase
+the packed sequence size and GPU memory usage. The paths, strengths, copies,
+and attachment order are saved in the project and captured in scene generation
+snapshots. Refmods follow ordinary references in native requests and consume
+neither Picture/Video labels nor Qwen vision tokens. Their metadata descriptions
+and trigger words are not automatically inserted into the prompt.
+
+**Render icon with refmod** creates an icon with all enabled refmods on that
+reference active, even without a written description. Other references in the
+project are not attached. Editing or removing its refmods invalidates its icon;
+an in-flight icon made with older settings is not attached. Existing icon queue,
+cancellation, and project persistence behavior applies.
+
+Refmods require a Ref2VA generator and a SlopFab DLL exposing the 1.8 refmod API.
+Both CUDA and Vulkan use the same attachment path. SlopFab validates the tensors
+and metadata, reporting incompatible files or bundles. Older DLLs remain usable
+without refmods and give an explicit compatibility error when an enabled refmod
+is submitted. Slopus does not encode, train, or optimize refmods.
diff --git a/docs/video-references.md b/docs/video-references.md
index 3deb39a..a5dcc0e 100644
--- a/docs/video-references.md
+++ b/docs/video-references.md
@@ -1,7 +1,7 @@
# Video references
In **References**, choose **Add a reference → Other → New** to create an
-**Uncategorized** reference, then choose **Add video** in its details. Video
+**Uncategorized** reference, then choose **Add file** in its details. Video
can also be attached to an existing reference. The reference card shows the
video's first source frame, including after reopening the project; replacing
the video updates the icon. MP4, M4V, and MOV containers are accepted; the computer's WebCodecs
diff --git a/src-tauri/src/agent.rs b/src-tauri/src/agent.rs
index 8c26158..27190f3 100644
--- a/src-tauri/src/agent.rs
+++ b/src-tauri/src/agent.rs
@@ -1866,6 +1866,7 @@ mod tests {
relative_path: None,
source_path: None,
images: Vec::new(),
+ refmods: Vec::new(),
video: None,
intended_use: vec!["character".into()],
subcategory: None,
diff --git a/src-tauri/src/agent_commands.rs b/src-tauri/src/agent_commands.rs
index 265419c..df16637 100644
--- a/src-tauri/src/agent_commands.rs
+++ b/src-tauri/src/agent_commands.rs
@@ -453,6 +453,7 @@ fn apply_command(
relative_path: None,
source_path: None,
images: Vec::new(),
+ refmods: Vec::new(),
video: None,
intended_use: intended_use.clone(),
subcategory: None,
diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs
index 723449b..3ef2159 100644
--- a/src-tauri/src/lib.rs
+++ b/src-tauri/src/lib.rs
@@ -226,6 +226,8 @@ struct ReusableReference {
/// readable for projects made when an image was a separate reference kind.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
images: Vec<ReferenceImage>,
+ #[serde(default, skip_serializing_if = "Vec::is_empty")]
+ refmods: Vec<ReferenceRefmod>,
#[serde(default, skip_serializing_if = "Option::is_none")]
video: Option<ReferenceVideoOptions>,
#[serde(default)]
@@ -256,6 +258,19 @@ struct ReferenceVideoOptions {
include_audio: bool,
}
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+struct ReferenceRefmod {
+ #[serde(flatten)]
+ file: ReferenceImage,
+ #[serde(default = "default_refmod_strength")]
+ strength: f64,
+ #[serde(default = "default_refmod_copies")]
+ copies: u32,
+}
+fn default_refmod_strength() -> f64 { 1.0 }
+fn default_refmod_copies() -> u32 { 1 }
+
/// One shot inside a scene. Mirrors `sceneShotSchema` in src/lib/project.ts.
///
/// `startSeconds` and `action` are REQUIRED on the zod side and are plain
@@ -836,6 +851,18 @@ fn validate_and_normalize_config(mut config: ProjectConfig) -> Result<ProjectCon
));
}
let mut image_ids = BTreeSet::new();
+ let mut refmod_ids = BTreeSet::new();
+ for refmod in &mut reference.refmods {
+ let file = &mut refmod.file;
+ if file.id.trim().is_empty() || file.name.trim().is_empty() || !refmod_ids.insert(file.id.clone())
+ || !refmod.strength.is_finite() || !(0.0..=1.0).contains(&refmod.strength)
+ || !(1..=10).contains(&refmod.copies) {
+ return Err("Refmods need unique ids, names, strength 0 to 1 and copies 1 to 10.".into());
+ }
+ file.relative_path = file.relative_path.as_deref().map(normalize_project_path).transpose()?;
+ file.source_path = file.source_path.as_deref().map(normalize_external_path).transpose()?;
+ check_one_location("Refmod", file.relative_path.as_ref(), file.source_path.as_ref())?;
+ }
if let Some(video) = &reference.video {
if reference.kind != "video" || !video.start_seconds.is_finite() || video.start_seconds < 0.0
|| !video.duration_seconds.is_finite() || !(2.0..=15.0).contains(&video.duration_seconds) {
@@ -1678,7 +1705,8 @@ fn choose_reference_images(
/// One reference the user added: an image copied into `references/`, or a
/// video/audio file left where it is and pointed at.
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Serialize)]
+#[serde(rename_all = "camelCase")]
struct ImportedReference {
kind: &'static str,
name: String,
@@ -2166,6 +2194,7 @@ fn create_project_at_with_references(
relative_path: imported.relative_path,
source_path: imported.source_path,
images: Vec::new(),
+ refmods: Vec::new(),
video: None,
intended_use: vec!["style".into()],
subcategory: None,
@@ -2355,6 +2384,45 @@ fn choose_reference_video(app: AppHandle, folder_path: String) -> Result<Option<
picked_external_source_path(&path, &root).map(Some)
}
+fn reference_attachment_kind(path: &Path) -> Result<&'static str, String> {
+ match path.extension().and_then(|ext| ext.to_str()).unwrap_or("").to_ascii_lowercase().as_str() {
+ "png" | "jpg" | "jpeg" | "webp" => Ok("image"),
+ "mp4" | "m4v" | "mov" => Ok("video"),
+ "safetensors" => Ok("refmod"),
+ _ => Err("Choose an image, MP4/MOV video, or refmod safetensors file.".into()),
+ }
+}
+
+fn import_reference_attachments(root: &Path, paths: &[PathBuf]) -> Result<Vec<ImportedReference>, String> {
+ let kinds = paths.iter().map(|path| {
+ if !path.is_file() { return Err("The selected reference file does not exist.".into()); }
+ reference_attachment_kind(path)
+ }).collect::<Result<Vec<_>, String>>()?;
+ if kinds.iter().filter(|kind| **kind == "video").count() > 1 {
+ return Err("Add one video per reference. Use another reference for additional videos.".into());
+ }
+ paths.iter().zip(kinds).map(|(path, kind)| {
+ if kind == "image" { return import_reference_file(path, root); }
+ Ok(ImportedReference {
+ kind,
+ name: path.file_name().and_then(|name| name.to_str()).unwrap_or("Reference").into(),
+ relative_path: None,
+ source_path: Some(picked_external_source_path(path, root)?),
+ })
+ }).collect()
+}
+
+#[tauri::command]
+fn choose_reference_files(app: AppHandle, folder_path: String) -> Result<Vec<ImportedReference>, String> {
+ let root = project_root(&folder_path)?;
+ let selected = app.dialog().file().set_title("Add reference files")
+ .add_filter("Images, videos and refmods", &["png", "jpg", "jpeg", "webp", "mp4", "m4v", "mov", "safetensors"])
+ .blocking_pick_files().unwrap_or_default();
+ let paths = selected.into_iter().map(|path| path.into_path().map_err(|error| format!("Could not access selected file: {error}")))
+ .collect::<Result<Vec<_>, _>>()?;
+ import_reference_attachments(&root, &paths)
+}
+
#[tauri::command]
fn create_reference_video(duration_seconds: f64, config: ProjectConfig) -> Result<String, String> {
let config = validate_and_normalize_config(config)?;
@@ -2956,6 +3024,7 @@ pub fn run() {
choose_reference_image,
choose_reference_images,
choose_reference_video,
+ choose_reference_files,
create_reference_video,
append_reference_video,
set_reference_video_audio,
@@ -4425,6 +4494,7 @@ mod tests {
relative_path: None,
source_path: None,
images: Vec::new(),
+ refmods: Vec::new(),
video: None,
intended_use: Vec::new(),
subcategory: None,
@@ -4437,6 +4507,47 @@ mod tests {
);
}
+ #[test]
+ fn refmod_attachments_round_trip_and_validate_ranges_and_paths() {
+ let mut value = serde_json::to_value(fixture()).unwrap();
+ value["references"][0]["refmods"] = serde_json::json!([
+ { "id": "latent", "name": "Person", "sourcePath": "D:/person.safetensors", "strength": 0.7, "copies": 2 }
+ ]);
+ let config = validate_and_normalize_config(serde_json::from_value(value.clone()).unwrap()).unwrap();
+ let root = tempfile::tempdir().unwrap();
+ let created = create_project_in(root.path(), &config).unwrap();
+ let folder = Path::new(&created.folder_path);
+ write_project(folder, &created.config).unwrap();
+ assert_eq!(read_project(folder).unwrap().config.references[0].refmods, config.references[0].refmods);
+ for (key, invalid) in [
+ ("strength", serde_json::json!(-0.1)), ("strength", serde_json::json!(1.1)),
+ ("copies", serde_json::json!(0)), ("copies", serde_json::json!(11)),
+ ("sourcePath", serde_json::json!("../outside.safetensors")),
+ ("sourcePath", serde_json::Value::Null),
+ ] {
+ let mut broken = value.clone();
+ broken["references"][0]["refmods"][0][key] = invalid;
+ assert!(serde_json::from_value(broken).map_err(|error| error.to_string()).and_then(validate_and_normalize_config).is_err());
+ }
+ }
+
+ #[test]
+ fn unified_reference_import_copies_images_and_keeps_video_and_refmod_sources() {
+ let root = tempfile::tempdir().unwrap();
+ let project = root.path().join("project");
+ fs::create_dir_all(&project).unwrap();
+ let paths = [root.path().join("photo.png"), root.path().join("motion.mp4"), root.path().join("person.safetensors")];
+ for path in &paths { fs::write(path, b"source").unwrap(); }
+ let imported = import_reference_attachments(&project, &paths).unwrap();
+ assert_eq!(imported.iter().map(|file| file.kind).collect::<Vec<_>>(), ["image", "video", "refmod"]);
+ assert!(project.join(imported[0].relative_path.as_ref().unwrap()).is_file());
+ assert!(imported[2].relative_path.is_none());
+ assert_eq!(imported[2].source_path.as_deref(), Some(external_source_path(&paths[2]).unwrap().as_str()));
+ assert!(paths.iter().all(|path| path.is_file()));
+ assert!(import_reference_attachments(&project, &[paths[1].clone(), paths[1].clone()]).unwrap_err().contains("one video"));
+ assert!(reference_attachment_kind(Path::new("script.exe")).is_err());
+ }
+
#[test]
fn video_reference_clip_settings_survive_save_and_reopen() {
let mut config = fixture();
diff --git a/src-tauri/src/slopfab.rs b/src-tauri/src/slopfab.rs
index 506dd57..fa884f9 100644
--- a/src-tauri/src/slopfab.rs
+++ b/src-tauri/src/slopfab.rs
@@ -127,6 +127,15 @@ pub struct GenerationRequest {
pub reference_paths: Vec<String>,
#[serde(default)]
pub reference_video_ids: Vec<String>,
+ #[serde(default)]
+ pub refmods: Vec<RefmodInput>,
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct RefmodInput {
+ pub path: String,
+ pub strength: f32,
+ pub copies: u32,
}
impl GenerationRequest {
@@ -710,6 +719,14 @@ fn configure_request(
.attach(handle, &configuration.dll_path)?;
}
}
+ for refmod in &request.refmods {
+ if refmod.path.trim().is_empty() || !refmod.strength.is_finite() || !(0.0..=1.0).contains(&refmod.strength)
+ || !(1..=10).contains(&refmod.copies) {
+ return Err("Refmods need a path, strength 0 to 1 and copies 1 to 10.".into());
+ }
+ if refmod.strength == 0.0 { continue; }
+ api.add_refmod(handle, Path::new(&refmod.path), refmod.strength, refmod.copies as i32)?;
+ }
Ok(())
}
@@ -938,6 +955,7 @@ fn write_reference_icon(
canvas_width: crate::reference_icons::RENDER_SIZE as i32,
canvas_height: crate::reference_icons::RENDER_SIZE as i32,
reference_paths: Vec::new(),
+ refmods: Vec::new(),
reference_video_ids: Vec::new(),
};
let handle = RequestHandle::new(api)?;
@@ -1279,6 +1297,7 @@ mod ffi {
set_seed: unsafe extern "C" fn(*mut Request, u64) -> i32,
set_model: unsafe extern "C" fn(*mut Request, i32, *const c_char) -> i32,
add_lora: Option<unsafe extern "C" fn(*mut Request, *const c_char, f32) -> i32>,
+ add_refmod: Option<unsafe extern "C" fn(*mut Request, *const c_char, f32, i32) -> i32>,
set_schedule: Option<unsafe extern "C" fn(*mut Request, i32) -> i32>,
add_reference: unsafe extern "C" fn(*mut Request, *const c_char) -> i32,
set_attention: unsafe extern "C" fn(*mut Request, *const c_char) -> i32,
@@ -1377,6 +1396,9 @@ mod ffi {
add_lora: library
.get::<unsafe extern "C" fn(*mut Request, *const c_char, f32) -> i32>(b"slopfab_request_add_lora\0")
.ok().map(|symbol| *symbol),
+ add_refmod: library
+ .get::<unsafe extern "C" fn(*mut Request, *const c_char, f32, i32) -> i32>(b"slopfab_request_add_refmod\0")
+ .ok().map(|symbol| *symbol),
set_schedule: library
.get::<unsafe extern "C" fn(*mut Request, i32) -> i32>(b"slopfab_request_set_schedule\0")
.ok().map(|symbol| *symbol),
@@ -1541,6 +1563,11 @@ mod ffi {
let path = path_cstring(path)?;
self.error(unsafe { add(r, path.as_ptr(), strength) })
}
+ pub fn add_refmod(&self, r: *mut Request, path: &Path, strength: f32, copies: i32) -> Result<(), String> {
+ let add = self.add_refmod.ok_or("This slopfab.dll does not support refmods. Install a build with the refmod API (1.8 or later).")?;
+ let path = path_cstring(path)?;
+ self.error(unsafe { add(r, path.as_ptr(), strength, copies) })
+ }
pub fn set_taomate_schedule(&self, r: *mut Request) -> Result<(), String> {
let set = self.set_schedule.ok_or("This slopfab.dll does not support the TaoMate three-step schedule.")?;
self.error(unsafe { set(r, 1) })
@@ -1673,6 +1700,54 @@ mod ffi {
#[cfg(test)]
mod tests {
use super::*;
+ #[test]
+ fn refmods_attach_for_plans_and_icons_or_report_an_older_dll() {
+ let configuration = Configuration::from_settings(&BTreeMap::new());
+ let api = ffi::Api::load(&configuration.dll_path).unwrap();
+ let root = tempfile::tempdir().unwrap();
+ let file = root.path().join("person.safetensors");
+ let header = serde_json::json!({
+ "__metadata__": { "refmod_meta": serde_json::json!({ "kind": "image", "name": "Person", "_format_version": 4 }).to_string() },
+ "latent": { "dtype": "F32", "shape": [1, 24, 1, 2, 2], "data_offsets": [0, 384] }
+ }).to_string();
+ let mut bytes = (header.len() as u64).to_le_bytes().to_vec();
+ bytes.extend_from_slice(header.as_bytes());
+ for _ in 0..96 { bytes.extend_from_slice(&0.25_f32.to_le_bytes()); }
+ std::fs::write(&file, bytes).unwrap();
+ let mut request = GenerationRequest {
+ job_id: "refmod-test".into(), prompt: "A portrait.".into(), still_image: true, frames: 1,
+ steps: 20, seed: 1, canvas_width: 768, canvas_height: 768,
+ reference_paths: Vec::new(), reference_video_ids: Vec::new(),
+ refmods: vec![RefmodInput { path: file.to_string_lossy().into_owned(), strength: 0.7, copies: 2 }],
+ };
+ let probe = RequestHandle::new(&api).unwrap();
+ let added = api.add_refmod(probe.0, &file, 0.7, 2);
+ if let Err(error) = &added {
+ assert!(error.contains("does not support refmods"), "{error}");
+ eprintln!("Refmod export not yet available in the installed DLL; checked compatibility error.");
+ assert!(configure_request(&api, probe.0, &request, &configuration, ComputePlatform::Vulkan, false).unwrap_err().contains("does not support refmods"));
+ } else {
+ for platform in [ComputePlatform::Cuda13, ComputePlatform::Vulkan] {
+ for still_image in [false, true] {
+ request.still_image = still_image;
+ request.frames = if still_image { 1 } else { 120 };
+ for include_models in [false, true] {
+ let handle = RequestHandle::new(&api).unwrap();
+ configure_request(&api, handle.0, &request, &configuration, platform, include_models).unwrap();
+ let plan = api.resolve(handle.0).unwrap();
+ assert!(plan.aligned_frames >= request.frames);
+ }
+ }
+ }
+ }
+ request.refmods[0].strength = 0.0;
+ std::fs::remove_file(&file).unwrap();
+ let handle = RequestHandle::new(&api).unwrap();
+ configure_request(&api, handle.0, &request, &configuration, ComputePlatform::Vulkan, true).unwrap();
+ request.refmods[0].copies = 11;
+ assert!(configure_request(&api, handle.0, &request, &configuration, ComputePlatform::Vulkan, false).is_err());
+ }
+
#[test]
fn ordered_loras_and_taomate_schedule_reach_cuda_and_vulkan_requests() {
let root = tempfile::tempdir().unwrap();
@@ -1695,6 +1770,7 @@ mod tests {
frames: 120, still_image: false, steps: 20, seed: 1,
canvas_width: 736, canvas_height: 416,
reference_paths: Vec::new(), reference_video_ids: Vec::new(),
+ refmods: Vec::new(),
};
for platform in [ComputePlatform::Cuda13, ComputePlatform::Vulkan] {
for include_models in [false, true] {
@@ -1733,6 +1809,7 @@ mod tests {
frames: 120, still_image: false, steps: 12, seed: 1,
canvas_width: 736, canvas_height: 416,
reference_paths: Vec::new(), reference_video_ids: vec![id.clone()],
+ refmods: Vec::new(),
};
let configuration = Configuration::from_settings(&settings);
let api = ffi::Api::load(&configuration.dll_path)?;
diff --git a/src/components/ProjectWorkspace.test.ts b/src/components/ProjectWorkspace.test.ts
index a15c043..0dbf689 100644
--- a/src/components/ProjectWorkspace.test.ts
+++ b/src/components/ProjectWorkspace.test.ts
@@ -27,7 +27,7 @@ afterEach(() => {
/** Runs `body` with the app believing it is inside the desktop shell, so the
* REAL Tauri import path executes instead of the browser fallback. */
-async function asDesktopApp(importedImage: { name: string; relativePath: string } | Array<{ name: string; relativePath: string }>, body: () => void | Promise<void>) {
+async function asDesktopApp(importedImage: { name: string; relativePath: string } | Array<{ kind?: "image"; name: string; relativePath: string }>, body: () => void | Promise<void>) {
const { invoke } = await import("@tauri-apps/api/core");
(window as unknown as Record<string, unknown>).__TAURI_INTERNALS__ = {};
vi.mocked(invoke).mockResolvedValue(importedImage);
@@ -1176,7 +1176,7 @@ describe("project workspace timecode", () => {
// which is exactly why this shipped.
const config = createProjectConfig({ name: "Ceramic lamp", prompt: "A quiet product film", aspectRatio: "16:9", resolution: "1080p", targetDurationSeconds: 30 });
const onChange = vi.fn();
- await asDesktopApp([{ name: "IMG_4821", relativePath: "references/IMG_4821.jpg" }], async () => {
+ await asDesktopApp([{ kind: "image", name: "IMG_4821", relativePath: "references/IMG_4821.jpg" }], async () => {
const view = render(createElement(ReferencesView, { config, folderPath: "C:\\Ceramic Lamp", onChange }));
fireEvent.click(screen.getByRole("button", { name: /Add a reference/ }));
const dialog = screen.getByRole("dialog", { name: "Add a reference" });
@@ -1185,7 +1185,7 @@ describe("project workspace timecode", () => {
const withReference = onChange.mock.calls[0][0];
view.rerender(createElement(ReferencesView, { config: withReference, folderPath: "C:\\Ceramic Lamp", onChange }));
onChange.mockClear();
- fireEvent.click(screen.getByRole("button", { name: "Add images" }));
+ fireEvent.click(screen.getByRole("button", { name: "Add file" }));
await waitFor(() => expect(onChange).toHaveBeenCalled());
});
diff --git a/src/components/workspace/GeneratorView.tsx b/src/components/workspace/GeneratorView.tsx
index 9269f3a..c8a6dda 100644
--- a/src/components/workspace/GeneratorView.tsx
+++ b/src/components/workspace/GeneratorView.tsx
@@ -2,6 +2,7 @@ import type { GenerationSubmission } from "../../lib/workQueue";
import { loadLoras, subscribeLoras } from "../../lib/loras";
import { refreshDownloadedLoras } from "../../lib/weightDownloads";
import { usableVideoReferences } from "../../lib/project";
+import { referenceRefmodInputs } from "../../lib/project";
import { ChevronDown, Plus, Square, Trash2, WandSparkles } from "lucide-react";
import { invoke } from "@tauri-apps/api/core";
import { useEffect, useMemo, useRef, useState, useSyncExternalStore } from "react";
@@ -311,6 +312,7 @@ export function GeneratorView({ config, folderPath, runtime = null, generationCo
referencePaths: usableReferenceImages(bound)
.map((image) => projectItemPath(folderPath, image) ?? "")
.filter((path) => path.length > 0),
+ refmods: referenceRefmodInputs(folderPath, bound),
referenceVideos: usableVideoReferences(bound).map((reference) => ({
name: reference.name, relativePath: reference.relativePath, sourcePath: reference.sourcePath,
startSeconds: reference.video?.startSeconds ?? 0,
diff --git a/src/components/workspace/ReferencesView.test.tsx b/src/components/workspace/ReferencesView.test.tsx
index d5c928a..53e00fc 100644
--- a/src/components/workspace/ReferencesView.test.tsx
+++ b/src/components/workspace/ReferencesView.test.tsx
@@ -321,8 +321,8 @@ describe("Reference type presets", () => {
it("attaches multiple images to a new product without changing its category or prompt", async () => {
(window as unknown as Record<string, unknown>).__TAURI_INTERNALS__ = {};
vi.mocked(invoke).mockResolvedValue([
- { name: "front", relativePath: "references/front.png" },
- { name: "side", relativePath: "references/side.png" },
+ { kind: "image", name: "front", relativePath: "references/front.png" },
+ { kind: "image", name: "side", relativePath: "references/side.png" },
]);
const state = setup();
@@ -330,10 +330,10 @@ describe("Reference type presets", () => {
const dialog = screen.getByRole("dialog", { name: "Add a reference" });
fireEvent.click(within(dialog).getByRole("button", { name: "Product" }));
fireEvent.click(within(dialog).getByRole("button", { name: "New" }));
- fireEvent.click(screen.getByRole("button", { name: "Add images" }));
+ fireEvent.click(screen.getByRole("button", { name: "Add file" }));
await waitFor(() => expect(state.latest().references[0].images).toHaveLength(2));
- expect(invoke).toHaveBeenCalledWith("choose_reference_images", { folderPath: "C:\\\\project" });
+ expect(invoke).toHaveBeenCalledWith("choose_reference_files", { folderPath: "C:\\\\project" });
expect(state.latest().references[0]).toMatchObject({
kind: "text",
name: "New product",
diff --git a/src/components/workspace/ReferencesView.tsx b/src/components/workspace/ReferencesView.tsx
index f8aeb8b..dbeb4df 100644
--- a/src/components/workspace/ReferencesView.tsx
+++ b/src/components/workspace/ReferencesView.tsx
@@ -1,7 +1,8 @@
-import { BookOpen, ChevronRight, FileText, ImagePlus, Plus, RefreshCw, Search, Trash2, Users, Video, X } from "lucide-react";
+import { BookOpen, ChevronRight, FileText, ImagePlus, Plus, RefreshCw, Search, Trash2, Users, X } from "lucide-react";
import { invoke } from "@tauri-apps/api/core";
import { useEffect, useMemo, useRef, useState, useSyncExternalStore } from "react";
import { isReferenceDescribed, referenceImages, type ProjectConfig, type ProjectReference, type ProjectReferenceImage } from "../../lib/project";
+import { activeReferenceRefmods } from "../../lib/project";
import {
composeLocationPrompt,
locationSelectionFromPrompt,
@@ -39,6 +40,7 @@ const PICKER_TYPES: ReadonlyArray<{ id: ReferenceType; label: string }> = [
* reference, and neither the card nor the inspector does anything with it. */
const referenceKindLabel = (reference: ProjectReference) => {
const count = referenceImages(reference).length;
+ if (reference.refmods?.length) return `${reference.refmods.length} refmod${reference.refmods.length === 1 ? "" : "s"}`;
if (reference.kind === "video") return `Video clip${count ? ` + ${count} image${count === 1 ? "" : "s"}` : ""}`;
if (count === 0) return reference.kind === "audio" ? "Sound" : "Text definition";
return `${count} image${count === 1 ? "" : "s"}${isReferenceDescribed(reference) ? " + text" : ""}`;
@@ -74,7 +76,7 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
const [showDebugIconPrompt, setShowDebugIconPrompt] = useState(false);
useEffect(() => setShowDebugIconPrompt(false), [selectedId, debugEnabled]);
const [importError, setImportError] = useState<string | null>(null);
- const [importingVideo, setImportingVideo] = useState(false);
+ const [importingFiles, setImportingFiles] = useState(false);
const [iconError, setIconError] = useState<string | null>(null);
const [presetDialog, setPresetDialog] = useState(false);
const catalogVisit = useRef(0);
@@ -126,27 +128,6 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
else if (action === "generate") onGenerateBuiltinIcons();
}).catch((reason) => setIconError(String(reason)));
};
- const addImages = async () => {
- if (!selected) return;
- setImportError(null);
- if (!isTauri()) {
- setImportError("Image import is available in the desktop app.");
- return;
- }
- try {
- const imported = await invoke<Array<{ name: string; relativePath: string }>>("choose_reference_images", { folderPath });
- if (imported.length === 0) return;
- const stamp = Date.now();
- const images: ProjectReferenceImage[] = imported.map((image, index) => ({
- id: `ref-image-${stamp}-${index + 1}`,
- name: image.name,
- relativePath: image.relativePath,
- }));
- update(selected.id, { images: [...(selected.images ?? []), ...images] });
- } catch (reason) {
- setImportError(reason instanceof Error ? reason.message : String(reason));
- }
- };
const remove = () => {
if (!selected) return;
onChange({ ...config, references: config.references.filter((ref) => ref.id !== selected.id), generationJobs: config.generationJobs.map((job) => ({
@@ -157,22 +138,32 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
})) });
setSelectedId(config.references.find((ref) => ref.id !== selected.id)?.id);
};
- const addVideo = async () => {
+ const addFiles = async () => {
if (!selected) return;
setImportError(null);
- if (!isTauri()) { setImportError("Video import is available in the desktop app."); return; }
- setImportingVideo(true);
+ if (!isTauri()) { setImportError("File import is available in the desktop app."); return; }
+ setImportingFiles(true);
try {
- const sourcePath = await invoke<string | null>("choose_reference_video", { folderPath });
- if (!sourcePath) return;
- const options = await inspectReferenceVideo(folderPath, sourcePath);
- const video = { startSeconds: 0, ...options };
+ const files = await invoke<Array<{ kind: "image" | "video" | "refmod"; name: string; relativePath?: string | null; sourcePath?: string | null }>>("choose_reference_files", { folderPath });
+ if (!files.length) return;
+ const clip = files.find((file) => file.kind === "video");
+ const video = clip ? { startSeconds: 0, ...await inspectReferenceVideo(folderPath, clip.sourcePath!) } : undefined;
const current = configRef.current;
const target = current.references.find((reference) => reference.id === selected.id);
- if (!target) throw new Error("The reference was removed while the video was importing.");
- update(target.id, { kind: "video", relativePath: null, sourcePath, video, images: referenceImages(target) });
+ if (!target) throw new Error("The reference was removed while the files were importing.");
+ const images: ProjectReferenceImage[] = files.filter((file) => file.kind === "image").map((file) => ({
+ id: `ref-image-${crypto.randomUUID()}`, name: file.name, relativePath: file.relativePath, sourcePath: file.sourcePath,
+ }));
+ const refmods = files.filter((file) => file.kind === "refmod").map((file) => ({
+ id: `refmod-${crypto.randomUUID()}`, name: file.name, relativePath: file.relativePath, sourcePath: file.sourcePath, strength: 1, copies: 1,
+ }));
+ update(target.id, {
+ ...(clip ? { kind: "video", relativePath: null, sourcePath: clip.sourcePath, video } : {}),
+ images: [...(clip ? referenceImages(target) : target.images ?? []), ...images],
+ ...(refmods.length ? { refmods: [...(target.refmods ?? []), ...refmods], iconRelativePath: undefined } : {}),
+ });
} catch (reason) { setImportError(reason instanceof Error ? reason.message : String(reason)); }
- finally { setImportingVideo(false); }
+ finally { setImportingFiles(false); }
};
const createEmptyReference = () => {
addTextReference(pickerType === "custom" ? "Uncategorized" : `New ${referenceTypeLabel(pickerType).toLocaleLowerCase()}`, "", pickerType, pickerSubcategory === "all" ? "" : pickerSubcategory);
@@ -218,6 +209,8 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
asset={{ id: ref.id, kind: "video", name: ref.name, sourcePath: ref.sourcePath, relativePath: ref.relativePath, mimeType: "video/mp4", createdAt: ref.createdAt }}
posterTimeSeconds={0}
/></span>
+ : ref.refmods?.length && ref.iconRelativePath
+ ? <span className="reference-art reference-art--photo"><ReferenceImage folderPath={folderPath} relativePath={ref.iconRelativePath} alt={`${ref.name} icon`} /></span>
: cover
? <span className="reference-art reference-art--photo"><ReferenceImage folderPath={folderPath} relativePath={cover.relativePath} sourcePath={cover.sourcePath} alt={cover.name} /></span>
: ref.iconRelativePath
@@ -225,9 +218,9 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
: presetIcon
? <span className="reference-art reference-art--photo"><PresetIcon preset={presetIcon} /></span>
: <span className="reference-copy-art"><FileText size={26} /></span>}
- <span className="reference-card__body"><span><b>{ref.name}</b>{(images.length > 0 || ref.kind === "video" || ref.kind === "audio") && <small>{referenceKindLabel(ref)}</small>}</span>{isReferenceDescribed(ref)
+ <span className="reference-card__body"><span><b>{ref.name}</b>{(images.length > 0 || ref.refmods?.length || ref.kind === "video" || ref.kind === "audio") && <small>{referenceKindLabel(ref)}</small>}</span>{isReferenceDescribed(ref)
? <p>{ref.description}</p>
- : <p className="reference-card__incomplete">{ref.kind === "video" ? "The clip is sent as a reference. Add a prompt to describe what to keep." : images.length > 0
+ : <p className="reference-card__incomplete">{ref.refmods?.length ? "Pre-encoded reference. Add a prompt to describe what to keep." : ref.kind === "video" ? "The clip is sent as a reference. Add a prompt to describe what to keep." : images.length > 0
? "Not described yet — the picture is sent, but nothing tells the engine what to keep."
: "Not described yet — it won’t be used until you add a definition."}</p>}<span className="use-tags"><i>{referenceTypeLabel(referenceType(ref))}</i></span></span>
</button>;
@@ -273,7 +266,7 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
to print the file's path over the thumbnail, which is neither
what the reference IS nor anything the user acts on. */}
{selectedImages.length === 0 && !selected.iconRelativePath && !selectedPresetIcon && <em>{referenceKindLabel(selected)}</em>}
- {selectedImages.length === 0 && onRegenerateIcon && <button
+ {selectedImages.length === 0 && !selected.refmods?.length && onRegenerateIcon && <button
type="button"
className="reference-icon-refresh"
aria-label="Regenerate reference icon"
@@ -289,10 +282,30 @@ export function ReferencesView({ config, folderPath, onChange, onRegenerateIcon,
<div className="reference-fields">
<label><span>Prompt</span><textarea value={selected.description} placeholder="Describe what should stay consistent — the traits, materials, colours, or wardrobe Slopus should preserve across shots." onChange={(event) => update(selected.id, { description: event.target.value, content: event.target.value || null, subcategory: selectedSubcategory })} /></label>
<div className="reference-fields__actions">
- <button className="secondary-button" onClick={() => void addImages()}><ImagePlus size={16} /> Add images</button>
- <button className="secondary-button" disabled={importingVideo} onClick={() => void addVideo()}><Video size={16} /> {importingVideo ? "Importing video…" : selected.kind === "video" ? "Replace video" : "Add video"}</button>
+ <button className="secondary-button" disabled={importingFiles} onClick={() => void addFiles()}><ImagePlus size={16} /> {importingFiles ? "Adding files…" : "Add file"}</button>
</div>
</div>
+ {Boolean(selected.refmods?.length) && <section className="reference-refmods" aria-label="Refmod attachments">
+ <h3>Refmods</h3>
+ <p>Requires a Ref2VA generator. Strength 0 disables a refmod. More copies use more GPU memory.</p>
+ {selected.refmods!.map((refmod) => <div key={refmod.id}>
+ <b>{refmod.name}</b>
+ <label>Strength<input aria-label={`${refmod.name} strength`} type="number" min={0} max={1} step={0.05} value={refmod.strength} onChange={(event) => {
+ const strength = Number(event.target.value);
+ if (event.target.value && Number.isFinite(strength) && strength >= 0 && strength <= 1) update(selected.id, { iconRelativePath: undefined, refmods: selected.refmods!.map((entry) => entry.id === refmod.id ? { ...entry, strength } : entry) });
+ }} /></label>
+ <label>Copies<input aria-label={`${refmod.name} copies`} type="number" min={1} max={10} step={1} value={refmod.copies} onChange={(event) => {
+ const copies = Number(event.target.value);
+ if (Number.isInteger(copies) && copies >= 1 && copies <= 10) update(selected.id, { iconRelativePath: undefined, refmods: selected.refmods!.map((entry) => entry.id === refmod.id ? { ...entry, copies } : entry) });
+ }} /></label>
+ <button className="secondary-button" onClick={() => update(selected.id, { iconRelativePath: undefined, refmods: selected.refmods!.filter((entry) => entry.id !== refmod.id) })}><Trash2 size={16} /> Remove {refmod.name}</button>
+ </div>)}
+ {selected.iconRelativePath && <ReferenceImage folderPath={folderPath} relativePath={selected.iconRelativePath} alt={`${selected.name} refmod icon`} />}
+ {onRegenerateIcon && <button className="secondary-button" disabled={!activeReferenceRefmods(selected).length || pendingIconIds.has(selected.id)} onClick={() => {
+ setIconError(null);
+ try { onRegenerateIcon(selected.id); } catch (reason) { setIconError(String(reason)); }
+ }}><RefreshCw size={16} /> {pendingIconIds.has(selected.id) ? "Rendering icon…" : "Render icon with refmod"}</button>}
+ </section>}
{selectedLocation && <section className="reference-location-settings" aria-label="Location settings">
{LOCATION_SETTING_GROUPS.map((group) => <label key={group.id} className="reference-location-settings__group">
<span>{group.label}</span>
diff --git a/src/components/workspace/ReferencesView.video.test.tsx b/src/components/workspace/ReferencesView.video.test.tsx
index c2479e7..e2ae696 100644
--- a/src/components/workspace/ReferencesView.video.test.tsx
+++ b/src/components/workspace/ReferencesView.video.test.tsx
@@ -14,9 +14,43 @@ vi.mock("./ReferenceVideo", () => ({ ReferenceVideo: () => <div>Video preview</d
vi.mock("./MediaThumbnail", () => ({ MediaThumbnail: ({ asset, posterTimeSeconds }: { asset: { sourcePath: string; name: string }; posterTimeSeconds: number }) => <img alt={`Frame from ${asset.name}`} src={`test:${asset.sourcePath}:${posterTimeSeconds}`} /> }));
afterEach(() => { cleanup(); vi.resetAllMocks(); delete (window as unknown as Record<string, unknown>).__TAURI_INTERNALS__; });
+it("attaches mixed files through Add file and renders an icon with an enabled refmod", async () => {
+ Object.assign(window, { __TAURI_INTERNALS__: {} });
+ vi.mocked(invoke).mockResolvedValue([
+ { kind: "image", name: "photo", relativePath: "references/photo.png" },
+ { kind: "video", name: "motion", sourcePath: "D:/motion.mp4" },
+ { kind: "refmod", name: "person.safetensors", sourcePath: "D:/person.safetensors" },
+ ]);
+ vi.mocked(inspectReferenceVideo).mockResolvedValue({ durationSeconds: 4, includeAudio: false });
+ const renderIcon = vi.fn();
+ let latest = createProjectConfig({ name: "Mixed", prompt: "", aspectRatio: "16:9", resolution: "416p", targetDurationSeconds: 10 });
+ latest.references = [{ id: "ref", name: "Uncategorized", description: "", kind: "text", intendedUse: [], createdAt: latest.createdAt }];
+ function Harness() {
+ const [config, setConfig] = useState(latest); latest = config;
+ return <ReferencesView folderPath="C:/mixed" config={config} onChange={setConfig} onRegenerateIcon={renderIcon} />;
+ }
+ render(<Harness />);
+ expect(screen.queryByRole("button", { name: "Add images" })).not.toBeInTheDocument();
+ expect(screen.queryByRole("button", { name: "Add video" })).not.toBeInTheDocument();
+ fireEvent.click(screen.getByRole("button", { name: "Add file" }));
+ const icon = await screen.findByRole("button", { name: "Render icon with refmod" });
+ expect(parseProjectConfig(latest).references[0]).toMatchObject({ kind: "video", sourcePath: "D:/motion.mp4", images: [{ relativePath: "references/photo.png" }], refmods: [{ sourcePath: "D:/person.safetensors", strength: 1, copies: 1 }] });
+ fireEvent.change(screen.getByLabelText("person.safetensors strength"), { target: { value: "0.7" } });
+ fireEvent.change(screen.getByLabelText("person.safetensors copies"), { target: { value: "2" } });
+ fireEvent.click(icon);
+ expect(renderIcon).toHaveBeenCalledWith("ref");
+ expect(latest.references[0].refmods![0]).toMatchObject({ strength: 0.7, copies: 2 });
+ fireEvent.change(screen.getByLabelText("person.safetensors strength"), { target: { value: "0" } });
+ expect(icon).toBeDisabled();
+ fireEvent.click(screen.getByRole("button", { name: "Remove person.safetensors" }));
+ expect(latest.references[0].refmods).toEqual([]);
+ expect(latest.references[0].images).toHaveLength(1);
+ expect(latest.references[0].kind).toBe("video");
+});
+
it("imports a video, edits its saved trim and soundtrack, then removes its attachment", async () => {
Object.assign(window, { __TAURI_INTERNALS__: {} });
- vi.mocked(invoke).mockResolvedValue("C:/motion.mp4");
+ vi.mocked(invoke).mockResolvedValue([{ kind: "video", name: "motion.mp4", sourcePath: "C:/motion.mp4" }]);
vi.mocked(inspectReferenceVideo).mockResolvedValue({ durationSeconds: 10, includeAudio: true });
let latest = createProjectConfig({ name: "Video", prompt: "", aspectRatio: "16:9", resolution: "1080p", targetDurationSeconds: 20 });
function Harness() {
@@ -33,9 +67,9 @@ it("imports a video, edits its saved trim and soundtrack, then removes its attac
expect(within(picker).queryByText("No options match that search.")).not.toBeInTheDocument();
fireEvent.click(within(picker).getByRole("button", { name: "New" }));
expect(latest.references[0]).toMatchObject({ name: "Uncategorized", kind: "text", intendedUse: [] });
- fireEvent.click(screen.getByRole("button", { name: "Add video" }));
+ fireEvent.click(screen.getByRole("button", { name: "Add file" }));
await screen.findByText("Video preview");
- expect(invoke).toHaveBeenCalledWith("choose_reference_video", { folderPath: "C:/project" });
+ expect(invoke).toHaveBeenCalledWith("choose_reference_files", { folderPath: "C:/project" });
expect(screen.getByRole("img", { name: "Frame from Uncategorized" })).toHaveAttribute("src", "test:C:/motion.mp4:0");
fireEvent.change(screen.getByLabelText("Clip start (seconds)"), { target: { value: "2" } });
fireEvent.blur(screen.getByLabelText("Clip start (seconds)"));
@@ -46,8 +80,8 @@ it("imports a video, edits its saved trim and soundtrack, then removes its attac
name: "Uncategorized", intendedUse: [], kind: "video", sourcePath: "C:/motion.mp4", video: { startSeconds: 2, durationSeconds: 4, includeAudio: false },
});
expect(screen.getByRole("img", { name: "Frame from Uncategorized" })).toHaveAttribute("src", "test:C:/motion.mp4:0");
- vi.mocked(invoke).mockResolvedValueOnce("C:/replacement.mp4");
- fireEvent.click(screen.getByRole("button", { name: "Replace video" }));
+ vi.mocked(invoke).mockResolvedValueOnce([{ kind: "video", name: "replacement.mp4", sourcePath: "C:/replacement.mp4" }]);
+ fireEvent.click(screen.getByRole("button", { name: "Add file" }));
await waitFor(() => expect(screen.getByRole("img", { name: "Frame from Uncategorized" })).toHaveAttribute("src", "test:C:/replacement.mp4:0"));
fireEvent.click(screen.getByRole("button", { name: "Remove video" }));
await waitFor(() => expect(latest.references[0].kind).toBe("text"));
diff --git a/src/lib/project.refmods.test.ts b/src/lib/project.refmods.test.ts
new file mode 100644
index 0000000..64e29c7
--- /dev/null
+++ b/src/lib/project.refmods.test.ts
@@ -0,0 +1,37 @@
+import { expect, it } from "vitest";
+import { activeReferenceRefmods, compileMiniMaxH3Prompt, createProjectConfig, isReferenceUsable, parseProjectConfig, projectReferenceSchema, referenceRefmodInputs, sceneGenerationSnapshot, usableReferenceImages } from "./project";
+
+const reference = () => projectReferenceSchema.parse({
+ id: "subject", name: "Uncategorized", description: "", kind: "text", intendedUse: [], createdAt: "2026-09-13T00:00:00.000Z",
+ refmods: [{ id: "latent", name: "person.safetensors", sourcePath: "D:/person.safetensors" }],
+});
+
+it("round-trips refmod attachments with defaults without inventing image inputs", () => {
+ const config = createProjectConfig({ name: "Refmod", prompt: "A scene", aspectRatio: "16:9", resolution: "416p", targetDurationSeconds: 10 });
+ config.references = [reference()];
+ expect(parseProjectConfig(JSON.parse(JSON.stringify(config)))).toEqual(config);
+ expect(reference().refmods![0]).toMatchObject({ strength: 1, copies: 1 });
+ expect(isReferenceUsable(reference())).toBe(true);
+ expect(usableReferenceImages(config.references)).toEqual([]);
+ expect(referenceRefmodInputs("C:/project", config.references)).toEqual([{ path: "D:/person.safetensors", strength: 1, copies: 1 }]);
+ const prompt = compileMiniMaxH3Prompt("A walking person", config.references);
+ expect(prompt).not.toContain("<Picture");
+ expect(prompt).not.toContain("person.safetensors");
+ expect(prompt).toContain("pre-encoded reference");
+});
+
+it("omits disabled refmods and includes their settings in generation snapshots", () => {
+ const ref = reference();
+ ref.refmods![0].strength = 0;
+ expect(activeReferenceRefmods(ref)).toEqual([]);
+ expect(isReferenceUsable(ref)).toBe(false);
+ const config = createProjectConfig({ name: "Refmod", prompt: "A scene", aspectRatio: "16:9", resolution: "416p", targetDurationSeconds: 10 });
+ const request = { prompt: "A scene", frames: 120, steps: 20, seed: 1, canvasWidth: 736, canvasHeight: 416, referencePaths: [], refmods: [{ path: "D:/ref.safetensors", strength: 0.7, copies: 2 }] };
+ const before = sceneGenerationSnapshot(config.generationJobs[0], request);
+ expect(sceneGenerationSnapshot(config.generationJobs[0], { ...request, refmods: [{ ...request.refmods[0], copies: 3 }] })).not.toBe(before);
+});
+
+it.each([{ strength: -1 }, { strength: 1.1 }, { strength: Infinity }, { copies: 0 }, { copies: 11 }, { copies: 1.5 }, { sourcePath: "../escape.safetensors" }, { sourcePath: null }])("rejects invalid refmod settings %j", (patch) => {
+ const ref = reference();
+ expect(() => projectReferenceSchema.parse({ ...ref, refmods: [{ ...ref.refmods![0], ...patch }] })).toThrow();
+});
diff --git a/src/lib/project.ts b/src/lib/project.ts
index 89b0b1d..3ba2d50 100644
--- a/src/lib/project.ts
+++ b/src/lib/project.ts
@@ -252,6 +252,18 @@ export const projectReferenceImageSchema = z.object({
}
});
+export const projectReferenceRefmodSchema = z.object({
+ id: idSchema,
+ name: z.string().min(1),
+ relativePath: projectRelativePathSchema.nullish(),
+ sourcePath: externalPathSchema.nullish(),
+ strength: z.number().finite().min(0).max(1).default(1),
+ copies: z.number().int().min(1).max(10).default(1),
+}).superRefine((refmod, context) => {
+ checkOneLocation(refmod, context, `Refmod '${refmod.id}'`);
+ if (!refmod.relativePath && !refmod.sourcePath) context.addIssue({ code: z.ZodIssueCode.custom, path: ["sourcePath"], message: "A refmod needs a file." });
+});
+
export const projectReferenceSchema = z.object({
id: idSchema,
// `image` remains readable for legacy projects. New pictures live in
@@ -269,6 +281,7 @@ export const projectReferenceSchema = z.object({
// written definition can carry any number of images, and an image-only
// reference is simply a definition whose description is still blank.
images: z.array(projectReferenceImageSchema).optional(),
+ refmods: z.array(projectReferenceRefmodSchema).optional(),
video: z.object({
startSeconds: z.number().finite().min(0),
durationSeconds: z.number().finite().min(2).max(15),
@@ -285,6 +298,11 @@ export const projectReferenceSchema = z.object({
context.addIssue({ code: z.ZodIssueCode.custom, path: ["video"], message: "Clip settings require a video reference." });
}
const imageIds = new Set<string>();
+ const refmodIds = new Set<string>();
+ for (const refmod of reference.refmods ?? []) {
+ if (refmodIds.has(refmod.id)) context.addIssue({ code: z.ZodIssueCode.custom, path: ["refmods"], message: `Duplicate refmod id '${refmod.id}'.` });
+ refmodIds.add(refmod.id);
+ }
for (const image of reference.images ?? []) {
if (imageIds.has(image.id)) {
context.addIssue({ code: z.ZodIssueCode.custom, path: ["images"], message: `Reference '${reference.id}' has two images with the id '${image.id}'.` });
@@ -556,6 +574,7 @@ export const clipLook = (clip: TimelineClip): ClipLook => clip.look ?? DEFAULT_C
export const clipTransition = (clip: TimelineClip): ClipTransition => clip.transition ?? DEFAULT_CLIP_TRANSITION;
export type GenerationBrief = z.infer<typeof generationBriefSchema>;
export type ProjectReferenceImage = z.infer<typeof projectReferenceImageSchema>;
+export type ProjectReferenceRefmod = z.infer<typeof projectReferenceRefmodSchema>;
export type ProjectReference = z.infer<typeof projectReferenceSchema>;
export type GenerationJob = z.infer<typeof generationJobSchema>;
@@ -644,9 +663,19 @@ export function isReferenceDescribed(reference: ProjectReference): boolean {
* An image qualifies on its file alone: it can be sent to the engine and cited
* as <Picture N> with nothing written about it. */
export function isReferenceUsable(reference: ProjectReference): boolean {
- return referenceImages(reference).length > 0 || (reference.kind === "video" && Boolean(reference.sourcePath || reference.relativePath)) || isReferenceDescribed(reference);
+ return referenceImages(reference).length > 0 || (reference.kind === "video" && Boolean(reference.sourcePath || reference.relativePath)) || activeReferenceRefmods(reference).length > 0 || isReferenceDescribed(reference);
}
+export const activeReferenceRefmods = (reference: ProjectReference): ProjectReferenceRefmod[] =>
+ (reference.refmods ?? []).filter((refmod) => refmod.strength > 0);
+
+/** Refmods follow ordinary references, in attachment order. They never consume
+ * Picture/Video labels or run a reference encoder. */
+export const referenceRefmodInputs = (folderPath: string, references: ProjectReference[]) =>
+ references.flatMap(activeReferenceRefmods).map((refmod) => ({
+ path: projectItemPath(folderPath, refmod)!, strength: refmod.strength, copies: refmod.copies,
+ }));
+
/** Reference guide §2.1: `<Subject N>` is VISIBLE content. A reference tagged
* only "audio" is a note about sound, so compiling it into subject_definitions
* announced a soundtrack as something on screen ("<Subject 1> is Score idea.
@@ -873,6 +902,7 @@ export interface SceneGenerationInput {
canvasHeight: number;
referencePaths: readonly string[];
referenceVideos?: readonly { name: string; relativePath?: string | null; sourcePath?: string | null; startSeconds: number; durationSeconds?: number; includeAudio: boolean }[];
+ refmods?: readonly { path: string; strength: number; copies: number }[];
previousSceneId?: string;
}
@@ -894,6 +924,7 @@ export function sceneGenerationSnapshot(job: GenerationJob, input: SceneGenerati
canvasHeight: input.canvasHeight,
referencePaths: input.referencePaths,
...(input.referenceVideos?.length ? { referenceVideos: input.referenceVideos } : {}),
+ ...(input.refmods?.length ? { refmods: input.refmods } : {}),
...(input.previousSceneId ? { previousSceneId: input.previousSceneId } : {}),
shots,
});
@@ -1088,6 +1119,9 @@ export function compileScenePromptSegments(scene: ScenePrompt, references: Proje
const video = videoNumbers.get(reference.id);
const label = referenceLabel(reference, index);
const lead = index === 0 ? "" : "\n";
+ if (!detail && pictures.length === 0 && !video && activeReferenceRefmods(reference).length) {
+ return [frame(`${lead}${label} uses the attached pre-encoded reference.`)];
+ }
if (pictures.length === 0 && !video) return [frame(`${lead}${label}: `), own(detail), ...addedStop(detail)];
const pictureList = listOf([...pictures.map((picture) => `<Picture ${picture}>`), ...(video ? [`<Video ${video}>`] : [])]);
return [
diff --git a/src/lib/referenceIconWork.test.ts b/src/lib/referenceIconWork.test.ts
index 92cf3e5..9276bd2 100644
--- a/src/lib/referenceIconWork.test.ts
+++ b/src/lib/referenceIconWork.test.ts
@@ -56,6 +56,22 @@ beforeEach(() => {
afterEach(() => {
REFERENCE_PRESETS.forEach((preset) => { preset.icon = originalPresetIcons.get(preset.id); }); stop?.(); vi.restoreAllMocks(); vi.useRealTimers(); delete (window as any).__TAURI_INTERNALS__; });
+it("renders a refmod icon without a description and rejects a result after its settings change", async () => {
+ const ref = { ...reference("latent"), description: "", refmods: [{ id: "r", name: "Person", sourcePath: "D:/person.safetensors", strength: 0.7, copies: 2 }] };
+ const { queue, session } = setup([ref]);
+ queue.regenerateReferenceIcon(session, "latent");
+ await flush();
+ expect(requests()[0]).toMatchObject({ stillImage: true, referencePaths: [], refmods: [{ path: "D:/person.safetensors", strength: 0.7, copies: 2 }] });
+ session.update((config) => ({ ...config, references: config.references.map((entry) => ({ ...entry, refmods: entry.refmods!.map((latent) => ({ ...latent, copies: 3 })) })) }));
+ await finish(requests()[0].jobId);
+ expect(session.getSnapshot().config.references[0].iconRelativePath).toBeUndefined();
+ queue.regenerateReferenceIcon(session, "latent");
+ await flush();
+ expect(requests()[1].refmods![0].copies).toBe(3);
+ await finish(requests()[1].jobId);
+ expect(session.getSnapshot().config.references[0].iconRelativePath).toContain("references/icons/");
+});
+
it("waits for batch confirmation without blocking video generation", async () => {
saveReferenceIconAutomation("ask");
const { queue, session } = setup([reference("hero"), reference("place", "location")]);
diff --git a/src/lib/referenceIconWork.ts b/src/lib/referenceIconWork.ts
index 68be7a3..12c06b8 100644
--- a/src/lib/referenceIconWork.ts
+++ b/src/lib/referenceIconWork.ts
@@ -6,6 +6,7 @@ import { referenceImages, type ProjectReference } from "./project";
import { needsReferenceIcon, referenceIconPrompt, REFERENCE_ICON_RENDER_SIZE, REFERENCE_ICON_STEPS } from "./referenceIcons";
import { cancelSlopfabGeneration, enqueueSlopfabGeneration, saveReferenceIcon } from "./runtime";
import { withEngineSettings } from "./settings";
+import { activeReferenceRefmods, referenceRefmodInputs } from "./project";
import { loadReferenceIconAutomation, saveReferenceIconAutomation, subscribeReferenceIconAutomation } from "./referenceIconSettings";
import type { WorkItem } from "./workQueue";
import { hasPresetIcon, REFERENCE_PRESETS, type ReferencePreset } from "./reference-presets";
@@ -23,6 +24,7 @@ interface IconTask {
status: TaskStatus;
nativeId?: string;
submitted: boolean;
+ refmodsSnapshot?: string;
cancelled: boolean;
force?: boolean;
approved?: boolean;
@@ -142,7 +144,8 @@ export class ReferenceIconWork {
}
private shouldGenerate(reference: ProjectReference, task: { force?: boolean }) {
- return task.force ? Boolean(reference.description.trim()) && referenceImages(reference).length === 0 : needsReferenceIcon(reference);
+ if (reference.refmods?.length && !activeReferenceRefmods(reference).length) return false;
+ return task.force ? Boolean(reference.description.trim() || activeReferenceRefmods(reference).length) && (referenceImages(reference).length === 0 || activeReferenceRefmods(reference).length > 0) : needsReferenceIcon(reference);
}
isPending(session: ProjectSession, referenceId: string) {
@@ -237,12 +240,14 @@ export class ReferenceIconWork {
const reference = this.referenceFor(task);
if (!reference || !this.shouldGenerate(reference, task)) { task.status = "skipped"; return; }
task.prompt = referenceIconPrompt(reference);
+ task.refmodsSnapshot = JSON.stringify(activeReferenceRefmods(reference));
task.name = reference.name;
task.nativeId = `icon-${crypto.randomUUID()}`;
const done = new Promise<void>((resolve) => { task.finish = resolve; });
await enqueueSlopfabGeneration({
jobId: task.nativeId, prompt: task.prompt, stillImage: true, frames: 1,
steps: REFERENCE_ICON_STEPS, seed: -1, canvasWidth: REFERENCE_ICON_RENDER_SIZE, canvasHeight: REFERENCE_ICON_RENDER_SIZE, referencePaths: [],
+ ...(activeReferenceRefmods(reference).length ? { refmods: referenceRefmodInputs(task.session.record.folderPath, [reference]) } : {}),
}, withEngineSettings(task.session.getSnapshot().config));
task.submitted = true;
if (task.cancelled) await this.cancelNative(task);
@@ -293,7 +298,7 @@ export class ReferenceIconWork {
try {
const reference = this.referenceFor(task);
if (task.cancelled) { task.status = "cancelled"; return; }
- if (!reference || !this.shouldGenerate(reference, task) || referenceIconPrompt(reference) !== task.prompt) { task.status = "skipped"; return; }
+ if (!reference || !this.shouldGenerate(reference, task) || referenceIconPrompt(reference) !== task.prompt || JSON.stringify(activeReferenceRefmods(reference)) !== task.refmodsSnapshot) { task.status = "skipped"; return; }
if (task.preset) {
await saveBuiltinIcon(task.preset.id, task.nativeId!);
task.status = "completed";
@@ -302,7 +307,7 @@ export class ReferenceIconWork {
const iconRelativePath = await saveReferenceIcon(task.session.record.folderPath, task.nativeId!);
let attached = false;
task.session.update((current) => ({ ...current, references: current.references.map((reference) => {
- if (reference.id !== task.referenceId || !this.shouldGenerate(reference, task) || referenceIconPrompt(reference) !== task.prompt) return reference;
+ if (reference.id !== task.referenceId || !this.shouldGenerate(reference, task) || referenceIconPrompt(reference) !== task.prompt || JSON.stringify(activeReferenceRefmods(reference)) !== task.refmodsSnapshot) return reference;
attached = true;
return { ...reference, iconRelativePath };
}) }));
diff --git a/src/lib/runtime.ts b/src/lib/runtime.ts
index 4e9214a..34b2c47 100644
--- a/src/lib/runtime.ts
+++ b/src/lib/runtime.ts
@@ -89,6 +89,7 @@ export interface SlopfabGenerationRequest {
canvasWidth: number;
canvasHeight: number;
referencePaths: string[];
+ refmods?: Array<{ path: string; strength: number; copies: number }>;
referenceVideos?: Array<{ name: string; relativePath?: string | null; sourcePath?: string | null; startSeconds: number; durationSeconds?: number; includeAudio: boolean }>;
/** Temporary native handles, populated by the work queue and never saved. */
referenceVideoIds?: string[];
diff --git a/src/styles/references.css b/src/styles/references.css
index 7b70100..1c7dce8 100644
--- a/src/styles/references.css
+++ b/src/styles/references.css
@@ -200,6 +200,14 @@
height: 100%;
}
+.reference-refmods, .reference-refmods > div { display: grid; gap: var(--space-2); }
+.reference-refmods { padding: var(--space-4); border-top: 1px solid var(--line); }
+.reference-refmods h3, .reference-refmods p { margin: 0; }
+.reference-refmods p { color: var(--text-muted); font-size: var(--text-sm); }
+.reference-refmods label { display: flex; align-items: center; justify-content: space-between; gap: var(--space-2); }
+.reference-refmods input { width: 88px; min-height: var(--field-height); padding: var(--space-2); color: inherit; background: var(--panel); border: 1px solid var(--line); border-radius: var(--radius-sm); }
+.reference-refmods > img { width: 100%; max-height: 256px; object-fit: contain; }
+
.reference-detail-art--photo img {
width: 100%;
height: 100%;