Skip to content

AI media harness for planning, generating, and editing locally

TypeScript

Document refmod export and add a real-VAE export test

Bittia6763133def9
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 1595d48..6f518c3 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -5,6 +5,7 @@ This alpha update adds DMAD LoRA support, resumable model downloads, and improve
## What's new
- Continue scenes from a selected scene's saved latents, with adjustable overlap and a choice of beginning or end. This replaces Previous scene in First & last frame.
+- Export any reference with images, a video or a sound as a refmod `.safetensors` file from References. Exported refmods can be attached to other references and projects.
- Copying or cutting prompts preserves reference chips when pasted into other prompt fields.
- Choose Make Primary on an image version to show it in the main image bar and use it when creating scenes or references from that family.
- Images imported through Timeline or References, including selected video reference frames, appear in the image bar and open for editing.
diff --git a/docs/refmods.md b/docs/refmods.md
index 9a2fab5..694dec8 100644
--- a/docs/refmods.md
+++ b/docs/refmods.md
@@ -30,4 +30,29 @@ Refmods require a Ref2VA generator and a SlopFab DLL exposing the 1.8 refmod API
Both CUDA and Vulkan use the same attachment path. SlopFab validates the tensors
and metadata, reporting incompatible files or bundles. Older DLLs remain usable
without refmods and give an explicit compatibility error when an enabled refmod
-is submitted. Slopus does not encode, train, or optimize refmods.
+is submitted. Slopus does not train, pool, or optimize refmods.
+
+## Exporting a reference as a refmod
+
+Select a reference in References and choose **Export refmod** in the command
+bar, or **Export as refmod…** in its context menu, then pick where to save the
+`.safetensors` file. The export encodes the reference's own media with the
+selected generator's video and audio VAEs: up to nine images, its video clip
+(using the saved trim, with the soundtrack when **Include sound** is on), or
+its sound clip. Text-only references and references that already use refmods
+cannot be exported. The prompt is stored only as the file's description.
+
+One image, a silent clip or a sound gives a standalone refmod. Several items
+give a version-5 bundle: images first, then the clip and its soundtrack.
+Frames-mode video references export their selected stills. Latents are saved
+as F32 at SlopFab's native reference size (768-pixel short edge), without
+pooling or refinement.
+
+The export is a work queue item, so it waits behind running generations. It
+cannot be cancelled once encoding starts. It needs SlopFab API 1.18, a
+floating-point video VAE for images and video, and a floating-point audio VAE
+for sound; no transformer or text encoder is loaded. With a LAN worker
+selected, the worker encodes and the file is saved on this computer. Only the
+VAEs and the reference's media are sent to it.
+
+Attach the exported file to any reference with **Add file**.
diff --git a/docs/slopfab-compatibility.md b/docs/slopfab-compatibility.md
index 18919c9..d9c6828 100644
--- a/docs/slopfab-compatibility.md
+++ b/docs/slopfab-compatibility.md
@@ -1,5 +1,25 @@
# Slopfab compatibility
+## API 1.18: refmod export
+
+The bundled runtime comes from SlopFab commit `9d45da8`, built with
+`SLOPFAB_WITH_FFMPEG=OFF`. API 1.18 adds `slopfab_export_refmod`, which encodes
+a request's raw image, video and audio references into a refmod file using only
+the VAEs. API 1.17 added version-5 refmod bundles, which `add_refmod` loads.
+
+Slopus calls the export from the work queue for **Export refmod** in References
+(see [refmods](refmods.md)). It reuses the existing reference image paths,
+prepared video handles and sound PCM, so video and audio are still decoded with
+mp4box.js/WebCodecs. Both backends are supported, locally or through the
+worker's `/v1/refmods` route. Older runtimes keep working and report that API
+1.18 is needed; an older worker asks to be updated.
+
+Tests cover the missing-media, missing-VAE and older-runtime errors, the worker
+round trip against a stub, queue ordering and the References UI. An ignored
+test (`SLOPUS_E2E_WEIGHTS`) encodes an image, a clip and a sound with real VAEs
+and reloads the resulting bundle; it was run once locally. Generation quality
+with exported refmods was not evaluated.
+
## API 1.16: DMAD
The bundled runtime comes from SlopFab commit `017d2b6`, built with
diff --git a/src-tauri/core/src/slopfab/tests.rs b/src-tauri/core/src/slopfab/tests.rs
index 39f77ba..bc8ba00 100644
--- a/src-tauri/core/src/slopfab/tests.rs
+++ b/src-tauri/core/src/slopfab/tests.rs
@@ -672,6 +672,51 @@ fn refmod_export_needs_raw_media_and_vaes_or_reports_an_older_dll() {
references.release_reference_audios(&[audio]).unwrap();
}
+/// Encodes real media on the GPU. Needs SLOPUS_E2E_WEIGHTS: a folder holding
+/// floating-point H3 video and audio VAEs.
+#[test]
+#[ignore]
+fn refmod_export_encodes_an_image_a_clip_and_a_sound_into_a_loadable_bundle() {
+ let weights = std::path::PathBuf::from(std::env::var("SLOPUS_E2E_WEIGHTS").expect("SLOPUS_E2E_WEIGHTS"));
+ let find = |part: &str| std::fs::read_dir(&weights).unwrap().filter_map(Result::ok).map(|entry| entry.path())
+ .find(|path| path.to_string_lossy().contains(part) && path.extension().is_some_and(|ext| ext == "safetensors"))
+ .unwrap_or_else(|| panic!("no {part} weights")).to_string_lossy().into_owned();
+ let options = BTreeMap::from([
+ ("videoVae".to_string(), ProviderOption::String(find("video_vae_fp16"))),
+ ("audioVae".to_string(), ProviderOption::String(find("audio_vae_fp32"))),
+ ]);
+ let settings = BTreeMap::from([("slopfab".to_string(), ProviderSetting { enabled: true, model: None, options })]);
+ let root = tempfile::tempdir().unwrap();
+ let picture = root.path().join("hero.png");
+ image::RgbImage::from_fn(256, 384, |x, y| image::Rgb([x as u8, y as u8, 128])).save(&picture).unwrap();
+ let references = ReferenceVideos::default();
+ let video = references.create_reference_video(2.0, &settings).unwrap();
+ for frame in 0..48_u8 {
+ references.append_reference_video(&video, &vec![frame * 5; 320 * 192 * 4], 320, 192, f64::from(frame) / 24.0).unwrap();
+ }
+ let tone: Vec<u8> = (0..32_000 * 2).flat_map(|index| ((index as f32 * 0.05).sin() * 0.25).to_le_bytes()).collect();
+ let audio = references.create_reference_audio(&tone, 1, 32_000).unwrap();
+ let request = GenerationRequest {
+ job_id: "refmod-export-e2e".into(),
+ reference_paths: vec![picture.to_string_lossy().into_owned()],
+ reference_video_ids: vec![video],
+ reference_audio_ids: vec![audio],
+ ..Default::default()
+ };
+ let output = root.path().join("hero.safetensors");
+ export_refmod(&request, &settings, &references, &output, "Hero", "A traveler").unwrap();
+ let bytes = std::fs::read(&output).unwrap();
+ let length = u64::from_le_bytes(bytes[..8].try_into().unwrap()) as usize;
+ let header: serde_json::Value = serde_json::from_slice(&bytes[8..8 + length]).unwrap();
+ let meta: serde_json::Value = serde_json::from_str(header["__metadata__"]["refmod_meta"].as_str().unwrap()).unwrap();
+ assert_eq!((meta["kind"].as_str(), meta["name"].as_str()), (Some("bundle"), Some("Hero")));
+ let kinds: Vec<_> = meta["members"].as_array().unwrap().iter().map(|member| member["kind"].as_str().unwrap()).collect();
+ assert_eq!(kinds, ["image", "video", "audio"]);
+ assert_eq!(header["ref_0"]["shape"], serde_json::json!([1, 24, 1, 72, 48]));
+ let api = ffi::Api::load(&Configuration::from_settings(&settings).dll_path).unwrap();
+ api.add_refmod(&RequestHandle::new(&api).unwrap(), &output, 1.0, 1).unwrap();
+}
+
#[test]
fn motion_cache_reaches_cuda_and_vulkan_requests_and_changes_timing_profile() {
let mut configuration = Configuration::from_settings(&BTreeMap::from([(