EffectCraft: An open-source motion graphics and compositing tool written by scratch in pure Rust
Roto Brush: don't run the classic engine silently under a selected model
Claudea7417b578df6
diff --git a/crates/effects/src/roto.rs b/crates/effects/src/roto.rsindex 96d6b467c..ffc2be376 100644--- a/crates/effects/src/roto.rs+++ b/crates/effects/src/roto.rs@@ -191,9 +191,15 @@ pub fn set_model(m: Option<Arc<dyn MaskModel>>) {}}+/// Whether an instance with `params` segments with the model Settings chose: Roto Brush 2.0 and+/// 3.0 do, 1.0 is the classic engine.+pub fn uses_model(params: &Params) -> bool {+ e(params, "version") != 0+}+/// The model an instance segments with: its version's choice of the installed one.pub fn model_for(params: &Params) -> Option<Arc<dyn MaskModel>> {- if e(params, "version") == 0 {+ if !uses_model(params) {return None;}installed_model()diff --git a/crates/engine/src/analysis/models.rs b/crates/engine/src/analysis/models.rsindex dbe8a1d5f..11f856d5f 100644--- a/crates/engine/src/analysis/models.rs+++ b/crates/engine/src/analysis/models.rs@@ -14,10 +14,12 @@use std::path::{Path, PathBuf};use std::sync::{Arc, Mutex};+use std::time::Duration;use effectcraft_segment::face::FaceModel;use effectcraft_segment::{self as seg, Loaded, ModelInfo, Task};use serde_json::{Value, json};+use web_time::Instant;use crate::commands::{CommandSpec, always, b_p, bad, str_p};use crate::{EngineError, Result, Session, cmd, query};@@ -126,6 +128,10 @@ fn kind(task: Task) -> &'static Kind {}}+/// How long a job that needs the Roto Brush model waits for it to load (or download) before it+/// gives up. A worker that never reports back must not hang the command.+const ROTO_MODEL_WAIT: Duration = Duration::from_secs(600);+impl Session {/// The folder installed models live in: the host's, else `models` next to the settings.pub fn models_dir(&self) -> Option<PathBuf> {@@ -193,6 +199,85 @@ impl Session {self.models.status(Task::Mask).active}+ /// The Roto Brush model Settings chose, when it is one (not the classic engine).+ fn roto_model_info(&self) -> Option<&'static ModelInfo> {+ seg::info(self.chosen_model(Task::Mask)).filter(|m| m.task == Task::Mask)+ }++ /// Whether the chosen Roto Brush model is still loading or downloading.+ pub fn roto_model_loading(&self) -> bool {+ self.models.status(Task::Mask).busy.is_some() && self.chosen_model(Task::Mask) != seg::CLASSICAL+ }++ /// The Roto Brush model Settings chose and its state (`roto.status`).+ pub fn roto_model(&self) -> Value {+ let st = self.models.status(Task::Mask);+ json!({+ "selected": self.chosen_model(Task::Mask),+ "installed": self.roto_model_info().is_none_or(|m| self.model_path(m).is_some()),+ "active": st.active,+ "loading": st.busy.is_some(),+ "error": st.error,+ })+ }++ /// What `roto.status` says when the chosen model is not the one in use for this layer (`None`:+ /// nothing to say). `uses_model`: the layer segments with a model at all (Roto Brush 1.0 does not).+ pub fn roto_model_notice(&self, uses_model: bool) -> Option<String> {+ let info = self.roto_model_info()?;+ if !uses_model {+ return Some(format!("{} is selected, but this layer is Roto Brush 1.0: it segments with the classic engine", info.name));+ }+ let st = self.models.status(Task::Mask);+ if st.active.as_deref() == Some(info.id) {+ None+ } else if self.model_path(info).is_none() {+ Some(format!("{} is selected but not installed: propagate refuses until it is installed or Classic is chosen", info.name))+ } else if let Some(e) = st.error {+ Some(format!("{} is selected but failed to load ({e}): propagate refuses until it is selected again", info.name))+ } else {+ Some(format!("{} is selected but not loaded yet: propagate loads it first with wait: true, and refuses without it", info.name))+ }+ }++ /// Make sure the Roto Brush model Settings chose is in use before a job that segments with it+ /// starts, so the job never runs on the classic engine without saying so. With `wait`, a load+ /// under way is waited for and one not started is loaded here; without it, a model that is not+ /// ready is refused (and starts loading). A model that is not installed, or failed to load, is+ /// refused either way. The classic engine needs nothing.+ pub fn ensure_roto_model(&mut self, wait: bool) -> std::result::Result<(), String> {+ let Some(info) = self.roto_model_info() else { return Ok(()) };+ // A download or load under way (from Settings, or `select` without `wait`) finishes first.+ // Nothing is in flight in the browser build, so this never sleeps there.+ if wait && self.models.status(Task::Mask).busy.is_some() {+ let started = Instant::now();+ while self.models.status(Task::Mask).busy.is_some() {+ if started.elapsed() >= ROTO_MODEL_WAIT {+ return Err(format!("{} is still loading after {} minutes", info.name, ROTO_MODEL_WAIT.as_secs() / 60));+ }+ std::thread::sleep(Duration::from_millis(5));+ }+ }+ if self.model_path(info).is_none() {+ if self.models.status(Task::Mask).busy.is_some() {+ return Err(format!("{} is still downloading: try again when it is ready, or pass wait: true", info.name));+ }+ return Err(format!("{} is selected for Roto Brush but not installed: install or download it, or choose {}", info.name, ROTO.classic_name));+ }+ let st = self.models.status(Task::Mask);+ if st.active.as_deref() != Some(info.id) && st.busy.is_none() && st.error.is_none() {+ self.apply_model(Task::Mask, wait);+ }+ let st = self.models.status(Task::Mask);+ if let Some(e) = st.error {+ return Err(format!("{} failed to load: {e}", info.name));+ }+ if st.active.as_deref() == Some(info.id) {+ return Ok(());+ }+ Err(format!("{} is still loading: try again when it is ready, or pass wait: true", info.name))+ }+/// Finish what the model workers did: a toast, and loading a model that just arrived.pub fn poll_models(&mut self) {for task in [Task::Mask, Task::Face] {@@ -417,7 +502,7 @@ pub fn specs() -> Vec<CommandSpec> {"Use Roto Brush Model",[],None,- "{id: classical|mobilesam, wait?} (Roto Brush 2.0 / 3.0 use it; wait: loaded before returning)",+ "{id: classical|mobilesam|sam2.1-hiera-tiny, wait?} (Roto Brush 2.0 / 3.0 use it; wait: loaded before returning)",always,|s, p| select(s, p, Task::Mask)),diff --git a/crates/engine/src/analysis/roto.rs b/crates/engine/src/analysis/roto.rsindex 12ce21260..357e77b13 100644--- a/crates/engine/src/analysis/roto.rs+++ b/crates/engine/src/analysis/roto.rs@@ -621,9 +621,21 @@ impl Session {}}+ /// Whether `effect` on `layer` segments with the model Settings chose: Object Matte always,+ /// Roto Brush unless it is version 1.0 (the classic engine).+ pub(crate) fn uses_roto_model(&self, comp: ItemId, layer: LayerId, effect: Uid) -> bool {+ let g = self.project.comp(comp).and_then(|c| c.layer(layer)).and_then(|l| l.effects()).and_then(|f| f.groups().find(|g| g.uid == effect));+ g.is_some_and(|g| is_object_matte(g) || fx::uses_model(¶ms_static(g)))+ }+/// Start propagating (or freezing) Roto Brush `effect` on `layer`. `wait`: block until done/// (wasm32 always does). Returns the number of frames.pub fn start_roto(&mut self, comp: ItemId, layer: LayerId, effect: Uid, task: RotoTask, dir: Direction, wait: bool) -> Result<usize, String> {+ // The model must be in use before the job starts: otherwise it would run on the classic+ // engine without saying so.+ if self.uses_roto_model(comp, layer, effect) {+ self.ensure_roto_model(wait)?;+ }if self.is_roto_running() {if !wait {return Err("a Roto Brush job is already running".into());@@ -786,11 +798,14 @@ impl Session {pub fn poll_roto(&mut self, auto: bool) -> bool {self.poll_models();let changed = self.poll_roto_job();- if auto && self.roto_job.is_none() && self.offloaded_roto().is_none() && self.state.roto.auto_propagate {+ // Edited instances stay pending while the model they will segment with is still loading.+ let loading = self.roto_model_loading();+ if auto && self.roto_job.is_none() && self.offloaded_roto().is_none() && self.state.roto.auto_propagate && !loading {while let Some((c, l, e)) = self.roto_pending.first().copied() {self.roto_pending.remove(0);- if self.start_roto(c, l, e, RotoTask::Propagate, Direction::Both, false).is_ok() {- return true;+ match self.start_roto(c, l, e, RotoTask::Propagate, Direction::Both, false) {+ Ok(_) => return true,+ Err(message) => self.events.push(Event::Toast { message, error: true }),}}}diff --git a/crates/engine/src/cmd/roto_cmds.rs b/crates/engine/src/cmd/roto_cmds.rsindex c37adb0c2..c7f190fde 100644--- a/crates/engine/src/cmd/roto_cmds.rs+++ b/crates/engine/src/cmd/roto_cmds.rs@@ -431,6 +431,9 @@ fn status(s: &mut Session, p: &Value) -> Result<Value> {out["strokeFrames"] = json!(d.stroke_frames());out["frozen"] = json!(frozen_of(&g));out["pending"] = json!(s.roto_pending.contains(&(cid, lid, uid)));+ // The model Settings chose and whether this layer segments with it (propagate refuses or waits otherwise).+ out["model"] = s.roto_model();+ out["notice"] = json!(s.roto_model_notice(s.uses_roto_model(cid, lid, uid)));out["limits"] = json!(rt::frame_limits(&comp, &layer));let frame = frame_p(s, p, cid, &layer);out["frame"] = json!(frame);@@ -464,7 +467,8 @@ fn status(s: &mut Session, p: &Value) -> Result<Value> {}}// Where each frame's matte came from, and how far to trust it: frames a tracker followed- // report its confidence and whether the object was absent; the rest came from the classic engine.+ // report its confidence and whether the object was absent; frames the model segmented on+ // their own report "model"; the rest came from the classic engine.{let size = rt::layer_size(&s.project, &comp, &layer);let chain = fx::Chain::new(¶ms_static(&g), size, comp.frame_rate.as_f64(), 1.0);@@ -474,7 +478,13 @@ fn status(s: &mut Session, p: &Value) -> Result<Value> {.filter_map(|(f, k)| fx::trust(*k).map(|t| (f.to_string(), json!({"confidence": t.confidence, "occluded": t.occluded})))).collect();out["tracker"] = json!({"model": s.models_roto_active(), "frames": tracked});- out["source"] = json!(if tracked.contains_key(&frame.to_string()) { "tracker" } else { "classical" });+ out["source"] = json!(if tracked.contains_key(&frame.to_string()) {+ "tracker"+ } else if chain.model.is_some() {+ "model"+ } else {+ "classical"+ });}let computed = s.roto_computed(cid, lid, uid);out["computed"] = json!(computed.iter().filter(|(_, c)| **c).map(|(f, _)| *f).collect::<Vec<_>>());diff --git a/crates/engine/src/tests/roto.rs b/crates/engine/src/tests/roto.rsindex b8cef6677..9ef6a848f 100644--- a/crates/engine/src/tests/roto.rs+++ b/crates/engine/src/tests/roto.rs@@ -677,6 +677,7 @@ fn object_matte_mattes_persist_in_the_disk_cache() {fn sam2_tracks_the_clip_through_roto_propagate() {let Ok(weights) = std::env::var("EFFECTCRAFT_SAM2") else { return };let dir = std::env::temp_dir().join(format!("ec-roto-sam2-{}", std::process::id()));+ hold_roto_cache_test_lock();let (mut s, clip, _) = painted(center, "disk-sam2.mov");s.models_dir = Some(dir.join("models"));std::fs::create_dir_all(dir.join("models")).unwrap();@@ -707,6 +708,49 @@ fn sam2_tracks_the_clip_through_roto_propagate() {let _ = std::fs::remove_dir_all(dir);}+/// Choosing a model that is not installed must not fall back to the classic engine silently:+/// propagate refuses and `roto.status` says why. Choosing the classic engine runs it again.+#[test]+fn a_selected_model_that_is_not_installed_refuses_to_propagate() {+ hold_roto_cache_test_lock();+ let (mut s, clip, _) = painted(center, "disk-not-installed.mov");+ s.models_dir = Some(std::env::temp_dir().join(format!("ec-roto-none-{}", std::process::id())));+ s.execute_checked("roto.model.select", json!({"id": "sam2.1-hiera-tiny"})).unwrap();+ let st = s.execute("roto.status", json!({"layer": clip.0})).unwrap();+ assert_eq!(st["model"]["selected"], "sam2.1-hiera-tiny", "{st}");+ assert_eq!(st["model"]["installed"], false, "{st}");+ assert!(st["notice"].as_str().unwrap().contains("not installed"), "{st}");+ let e = s.execute_checked("roto.propagate", json!({"layer": clip.0, "direction": "both", "wait": true})).unwrap_err().to_string();+ assert!(e.contains("not installed"), "{e}");+ assert_eq!(s.execute("roto.status", json!({"layer": clip.0, "frame": 10})).unwrap()["source"], "classical");+ s.execute_checked("roto.model.select", json!({"id": "classical"})).unwrap();+ let r = s.execute_checked("roto.propagate", json!({"layer": clip.0, "direction": "both", "wait": true})).unwrap();+ assert_eq!(r["frames"], 21, "{r}");+ assert!(s.execute("roto.status", json!({"layer": clip.0})).unwrap()["notice"].is_null());+}++/// Selecting SAM 2 without `wait` and propagating at once: propagate loads the model first, so the+/// tracker (not the classic engine) does the work. Needs EFFECTCRAFT_SAM2 like the test above.+#[cfg(not(target_arch = "wasm32"))]+#[test]+fn sam2_selected_without_wait_is_loaded_before_propagate() {+ let Ok(weights) = std::env::var("EFFECTCRAFT_SAM2") else { return };+ hold_roto_cache_test_lock();+ let dir = std::env::temp_dir().join(format!("ec-roto-sam2-wait-{}", std::process::id()));+ let (mut s, clip, _) = painted(center, "disk-sam2-wait.mov");+ s.models_dir = Some(dir.join("models"));+ std::fs::create_dir_all(dir.join("models")).unwrap();+ s.execute_checked("roto.model.install", json!({"path": weights})).unwrap();+ s.execute_checked("roto.model.select", json!({"id": "sam2.1-hiera-tiny"})).unwrap();+ let r = s.execute_checked("roto.propagate", json!({"layer": clip.0, "direction": "both", "wait": true})).unwrap();+ assert_eq!(r["frames"], 21, "{r}");+ let st = s.execute("roto.status", json!({"layer": clip.0, "frame": 12})).unwrap();+ assert_eq!(st["source"], "tracker", "{st}");+ assert_eq!(st["tracker"]["model"], "sam2.1-hiera-tiny");+ assert!(st["notice"].is_null(), "{st}");+ let _ = std::fs::remove_dir_all(dir);+}+/// A model whose tracker always fails: Roto Brush must carry on with the classic engine and say so.#[test]fn a_failing_tracker_falls_back_to_the_classic_engine() {@@ -735,6 +779,7 @@ fn a_failing_tracker_falls_back_to_the_classic_engine() {Err("the tracker broke".into())}}+ hold_roto_cache_test_lock();let (mut s, clip, _) = painted(center, "disk-broken-tracker.mov");effectcraft_effects::roto::set_model(Some(Arc::new(Broken)));let r = s.execute_checked("roto.propagate", json!({"layer": clip.0, "direction": "both", "wait": true}));diff --git a/docs/agents.md b/docs/agents.mdindex e0c0055ec..54ba21afa 100644--- a/docs/agents.md+++ b/docs/agents.md@@ -508,6 +508,9 @@ Trained model (Roto Brush 2.0 / 3.0):`roto.model.install {path, id?}` installs a file you already have.- `roto.model.select {id: classical|mobilesam|sam2.1-hiera-tiny}` chooses which one Roto Brush uses (SAM 2 also tracks the object through the clip, see [roto-brush-sam2.md](roto-brush-sam2.md); `roto.status` then reports `source` and per-frame `tracker.frames` with `confidence` and `occluded`). Loading runsin the background; `"wait": true` (also on download and install) returns once it is in use.+- `roto.propagate` never falls back to the classic engine silently: a chosen model that is not installed, or failed+ to load, is an error, and without `"wait": true` so is a model still loading. `roto.status` reports `model`+ (`selected`, `installed`, `active`, `loading`, `error`) and a `notice` while the chosen model is not the one in use.- `roto.model.remove {id}` deletes the installed weights.- The `version` property (`effects/#1/version`: 0 = 1.0, 1 = 2.0, 2 = 3.0) picks between theclassic engine (1.0) and the chosen model (2.0, 3.0).diff --git a/docs/roto-brush-sam2.md b/docs/roto-brush-sam2.mdindex efa7a5eb8..f5946a98e 100644--- a/docs/roto-brush-sam2.md+++ b/docs/roto-brush-sam2.md@@ -12,12 +12,19 @@ Apache-2.0). It is optional: nothing changes until you install the model and chowhole span in both directions, using SAM 2's streaming memory, so it keeps tracking through occlusion and when theobject leaves the frame and returns. Each frame's probability map is then refined by the same graph cut as every otherRoto Brush frame: your strokes stay hard constraints and edges snap to colour.-4. `roto.status {frame}` reports `source` (`tracker` or `classical`) and, for tracked frames, `tracker.frames[f]`:- `confidence` (0–1, the model's object score times its predicted mask quality) and `occluded` (the object is not- in the frame: an empty matte).+4. `roto.status {frame}` reports `source` (`tracker`, `model` for a frame the model segmented on its own, or+ `classical`) and, for tracked frames, `tracker.frames[f]`: `confidence` (0–1, the model's object score times its+ predicted mask quality) and `occluded` (the object is not in the frame: an empty matte). `model` says what Settings+ chose and whether it is installed, loaded and loading; `notice` says when it is not the one in use for the layer.-If the model is missing, fails, or is deselected, the classic engine runs exactly as before (`roto.status` says-`classical`, and the job's `trackerNote` says why).+Propagate never falls back to the classic engine without saying so. With SAM 2 chosen but not installed, or failing to+load, it refuses with the reason. While it is still loading, `wait: true` (agents) loads it first; without `wait` it+refuses, and the load goes on in the background. Deselecting SAM 2 (choosing Classic) runs Propagate exactly as before,+and Roto Brush 1.0 never uses a model.++Tracked frames live in the session that ran Propagate and are not saved in the project file. A separate render process+(`effectcraft-cli render-frame`) does not have them, so render in the same session: in the app, or in one `run` that+includes `roto.propagate` and then `comp.saveFrameAs`.## How it works`crates/sam2` is a Rust port (on [candle](https://github.com/huggingface/candle), native only) of the Hugging Face