The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.
Rust ffmpegfree-video-editormodel-context-protocolnon-linear-editoroffline-firstopen-source-video-editorself-hostedvideo-editingvideo-editing-automationvideo-editing-softwarevideo-editing-toolsvideo-editorvoice-cloningyoutube-automationyoutube-shorts
A smart stroke's region follows its thing along the clip
Jareer3b2dccf17ef4
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.mdindex 7776bb2c..d920884c 100644--- a/ARCHITECTURE.md+++ b/ARCHITECTURE.md@@ -438,15 +438,19 @@ either, because `apply::Mapping` is the one place a decoded pixel iswalked back to a source fraction. A mask the model left blank shows thepicture as shot rather than nothing.-**The smart brushes name a thing.** A stroke records the source instant-it was painted at; for a smart brush or smart eraser, `host::brush`-decodes that frame, runs SlimSAM's encoder on it once (the embedding is-kept for the next stroke on the same frame) and its decoder on sixteen-points along the stroke, and writes the region it answers - the whole-thing the stroke touched - beside the media's masks, named by a hash of-the stroke. `strokes::paint` keeps or drops that region whole; until it-has been read, or when the model is not to hand, the smart tools fall-back to the mask's own confidence under a disc. The plain brush and+**The smart brushes name a thing, and follow it.** A stroke records the+source instant it was painted at; for a smart brush or smart eraser,+`host::brush` decodes that frame, runs SlimSAM's encoder on it and its+decoder on sixteen points along the stroke, and then walks the clip's+analysed instants forward and backward from there, each frame prompted+with points from inside the region the frame beside it had and keeping+the candidate that overlaps it most, so the region moves with its thing.+The regions go beside the media's masks, one per instant in a directory+named by a hash of the stroke; where the thing is lost the rest of the+run gets an empty region, so the grid is whole and nothing is asked+twice. `strokes::paint` keeps or drops the instant's region whole; until+one has been read, or when the model is not to hand, the smart tools+fall back to the mask's own confidence under a disc. The plain brush anderaser are discs and nothing more.**The store is the project's.** Each media file's masks are PNGs indiff --git a/engine/crates/concat-export/src/lib.rs b/engine/crates/concat-export/src/lib.rsindex faa20979..ebd10062 100644--- a/engine/crates/concat-export/src/lib.rs+++ b/engine/crates/concat-export/src/lib.rs@@ -661,9 +661,9 @@ struct Source<F> {passes: Vec<ShaderPass>,}-/// The tint a highlighted cutout wears: a teal that no footage is likely-/// to be, and the colour the brushes themselves are drawn in.-const HIGHLIGHT: [u8; 3] = [0, 196, 204];+/// The tint a highlighted cutout wears: the interface's accent, the same+/// lime the brushes and the selection are drawn in.+const HIGHLIGHT: [u8; 3] = [0xcb, 0xf5, 0x3f];/// A cutout as the frame loop runs it: the masks, what to paint on them,/// and how a decoded pixel finds its place in the source.diff --git a/engine/crates/concat-host/src/brush.rs b/engine/crates/concat-host/src/brush.rsindex faa5bf13..0889913f 100644--- a/engine/crates/concat-host/src/brush.rs+++ b/engine/crates/concat-host/src/brush.rs@@ -4,25 +4,32 @@//! Reading what a smart stroke lies on.//!//! A smart brush or smart eraser names a thing in one frame, and the-//! whole of it is kept or dropped. This is the job that finds the thing:-//! it decodes the frame the stroke was painted at, runs the brush model's-//! encoder on it - once per frame, the embedding kept for the next stroke-//! on the same one - and its decoder on the stroke's points, and writes-//! the region beside the media's masks, where `concat-vision` paints it-//! in. The model's two files download on first use, like the segmenters'.+//! whole of it is kept or dropped. This is the job that finds the thing+//! and follows it: it decodes the frame the stroke was painted at, runs+//! the brush model's encoder on it and its decoder on the stroke's+//! points, and then walks the clip's analysed instants forward and+//! backward from there, each frame prompted from the region the frame+//! beside it had, so the region moves with its thing. Every region goes+//! beside the media's masks, one per instant, where `concat-vision`+//! paints it in. Where the thing is lost the rest of the run gets an+//! empty region: nothing kept, nothing dropped, and the grid complete so+//! the job is not asked again. The model's two files download on first+//! use, like the segmenters'.//!-//! One region at a time through a [`SingleFlight`]; the window queues the-//! rest behind it.+//! One stroke at a time through a [`SingleFlight`]; the window queues the+//! rest behind it. An embedding is kept for a few frames, so a second+//! stroke on the same frame costs the decoder alone.use std::collections::VecDeque;use std::path::{Path, PathBuf};use std::sync::atomic::{AtomicBool, Ordering};use std::sync::{Arc, Mutex};-use concat_core::time::Rational;+use concat_core::frame::Frame;+use concat_core::time::{FrameRate, Rational};use concat_media::{DecodeOptions, Decoder, FrameSource};use concat_project::model::{Stroke, Subject};-use concat_vision::{Brush, Embedding, ModelId, mask_dir, models, region_file};+use concat_vision::{Brush, Embedding, MASK_RATE, Mask, MaskStore, ModelId, mask_dir, models, region_dir};use crate::cutout::Progress;use crate::jobs::SingleFlight;@@ -34,32 +41,47 @@ pub struct RegionRequest {pub project: PathBuf,/// The media file the stroke was painted on.pub media_path: String,+ /// The media's picture size.+ pub media_size: (u32, u32),/// A still: its one frame, whatever the stroke's instant.pub still: bool,/// The cutout's subject, which names the masks' directory.pub subject: Subject,+ /// The stretches of source, in seconds, the clip shows: where the+ /// region is followed to.+ pub ranges: Vec<(f64, f64)>,/// The stroke, with its instant.pub stroke: Stroke,}impl RegionRequest {- /// Where the region goes.- pub fn file(&self) -> PathBuf {- region_file(+ /// Where the regions go.+ pub fn dir(&self) -> PathBuf {+ region_dir(&mask_dir(&self.project, &self.media_path, self.subject),&self.stroke,)}- /// Whether the region is still to be read.+ /// Whether any instant the clip shows still wants a region.pub fn outstanding(&self) -> bool {- self.stroke.is_smart() && self.stroke.at.is_some() && !self.file().is_file()+ if !self.stroke.is_smart() || self.stroke.at.is_none() {+ return false;+ }+ let store = MaskStore::open(&self.dir());+ if self.still {+ return store.is_empty();+ }+ !missing(&store, &self.ranges).is_empty()}}/// How many frames' embeddings stay in memory. Each is a few megabytes;/// a few frames is a session of strokes on one shot.const EMBEDDINGS: usize = 6;+/// The backward walk decodes this many frames at a time and holds their+/// embeddings, since a decoder only runs forward.+const BACKWARD_CHUNK: usize = 8;/// A frame read lately: its media and source millisecond, and what the/// encoder made of it.@@ -85,7 +107,7 @@ impl Brushes {}}- /// Whether a region is being read.+ /// Whether a stroke is being read.pub fn is_busy(&self) -> bool {self.gate.is_busy()}@@ -116,8 +138,39 @@ impl Brushes {Ok(brush)}- /// Reads the region under `request`'s stroke and writes it. Blocks- /// for the whole run, so run it on its own thread.+ /// The encoder's reading of `frame` at `millis` of `media`, from the+ /// recent few when it is there.+ fn embed(+ &self,+ brush: &Brush,+ media: &str,+ millis: u64,+ frame: &Frame,+ ) -> Result<Arc<Embedding>, String> {+ let key = (media.to_owned(), millis);+ if let Some(held) = self+ .embeddings+ .lock()+ .map_err(|_| "embeddings poisoned")?+ .iter()+ .find(|(k, _)| *k == key)+ .map(|(_, embedding)| Arc::clone(embedding))+ {+ return Ok(held);+ }+ let embedding = Arc::new(brush.embed(frame)?);+ if let Ok(mut held) = self.embeddings.lock() {+ held.push_back((key, Arc::clone(&embedding)));+ while held.len() > EMBEDDINGS {+ held.pop_front();+ }+ }+ Ok(embedding)+ }++ /// Reads the regions under `request`'s stroke, along the clip, and+ /// writes them. Blocks for the whole run, so run it on its own+ /// thread; `progress` is called as it goes.pub fn read(&self,request: &RegionRequest,@@ -128,65 +181,171 @@ impl Brushes {let Some(at) = request.stroke.at else {return Err("the stroke has no instant".to_owned());};- let file = request.file();- if file.is_file() {- return Ok(());- }+ let mut store = MaskStore::open(&request.dir());let brush = self.brush(&cancel, progress)?;progress(Progress::Analysing(0.0));- let millis = if request.still {+ let (width, height) = Brush::input_size(request.media_size.0, request.media_size.1);+ let step = 1000 / u64::from(MASK_RATE);+ let stopped = || cancel.load(Ordering::Relaxed);+ let open = |from: u64, count: usize| -> Result<Decoder, String> {+ let mut options = DecodeOptions::default()+ .scaled_to(width, height)+ .limited_to(count as u64);+ if !request.still {+ options = options+ .starting_at(Rational::new(from as i64, 1000))+ .at_rate(FrameRate::new(Rational::new(i64::from(MASK_RATE), 1)));+ }+ Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())+ };++ // The seed: the thing under the stroke, in the frame it was+ // painted on, held to the analysis grid.+ let seed_ms = if request.still {0} else {- (at * 1000.0).round() as u64+ ((at * 1000.0) / step as f64).round() as u64 * step};- let key = (request.media_path.clone(), millis);- let held = self- .embeddings- .lock()- .map_err(|_| "embeddings poisoned")?- .iter()- .find(|(k, _)| *k == key)- .map(|(_, embedding)| Arc::clone(embedding));- let embedding = match held {- Some(embedding) => embedding,- None => {- let facts = concat_media::probe(&request.media_path)- .map_err(|error| error.to_string())?;- let video = facts.require_video().map_err(|error| error.to_string())?;- let (w, h) = Brush::input_size(video.width, video.height);- let mut options = DecodeOptions::default().scaled_to(w, h).limited_to(1);- if !request.still {- options = options.starting_at(Rational::new(millis as i64, 1000));- }- let mut decoder = Decoder::open(&request.media_path, &options)- .map_err(|error| error.to_string())?;- let frame = decoder- .next_frame()- .map_err(|error| error.to_string())?- .ok_or_else(|| format!("{}: no picture at the stroke", request.media_path))?;- if cancel.load(Ordering::Relaxed) {+ if store.mask_at(seed_ms as f64 / 1000.0).is_none() || request.still && store.is_empty() {+ let frame = open(seed_ms, 1)?+ .next_frame()+ .map_err(|error| error.to_string())?+ .ok_or_else(|| format!("{}: no picture at the stroke", request.media_path))?;+ if stopped() {+ return Err("brush reading cancelled".to_owned());+ }+ let embedding = self.embed(&brush, &request.media_path, seed_ms, &frame)?;+ let region = brush.region(&embedding, &request.stroke.points)?;+ store.put(seed_ms, ®ion)?;+ }+ if request.still {+ progress(Progress::Analysing(1.0));+ return Ok(());+ }++ // Then the rest of what the clip shows, followed from the seed:+ // forward in one decode, backward in chunks.+ let wanted = missing(&store, &request.ranges);+ let total = wanted.len();+ if total == 0 {+ progress(Progress::Analysing(1.0));+ return Ok(());+ }+ let mut done = 0usize;+ let lost_after = |store: &mut MaskStore, instants: &[u64]| -> Result<(), String> {+ for &millis in instants {+ store.put(millis, &Mask::filled(1, 1, 0))?;+ }+ Ok(())+ };++ let forward: Vec<u64> = wanted.iter().copied().filter(|&m| m > seed_ms).collect();+ for run in runs(&forward, step) {+ let mut decoder = open(run[0], run.len())?;+ let mut previous = store+ .mask_at((run[0] - step) as f64 / 1000.0)+ .ok_or_else(|| "brush: the region before is missing".to_owned())?;+ for (index, &millis) in run.iter().enumerate() {+ if stopped() {return Err("brush reading cancelled".to_owned());}- let embedding = Arc::new(brush.embed(&frame)?);- if let Ok(mut held) = self.embeddings.lock() {- held.push_back((key, Arc::clone(&embedding)));- while held.len() > EMBEDDINGS {- held.pop_front();+ let Some(frame) = decoder.next_frame().map_err(|error| error.to_string())? else {+ lost_after(&mut store, &run[index..])?;+ done += run.len() - index;+ break;+ };+ let embedding = self.embed(&brush, &request.media_path, millis, &frame)?;+ match brush.track(&embedding, &previous)? {+ Some(region) => {+ store.put(millis, ®ion)?;+ previous = Arc::new(region);+ done += 1;+ progress(Progress::Analysing(done as f32 / total as f32));+ }+ None => {+ lost_after(&mut store, &run[index..])?;+ done += run.len() - index;+ break;}}- embedding}- };- progress(Progress::Analysing(0.5));- let region = brush.region(&embedding, &request.stroke.points)?;- if let Some(parent) = file.parent() {- std::fs::create_dir_all(parent)- .map_err(|error| format!("could not create {}: {error}", parent.display()))?;}- std::fs::write(&file, region.to_png())- .map_err(|error| format!("could not write {}: {error}", file.display()))?;++ let backward: Vec<u64> = wanted.iter().copied().filter(|&m| m < seed_ms).collect();+ for run in runs(&backward, step) {+ // Nearest the seed first: the run's tail, in chunks, each+ // decoded forward and walked backward.+ let mut chunk_end = run.len();+ let mut lost = false;+ while chunk_end > 0 && !lost {+ let chunk_start = chunk_end.saturating_sub(BACKWARD_CHUNK);+ let chunk = &run[chunk_start..chunk_end];+ let mut decoder = open(chunk[0], chunk.len())?;+ let mut frames: Vec<(u64, Frame)> = Vec::with_capacity(chunk.len());+ for &millis in chunk {+ if stopped() {+ return Err("brush reading cancelled".to_owned());+ }+ match decoder.next_frame().map_err(|error| error.to_string())? {+ Some(frame) => frames.push((millis, frame)),+ None => break,+ }+ }+ for &(millis, ref frame) in frames.iter().rev() {+ let Some(previous) = store.mask_at((millis + step) as f64 / 1000.0) else {+ lost = true;+ break;+ };+ let embedding = self.embed(&brush, &request.media_path, millis, frame)?;+ match brush.track(&embedding, &previous)? {+ Some(region) => {+ store.put(millis, ®ion)?;+ done += 1;+ progress(Progress::Analysing(done as f32 / total as f32));+ }+ None => {+ lost = true;+ break;+ }+ }+ }+ chunk_end = chunk_start;+ }+ // Whatever the walk did not reach - lost, or the file short -+ // is empty, so the grid is whole.+ let left: Vec<u64> = run+ .iter()+ .copied()+ .filter(|&m| store.mask_at(m as f64 / 1000.0).is_none())+ .collect();+ done += left.len();+ lost_after(&mut store, &left)?;+ }progress(Progress::Analysing(1.0));Ok(())}}++/// Every instant the ranges need that the store lacks, ascending, once.+fn missing(store: &MaskStore, ranges: &[(f64, f64)]) -> Vec<u64> {+ let mut wanted: Vec<u64> = ranges+ .iter()+ .flat_map(|&(from, to)| store.missing(from, to))+ .collect();+ wanted.sort_unstable();+ wanted.dedup();+ wanted+}++/// The instants split into runs a step apart.+fn runs(instants: &[u64], step: u64) -> Vec<Vec<u64>> {+ let mut out: Vec<Vec<u64>> = Vec::new();+ for &at in instants {+ match out.last_mut() {+ Some(run) if run.last().is_some_and(|&last| last + step == at) => run.push(at),+ _ => out.push(vec![at]),+ }+ }+ out+}diff --git a/engine/crates/concat-vision/src/brush.rs b/engine/crates/concat-vision/src/brush.rsindex f0dc37fe..1fd3a23c 100644--- a/engine/crates/concat-vision/src/brush.rs+++ b/engine/crates/concat-vision/src/brush.rs@@ -17,6 +17,12 @@//! long edge and the rest is padding. The mask it answers with covers//! that square; [`Brush::region`] reads it back as a mask over the source//! picture alone, in the fractions every other mask here uses.+//!+//! Footage moves. A stroke names the thing in one frame; [`Brush::track`]+//! finds the same thing in the next one, prompting the model with points+//! from inside the region it had and keeping the candidate that overlaps+//! it most, so a region follows its thing along the clip rather than+//! staying where the stroke was.use std::path::Path;use std::sync::Mutex;@@ -125,6 +131,42 @@ impl Brush {.map(|i| points[i * (points.len() - 1) / (PROMPTS - 1)]).collect()};+ // Three candidates, scored; the model's own best guess wins.+ let candidates = self.candidates(embedding, &picked)?;+ candidates+ .into_iter()+ .max_by(|a, b| a.0.total_cmp(&b.0))+ .map(|(_, mask)| mask)+ .ok_or_else(|| "brush: the model answered nothing".to_owned())+ }++ /// The thing `previous` covered, found again in another frame: the+ /// model is prompted with points from inside the old region and the+ /// candidate that overlaps it most is the answer. `None` when nothing+ /// overlaps enough - the thing has left, or changed past knowing.+ pub fn track(&self, embedding: &Embedding, previous: &Mask) -> Result<Option<Mask>, String> {+ /// Below this much overlap with the region before, the thing is+ /// lost rather than moved.+ const STILL_IT: f32 = 0.25;+ let prompts = interior_points(previous);+ if prompts.is_empty() {+ return Ok(None);+ }+ let candidates = self.candidates(embedding, &prompts)?;+ let best = candidates+ .into_iter()+ .map(|(_, mask)| (overlap(&mask, previous), mask))+ .max_by(|a, b| a.0.total_cmp(&b.0));+ Ok(best.filter(|(iou, _)| *iou >= STILL_IT).map(|(_, mask)| mask))+ }++ /// The model's three answers for `picked` points, each with the+ /// score it gives itself, as masks over the source picture.+ fn candidates(+ &self,+ embedding: &Embedding,+ picked: &[[f64; 2]],+ ) -> Result<Vec<(f32, Mask)>, String> {if picked.is_empty() {return Err("brush: a stroke with no points".to_owned());}@@ -166,36 +208,101 @@ impl Brush {.lock().map_err(|_| "brush decoder poisoned".to_owned())?.run(inputs, &["iou_scores", "pred_masks"])?;- // Three candidate masks, scored; the model's own best guess wins.- let best = outputs[0]- .data- .iter()- .enumerate()- .max_by(|a, b| a.1.total_cmp(b.1))- .map(|(index, _)| index)- .unwrap_or(0);+ let scores = &outputs[0].data;let masks = &outputs[1].data;let plane = SAM_MASK * SAM_MASK;- if masks.len() < (best + 1) * plane {+ let count = scores.len().min(masks.len() / plane.max(1));+ if count == 0 {return Err("brush: the model's answer is the wrong size".to_owned());}- let logits = &masks[best * plane..(best + 1) * plane];- // The answer covers the square; the picture is its top-left+ // Each answer covers the square; the picture is its top-left// corner. Read the region over the picture alone.let size = REGION_SIZE as usize;- let mut bytes = Vec::with_capacity(size * size);- for y in 0..size {- let py = ((y as f32 + 0.5) / size as f32) * sh / SAM_SIZE as f32 * SAM_MASK as f32;- for x in 0..size {- let px = ((x as f32 + 0.5) / size as f32) * sw / SAM_SIZE as f32 * SAM_MASK as f32;- let logit = bilinear(logits, SAM_MASK, px - 0.5, py - 0.5);- let probability = 1.0 / (1.0 + (-logit).exp());- bytes.push((probability.clamp(0.0, 1.0) * 255.0 + 0.5) as u8);+ (0..count)+ .map(|index| {+ let logits = &masks[index * plane..(index + 1) * plane];+ let mut bytes = Vec::with_capacity(size * size);+ for y in 0..size {+ let py = ((y as f32 + 0.5) / size as f32) * sh / SAM_SIZE as f32+ * SAM_MASK as f32;+ for x in 0..size {+ let px = ((x as f32 + 0.5) / size as f32) * sw / SAM_SIZE as f32+ * SAM_MASK as f32;+ let logit = bilinear(logits, SAM_MASK, px - 0.5, py - 0.5);+ let probability = 1.0 / (1.0 + (-logit).exp());+ bytes.push((probability.clamp(0.0, 1.0) * 255.0 + 0.5) as u8);+ }+ }+ Mask::from_bytes(REGION_SIZE, REGION_SIZE, bytes)+ .map(|mask| (scores[index], mask))+ .ok_or_else(|| "brush: the region is the wrong size".to_owned())+ })+ .collect()+ }+}++/// Points well inside a region, as source fractions: a handful spread+/// over its sure pixels, to prompt the model with where the thing was.+fn interior_points(region: &Mask) -> Vec<[f64; 2]> {+ /// A pixel this sure is inside, not on the edge.+ const INSIDE: u8 = 230;+ /// At most this many prompts, on a grid over the region.+ const AT_MOST: usize = 9;+ let (width, height) = (region.width() as usize, region.height() as usize);+ let bytes = region.bytes();+ let sure: Vec<(usize, usize)> = (0..height)+ .flat_map(|y| (0..width).map(move |x| (x, y)))+ .filter(|&(x, y)| bytes[y * width + x] >= INSIDE)+ .collect();+ if sure.is_empty() {+ return Vec::new();+ }+ // The region's box, cut into a grid; the sure pixel nearest each+ // cell's centre is a prompt, so the prompts cover the thing rather+ // than cluster.+ let (x0, x1) = sure.iter().fold((usize::MAX, 0), |(lo, hi), &(x, _)| (lo.min(x), hi.max(x)));+ let (y0, y1) = sure.iter().fold((usize::MAX, 0), |(lo, hi), &(_, y)| (lo.min(y), hi.max(y)));+ let cells = (AT_MOST as f64).sqrt().round() as usize;+ let mut out = Vec::new();+ for cy in 0..cells {+ for cx in 0..cells {+ let tx = x0 as f64 + (x1 - x0) as f64 * (cx as f64 + 0.5) / cells as f64;+ let ty = y0 as f64 + (y1 - y0) as f64 * (cy as f64 + 0.5) / cells as f64;+ let nearest = sure+ .iter()+ .map(|&(x, y)| ((x as f64 - tx).powi(2) + (y as f64 - ty).powi(2), x, y))+ .min_by(|a, b| a.0.total_cmp(&b.0));+ if let Some((distance, x, y)) = nearest {+ // A cell whose nearest sure pixel is far away is not the+ // thing's; skip it rather than prompt beside the thing.+ let reach = ((x1 - x0).max(y1 - y0) as f64 / cells as f64).max(2.0);+ if distance.sqrt() <= reach {+ out.push([(x as f64 + 0.5) / width as f64, (y as f64 + 0.5) / height as f64]);+ }}}- Mask::from_bytes(REGION_SIZE, REGION_SIZE, bytes)- .ok_or_else(|| "brush: the region is the wrong size".to_owned())}+ out.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));+ out.dedup();+ out+}++/// Intersection over union of two regions' sure halves, sampled on the+/// first's grid.+fn overlap(a: &Mask, b: &Mask) -> f32 {+ let (width, height) = (a.width(), a.height());+ let (mut both, mut either) = (0u32, 0u32);+ for y in 0..height {+ let v = (y as f32 + 0.5) / height as f32;+ for x in 0..width {+ let u = (x as f32 + 0.5) / width as f32;+ let in_a = a.at(i64::from(x), i64::from(y)) >= 128;+ let in_b = b.sample(u, v) >= 0.5;+ both += u32::from(in_a && in_b);+ either += u32::from(in_a || in_b);+ }+ }+ if either == 0 { 0.0 } else { both as f32 / either as f32 }}/// A value from a square plane, bilinear between the four around it,@@ -225,6 +332,25 @@ mod tests {assert_eq!(Brush::input_size(500, 500), (1024, 1024));}+ #[test]+ fn interior_points_lie_inside_the_region_and_overlap_is_one_with_itself() {+ let mut region = Mask::filled(32, 32, 0);+ for y in 8..24 {+ for x in 8..24 {+ region.bytes_mut()[y * 32 + x] = 255;+ }+ }+ let points = interior_points(®ion);+ assert!(!points.is_empty() && points.len() <= 9);+ for [x, y] in &points {+ assert!((0.25..0.75).contains(x) && (0.25..0.75).contains(y), "{x} {y}");+ }+ assert_eq!(overlap(®ion, ®ion), 1.0);+ let empty = Mask::filled(32, 32, 0);+ assert!(interior_points(&empty).is_empty());+ assert_eq!(overlap(®ion, &empty), 0.0);+ }+#[test]fn bilinear_reads_between_pixels_and_clamps_at_the_edge() {let plane = [0.0, 1.0, 2.0, 3.0];diff --git a/engine/crates/concat-vision/src/lib.rs b/engine/crates/concat-vision/src/lib.rsindex 9b28879a..a89c0e8e 100644--- a/engine/crates/concat-vision/src/lib.rs+++ b/engine/crates/concat-vision/src/lib.rs@@ -58,7 +58,7 @@ pub use mask::Mask;pub use models::ModelId;#[cfg(feature = "infer")]pub use segment::Segmenter;-pub use store::{MaskStore, mask_dir, region_file};+pub use store::{MaskStore, mask_dir, region_dir};/// Masks are found this many times a second of source. Ten is where a/// person's outline stops visibly lagging their movement, and where adiff --git a/engine/crates/concat-vision/src/store.rs b/engine/crates/concat-vision/src/store.rsindex 551a5d1b..47b2f551 100644--- a/engine/crates/concat-vision/src/store.rs+++ b/engine/crates/concat-vision/src/store.rs@@ -51,11 +51,13 @@ fn model_file(dir: &Path) -> PathBuf {dir.join("model")}-/// The file a smart stroke's region is kept in, under the masks' own-/// directory: the thing the brush model read under that stroke.-pub fn region_file(dir: &Path, stroke: &Stroke) -> PathBuf {+/// The directory a smart stroke's regions are kept in, under the masks'+/// own directory: the thing the brush model read under that stroke, one+/// PNG per analysed instant like the masks themselves, since the thing+/// moves and the region follows it.+pub fn region_dir(dir: &Path, stroke: &Stroke) -> PathBuf {dir.join("strokes")- .join(format!("{:016x}.png", strokes::stroke_key(stroke)))+ .join(format!("{:016x}", strokes::stroke_key(stroke)))}/// The file for the mask at `millis` of source.@@ -183,9 +185,27 @@ impl MaskStore {pub fn resolved(&self, seconds: f64, cutout: &Cutout, aspect: f32) -> Option<Arc<Mask>> {let millis = self.nearest_millis(seconds)?;let file = mask_file(&self.dir, millis);+ // The smart strokes' regions, opened once for the key and the+ // painting both: what is there changes as the brush model works+ // its way along the clip.+ let region_stores: Vec<(u64, MaskStore)> = if cutout.mode == CutoutMode::Custom {+ cutout+ .strokes+ .iter()+ .filter(|stroke| stroke.is_smart())+ .map(|stroke| {+ (+ strokes::stroke_key(stroke),+ MaskStore::open(®ion_dir(&self.dir, stroke)),+ )+ })+ .collect()+ } else {+ Vec::new()+ };let key = ResolvedKey {file: file.clone(),- settings: settings_key(&self.dir, cutout, aspect),+ settings: settings_key(cutout, aspect, ®ion_stores, millis),};if let Some(hit) = resolved_cache().lock()@@ -201,8 +221,12 @@ impl MaskStore {}let painted = match cutout.mode {CutoutMode::Custom if !cutout.strokes.is_empty() => {- let dir = self.dir.clone();- let regions = move |stroke: &Stroke| load(®ion_file(&dir, stroke));+ let regions = move |stroke: &Stroke| {+ region_stores+ .iter()+ .find(|(key, _)| *key == strokes::stroke_key(stroke))+ .and_then(|(_, store)| store.mask_at(seconds))+ };strokes::paint(&auto, &cutout.strokes, aspect, ®ions)}_ => (*auto).clone(),@@ -276,8 +300,14 @@ impl MaskStore {/// What besides the file decides a resolved mask: the mode, the feather/// and every stroke, hashed, with the aspect the brushes were sized by,-/// and for each smart stroke whether its region has been read yet.-fn settings_key(dir: &Path, cutout: &Cutout, aspect: f32) -> u64 {+/// and for each smart stroke which region, if any, answers for the+/// instant.+fn settings_key(+ cutout: &Cutout,+ aspect: f32,+ region_stores: &[(u64, MaskStore)],+ millis: u64,+) -> u64 {use std::hash::{Hash, Hasher};let mut hasher = std::collections::hash_map::DefaultHasher::new();(cutout.mode == CutoutMode::Custom).hash(&mut hasher);@@ -286,9 +316,10 @@ fn settings_key(dir: &Path, cutout: &Cutout, aspect: f32) -> u64 {if cutout.mode == CutoutMode::Custom {for stroke in &cutout.strokes {strokes::stroke_key(stroke).hash(&mut hasher);- if stroke.is_smart() {- region_file(dir, stroke).is_file().hash(&mut hasher);- }+ }+ for (key, store) in region_stores {+ key.hash(&mut hasher);+ store.nearest_millis(millis as f64 / 1000.0).hash(&mut hasher);}}hasher.finish()diff --git a/engine/crates/concat/src/studio.rs b/engine/crates/concat/src/studio.rsindex 2b02299f..afd1d381 100644--- a/engine/crates/concat/src/studio.rs+++ b/engine/crates/concat/src/studio.rs@@ -5296,12 +5296,19 @@ impl Studio {let Some(media) = self.project().media_by_id(&clip.media_id) else {continue;};+ // The source the clip shows, as the mask analysis reckons it.+ let range = (+ clip.source_start,+ clip.source_start + clip.duration * clip.speed.max(0.0625),+ );for stroke in &cutout.strokes {let request = RegionRequest {project: project.clone(),media_path: media.path.clone(),+ media_size: (media.width.unwrap_or(0), media.height.unwrap_or(0)),still: media.kind == model::MediaKind::Image,subject: cutout.subject,+ ranges: vec![range],stroke: stroke.clone(),};if request.outstanding() {