Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

A smart stroke's region follows its thing along the clip

Jareer3b2dccf17ef4
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index 7776bb2c..d920884c 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -438,15 +438,19 @@ either, because `apply::Mapping` is the one place a decoded pixel is
walked back to a source fraction. A mask the model left blank shows the
picture as shot rather than nothing.
-**The smart brushes name a thing.** A stroke records the source instant
-it was painted at; for a smart brush or smart eraser, `host::brush`
-decodes that frame, runs SlimSAM's encoder on it once (the embedding is
-kept for the next stroke on the same frame) and its decoder on sixteen
-points along the stroke, and writes the region it answers - the whole
-thing the stroke touched - beside the media's masks, named by a hash of
-the stroke. `strokes::paint` keeps or drops that region whole; until it
-has been read, or when the model is not to hand, the smart tools fall
-back to the mask's own confidence under a disc. The plain brush and
+**The smart brushes name a thing, and follow it.** A stroke records the
+source instant it was painted at; for a smart brush or smart eraser,
+`host::brush` decodes that frame, runs SlimSAM's encoder on it and its
+decoder on sixteen points along the stroke, and then walks the clip's
+analysed instants forward and backward from there, each frame prompted
+with points from inside the region the frame beside it had and keeping
+the candidate that overlaps it most, so the region moves with its thing.
+The regions go beside the media's masks, one per instant in a directory
+named by a hash of the stroke; where the thing is lost the rest of the
+run gets an empty region, so the grid is whole and nothing is asked
+twice. `strokes::paint` keeps or drops the instant's region whole; until
+one has been read, or when the model is not to hand, the smart tools
+fall back to the mask's own confidence under a disc. The plain brush and
eraser are discs and nothing more.
**The store is the project's.** Each media file's masks are PNGs in
diff --git a/engine/crates/concat-export/src/lib.rs b/engine/crates/concat-export/src/lib.rs
index faa20979..ebd10062 100644
--- a/engine/crates/concat-export/src/lib.rs
+++ b/engine/crates/concat-export/src/lib.rs
@@ -661,9 +661,9 @@ struct Source<F> {
passes: Vec<ShaderPass>,
}
-/// The tint a highlighted cutout wears: a teal that no footage is likely
-/// to be, and the colour the brushes themselves are drawn in.
-const HIGHLIGHT: [u8; 3] = [0, 196, 204];
+/// The tint a highlighted cutout wears: the interface's accent, the same
+/// lime the brushes and the selection are drawn in.
+const HIGHLIGHT: [u8; 3] = [0xcb, 0xf5, 0x3f];
/// A cutout as the frame loop runs it: the masks, what to paint on them,
/// and how a decoded pixel finds its place in the source.
diff --git a/engine/crates/concat-host/src/brush.rs b/engine/crates/concat-host/src/brush.rs
index faa5bf13..0889913f 100644
--- a/engine/crates/concat-host/src/brush.rs
+++ b/engine/crates/concat-host/src/brush.rs
@@ -4,25 +4,32 @@
//! Reading what a smart stroke lies on.
//!
//! A smart brush or smart eraser names a thing in one frame, and the
-//! whole of it is kept or dropped. This is the job that finds the thing:
-//! it decodes the frame the stroke was painted at, runs the brush model's
-//! encoder on it - once per frame, the embedding kept for the next stroke
-//! on the same one - and its decoder on the stroke's points, and writes
-//! the region beside the media's masks, where `concat-vision` paints it
-//! in. The model's two files download on first use, like the segmenters'.
+//! whole of it is kept or dropped. This is the job that finds the thing
+//! and follows it: it decodes the frame the stroke was painted at, runs
+//! the brush model's encoder on it and its decoder on the stroke's
+//! points, and then walks the clip's analysed instants forward and
+//! backward from there, each frame prompted from the region the frame
+//! beside it had, so the region moves with its thing. Every region goes
+//! beside the media's masks, one per instant, where `concat-vision`
+//! paints it in. Where the thing is lost the rest of the run gets an
+//! empty region: nothing kept, nothing dropped, and the grid complete so
+//! the job is not asked again. The model's two files download on first
+//! use, like the segmenters'.
//!
-//! One region at a time through a [`SingleFlight`]; the window queues the
-//! rest behind it.
+//! One stroke at a time through a [`SingleFlight`]; the window queues the
+//! rest behind it. An embedding is kept for a few frames, so a second
+//! stroke on the same frame costs the decoder alone.
use std::collections::VecDeque;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
-use concat_core::time::Rational;
+use concat_core::frame::Frame;
+use concat_core::time::{FrameRate, Rational};
use concat_media::{DecodeOptions, Decoder, FrameSource};
use concat_project::model::{Stroke, Subject};
-use concat_vision::{Brush, Embedding, ModelId, mask_dir, models, region_file};
+use concat_vision::{Brush, Embedding, MASK_RATE, Mask, MaskStore, ModelId, mask_dir, models, region_dir};
use crate::cutout::Progress;
use crate::jobs::SingleFlight;
@@ -34,32 +41,47 @@ pub struct RegionRequest {
pub project: PathBuf,
/// The media file the stroke was painted on.
pub media_path: String,
+ /// The media's picture size.
+ pub media_size: (u32, u32),
/// A still: its one frame, whatever the stroke's instant.
pub still: bool,
/// The cutout's subject, which names the masks' directory.
pub subject: Subject,
+ /// The stretches of source, in seconds, the clip shows: where the
+ /// region is followed to.
+ pub ranges: Vec<(f64, f64)>,
/// The stroke, with its instant.
pub stroke: Stroke,
}
impl RegionRequest {
- /// Where the region goes.
- pub fn file(&self) -> PathBuf {
- region_file(
+ /// Where the regions go.
+ pub fn dir(&self) -> PathBuf {
+ region_dir(
&mask_dir(&self.project, &self.media_path, self.subject),
&self.stroke,
)
}
- /// Whether the region is still to be read.
+ /// Whether any instant the clip shows still wants a region.
pub fn outstanding(&self) -> bool {
- self.stroke.is_smart() && self.stroke.at.is_some() && !self.file().is_file()
+ if !self.stroke.is_smart() || self.stroke.at.is_none() {
+ return false;
+ }
+ let store = MaskStore::open(&self.dir());
+ if self.still {
+ return store.is_empty();
+ }
+ !missing(&store, &self.ranges).is_empty()
}
}
/// How many frames' embeddings stay in memory. Each is a few megabytes;
/// a few frames is a session of strokes on one shot.
const EMBEDDINGS: usize = 6;
+/// The backward walk decodes this many frames at a time and holds their
+/// embeddings, since a decoder only runs forward.
+const BACKWARD_CHUNK: usize = 8;
/// A frame read lately: its media and source millisecond, and what the
/// encoder made of it.
@@ -85,7 +107,7 @@ impl Brushes {
}
}
- /// Whether a region is being read.
+ /// Whether a stroke is being read.
pub fn is_busy(&self) -> bool {
self.gate.is_busy()
}
@@ -116,8 +138,39 @@ impl Brushes {
Ok(brush)
}
- /// Reads the region under `request`'s stroke and writes it. Blocks
- /// for the whole run, so run it on its own thread.
+ /// The encoder's reading of `frame` at `millis` of `media`, from the
+ /// recent few when it is there.
+ fn embed(
+ &self,
+ brush: &Brush,
+ media: &str,
+ millis: u64,
+ frame: &Frame,
+ ) -> Result<Arc<Embedding>, String> {
+ let key = (media.to_owned(), millis);
+ if let Some(held) = self
+ .embeddings
+ .lock()
+ .map_err(|_| "embeddings poisoned")?
+ .iter()
+ .find(|(k, _)| *k == key)
+ .map(|(_, embedding)| Arc::clone(embedding))
+ {
+ return Ok(held);
+ }
+ let embedding = Arc::new(brush.embed(frame)?);
+ if let Ok(mut held) = self.embeddings.lock() {
+ held.push_back((key, Arc::clone(&embedding)));
+ while held.len() > EMBEDDINGS {
+ held.pop_front();
+ }
+ }
+ Ok(embedding)
+ }
+
+ /// Reads the regions under `request`'s stroke, along the clip, and
+ /// writes them. Blocks for the whole run, so run it on its own
+ /// thread; `progress` is called as it goes.
pub fn read(
&self,
request: &RegionRequest,
@@ -128,65 +181,171 @@ impl Brushes {
let Some(at) = request.stroke.at else {
return Err("the stroke has no instant".to_owned());
};
- let file = request.file();
- if file.is_file() {
- return Ok(());
- }
+ let mut store = MaskStore::open(&request.dir());
let brush = self.brush(&cancel, progress)?;
progress(Progress::Analysing(0.0));
- let millis = if request.still {
+ let (width, height) = Brush::input_size(request.media_size.0, request.media_size.1);
+ let step = 1000 / u64::from(MASK_RATE);
+ let stopped = || cancel.load(Ordering::Relaxed);
+ let open = |from: u64, count: usize| -> Result<Decoder, String> {
+ let mut options = DecodeOptions::default()
+ .scaled_to(width, height)
+ .limited_to(count as u64);
+ if !request.still {
+ options = options
+ .starting_at(Rational::new(from as i64, 1000))
+ .at_rate(FrameRate::new(Rational::new(i64::from(MASK_RATE), 1)));
+ }
+ Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())
+ };
+
+ // The seed: the thing under the stroke, in the frame it was
+ // painted on, held to the analysis grid.
+ let seed_ms = if request.still {
0
} else {
- (at * 1000.0).round() as u64
+ ((at * 1000.0) / step as f64).round() as u64 * step
};
- let key = (request.media_path.clone(), millis);
- let held = self
- .embeddings
- .lock()
- .map_err(|_| "embeddings poisoned")?
- .iter()
- .find(|(k, _)| *k == key)
- .map(|(_, embedding)| Arc::clone(embedding));
- let embedding = match held {
- Some(embedding) => embedding,
- None => {
- let facts = concat_media::probe(&request.media_path)
- .map_err(|error| error.to_string())?;
- let video = facts.require_video().map_err(|error| error.to_string())?;
- let (w, h) = Brush::input_size(video.width, video.height);
- let mut options = DecodeOptions::default().scaled_to(w, h).limited_to(1);
- if !request.still {
- options = options.starting_at(Rational::new(millis as i64, 1000));
- }
- let mut decoder = Decoder::open(&request.media_path, &options)
- .map_err(|error| error.to_string())?;
- let frame = decoder
- .next_frame()
- .map_err(|error| error.to_string())?
- .ok_or_else(|| format!("{}: no picture at the stroke", request.media_path))?;
- if cancel.load(Ordering::Relaxed) {
+ if store.mask_at(seed_ms as f64 / 1000.0).is_none() || request.still && store.is_empty() {
+ let frame = open(seed_ms, 1)?
+ .next_frame()
+ .map_err(|error| error.to_string())?
+ .ok_or_else(|| format!("{}: no picture at the stroke", request.media_path))?;
+ if stopped() {
+ return Err("brush reading cancelled".to_owned());
+ }
+ let embedding = self.embed(&brush, &request.media_path, seed_ms, &frame)?;
+ let region = brush.region(&embedding, &request.stroke.points)?;
+ store.put(seed_ms, &region)?;
+ }
+ if request.still {
+ progress(Progress::Analysing(1.0));
+ return Ok(());
+ }
+
+ // Then the rest of what the clip shows, followed from the seed:
+ // forward in one decode, backward in chunks.
+ let wanted = missing(&store, &request.ranges);
+ let total = wanted.len();
+ if total == 0 {
+ progress(Progress::Analysing(1.0));
+ return Ok(());
+ }
+ let mut done = 0usize;
+ let lost_after = |store: &mut MaskStore, instants: &[u64]| -> Result<(), String> {
+ for &millis in instants {
+ store.put(millis, &Mask::filled(1, 1, 0))?;
+ }
+ Ok(())
+ };
+
+ let forward: Vec<u64> = wanted.iter().copied().filter(|&m| m > seed_ms).collect();
+ for run in runs(&forward, step) {
+ let mut decoder = open(run[0], run.len())?;
+ let mut previous = store
+ .mask_at((run[0] - step) as f64 / 1000.0)
+ .ok_or_else(|| "brush: the region before is missing".to_owned())?;
+ for (index, &millis) in run.iter().enumerate() {
+ if stopped() {
return Err("brush reading cancelled".to_owned());
}
- let embedding = Arc::new(brush.embed(&frame)?);
- if let Ok(mut held) = self.embeddings.lock() {
- held.push_back((key, Arc::clone(&embedding)));
- while held.len() > EMBEDDINGS {
- held.pop_front();
+ let Some(frame) = decoder.next_frame().map_err(|error| error.to_string())? else {
+ lost_after(&mut store, &run[index..])?;
+ done += run.len() - index;
+ break;
+ };
+ let embedding = self.embed(&brush, &request.media_path, millis, &frame)?;
+ match brush.track(&embedding, &previous)? {
+ Some(region) => {
+ store.put(millis, &region)?;
+ previous = Arc::new(region);
+ done += 1;
+ progress(Progress::Analysing(done as f32 / total as f32));
+ }
+ None => {
+ lost_after(&mut store, &run[index..])?;
+ done += run.len() - index;
+ break;
}
}
- embedding
}
- };
- progress(Progress::Analysing(0.5));
- let region = brush.region(&embedding, &request.stroke.points)?;
- if let Some(parent) = file.parent() {
- std::fs::create_dir_all(parent)
- .map_err(|error| format!("could not create {}: {error}", parent.display()))?;
}
- std::fs::write(&file, region.to_png())
- .map_err(|error| format!("could not write {}: {error}", file.display()))?;
+
+ let backward: Vec<u64> = wanted.iter().copied().filter(|&m| m < seed_ms).collect();
+ for run in runs(&backward, step) {
+ // Nearest the seed first: the run's tail, in chunks, each
+ // decoded forward and walked backward.
+ let mut chunk_end = run.len();
+ let mut lost = false;
+ while chunk_end > 0 && !lost {
+ let chunk_start = chunk_end.saturating_sub(BACKWARD_CHUNK);
+ let chunk = &run[chunk_start..chunk_end];
+ let mut decoder = open(chunk[0], chunk.len())?;
+ let mut frames: Vec<(u64, Frame)> = Vec::with_capacity(chunk.len());
+ for &millis in chunk {
+ if stopped() {
+ return Err("brush reading cancelled".to_owned());
+ }
+ match decoder.next_frame().map_err(|error| error.to_string())? {
+ Some(frame) => frames.push((millis, frame)),
+ None => break,
+ }
+ }
+ for &(millis, ref frame) in frames.iter().rev() {
+ let Some(previous) = store.mask_at((millis + step) as f64 / 1000.0) else {
+ lost = true;
+ break;
+ };
+ let embedding = self.embed(&brush, &request.media_path, millis, frame)?;
+ match brush.track(&embedding, &previous)? {
+ Some(region) => {
+ store.put(millis, &region)?;
+ done += 1;
+ progress(Progress::Analysing(done as f32 / total as f32));
+ }
+ None => {
+ lost = true;
+ break;
+ }
+ }
+ }
+ chunk_end = chunk_start;
+ }
+ // Whatever the walk did not reach - lost, or the file short -
+ // is empty, so the grid is whole.
+ let left: Vec<u64> = run
+ .iter()
+ .copied()
+ .filter(|&m| store.mask_at(m as f64 / 1000.0).is_none())
+ .collect();
+ done += left.len();
+ lost_after(&mut store, &left)?;
+ }
progress(Progress::Analysing(1.0));
Ok(())
}
}
+
+/// Every instant the ranges need that the store lacks, ascending, once.
+fn missing(store: &MaskStore, ranges: &[(f64, f64)]) -> Vec<u64> {
+ let mut wanted: Vec<u64> = ranges
+ .iter()
+ .flat_map(|&(from, to)| store.missing(from, to))
+ .collect();
+ wanted.sort_unstable();
+ wanted.dedup();
+ wanted
+}
+
+/// The instants split into runs a step apart.
+fn runs(instants: &[u64], step: u64) -> Vec<Vec<u64>> {
+ let mut out: Vec<Vec<u64>> = Vec::new();
+ for &at in instants {
+ match out.last_mut() {
+ Some(run) if run.last().is_some_and(|&last| last + step == at) => run.push(at),
+ _ => out.push(vec![at]),
+ }
+ }
+ out
+}
diff --git a/engine/crates/concat-vision/src/brush.rs b/engine/crates/concat-vision/src/brush.rs
index f0dc37fe..1fd3a23c 100644
--- a/engine/crates/concat-vision/src/brush.rs
+++ b/engine/crates/concat-vision/src/brush.rs
@@ -17,6 +17,12 @@
//! long edge and the rest is padding. The mask it answers with covers
//! that square; [`Brush::region`] reads it back as a mask over the source
//! picture alone, in the fractions every other mask here uses.
+//!
+//! Footage moves. A stroke names the thing in one frame; [`Brush::track`]
+//! finds the same thing in the next one, prompting the model with points
+//! from inside the region it had and keeping the candidate that overlaps
+//! it most, so a region follows its thing along the clip rather than
+//! staying where the stroke was.
use std::path::Path;
use std::sync::Mutex;
@@ -125,6 +131,42 @@ impl Brush {
.map(|i| points[i * (points.len() - 1) / (PROMPTS - 1)])
.collect()
};
+ // Three candidates, scored; the model's own best guess wins.
+ let candidates = self.candidates(embedding, &picked)?;
+ candidates
+ .into_iter()
+ .max_by(|a, b| a.0.total_cmp(&b.0))
+ .map(|(_, mask)| mask)
+ .ok_or_else(|| "brush: the model answered nothing".to_owned())
+ }
+
+ /// The thing `previous` covered, found again in another frame: the
+ /// model is prompted with points from inside the old region and the
+ /// candidate that overlaps it most is the answer. `None` when nothing
+ /// overlaps enough - the thing has left, or changed past knowing.
+ pub fn track(&self, embedding: &Embedding, previous: &Mask) -> Result<Option<Mask>, String> {
+ /// Below this much overlap with the region before, the thing is
+ /// lost rather than moved.
+ const STILL_IT: f32 = 0.25;
+ let prompts = interior_points(previous);
+ if prompts.is_empty() {
+ return Ok(None);
+ }
+ let candidates = self.candidates(embedding, &prompts)?;
+ let best = candidates
+ .into_iter()
+ .map(|(_, mask)| (overlap(&mask, previous), mask))
+ .max_by(|a, b| a.0.total_cmp(&b.0));
+ Ok(best.filter(|(iou, _)| *iou >= STILL_IT).map(|(_, mask)| mask))
+ }
+
+ /// The model's three answers for `picked` points, each with the
+ /// score it gives itself, as masks over the source picture.
+ fn candidates(
+ &self,
+ embedding: &Embedding,
+ picked: &[[f64; 2]],
+ ) -> Result<Vec<(f32, Mask)>, String> {
if picked.is_empty() {
return Err("brush: a stroke with no points".to_owned());
}
@@ -166,36 +208,101 @@ impl Brush {
.lock()
.map_err(|_| "brush decoder poisoned".to_owned())?
.run(inputs, &["iou_scores", "pred_masks"])?;
- // Three candidate masks, scored; the model's own best guess wins.
- let best = outputs[0]
- .data
- .iter()
- .enumerate()
- .max_by(|a, b| a.1.total_cmp(b.1))
- .map(|(index, _)| index)
- .unwrap_or(0);
+ let scores = &outputs[0].data;
let masks = &outputs[1].data;
let plane = SAM_MASK * SAM_MASK;
- if masks.len() < (best + 1) * plane {
+ let count = scores.len().min(masks.len() / plane.max(1));
+ if count == 0 {
return Err("brush: the model's answer is the wrong size".to_owned());
}
- let logits = &masks[best * plane..(best + 1) * plane];
- // The answer covers the square; the picture is its top-left
+ // Each answer covers the square; the picture is its top-left
// corner. Read the region over the picture alone.
let size = REGION_SIZE as usize;
- let mut bytes = Vec::with_capacity(size * size);
- for y in 0..size {
- let py = ((y as f32 + 0.5) / size as f32) * sh / SAM_SIZE as f32 * SAM_MASK as f32;
- for x in 0..size {
- let px = ((x as f32 + 0.5) / size as f32) * sw / SAM_SIZE as f32 * SAM_MASK as f32;
- let logit = bilinear(logits, SAM_MASK, px - 0.5, py - 0.5);
- let probability = 1.0 / (1.0 + (-logit).exp());
- bytes.push((probability.clamp(0.0, 1.0) * 255.0 + 0.5) as u8);
+ (0..count)
+ .map(|index| {
+ let logits = &masks[index * plane..(index + 1) * plane];
+ let mut bytes = Vec::with_capacity(size * size);
+ for y in 0..size {
+ let py = ((y as f32 + 0.5) / size as f32) * sh / SAM_SIZE as f32
+ * SAM_MASK as f32;
+ for x in 0..size {
+ let px = ((x as f32 + 0.5) / size as f32) * sw / SAM_SIZE as f32
+ * SAM_MASK as f32;
+ let logit = bilinear(logits, SAM_MASK, px - 0.5, py - 0.5);
+ let probability = 1.0 / (1.0 + (-logit).exp());
+ bytes.push((probability.clamp(0.0, 1.0) * 255.0 + 0.5) as u8);
+ }
+ }
+ Mask::from_bytes(REGION_SIZE, REGION_SIZE, bytes)
+ .map(|mask| (scores[index], mask))
+ .ok_or_else(|| "brush: the region is the wrong size".to_owned())
+ })
+ .collect()
+ }
+}
+
+/// Points well inside a region, as source fractions: a handful spread
+/// over its sure pixels, to prompt the model with where the thing was.
+fn interior_points(region: &Mask) -> Vec<[f64; 2]> {
+ /// A pixel this sure is inside, not on the edge.
+ const INSIDE: u8 = 230;
+ /// At most this many prompts, on a grid over the region.
+ const AT_MOST: usize = 9;
+ let (width, height) = (region.width() as usize, region.height() as usize);
+ let bytes = region.bytes();
+ let sure: Vec<(usize, usize)> = (0..height)
+ .flat_map(|y| (0..width).map(move |x| (x, y)))
+ .filter(|&(x, y)| bytes[y * width + x] >= INSIDE)
+ .collect();
+ if sure.is_empty() {
+ return Vec::new();
+ }
+ // The region's box, cut into a grid; the sure pixel nearest each
+ // cell's centre is a prompt, so the prompts cover the thing rather
+ // than cluster.
+ let (x0, x1) = sure.iter().fold((usize::MAX, 0), |(lo, hi), &(x, _)| (lo.min(x), hi.max(x)));
+ let (y0, y1) = sure.iter().fold((usize::MAX, 0), |(lo, hi), &(_, y)| (lo.min(y), hi.max(y)));
+ let cells = (AT_MOST as f64).sqrt().round() as usize;
+ let mut out = Vec::new();
+ for cy in 0..cells {
+ for cx in 0..cells {
+ let tx = x0 as f64 + (x1 - x0) as f64 * (cx as f64 + 0.5) / cells as f64;
+ let ty = y0 as f64 + (y1 - y0) as f64 * (cy as f64 + 0.5) / cells as f64;
+ let nearest = sure
+ .iter()
+ .map(|&(x, y)| ((x as f64 - tx).powi(2) + (y as f64 - ty).powi(2), x, y))
+ .min_by(|a, b| a.0.total_cmp(&b.0));
+ if let Some((distance, x, y)) = nearest {
+ // A cell whose nearest sure pixel is far away is not the
+ // thing's; skip it rather than prompt beside the thing.
+ let reach = ((x1 - x0).max(y1 - y0) as f64 / cells as f64).max(2.0);
+ if distance.sqrt() <= reach {
+ out.push([(x as f64 + 0.5) / width as f64, (y as f64 + 0.5) / height as f64]);
+ }
}
}
- Mask::from_bytes(REGION_SIZE, REGION_SIZE, bytes)
- .ok_or_else(|| "brush: the region is the wrong size".to_owned())
}
+ out.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
+ out.dedup();
+ out
+}
+
+/// Intersection over union of two regions' sure halves, sampled on the
+/// first's grid.
+fn overlap(a: &Mask, b: &Mask) -> f32 {
+ let (width, height) = (a.width(), a.height());
+ let (mut both, mut either) = (0u32, 0u32);
+ for y in 0..height {
+ let v = (y as f32 + 0.5) / height as f32;
+ for x in 0..width {
+ let u = (x as f32 + 0.5) / width as f32;
+ let in_a = a.at(i64::from(x), i64::from(y)) >= 128;
+ let in_b = b.sample(u, v) >= 0.5;
+ both += u32::from(in_a && in_b);
+ either += u32::from(in_a || in_b);
+ }
+ }
+ if either == 0 { 0.0 } else { both as f32 / either as f32 }
}
/// A value from a square plane, bilinear between the four around it,
@@ -225,6 +332,25 @@ mod tests {
assert_eq!(Brush::input_size(500, 500), (1024, 1024));
}
+ #[test]
+ fn interior_points_lie_inside_the_region_and_overlap_is_one_with_itself() {
+ let mut region = Mask::filled(32, 32, 0);
+ for y in 8..24 {
+ for x in 8..24 {
+ region.bytes_mut()[y * 32 + x] = 255;
+ }
+ }
+ let points = interior_points(&region);
+ assert!(!points.is_empty() && points.len() <= 9);
+ for [x, y] in &points {
+ assert!((0.25..0.75).contains(x) && (0.25..0.75).contains(y), "{x} {y}");
+ }
+ assert_eq!(overlap(&region, &region), 1.0);
+ let empty = Mask::filled(32, 32, 0);
+ assert!(interior_points(&empty).is_empty());
+ assert_eq!(overlap(&region, &empty), 0.0);
+ }
+
#[test]
fn bilinear_reads_between_pixels_and_clamps_at_the_edge() {
let plane = [0.0, 1.0, 2.0, 3.0];
diff --git a/engine/crates/concat-vision/src/lib.rs b/engine/crates/concat-vision/src/lib.rs
index 9b28879a..a89c0e8e 100644
--- a/engine/crates/concat-vision/src/lib.rs
+++ b/engine/crates/concat-vision/src/lib.rs
@@ -58,7 +58,7 @@ pub use mask::Mask;
pub use models::ModelId;
#[cfg(feature = "infer")]
pub use segment::Segmenter;
-pub use store::{MaskStore, mask_dir, region_file};
+pub use store::{MaskStore, mask_dir, region_dir};
/// Masks are found this many times a second of source. Ten is where a
/// person's outline stops visibly lagging their movement, and where a
diff --git a/engine/crates/concat-vision/src/store.rs b/engine/crates/concat-vision/src/store.rs
index 551a5d1b..47b2f551 100644
--- a/engine/crates/concat-vision/src/store.rs
+++ b/engine/crates/concat-vision/src/store.rs
@@ -51,11 +51,13 @@ fn model_file(dir: &Path) -> PathBuf {
dir.join("model")
}
-/// The file a smart stroke's region is kept in, under the masks' own
-/// directory: the thing the brush model read under that stroke.
-pub fn region_file(dir: &Path, stroke: &Stroke) -> PathBuf {
+/// The directory a smart stroke's regions are kept in, under the masks'
+/// own directory: the thing the brush model read under that stroke, one
+/// PNG per analysed instant like the masks themselves, since the thing
+/// moves and the region follows it.
+pub fn region_dir(dir: &Path, stroke: &Stroke) -> PathBuf {
dir.join("strokes")
- .join(format!("{:016x}.png", strokes::stroke_key(stroke)))
+ .join(format!("{:016x}", strokes::stroke_key(stroke)))
}
/// The file for the mask at `millis` of source.
@@ -183,9 +185,27 @@ impl MaskStore {
pub fn resolved(&self, seconds: f64, cutout: &Cutout, aspect: f32) -> Option<Arc<Mask>> {
let millis = self.nearest_millis(seconds)?;
let file = mask_file(&self.dir, millis);
+ // The smart strokes' regions, opened once for the key and the
+ // painting both: what is there changes as the brush model works
+ // its way along the clip.
+ let region_stores: Vec<(u64, MaskStore)> = if cutout.mode == CutoutMode::Custom {
+ cutout
+ .strokes
+ .iter()
+ .filter(|stroke| stroke.is_smart())
+ .map(|stroke| {
+ (
+ strokes::stroke_key(stroke),
+ MaskStore::open(&region_dir(&self.dir, stroke)),
+ )
+ })
+ .collect()
+ } else {
+ Vec::new()
+ };
let key = ResolvedKey {
file: file.clone(),
- settings: settings_key(&self.dir, cutout, aspect),
+ settings: settings_key(cutout, aspect, &region_stores, millis),
};
if let Some(hit) = resolved_cache()
.lock()
@@ -201,8 +221,12 @@ impl MaskStore {
}
let painted = match cutout.mode {
CutoutMode::Custom if !cutout.strokes.is_empty() => {
- let dir = self.dir.clone();
- let regions = move |stroke: &Stroke| load(&region_file(&dir, stroke));
+ let regions = move |stroke: &Stroke| {
+ region_stores
+ .iter()
+ .find(|(key, _)| *key == strokes::stroke_key(stroke))
+ .and_then(|(_, store)| store.mask_at(seconds))
+ };
strokes::paint(&auto, &cutout.strokes, aspect, &regions)
}
_ => (*auto).clone(),
@@ -276,8 +300,14 @@ impl MaskStore {
/// What besides the file decides a resolved mask: the mode, the feather
/// and every stroke, hashed, with the aspect the brushes were sized by,
-/// and for each smart stroke whether its region has been read yet.
-fn settings_key(dir: &Path, cutout: &Cutout, aspect: f32) -> u64 {
+/// and for each smart stroke which region, if any, answers for the
+/// instant.
+fn settings_key(
+ cutout: &Cutout,
+ aspect: f32,
+ region_stores: &[(u64, MaskStore)],
+ millis: u64,
+) -> u64 {
use std::hash::{Hash, Hasher};
let mut hasher = std::collections::hash_map::DefaultHasher::new();
(cutout.mode == CutoutMode::Custom).hash(&mut hasher);
@@ -286,9 +316,10 @@ fn settings_key(dir: &Path, cutout: &Cutout, aspect: f32) -> u64 {
if cutout.mode == CutoutMode::Custom {
for stroke in &cutout.strokes {
strokes::stroke_key(stroke).hash(&mut hasher);
- if stroke.is_smart() {
- region_file(dir, stroke).is_file().hash(&mut hasher);
- }
+ }
+ for (key, store) in region_stores {
+ key.hash(&mut hasher);
+ store.nearest_millis(millis as f64 / 1000.0).hash(&mut hasher);
}
}
hasher.finish()
diff --git a/engine/crates/concat/src/studio.rs b/engine/crates/concat/src/studio.rs
index 2b02299f..afd1d381 100644
--- a/engine/crates/concat/src/studio.rs
+++ b/engine/crates/concat/src/studio.rs
@@ -5296,12 +5296,19 @@ impl Studio {
let Some(media) = self.project().media_by_id(&clip.media_id) else {
continue;
};
+ // The source the clip shows, as the mask analysis reckons it.
+ let range = (
+ clip.source_start,
+ clip.source_start + clip.duration * clip.speed.max(0.0625),
+ );
for stroke in &cutout.strokes {
let request = RegionRequest {
project: project.clone(),
media_path: media.path.clone(),
+ media_size: (media.width.unwrap_or(0), media.height.unwrap_or(0)),
still: media.kind == model::MediaKind::Image,
subject: cutout.subject,
+ ranges: vec![range],
stroke: stroke.clone(),
};
if request.outstanding() {