The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.
Rust ffmpegfree-video-editormodel-context-protocolnon-linear-editoroffline-firstopen-source-video-editorself-hostedvideo-editingvideo-editing-automationvideo-editing-softwarevideo-editing-toolsvideo-editorvoice-cloningyoutube-automationyoutube-shorts
Three cutout models: a person matte, an object model, a subject to choose
Jareera54a880543eb
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.mdindex fbf7ae48..49cfafd5 100644--- a/ARCHITECTURE.md+++ b/ARCHITECTURE.md@@ -421,16 +421,22 @@ look repaints.## 10. Cutouts: `concat-vision` and `host::cutout`-A cutout takes a picture's background away without a key colour. The-model is MediaPipe's selfie segmentation, compiled into `concat-vision`-and run by ONNX Runtime: a 256 × 256 picture in, a probability per pixel-out, five milliseconds a frame on an Apple GPU and about twenty on a-laptop's cores. Masks are square-whatever the picture's shape, and every position in one is a fraction of-the source picture, which is also how strokes are stored; a crop, a flip-or a change of output size changes nothing about either, because-`apply::Mapping` is the one place a decoded pixel is walked back to a-source fraction.+A cutout takes a picture's background away without a key colour. Three+models, all run by ONNX Runtime, all one interface in `segment`: a frame+in, a probability per pixel out. Robust Video Matting is the person+model, at the picture's own aspect on a 512 edge, carrying a recurrent+state between frames so an outline holds still; IS-Net is the object+model, 1024 square, for a car or a cartoon; MediaPipe's selfie+segmentation is compiled in and answers when nothing has been+downloaded. A clip's `Subject` picks between the first two, and on+automatic the person model looks at the first frame and hands over to the+object model when it finds no one; the choice is written into the mask+store, so every run agrees. Every position in a mask is a fraction of the+source picture whatever the mask's shape, which is also how strokes are+stored; a crop, a flip or a change of output size changes nothing about+either, because `apply::Mapping` is the one place a decoded pixel is+walked back to a source fraction. A mask the model left blank shows the+picture as shot rather than nothing.**The store is the project's.** Each media file's masks are PNGs in`cache/masks/<hash>-<model>/`, one per analysed source instant, named bydiff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 1a005266..74382913 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -87,13 +87,31 @@ https://huggingface.co/hexgrad/Kokoro-82M) are downloaded on demand from thesherpa-onnx releases - including espeak-ng's data files - and are neverbundled with the app.-## The cutout model--Remove background's automatic and custom modes run Google's MediaPipe-Selfie Segmentation model (Apache-2.0), in the ONNX conversion published by-the ONNX Community (https://huggingface.co/onnx-community/mediapipe_selfie_segmentation,-Apache-2.0). The model file is compiled into the `concat-vision` crate; see-`engine/crates/concat-vision/models/NOTICE.md`. It is run by ONNX Runtime+## The cutout models++Remove background runs three models, none of which ship inside the app+except the first:++- Google's MediaPipe Selfie Segmentation (Apache-2.0), in the ONNX+ conversion published by the ONNX Community+ (https://huggingface.co/onnx-community/mediapipe_selfie_segmentation,+ Apache-2.0), compiled into the `concat-vision` crate; see+ `engine/crates/concat-vision/models/NOTICE.md`. The answer when nothing+ has been downloaded.+- Robust Video Matting, the MobileNetV3 variant, by Peter Lin and others+ (https://github.com/PeterL1n/RobustVideoMatting, GPL-3.0), downloaded on+ first use from that repository's releases. The person model.+- IS-Net from "Highly Accurate Dichotomous Image Segmentation" by Qin and+ others (https://github.com/xuebinqin/DIS, Apache-2.0), in the ONNX+ export the rembg project publishes+ (https://github.com/danielgatis/rembg, MIT), downloaded on first use.+ The object model.+- SlimSAM (https://github.com/czg1225/SlimSAM, Apache-2.0), in the ONNX+ export published at https://huggingface.co/Xenova/slimsam-77-uniform+ (Apache-2.0), downloaded on first use. The brushes' model.++Downloaded models live in the app's data directory under `cutout-models`+and are never bundled. They are all run by ONNX Runtime(https://github.com/microsoft/onnxruntime, MIT) through the `ort` crate(https://github.com/pykeio/ort, MIT OR Apache-2.0), with the platform'sown accelerator behind it: CoreML on macOS and iOS, DirectML on Windows,diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex f16caf51..95575f11 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -1229,6 +1229,7 @@ dependencies = ["memmap2","serde","serde_json",+ "ureq 2.12.1",][[package]]diff --git a/engine/crates/concat-export/src/flatten.rs b/engine/crates/concat-export/src/flatten.rsindex 04cacff6..44cd85ff 100644--- a/engine/crates/concat-export/src/flatten.rs+++ b/engine/crates/concat-export/src/flatten.rs@@ -166,9 +166,11 @@ pub fn flatten_timeline_in(has_audio: Some(media.has_audio),cutout: clip.cutout.clone(),mask_dir: match (&clip.cutout, project_dir) {- (Some(_), Some(dir)) => concat_vision::mask_dir(dir, &media.path)- .to_string_lossy()- .into_owned(),+ (Some(cutout), Some(dir)) => {+ concat_vision::mask_dir(dir, &media.path, cutout.subject)+ .to_string_lossy()+ .into_owned()+ }_ => String::new(),},})diff --git a/engine/crates/concat-host/Cargo.toml b/engine/crates/concat-host/Cargo.tomlindex 767be51c..5df82db3 100644--- a/engine/crates/concat-host/Cargo.toml+++ b/engine/crates/concat-host/Cargo.toml@@ -18,8 +18,10 @@ concat-project.workspace = trueconcat-export.workspace = true# Titles become pictures here; see titles.rs.concat-text.workspace = true-# The cutout model and its masks; `infer` because the host is what runs it.+# The cutout models and their masks; `infer` because the host is what+# runs them, and ureq because the host is what fetches them.concat-vision = { workspace = true, features = ["infer"] }+ureq = "2"concat-render = { workspace = true, optional = true }serde = { version = "1", features = ["derive"] }serde_json.workspace = truediff --git a/engine/crates/concat-host/src/cutout.rs b/engine/crates/concat-host/src/cutout.rsindex 04199a20..2ce6ea70 100644--- a/engine/crates/concat-host/src/cutout.rs+++ b/engine/crates/concat-host/src/cutout.rs@@ -5,24 +5,35 @@//!//! A clip with a cutout needs a mask for every source instant it shows, on//! the analysis grid `concat-vision` reads by. This is the job that fills-//! the gaps: it decodes the media at the model's size along each missing-//! stretch, runs the model on several frames at once, and writes what it-//! finds into the media's mask store in the project folder. A still is one-//! frame; footage is [`concat_vision::MASK_RATE`] frames a second of source.+//! the gaps: it picks the model the clip's subject asks for, fetches it if+//! it is not on disk yet, decodes the media at the model's size along each+//! missing stretch, runs the model frame by frame, and writes what it+//! finds into the media's mask store in the project folder. A still is+//! one frame; footage is [`concat_vision::MASK_RATE`] frames a second of+//! source.+//!+//! Which model: the person model for a person, the object model for an+//! object, and for a subject left on automatic the person model is tried+//! on the first frame and the object model takes over when it finds no+//! one. The choice is written into the store, so the rest of the run and+//! every later one agree. With nothing downloaded and no network, the+//! compiled-in person model answers instead.//!//! One job at a time through a [`SingleFlight`], like every long job the-//! host runs; the window queues the next media behind it. The model loads+//! host runs; the window queues the next media behind it. A model loads//! on first use and stays loaded.-use std::path::PathBuf;-use std::sync::{Arc, OnceLock};+use std::path::{Path, PathBuf};+use std::sync::atomic::{AtomicBool, Ordering};+use std::sync::{Arc, Mutex, OnceLock};-use concat_core::frame::Frame;use concat_core::time::{FrameRate, Rational};use concat_media::{DecodeOptions, Decoder, FrameSource};-use concat_vision::{MASK_RATE, MODEL_SIZE, Mask, MaskStore, Segmenter, mask_dir};+use concat_project::model::Subject;+use concat_vision::segment::Kind;+use concat_vision::{MASK_RATE, MaskStore, ModelId, Segmenter, mask_dir, models};-use crate::jobs::{Job, SingleFlight};+use crate::jobs::SingleFlight;/// What one analysis covers.#[derive(Clone, Debug)]@@ -31,31 +42,51 @@ pub struct AnalyseRequest {pub project: PathBuf,/// The media file to analyse.pub media_path: String,+ /// The media's picture size, for the person model's working size.+ pub media_size: (u32, u32),/// A still: one frame answers for every instant.pub still: bool,+ /// What to keep.+ pub subject: Subject,/// The stretches of source, in seconds, that clips show. Instants/// already analysed inside them are skipped.pub ranges: Vec<(f64, f64)>,}-/// The analysis service: the model, loaded once, and the one-job slot.-pub struct Cutouts {- gate: Arc<SingleFlight>,- segmenter: OnceLock<Result<Arc<Segmenter>, String>>,+/// How far an analysis has got, as it reports it.+#[derive(Clone, Copy, Debug, PartialEq)]+pub enum Progress {+ /// A model is being downloaded: bytes so far, of about this many.+ Fetching {+ /// Bytes received.+ received: u64,+ /// Bytes expected.+ total: u64,+ },+ /// Masks are being found: `0..=1` of the instants wanted.+ Analysing(f32),}-impl Default for Cutouts {- fn default() -> Self {- Self::new()- }+/// The analysis service: the models, loaded once each, and the one-job+/// slot.+pub struct Cutouts {+ gate: Arc<SingleFlight>,+ /// The app's data directory, where downloaded models live.+ data: PathBuf,+ selfie: OnceLock<Result<Arc<Segmenter>, String>>,+ person: Mutex<Option<Arc<Segmenter>>>,+ object: Mutex<Option<Arc<Segmenter>>>,}impl Cutouts {- /// A service with nothing loaded yet.- pub fn new() -> Cutouts {+ /// A service with nothing loaded yet, keeping its models under `data`.+ pub fn new(data: &Path) -> Cutouts {Cutouts {gate: Arc::new(SingleFlight::new()),- segmenter: OnceLock::new(),+ data: data.to_path_buf(),+ selfie: OnceLock::new(),+ person: Mutex::new(None),+ object: Mutex::new(None),}}@@ -64,20 +95,81 @@ impl Cutouts {self.gate.is_busy()}- /// Asks the running analysis to stop after the frames in hand.+ /// Asks the running analysis to stop after the frame in hand.pub fn cancel(&self) {self.gate.cancel();}- fn segmenter(&self) -> Result<Arc<Segmenter>, String> {- self.segmenter- .get_or_init(|| Segmenter::load().map(Arc::new))+ /// Whether a downloadable model is on disk.+ pub fn installed(&self, id: ModelId) -> bool {+ models::installed(&self.data, id)+ }++ fn selfie(&self) -> Result<Arc<Segmenter>, String> {+ self.selfie+ .get_or_init(|| Segmenter::selfie().map(Arc::new)).clone()}+ /// The person or object model, fetched first when it is not on disk.+ fn downloaded(+ &self,+ id: ModelId,+ cancel: &AtomicBool,+ progress: &mut dyn FnMut(Progress),+ ) -> Result<Arc<Segmenter>, String> {+ let slot = match id {+ ModelId::Person => &self.person,+ ModelId::Object => &self.object,+ _ => return Err("not a segmenting model".to_owned()),+ };+ if let Some(loaded) = slot.lock().map_err(|_| "model slot poisoned")?.as_ref() {+ return Ok(Arc::clone(loaded));+ }+ let file = models::model_file(&self.data, id);+ if !models::installed(&self.data, id) {+ fetch(id, &file, cancel, progress)?;+ }+ let segmenter = Arc::new(match id {+ ModelId::Person => Segmenter::person(&file)?,+ _ => Segmenter::object(&file)?,+ });+ *slot.lock().map_err(|_| "model slot poisoned")? = Some(Arc::clone(&segmenter));+ Ok(segmenter)+ }++ /// The person model, or the compiled-in one when the download cannot+ /// happen: a cutout with no network is a worse cutout, not none.+ fn person_or_selfie(+ &self,+ cancel: &AtomicBool,+ progress: &mut dyn FnMut(Progress),+ ) -> Result<Arc<Segmenter>, String> {+ match self.downloaded(ModelId::Person, cancel, progress) {+ Ok(model) => Ok(model),+ Err(error) if error.contains("cancelled") => Err(error),+ Err(_) => self.selfie(),+ }+ }+/// How many instants `request` still needs, without doing anything.+ /// A store made by a model the subject would not pick now needs all+ /// of them again.pub fn outstanding(request: &AnalyseRequest) -> usize {- let store = MaskStore::open(&mask_dir(&request.project, &request.media_path));+ let store = MaskStore::open(&mask_dir(+ &request.project,+ &request.media_path,+ request.subject,+ ));+ let stale = match (request.subject, store.model()) {+ (_, None) => false,+ (Subject::Person, Some(model)) => model == ModelId::Object.name(),+ (Subject::Object, Some(model)) => model != ModelId::Object.name(),+ (Subject::Auto, _) => false,+ };+ if stale {+ return usize::MAX;+ }if request.still {return usize::from(store.is_empty());}@@ -85,32 +177,76 @@ impl Cutouts {}/// Fills the masks `request` is missing. Blocks for the whole run, so- /// run it on its own thread; `progress` is called with `0..=1` as it- /// goes. Returns how many masks were written.+ /// run it on its own thread; `progress` is called as it goes. Returns+ /// how many masks were written.pub fn analyse(&self,request: &AnalyseRequest,- progress: &mut dyn FnMut(f32),+ progress: &mut dyn FnMut(Progress),) -> Result<usize, String> {let job = self.gate.begin("cutout analysis")?;- let segmenter = self.segmenter()?;- let mut store = MaskStore::open(&mask_dir(&request.project, &request.media_path));+ let cancel = job.cancel_handle();+ let mut store = MaskStore::open(&mask_dir(+ &request.project,+ &request.media_path,+ request.subject,+ ));+ if Self::outstanding(request) == usize::MAX {+ store.clear();+ }++ // The model the subject names, or for automatic the one the+ // store already records; the first frame decides otherwise.+ let mut segmenter = match request.subject {+ Subject::Person => Some(self.person_or_selfie(&cancel, progress)?),+ Subject::Object => Some(self.downloaded(ModelId::Object, &cancel, progress)?),+ Subject::Auto => match store.model() {+ Some(model) if model == ModelId::Object.name() => {+ Some(self.downloaded(ModelId::Object, &cancel, progress)?)+ }+ Some(_) => Some(self.person_or_selfie(&cancel, progress)?),+ None => None,+ },+ };++ let (width, height) = request.media_size;+ let size_for = |segmenter: &Segmenter| segmenter.input_size(width, height);if request.still {if !store.is_empty() {return Ok(0);}- let options = DecodeOptions::default()- .scaled_to(MODEL_SIZE, MODEL_SIZE)- .limited_to(1);- let mut decoder =- Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())?;- let frame = decoder- .next_frame()- .map_err(|error| error.to_string())?- .ok_or_else(|| format!("{}: no picture to analyse", request.media_path))?;- store.put(0, &segmenter.mask(&frame)?)?;- progress(1.0);+ let frame_with = |segmenter: &Segmenter| {+ let (w, h) = size_for(segmenter);+ let options = DecodeOptions::default().scaled_to(w, h).limited_to(1);+ let mut decoder = Decoder::open(&request.media_path, &options)+ .map_err(|error| error.to_string())?;+ decoder+ .next_frame()+ .map_err(|error| error.to_string())?+ .ok_or_else(|| format!("{}: no picture to analyse", request.media_path))+ };+ let segmenter = match segmenter {+ Some(segmenter) => segmenter,+ None => {+ let person = self.person_or_selfie(&cancel, progress)?;+ person.begin();+ let mask = person.mask(&frame_with(&person)?)?;+ if mask.is_blank() {+ self.downloaded(ModelId::Object, &cancel, progress)?+ } else {+ store.set_model(person.name())?;+ store.put(0, &mask)?;+ progress(Progress::Analysing(1.0));+ return Ok(1);+ }+ }+ };+ segmenter.begin();+ let mask = segmenter.mask(&frame_with(&segmenter)?)?;+ store.set_model(segmenter.name())?;+ store.put(0, &mask)?;+ progress(Progress::Analysing(1.0));return Ok(1);}@@ -120,54 +256,148 @@ impl Cutouts {return Ok(0);}let step_ms = 1000 / u64::from(MASK_RATE);- let workers = std::thread::available_parallelism()- .map(|count| count.get())- .unwrap_or(1)- .clamp(1, 8);let mut done = 0usize;+ progress(Progress::Analysing(0.0));// Each unbroken run of missing instants is one pass of a decoder- // paced to the analysis grid, so no frame is sought twice.+ // paced to the analysis grid, so no frame is sought twice, and one+ // run of the person model's memory.for run in runs(&wanted, step_ms) {let start = Rational::new(run[0] as i64, 1000);- let options = DecodeOptions::default()- .starting_at(start)- .scaled_to(MODEL_SIZE, MODEL_SIZE)- .at_rate(FrameRate::new(Rational::new(i64::from(MASK_RATE), 1)))- .limited_to(run.len() as u64);- let mut decoder =- Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())?;+ let mut active = match &segmenter {+ Some(segmenter) => Arc::clone(segmenter),+ None => self.person_or_selfie(&cancel, progress)?,+ };+ let open = |segmenter: &Segmenter| {+ let (w, h) = size_for(segmenter);+ let options = DecodeOptions::default()+ .starting_at(start)+ .scaled_to(w, h)+ .at_rate(FrameRate::new(Rational::new(i64::from(MASK_RATE), 1)))+ .limited_to(run.len() as u64);+ Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())+ };+ let mut decoder = open(&active)?;+ active.begin();let mut next = 0usize;while next < run.len() {if job.cancelled() {return Err("cutout analysis cancelled".to_owned());}- // A batch of frames, one per core, found together.- let mut batch: Vec<(u64, Frame)> = Vec::with_capacity(workers);- while batch.len() < workers && next < run.len() {- match decoder.next_frame().map_err(|error| error.to_string())? {- Some(frame) => batch.push((run[next], frame)),- // The file ran out before its stated length: what- // was found is what there is.- None => break,- }- next += 1;- }- if batch.is_empty() {+ let Some(frame) = decoder.next_frame().map_err(|error| error.to_string())? else {+ // The file ran out before its stated length: what was+ // found is what there is.break;+ };+ let mut mask = active.mask(&frame)?;+ if segmenter.is_none() {+ // Automatic, undecided: the person model has just+ // looked at the first frame. Nobody there means the+ // object model takes the whole run, from this frame.+ if mask.is_blank() && active.kind() != Kind::Object {+ active = self.downloaded(ModelId::Object, &cancel, progress)?;+ decoder = open(&active)?;+ active.begin();+ let Some(again) = decoder.next_frame().map_err(|error| error.to_string())?+ else {+ break;+ };+ mask = active.mask(&again)?;+ }+ store.set_model(active.name())?;+ segmenter = Some(Arc::clone(&active));+ } else if store.model().is_none() {+ store.set_model(active.name())?;}- for (millis, mask) in find_masks(&segmenter, &batch, &job)? {- store.put(millis, &mask)?;- done += 1;- }- progress(done as f32 / total as f32);+ store.put(run[next], &mask)?;+ next += 1;+ done += 1;+ progress(Progress::Analysing(done as f32 / total as f32));}}- progress(1.0);+ progress(Progress::Analysing(1.0));Ok(done)}}+/// Streams a model into `file`, by way of a `.part` beside it, reporting+/// every couple of megabytes and stopping when `cancel` is set.+fn fetch(+ id: ModelId,+ file: &Path,+ cancel: &AtomicBool,+ progress: &mut dyn FnMut(Progress),+) -> Result<(), String> {+ use std::io::{Read, Write};++ let spec = id.spec();+ if let Some(parent) = file.parent() {+ std::fs::create_dir_all(parent)+ .map_err(|error| format!("could not create {}: {error}", parent.display()))?;+ }+ let partial = file.with_extension("part");+ progress(Progress::Fetching {+ received: 0,+ total: spec.bytes,+ });++ // A stalled connection blocks in `read` with the cancel flag+ // unreachable; thirty seconds without a byte means it is dead.+ let agent = ureq::AgentBuilder::new()+ .timeout_connect(std::time::Duration::from_secs(20))+ .timeout_read(std::time::Duration::from_secs(30))+ .build();+ let response = agent+ .get(spec.url)+ .call()+ .map_err(|error| format!("could not fetch the {} model: {error}", spec.file))?;+ let total = response+ .header("Content-Length")+ .and_then(|value| value.parse::<u64>().ok())+ .unwrap_or(spec.bytes);++ let result = (|| {+ let mut out = std::fs::File::create(&partial)+ .map_err(|error| format!("could not create {}: {error}", partial.display()))?;+ let mut reader = response.into_reader();+ let mut buffer = [0u8; 64 * 1024];+ let mut received = 0u64;+ let mut reported = 0u64;+ loop {+ if cancel.load(Ordering::Relaxed) {+ return Err("cutout analysis cancelled".to_owned());+ }+ let read = reader+ .read(&mut buffer)+ .map_err(|error| format!("model download interrupted: {error}"))?;+ if read == 0 {+ break;+ }+ out.write_all(&buffer[..read])+ .map_err(|error| format!("could not write the model: {error}"))?;+ received += read as u64;+ if received - reported >= 2 * 1024 * 1024 {+ reported = received;+ progress(Progress::Fetching { received, total });+ }+ }+ out.flush()+ .map_err(|error| format!("could not write the model: {error}"))?;+ if received == 0 {+ return Err(format!("the {} model came back empty", spec.file));+ }+ Ok(())+ })();+ match result {+ Ok(()) => std::fs::rename(&partial, file)+ .map_err(|error| format!("could not finish {}: {error}", file.display())),+ Err(error) => {+ let _ = std::fs::remove_file(&partial);+ Err(error)+ }+ }+}+/// Every instant the ranges need that the store lacks, ascending, once.fn missing(store: &MaskStore, ranges: &[(f64, f64)]) -> Vec<u64> {let mut wanted: Vec<u64> = ranges@@ -191,34 +421,6 @@ fn runs(instants: &[u64], step: u64) -> Vec<Vec<u64>> {out}-/// The masks of a batch of frames, one thread each.-fn find_masks(- segmenter: &Arc<Segmenter>,- batch: &[(u64, Frame)],- job: &Job,-) -> Result<Vec<(u64, Mask)>, String> {- if job.cancelled() {- return Err("cutout analysis cancelled".to_owned());- }- std::thread::scope(|scope| {- let handles: Vec<_> = batch- .iter()- .map(|(millis, frame)| {- let segmenter = Arc::clone(segmenter);- scope.spawn(move || segmenter.mask(frame).map(|mask| (*millis, mask)))- })- .collect();- handles- .into_iter()- .map(|handle| {- handle- .join()- .map_err(|_| "cutout analysis panicked".to_owned())?- })- .collect()- })-}-#[cfg(test)]mod tests {use super::*;diff --git a/engine/crates/concat-project/src/model.rs b/engine/crates/concat-project/src/model.rsindex d3aebc40..b6a673cc 100644--- a/engine/crates/concat-project/src/model.rs+++ b/engine/crates/concat-project/src/model.rs@@ -155,6 +155,9 @@ fn yes() -> bool {pub struct Cutout {/// Automatic keeps what the model finds; custom adds the strokes.pub mode: CutoutMode,+ /// What the model looks for.+ #[serde(default, skip_serializing_if = "Subject::is_auto")]+ pub subject: Subject,/// How far the edge is softened, as a fraction of the picture's width.#[serde(default = "default_feather")]pub feather: f64,@@ -173,6 +176,37 @@ pub enum CutoutMode {Custom,}+/// What a cutout keeps: the person the matting model finds, the one+/// thing the picture is of whatever it is, or whichever of those the+/// analysis decides fits the footage.+#[derive(Clone, Copy, PartialEq, Eq, Debug, Default, Serialize, Deserialize)]+#[serde(rename_all = "lowercase")]+pub enum Subject {+ /// A person when the person model finds one, the object otherwise.+ #[default]+ Auto,+ /// The person model, always.+ Person,+ /// The object model, always.+ Object,+}++impl Subject {+ /// The default, left out of the document.+ pub fn is_auto(&self) -> bool {+ *self == Subject::Auto+ }++ /// The name the mask cache is keyed by.+ pub fn key(&self) -> &'static str {+ match self {+ Subject::Auto => "auto",+ Subject::Person => "person",+ Subject::Object => "object",+ }+ }+}+/// One brush stroke over a cutout: which tool, how wide, and where it went./// Points are fractions of the source picture, `(0, 0)` its top-left, so a/// stroke survives a change of output size, a crop or a flip.@@ -219,6 +253,7 @@ impl Cutout {pub fn auto() -> Cutout {Cutout {mode: CutoutMode::Auto,+ subject: Subject::Auto,feather: DEFAULT_FEATHER,strokes: Vec::new(),}diff --git a/engine/crates/concat-vision/models/NOTICE.md b/engine/crates/concat-vision/models/NOTICE.mdindex ffbf636c..4e64def3 100644--- a/engine/crates/concat-vision/models/NOTICE.md+++ b/engine/crates/concat-vision/models/NOTICE.md@@ -1,4 +1,7 @@-# The cutout model+# The compiled-in cutout model++The downloaded models are listed in THIRD_PARTY_NOTICES.md at the+repository root and named in `src/models.rs`.`selfie-segmentation.onnx` is Google's MediaPipe Selfie Segmentation model(the general, 256 × 256 variant), converted to ONNX and published by thediff --git a/engine/crates/concat-vision/src/lib.rs b/engine/crates/concat-vision/src/lib.rsindex 9da8abfa..18cde303 100644--- a/engine/crates/concat-vision/src/lib.rs+++ b/engine/crates/concat-vision/src/lib.rs@@ -19,22 +19,26 @@//! file and the source instant, and cached in the project folder like//! its waveforms, so a cutout is found once and travels with the edit.//!-//! Finding the mask is [`segment`], behind the `infer` feature: the model-//! is compiled in and run by ONNX Runtime through [`runtime`], on the-//! platform's accelerator where there is one, so a cutout needs no-//! download, on a desk or a phone. The renderer reads masks and never-//! infers; the host infers and writes them.+//! Finding the mask is [`segment`], behind the `infer` feature: three+//! models run by ONNX Runtime through [`runtime`], on the platform's+//! accelerator where there is one. A person matting model and a general+//! object model are downloaded on first use ([`models`] says where from);+//! a small person model is compiled in so a cutout works with nothing+//! downloaded at all. The renderer reads masks and never infers; the+//! host infers and writes them.//!-//! Masks are square at the model's resolution whatever the picture's-//! shape, and every position in them is a fraction of the source picture:-//! `(0, 0)` its top-left, `(1, 1)` its bottom-right. Strokes are stored in-//! the same fractions. A crop, a flip or a change of output size therefore-//! changes nothing about a mask - the mapping from a decoded pixel back to-//! a source fraction is [`apply::Mapping`], and it is the one place those+//! A mask is whatever shape its model answers in - square for the object+//! model, the picture's own aspect for the person model - and every+//! position in one is a fraction of the source picture: `(0, 0)` its+//! top-left, `(1, 1)` its bottom-right. Strokes are stored in the same+//! fractions. A crop, a flip or a change of output size therefore changes+//! nothing about a mask - the mapping from a decoded pixel back to a+//! source fraction is [`apply::Mapping`], and it is the one place those//! are undone.pub mod apply;pub mod mask;+pub mod models;#[cfg(feature = "infer")]pub mod runtime;#[cfg(feature = "infer")]@@ -44,6 +48,7 @@ pub mod strokes;pub use apply::{Mapping, cut};pub use mask::Mask;+pub use models::ModelId;#[cfg(feature = "infer")]pub use segment::Segmenter;pub use store::{MaskStore, mask_dir};@@ -53,9 +58,9 @@ pub use store::{MaskStore, mask_dir};/// minute of footage is six hundred inferences rather than eighteen hundred.pub const MASK_RATE: u32 = 10;-/// The model's input and output edge, in pixels.+/// The compiled-in person model's input and output edge, in pixels.pub const MODEL_SIZE: u32 = 256;-/// Names the model a mask came from. Part of the cache directory's name,-/// so a different model never serves another's masks.+/// Names the compiled-in model. A mask store records which model made+/// its masks, so a different model never serves another's.pub const MODEL_ID: &str = "selfie-256";diff --git a/engine/crates/concat-vision/src/mask.rs b/engine/crates/concat-vision/src/mask.rsindex 97b7e098..7be3ab67 100644--- a/engine/crates/concat-vision/src/mask.rs+++ b/engine/crates/concat-vision/src/mask.rs@@ -5,6 +5,10 @@use std::io::Cursor;+/// A mask with every value under this is nobody's: the model found+/// nothing, and the picture should show as shot rather than vanish.+const BLANK_BELOW: u8 = 10;+/// An eight-bit picture of probabilities, `0` background and `255` subject,/// covering the whole source picture whatever its shape.#[derive(Clone, PartialEq, Eq, Debug)]@@ -55,6 +59,37 @@ impl Mask {&mut self.data}+ /// True when the model found nothing: no pixel is more than a few+ /// percent likely to be the subject.+ pub fn is_blank(&self) -> bool {+ self.data.iter().all(|&v| v < BLANK_BELOW)+ }++ /// The mask shrunk by `factor` in each direction, each new pixel the+ /// mean of the block it replaces. A large model's answer stored at a+ /// size the renderer can read quickly.+ pub fn shrunk(&self, factor: u32) -> Mask {+ let factor = factor.max(1);+ if factor == 1 {+ return self.clone();+ }+ let width = (self.width / factor).max(1);+ let height = (self.height / factor).max(1);+ let mut data = Vec::with_capacity((width * height) as usize);+ for y in 0..height {+ for x in 0..width {+ let mut sum = 0u32;+ for dy in 0..factor {+ for dx in 0..factor {+ sum += u32::from(self.at(i64::from(x * factor + dx), i64::from(y * factor + dy)));+ }+ }+ data.push((sum / (factor * factor)) as u8);+ }+ }+ Mask { width, height, data }+ }+/// The value at a pixel, clamped to the edge.pub fn at(&self, x: i64, y: i64) -> u8 {let x = x.clamp(0, i64::from(self.width) - 1) as usize;diff --git a/engine/crates/concat-vision/src/models.rs b/engine/crates/concat-vision/src/models.rsnew file mode 100644index 00000000..1f9bb98b--- /dev/null+++ b/engine/crates/concat-vision/src/models.rs@@ -0,0 +1,127 @@+// SPDX-License-Identifier: AGPL-3.0-or-later+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors++//! The models that are downloaded rather than compiled in: what they are+//! called, where they come from, and where they live once fetched.+//!+//! Each is one ONNX file in `<app data>/cutout-models/`, named as the+//! table says; a file that is there and whole is an installed model, and+//! a fetch lands in a `.part` beside it and renames at the end, so a+//! killed download leaves nothing that could be mistaken for one. The+//! fetching itself is the host's, since it is the host that has a network+//! and a job slot; this crate only knows the table.++use std::path::{Path, PathBuf};++/// The downloadable models.+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]+pub enum ModelId {+ /// Robust Video Matting on MobileNetV3: a person, with a true alpha+ /// edge, carrying state from one frame to the next.+ Person,+ /// IS-Net from the DIS paper: the one thing a picture is of, whatever+ /// it is.+ Object,+ /// SlimSAM's image encoder: a picture into the embeddings the brush+ /// decoder reads.+ BrushEncoder,+ /// SlimSAM's prompt encoder and mask decoder: points on the picture+ /// into the mask of what is under them.+ BrushDecoder,+}++/// One model as the table describes it.+#[derive(Clone, Copy, Debug)]+pub struct ModelSpec {+ /// Which model.+ pub id: ModelId,+ /// The file name on disk, and what a mask store records.+ pub file: &'static str,+ /// Where it is fetched from.+ pub url: &'static str,+ /// Its size, for a progress bar before the server says.+ pub bytes: u64,+ /// The licence it comes under, for the notices.+ pub licence: &'static str,+}++/// Every downloadable model.+pub const MODELS: [ModelSpec; 4] = [+ ModelSpec {+ id: ModelId::Person,+ file: "rvm-mobilenetv3.onnx",+ url: "https://github.com/PeterL1n/RobustVideoMatting/releases/download/v1.0.0/rvm_mobilenetv3_fp32.onnx",+ bytes: 14_975_696,+ licence: "GPL-3.0",+ },+ ModelSpec {+ id: ModelId::Object,+ file: "isnet-general-use.onnx",+ url: "https://github.com/danielgatis/rembg/releases/download/v0.0.0/isnet-general-use.onnx",+ bytes: 178_648_008,+ licence: "Apache-2.0",+ },+ ModelSpec {+ id: ModelId::BrushEncoder,+ file: "slimsam-77-encoder.onnx",+ url: "https://huggingface.co/Xenova/slimsam-77-uniform/resolve/main/onnx/vision_encoder.onnx",+ bytes: 23_276_014,+ licence: "Apache-2.0",+ },+ ModelSpec {+ id: ModelId::BrushDecoder,+ file: "slimsam-77-decoder.onnx",+ url: "https://huggingface.co/Xenova/slimsam-77-uniform/resolve/main/onnx/prompt_encoder_mask_decoder.onnx",+ bytes: 16_557_892,+ licence: "Apache-2.0",+ },+];++impl ModelId {+ /// The table's row for this model.+ pub fn spec(self) -> &'static ModelSpec {+ MODELS+ .iter()+ .find(|spec| spec.id == self)+ .expect("every model is in the table")+ }++ /// The name a mask store records for masks this model made.+ pub fn name(self) -> &'static str {+ match self {+ ModelId::Person => "rvm-mobilenetv3",+ ModelId::Object => "isnet-1024",+ ModelId::BrushEncoder => "slimsam-77-encoder",+ ModelId::BrushDecoder => "slimsam-77-decoder",+ }+ }+}++/// Where the models live: `<app data>/cutout-models/`.+pub fn models_dir(data: &Path) -> PathBuf {+ data.join("cutout-models")+}++/// The file a model is, once fetched.+pub fn model_file(data: &Path, id: ModelId) -> PathBuf {+ models_dir(data).join(id.spec().file)+}++/// Whether a model is on disk and whole.+pub fn installed(data: &Path, id: ModelId) -> bool {+ std::fs::metadata(model_file(data, id)).is_ok_and(|meta| meta.is_file() && meta.len() > 0)+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn every_model_has_its_own_file_and_name() {+ let files: std::collections::HashSet<_> = MODELS.iter().map(|m| m.file).collect();+ assert_eq!(files.len(), MODELS.len());+ let names: std::collections::HashSet<_> = MODELS.iter().map(|m| m.id.name()).collect();+ assert_eq!(names.len(), MODELS.len());+ assert!(!installed(Path::new("/nowhere"), ModelId::Person));+ }+}diff --git a/engine/crates/concat-vision/src/runtime.rs b/engine/crates/concat-vision/src/runtime.rsindex 128dd092..a4cb4f42 100644--- a/engine/crates/concat-vision/src/runtime.rs+++ b/engine/crates/concat-vision/src/runtime.rs@@ -102,13 +102,28 @@ impl Model {/// cannot supply is skipped, not an error, so the same build runs on a/// box without one.fn accelerators() -> Vec<ort::ep::ExecutionProviderDispatch> {- let mut providers = Vec::new();- #[cfg(target_vendor = "apple")]- providers.push(ort::ep::CoreML::default().build());- #[cfg(target_os = "windows")]- providers.push(ort::ep::DirectML::default().build());- #[cfg(target_os = "android")]- providers.push(ort::ep::NNAPI::default().build());- providers.push(ort::ep::CPU::default().build());- providers+ platform_accelerator()+ .into_iter()+ .chain(std::iter::once(ort::ep::CPU::default().build()))+ .collect()+}++#[cfg(target_vendor = "apple")]+fn platform_accelerator() -> Option<ort::ep::ExecutionProviderDispatch> {+ Some(ort::ep::CoreML::default().build())+}++#[cfg(target_os = "windows")]+fn platform_accelerator() -> Option<ort::ep::ExecutionProviderDispatch> {+ Some(ort::ep::DirectML::default().build())+}++#[cfg(target_os = "android")]+fn platform_accelerator() -> Option<ort::ep::ExecutionProviderDispatch> {+ Some(ort::ep::NNAPI::default().build())+}++#[cfg(not(any(target_vendor = "apple", target_os = "windows", target_os = "android")))]+fn platform_accelerator() -> Option<ort::ep::ExecutionProviderDispatch> {+ None}diff --git a/engine/crates/concat-vision/src/segment.rs b/engine/crates/concat-vision/src/segment.rsindex e4dab2f6..cf46910c 100644--- a/engine/crates/concat-vision/src/segment.rs+++ b/engine/crates/concat-vision/src/segment.rs@@ -1,93 +1,290 @@// SPDX-License-Identifier: AGPL-3.0-or-later// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors-//! Finding the person in a picture.+//! Finding the subject in a picture.//!-//! The model is MediaPipe's selfie segmentation, compiled into the binary-//! (see `models/NOTICE.md`) and run by ONNX Runtime. It takes a 256 × 256-//! RGB picture and answers with a probability per pixel; a source of any-//! shape is squashed to the square on the way in and the mask is read as-//! covering the whole source on the way out, which is the convention every-//! other part of this crate keeps.+//! Three models, one interface: a frame in, a probability per pixel out.//!-//! One inference is a few milliseconds with the platform's accelerator and-//! about twenty on a laptop's cores without.+//! - **Selfie**: MediaPipe's selfie segmentation, compiled into the binary+//! (see `models/NOTICE.md`). A person, at 256 × 256, in a few+//! milliseconds; the answer when nothing has been downloaded.+//! - **Person**: Robust Video Matting on MobileNetV3. A person with a true+//! alpha edge - hair, motion blur - at the picture's own aspect, carrying+//! a recurrent state from one frame to the next so an outline does not+//! flicker. Frames must therefore arrive in order: [`Segmenter::begin`]+//! starts a run.+//! - **Object**: IS-Net. The one thing the picture is of, whatever it is,+//! at 1024 × 1024; the answer for a car, a product, a cartoon.+//!+//! A source of any shape is resampled to the model's size on the way in+//! and the mask is read as covering the whole source on the way out, which+//! is the convention every other part of this crate keeps.+use std::path::Path;use std::sync::Mutex;use concat_core::frame::{BYTES_PER_PIXEL, Frame};-use crate::runtime::{Input, Model};-use crate::{MODEL_SIZE, Mask};+use crate::models::ModelId;+use crate::runtime::{Input, Model, Output};+use crate::{MODEL_ID, MODEL_SIZE, Mask};++/// The compiled-in model.+const SELFIE: &[u8] = include_bytes!("../models/selfie-segmentation.onnx");-/// The model as the crate ships it.-const MODEL: &[u8] = include_bytes!("../models/selfie-segmentation.onnx");+/// The person model's long edge: its stage one works at up to 512, and a+/// larger frame would only be shrunk inside it.+const PERSON_EDGE: u32 = 512;+/// The object model's fixed input.+const OBJECT_SIZE: u32 = 1024;+/// The object model's answer is stored at half its size: a quarter of the+/// bytes, and still twice the person model's.+const OBJECT_SHRINK: u32 = 2;++/// Which model a segmenter runs.+#[derive(Clone, Copy, PartialEq, Eq, Debug)]+pub enum Kind {+ /// The compiled-in person model.+ Selfie,+ /// The downloaded person matting model.+ Person,+ /// The downloaded object model.+ Object,+}-/// A loaded model, ready to run. `Send + Sync`: one per process is enough,-/// and the runtime's own threads take a frame across the cores.+/// A loaded model, ready to run. `Send + Sync`; the runtime's own threads+/// take a frame across the cores, so one per process is enough.pub struct Segmenter {+ kind: Kind,model: Mutex<Model>,+ /// The person model's recurrent state from the frame before, within+ /// one run.+ state: Mutex<Option<Vec<Output>>>,}impl Segmenter {- /// Loads the compiled-in model. A few tens of milliseconds, once.+ /// The compiled-in person model. A few tens of milliseconds, once.+ pub fn selfie() -> Result<Segmenter, String> {+ Ok(Segmenter {+ kind: Kind::Selfie,+ model: Mutex::new(Model::from_bytes(SELFIE)?),+ state: Mutex::new(None),+ })+ }++ /// The compiled-in model; see [`Segmenter::selfie`].pub fn load() -> Result<Segmenter, String> {+ Segmenter::selfie()+ }++ /// The person matting model, from its downloaded file.+ pub fn person(file: &Path) -> Result<Segmenter, String> {+ Ok(Segmenter {+ kind: Kind::Person,+ model: Mutex::new(Model::from_file(file)?),+ state: Mutex::new(None),+ })+ }++ /// The object model, from its downloaded file.+ pub fn object(file: &Path) -> Result<Segmenter, String> {Ok(Segmenter {- model: Mutex::new(Model::from_bytes(MODEL)?),+ kind: Kind::Object,+ model: Mutex::new(Model::from_file(file)?),+ state: Mutex::new(None),})}- /// The person mask of `frame`, at the model's resolution. A frame of- /// any size: it is resampled to the square first.+ /// Which model this is.+ pub fn kind(&self) -> Kind {+ self.kind+ }++ /// The name a mask store records for this model's masks.+ pub fn name(&self) -> &'static str {+ match self.kind {+ Kind::Selfie => MODEL_ID,+ Kind::Person => ModelId::Person.name(),+ Kind::Object => ModelId::Object.name(),+ }+ }++ /// The size to decode a source of `width` × `height` to for this+ /// model: its fixed square, or the source's shape at the person model's+ /// working edge.+ pub fn input_size(&self, width: u32, height: u32) -> (u32, u32) {+ match self.kind {+ Kind::Selfie => (MODEL_SIZE, MODEL_SIZE),+ Kind::Object => (OBJECT_SIZE, OBJECT_SIZE),+ Kind::Person => {+ let (width, height) = (width.max(1) as f64, height.max(1) as f64);+ let scale = f64::from(PERSON_EDGE) / width.max(height);+ // Multiples of sixteen, so every stride inside the network+ // divides evenly and nothing is padded off the edge.+ let round = |edge: f64| (((edge * scale) / 16.0).round() as u32 * 16).max(64);+ (round(width), round(height))+ }+ }+ }++ /// Starts a run of frames in source order. The person model forgets+ /// what it carried from the frame before; the others carry nothing.+ pub fn begin(&self) {+ if let Ok(mut state) = self.state.lock() {+ *state = None;+ }+ }++ /// The subject mask of `frame`. A frame of any size: it is resampled+ /// to the model's size first, so the decoder is best asked for+ /// [`Segmenter::input_size`] and the resample is a copy.pub fn mask(&self, frame: &Frame) -> Result<Mask, String> {- let size = MODEL_SIZE as usize;- let width = frame.width() as usize;- let height = frame.height() as usize;- if width == 0 || height == 0 {+ if frame.width() == 0 || frame.height() == 0 {return Err("cutout: an empty frame".to_owned());}- let pixels = frame.pixels();- let mut data = vec![0f32; 3 * size * size];- for c in 0..3 {- for y in 0..size {- // Nearest sample: the decoder already scaled to the square- // in the analysis path, and a still is fine at this size.- let sy = (y * height / size).min(height - 1);- for x in 0..size {- let sx = (x * width / size).min(width - 1);- data[(c * size + y) * size + x] =- f32::from(pixels[(sy * width + sx) * BYTES_PER_PIXEL + c]) / 255.0;- }- }+ let (width, height) = self.input_size(frame.width(), frame.height());+ let planes = planes(frame, width, height);+ match self.kind {+ Kind::Selfie => self.selfie_mask(planes, width, height),+ Kind::Object => self.object_mask(planes, width, height),+ Kind::Person => self.person_mask(planes, width, height),}+ }++ fn run(&self, inputs: Vec<Input<'_>>, outputs: &[&str]) -> Result<Vec<Output>, String> {+ self.model+ .lock()+ .map_err(|_| "cutout model poisoned".to_owned())?+ .run(inputs, outputs)+ }++ fn selfie_mask(&self, planes: Vec<f32>, width: u32, height: u32) -> Result<Mask, String> {+ let size = (width * height) as usize;let input = Input {name: "pixel_values",- dims: vec![1, 3, size, size],- data,+ dims: vec![1, 3, height as usize, width as usize],+ data: planes,};- let outputs = self- .model- .lock()- .map_err(|_| "cutout model poisoned".to_owned())?- .run(vec![input], &["alphas"])?;- let data: Vec<u8> = outputs[0]- .data+ let outputs = self.run(vec![input], &["alphas"])?;+ probabilities(&outputs[0].data, size, width, height)+ }++ fn object_mask(&self, mut planes: Vec<f32>, width: u32, height: u32) -> Result<Mask, String> {+ // As the model was trained: the picture scaled by its brightest+ // value, then centred on a half.+ let brightest = planes.iter().copied().fold(0.0f32, f32::max).max(1e-6);+ for value in &mut planes {+ *value = *value / brightest - 0.5;+ }+ let size = (width * height) as usize;+ let input = Input {+ name: "input_image",+ dims: vec![1, 3, height as usize, width as usize],+ data: planes,+ };+ let outputs = self.run(vec![input], &["output_image"])?;+ // The model answers in its own range; the span of the answer is+ // the mask, as the reference implementation reads it.+ let data = &outputs[0].data[..size.min(outputs[0].data.len())];+ let low = data.iter().copied().fold(f32::INFINITY, f32::min);+ let high = data.iter().copied().fold(f32::NEG_INFINITY, f32::max);+ let span = (high - low).max(1e-6);+ let bytes: Vec<u8> = data.iter()- .take(size * size)- .map(|&p: &f32| (p.clamp(0.0, 1.0) * 255.0 + 0.5) as u8)+ .map(|&p| (((p - low) / span).clamp(0.0, 1.0) * 255.0 + 0.5) as u8).collect();- Mask::from_bytes(MODEL_SIZE, MODEL_SIZE, data)- .ok_or_else(|| "cutout: the model's answer is the wrong size".to_owned())+ let mask = Mask::from_bytes(width, height, bytes)+ .ok_or_else(|| "cutout: the model's answer is the wrong size".to_owned())?;+ Ok(mask.shrunk(OBJECT_SHRINK))+ }++ fn person_mask(&self, planes: Vec<f32>, width: u32, height: u32) -> Result<Mask, String> {+ let size = (width * height) as usize;+ let mut inputs = vec![Input {+ name: "src",+ dims: vec![1, 3, height as usize, width as usize],+ data: planes,+ }];+ let carried = self+ .state+ .lock()+ .map_err(|_| "cutout state poisoned".to_owned())?+ .take();+ const STATES: [&str; 4] = ["r1i", "r2i", "r3i", "r4i"];+ match carried {+ Some(state) if state.len() == 4 => {+ for (name, output) in STATES.iter().zip(state) {+ inputs.push(Input {+ name,+ dims: output.dims,+ data: output.data,+ });+ }+ }+ _ => {+ for name in STATES {+ inputs.push(Input {+ name,+ dims: vec![1, 1, 1, 1],+ data: vec![0.0],+ });+ }+ }+ }+ // The frame is already at the working size; the model's own+ // downsample stays at one.+ inputs.push(Input {+ name: "downsample_ratio",+ dims: vec![1],+ data: vec![1.0],+ });+ let mut outputs = self.run(inputs, &["pha", "r1o", "r2o", "r3o", "r4o"])?;+ let alpha = outputs.remove(0);+ if let Ok(mut state) = self.state.lock() {+ *state = Some(outputs);+ }+ probabilities(&alpha.data, size, width, height)}}+/// The frame as the models read it: three planes of `0..=1`, red then+/// green then blue, at `width` × `height`, nearest-sampled from the frame.+fn planes(frame: &Frame, width: u32, height: u32) -> Vec<f32> {+ let (fw, fh) = (frame.width() as usize, frame.height() as usize);+ let (width, height) = (width as usize, height as usize);+ let pixels = frame.pixels();+ let mut data = vec![0f32; 3 * width * height];+ for c in 0..3 {+ for y in 0..height {+ let sy = (y * fh / height).min(fh - 1);+ for x in 0..width {+ let sx = (x * fw / width).min(fw - 1);+ data[(c * height + y) * width + x] =+ f32::from(pixels[(sy * fw + sx) * BYTES_PER_PIXEL + c]) / 255.0;+ }+ }+ }+ data+}++/// A model's `0..=1` answer as a mask.+fn probabilities(data: &[f32], size: usize, width: u32, height: u32) -> Result<Mask, String> {+ let bytes: Vec<u8> = data+ .iter()+ .take(size)+ .map(|&p| (p.clamp(0.0, 1.0) * 255.0 + 0.5) as u8)+ .collect();+ Mask::from_bytes(width, height, bytes)+ .ok_or_else(|| "cutout: the model's answer is the wrong size".to_owned())+}+#[cfg(test)]mod tests {use super::*;#[test]fn the_model_loads_and_answers_a_mask_of_its_size() {- let segmenter = Segmenter::load().expect("loads");+ let segmenter = Segmenter::selfie().expect("loads");// A flat grey picture is nobody: the mask should lean background.let mut frame = Frame::black(64, 48);for pixel in frame.pixels_mut().chunks_exact_mut(4) {@@ -98,4 +295,16 @@ mod tests {let mean = mask.bytes().iter().map(|&v| u32::from(v)).sum::<u32>() / (256 * 256);assert!(mean < 128, "a blank picture read as mostly person: {mean}");}++ #[test]+ fn the_person_model_works_at_the_pictures_shape() {+ let segmenter = Segmenter {+ kind: Kind::Person,+ model: Mutex::new(Model::from_bytes(SELFIE).expect("loads")),+ state: Mutex::new(None),+ };+ assert_eq!(segmenter.input_size(1920, 1080), (512, 288));+ assert_eq!(segmenter.input_size(1080, 1920), (288, 512));+ assert_eq!(segmenter.input_size(1000, 1000), (512, 512));+ }}diff --git a/engine/crates/concat-vision/src/store.rs b/engine/crates/concat-vision/src/store.rsindex 2aa52eca..8c49214a 100644--- a/engine/crates/concat-vision/src/store.rs+++ b/engine/crates/concat-vision/src/store.rs@@ -3,12 +3,13 @@//! Where masks live between runs.//!-//! One directory per media file per model, inside the project's `cache/`+//! One directory per media file per subject, inside the project's `cache/`//! folder so it travels with the project and vanishes with it, holding one-//! PNG per analysed source instant, named by that instant in milliseconds.-//! The host writes them as it analyses; the renderer opens the directory,-//! reads the names, and asks for the mask nearest a source time. Nothing-//! is ever indexed: the file names are the index.+//! PNG per analysed source instant, named by that instant in milliseconds,+//! and a `model` file naming the model that made them. The host writes+//! them as it analyses; the renderer opens the directory, reads the names,+//! and asks for the mask nearest a source time. Nothing is ever indexed:+//! the file names are the index.//!//! Decoded masks are kept in a process-wide cache, since a preview asks for//! the same handful over and over while the playhead sits still, and an@@ -18,9 +19,9 @@ use std::collections::{HashMap, VecDeque};use std::path::{Path, PathBuf};use std::sync::{Arc, Mutex, OnceLock};-use concat_project::model::{Cutout, CutoutMode};+use concat_project::model::{Cutout, CutoutMode, Subject};-use crate::{MASK_RATE, MODEL_ID, Mask, strokes};+use crate::{MASK_RATE, Mask, strokes};/// A mask further than this from the instant asked for is nobody's answer:/// two and a half analysis steps, so a gap in the cache reads as "not yet"@@ -31,17 +32,23 @@ const REACH_MS: u64 = 2500 / MASK_RATE as u64;/// mask is 64 KB; both together stay under forty megabytes.const CACHED: usize = 300;-/// The directory a media file's masks live in under `project`.+/// The directory a media file's masks for `subject` live in under+/// `project`.////// Named by a hash of the path rather than the path, like the peaks cache/// and for the same reason: a flat, portable name that cannot escape the-/// folder. The model's id is in it too, so a different model never-/// answers for this one.-pub fn mask_dir(project: &Path, media_path: &str) -> PathBuf {+/// folder. The subject is in it too, so a clip asked to keep the person+/// and one asked to keep the object never read each other's answer.+pub fn mask_dir(project: &Path, media_path: &str, subject: Subject) -> PathBuf {project.join("cache").join("masks")- .join(format!("{:016x}-{MODEL_ID}", fnv1a(media_path.as_bytes())))+ .join(format!("{:016x}-{}", fnv1a(media_path.as_bytes()), subject.key()))+}++/// The file naming the model that made a directory's masks.+fn model_file(dir: &Path) -> PathBuf {+ dir.join("model")}/// The file for the mask at `millis` of source.@@ -65,6 +72,8 @@ pub struct MaskStore {dir: PathBuf,/// The analysed instants, ascending, in milliseconds of source.times: Vec<u64>,+ /// The model that made them, as the directory's `model` file says.+ model: Option<String>,}impl MaskStore {@@ -83,9 +92,14 @@ impl MaskStore {.collect();times.sort_unstable();times.dedup();+ let model = std::fs::read_to_string(model_file(dir))+ .ok()+ .map(|name| name.trim().to_owned())+ .filter(|name| !name.is_empty());MaskStore {dir: dir.to_path_buf(),times,+ model,}}@@ -94,6 +108,28 @@ impl MaskStore {&self.dir}+ /// The model that made the masks here, once one has.+ pub fn model(&self) -> Option<&str> {+ self.model.as_deref()+ }++ /// Records the model about to fill the store. Masks from another+ /// model go first: one directory, one model.+ pub fn set_model(&mut self, name: &str) -> Result<(), String> {+ if self.model.as_deref() == Some(name) {+ return Ok(());+ }+ if self.model.is_some() {+ self.clear();+ }+ std::fs::create_dir_all(&self.dir)+ .map_err(|error| format!("could not create {}: {error}", self.dir.display()))?;+ std::fs::write(model_file(&self.dir), name)+ .map_err(|error| format!("could not write {}: {error}", self.dir.display()))?;+ self.model = Some(name.to_owned());+ Ok(())+ }+/// How many instants are analysed.pub fn len(&self) -> usize {self.times.len()@@ -133,6 +169,9 @@ impl MaskStore {/// The mask to cut with at `seconds`: the model's, with the cutout's/// strokes painted on when it is custom, and softened by its feather.+ /// `None` when the instant has no mask yet, and also when the model+ /// found nothing there and no stroke says otherwise: a picture with+ /// no subject shows as shot rather than vanishing./// `aspect` is the source's width over its height, for round brushes.pub fn resolved(&self, seconds: f64, cutout: &Cutout, aspect: f32) -> Option<Arc<Mask>> {let millis = self.nearest_millis(seconds)?;@@ -149,6 +188,10 @@ impl MaskStore {return Some(hit);}let auto = load(&file)?;+ let corrected = cutout.mode == CutoutMode::Custom && !cutout.strokes.is_empty();+ if !corrected && auto.is_blank() {+ return None;+ }let painted = match cutout.mode {CutoutMode::Custom if !cutout.strokes.is_empty() => {strokes::paint(&auto, &cutout.strokes, aspect)@@ -213,6 +256,8 @@ impl MaskStore {cache.retain(|key| !key.file.starts_with(&self.dir));}self.times.clear();+ let _ = std::fs::remove_file(model_file(&self.dir));+ self.model = None;let _ = std::fs::remove_dir(&self.dir);}}@@ -332,11 +377,13 @@ mod tests {#[test]fn the_directory_is_named_by_the_file_and_the_model() {- let a = mask_dir(Path::new("/p"), "/footage/a.mp4");- let b = mask_dir(Path::new("/p"), "/footage/b.mp4");+ let a = mask_dir(Path::new("/p"), "/footage/a.mp4", Subject::Auto);+ let b = mask_dir(Path::new("/p"), "/footage/b.mp4", Subject::Auto);+ let c = mask_dir(Path::new("/p"), "/footage/a.mp4", Subject::Object);+ assert_ne!(a, c);assert_ne!(a, b);assert!(a.starts_with("/p/cache/masks"));- assert!(a.to_string_lossy().ends_with(MODEL_ID));+ assert!(a.to_string_lossy().ends_with("-auto"));}#[test]@@ -374,14 +421,15 @@ mod tests {fn a_resolved_mask_carries_the_strokes_and_the_feather() {let dir = scratch("resolved");let mut store = MaskStore::open(&dir);- store.put(0, &Mask::filled(16, 16, 0)).expect("writes");+ store.put(0, &Mask::filled(16, 16, 20)).expect("writes");let plain = Cutout::auto();assert_eq!(store.resolved(0.0, &plain, 1.0).map(|m| m.at(8, 8)),- Some(0)+ Some(20));let custom = Cutout {mode: CutoutMode::Custom,+ subject: Subject::Auto,feather: 0.0,strokes: vec![Stroke {tool: BrushTool::Brush,@@ -400,11 +448,47 @@ mod tests {};assert_eq!(store.resolved(0.0, &back_to_auto, 1.0).map(|m| m.at(8, 8)),- Some(0)+ Some(20)+ );+ let _ = std::fs::remove_dir_all(&dir);+ }++ #[test]+ fn a_blank_mask_shows_the_picture_as_shot_unless_painted() {+ let dir = scratch("blank");+ let mut store = MaskStore::open(&dir);+ store.put(0, &Mask::filled(16, 16, 0)).expect("writes");+ assert!(store.resolved(0.0, &Cutout::auto(), 1.0).is_none());+ let painted = Cutout {+ mode: CutoutMode::Custom,+ subject: Subject::Auto,+ feather: 0.0,+ strokes: vec![Stroke {+ tool: BrushTool::Brush,+ size: 0.5,+ points: vec![[0.5, 0.5]],+ }],+ };+ assert_eq!(+ store.resolved(0.0, &painted, 1.0).map(|m| m.at(8, 8)),+ Some(255));let _ = std::fs::remove_dir_all(&dir);}+ #[test]+ fn the_store_remembers_its_model_and_clears_for_another() {+ let dir = scratch("model");+ let mut store = MaskStore::open(&dir);+ store.set_model("first").expect("writes");+ store.put(0, &Mask::filled(4, 4, 200)).expect("writes");+ assert_eq!(MaskStore::open(&dir).model(), Some("first"));+ store.set_model("second").expect("writes");+ assert!(store.is_empty());+ assert_eq!(store.model(), Some("second"));+ let _ = std::fs::remove_dir_all(&dir);+ }+#[test]fn clearing_forgets_disk_and_memory() {let dir = scratch("clear");diff --git a/engine/crates/concat/locales/de.json b/engine/crates/concat/locales/de.jsonindex 8ec7a14a..4b1199f4 100644--- a/engine/crates/concat/locales/de.json+++ b/engine/crates/concat/locales/de.json@@ -181,6 +181,7 @@"Favourites": "Favoriten","Feather": "Weiche Kante","female": "weiblich",+ "Fetching the model… {0}%": "Modell wird geladen… {0}%","File": "Datei","Fill": "Füllung","Film Grain": "Filmkorn",@@ -312,12 +313,14 @@"Normalize Loudness": "Lautheit normalisieren","Normalize loudness": "Lautheit normalisieren","not installed": "nicht installiert",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Hier wurde nichts gefunden, das bleiben könnte, daher wird das Bild wie aufgenommen gezeigt. Versuche ein anderes Motiv oder das manuelle Entfernen.","Nothing of that kind in this project.": "Nichts dieser Art in diesem Projekt.","Nothing starred yet — hover a tile and press its star.": "Noch nichts markiert – über eine Kachel fahren und den Stern drücken.","Nothing to export": "Nichts zu exportieren","Nothing to read yet": "Noch nichts zu lesen","Nothing under the playhead": "Nichts unter dem Abspielkopf","Nothing was said in that clip": "In diesem Clip wurde nichts gesagt",+ "Object": "Objekt","off": "aus","One repeat, fading.": "Eine Wiederholung, ausklingend.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Ein Titel pro gesprochenem Satz, im Takt der Wörter geschnitten, auf der ersten freien Spur.",@@ -332,6 +335,7 @@"Pace": "Tempo","Paste": "Einfügen","Pause": "Pause",+ "Person": "Person","Pick an effect or filter from the library": "Wähle einen Effekt oder Filter aus der Bibliothek","Pitch": "Tonhöhe","Pixelate": "Verpixeln",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "Stein und Höhe: Reflexionen, die eine Weile brauchen.","Strength": "Stärke","Stylize": "Stilisieren",+ "Subject": "Motiv","Subtitle": "Untertitel","Summary": "Zusammenfassung","Sweet": "Süß",diff --git a/engine/crates/concat/locales/en.json b/engine/crates/concat/locales/en.jsonindex 5937f17f..5d526879 100644--- a/engine/crates/concat/locales/en.json+++ b/engine/crates/concat/locales/en.json@@ -181,6 +181,7 @@"Favourites": "Favourites","Feather": "Feather","female": "female",+ "Fetching the model… {0}%": "Fetching the model… {0}%","File": "File","Fill": "Fill","Film Grain": "Film Grain",@@ -312,12 +313,14 @@"Normalize Loudness": "Normalize Loudness","Normalize loudness": "Normalize loudness","not installed": "not installed",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.","Nothing of that kind in this project.": "Nothing of that kind in this project.","Nothing starred yet — hover a tile and press its star.": "Nothing starred yet — hover a tile and press its star.","Nothing to export": "Nothing to export","Nothing to read yet": "Nothing to read yet","Nothing under the playhead": "Nothing under the playhead","Nothing was said in that clip": "Nothing was said in that clip",+ "Object": "Object","off": "off","One repeat, fading.": "One repeat, fading.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "One title per phrase spoken, cut to the words' own timing, on the first lane with room.",@@ -332,6 +335,7 @@"Pace": "Pace","Paste": "Paste","Pause": "Pause",+ "Person": "Person","Pick an effect or filter from the library": "Pick an effect or filter from the library","Pitch": "Pitch","Pixelate": "Pixelate",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "Stone and height: reflections that take a while to arrive.","Strength": "Strength","Stylize": "Stylize",+ "Subject": "Subject","Subtitle": "Subtitle","Summary": "Summary","Sweet": "Sweet",diff --git a/engine/crates/concat/locales/es.json b/engine/crates/concat/locales/es.jsonindex 37a2f089..f951c183 100644--- a/engine/crates/concat/locales/es.json+++ b/engine/crates/concat/locales/es.json@@ -181,6 +181,7 @@"Favourites": "Favoritos","Feather": "Difuminado","female": "femenina",+ "Fetching the model… {0}%": "Descargando el modelo… {0}%","File": "Archivo","Fill": "Relleno","Film Grain": "Grano de película",@@ -312,12 +313,14 @@"Normalize Loudness": "Normalizar sonoridad","Normalize loudness": "Normalizar sonoridad","not installed": "no instalado",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "No se encontró nada que conservar, así que la imagen se muestra tal cual. Prueba otro sujeto o la eliminación manual.","Nothing of that kind in this project.": "No hay nada de ese tipo en este proyecto.","Nothing starred yet — hover a tile and press its star.": "Aún no hay favoritos: pasa el cursor por una tarjeta y pulsa su estrella.","Nothing to export": "Nada que exportar","Nothing to read yet": "Aún no hay nada que leer","Nothing under the playhead": "Nada bajo el cabezal","Nothing was said in that clip": "No se dijo nada en ese clip",+ "Object": "Objeto","off": "desactivado","One repeat, fading.": "Una repetición que se desvanece.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Un título por frase hablada, ajustado al ritmo de las palabras, en la primera pista con espacio.",@@ -332,6 +335,7 @@"Pace": "Ritmo","Paste": "Pegar","Pause": "Pausa",+ "Person": "Persona","Pick an effect or filter from the library": "Elige un efecto o filtro de la biblioteca","Pitch": "Tono","Pixelate": "Pixelar",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "Piedra y altura: reflexiones que tardan en llegar.","Strength": "Intensidad","Stylize": "Estilizar",+ "Subject": "Sujeto","Subtitle": "Subtítulo","Summary": "Resumen","Sweet": "Dulce",diff --git a/engine/crates/concat/locales/fr.json b/engine/crates/concat/locales/fr.jsonindex c535c885..763f71bc 100644--- a/engine/crates/concat/locales/fr.json+++ b/engine/crates/concat/locales/fr.json@@ -181,6 +181,7 @@"Favourites": "Favoris","Feather": "Contour progressif","female": "féminine",+ "Fetching the model… {0}%": "Téléchargement du modèle… {0}%","File": "Fichier","Fill": "Remplissage","Film Grain": "Grain de film",@@ -312,12 +313,14 @@"Normalize Loudness": "Normaliser la sonie","Normalize loudness": "Normaliser la sonie","not installed": "non installé",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Rien à garder ici, l'image est donc montrée telle quelle. Essayez un autre sujet ou la suppression manuelle.","Nothing of that kind in this project.": "Rien de ce genre dans ce projet.","Nothing starred yet — hover a tile and press its star.": "Rien en favori pour l'instant — survolez une vignette et appuyez sur son étoile.","Nothing to export": "Rien à exporter","Nothing to read yet": "Rien à lire pour l'instant","Nothing under the playhead": "Rien sous la tête de lecture","Nothing was said in that clip": "Rien n'a été dit dans ce clip",+ "Object": "Objet","off": "désactivé","One repeat, fading.": "Une répétition qui s'estompe.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Un titre par phrase prononcée, calé sur le rythme des mots, sur la première piste libre.",@@ -332,6 +335,7 @@"Pace": "Rythme","Paste": "Coller","Pause": "Pause",+ "Person": "Personne","Pick an effect or filter from the library": "Choisissez un effet ou un filtre dans la bibliothèque","Pitch": "Hauteur","Pixelate": "Pixeliser",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "Pierre et hauteur : des réflexions qui mettent du temps à arriver.","Strength": "Intensité","Stylize": "Styliser",+ "Subject": "Sujet","Subtitle": "Sous-titre","Summary": "Résumé","Sweet": "Douce",diff --git a/engine/crates/concat/locales/it.json b/engine/crates/concat/locales/it.jsonindex e19e692c..c187b2d4 100644--- a/engine/crates/concat/locales/it.json+++ b/engine/crates/concat/locales/it.json@@ -181,6 +181,7 @@"Favourites": "Preferiti","Feather": "Sfumatura","female": "femminile",+ "Fetching the model… {0}%": "Scaricamento del modello… {0}%","File": "File","Fill": "Riempimento","Film Grain": "Grana pellicola",@@ -312,12 +313,14 @@"Normalize Loudness": "Normalizza loudness","Normalize loudness": "Normalizza loudness","not installed": "non installato",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Qui non c'è nulla da tenere, quindi l'immagine è mostrata com'è. Prova un altro soggetto o la rimozione manuale.","Nothing of that kind in this project.": "Niente di questo tipo in questo progetto.","Nothing starred yet — hover a tile and press its star.": "Nessun preferito ancora: passa su una scheda e premi la sua stella.","Nothing to export": "Niente da esportare","Nothing to read yet": "Ancora niente da leggere","Nothing under the playhead": "Niente sotto la testina","Nothing was said in that clip": "In quella clip non è stato detto nulla",+ "Object": "Oggetto","off": "disattivato","One repeat, fading.": "Una ripetizione che sfuma.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Un titolo per ogni frase pronunciata, tagliato sul ritmo delle parole, sulla prima traccia libera.",@@ -332,6 +335,7 @@"Pace": "Ritmo","Paste": "Incolla","Pause": "Pausa",+ "Person": "Persona","Pick an effect or filter from the library": "Scegli un effetto o un filtro dalla libreria","Pitch": "Intonazione","Pixelate": "Pixel",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "Pietra e altezza: riflessioni che tardano ad arrivare.","Strength": "Intensità","Stylize": "Stilizza",+ "Subject": "Soggetto","Subtitle": "Sottotitolo","Summary": "Riepilogo","Sweet": "Dolce",diff --git a/engine/crates/concat/locales/ja.json b/engine/crates/concat/locales/ja.jsonindex 63bf5eec..0373456c 100644--- a/engine/crates/concat/locales/ja.json+++ b/engine/crates/concat/locales/ja.json@@ -181,6 +181,7 @@"Favourites": "お気に入り","Feather": "ぼかし","female": "女性",+ "Fetching the model… {0}%": "モデルを取得中… {0}%","File": "ファイル","Fill": "塗り","Film Grain": "フィルムグレイン",@@ -312,12 +313,14 @@"Normalize Loudness": "ラウドネスを正規化","Normalize loudness": "ラウドネスを正規化","not installed": "未インストール",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "残すものが見つからなかったため、画像はそのまま表示されています。別の対象かカスタム削除を試してください。","Nothing of that kind in this project.": "この種類のものはこのプロジェクトにありません。","Nothing starred yet — hover a tile and press its star.": "まだお気に入りはありません。タイルにカーソルを合わせて星を押してください。","Nothing to export": "書き出すものがありません","Nothing to read yet": "読み上げる内容がまだありません","Nothing under the playhead": "再生ヘッドの下に何もありません","Nothing was said in that clip": "そのクリップには発話がありませんでした",+ "Object": "物体","off": "オフ","One repeat, fading.": "一回の繰り返しがフェードします。","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "発話されたフレーズごとにタイトルを一つ、言葉のタイミングどおりに、空いている最初のレーンへ。",@@ -332,6 +335,7 @@"Pace": "速さ","Paste": "ペースト","Pause": "一時停止",+ "Person": "人物","Pick an effect or filter from the library": "ライブラリからエフェクトまたはフィルターを選んでください","Pitch": "ピッチ","Pixelate": "モザイク",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "石と高さ:届くまでに時間のかかる反射。","Strength": "強さ","Stylize": "スタイライズ",+ "Subject": "対象","Subtitle": "サブタイトル","Summary": "概要","Sweet": "スイート",diff --git a/engine/crates/concat/locales/ko.json b/engine/crates/concat/locales/ko.jsonindex e9ebcf52..91232d2e 100644--- a/engine/crates/concat/locales/ko.json+++ b/engine/crates/concat/locales/ko.json@@ -181,6 +181,7 @@"Favourites": "즐겨찾기","Feather": "페더","female": "여성",+ "Fetching the model… {0}%": "모델 가져오는 중… {0}%","File": "파일","Fill": "채우기","Film Grain": "필름 그레인",@@ -312,12 +313,14 @@"Normalize Loudness": "라우드니스 정규화","Normalize loudness": "라우드니스 정규화","not installed": "설치되지 않음",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "남길 것을 찾지 못해 그림을 촬영한 그대로 보여줍니다. 다른 대상이나 사용자 지정 제거를 시도해 보세요.","Nothing of that kind in this project.": "이 프로젝트에 해당 종류의 항목이 없습니다.","Nothing starred yet — hover a tile and press its star.": "아직 즐겨찾기가 없습니다. 타일에 마우스를 올리고 별을 누르세요.","Nothing to export": "내보낼 것이 없습니다","Nothing to read yet": "아직 읽을 내용이 없습니다","Nothing under the playhead": "재생 헤드 아래에 아무것도 없습니다","Nothing was said in that clip": "그 클립에는 말소리가 없습니다",+ "Object": "사물","off": "끔","One repeat, fading.": "한 번 반복하며 사라집니다.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "말한 구절마다 자막 하나씩, 말의 타이밍에 맞춰, 자리가 있는 첫 레인에 놓습니다.",@@ -332,6 +335,7 @@"Pace": "속도","Paste": "붙여넣기","Pause": "일시 정지",+ "Person": "인물","Pick an effect or filter from the library": "라이브러리에서 효과나 필터를 선택하세요","Pitch": "피치","Pixelate": "픽셀화",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "돌과 높이: 도착하기까지 시간이 걸리는 반사.","Strength": "강도","Stylize": "스타일화",+ "Subject": "대상","Subtitle": "부제","Summary": "요약","Sweet": "달콤하게",diff --git a/engine/crates/concat/locales/pt-BR.json b/engine/crates/concat/locales/pt-BR.jsonindex 4b221606..7218e617 100644--- a/engine/crates/concat/locales/pt-BR.json+++ b/engine/crates/concat/locales/pt-BR.json@@ -181,6 +181,7 @@"Favourites": "Favoritos","Feather": "Suavização","female": "feminina",+ "Fetching the model… {0}%": "Baixando o modelo… {0}%","File": "Arquivo","Fill": "Preenchimento","Film Grain": "Granulação de filme",@@ -312,12 +313,14 @@"Normalize Loudness": "Normalizar loudness","Normalize loudness": "Normalizar loudness","not installed": "não instalado",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Nada foi encontrado para manter, então a imagem é mostrada como gravada. Tente outro assunto ou a remoção manual.","Nothing of that kind in this project.": "Nada desse tipo neste projeto.","Nothing starred yet — hover a tile and press its star.": "Nada favoritado ainda — passe o mouse sobre um cartão e toque na estrela.","Nothing to export": "Nada para exportar","Nothing to read yet": "Nada para ler ainda","Nothing under the playhead": "Nada sob o cursor","Nothing was said in that clip": "Nada foi dito nesse clipe",+ "Object": "Objeto","off": "desligado","One repeat, fading.": "Uma repetição que se esvai.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Um título por frase falada, no ritmo das palavras, na primeira faixa com espaço.",@@ -332,6 +335,7 @@"Pace": "Ritmo","Paste": "Colar","Pause": "Pausar",+ "Person": "Pessoa","Pick an effect or filter from the library": "Escolha um efeito ou filtro na biblioteca","Pitch": "Tom","Pixelate": "Pixelizar",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "Pedra e altura: reflexões que demoram a chegar.","Strength": "Intensidade","Stylize": "Estilizar",+ "Subject": "Assunto","Subtitle": "Subtítulo","Summary": "Resumo","Sweet": "Doce",diff --git a/engine/crates/concat/locales/ru.json b/engine/crates/concat/locales/ru.jsonindex abd35055..5d7eb4f6 100644--- a/engine/crates/concat/locales/ru.json+++ b/engine/crates/concat/locales/ru.json@@ -181,6 +181,7 @@"Favourites": "Избранное","Feather": "Растушёвка","female": "женский",+ "Fetching the model… {0}%": "Загрузка модели… {0}%","File": "Файл","Fill": "Заливка","Film Grain": "Зерно плёнки",@@ -312,12 +313,14 @@"Normalize Loudness": "Нормализовать громкость","Normalize loudness": "Нормализовать громкость","not installed": "не установлено",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Здесь нечего оставить, поэтому кадр показан как снят. Попробуйте другой объект съёмки или ручное удаление.","Nothing of that kind in this project.": "В этом проекте ничего такого нет.","Nothing starred yet — hover a tile and press its star.": "Пока ничего нет — наведите на плитку и нажмите звезду.","Nothing to export": "Нечего экспортировать","Nothing to read yet": "Пока нечего читать","Nothing under the playhead": "Под курсором ничего нет","Nothing was said in that clip": "В этом клипе ничего не сказано",+ "Object": "Предмет","off": "выкл.","One repeat, fading.": "Один повтор, затухающий.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Один титр на каждую произнесённую фразу, в такт словам, на первой свободной дорожке.",@@ -332,6 +335,7 @@"Pace": "Темп","Paste": "Вставить","Pause": "Пауза",+ "Person": "Человек","Pick an effect or filter from the library": "Выберите эффект или фильтр в библиотеке","Pitch": "Высота","Pixelate": "Пикселизация",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "Камень и высота: отражения, которые долго идут.","Strength": "Сила","Stylize": "Стилизация",+ "Subject": "Объект съёмки","Subtitle": "Подзаголовок","Summary": "Сводка","Sweet": "Сладкий",diff --git a/engine/crates/concat/locales/tr.json b/engine/crates/concat/locales/tr.jsonindex b188f773..4596d007 100644--- a/engine/crates/concat/locales/tr.json+++ b/engine/crates/concat/locales/tr.json@@ -181,6 +181,7 @@"Favourites": "Favoriler","Feather": "Yumuşatma","female": "kadın",+ "Fetching the model… {0}%": "Model indiriliyor… {0}%","File": "Dosya","Fill": "Dolgu","Film Grain": "Film greni",@@ -312,12 +313,14 @@"Normalize Loudness": "Ses yüksekliğini normalleştir","Normalize loudness": "Gürlüğü normalleştir","not installed": "kurulu değil",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Burada tutulacak bir şey bulunamadı, bu yüzden görüntü çekildiği gibi gösteriliyor. Başka bir konu ya da özel kaldırmayı deneyin.","Nothing of that kind in this project.": "Bu projede o türden bir şey yok.","Nothing starred yet — hover a tile and press its star.": "Henüz favori yok — bir kartın üzerine gelip yıldızına basın.","Nothing to export": "Dışa aktarılacak bir şey yok","Nothing to read yet": "Henüz okunacak bir şey yok","Nothing under the playhead": "Oynatma imlecinin altında bir şey yok","Nothing was said in that clip": "O klipte hiçbir şey söylenmedi",+ "Object": "Nesne","off": "kapalı","One repeat, fading.": "Sönen tek bir tekrar.","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Söylenen her cümle için bir yazı, kelimelerin kendi zamanlamasıyla, yeri olan ilk şeritte.",@@ -332,6 +335,7 @@"Pace": "Tempo","Paste": "Yapıştır","Pause": "Duraklat",+ "Person": "Kişi","Pick an effect or filter from the library": "Kitaplıktan bir efekt veya filtre seçin","Pitch": "Perde","Pixelate": "Pikselleştir",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "Taş ve yükseklik: gelmesi zaman alan yansımalar.","Strength": "Güç","Stylize": "Stilize",+ "Subject": "Konu","Subtitle": "Alt başlık","Summary": "Özet","Sweet": "Tatlı",diff --git a/engine/crates/concat/locales/zh-Hans.json b/engine/crates/concat/locales/zh-Hans.jsonindex 3978676d..50073d1f 100644--- a/engine/crates/concat/locales/zh-Hans.json+++ b/engine/crates/concat/locales/zh-Hans.json@@ -181,6 +181,7 @@"Favourites": "收藏","Feather": "羽化","female": "女声",+ "Fetching the model… {0}%": "正在获取模型… {0}%","File": "文件","Fill": "填充","Film Grain": "胶片颗粒",@@ -312,12 +313,14 @@"Normalize Loudness": "响度标准化","Normalize loudness": "响度标准化","not installed": "未安装",+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "这里没有找到可保留的内容,因此画面按原样显示。请尝试其他主体或自定义移除。","Nothing of that kind in this project.": "此项目中没有该类型的内容。","Nothing starred yet — hover a tile and press its star.": "还没有收藏——将鼠标悬停在卡片上并点击星标。","Nothing to export": "没有可导出的内容","Nothing to read yet": "还没有可朗读的内容","Nothing under the playhead": "播放头下方没有内容","Nothing was said in that clip": "该片段中没有说话内容",+ "Object": "物体","off": "关","One repeat, fading.": "一次重复,逐渐消失。","One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "每说一句生成一条字幕,按语句自身的时间切分,放在第一条有空位的轨道上。",@@ -332,6 +335,7 @@"Pace": "语速","Paste": "粘贴","Pause": "暂停",+ "Person": "人物","Pick an effect or filter from the library": "请从素材库中选择效果或滤镜","Pitch": "音高","Pixelate": "像素化",@@ -447,6 +451,7 @@"Stone and height: reflections that take a while to arrive.": "石壁与高度:需要一段时间才到达的反射。","Strength": "强度","Stylize": "风格化",+ "Subject": "主体","Subtitle": "副标题","Summary": "摘要","Sweet": "甜美",diff --git a/engine/crates/concat/src/host.rs b/engine/crates/concat/src/host.rsindex c463355d..a8bc507a 100644--- a/engine/crates/concat/src/host.rs+++ b/engine/crates/concat/src/host.rs@@ -55,6 +55,7 @@ impl Host {let _ = std::fs::create_dir_all(&dirs.config);Ok(Host {titles: concat_host::Titles::new(&dirs),+ cutouts: Arc::new(concat_host::Cutouts::new(&dirs.data)),dirs,playback: Playback::start(Arc::new(Events))?,monitor: match gpu {@@ -64,7 +65,6 @@ impl Host {exporter: Exporter::new(),transcriber: Arc::new(Transcriber::new()),speech: Arc::new(Speech::new()),- cutouts: Arc::new(concat_host::Cutouts::new()),})}}diff --git a/engine/crates/concat/src/lib.rs b/engine/crates/concat/src/lib.rsindex f6081414..ea785fa2 100644--- a/engine/crates/concat/src/lib.rs+++ b/engine/crates/concat/src/lib.rs@@ -774,6 +774,9 @@ pub fn run() -> Result<(), slint::PlatformError> {editor.on_cutout_painting(on_window!(|state, on: bool| {state.cutout_painting(on);}));+ editor.on_cutout_subject(on_window!(|state, index: i32| {+ state.cutout_subject(index);+ }));editor.on_cutout_clear(on_window!(|state| {state.cutout_clear();}));diff --git a/engine/crates/concat/src/studio.rs b/engine/crates/concat/src/studio.rsindex 3606f72a..17ac5b7c 100644--- a/engine/crates/concat/src/studio.rs+++ b/engine/crates/concat/src/studio.rs@@ -719,7 +719,7 @@ pub struct Studio {/// rather than moving the picture.pub painting: bool,/// The mask analyses running, by media id, and how far each has got.- cutout_jobs: HashMap<String, f32>,+ cutout_jobs: HashMap<String, (bool, f32)>,}// ── conversions between the document and the window ─────────────────────@@ -3471,12 +3471,18 @@ impl Studio {let project = std::path::PathBuf::from(session.path());let mut wanted: Vec<(String, AnalyseRequest)> = Vec::new();for clip in &self.timeline().clips {- if clip.cutout.is_none() || !clip.kind.is_visual() {+ let Some(cutout) = clip.cutout.as_ref() else {+ continue;+ };+ if !clip.kind.is_visual() {continue;}let Some(media) = self.project().media_by_id(&clip.media_id) else {continue;};+ // One analysis per media and subject: two clips of one file+ // that keep different things need different masks.+ let key = Self::analysis_key(&media.id, cutout.subject);// The source the clip shows: its in-point, for as long as it// runs at its speed. A curve's mean is its speed, so this// covers a curved clip too.@@ -3484,14 +3490,16 @@ impl Studio {clip.source_start,clip.source_start + clip.duration * clip.speed.max(0.0625),);- match wanted.iter_mut().find(|(id, _)| *id == media.id) {+ match wanted.iter_mut().find(|(id, _)| *id == key) {Some((_, request)) => request.ranges.push(range),None => wanted.push((- media.id.clone(),+ key,AnalyseRequest {project: project.clone(),media_path: media.path.clone(),+ media_size: (media.width.unwrap_or(0), media.height.unwrap_or(0)),still: media.kind == model::MediaKind::Image,+ subject: cutout.subject,ranges: vec![range],},)),@@ -3503,21 +3511,27 @@ impl Studio {else {return;};- self.cutout_jobs.insert(id.clone(), 0.0);+ self.cutout_jobs.insert(id.clone(), (false, 0.0));let cutouts = Arc::clone(&self.host.cutouts);spawn(move || {- let mut last = -1.0f32;+ let mut last = (false, -1.0f32);let reporting = id.clone();let result = cutouts.analyse(&request, &mut |progress| {// Every percent, not every frame: the readout cannot// use more and the event loop has other work.- if progress - last >= 0.01 {- last = progress;+ let now = match progress {+ concat_host::cutout::Progress::Fetching { received, total } => {+ (true, received as f32 / total.max(1) as f32)+ }+ concat_host::cutout::Progress::Analysing(fraction) => (false, fraction),+ };+ if now.0 != last.0 || now.1 - last.1 >= 0.01 {+ last = now;let id = reporting.clone();on_ui(move |studio, _, _| {if let Some(held) = studio.cutout_jobs.get_mut(&id) {- *held = progress;+ *held = now;}});}@@ -3572,6 +3586,34 @@ impl Studio {}}+ /// The name an analysis runs under: the media and what it keeps.+ fn analysis_key(media_id: &str, subject: model::Subject) -> String {+ format!("{media_id}:{}", subject.key())+ }++ /// The Subject row: 0 automatic, 1 person, 2 object. A change means+ /// other masks, which the analysis notices on its own.+ pub fn cutout_subject(&mut self, index: i32) {+ let Some(id) = self.sole_selection() else {+ return;+ };+ let Some(cutout) = self.clip(&id).and_then(|clip| clip.cutout.clone()) else {+ return;+ };+ let subject = match index {+ 1 => model::Subject::Person,+ 2 => model::Subject::Object,+ _ => model::Subject::Auto,+ };+ if cutout.subject == subject {+ return;+ }+ self.apply(Command::SetClipCutout {+ clip_id: id,+ cutout: Some(model::Cutout { subject, ..cutout }),+ });+ }+/// Takes every stroke off the selected clip's cutout, keeping it custom.pub fn cutout_clear(&mut self) {let Some(id) = self.sole_selection() else {@@ -5034,6 +5076,11 @@ impl Studio {let fill = colour_of(&text.color);let stroke = colour_of(&text.stroke_color);let plate = colour_of(&text.background);+ let analysis = clip.cutout.as_ref().and_then(|cutout| {+ self.cutout_jobs+ .get(&Self::analysis_key(&clip.media_id, cutout.subject))+ .copied()+ });SelectedClipData {present: true,id: clip.id.as_str().into(),@@ -5121,14 +5168,48 @@ impl Studio {.as_ref().map(|cutout| cutout.strokes.len() as i32).unwrap_or(0),- cutout_progress: self- .cutout_jobs- .get(&clip.media_id)- .copied()- .unwrap_or(-1.0),+ cutout_subject: clip+ .cutout+ .as_ref()+ .map(|cutout| match cutout.subject {+ model::Subject::Auto => 0,+ model::Subject::Person => 1,+ model::Subject::Object => 2,+ })+ .unwrap_or(0),+ cutout_progress: analysis.map(|(_, fraction)| fraction).unwrap_or(-1.0),+ cutout_fetching: analysis.is_some_and(|(fetching, _)| fetching),+ cutout_empty: self.cutout_empty(clip),}}+ /// Whether the cutout model found nothing at the playhead's frame of+ /// `clip`, so the picture is showing as shot. Only an automatic cutout+ /// says so: a custom one is whatever was painted.+ fn cutout_empty(&self, clip: &Clip) -> bool {+ let Some(cutout) = clip.cutout.as_ref() else {+ return false;+ };+ if cutout.mode != model::CutoutMode::Auto {+ return false;+ }+ let (Some(session), Some(media)) = (+ self.session.as_ref(),+ self.project().media_by_id(&clip.media_id),+ ) else {+ return false;+ };+ let project = std::path::Path::new(session.path());+ let store = concat_vision::MaskStore::open(&concat_vision::mask_dir(+ project,+ &media.path,+ cutout.subject,+ ));+ let along = (f64::from(self.playhead) - clip.start).clamp(0.0, clip.duration);+ let source = clip.source_start + along * clip.speed;+ store.mask_at(source).is_some_and(|mask| mask.is_blank())+ }+/// The menus, the dialogs, the bin and the engine lists.pub fn publish_chrome(&self, app: &App, models: &Models) {let editor = app.global::<Editor>();diff --git a/engine/crates/concat/ui/editor.slint b/engine/crates/concat/ui/editor.slintindex afe9fff8..a130d1cc 100644--- a/engine/crates/concat/ui/editor.slint+++ b/engine/crates/concat/ui/editor.slint@@ -201,6 +201,7 @@ export global Editor {callback cutout-tool(index: int);callback cutout-size(size: float);callback cutout-painting(on: bool);+ callback cutout-subject(index: int);callback cutout-clear();// ── the timeline ──diff --git a/engine/crates/concat/ui/inspector/clip-inspector.slint b/engine/crates/concat/ui/inspector/clip-inspector.slintindex 3c5b7e08..f8f3b9dc 100644--- a/engine/crates/concat/ui/inspector/clip-inspector.slint+++ b/engine/crates/concat/ui/inspector/clip-inspector.slint@@ -649,10 +649,14 @@ component ToolButton inherits Rectangle {// background and the picture is shown as shot until they are.component AnalysisRow inherits Row {in property <float> progress;+ /// The model is still being downloaded; the frames come after.+ in property <bool> fetching;label: I18n.t("Analysis");Text {- text: I18n.t1("Finding the subject… {0}%", "\{Math.round(root.progress * 100)}");+ text: root.fetching+ ? I18n.t1("Fetching the model… {0}%", "\{Math.round(root.progress * 100)}")+ : I18n.t1("Finding the subject… {0}%", "\{Math.round(root.progress * 100)}");color: Theme.fg-muted;font-size: Theme.fs;vertical-alignment: center;@@ -695,6 +699,8 @@ export component VideoPanel inherits VerticalLayout {callback cutout-tool(index: int);callback cutout-size(size: float);callback cutout-painting(on: bool);+ /// What the automatic cutout keeps: 0 auto, 1 person, 2 object.+ callback cutout-subject(index: int);callback cutout-clear();/// Which keyer is on the chain, and where: -1 for none. Scans the first@@ -939,6 +945,14 @@ export component VideoPanel inherits VerticalLayout {on: root.removal == 1;toggled(on) => { root.remove-by(on ? 1 : 0); }}+ if root.removal == 1: Row {+ label: I18n.t("Subject");+ Select {+ options: [I18n.t("Auto"), I18n.t("Person"), I18n.t("Object")];+ current: root.selected.cutout-subject;+ changed(index) => { root.cutout-subject(index); }+ }+ }if root.removal == 1: Param {label: I18n.t("Feather");value <=> root.feather;@@ -951,6 +965,10 @@ export component VideoPanel inherits VerticalLayout {}if root.removal == 1 && root.selected.cutout-progress >= 0: AnalysisRow {progress: root.selected.cutout-progress;+ fetching: root.selected.cutout-fetching;+ }+ if root.removal == 1 && root.selected.cutout-progress < 0 && root.selected.cutout-empty: Note {+ text: I18n.t("Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.");}Hairline { }@@ -1022,6 +1040,7 @@ export component VideoPanel inherits VerticalLayout {}if root.removal == 2 && root.selected.cutout-progress >= 0: AnalysisRow {progress: root.selected.cutout-progress;+ fetching: root.selected.cutout-fetching;}if root.removal == 2: Row {label: root.selected.cutout-strokes == 1 ? I18n.t("1 stroke")diff --git a/engine/crates/concat/ui/timeline/model.slint b/engine/crates/concat/ui/timeline/model.slintindex 64dee708..e79d24ad 100644--- a/engine/crates/concat/ui/timeline/model.slint+++ b/engine/crates/concat/ui/timeline/model.slint@@ -281,13 +281,19 @@ export struct SelectedClipData {fade-in: float,fade-out: float,- // the cutout: 0 none, 1 automatic, 2 custom; how soft its edge is; how- // many strokes it carries; and the analysis finding its masks, -1 when- // none is running, else how far along it is+ // the cutout: 0 none, 1 automatic, 2 custom; what it keeps, 0 auto,+ // 1 person, 2 object; how soft its edge is; how many strokes it+ // carries; the analysis finding its masks, -1 when none is running,+ // else how far along it is, and whether that is a model downloading+ // rather than frames being read; and whether the model found nothing+ // at the playhead, so the picture is showing as shotcutout: int,+ cutout-subject: int,cutout-feather: float,cutout-strokes: int,cutout-progress: float,+ cutout-fetching: bool,+ cutout-empty: bool,// the overlay, when this is a text clipcontent: string,diff --git a/engine/crates/concat/ui/workspace/config-pane.slint b/engine/crates/concat/ui/workspace/config-pane.slintindex 79be3b45..06fb300c 100644--- a/engine/crates/concat/ui/workspace/config-pane.slint+++ b/engine/crates/concat/ui/workspace/config-pane.slint@@ -172,6 +172,7 @@ export component ConfigPane inherits Rectangle {callback cutout-tool(index: int);callback cutout-size(size: float);callback cutout-painting(on: bool);+ callback cutout-subject(index: int);callback cutout-clear();@@ -524,6 +525,7 @@ export component ConfigPane inherits Rectangle {cutout-tool(index) => { root.cutout-tool(index); }cutout-size(size) => { root.cutout-size(size); }cutout-painting(on) => { root.cutout-painting(on); }+ cutout-subject(index) => { root.cutout-subject(index); }cutout-clear => { root.cutout-clear(); }}diff --git a/engine/crates/concat/ui/workspace/seat.slint b/engine/crates/concat/ui/workspace/seat.slintindex f359649f..dbcff09d 100644--- a/engine/crates/concat/ui/workspace/seat.slint+++ b/engine/crates/concat/ui/workspace/seat.slint@@ -217,6 +217,7 @@ export component Seat inherits Rectangle {cutout-tool(index) => { Editor.cutout-tool(index); }cutout-size(size) => { Editor.cutout-size(size); }cutout-painting(on) => { Editor.cutout-painting(on); }+ cutout-subject(index) => { Editor.cutout-subject(index); }cutout-clear => { Editor.cutout-clear(); }project-name: Editor.project-name;