Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

Three cutout models: a person matte, an object model, a subject to choose

Jareera54a880543eb
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index fbf7ae48..49cfafd5 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -421,16 +421,22 @@ look repaints.
## 10. Cutouts: `concat-vision` and `host::cutout`
-A cutout takes a picture's background away without a key colour. The
-model is MediaPipe's selfie segmentation, compiled into `concat-vision`
-and run by ONNX Runtime: a 256 × 256 picture in, a probability per pixel
-out, five milliseconds a frame on an Apple GPU and about twenty on a
-laptop's cores. Masks are square
-whatever the picture's shape, and every position in one is a fraction of
-the source picture, which is also how strokes are stored; a crop, a flip
-or a change of output size changes nothing about either, because
-`apply::Mapping` is the one place a decoded pixel is walked back to a
-source fraction.
+A cutout takes a picture's background away without a key colour. Three
+models, all run by ONNX Runtime, all one interface in `segment`: a frame
+in, a probability per pixel out. Robust Video Matting is the person
+model, at the picture's own aspect on a 512 edge, carrying a recurrent
+state between frames so an outline holds still; IS-Net is the object
+model, 1024 square, for a car or a cartoon; MediaPipe's selfie
+segmentation is compiled in and answers when nothing has been
+downloaded. A clip's `Subject` picks between the first two, and on
+automatic the person model looks at the first frame and hands over to the
+object model when it finds no one; the choice is written into the mask
+store, so every run agrees. Every position in a mask is a fraction of the
+source picture whatever the mask's shape, which is also how strokes are
+stored; a crop, a flip or a change of output size changes nothing about
+either, because `apply::Mapping` is the one place a decoded pixel is
+walked back to a source fraction. A mask the model left blank shows the
+picture as shot rather than nothing.
**The store is the project's.** Each media file's masks are PNGs in
`cache/masks/<hash>-<model>/`, one per analysed source instant, named by
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md
index 1a005266..74382913 100644
--- a/THIRD_PARTY_NOTICES.md
+++ b/THIRD_PARTY_NOTICES.md
@@ -87,13 +87,31 @@ https://huggingface.co/hexgrad/Kokoro-82M) are downloaded on demand from the
sherpa-onnx releases - including espeak-ng's data files - and are never
bundled with the app.
-## The cutout model
-
-Remove background's automatic and custom modes run Google's MediaPipe
-Selfie Segmentation model (Apache-2.0), in the ONNX conversion published by
-the ONNX Community (https://huggingface.co/onnx-community/mediapipe_selfie_segmentation,
-Apache-2.0). The model file is compiled into the `concat-vision` crate; see
-`engine/crates/concat-vision/models/NOTICE.md`. It is run by ONNX Runtime
+## The cutout models
+
+Remove background runs three models, none of which ship inside the app
+except the first:
+
+- Google's MediaPipe Selfie Segmentation (Apache-2.0), in the ONNX
+ conversion published by the ONNX Community
+ (https://huggingface.co/onnx-community/mediapipe_selfie_segmentation,
+ Apache-2.0), compiled into the `concat-vision` crate; see
+ `engine/crates/concat-vision/models/NOTICE.md`. The answer when nothing
+ has been downloaded.
+- Robust Video Matting, the MobileNetV3 variant, by Peter Lin and others
+ (https://github.com/PeterL1n/RobustVideoMatting, GPL-3.0), downloaded on
+ first use from that repository's releases. The person model.
+- IS-Net from "Highly Accurate Dichotomous Image Segmentation" by Qin and
+ others (https://github.com/xuebinqin/DIS, Apache-2.0), in the ONNX
+ export the rembg project publishes
+ (https://github.com/danielgatis/rembg, MIT), downloaded on first use.
+ The object model.
+- SlimSAM (https://github.com/czg1225/SlimSAM, Apache-2.0), in the ONNX
+ export published at https://huggingface.co/Xenova/slimsam-77-uniform
+ (Apache-2.0), downloaded on first use. The brushes' model.
+
+Downloaded models live in the app's data directory under `cutout-models`
+and are never bundled. They are all run by ONNX Runtime
(https://github.com/microsoft/onnxruntime, MIT) through the `ort` crate
(https://github.com/pykeio/ort, MIT OR Apache-2.0), with the platform's
own accelerator behind it: CoreML on macOS and iOS, DirectML on Windows,
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index f16caf51..95575f11 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -1229,6 +1229,7 @@ dependencies = [
"memmap2",
"serde",
"serde_json",
+ "ureq 2.12.1",
]
[[package]]
diff --git a/engine/crates/concat-export/src/flatten.rs b/engine/crates/concat-export/src/flatten.rs
index 04cacff6..44cd85ff 100644
--- a/engine/crates/concat-export/src/flatten.rs
+++ b/engine/crates/concat-export/src/flatten.rs
@@ -166,9 +166,11 @@ pub fn flatten_timeline_in(
has_audio: Some(media.has_audio),
cutout: clip.cutout.clone(),
mask_dir: match (&clip.cutout, project_dir) {
- (Some(_), Some(dir)) => concat_vision::mask_dir(dir, &media.path)
- .to_string_lossy()
- .into_owned(),
+ (Some(cutout), Some(dir)) => {
+ concat_vision::mask_dir(dir, &media.path, cutout.subject)
+ .to_string_lossy()
+ .into_owned()
+ }
_ => String::new(),
},
})
diff --git a/engine/crates/concat-host/Cargo.toml b/engine/crates/concat-host/Cargo.toml
index 767be51c..5df82db3 100644
--- a/engine/crates/concat-host/Cargo.toml
+++ b/engine/crates/concat-host/Cargo.toml
@@ -18,8 +18,10 @@ concat-project.workspace = true
concat-export.workspace = true
# Titles become pictures here; see titles.rs.
concat-text.workspace = true
-# The cutout model and its masks; `infer` because the host is what runs it.
+# The cutout models and their masks; `infer` because the host is what
+# runs them, and ureq because the host is what fetches them.
concat-vision = { workspace = true, features = ["infer"] }
+ureq = "2"
concat-render = { workspace = true, optional = true }
serde = { version = "1", features = ["derive"] }
serde_json.workspace = true
diff --git a/engine/crates/concat-host/src/cutout.rs b/engine/crates/concat-host/src/cutout.rs
index 04199a20..2ce6ea70 100644
--- a/engine/crates/concat-host/src/cutout.rs
+++ b/engine/crates/concat-host/src/cutout.rs
@@ -5,24 +5,35 @@
//!
//! A clip with a cutout needs a mask for every source instant it shows, on
//! the analysis grid `concat-vision` reads by. This is the job that fills
-//! the gaps: it decodes the media at the model's size along each missing
-//! stretch, runs the model on several frames at once, and writes what it
-//! finds into the media's mask store in the project folder. A still is one
-//! frame; footage is [`concat_vision::MASK_RATE`] frames a second of source.
+//! the gaps: it picks the model the clip's subject asks for, fetches it if
+//! it is not on disk yet, decodes the media at the model's size along each
+//! missing stretch, runs the model frame by frame, and writes what it
+//! finds into the media's mask store in the project folder. A still is
+//! one frame; footage is [`concat_vision::MASK_RATE`] frames a second of
+//! source.
+//!
+//! Which model: the person model for a person, the object model for an
+//! object, and for a subject left on automatic the person model is tried
+//! on the first frame and the object model takes over when it finds no
+//! one. The choice is written into the store, so the rest of the run and
+//! every later one agree. With nothing downloaded and no network, the
+//! compiled-in person model answers instead.
//!
//! One job at a time through a [`SingleFlight`], like every long job the
-//! host runs; the window queues the next media behind it. The model loads
+//! host runs; the window queues the next media behind it. A model loads
//! on first use and stays loaded.
-use std::path::PathBuf;
-use std::sync::{Arc, OnceLock};
+use std::path::{Path, PathBuf};
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::sync::{Arc, Mutex, OnceLock};
-use concat_core::frame::Frame;
use concat_core::time::{FrameRate, Rational};
use concat_media::{DecodeOptions, Decoder, FrameSource};
-use concat_vision::{MASK_RATE, MODEL_SIZE, Mask, MaskStore, Segmenter, mask_dir};
+use concat_project::model::Subject;
+use concat_vision::segment::Kind;
+use concat_vision::{MASK_RATE, MaskStore, ModelId, Segmenter, mask_dir, models};
-use crate::jobs::{Job, SingleFlight};
+use crate::jobs::SingleFlight;
/// What one analysis covers.
#[derive(Clone, Debug)]
@@ -31,31 +42,51 @@ pub struct AnalyseRequest {
pub project: PathBuf,
/// The media file to analyse.
pub media_path: String,
+ /// The media's picture size, for the person model's working size.
+ pub media_size: (u32, u32),
/// A still: one frame answers for every instant.
pub still: bool,
+ /// What to keep.
+ pub subject: Subject,
/// The stretches of source, in seconds, that clips show. Instants
/// already analysed inside them are skipped.
pub ranges: Vec<(f64, f64)>,
}
-/// The analysis service: the model, loaded once, and the one-job slot.
-pub struct Cutouts {
- gate: Arc<SingleFlight>,
- segmenter: OnceLock<Result<Arc<Segmenter>, String>>,
+/// How far an analysis has got, as it reports it.
+#[derive(Clone, Copy, Debug, PartialEq)]
+pub enum Progress {
+ /// A model is being downloaded: bytes so far, of about this many.
+ Fetching {
+ /// Bytes received.
+ received: u64,
+ /// Bytes expected.
+ total: u64,
+ },
+ /// Masks are being found: `0..=1` of the instants wanted.
+ Analysing(f32),
}
-impl Default for Cutouts {
- fn default() -> Self {
- Self::new()
- }
+/// The analysis service: the models, loaded once each, and the one-job
+/// slot.
+pub struct Cutouts {
+ gate: Arc<SingleFlight>,
+ /// The app's data directory, where downloaded models live.
+ data: PathBuf,
+ selfie: OnceLock<Result<Arc<Segmenter>, String>>,
+ person: Mutex<Option<Arc<Segmenter>>>,
+ object: Mutex<Option<Arc<Segmenter>>>,
}
impl Cutouts {
- /// A service with nothing loaded yet.
- pub fn new() -> Cutouts {
+ /// A service with nothing loaded yet, keeping its models under `data`.
+ pub fn new(data: &Path) -> Cutouts {
Cutouts {
gate: Arc::new(SingleFlight::new()),
- segmenter: OnceLock::new(),
+ data: data.to_path_buf(),
+ selfie: OnceLock::new(),
+ person: Mutex::new(None),
+ object: Mutex::new(None),
}
}
@@ -64,20 +95,81 @@ impl Cutouts {
self.gate.is_busy()
}
- /// Asks the running analysis to stop after the frames in hand.
+ /// Asks the running analysis to stop after the frame in hand.
pub fn cancel(&self) {
self.gate.cancel();
}
- fn segmenter(&self) -> Result<Arc<Segmenter>, String> {
- self.segmenter
- .get_or_init(|| Segmenter::load().map(Arc::new))
+ /// Whether a downloadable model is on disk.
+ pub fn installed(&self, id: ModelId) -> bool {
+ models::installed(&self.data, id)
+ }
+
+ fn selfie(&self) -> Result<Arc<Segmenter>, String> {
+ self.selfie
+ .get_or_init(|| Segmenter::selfie().map(Arc::new))
.clone()
}
+ /// The person or object model, fetched first when it is not on disk.
+ fn downloaded(
+ &self,
+ id: ModelId,
+ cancel: &AtomicBool,
+ progress: &mut dyn FnMut(Progress),
+ ) -> Result<Arc<Segmenter>, String> {
+ let slot = match id {
+ ModelId::Person => &self.person,
+ ModelId::Object => &self.object,
+ _ => return Err("not a segmenting model".to_owned()),
+ };
+ if let Some(loaded) = slot.lock().map_err(|_| "model slot poisoned")?.as_ref() {
+ return Ok(Arc::clone(loaded));
+ }
+ let file = models::model_file(&self.data, id);
+ if !models::installed(&self.data, id) {
+ fetch(id, &file, cancel, progress)?;
+ }
+ let segmenter = Arc::new(match id {
+ ModelId::Person => Segmenter::person(&file)?,
+ _ => Segmenter::object(&file)?,
+ });
+ *slot.lock().map_err(|_| "model slot poisoned")? = Some(Arc::clone(&segmenter));
+ Ok(segmenter)
+ }
+
+ /// The person model, or the compiled-in one when the download cannot
+ /// happen: a cutout with no network is a worse cutout, not none.
+ fn person_or_selfie(
+ &self,
+ cancel: &AtomicBool,
+ progress: &mut dyn FnMut(Progress),
+ ) -> Result<Arc<Segmenter>, String> {
+ match self.downloaded(ModelId::Person, cancel, progress) {
+ Ok(model) => Ok(model),
+ Err(error) if error.contains("cancelled") => Err(error),
+ Err(_) => self.selfie(),
+ }
+ }
+
/// How many instants `request` still needs, without doing anything.
+ /// A store made by a model the subject would not pick now needs all
+ /// of them again.
pub fn outstanding(request: &AnalyseRequest) -> usize {
- let store = MaskStore::open(&mask_dir(&request.project, &request.media_path));
+ let store = MaskStore::open(&mask_dir(
+ &request.project,
+ &request.media_path,
+ request.subject,
+ ));
+ let stale = match (request.subject, store.model()) {
+ (_, None) => false,
+ (Subject::Person, Some(model)) => model == ModelId::Object.name(),
+ (Subject::Object, Some(model)) => model != ModelId::Object.name(),
+ (Subject::Auto, _) => false,
+ };
+ if stale {
+ return usize::MAX;
+ }
if request.still {
return usize::from(store.is_empty());
}
@@ -85,32 +177,76 @@ impl Cutouts {
}
/// Fills the masks `request` is missing. Blocks for the whole run, so
- /// run it on its own thread; `progress` is called with `0..=1` as it
- /// goes. Returns how many masks were written.
+ /// run it on its own thread; `progress` is called as it goes. Returns
+ /// how many masks were written.
pub fn analyse(
&self,
request: &AnalyseRequest,
- progress: &mut dyn FnMut(f32),
+ progress: &mut dyn FnMut(Progress),
) -> Result<usize, String> {
let job = self.gate.begin("cutout analysis")?;
- let segmenter = self.segmenter()?;
- let mut store = MaskStore::open(&mask_dir(&request.project, &request.media_path));
+ let cancel = job.cancel_handle();
+ let mut store = MaskStore::open(&mask_dir(
+ &request.project,
+ &request.media_path,
+ request.subject,
+ ));
+ if Self::outstanding(request) == usize::MAX {
+ store.clear();
+ }
+
+ // The model the subject names, or for automatic the one the
+ // store already records; the first frame decides otherwise.
+ let mut segmenter = match request.subject {
+ Subject::Person => Some(self.person_or_selfie(&cancel, progress)?),
+ Subject::Object => Some(self.downloaded(ModelId::Object, &cancel, progress)?),
+ Subject::Auto => match store.model() {
+ Some(model) if model == ModelId::Object.name() => {
+ Some(self.downloaded(ModelId::Object, &cancel, progress)?)
+ }
+ Some(_) => Some(self.person_or_selfie(&cancel, progress)?),
+ None => None,
+ },
+ };
+
+ let (width, height) = request.media_size;
+ let size_for = |segmenter: &Segmenter| segmenter.input_size(width, height);
if request.still {
if !store.is_empty() {
return Ok(0);
}
- let options = DecodeOptions::default()
- .scaled_to(MODEL_SIZE, MODEL_SIZE)
- .limited_to(1);
- let mut decoder =
- Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())?;
- let frame = decoder
- .next_frame()
- .map_err(|error| error.to_string())?
- .ok_or_else(|| format!("{}: no picture to analyse", request.media_path))?;
- store.put(0, &segmenter.mask(&frame)?)?;
- progress(1.0);
+ let frame_with = |segmenter: &Segmenter| {
+ let (w, h) = size_for(segmenter);
+ let options = DecodeOptions::default().scaled_to(w, h).limited_to(1);
+ let mut decoder = Decoder::open(&request.media_path, &options)
+ .map_err(|error| error.to_string())?;
+ decoder
+ .next_frame()
+ .map_err(|error| error.to_string())?
+ .ok_or_else(|| format!("{}: no picture to analyse", request.media_path))
+ };
+ let segmenter = match segmenter {
+ Some(segmenter) => segmenter,
+ None => {
+ let person = self.person_or_selfie(&cancel, progress)?;
+ person.begin();
+ let mask = person.mask(&frame_with(&person)?)?;
+ if mask.is_blank() {
+ self.downloaded(ModelId::Object, &cancel, progress)?
+ } else {
+ store.set_model(person.name())?;
+ store.put(0, &mask)?;
+ progress(Progress::Analysing(1.0));
+ return Ok(1);
+ }
+ }
+ };
+ segmenter.begin();
+ let mask = segmenter.mask(&frame_with(&segmenter)?)?;
+ store.set_model(segmenter.name())?;
+ store.put(0, &mask)?;
+ progress(Progress::Analysing(1.0));
return Ok(1);
}
@@ -120,54 +256,148 @@ impl Cutouts {
return Ok(0);
}
let step_ms = 1000 / u64::from(MASK_RATE);
- let workers = std::thread::available_parallelism()
- .map(|count| count.get())
- .unwrap_or(1)
- .clamp(1, 8);
let mut done = 0usize;
+ progress(Progress::Analysing(0.0));
// Each unbroken run of missing instants is one pass of a decoder
- // paced to the analysis grid, so no frame is sought twice.
+ // paced to the analysis grid, so no frame is sought twice, and one
+ // run of the person model's memory.
for run in runs(&wanted, step_ms) {
let start = Rational::new(run[0] as i64, 1000);
- let options = DecodeOptions::default()
- .starting_at(start)
- .scaled_to(MODEL_SIZE, MODEL_SIZE)
- .at_rate(FrameRate::new(Rational::new(i64::from(MASK_RATE), 1)))
- .limited_to(run.len() as u64);
- let mut decoder =
- Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())?;
+ let mut active = match &segmenter {
+ Some(segmenter) => Arc::clone(segmenter),
+ None => self.person_or_selfie(&cancel, progress)?,
+ };
+ let open = |segmenter: &Segmenter| {
+ let (w, h) = size_for(segmenter);
+ let options = DecodeOptions::default()
+ .starting_at(start)
+ .scaled_to(w, h)
+ .at_rate(FrameRate::new(Rational::new(i64::from(MASK_RATE), 1)))
+ .limited_to(run.len() as u64);
+ Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())
+ };
+ let mut decoder = open(&active)?;
+ active.begin();
let mut next = 0usize;
while next < run.len() {
if job.cancelled() {
return Err("cutout analysis cancelled".to_owned());
}
- // A batch of frames, one per core, found together.
- let mut batch: Vec<(u64, Frame)> = Vec::with_capacity(workers);
- while batch.len() < workers && next < run.len() {
- match decoder.next_frame().map_err(|error| error.to_string())? {
- Some(frame) => batch.push((run[next], frame)),
- // The file ran out before its stated length: what
- // was found is what there is.
- None => break,
- }
- next += 1;
- }
- if batch.is_empty() {
+ let Some(frame) = decoder.next_frame().map_err(|error| error.to_string())? else {
+ // The file ran out before its stated length: what was
+ // found is what there is.
break;
+ };
+ let mut mask = active.mask(&frame)?;
+ if segmenter.is_none() {
+ // Automatic, undecided: the person model has just
+ // looked at the first frame. Nobody there means the
+ // object model takes the whole run, from this frame.
+ if mask.is_blank() && active.kind() != Kind::Object {
+ active = self.downloaded(ModelId::Object, &cancel, progress)?;
+ decoder = open(&active)?;
+ active.begin();
+ let Some(again) = decoder.next_frame().map_err(|error| error.to_string())?
+ else {
+ break;
+ };
+ mask = active.mask(&again)?;
+ }
+ store.set_model(active.name())?;
+ segmenter = Some(Arc::clone(&active));
+ } else if store.model().is_none() {
+ store.set_model(active.name())?;
}
- for (millis, mask) in find_masks(&segmenter, &batch, &job)? {
- store.put(millis, &mask)?;
- done += 1;
- }
- progress(done as f32 / total as f32);
+ store.put(run[next], &mask)?;
+ next += 1;
+ done += 1;
+ progress(Progress::Analysing(done as f32 / total as f32));
}
}
- progress(1.0);
+ progress(Progress::Analysing(1.0));
Ok(done)
}
}
+/// Streams a model into `file`, by way of a `.part` beside it, reporting
+/// every couple of megabytes and stopping when `cancel` is set.
+fn fetch(
+ id: ModelId,
+ file: &Path,
+ cancel: &AtomicBool,
+ progress: &mut dyn FnMut(Progress),
+) -> Result<(), String> {
+ use std::io::{Read, Write};
+
+ let spec = id.spec();
+ if let Some(parent) = file.parent() {
+ std::fs::create_dir_all(parent)
+ .map_err(|error| format!("could not create {}: {error}", parent.display()))?;
+ }
+ let partial = file.with_extension("part");
+ progress(Progress::Fetching {
+ received: 0,
+ total: spec.bytes,
+ });
+
+ // A stalled connection blocks in `read` with the cancel flag
+ // unreachable; thirty seconds without a byte means it is dead.
+ let agent = ureq::AgentBuilder::new()
+ .timeout_connect(std::time::Duration::from_secs(20))
+ .timeout_read(std::time::Duration::from_secs(30))
+ .build();
+ let response = agent
+ .get(spec.url)
+ .call()
+ .map_err(|error| format!("could not fetch the {} model: {error}", spec.file))?;
+ let total = response
+ .header("Content-Length")
+ .and_then(|value| value.parse::<u64>().ok())
+ .unwrap_or(spec.bytes);
+
+ let result = (|| {
+ let mut out = std::fs::File::create(&partial)
+ .map_err(|error| format!("could not create {}: {error}", partial.display()))?;
+ let mut reader = response.into_reader();
+ let mut buffer = [0u8; 64 * 1024];
+ let mut received = 0u64;
+ let mut reported = 0u64;
+ loop {
+ if cancel.load(Ordering::Relaxed) {
+ return Err("cutout analysis cancelled".to_owned());
+ }
+ let read = reader
+ .read(&mut buffer)
+ .map_err(|error| format!("model download interrupted: {error}"))?;
+ if read == 0 {
+ break;
+ }
+ out.write_all(&buffer[..read])
+ .map_err(|error| format!("could not write the model: {error}"))?;
+ received += read as u64;
+ if received - reported >= 2 * 1024 * 1024 {
+ reported = received;
+ progress(Progress::Fetching { received, total });
+ }
+ }
+ out.flush()
+ .map_err(|error| format!("could not write the model: {error}"))?;
+ if received == 0 {
+ return Err(format!("the {} model came back empty", spec.file));
+ }
+ Ok(())
+ })();
+ match result {
+ Ok(()) => std::fs::rename(&partial, file)
+ .map_err(|error| format!("could not finish {}: {error}", file.display())),
+ Err(error) => {
+ let _ = std::fs::remove_file(&partial);
+ Err(error)
+ }
+ }
+}
+
/// Every instant the ranges need that the store lacks, ascending, once.
fn missing(store: &MaskStore, ranges: &[(f64, f64)]) -> Vec<u64> {
let mut wanted: Vec<u64> = ranges
@@ -191,34 +421,6 @@ fn runs(instants: &[u64], step: u64) -> Vec<Vec<u64>> {
out
}
-/// The masks of a batch of frames, one thread each.
-fn find_masks(
- segmenter: &Arc<Segmenter>,
- batch: &[(u64, Frame)],
- job: &Job,
-) -> Result<Vec<(u64, Mask)>, String> {
- if job.cancelled() {
- return Err("cutout analysis cancelled".to_owned());
- }
- std::thread::scope(|scope| {
- let handles: Vec<_> = batch
- .iter()
- .map(|(millis, frame)| {
- let segmenter = Arc::clone(segmenter);
- scope.spawn(move || segmenter.mask(frame).map(|mask| (*millis, mask)))
- })
- .collect();
- handles
- .into_iter()
- .map(|handle| {
- handle
- .join()
- .map_err(|_| "cutout analysis panicked".to_owned())?
- })
- .collect()
- })
-}
-
#[cfg(test)]
mod tests {
use super::*;
diff --git a/engine/crates/concat-project/src/model.rs b/engine/crates/concat-project/src/model.rs
index d3aebc40..b6a673cc 100644
--- a/engine/crates/concat-project/src/model.rs
+++ b/engine/crates/concat-project/src/model.rs
@@ -155,6 +155,9 @@ fn yes() -> bool {
pub struct Cutout {
/// Automatic keeps what the model finds; custom adds the strokes.
pub mode: CutoutMode,
+ /// What the model looks for.
+ #[serde(default, skip_serializing_if = "Subject::is_auto")]
+ pub subject: Subject,
/// How far the edge is softened, as a fraction of the picture's width.
#[serde(default = "default_feather")]
pub feather: f64,
@@ -173,6 +176,37 @@ pub enum CutoutMode {
Custom,
}
+/// What a cutout keeps: the person the matting model finds, the one
+/// thing the picture is of whatever it is, or whichever of those the
+/// analysis decides fits the footage.
+#[derive(Clone, Copy, PartialEq, Eq, Debug, Default, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+pub enum Subject {
+ /// A person when the person model finds one, the object otherwise.
+ #[default]
+ Auto,
+ /// The person model, always.
+ Person,
+ /// The object model, always.
+ Object,
+}
+
+impl Subject {
+ /// The default, left out of the document.
+ pub fn is_auto(&self) -> bool {
+ *self == Subject::Auto
+ }
+
+ /// The name the mask cache is keyed by.
+ pub fn key(&self) -> &'static str {
+ match self {
+ Subject::Auto => "auto",
+ Subject::Person => "person",
+ Subject::Object => "object",
+ }
+ }
+}
+
/// One brush stroke over a cutout: which tool, how wide, and where it went.
/// Points are fractions of the source picture, `(0, 0)` its top-left, so a
/// stroke survives a change of output size, a crop or a flip.
@@ -219,6 +253,7 @@ impl Cutout {
pub fn auto() -> Cutout {
Cutout {
mode: CutoutMode::Auto,
+ subject: Subject::Auto,
feather: DEFAULT_FEATHER,
strokes: Vec::new(),
}
diff --git a/engine/crates/concat-vision/models/NOTICE.md b/engine/crates/concat-vision/models/NOTICE.md
index ffbf636c..4e64def3 100644
--- a/engine/crates/concat-vision/models/NOTICE.md
+++ b/engine/crates/concat-vision/models/NOTICE.md
@@ -1,4 +1,7 @@
-# The cutout model
+# The compiled-in cutout model
+
+The downloaded models are listed in THIRD_PARTY_NOTICES.md at the
+repository root and named in `src/models.rs`.
`selfie-segmentation.onnx` is Google's MediaPipe Selfie Segmentation model
(the general, 256 × 256 variant), converted to ONNX and published by the
diff --git a/engine/crates/concat-vision/src/lib.rs b/engine/crates/concat-vision/src/lib.rs
index 9da8abfa..18cde303 100644
--- a/engine/crates/concat-vision/src/lib.rs
+++ b/engine/crates/concat-vision/src/lib.rs
@@ -19,22 +19,26 @@
//! file and the source instant, and cached in the project folder like
//! its waveforms, so a cutout is found once and travels with the edit.
//!
-//! Finding the mask is [`segment`], behind the `infer` feature: the model
-//! is compiled in and run by ONNX Runtime through [`runtime`], on the
-//! platform's accelerator where there is one, so a cutout needs no
-//! download, on a desk or a phone. The renderer reads masks and never
-//! infers; the host infers and writes them.
+//! Finding the mask is [`segment`], behind the `infer` feature: three
+//! models run by ONNX Runtime through [`runtime`], on the platform's
+//! accelerator where there is one. A person matting model and a general
+//! object model are downloaded on first use ([`models`] says where from);
+//! a small person model is compiled in so a cutout works with nothing
+//! downloaded at all. The renderer reads masks and never infers; the
+//! host infers and writes them.
//!
-//! Masks are square at the model's resolution whatever the picture's
-//! shape, and every position in them is a fraction of the source picture:
-//! `(0, 0)` its top-left, `(1, 1)` its bottom-right. Strokes are stored in
-//! the same fractions. A crop, a flip or a change of output size therefore
-//! changes nothing about a mask - the mapping from a decoded pixel back to
-//! a source fraction is [`apply::Mapping`], and it is the one place those
+//! A mask is whatever shape its model answers in - square for the object
+//! model, the picture's own aspect for the person model - and every
+//! position in one is a fraction of the source picture: `(0, 0)` its
+//! top-left, `(1, 1)` its bottom-right. Strokes are stored in the same
+//! fractions. A crop, a flip or a change of output size therefore changes
+//! nothing about a mask - the mapping from a decoded pixel back to a
+//! source fraction is [`apply::Mapping`], and it is the one place those
//! are undone.
pub mod apply;
pub mod mask;
+pub mod models;
#[cfg(feature = "infer")]
pub mod runtime;
#[cfg(feature = "infer")]
@@ -44,6 +48,7 @@ pub mod strokes;
pub use apply::{Mapping, cut};
pub use mask::Mask;
+pub use models::ModelId;
#[cfg(feature = "infer")]
pub use segment::Segmenter;
pub use store::{MaskStore, mask_dir};
@@ -53,9 +58,9 @@ pub use store::{MaskStore, mask_dir};
/// minute of footage is six hundred inferences rather than eighteen hundred.
pub const MASK_RATE: u32 = 10;
-/// The model's input and output edge, in pixels.
+/// The compiled-in person model's input and output edge, in pixels.
pub const MODEL_SIZE: u32 = 256;
-/// Names the model a mask came from. Part of the cache directory's name,
-/// so a different model never serves another's masks.
+/// Names the compiled-in model. A mask store records which model made
+/// its masks, so a different model never serves another's.
pub const MODEL_ID: &str = "selfie-256";
diff --git a/engine/crates/concat-vision/src/mask.rs b/engine/crates/concat-vision/src/mask.rs
index 97b7e098..7be3ab67 100644
--- a/engine/crates/concat-vision/src/mask.rs
+++ b/engine/crates/concat-vision/src/mask.rs
@@ -5,6 +5,10 @@
use std::io::Cursor;
+/// A mask with every value under this is nobody's: the model found
+/// nothing, and the picture should show as shot rather than vanish.
+const BLANK_BELOW: u8 = 10;
+
/// An eight-bit picture of probabilities, `0` background and `255` subject,
/// covering the whole source picture whatever its shape.
#[derive(Clone, PartialEq, Eq, Debug)]
@@ -55,6 +59,37 @@ impl Mask {
&mut self.data
}
+ /// True when the model found nothing: no pixel is more than a few
+ /// percent likely to be the subject.
+ pub fn is_blank(&self) -> bool {
+ self.data.iter().all(|&v| v < BLANK_BELOW)
+ }
+
+ /// The mask shrunk by `factor` in each direction, each new pixel the
+ /// mean of the block it replaces. A large model's answer stored at a
+ /// size the renderer can read quickly.
+ pub fn shrunk(&self, factor: u32) -> Mask {
+ let factor = factor.max(1);
+ if factor == 1 {
+ return self.clone();
+ }
+ let width = (self.width / factor).max(1);
+ let height = (self.height / factor).max(1);
+ let mut data = Vec::with_capacity((width * height) as usize);
+ for y in 0..height {
+ for x in 0..width {
+ let mut sum = 0u32;
+ for dy in 0..factor {
+ for dx in 0..factor {
+ sum += u32::from(self.at(i64::from(x * factor + dx), i64::from(y * factor + dy)));
+ }
+ }
+ data.push((sum / (factor * factor)) as u8);
+ }
+ }
+ Mask { width, height, data }
+ }
+
/// The value at a pixel, clamped to the edge.
pub fn at(&self, x: i64, y: i64) -> u8 {
let x = x.clamp(0, i64::from(self.width) - 1) as usize;
diff --git a/engine/crates/concat-vision/src/models.rs b/engine/crates/concat-vision/src/models.rs
new file mode 100644
index 00000000..1f9bb98b
--- /dev/null
+++ b/engine/crates/concat-vision/src/models.rs
@@ -0,0 +1,127 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! The models that are downloaded rather than compiled in: what they are
+//! called, where they come from, and where they live once fetched.
+//!
+//! Each is one ONNX file in `<app data>/cutout-models/`, named as the
+//! table says; a file that is there and whole is an installed model, and
+//! a fetch lands in a `.part` beside it and renames at the end, so a
+//! killed download leaves nothing that could be mistaken for one. The
+//! fetching itself is the host's, since it is the host that has a network
+//! and a job slot; this crate only knows the table.
+
+use std::path::{Path, PathBuf};
+
+/// The downloadable models.
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
+pub enum ModelId {
+ /// Robust Video Matting on MobileNetV3: a person, with a true alpha
+ /// edge, carrying state from one frame to the next.
+ Person,
+ /// IS-Net from the DIS paper: the one thing a picture is of, whatever
+ /// it is.
+ Object,
+ /// SlimSAM's image encoder: a picture into the embeddings the brush
+ /// decoder reads.
+ BrushEncoder,
+ /// SlimSAM's prompt encoder and mask decoder: points on the picture
+ /// into the mask of what is under them.
+ BrushDecoder,
+}
+
+/// One model as the table describes it.
+#[derive(Clone, Copy, Debug)]
+pub struct ModelSpec {
+ /// Which model.
+ pub id: ModelId,
+ /// The file name on disk, and what a mask store records.
+ pub file: &'static str,
+ /// Where it is fetched from.
+ pub url: &'static str,
+ /// Its size, for a progress bar before the server says.
+ pub bytes: u64,
+ /// The licence it comes under, for the notices.
+ pub licence: &'static str,
+}
+
+/// Every downloadable model.
+pub const MODELS: [ModelSpec; 4] = [
+ ModelSpec {
+ id: ModelId::Person,
+ file: "rvm-mobilenetv3.onnx",
+ url: "https://github.com/PeterL1n/RobustVideoMatting/releases/download/v1.0.0/rvm_mobilenetv3_fp32.onnx",
+ bytes: 14_975_696,
+ licence: "GPL-3.0",
+ },
+ ModelSpec {
+ id: ModelId::Object,
+ file: "isnet-general-use.onnx",
+ url: "https://github.com/danielgatis/rembg/releases/download/v0.0.0/isnet-general-use.onnx",
+ bytes: 178_648_008,
+ licence: "Apache-2.0",
+ },
+ ModelSpec {
+ id: ModelId::BrushEncoder,
+ file: "slimsam-77-encoder.onnx",
+ url: "https://huggingface.co/Xenova/slimsam-77-uniform/resolve/main/onnx/vision_encoder.onnx",
+ bytes: 23_276_014,
+ licence: "Apache-2.0",
+ },
+ ModelSpec {
+ id: ModelId::BrushDecoder,
+ file: "slimsam-77-decoder.onnx",
+ url: "https://huggingface.co/Xenova/slimsam-77-uniform/resolve/main/onnx/prompt_encoder_mask_decoder.onnx",
+ bytes: 16_557_892,
+ licence: "Apache-2.0",
+ },
+];
+
+impl ModelId {
+ /// The table's row for this model.
+ pub fn spec(self) -> &'static ModelSpec {
+ MODELS
+ .iter()
+ .find(|spec| spec.id == self)
+ .expect("every model is in the table")
+ }
+
+ /// The name a mask store records for masks this model made.
+ pub fn name(self) -> &'static str {
+ match self {
+ ModelId::Person => "rvm-mobilenetv3",
+ ModelId::Object => "isnet-1024",
+ ModelId::BrushEncoder => "slimsam-77-encoder",
+ ModelId::BrushDecoder => "slimsam-77-decoder",
+ }
+ }
+}
+
+/// Where the models live: `<app data>/cutout-models/`.
+pub fn models_dir(data: &Path) -> PathBuf {
+ data.join("cutout-models")
+}
+
+/// The file a model is, once fetched.
+pub fn model_file(data: &Path, id: ModelId) -> PathBuf {
+ models_dir(data).join(id.spec().file)
+}
+
+/// Whether a model is on disk and whole.
+pub fn installed(data: &Path, id: ModelId) -> bool {
+ std::fs::metadata(model_file(data, id)).is_ok_and(|meta| meta.is_file() && meta.len() > 0)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn every_model_has_its_own_file_and_name() {
+ let files: std::collections::HashSet<_> = MODELS.iter().map(|m| m.file).collect();
+ assert_eq!(files.len(), MODELS.len());
+ let names: std::collections::HashSet<_> = MODELS.iter().map(|m| m.id.name()).collect();
+ assert_eq!(names.len(), MODELS.len());
+ assert!(!installed(Path::new("/nowhere"), ModelId::Person));
+ }
+}
diff --git a/engine/crates/concat-vision/src/runtime.rs b/engine/crates/concat-vision/src/runtime.rs
index 128dd092..a4cb4f42 100644
--- a/engine/crates/concat-vision/src/runtime.rs
+++ b/engine/crates/concat-vision/src/runtime.rs
@@ -102,13 +102,28 @@ impl Model {
/// cannot supply is skipped, not an error, so the same build runs on a
/// box without one.
fn accelerators() -> Vec<ort::ep::ExecutionProviderDispatch> {
- let mut providers = Vec::new();
- #[cfg(target_vendor = "apple")]
- providers.push(ort::ep::CoreML::default().build());
- #[cfg(target_os = "windows")]
- providers.push(ort::ep::DirectML::default().build());
- #[cfg(target_os = "android")]
- providers.push(ort::ep::NNAPI::default().build());
- providers.push(ort::ep::CPU::default().build());
- providers
+ platform_accelerator()
+ .into_iter()
+ .chain(std::iter::once(ort::ep::CPU::default().build()))
+ .collect()
+}
+
+#[cfg(target_vendor = "apple")]
+fn platform_accelerator() -> Option<ort::ep::ExecutionProviderDispatch> {
+ Some(ort::ep::CoreML::default().build())
+}
+
+#[cfg(target_os = "windows")]
+fn platform_accelerator() -> Option<ort::ep::ExecutionProviderDispatch> {
+ Some(ort::ep::DirectML::default().build())
+}
+
+#[cfg(target_os = "android")]
+fn platform_accelerator() -> Option<ort::ep::ExecutionProviderDispatch> {
+ Some(ort::ep::NNAPI::default().build())
+}
+
+#[cfg(not(any(target_vendor = "apple", target_os = "windows", target_os = "android")))]
+fn platform_accelerator() -> Option<ort::ep::ExecutionProviderDispatch> {
+ None
}
diff --git a/engine/crates/concat-vision/src/segment.rs b/engine/crates/concat-vision/src/segment.rs
index e4dab2f6..cf46910c 100644
--- a/engine/crates/concat-vision/src/segment.rs
+++ b/engine/crates/concat-vision/src/segment.rs
@@ -1,93 +1,290 @@
// SPDX-License-Identifier: AGPL-3.0-or-later
// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
-//! Finding the person in a picture.
+//! Finding the subject in a picture.
//!
-//! The model is MediaPipe's selfie segmentation, compiled into the binary
-//! (see `models/NOTICE.md`) and run by ONNX Runtime. It takes a 256 × 256
-//! RGB picture and answers with a probability per pixel; a source of any
-//! shape is squashed to the square on the way in and the mask is read as
-//! covering the whole source on the way out, which is the convention every
-//! other part of this crate keeps.
+//! Three models, one interface: a frame in, a probability per pixel out.
//!
-//! One inference is a few milliseconds with the platform's accelerator and
-//! about twenty on a laptop's cores without.
+//! - **Selfie**: MediaPipe's selfie segmentation, compiled into the binary
+//! (see `models/NOTICE.md`). A person, at 256 × 256, in a few
+//! milliseconds; the answer when nothing has been downloaded.
+//! - **Person**: Robust Video Matting on MobileNetV3. A person with a true
+//! alpha edge - hair, motion blur - at the picture's own aspect, carrying
+//! a recurrent state from one frame to the next so an outline does not
+//! flicker. Frames must therefore arrive in order: [`Segmenter::begin`]
+//! starts a run.
+//! - **Object**: IS-Net. The one thing the picture is of, whatever it is,
+//! at 1024 × 1024; the answer for a car, a product, a cartoon.
+//!
+//! A source of any shape is resampled to the model's size on the way in
+//! and the mask is read as covering the whole source on the way out, which
+//! is the convention every other part of this crate keeps.
+use std::path::Path;
use std::sync::Mutex;
use concat_core::frame::{BYTES_PER_PIXEL, Frame};
-use crate::runtime::{Input, Model};
-use crate::{MODEL_SIZE, Mask};
+use crate::models::ModelId;
+use crate::runtime::{Input, Model, Output};
+use crate::{MODEL_ID, MODEL_SIZE, Mask};
+
+/// The compiled-in model.
+const SELFIE: &[u8] = include_bytes!("../models/selfie-segmentation.onnx");
-/// The model as the crate ships it.
-const MODEL: &[u8] = include_bytes!("../models/selfie-segmentation.onnx");
+/// The person model's long edge: its stage one works at up to 512, and a
+/// larger frame would only be shrunk inside it.
+const PERSON_EDGE: u32 = 512;
+/// The object model's fixed input.
+const OBJECT_SIZE: u32 = 1024;
+/// The object model's answer is stored at half its size: a quarter of the
+/// bytes, and still twice the person model's.
+const OBJECT_SHRINK: u32 = 2;
+
+/// Which model a segmenter runs.
+#[derive(Clone, Copy, PartialEq, Eq, Debug)]
+pub enum Kind {
+ /// The compiled-in person model.
+ Selfie,
+ /// The downloaded person matting model.
+ Person,
+ /// The downloaded object model.
+ Object,
+}
-/// A loaded model, ready to run. `Send + Sync`: one per process is enough,
-/// and the runtime's own threads take a frame across the cores.
+/// A loaded model, ready to run. `Send + Sync`; the runtime's own threads
+/// take a frame across the cores, so one per process is enough.
pub struct Segmenter {
+ kind: Kind,
model: Mutex<Model>,
+ /// The person model's recurrent state from the frame before, within
+ /// one run.
+ state: Mutex<Option<Vec<Output>>>,
}
impl Segmenter {
- /// Loads the compiled-in model. A few tens of milliseconds, once.
+ /// The compiled-in person model. A few tens of milliseconds, once.
+ pub fn selfie() -> Result<Segmenter, String> {
+ Ok(Segmenter {
+ kind: Kind::Selfie,
+ model: Mutex::new(Model::from_bytes(SELFIE)?),
+ state: Mutex::new(None),
+ })
+ }
+
+ /// The compiled-in model; see [`Segmenter::selfie`].
pub fn load() -> Result<Segmenter, String> {
+ Segmenter::selfie()
+ }
+
+ /// The person matting model, from its downloaded file.
+ pub fn person(file: &Path) -> Result<Segmenter, String> {
+ Ok(Segmenter {
+ kind: Kind::Person,
+ model: Mutex::new(Model::from_file(file)?),
+ state: Mutex::new(None),
+ })
+ }
+
+ /// The object model, from its downloaded file.
+ pub fn object(file: &Path) -> Result<Segmenter, String> {
Ok(Segmenter {
- model: Mutex::new(Model::from_bytes(MODEL)?),
+ kind: Kind::Object,
+ model: Mutex::new(Model::from_file(file)?),
+ state: Mutex::new(None),
})
}
- /// The person mask of `frame`, at the model's resolution. A frame of
- /// any size: it is resampled to the square first.
+ /// Which model this is.
+ pub fn kind(&self) -> Kind {
+ self.kind
+ }
+
+ /// The name a mask store records for this model's masks.
+ pub fn name(&self) -> &'static str {
+ match self.kind {
+ Kind::Selfie => MODEL_ID,
+ Kind::Person => ModelId::Person.name(),
+ Kind::Object => ModelId::Object.name(),
+ }
+ }
+
+ /// The size to decode a source of `width` × `height` to for this
+ /// model: its fixed square, or the source's shape at the person model's
+ /// working edge.
+ pub fn input_size(&self, width: u32, height: u32) -> (u32, u32) {
+ match self.kind {
+ Kind::Selfie => (MODEL_SIZE, MODEL_SIZE),
+ Kind::Object => (OBJECT_SIZE, OBJECT_SIZE),
+ Kind::Person => {
+ let (width, height) = (width.max(1) as f64, height.max(1) as f64);
+ let scale = f64::from(PERSON_EDGE) / width.max(height);
+ // Multiples of sixteen, so every stride inside the network
+ // divides evenly and nothing is padded off the edge.
+ let round = |edge: f64| (((edge * scale) / 16.0).round() as u32 * 16).max(64);
+ (round(width), round(height))
+ }
+ }
+ }
+
+ /// Starts a run of frames in source order. The person model forgets
+ /// what it carried from the frame before; the others carry nothing.
+ pub fn begin(&self) {
+ if let Ok(mut state) = self.state.lock() {
+ *state = None;
+ }
+ }
+
+ /// The subject mask of `frame`. A frame of any size: it is resampled
+ /// to the model's size first, so the decoder is best asked for
+ /// [`Segmenter::input_size`] and the resample is a copy.
pub fn mask(&self, frame: &Frame) -> Result<Mask, String> {
- let size = MODEL_SIZE as usize;
- let width = frame.width() as usize;
- let height = frame.height() as usize;
- if width == 0 || height == 0 {
+ if frame.width() == 0 || frame.height() == 0 {
return Err("cutout: an empty frame".to_owned());
}
- let pixels = frame.pixels();
- let mut data = vec![0f32; 3 * size * size];
- for c in 0..3 {
- for y in 0..size {
- // Nearest sample: the decoder already scaled to the square
- // in the analysis path, and a still is fine at this size.
- let sy = (y * height / size).min(height - 1);
- for x in 0..size {
- let sx = (x * width / size).min(width - 1);
- data[(c * size + y) * size + x] =
- f32::from(pixels[(sy * width + sx) * BYTES_PER_PIXEL + c]) / 255.0;
- }
- }
+ let (width, height) = self.input_size(frame.width(), frame.height());
+ let planes = planes(frame, width, height);
+ match self.kind {
+ Kind::Selfie => self.selfie_mask(planes, width, height),
+ Kind::Object => self.object_mask(planes, width, height),
+ Kind::Person => self.person_mask(planes, width, height),
}
+ }
+
+ fn run(&self, inputs: Vec<Input<'_>>, outputs: &[&str]) -> Result<Vec<Output>, String> {
+ self.model
+ .lock()
+ .map_err(|_| "cutout model poisoned".to_owned())?
+ .run(inputs, outputs)
+ }
+
+ fn selfie_mask(&self, planes: Vec<f32>, width: u32, height: u32) -> Result<Mask, String> {
+ let size = (width * height) as usize;
let input = Input {
name: "pixel_values",
- dims: vec![1, 3, size, size],
- data,
+ dims: vec![1, 3, height as usize, width as usize],
+ data: planes,
};
- let outputs = self
- .model
- .lock()
- .map_err(|_| "cutout model poisoned".to_owned())?
- .run(vec![input], &["alphas"])?;
- let data: Vec<u8> = outputs[0]
- .data
+ let outputs = self.run(vec![input], &["alphas"])?;
+ probabilities(&outputs[0].data, size, width, height)
+ }
+
+ fn object_mask(&self, mut planes: Vec<f32>, width: u32, height: u32) -> Result<Mask, String> {
+ // As the model was trained: the picture scaled by its brightest
+ // value, then centred on a half.
+ let brightest = planes.iter().copied().fold(0.0f32, f32::max).max(1e-6);
+ for value in &mut planes {
+ *value = *value / brightest - 0.5;
+ }
+ let size = (width * height) as usize;
+ let input = Input {
+ name: "input_image",
+ dims: vec![1, 3, height as usize, width as usize],
+ data: planes,
+ };
+ let outputs = self.run(vec![input], &["output_image"])?;
+ // The model answers in its own range; the span of the answer is
+ // the mask, as the reference implementation reads it.
+ let data = &outputs[0].data[..size.min(outputs[0].data.len())];
+ let low = data.iter().copied().fold(f32::INFINITY, f32::min);
+ let high = data.iter().copied().fold(f32::NEG_INFINITY, f32::max);
+ let span = (high - low).max(1e-6);
+ let bytes: Vec<u8> = data
.iter()
- .take(size * size)
- .map(|&p: &f32| (p.clamp(0.0, 1.0) * 255.0 + 0.5) as u8)
+ .map(|&p| (((p - low) / span).clamp(0.0, 1.0) * 255.0 + 0.5) as u8)
.collect();
- Mask::from_bytes(MODEL_SIZE, MODEL_SIZE, data)
- .ok_or_else(|| "cutout: the model's answer is the wrong size".to_owned())
+ let mask = Mask::from_bytes(width, height, bytes)
+ .ok_or_else(|| "cutout: the model's answer is the wrong size".to_owned())?;
+ Ok(mask.shrunk(OBJECT_SHRINK))
+ }
+
+ fn person_mask(&self, planes: Vec<f32>, width: u32, height: u32) -> Result<Mask, String> {
+ let size = (width * height) as usize;
+ let mut inputs = vec![Input {
+ name: "src",
+ dims: vec![1, 3, height as usize, width as usize],
+ data: planes,
+ }];
+ let carried = self
+ .state
+ .lock()
+ .map_err(|_| "cutout state poisoned".to_owned())?
+ .take();
+ const STATES: [&str; 4] = ["r1i", "r2i", "r3i", "r4i"];
+ match carried {
+ Some(state) if state.len() == 4 => {
+ for (name, output) in STATES.iter().zip(state) {
+ inputs.push(Input {
+ name,
+ dims: output.dims,
+ data: output.data,
+ });
+ }
+ }
+ _ => {
+ for name in STATES {
+ inputs.push(Input {
+ name,
+ dims: vec![1, 1, 1, 1],
+ data: vec![0.0],
+ });
+ }
+ }
+ }
+ // The frame is already at the working size; the model's own
+ // downsample stays at one.
+ inputs.push(Input {
+ name: "downsample_ratio",
+ dims: vec![1],
+ data: vec![1.0],
+ });
+ let mut outputs = self.run(inputs, &["pha", "r1o", "r2o", "r3o", "r4o"])?;
+ let alpha = outputs.remove(0);
+ if let Ok(mut state) = self.state.lock() {
+ *state = Some(outputs);
+ }
+ probabilities(&alpha.data, size, width, height)
}
}
+/// The frame as the models read it: three planes of `0..=1`, red then
+/// green then blue, at `width` × `height`, nearest-sampled from the frame.
+fn planes(frame: &Frame, width: u32, height: u32) -> Vec<f32> {
+ let (fw, fh) = (frame.width() as usize, frame.height() as usize);
+ let (width, height) = (width as usize, height as usize);
+ let pixels = frame.pixels();
+ let mut data = vec![0f32; 3 * width * height];
+ for c in 0..3 {
+ for y in 0..height {
+ let sy = (y * fh / height).min(fh - 1);
+ for x in 0..width {
+ let sx = (x * fw / width).min(fw - 1);
+ data[(c * height + y) * width + x] =
+ f32::from(pixels[(sy * fw + sx) * BYTES_PER_PIXEL + c]) / 255.0;
+ }
+ }
+ }
+ data
+}
+
+/// A model's `0..=1` answer as a mask.
+fn probabilities(data: &[f32], size: usize, width: u32, height: u32) -> Result<Mask, String> {
+ let bytes: Vec<u8> = data
+ .iter()
+ .take(size)
+ .map(|&p| (p.clamp(0.0, 1.0) * 255.0 + 0.5) as u8)
+ .collect();
+ Mask::from_bytes(width, height, bytes)
+ .ok_or_else(|| "cutout: the model's answer is the wrong size".to_owned())
+}
+
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_model_loads_and_answers_a_mask_of_its_size() {
- let segmenter = Segmenter::load().expect("loads");
+ let segmenter = Segmenter::selfie().expect("loads");
// A flat grey picture is nobody: the mask should lean background.
let mut frame = Frame::black(64, 48);
for pixel in frame.pixels_mut().chunks_exact_mut(4) {
@@ -98,4 +295,16 @@ mod tests {
let mean = mask.bytes().iter().map(|&v| u32::from(v)).sum::<u32>() / (256 * 256);
assert!(mean < 128, "a blank picture read as mostly person: {mean}");
}
+
+ #[test]
+ fn the_person_model_works_at_the_pictures_shape() {
+ let segmenter = Segmenter {
+ kind: Kind::Person,
+ model: Mutex::new(Model::from_bytes(SELFIE).expect("loads")),
+ state: Mutex::new(None),
+ };
+ assert_eq!(segmenter.input_size(1920, 1080), (512, 288));
+ assert_eq!(segmenter.input_size(1080, 1920), (288, 512));
+ assert_eq!(segmenter.input_size(1000, 1000), (512, 512));
+ }
}
diff --git a/engine/crates/concat-vision/src/store.rs b/engine/crates/concat-vision/src/store.rs
index 2aa52eca..8c49214a 100644
--- a/engine/crates/concat-vision/src/store.rs
+++ b/engine/crates/concat-vision/src/store.rs
@@ -3,12 +3,13 @@
//! Where masks live between runs.
//!
-//! One directory per media file per model, inside the project's `cache/`
+//! One directory per media file per subject, inside the project's `cache/`
//! folder so it travels with the project and vanishes with it, holding one
-//! PNG per analysed source instant, named by that instant in milliseconds.
-//! The host writes them as it analyses; the renderer opens the directory,
-//! reads the names, and asks for the mask nearest a source time. Nothing
-//! is ever indexed: the file names are the index.
+//! PNG per analysed source instant, named by that instant in milliseconds,
+//! and a `model` file naming the model that made them. The host writes
+//! them as it analyses; the renderer opens the directory, reads the names,
+//! and asks for the mask nearest a source time. Nothing is ever indexed:
+//! the file names are the index.
//!
//! Decoded masks are kept in a process-wide cache, since a preview asks for
//! the same handful over and over while the playhead sits still, and an
@@ -18,9 +19,9 @@ use std::collections::{HashMap, VecDeque};
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex, OnceLock};
-use concat_project::model::{Cutout, CutoutMode};
+use concat_project::model::{Cutout, CutoutMode, Subject};
-use crate::{MASK_RATE, MODEL_ID, Mask, strokes};
+use crate::{MASK_RATE, Mask, strokes};
/// A mask further than this from the instant asked for is nobody's answer:
/// two and a half analysis steps, so a gap in the cache reads as "not yet"
@@ -31,17 +32,23 @@ const REACH_MS: u64 = 2500 / MASK_RATE as u64;
/// mask is 64 KB; both together stay under forty megabytes.
const CACHED: usize = 300;
-/// The directory a media file's masks live in under `project`.
+/// The directory a media file's masks for `subject` live in under
+/// `project`.
///
/// Named by a hash of the path rather than the path, like the peaks cache
/// and for the same reason: a flat, portable name that cannot escape the
-/// folder. The model's id is in it too, so a different model never
-/// answers for this one.
-pub fn mask_dir(project: &Path, media_path: &str) -> PathBuf {
+/// folder. The subject is in it too, so a clip asked to keep the person
+/// and one asked to keep the object never read each other's answer.
+pub fn mask_dir(project: &Path, media_path: &str, subject: Subject) -> PathBuf {
project
.join("cache")
.join("masks")
- .join(format!("{:016x}-{MODEL_ID}", fnv1a(media_path.as_bytes())))
+ .join(format!("{:016x}-{}", fnv1a(media_path.as_bytes()), subject.key()))
+}
+
+/// The file naming the model that made a directory's masks.
+fn model_file(dir: &Path) -> PathBuf {
+ dir.join("model")
}
/// The file for the mask at `millis` of source.
@@ -65,6 +72,8 @@ pub struct MaskStore {
dir: PathBuf,
/// The analysed instants, ascending, in milliseconds of source.
times: Vec<u64>,
+ /// The model that made them, as the directory's `model` file says.
+ model: Option<String>,
}
impl MaskStore {
@@ -83,9 +92,14 @@ impl MaskStore {
.collect();
times.sort_unstable();
times.dedup();
+ let model = std::fs::read_to_string(model_file(dir))
+ .ok()
+ .map(|name| name.trim().to_owned())
+ .filter(|name| !name.is_empty());
MaskStore {
dir: dir.to_path_buf(),
times,
+ model,
}
}
@@ -94,6 +108,28 @@ impl MaskStore {
&self.dir
}
+ /// The model that made the masks here, once one has.
+ pub fn model(&self) -> Option<&str> {
+ self.model.as_deref()
+ }
+
+ /// Records the model about to fill the store. Masks from another
+ /// model go first: one directory, one model.
+ pub fn set_model(&mut self, name: &str) -> Result<(), String> {
+ if self.model.as_deref() == Some(name) {
+ return Ok(());
+ }
+ if self.model.is_some() {
+ self.clear();
+ }
+ std::fs::create_dir_all(&self.dir)
+ .map_err(|error| format!("could not create {}: {error}", self.dir.display()))?;
+ std::fs::write(model_file(&self.dir), name)
+ .map_err(|error| format!("could not write {}: {error}", self.dir.display()))?;
+ self.model = Some(name.to_owned());
+ Ok(())
+ }
+
/// How many instants are analysed.
pub fn len(&self) -> usize {
self.times.len()
@@ -133,6 +169,9 @@ impl MaskStore {
/// The mask to cut with at `seconds`: the model's, with the cutout's
/// strokes painted on when it is custom, and softened by its feather.
+ /// `None` when the instant has no mask yet, and also when the model
+ /// found nothing there and no stroke says otherwise: a picture with
+ /// no subject shows as shot rather than vanishing.
/// `aspect` is the source's width over its height, for round brushes.
pub fn resolved(&self, seconds: f64, cutout: &Cutout, aspect: f32) -> Option<Arc<Mask>> {
let millis = self.nearest_millis(seconds)?;
@@ -149,6 +188,10 @@ impl MaskStore {
return Some(hit);
}
let auto = load(&file)?;
+ let corrected = cutout.mode == CutoutMode::Custom && !cutout.strokes.is_empty();
+ if !corrected && auto.is_blank() {
+ return None;
+ }
let painted = match cutout.mode {
CutoutMode::Custom if !cutout.strokes.is_empty() => {
strokes::paint(&auto, &cutout.strokes, aspect)
@@ -213,6 +256,8 @@ impl MaskStore {
cache.retain(|key| !key.file.starts_with(&self.dir));
}
self.times.clear();
+ let _ = std::fs::remove_file(model_file(&self.dir));
+ self.model = None;
let _ = std::fs::remove_dir(&self.dir);
}
}
@@ -332,11 +377,13 @@ mod tests {
#[test]
fn the_directory_is_named_by_the_file_and_the_model() {
- let a = mask_dir(Path::new("/p"), "/footage/a.mp4");
- let b = mask_dir(Path::new("/p"), "/footage/b.mp4");
+ let a = mask_dir(Path::new("/p"), "/footage/a.mp4", Subject::Auto);
+ let b = mask_dir(Path::new("/p"), "/footage/b.mp4", Subject::Auto);
+ let c = mask_dir(Path::new("/p"), "/footage/a.mp4", Subject::Object);
+ assert_ne!(a, c);
assert_ne!(a, b);
assert!(a.starts_with("/p/cache/masks"));
- assert!(a.to_string_lossy().ends_with(MODEL_ID));
+ assert!(a.to_string_lossy().ends_with("-auto"));
}
#[test]
@@ -374,14 +421,15 @@ mod tests {
fn a_resolved_mask_carries_the_strokes_and_the_feather() {
let dir = scratch("resolved");
let mut store = MaskStore::open(&dir);
- store.put(0, &Mask::filled(16, 16, 0)).expect("writes");
+ store.put(0, &Mask::filled(16, 16, 20)).expect("writes");
let plain = Cutout::auto();
assert_eq!(
store.resolved(0.0, &plain, 1.0).map(|m| m.at(8, 8)),
- Some(0)
+ Some(20)
);
let custom = Cutout {
mode: CutoutMode::Custom,
+ subject: Subject::Auto,
feather: 0.0,
strokes: vec![Stroke {
tool: BrushTool::Brush,
@@ -400,11 +448,47 @@ mod tests {
};
assert_eq!(
store.resolved(0.0, &back_to_auto, 1.0).map(|m| m.at(8, 8)),
- Some(0)
+ Some(20)
+ );
+ let _ = std::fs::remove_dir_all(&dir);
+ }
+
+ #[test]
+ fn a_blank_mask_shows_the_picture_as_shot_unless_painted() {
+ let dir = scratch("blank");
+ let mut store = MaskStore::open(&dir);
+ store.put(0, &Mask::filled(16, 16, 0)).expect("writes");
+ assert!(store.resolved(0.0, &Cutout::auto(), 1.0).is_none());
+ let painted = Cutout {
+ mode: CutoutMode::Custom,
+ subject: Subject::Auto,
+ feather: 0.0,
+ strokes: vec![Stroke {
+ tool: BrushTool::Brush,
+ size: 0.5,
+ points: vec![[0.5, 0.5]],
+ }],
+ };
+ assert_eq!(
+ store.resolved(0.0, &painted, 1.0).map(|m| m.at(8, 8)),
+ Some(255)
);
let _ = std::fs::remove_dir_all(&dir);
}
+ #[test]
+ fn the_store_remembers_its_model_and_clears_for_another() {
+ let dir = scratch("model");
+ let mut store = MaskStore::open(&dir);
+ store.set_model("first").expect("writes");
+ store.put(0, &Mask::filled(4, 4, 200)).expect("writes");
+ assert_eq!(MaskStore::open(&dir).model(), Some("first"));
+ store.set_model("second").expect("writes");
+ assert!(store.is_empty());
+ assert_eq!(store.model(), Some("second"));
+ let _ = std::fs::remove_dir_all(&dir);
+ }
+
#[test]
fn clearing_forgets_disk_and_memory() {
let dir = scratch("clear");
diff --git a/engine/crates/concat/locales/de.json b/engine/crates/concat/locales/de.json
index 8ec7a14a..4b1199f4 100644
--- a/engine/crates/concat/locales/de.json
+++ b/engine/crates/concat/locales/de.json
@@ -181,6 +181,7 @@
"Favourites": "Favoriten",
"Feather": "Weiche Kante",
"female": "weiblich",
+ "Fetching the model… {0}%": "Modell wird geladen… {0}%",
"File": "Datei",
"Fill": "Füllung",
"Film Grain": "Filmkorn",
@@ -312,12 +313,14 @@
"Normalize Loudness": "Lautheit normalisieren",
"Normalize loudness": "Lautheit normalisieren",
"not installed": "nicht installiert",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Hier wurde nichts gefunden, das bleiben könnte, daher wird das Bild wie aufgenommen gezeigt. Versuche ein anderes Motiv oder das manuelle Entfernen.",
"Nothing of that kind in this project.": "Nichts dieser Art in diesem Projekt.",
"Nothing starred yet — hover a tile and press its star.": "Noch nichts markiert – über eine Kachel fahren und den Stern drücken.",
"Nothing to export": "Nichts zu exportieren",
"Nothing to read yet": "Noch nichts zu lesen",
"Nothing under the playhead": "Nichts unter dem Abspielkopf",
"Nothing was said in that clip": "In diesem Clip wurde nichts gesagt",
+ "Object": "Objekt",
"off": "aus",
"One repeat, fading.": "Eine Wiederholung, ausklingend.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Ein Titel pro gesprochenem Satz, im Takt der Wörter geschnitten, auf der ersten freien Spur.",
@@ -332,6 +335,7 @@
"Pace": "Tempo",
"Paste": "Einfügen",
"Pause": "Pause",
+ "Person": "Person",
"Pick an effect or filter from the library": "Wähle einen Effekt oder Filter aus der Bibliothek",
"Pitch": "Tonhöhe",
"Pixelate": "Verpixeln",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "Stein und Höhe: Reflexionen, die eine Weile brauchen.",
"Strength": "Stärke",
"Stylize": "Stilisieren",
+ "Subject": "Motiv",
"Subtitle": "Untertitel",
"Summary": "Zusammenfassung",
"Sweet": "Süß",
diff --git a/engine/crates/concat/locales/en.json b/engine/crates/concat/locales/en.json
index 5937f17f..5d526879 100644
--- a/engine/crates/concat/locales/en.json
+++ b/engine/crates/concat/locales/en.json
@@ -181,6 +181,7 @@
"Favourites": "Favourites",
"Feather": "Feather",
"female": "female",
+ "Fetching the model… {0}%": "Fetching the model… {0}%",
"File": "File",
"Fill": "Fill",
"Film Grain": "Film Grain",
@@ -312,12 +313,14 @@
"Normalize Loudness": "Normalize Loudness",
"Normalize loudness": "Normalize loudness",
"not installed": "not installed",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.",
"Nothing of that kind in this project.": "Nothing of that kind in this project.",
"Nothing starred yet — hover a tile and press its star.": "Nothing starred yet — hover a tile and press its star.",
"Nothing to export": "Nothing to export",
"Nothing to read yet": "Nothing to read yet",
"Nothing under the playhead": "Nothing under the playhead",
"Nothing was said in that clip": "Nothing was said in that clip",
+ "Object": "Object",
"off": "off",
"One repeat, fading.": "One repeat, fading.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "One title per phrase spoken, cut to the words' own timing, on the first lane with room.",
@@ -332,6 +335,7 @@
"Pace": "Pace",
"Paste": "Paste",
"Pause": "Pause",
+ "Person": "Person",
"Pick an effect or filter from the library": "Pick an effect or filter from the library",
"Pitch": "Pitch",
"Pixelate": "Pixelate",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "Stone and height: reflections that take a while to arrive.",
"Strength": "Strength",
"Stylize": "Stylize",
+ "Subject": "Subject",
"Subtitle": "Subtitle",
"Summary": "Summary",
"Sweet": "Sweet",
diff --git a/engine/crates/concat/locales/es.json b/engine/crates/concat/locales/es.json
index 37a2f089..f951c183 100644
--- a/engine/crates/concat/locales/es.json
+++ b/engine/crates/concat/locales/es.json
@@ -181,6 +181,7 @@
"Favourites": "Favoritos",
"Feather": "Difuminado",
"female": "femenina",
+ "Fetching the model… {0}%": "Descargando el modelo… {0}%",
"File": "Archivo",
"Fill": "Relleno",
"Film Grain": "Grano de película",
@@ -312,12 +313,14 @@
"Normalize Loudness": "Normalizar sonoridad",
"Normalize loudness": "Normalizar sonoridad",
"not installed": "no instalado",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "No se encontró nada que conservar, así que la imagen se muestra tal cual. Prueba otro sujeto o la eliminación manual.",
"Nothing of that kind in this project.": "No hay nada de ese tipo en este proyecto.",
"Nothing starred yet — hover a tile and press its star.": "Aún no hay favoritos: pasa el cursor por una tarjeta y pulsa su estrella.",
"Nothing to export": "Nada que exportar",
"Nothing to read yet": "Aún no hay nada que leer",
"Nothing under the playhead": "Nada bajo el cabezal",
"Nothing was said in that clip": "No se dijo nada en ese clip",
+ "Object": "Objeto",
"off": "desactivado",
"One repeat, fading.": "Una repetición que se desvanece.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Un título por frase hablada, ajustado al ritmo de las palabras, en la primera pista con espacio.",
@@ -332,6 +335,7 @@
"Pace": "Ritmo",
"Paste": "Pegar",
"Pause": "Pausa",
+ "Person": "Persona",
"Pick an effect or filter from the library": "Elige un efecto o filtro de la biblioteca",
"Pitch": "Tono",
"Pixelate": "Pixelar",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "Piedra y altura: reflexiones que tardan en llegar.",
"Strength": "Intensidad",
"Stylize": "Estilizar",
+ "Subject": "Sujeto",
"Subtitle": "Subtítulo",
"Summary": "Resumen",
"Sweet": "Dulce",
diff --git a/engine/crates/concat/locales/fr.json b/engine/crates/concat/locales/fr.json
index c535c885..763f71bc 100644
--- a/engine/crates/concat/locales/fr.json
+++ b/engine/crates/concat/locales/fr.json
@@ -181,6 +181,7 @@
"Favourites": "Favoris",
"Feather": "Contour progressif",
"female": "féminine",
+ "Fetching the model… {0}%": "Téléchargement du modèle… {0}%",
"File": "Fichier",
"Fill": "Remplissage",
"Film Grain": "Grain de film",
@@ -312,12 +313,14 @@
"Normalize Loudness": "Normaliser la sonie",
"Normalize loudness": "Normaliser la sonie",
"not installed": "non installé",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Rien à garder ici, l'image est donc montrée telle quelle. Essayez un autre sujet ou la suppression manuelle.",
"Nothing of that kind in this project.": "Rien de ce genre dans ce projet.",
"Nothing starred yet — hover a tile and press its star.": "Rien en favori pour l'instant — survolez une vignette et appuyez sur son étoile.",
"Nothing to export": "Rien à exporter",
"Nothing to read yet": "Rien à lire pour l'instant",
"Nothing under the playhead": "Rien sous la tête de lecture",
"Nothing was said in that clip": "Rien n'a été dit dans ce clip",
+ "Object": "Objet",
"off": "désactivé",
"One repeat, fading.": "Une répétition qui s'estompe.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Un titre par phrase prononcée, calé sur le rythme des mots, sur la première piste libre.",
@@ -332,6 +335,7 @@
"Pace": "Rythme",
"Paste": "Coller",
"Pause": "Pause",
+ "Person": "Personne",
"Pick an effect or filter from the library": "Choisissez un effet ou un filtre dans la bibliothèque",
"Pitch": "Hauteur",
"Pixelate": "Pixeliser",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "Pierre et hauteur : des réflexions qui mettent du temps à arriver.",
"Strength": "Intensité",
"Stylize": "Styliser",
+ "Subject": "Sujet",
"Subtitle": "Sous-titre",
"Summary": "Résumé",
"Sweet": "Douce",
diff --git a/engine/crates/concat/locales/it.json b/engine/crates/concat/locales/it.json
index e19e692c..c187b2d4 100644
--- a/engine/crates/concat/locales/it.json
+++ b/engine/crates/concat/locales/it.json
@@ -181,6 +181,7 @@
"Favourites": "Preferiti",
"Feather": "Sfumatura",
"female": "femminile",
+ "Fetching the model… {0}%": "Scaricamento del modello… {0}%",
"File": "File",
"Fill": "Riempimento",
"Film Grain": "Grana pellicola",
@@ -312,12 +313,14 @@
"Normalize Loudness": "Normalizza loudness",
"Normalize loudness": "Normalizza loudness",
"not installed": "non installato",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Qui non c'è nulla da tenere, quindi l'immagine è mostrata com'è. Prova un altro soggetto o la rimozione manuale.",
"Nothing of that kind in this project.": "Niente di questo tipo in questo progetto.",
"Nothing starred yet — hover a tile and press its star.": "Nessun preferito ancora: passa su una scheda e premi la sua stella.",
"Nothing to export": "Niente da esportare",
"Nothing to read yet": "Ancora niente da leggere",
"Nothing under the playhead": "Niente sotto la testina",
"Nothing was said in that clip": "In quella clip non è stato detto nulla",
+ "Object": "Oggetto",
"off": "disattivato",
"One repeat, fading.": "Una ripetizione che sfuma.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Un titolo per ogni frase pronunciata, tagliato sul ritmo delle parole, sulla prima traccia libera.",
@@ -332,6 +335,7 @@
"Pace": "Ritmo",
"Paste": "Incolla",
"Pause": "Pausa",
+ "Person": "Persona",
"Pick an effect or filter from the library": "Scegli un effetto o un filtro dalla libreria",
"Pitch": "Intonazione",
"Pixelate": "Pixel",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "Pietra e altezza: riflessioni che tardano ad arrivare.",
"Strength": "Intensità",
"Stylize": "Stilizza",
+ "Subject": "Soggetto",
"Subtitle": "Sottotitolo",
"Summary": "Riepilogo",
"Sweet": "Dolce",
diff --git a/engine/crates/concat/locales/ja.json b/engine/crates/concat/locales/ja.json
index 63bf5eec..0373456c 100644
--- a/engine/crates/concat/locales/ja.json
+++ b/engine/crates/concat/locales/ja.json
@@ -181,6 +181,7 @@
"Favourites": "お気に入り",
"Feather": "ぼかし",
"female": "女性",
+ "Fetching the model… {0}%": "モデルを取得中… {0}%",
"File": "ファイル",
"Fill": "塗り",
"Film Grain": "フィルムグレイン",
@@ -312,12 +313,14 @@
"Normalize Loudness": "ラウドネスを正規化",
"Normalize loudness": "ラウドネスを正規化",
"not installed": "未インストール",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "残すものが見つからなかったため、画像はそのまま表示されています。別の対象かカスタム削除を試してください。",
"Nothing of that kind in this project.": "この種類のものはこのプロジェクトにありません。",
"Nothing starred yet — hover a tile and press its star.": "まだお気に入りはありません。タイルにカーソルを合わせて星を押してください。",
"Nothing to export": "書き出すものがありません",
"Nothing to read yet": "読み上げる内容がまだありません",
"Nothing under the playhead": "再生ヘッドの下に何もありません",
"Nothing was said in that clip": "そのクリップには発話がありませんでした",
+ "Object": "物体",
"off": "オフ",
"One repeat, fading.": "一回の繰り返しがフェードします。",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "発話されたフレーズごとにタイトルを一つ、言葉のタイミングどおりに、空いている最初のレーンへ。",
@@ -332,6 +335,7 @@
"Pace": "速さ",
"Paste": "ペースト",
"Pause": "一時停止",
+ "Person": "人物",
"Pick an effect or filter from the library": "ライブラリからエフェクトまたはフィルターを選んでください",
"Pitch": "ピッチ",
"Pixelate": "モザイク",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "石と高さ:届くまでに時間のかかる反射。",
"Strength": "強さ",
"Stylize": "スタイライズ",
+ "Subject": "対象",
"Subtitle": "サブタイトル",
"Summary": "概要",
"Sweet": "スイート",
diff --git a/engine/crates/concat/locales/ko.json b/engine/crates/concat/locales/ko.json
index e9ebcf52..91232d2e 100644
--- a/engine/crates/concat/locales/ko.json
+++ b/engine/crates/concat/locales/ko.json
@@ -181,6 +181,7 @@
"Favourites": "즐겨찾기",
"Feather": "페더",
"female": "여성",
+ "Fetching the model… {0}%": "모델 가져오는 중… {0}%",
"File": "파일",
"Fill": "채우기",
"Film Grain": "필름 그레인",
@@ -312,12 +313,14 @@
"Normalize Loudness": "라우드니스 정규화",
"Normalize loudness": "라우드니스 정규화",
"not installed": "설치되지 않음",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "남길 것을 찾지 못해 그림을 촬영한 그대로 보여줍니다. 다른 대상이나 사용자 지정 제거를 시도해 보세요.",
"Nothing of that kind in this project.": "이 프로젝트에 해당 종류의 항목이 없습니다.",
"Nothing starred yet — hover a tile and press its star.": "아직 즐겨찾기가 없습니다. 타일에 마우스를 올리고 별을 누르세요.",
"Nothing to export": "내보낼 것이 없습니다",
"Nothing to read yet": "아직 읽을 내용이 없습니다",
"Nothing under the playhead": "재생 헤드 아래에 아무것도 없습니다",
"Nothing was said in that clip": "그 클립에는 말소리가 없습니다",
+ "Object": "사물",
"off": "끔",
"One repeat, fading.": "한 번 반복하며 사라집니다.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "말한 구절마다 자막 하나씩, 말의 타이밍에 맞춰, 자리가 있는 첫 레인에 놓습니다.",
@@ -332,6 +335,7 @@
"Pace": "속도",
"Paste": "붙여넣기",
"Pause": "일시 정지",
+ "Person": "인물",
"Pick an effect or filter from the library": "라이브러리에서 효과나 필터를 선택하세요",
"Pitch": "피치",
"Pixelate": "픽셀화",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "돌과 높이: 도착하기까지 시간이 걸리는 반사.",
"Strength": "강도",
"Stylize": "스타일화",
+ "Subject": "대상",
"Subtitle": "부제",
"Summary": "요약",
"Sweet": "달콤하게",
diff --git a/engine/crates/concat/locales/pt-BR.json b/engine/crates/concat/locales/pt-BR.json
index 4b221606..7218e617 100644
--- a/engine/crates/concat/locales/pt-BR.json
+++ b/engine/crates/concat/locales/pt-BR.json
@@ -181,6 +181,7 @@
"Favourites": "Favoritos",
"Feather": "Suavização",
"female": "feminina",
+ "Fetching the model… {0}%": "Baixando o modelo… {0}%",
"File": "Arquivo",
"Fill": "Preenchimento",
"Film Grain": "Granulação de filme",
@@ -312,12 +313,14 @@
"Normalize Loudness": "Normalizar loudness",
"Normalize loudness": "Normalizar loudness",
"not installed": "não instalado",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Nada foi encontrado para manter, então a imagem é mostrada como gravada. Tente outro assunto ou a remoção manual.",
"Nothing of that kind in this project.": "Nada desse tipo neste projeto.",
"Nothing starred yet — hover a tile and press its star.": "Nada favoritado ainda — passe o mouse sobre um cartão e toque na estrela.",
"Nothing to export": "Nada para exportar",
"Nothing to read yet": "Nada para ler ainda",
"Nothing under the playhead": "Nada sob o cursor",
"Nothing was said in that clip": "Nada foi dito nesse clipe",
+ "Object": "Objeto",
"off": "desligado",
"One repeat, fading.": "Uma repetição que se esvai.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Um título por frase falada, no ritmo das palavras, na primeira faixa com espaço.",
@@ -332,6 +335,7 @@
"Pace": "Ritmo",
"Paste": "Colar",
"Pause": "Pausar",
+ "Person": "Pessoa",
"Pick an effect or filter from the library": "Escolha um efeito ou filtro na biblioteca",
"Pitch": "Tom",
"Pixelate": "Pixelizar",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "Pedra e altura: reflexões que demoram a chegar.",
"Strength": "Intensidade",
"Stylize": "Estilizar",
+ "Subject": "Assunto",
"Subtitle": "Subtítulo",
"Summary": "Resumo",
"Sweet": "Doce",
diff --git a/engine/crates/concat/locales/ru.json b/engine/crates/concat/locales/ru.json
index abd35055..5d7eb4f6 100644
--- a/engine/crates/concat/locales/ru.json
+++ b/engine/crates/concat/locales/ru.json
@@ -181,6 +181,7 @@
"Favourites": "Избранное",
"Feather": "Растушёвка",
"female": "женский",
+ "Fetching the model… {0}%": "Загрузка модели… {0}%",
"File": "Файл",
"Fill": "Заливка",
"Film Grain": "Зерно плёнки",
@@ -312,12 +313,14 @@
"Normalize Loudness": "Нормализовать громкость",
"Normalize loudness": "Нормализовать громкость",
"not installed": "не установлено",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Здесь нечего оставить, поэтому кадр показан как снят. Попробуйте другой объект съёмки или ручное удаление.",
"Nothing of that kind in this project.": "В этом проекте ничего такого нет.",
"Nothing starred yet — hover a tile and press its star.": "Пока ничего нет — наведите на плитку и нажмите звезду.",
"Nothing to export": "Нечего экспортировать",
"Nothing to read yet": "Пока нечего читать",
"Nothing under the playhead": "Под курсором ничего нет",
"Nothing was said in that clip": "В этом клипе ничего не сказано",
+ "Object": "Предмет",
"off": "выкл.",
"One repeat, fading.": "Один повтор, затухающий.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Один титр на каждую произнесённую фразу, в такт словам, на первой свободной дорожке.",
@@ -332,6 +335,7 @@
"Pace": "Темп",
"Paste": "Вставить",
"Pause": "Пауза",
+ "Person": "Человек",
"Pick an effect or filter from the library": "Выберите эффект или фильтр в библиотеке",
"Pitch": "Высота",
"Pixelate": "Пикселизация",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "Камень и высота: отражения, которые долго идут.",
"Strength": "Сила",
"Stylize": "Стилизация",
+ "Subject": "Объект съёмки",
"Subtitle": "Подзаголовок",
"Summary": "Сводка",
"Sweet": "Сладкий",
diff --git a/engine/crates/concat/locales/tr.json b/engine/crates/concat/locales/tr.json
index b188f773..4596d007 100644
--- a/engine/crates/concat/locales/tr.json
+++ b/engine/crates/concat/locales/tr.json
@@ -181,6 +181,7 @@
"Favourites": "Favoriler",
"Feather": "Yumuşatma",
"female": "kadın",
+ "Fetching the model… {0}%": "Model indiriliyor… {0}%",
"File": "Dosya",
"Fill": "Dolgu",
"Film Grain": "Film greni",
@@ -312,12 +313,14 @@
"Normalize Loudness": "Ses yüksekliğini normalleştir",
"Normalize loudness": "Gürlüğü normalleştir",
"not installed": "kurulu değil",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "Burada tutulacak bir şey bulunamadı, bu yüzden görüntü çekildiği gibi gösteriliyor. Başka bir konu ya da özel kaldırmayı deneyin.",
"Nothing of that kind in this project.": "Bu projede o türden bir şey yok.",
"Nothing starred yet — hover a tile and press its star.": "Henüz favori yok — bir kartın üzerine gelip yıldızına basın.",
"Nothing to export": "Dışa aktarılacak bir şey yok",
"Nothing to read yet": "Henüz okunacak bir şey yok",
"Nothing under the playhead": "Oynatma imlecinin altında bir şey yok",
"Nothing was said in that clip": "O klipte hiçbir şey söylenmedi",
+ "Object": "Nesne",
"off": "kapalı",
"One repeat, fading.": "Sönen tek bir tekrar.",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "Söylenen her cümle için bir yazı, kelimelerin kendi zamanlamasıyla, yeri olan ilk şeritte.",
@@ -332,6 +335,7 @@
"Pace": "Tempo",
"Paste": "Yapıştır",
"Pause": "Duraklat",
+ "Person": "Kişi",
"Pick an effect or filter from the library": "Kitaplıktan bir efekt veya filtre seçin",
"Pitch": "Perde",
"Pixelate": "Pikselleştir",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "Taş ve yükseklik: gelmesi zaman alan yansımalar.",
"Strength": "Güç",
"Stylize": "Stilize",
+ "Subject": "Konu",
"Subtitle": "Alt başlık",
"Summary": "Özet",
"Sweet": "Tatlı",
diff --git a/engine/crates/concat/locales/zh-Hans.json b/engine/crates/concat/locales/zh-Hans.json
index 3978676d..50073d1f 100644
--- a/engine/crates/concat/locales/zh-Hans.json
+++ b/engine/crates/concat/locales/zh-Hans.json
@@ -181,6 +181,7 @@
"Favourites": "收藏",
"Feather": "羽化",
"female": "女声",
+ "Fetching the model… {0}%": "正在获取模型… {0}%",
"File": "文件",
"Fill": "填充",
"Film Grain": "胶片颗粒",
@@ -312,12 +313,14 @@
"Normalize Loudness": "响度标准化",
"Normalize loudness": "响度标准化",
"not installed": "未安装",
+ "Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.": "这里没有找到可保留的内容,因此画面按原样显示。请尝试其他主体或自定义移除。",
"Nothing of that kind in this project.": "此项目中没有该类型的内容。",
"Nothing starred yet — hover a tile and press its star.": "还没有收藏——将鼠标悬停在卡片上并点击星标。",
"Nothing to export": "没有可导出的内容",
"Nothing to read yet": "还没有可朗读的内容",
"Nothing under the playhead": "播放头下方没有内容",
"Nothing was said in that clip": "该片段中没有说话内容",
+ "Object": "物体",
"off": "关",
"One repeat, fading.": "一次重复,逐渐消失。",
"One title per phrase spoken, cut to the words' own timing, on the first lane with room.": "每说一句生成一条字幕,按语句自身的时间切分,放在第一条有空位的轨道上。",
@@ -332,6 +335,7 @@
"Pace": "语速",
"Paste": "粘贴",
"Pause": "暂停",
+ "Person": "人物",
"Pick an effect or filter from the library": "请从素材库中选择效果或滤镜",
"Pitch": "音高",
"Pixelate": "像素化",
@@ -447,6 +451,7 @@
"Stone and height: reflections that take a while to arrive.": "石壁与高度:需要一段时间才到达的反射。",
"Strength": "强度",
"Stylize": "风格化",
+ "Subject": "主体",
"Subtitle": "副标题",
"Summary": "摘要",
"Sweet": "甜美",
diff --git a/engine/crates/concat/src/host.rs b/engine/crates/concat/src/host.rs
index c463355d..a8bc507a 100644
--- a/engine/crates/concat/src/host.rs
+++ b/engine/crates/concat/src/host.rs
@@ -55,6 +55,7 @@ impl Host {
let _ = std::fs::create_dir_all(&dirs.config);
Ok(Host {
titles: concat_host::Titles::new(&dirs),
+ cutouts: Arc::new(concat_host::Cutouts::new(&dirs.data)),
dirs,
playback: Playback::start(Arc::new(Events))?,
monitor: match gpu {
@@ -64,7 +65,6 @@ impl Host {
exporter: Exporter::new(),
transcriber: Arc::new(Transcriber::new()),
speech: Arc::new(Speech::new()),
- cutouts: Arc::new(concat_host::Cutouts::new()),
})
}
}
diff --git a/engine/crates/concat/src/lib.rs b/engine/crates/concat/src/lib.rs
index f6081414..ea785fa2 100644
--- a/engine/crates/concat/src/lib.rs
+++ b/engine/crates/concat/src/lib.rs
@@ -774,6 +774,9 @@ pub fn run() -> Result<(), slint::PlatformError> {
editor.on_cutout_painting(on_window!(|state, on: bool| {
state.cutout_painting(on);
}));
+ editor.on_cutout_subject(on_window!(|state, index: i32| {
+ state.cutout_subject(index);
+ }));
editor.on_cutout_clear(on_window!(|state| {
state.cutout_clear();
}));
diff --git a/engine/crates/concat/src/studio.rs b/engine/crates/concat/src/studio.rs
index 3606f72a..17ac5b7c 100644
--- a/engine/crates/concat/src/studio.rs
+++ b/engine/crates/concat/src/studio.rs
@@ -719,7 +719,7 @@ pub struct Studio {
/// rather than moving the picture.
pub painting: bool,
/// The mask analyses running, by media id, and how far each has got.
- cutout_jobs: HashMap<String, f32>,
+ cutout_jobs: HashMap<String, (bool, f32)>,
}
// ── conversions between the document and the window ─────────────────────
@@ -3471,12 +3471,18 @@ impl Studio {
let project = std::path::PathBuf::from(session.path());
let mut wanted: Vec<(String, AnalyseRequest)> = Vec::new();
for clip in &self.timeline().clips {
- if clip.cutout.is_none() || !clip.kind.is_visual() {
+ let Some(cutout) = clip.cutout.as_ref() else {
+ continue;
+ };
+ if !clip.kind.is_visual() {
continue;
}
let Some(media) = self.project().media_by_id(&clip.media_id) else {
continue;
};
+ // One analysis per media and subject: two clips of one file
+ // that keep different things need different masks.
+ let key = Self::analysis_key(&media.id, cutout.subject);
// The source the clip shows: its in-point, for as long as it
// runs at its speed. A curve's mean is its speed, so this
// covers a curved clip too.
@@ -3484,14 +3490,16 @@ impl Studio {
clip.source_start,
clip.source_start + clip.duration * clip.speed.max(0.0625),
);
- match wanted.iter_mut().find(|(id, _)| *id == media.id) {
+ match wanted.iter_mut().find(|(id, _)| *id == key) {
Some((_, request)) => request.ranges.push(range),
None => wanted.push((
- media.id.clone(),
+ key,
AnalyseRequest {
project: project.clone(),
media_path: media.path.clone(),
+ media_size: (media.width.unwrap_or(0), media.height.unwrap_or(0)),
still: media.kind == model::MediaKind::Image,
+ subject: cutout.subject,
ranges: vec![range],
},
)),
@@ -3503,21 +3511,27 @@ impl Studio {
else {
return;
};
- self.cutout_jobs.insert(id.clone(), 0.0);
+ self.cutout_jobs.insert(id.clone(), (false, 0.0));
let cutouts = Arc::clone(&self.host.cutouts);
spawn(
move || {
- let mut last = -1.0f32;
+ let mut last = (false, -1.0f32);
let reporting = id.clone();
let result = cutouts.analyse(&request, &mut |progress| {
// Every percent, not every frame: the readout cannot
// use more and the event loop has other work.
- if progress - last >= 0.01 {
- last = progress;
+ let now = match progress {
+ concat_host::cutout::Progress::Fetching { received, total } => {
+ (true, received as f32 / total.max(1) as f32)
+ }
+ concat_host::cutout::Progress::Analysing(fraction) => (false, fraction),
+ };
+ if now.0 != last.0 || now.1 - last.1 >= 0.01 {
+ last = now;
let id = reporting.clone();
on_ui(move |studio, _, _| {
if let Some(held) = studio.cutout_jobs.get_mut(&id) {
- *held = progress;
+ *held = now;
}
});
}
@@ -3572,6 +3586,34 @@ impl Studio {
}
}
+ /// The name an analysis runs under: the media and what it keeps.
+ fn analysis_key(media_id: &str, subject: model::Subject) -> String {
+ format!("{media_id}:{}", subject.key())
+ }
+
+ /// The Subject row: 0 automatic, 1 person, 2 object. A change means
+ /// other masks, which the analysis notices on its own.
+ pub fn cutout_subject(&mut self, index: i32) {
+ let Some(id) = self.sole_selection() else {
+ return;
+ };
+ let Some(cutout) = self.clip(&id).and_then(|clip| clip.cutout.clone()) else {
+ return;
+ };
+ let subject = match index {
+ 1 => model::Subject::Person,
+ 2 => model::Subject::Object,
+ _ => model::Subject::Auto,
+ };
+ if cutout.subject == subject {
+ return;
+ }
+ self.apply(Command::SetClipCutout {
+ clip_id: id,
+ cutout: Some(model::Cutout { subject, ..cutout }),
+ });
+ }
+
/// Takes every stroke off the selected clip's cutout, keeping it custom.
pub fn cutout_clear(&mut self) {
let Some(id) = self.sole_selection() else {
@@ -5034,6 +5076,11 @@ impl Studio {
let fill = colour_of(&text.color);
let stroke = colour_of(&text.stroke_color);
let plate = colour_of(&text.background);
+ let analysis = clip.cutout.as_ref().and_then(|cutout| {
+ self.cutout_jobs
+ .get(&Self::analysis_key(&clip.media_id, cutout.subject))
+ .copied()
+ });
SelectedClipData {
present: true,
id: clip.id.as_str().into(),
@@ -5121,14 +5168,48 @@ impl Studio {
.as_ref()
.map(|cutout| cutout.strokes.len() as i32)
.unwrap_or(0),
- cutout_progress: self
- .cutout_jobs
- .get(&clip.media_id)
- .copied()
- .unwrap_or(-1.0),
+ cutout_subject: clip
+ .cutout
+ .as_ref()
+ .map(|cutout| match cutout.subject {
+ model::Subject::Auto => 0,
+ model::Subject::Person => 1,
+ model::Subject::Object => 2,
+ })
+ .unwrap_or(0),
+ cutout_progress: analysis.map(|(_, fraction)| fraction).unwrap_or(-1.0),
+ cutout_fetching: analysis.is_some_and(|(fetching, _)| fetching),
+ cutout_empty: self.cutout_empty(clip),
}
}
+ /// Whether the cutout model found nothing at the playhead's frame of
+ /// `clip`, so the picture is showing as shot. Only an automatic cutout
+ /// says so: a custom one is whatever was painted.
+ fn cutout_empty(&self, clip: &Clip) -> bool {
+ let Some(cutout) = clip.cutout.as_ref() else {
+ return false;
+ };
+ if cutout.mode != model::CutoutMode::Auto {
+ return false;
+ }
+ let (Some(session), Some(media)) = (
+ self.session.as_ref(),
+ self.project().media_by_id(&clip.media_id),
+ ) else {
+ return false;
+ };
+ let project = std::path::Path::new(session.path());
+ let store = concat_vision::MaskStore::open(&concat_vision::mask_dir(
+ project,
+ &media.path,
+ cutout.subject,
+ ));
+ let along = (f64::from(self.playhead) - clip.start).clamp(0.0, clip.duration);
+ let source = clip.source_start + along * clip.speed;
+ store.mask_at(source).is_some_and(|mask| mask.is_blank())
+ }
+
/// The menus, the dialogs, the bin and the engine lists.
pub fn publish_chrome(&self, app: &App, models: &Models) {
let editor = app.global::<Editor>();
diff --git a/engine/crates/concat/ui/editor.slint b/engine/crates/concat/ui/editor.slint
index afe9fff8..a130d1cc 100644
--- a/engine/crates/concat/ui/editor.slint
+++ b/engine/crates/concat/ui/editor.slint
@@ -201,6 +201,7 @@ export global Editor {
callback cutout-tool(index: int);
callback cutout-size(size: float);
callback cutout-painting(on: bool);
+ callback cutout-subject(index: int);
callback cutout-clear();
// ── the timeline ──
diff --git a/engine/crates/concat/ui/inspector/clip-inspector.slint b/engine/crates/concat/ui/inspector/clip-inspector.slint
index 3c5b7e08..f8f3b9dc 100644
--- a/engine/crates/concat/ui/inspector/clip-inspector.slint
+++ b/engine/crates/concat/ui/inspector/clip-inspector.slint
@@ -649,10 +649,14 @@ component ToolButton inherits Rectangle {
// background and the picture is shown as shot until they are.
component AnalysisRow inherits Row {
in property <float> progress;
+ /// The model is still being downloaded; the frames come after.
+ in property <bool> fetching;
label: I18n.t("Analysis");
Text {
- text: I18n.t1("Finding the subject… {0}%", "\{Math.round(root.progress * 100)}");
+ text: root.fetching
+ ? I18n.t1("Fetching the model… {0}%", "\{Math.round(root.progress * 100)}")
+ : I18n.t1("Finding the subject… {0}%", "\{Math.round(root.progress * 100)}");
color: Theme.fg-muted;
font-size: Theme.fs;
vertical-alignment: center;
@@ -695,6 +699,8 @@ export component VideoPanel inherits VerticalLayout {
callback cutout-tool(index: int);
callback cutout-size(size: float);
callback cutout-painting(on: bool);
+ /// What the automatic cutout keeps: 0 auto, 1 person, 2 object.
+ callback cutout-subject(index: int);
callback cutout-clear();
/// Which keyer is on the chain, and where: -1 for none. Scans the first
@@ -939,6 +945,14 @@ export component VideoPanel inherits VerticalLayout {
on: root.removal == 1;
toggled(on) => { root.remove-by(on ? 1 : 0); }
}
+ if root.removal == 1: Row {
+ label: I18n.t("Subject");
+ Select {
+ options: [I18n.t("Auto"), I18n.t("Person"), I18n.t("Object")];
+ current: root.selected.cutout-subject;
+ changed(index) => { root.cutout-subject(index); }
+ }
+ }
if root.removal == 1: Param {
label: I18n.t("Feather");
value <=> root.feather;
@@ -951,6 +965,10 @@ export component VideoPanel inherits VerticalLayout {
}
if root.removal == 1 && root.selected.cutout-progress >= 0: AnalysisRow {
progress: root.selected.cutout-progress;
+ fetching: root.selected.cutout-fetching;
+ }
+ if root.removal == 1 && root.selected.cutout-progress < 0 && root.selected.cutout-empty: Note {
+ text: I18n.t("Nothing found to keep here, so the picture shows as shot. Try another subject, or custom removal.");
}
Hairline { }
@@ -1022,6 +1040,7 @@ export component VideoPanel inherits VerticalLayout {
}
if root.removal == 2 && root.selected.cutout-progress >= 0: AnalysisRow {
progress: root.selected.cutout-progress;
+ fetching: root.selected.cutout-fetching;
}
if root.removal == 2: Row {
label: root.selected.cutout-strokes == 1 ? I18n.t("1 stroke")
diff --git a/engine/crates/concat/ui/timeline/model.slint b/engine/crates/concat/ui/timeline/model.slint
index 64dee708..e79d24ad 100644
--- a/engine/crates/concat/ui/timeline/model.slint
+++ b/engine/crates/concat/ui/timeline/model.slint
@@ -281,13 +281,19 @@ export struct SelectedClipData {
fade-in: float,
fade-out: float,
- // the cutout: 0 none, 1 automatic, 2 custom; how soft its edge is; how
- // many strokes it carries; and the analysis finding its masks, -1 when
- // none is running, else how far along it is
+ // the cutout: 0 none, 1 automatic, 2 custom; what it keeps, 0 auto,
+ // 1 person, 2 object; how soft its edge is; how many strokes it
+ // carries; the analysis finding its masks, -1 when none is running,
+ // else how far along it is, and whether that is a model downloading
+ // rather than frames being read; and whether the model found nothing
+ // at the playhead, so the picture is showing as shot
cutout: int,
+ cutout-subject: int,
cutout-feather: float,
cutout-strokes: int,
cutout-progress: float,
+ cutout-fetching: bool,
+ cutout-empty: bool,
// the overlay, when this is a text clip
content: string,
diff --git a/engine/crates/concat/ui/workspace/config-pane.slint b/engine/crates/concat/ui/workspace/config-pane.slint
index 79be3b45..06fb300c 100644
--- a/engine/crates/concat/ui/workspace/config-pane.slint
+++ b/engine/crates/concat/ui/workspace/config-pane.slint
@@ -172,6 +172,7 @@ export component ConfigPane inherits Rectangle {
callback cutout-tool(index: int);
callback cutout-size(size: float);
callback cutout-painting(on: bool);
+ callback cutout-subject(index: int);
callback cutout-clear();
@@ -524,6 +525,7 @@ export component ConfigPane inherits Rectangle {
cutout-tool(index) => { root.cutout-tool(index); }
cutout-size(size) => { root.cutout-size(size); }
cutout-painting(on) => { root.cutout-painting(on); }
+ cutout-subject(index) => { root.cutout-subject(index); }
cutout-clear => { root.cutout-clear(); }
}
diff --git a/engine/crates/concat/ui/workspace/seat.slint b/engine/crates/concat/ui/workspace/seat.slint
index f359649f..dbcff09d 100644
--- a/engine/crates/concat/ui/workspace/seat.slint
+++ b/engine/crates/concat/ui/workspace/seat.slint
@@ -217,6 +217,7 @@ export component Seat inherits Rectangle {
cutout-tool(index) => { Editor.cutout-tool(index); }
cutout-size(size) => { Editor.cutout-size(size); }
cutout-painting(on) => { Editor.cutout-painting(on); }
+ cutout-subject(index) => { Editor.cutout-subject(index); }
cutout-clear => { Editor.cutout-clear(); }
project-name: Editor.project-name;