The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.
Rust ffmpegfree-video-editormodel-context-protocolnon-linear-editoroffline-firstopen-source-video-editorself-hostedvideo-editingvideo-editing-automationvideo-editing-softwarevideo-editing-toolsvideo-editorvoice-cloningyoutube-automationyoutube-shorts
Remove background: automatic and custom cutouts on a person mask
Jareereb3078ec6d70
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.mdindex 968852fa..1f0f11c3 100644--- a/ARCHITECTURE.md+++ b/ARCHITECTURE.md@@ -40,7 +40,7 @@ Two consequences of the rule shape everything below:## 2. The map-Everything lives in `engine/`, one Cargo workspace of twelve crates.+Everything lives in `engine/`, one Cargo workspace of thirteen crates.| Crate | Owns | Lines ||---|---|---|@@ -51,6 +51,7 @@ Everything lives in `engine/`, one Cargo workspace of twelve crates.| `concat-render` | A timeline plus an instant into one frame: the plan, the CPU compositor, the wgpu compositor. | 1.9k || `concat-export` | Timeline to file. Transitions resolve here; the frame loop and the one-pass mix; the paused monitor's true frame. | 2.2k || `concat-text` | Titles as pixels: font lookup, shaping, rasterising onto a frame-sized transparent canvas. | 0.6k |+| `concat-vision` | What the engine sees in a picture: the person mask behind a cutout, brush strokes over it, the frame cut by it; the model itself behind `infer`. | 0.9k || `concat-host` | What a window needs that is not the edit: sessions, project folders, caches, the monitor, audio playback, the export driver, templates, job slots, app directories. | 3.8k || `concat-speech` | Transcription with whisper.cpp and text to speech with Kokoro, both in-process, models downloaded on demand. | 1.2k || `concat-cli` | A driver for the engine without a window: `probe` and `render`. | 0.2k |@@ -62,10 +63,13 @@ The dependency arrows point one way:```concat ──► concat-speech ──► concat-host ──► concat-export ──► concat-render ──► concat-core│ │ │ │ ▲- │ │ │ └──► concat-effects ──► concat-project ──► concat-core+ │ │ │ ├──► concat-effects ──► concat-project ──► concat-core+ │ │ │ └──► concat-vision ──► concat-project, concat-core│ ├──► concat-media ◄──┘- │ └──► concat-text- └──► concat-effects (the catalogue, for the inspector's shelves)+ │ ├──► concat-text+ │ └──► concat-vision (with `infer`: the model runs here)+ ├──► concat-effects (the catalogue, for the inspector's shelves)+ └──► concat-vision (a press on the stage, mapped into the source)concat-android ──► concat concat-cli ──► concat-core, concat-media, concat-render```@@ -74,7 +78,9 @@ concat-android ──► concat concat-cli ──► concat-core, conbecause the document needs serde and core's zero-dependency rule is worthmore than the adjacency. Five crates carry no native library and build forthe web as well (`concat-core`, `concat-project`, `concat-effects`,-`concat-render`, `concat-text`); CI keeps that true.+`concat-render`, `concat-text`); CI keeps that true. `concat-vision` is+pure Rust too, inference included: tract runs the model, so a cutout needs+no runtime library on any platform.---@@ -377,6 +383,15 @@ keeps its readers rolling forward instead of seeking.---+**Cutouts happen on the decoded frame.** A clip with a cutout names the+directory its media's masks live in (the flattener fills it in when it+knows the project folder), and the frame loop asks `concat-vision` for the+mask nearest the layer's source instant, with the clip's strokes painted+on and its feather applied, and multiplies the decoded frame's alpha by+it through the clip's crop and flips. That happens before the frame+becomes a layer, so both compositors draw the same cut pixels, and an+instant with no mask yet draws the picture as shot.+## 9. Titles: `concat-text` and `host::titles`A text clip is a style and some words; the compositor wants a picture.@@ -402,7 +417,41 @@ look repaints.----## 10. The host layer: `concat-host`+## 10. Cutouts: `concat-vision` and `host::cutout`++A cutout takes a picture's background away without a key colour. The+model is MediaPipe's selfie segmentation, compiled into `concat-vision`+and run by tract: a 256 × 256 picture in, a probability per pixel out,+about twenty-five milliseconds a frame on one core. Masks are square+whatever the picture's shape, and every position in one is a fraction of+the source picture, which is also how strokes are stored; a crop, a flip+or a change of output size changes nothing about either, because+`apply::Mapping` is the one place a decoded pixel is walked back to a+source fraction.++**The store is the project's.** Each media file's masks are PNGs in+`cache/masks/<hash>-<model>/`, one per analysed source instant, named by+that instant in milliseconds, ten a second of footage and one for a+still. The file names are the index. The renderer reads them; the host's+`cutout` module writes them, decoding the media at the model's size along+each missing stretch and running the model on a batch of frames at once,+one media at a time through a `SingleFlight`. The window notices which+media want masks they lack after every change and starts the next+analysis when the last reports back; the picture is shown as shot until+its masks arrive, and the inspector says how far along they are.++**Custom is automatic plus strokes.** A brush stroke is a tool, a+diameter and a path, painted as discs onto the model's mask: the brush+and eraser do exactly what was painted; the smart brush keeps only what+the model gave some chance of being the subject, the smart eraser removes+only what it was unsure of. Each stroke is one command and one undo step.+The feather is three box blurs over the resolved mask, and the resolved+mask is cached per instant and settings so a paused monitor never+recomputes it.++---++## 11. The host layer: `concat-host`What the window needs that is not the edit. Nothing here knows about awindow; long work reports through callbacks and cancels through flags, and@@ -457,6 +506,11 @@ Every lock in playback and titles recovers from poisoning rather thanpropagating it: each guards a cache or a list that is still valid after apanic elsewhere, and refusing it would end audio or titles for the session.+**Cutouts.** `Cutouts` holds the model, loaded on first use, and the+one-analysis slot; see section 10. `outstanding` says how many instants a+request still needs without doing anything, which is how the window+decides whether to start it.+**Export, templates, jobs, directories.** `Exporter` adapts a progressclosure into the engine's `Reporter` and runs under a `SingleFlight`,which refuses a second concurrent job and gives each run its own cancel@@ -470,7 +524,7 @@ platform's config and data directories under `app.concat.editor`.----## 11. Speech: `concat-speech`+## 12. Speech: `concat-speech`Transcription runs whisper.cpp in-process on a clip's audio decoded to16 kHz mono, greedy sampling, threads capped at eight, with the job's@@ -487,7 +541,7 @@ native libraries stay out of everything that does not speak.----## 12. The window: `concat`+## 13. The window: `concat`**One library, three entry points.** `concat::run()` is the program.`main.rs` calls it on the desktop and on iOS; `concat-android`'s@@ -579,7 +633,7 @@ previews are embedded by `build.rs`.----## 13. Build and delivery+## 14. Build and delivery- **Toolchain.** Pinned in `engine/rust-toolchain.toml`. `cargo build`needs the FFmpeg 7+ development libraries, cmake and a C++ toolchain@@ -617,7 +671,7 @@ previews are embedded by `build.rs`.----## 14. Testing+## 15. Testing240 unit tests, all in-file `#[cfg(test)]` modules; there are no `tests/`directories. The pure layers are where the coverage is dense: rational@@ -638,20 +692,20 @@ the app, which is the work in 15.1.----## 15. Next work+## 16. Next workOrdered by how much each matters. Each is either an invariant acontributor could break without knowing it, or a piece of work with aknown shape.-### 15.1 Put a real project through every flow in the window+### 16.1 Put a real project through every flow in the windowImport, place, trim, split, transitions, a title, an effect, a speedcurve, transcription, narration, a template, an export, on each rendererand each platform. The engine's seams are tested; the window's are used.What that pass turns up goes above everything below.-### 15.2 A phone layout+### 16.2 A phone layoutThe window builds and packages for Android and iOS, and what runs there isthe desktop's tree on a phone's screen. A phone wants one seat at a time,@@ -663,7 +717,7 @@ asked for by name (they are linked, and the software decoder is what isopened); and the monitor on Android, which composites on the CPU becauseSlint's Android backend creates its own device.-### 15.3 Two paths that build different chains+### 16.3 Two paths that build different chainsOn the GPU, effects with a shader leave the FFmpeg chain and travel aspasses; on the CPU they stay in the chain. A machine with an adapter and@@ -674,7 +728,7 @@ render on one machine and not the other. Either require the chain for`effect` kinds in validation, or teach the CPU path to refuse the packageloudly.-### 15.4 Small things the code already knows about+### 16.4 Small things the code already knows about- `MoveTimeline`'s documented index is counted with the timeline removed;the clamp is against the unremoved length.@@ -695,10 +749,15 @@ loudly.imported for one type.- No clip culling beyond the visible flag; a windowing scheme is worthwriting the day a timeline needs one.+- The cutout model is one, sized for footage. A finer, salient-object+ model for stills would slot in behind the same `Segmenter`, keyed by+ its own `MODEL_ID` so the two never share a cache.+- A custom cutout's strokes are painted at the mask's resolution, so a+ very fine brush on a large picture is as fine as 256 pixels allow.----## 16. How to change things+## 17. How to change things- **A new operation on the edit.** Add a `Command` variant in`concat-project/src/commands.rs`, make its `apply` report `applied`diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex dfe47c54..79be655d 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -87,6 +87,16 @@ https://huggingface.co/hexgrad/Kokoro-82M) are downloaded on demand from thesherpa-onnx releases - including espeak-ng's data files - and are neverbundled with the app.+## The cutout model++Remove background's automatic and custom modes run Google's MediaPipe+Selfie Segmentation model (Apache-2.0), in the ONNX conversion published by+the ONNX Community (https://huggingface.co/onnx-community/mediapipe_selfie_segmentation,+Apache-2.0). The model file is compiled into the `concat-vision` crate; see+`engine/crates/concat-vision/models/NOTICE.md`. It is run by tract+(https://github.com/sonos/tract, MIT OR Apache-2.0), in pure Rust, so no+inference runtime is linked or shipped for it.+## Effect preview photographThe effect catalogue thumbnails are rendered from a photograph bydiff --git a/TODO.MD b/TODO.MDindex 54b42ca6..66a286b3 100644--- a/TODO.MD+++ b/TODO.MD@@ -29,4 +29,5 @@ DO NOT DELETE ANY OF THE TODOs, JUST MARK THEM DONE.- [x] fix: a lot of inputs have a bug where when they are focused and you click somewhere else to de-focus, it does not work. fix that. _(Any press on a lane, the stage, a clip, the ruler or the bin's floor takes focus back.)_- [x] feature: right now visual-media (images, videos) can only be scaled (resized on both dimensions propotionally). We need to add a feature to allow single-axis scale/stretch as well using the 2d bounding box in preview. _(Edge grips on the stage pull one axis; Width and Height under Video › Transform do the same by number. Saved as stretchX / stretchY; exports and both compositors follow it.)_- [x] feature: add a "resize/scale" snapping too along with translate snapping. _(Scaling from a corner pulls the picture's edges to the frame's lines and other pictures' edges, with the same guides a move draws.)_-- [ ] we need to add a "Remove BG" toggle feature in Inspector for Images and Videos, enabling this removes the media's background like how CapCut does (CapCut has different options like Auto Removal, Custom Removal, Chroma key, etc.). Custom removal gives the user Smart Brush, Brush, Smart Eraser, Eraser to outline the target. _(Video › Basic › Remove background: off, green screen or blue screen, with similarity and softness. Automatic and brush removal are next, on a person mask.)_\ No newline at end of file+- [x] we need to add a "Remove BG" toggle feature in Inspector for Images and Videos, enabling this removes the media's background like how CapCut does (CapCut has different options like Auto Removal, Custom Removal, Chroma key, etc.). Custom removal gives the user Smart Brush, Brush, Smart Eraser, Eraser to outline the target. _(Video › Basic › Remove background: Off, Automatic, Custom, Green screen, Blue screen. Automatic finds the person with a segmentation model compiled into the app (no download, works offline and on phones); its masks are found in the background, ten a second of footage, and cached in the project folder. Custom adds Smart brush, Brush, Smart eraser and Eraser, painted on the monitor with a brush-size slider; each stroke is one undo step and Clear strokes takes them all off. Feather softens the edge. Exports and the preview cut the same pixels.)_+- [ ] Add i18n-equivalent feature for Settings language change with json locales.\ No newline at end of filediff --git a/engine/.cargo/config.toml b/engine/.cargo/config.tomlindex c15e7156..75413d2c 100644--- a/engine/.cargo/config.toml+++ b/engine/.cargo/config.toml@@ -19,3 +19,11 @@# (Deliberately not FFMPEG_DLL_PATH: rusty_ffmpeg turns that into the# linker's /LIBPATH, which hunts for import libraries in bin/ where only# the DLLs live. The DLLs still have to be findable at run time.)++# No incremental caches. They are a speed-up for the second build of one+# crate and a tax on the disk for everything else: a session of edits across+# the workspace leaves tens of gigabytes of them behind. The shipping builds+# run on GitHub Actions; a machine here needs `cargo check` and the tests of+# the crates it is changing, and those are quick without.+[build]+incremental = falsediff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 7dec9247..90eb546b 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -140,7 +140,7 @@ checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"dependencies = ["cfg-if","cipher",- "cpufeatures",+ "cpufeatures 0.2.17",][[package]]@@ -359,6 +359,12 @@ version = "1.0.104"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"+[[package]]+name = "anymap3"+version = "1.1.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "fb5dfbc6d8d2675589ccbe4d0fd61df2419075625f8c1a62325e718e2b0049f9"+[[package]]name = "arbitrary"version = "1.4.2"@@ -692,7 +698,16 @@ version = "0.9.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "34ddef2995421ab6a5c779542c81ee77c115206f4ad9d5a8e05f4ff49716a3dd"dependencies = [- "bit-vec",+ "bit-vec 0.9.1",+]++[[package]]+name = "bit-set"+version = "0.11.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "56d87354e4229f54a44f7bf2435906a4656dba36026ab6eaca629a2c436a691c"+dependencies = [+ "bit-vec 0.10.1",][[package]]@@ -701,6 +716,16 @@ version = "0.9.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "b71798fca2c1fe1086445a7258a4bc81e6e49dcd24c8d0dd9a1e57395b603f51"+[[package]]+name = "bit-vec"+version = "0.10.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5727b15fa97d4f4fee0a3b7c3d550ed0269f54329207b86388de918604e31269"+dependencies = [+ "borsh",+ "serde",+]+[[package]]name = "bit_field"version = "0.10.3"@@ -777,10 +802,24 @@ version = "1.8.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a"dependencies = [+ "borsh-derive","bytes","cfg_aliases",]+[[package]]+name = "borsh-derive"+version = "1.8.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56"+dependencies = [+ "once_cell",+ "proc-macro-crate",+ "proc-macro2",+ "quote",+ "syn 3.0.4",+]+[[package]]name = "bstr"version = "1.13.1"@@ -985,6 +1024,17 @@ dependencies = ["libc",]+[[package]]+name = "chacha20"+version = "0.10.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"+dependencies = [+ "cfg-if",+ "cpufeatures 0.3.1",+ "rand_core 0.10.1",+]+[[package]]name = "chrono"version = "0.4.45"@@ -1140,6 +1190,7 @@ dependencies = ["concat-media","concat-project","concat-speech",+ "concat-vision","opener","pollster","rfd",@@ -1197,6 +1248,7 @@ dependencies = ["concat-media","concat-project","concat-render",+ "concat-vision","serde",]@@ -1210,6 +1262,7 @@ dependencies = ["concat-project","concat-render","concat-text",+ "concat-vision","cpal","memmap2","serde",@@ -1271,6 +1324,16 @@ dependencies = ["ttf-parser",]+[[package]]+name = "concat-vision"+version = "0.2.1"+dependencies = [+ "concat-core",+ "concat-project",+ "png 0.17.16",+ "tract-onnx",+]+[[package]]name = "concurrent-queue"version = "2.5.0"@@ -1434,6 +1497,15 @@ dependencies = ["libc",]+[[package]]+name = "cpufeatures"+version = "0.3.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566"+dependencies = [+ "libc",+]+[[package]]name = "crc"version = "3.4.0"@@ -1553,6 +1625,17 @@ version = "0.5.8"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"+[[package]]+name = "derive-new"+version = "0.7.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "2cdc8d50f426189eef89dac62fabfa0abb27d5cc008f25bf4156a0203325becc"+dependencies = [+ "proc-macro2",+ "quote",+ "syn 2.0.119",+]+[[package]]name = "derive_arbitrary"version = "1.4.2"@@ -1662,6 +1745,12 @@ version = "1.2.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "75b325c5dbd37f80359721ad39aca5a29fb04c89279657cffdda8736d0c0b9d2"+[[package]]+name = "downcast-rs"+version = "2.0.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "117240f60069e65410b3ae1bb213295bd828f707b5bec6596a1afc8793ce0cbc"+[[package]]name = "dpi"version = "0.1.2"@@ -1714,6 +1803,24 @@ version = "0.8.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "edf234dd1594d6dd434a8fb8cada51ddbbc593e40e4a01556a0b31c62da2775b"+[[package]]+name = "dyn-clone"+version = "1.0.20"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"++[[package]]+name = "dyn-eq"+version = "0.1.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5c2d035d21af5cde1a6f5c7b444a5bf963520a9f142e5d06931178433d7d5388"++[[package]]+name = "dyn-hash"+version = "1.0.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5fdab65db9274e0168143841eb8f864a0a21f8b1b8d2ba6812bbe6024346e99e"+[[package]]name = "either"version = "1.18.0"@@ -1783,6 +1890,17 @@ version = "1.0.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"+[[package]]+name = "erased-serde"+version = "0.4.10"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d2add8a07dd6a8d93ff627029c51de145e12686fbc36ecb298ac22e74cf02dec"+dependencies = [+ "serde",+ "serde_core",+ "typeid",+]+[[package]]name = "errno"version = "0.3.14"@@ -1966,6 +2084,12 @@ version = "0.9.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "98de4bbd547a563b716d8dfa9aad1cb19bfab00f4fa09a6a4ed21dbcf44ce9c4"+[[package]]+name = "float-ord"+version = "0.3.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8ce81f49ae8a0482e4c55ea62ebbd7e5a686af544c00b9d090bba3ff9be97b3d"+[[package]]name = "fnv"version = "1.0.7"@@ -2275,6 +2399,7 @@ dependencies = ["cfg-if","libc","r-efi 6.0.0",+ "rand_core 0.10.1",][[package]]@@ -2470,6 +2595,8 @@ dependencies = ["allocator-api2","equivalent","foldhash 0.2.0",+ "serde",+ "serde_core",][[package]]@@ -3075,7 +3202,7 @@ dependencies = ["image-webp","moxcms","num-traits",- "png",+ "png 0.18.1","qoi","ravif","rayon",@@ -3165,6 +3292,15 @@ dependencies = ["syn 2.0.119",]+[[package]]+name = "inventory"+version = "0.3.24"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a4f0c30c76f2f4ccee3fe55a2435f691ca00c0e4bd87abe4f4a851b1d4dac39b"+dependencies = [+ "rustversion",+]+[[package]]name = "io-lifetimes"version = "1.0.11"@@ -3200,6 +3336,15 @@ dependencies = ["either",]+[[package]]+name = "itertools"+version = "0.15.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8b4baf93f58d4425749ca49a51c50ebab072c5df6994d08fed93541c331481dc"+dependencies = [+ "either",+]+[[package]]name = "itoa"version = "1.0.18"@@ -3577,6 +3722,22 @@ dependencies = ["libc",]+[[package]]+name = "maplit"+version = "1.0.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3e2e65a1a2e43cfcb47a895c4c8b10d1f4a61097f9f254f183aee60cad9c651d"++[[package]]+name = "matrixmultiply"+version = "0.3.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3f607c237553f086e7043417a51df26b2eb899d3caff94e6a67592ff992fedc7"+dependencies = [+ "autocfg",+ "rawpointer",+]+[[package]]name = "maybe-rayon"version = "0.1.1"@@ -3602,6 +3763,12 @@ dependencies = ["libc",]+[[package]]+name = "memo-map"+version = "0.3.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5449c8c750f1a07ea702bbd212bd999fceece9b3d1508b17023b3e174583124b"+[[package]]name = "memoffset"version = "0.9.1"@@ -3611,6 +3778,16 @@ dependencies = ["autocfg",]+[[package]]+name = "minijinja"+version = "2.24.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "86886cf6dbf4e614b19c9a1eec9775f021869d7eadde0fc73921a81b90c9b4c9"+dependencies = [+ "memo-map",+ "serde",+]+[[package]]name = "minimal-lexical"version = "0.2.1"@@ -3661,7 +3838,7 @@ dependencies = ["objc2-core-foundation","objc2-foundation 0.3.2","once_cell",- "png",+ "png 0.18.1","thiserror 2.0.20","windows-sys 0.61.2",]@@ -3673,7 +3850,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "b2bf919621e7975acb27d881bae2fb993e0d45c8e0446e85e6272971e00dc8df"dependencies = ["arrayvec",- "bit-set",+ "bit-set 0.9.1","bitflags 2.13.1","cfg-if","cfg_aliases",@@ -3698,6 +3875,21 @@ version = "1.0.9"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "308d96db8debc727c3fd9744aac51751243420e46edf401010908da7f8d5e57c"+[[package]]+name = "ndarray"+version = "0.17.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "520080814a7a6b4a6e9070823bb24b4531daac8c4627e08ba5de8c5ef2f2752d"+dependencies = [+ "matrixmultiply",+ "num-complex",+ "num-integer",+ "num-traits",+ "portable-atomic",+ "portable-atomic-util",+ "rawpointer",+]+[[package]]name = "ndk"version = "0.8.0"@@ -3797,6 +3989,15 @@ dependencies = ["memchr",]+[[package]]+name = "nom-language"+version = "0.1.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "2de2bc5b451bfedaef92c90b8939a8fff5770bdcc1fafd6239d086aab8fa6b29"+dependencies = [+ "nom 8.0.0",+]+[[package]]name = "noop_proc_macro"version = "0.3.0"@@ -4587,6 +4788,19 @@ version = "0.2.3"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6"+[[package]]+name = "png"+version = "0.17.16"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "82151a2fc869e011c153adc57cf2789ccb8d9906ce52c0b39a6b5697749d7526"+dependencies = [+ "bitflags 1.3.2",+ "crc32fast",+ "fdeflate",+ "flate2",+ "miniz_oxide 0.8.9",+]+[[package]]name = "png"version = "0.18.1"@@ -4689,6 +4903,15 @@ dependencies = ["syn 2.0.119",]+[[package]]+name = "primal-check"+version = "0.3.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "dc0d895b311e3af9902528fbb8f928688abbd95872819320517cc24ca6b2bd08"+dependencies = [+ "num-integer",+]+[[package]]name = "proc-macro-crate"version = "3.5.0"@@ -4726,6 +4949,29 @@ dependencies = ["syn 2.0.119",]+[[package]]+name = "prost"+version = "0.14.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "528ac67416ff8646872a3c02cad9cc4ee5dc9f9540c9b10771855c95cb2e5ae1"+dependencies = [+ "bytes",+ "prost-derive",+]++[[package]]+name = "prost-derive"+version = "0.14.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"+dependencies = [+ "anyhow",+ "itertools 0.14.0",+ "proc-macro2",+ "quote",+ "syn 2.0.119",+]+[[package]]name = "pulldown-cmark"version = "0.13.4"@@ -4826,7 +5072,18 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41"dependencies = ["rand_chacha",- "rand_core",+ "rand_core 0.9.5",+]++[[package]]+name = "rand"+version = "0.10.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"+dependencies = [+ "chacha20",+ "getrandom 0.4.3",+ "rand_core 0.10.1",][[package]]@@ -4836,7 +5093,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"dependencies = ["ppv-lite86",- "rand_core",+ "rand_core 0.9.5",][[package]]@@ -4848,6 +5105,22 @@ dependencies = ["getrandom 0.3.4",]+[[package]]+name = "rand_core"+version = "0.10.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"++[[package]]+name = "rand_distr"+version = "0.6.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "4d431c2703ccf129de4d45253c03f49ebb22b97d6ad79ee3ecfc7e3f4862c1d8"+dependencies = [+ "num-traits",+ "rand 0.10.2",+]+[[package]]name = "range-alloc"version = "0.1.5"@@ -4881,7 +5154,7 @@ dependencies = ["num-traits","paste","profiling",- "rand",+ "rand 0.9.5","rand_chacha","simd_helpers","thiserror 2.0.20",@@ -4931,6 +5204,12 @@ dependencies = ["objc2-quartz-core 0.3.2",]+[[package]]+name = "rawpointer"+version = "0.2.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3"+[[package]]name = "rayon"version = "1.12.0"@@ -5168,6 +5447,20 @@ dependencies = ["semver",]+[[package]]+name = "rustfft"+version = "6.4.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "21db5f9893e91f41798c88680037dba611ca6674703c1a18601b01a72c8adb89"+dependencies = [+ "num-complex",+ "num-integer",+ "num-traits",+ "primal-check",+ "strength_reduce",+ "transpose",+]+[[package]]name = "rustix"version = "0.38.44"@@ -5253,6 +5546,19 @@ dependencies = ["unicode-script",]+[[package]]+name = "safetensors"+version = "0.8.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "79b079b829cb27a1c3c374341345ed2e8b2c0c839034522cee576c140bd7f846"+dependencies = [+ "hashbrown 0.16.1",+ "libc",+ "serde",+ "serde_json",+ "tempfile",+]+[[package]]name = "same-file"version = "1.0.6"@@ -5262,6 +5568,15 @@ dependencies = ["winapi-util",]+[[package]]+name = "scan_fmt"+version = "0.2.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0b53b0a5db882a8e2fdaae0a43f7b39e7e9082389e978398bdf223a55b581248"+dependencies = [+ "regex",+]+[[package]]name = "scoped-tls"version = "1.0.1"@@ -5369,7 +5684,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8"dependencies = ["cfg-if",- "cpufeatures",+ "cpufeatures 0.2.17","digest",]@@ -5748,6 +6063,12 @@ version = "1.1.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"+[[package]]+name = "strength_reduce"+version = "0.2.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "fe895eb47f22e2ddd4dabc02bce419d2e643c8e3b585c78158b349195bc24d82"+[[package]]name = "strict-num"version = "0.1.1"@@ -5757,6 +6078,16 @@ dependencies = ["float-cmp",]+[[package]]+name = "string-interner"+version = "0.20.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ad3df9b59e2eded8d825c7c4363ad339a20fb6bc0b9a4778560f518f59910b15"+dependencies = [+ "hashbrown 0.16.1",+ "serde",+]+[[package]]name = "strsim"version = "0.11.1"@@ -6014,7 +6345,7 @@ dependencies = ["bytemuck","cfg-if","log",- "png",+ "png 0.18.1","tiny-skia-path 0.12.0",]@@ -6163,6 +6494,223 @@ dependencies = ["once_cell",]+[[package]]+name = "tract-core"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "49647af4b6a0c5522c0c9b8120ce517146f2a7fd67797a7c6ab14fe91f01c5d5"+dependencies = [+ "anyhow",+ "anymap3",+ "bit-set 0.11.1",+ "derive-new",+ "downcast-rs 2.0.2",+ "dyn-clone",+ "dyn-eq",+ "erased-serde",+ "inventory",+ "lazy_static",+ "log",+ "maplit",+ "ndarray",+ "num-complex",+ "num-integer",+ "num-traits",+ "pastey 0.2.3",+ "rustfft",+ "serde",+ "smallvec",+ "tract-data",+ "tract-linalg",+]++[[package]]+name = "tract-data"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "33bb72b6667b6a4a698a263044aec741dc2887425ada192874f7b80b8fb3f85c"+dependencies = [+ "anyhow",+ "downcast-rs 2.0.2",+ "dyn-clone",+ "dyn-eq",+ "dyn-hash",+ "half",+ "inventory",+ "itertools 0.15.0",+ "lazy_static",+ "libm",+ "maplit",+ "ndarray",+ "nom 8.0.0",+ "nom-language",+ "num-integer",+ "num-traits",+ "parking_lot",+ "scan_fmt",+ "smallvec",+ "string-interner",+]++[[package]]+name = "tract-extra"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "cdd571adedfa2108532688c12e94b95640c87793bdaffd3daef238fed5a071ad"+dependencies = [+ "tract-nnef",+ "tract-pulse",+]++[[package]]+name = "tract-hir"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c17fd20ade88f142bf0419d899c3ce56fe00ee7d208794d5eb9c86059649b89c"+dependencies = [+ "derive-new",+ "log",+ "tract-core",+]++[[package]]+name = "tract-linalg"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c4aa27179d6809f4f5797ed2d0df8d0a79588c5ec7a78a68eb98c3aec85fa9f4"+dependencies = [+ "byteorder",+ "cc",+ "derive-new",+ "downcast-rs 2.0.2",+ "dyn-clone",+ "dyn-eq",+ "dyn-hash",+ "half",+ "inventory",+ "lazy_static",+ "libc",+ "log",+ "minijinja",+ "num-traits",+ "pastey 0.2.3",+ "scan_fmt",+ "tract-data",+ "walkdir",+]++[[package]]+name = "tract-nnef"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0e746cd34227b14280dbd2ac176400224f6407ea304b6c16f87b5be757ce73b4"+dependencies = [+ "byteorder",+ "erased-serde",+ "flate2",+ "log",+ "minijinja",+ "nom 8.0.0",+ "nom-language",+ "safetensors",+ "serde",+ "serde_json",+ "simd-adler32",+ "tar",+ "tract-core",+ "walkdir",+]++[[package]]+name = "tract-onnx"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "320e35b04af77879b05921ab0509a44b77f52fcd452e92162e8052f5d1f4a762"+dependencies = [+ "bytes",+ "derive-new",+ "dyn-eq",+ "log",+ "memmap2",+ "nom 8.0.0",+ "nom-language",+ "num-integer",+ "num-rational",+ "num-traits",+ "prost",+ "smallvec",+ "tract-extra",+ "tract-hir",+ "tract-nnef",+ "tract-onnx-opl",+ "tract-transformers",+]++[[package]]+name = "tract-onnx-opl"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "02e723cf0756822155aff3f4046cb85c62ccd8fffd0ea39290b1640b67ddbf9f"+dependencies = [+ "dyn-eq",+ "getrandom 0.4.3",+ "log",+ "rand 0.10.2",+ "rand_distr",+ "rustfft",+ "tract-extra",+ "tract-nnef",+]++[[package]]+name = "tract-pulse"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "7af0b850fc74a39da37f6ace584cd54dce773be9b7ec71f45f4011cb0125db3c"+dependencies = [+ "downcast-rs 2.0.2",+ "dyn-eq",+ "erased-serde",+ "lazy_static",+ "log",+ "serde",+ "tract-pulse-opl",+ "tract-transformers",+]++[[package]]+name = "tract-pulse-opl"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e0629fb3c289fb906d95a2aa1a71b4389511e8620f8bde3c711073e393414729"+dependencies = [+ "downcast-rs 2.0.2",+ "dyn-eq",+ "lazy_static",+ "tract-nnef",+]++[[package]]+name = "tract-transformers"+version = "0.23.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a075c3a29baf4a859c67a69fd4643c43ccda51e9644d74af9481937a7e57f710"+dependencies = [+ "float-ord",+ "rayon",+ "tract-nnef",+]++[[package]]+name = "transpose"+version = "0.2.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "1ad61aed86bc3faea4300c7aee358b4c6d0c8d6ccc36524c96e4c92ccf26e77e"+dependencies = [+ "num-integer",+ "strength_reduce",+]+[[package]]name = "ttf-parser"version = "0.25.1"@@ -6182,6 +6730,12 @@ dependencies = ["serde",]+[[package]]+name = "typeid"+version = "1.0.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "bc7d623258602320d5c55d1bc22793b57daff0ec7efc270ea7d55ce1d5f5471c"+[[package]]name = "typenum"version = "1.20.1"@@ -6508,7 +7062,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "38a91b4eaddff87b1cd1074985e3713da4af2c49742d1b356b2c01670a67a078"dependencies = ["cc",- "downcast-rs",+ "downcast-rs 1.2.1","rustix 1.1.4","scoped-tls","smallvec",@@ -6733,8 +7287,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "2f519832254e56965a9940c4af57dcb75f702b6f6fa4a0b172f685395843a4d7"dependencies = ["arrayvec",- "bit-set",- "bit-vec",+ "bit-set 0.9.1",+ "bit-vec 0.9.1","bitflags 2.13.1","bytemuck","cfg_aliases",@@ -6795,7 +7349,7 @@ dependencies = ["android_system_properties","arrayvec","ash",- "bit-set",+ "bit-set 0.9.1","bitflags 2.13.1","block2 0.6.2","bytemuck",diff --git a/engine/Cargo.toml b/engine/Cargo.tomlindex 7c9287a6..e8c62d8d 100644--- a/engine/Cargo.toml+++ b/engine/Cargo.toml@@ -18,6 +18,7 @@ concat-host = { path = "crates/concat-host" }concat-speech = { path = "crates/concat-speech" }concat-effects = { path = "crates/concat-effects" }concat-text = { path = "crates/concat-text" }+concat-vision = { path = "crates/concat-vision" }concat = { path = "crates/concat" }serde_json = "1"diff --git a/engine/crates/concat-android/src/lib.rs b/engine/crates/concat-android/src/lib.rsindex 60d7d699..df216a50 100644--- a/engine/crates/concat-android/src/lib.rs+++ b/engine/crates/concat-android/src/lib.rs@@ -75,7 +75,10 @@ mod activity {std::thread::Builder::new().name("stderr-to-logcat".into()).spawn(move || {- for line in std::io::BufReader::new(reader).lines().map_while(Result::ok) {+ for line in std::io::BufReader::new(reader)+ .lines()+ .map_while(Result::ok)+ {log::warn!("{line}");}})diff --git a/engine/crates/concat-export/Cargo.toml b/engine/crates/concat-export/Cargo.tomlindex 54016d85..efa237d0 100644--- a/engine/crates/concat-export/Cargo.toml+++ b/engine/crates/concat-export/Cargo.toml@@ -17,6 +17,9 @@ concat-media.workspace = trueconcat-render.workspace = trueconcat-project = { path = "../concat-project" }concat-effects.workspace = true+# Masks for cutouts: read from the project's cache and applied to a decoded+# frame before it is placed. Reading only; the host finds them.+concat-vision.workspace = trueserde = { version = "1", features = ["derive"] }[lints]diff --git a/engine/crates/concat-export/src/flatten.rs b/engine/crates/concat-export/src/flatten.rsindex 6fc1d783..e458e694 100644--- a/engine/crates/concat-export/src/flatten.rs+++ b/engine/crates/concat-export/src/flatten.rs@@ -13,6 +13,8 @@//! separately and rejoin the exporter as plain image clips; see//! `ExportRequest`'s caller.+use std::path::Path;+use concat_project::model::{ClipKind as ModelClipKind, Project, Timeline};use crate::chains::{audio_filter_chain, video_effect_chain};@@ -23,6 +25,17 @@ use crate::{ClipKind, ExportClip, ExportKey, TransitionSpec};/// separately and rejoin as image clips); a clip whose media or track has/// vanished is skipped with the same tolerance the document reader extends.pub fn flatten_timeline(project: &Project, timeline_id: Option<&str>) -> Vec<ExportClip> {+ flatten_timeline_in(project, timeline_id, None)+}++/// [`flatten_timeline`], knowing the project folder: a clip with a cutout+/// is then told where its media's masks are, and renders cut. Without the+/// folder the cutout is carried but not rendered.+pub fn flatten_timeline_in(+ project: &Project,+ timeline_id: Option<&str>,+ project_dir: Option<&Path>,+) -> Vec<ExportClip> {let Some(timeline) = pick_timeline(project, timeline_id) else {return Vec::new();};@@ -80,6 +93,8 @@ pub fn flatten_timeline(project: &Project, timeline_id: Option<&str>) -> Vec<Expmedia_width: None,media_height: None,has_audio: Some(false),+ cutout: None,+ mask_dir: String::new(),});}@@ -149,6 +164,13 @@ pub fn flatten_timeline(project: &Project, timeline_id: Option<&str>) -> Vec<Expmedia_width: media.width,media_height: media.height,has_audio: Some(media.has_audio),+ cutout: clip.cutout.clone(),+ mask_dir: match (&clip.cutout, project_dir) {+ (Some(_), Some(dir)) => concat_vision::mask_dir(dir, &media.path)+ .to_string_lossy()+ .into_owned(),+ _ => String::new(),+ },})}).collect()diff --git a/engine/crates/concat-export/src/lib.rs b/engine/crates/concat-export/src/lib.rsindex 687637ac..504086cf 100644--- a/engine/crates/concat-export/src/lib.rs+++ b/engine/crates/concat-export/src/lib.rs@@ -36,8 +36,9 @@ use concat_core::timeline::{Clip, ClipId, MediaRef, Timeline, Track, TrackKind,use concat_effects::Catalogue;use concat_media::audio::{self, AudioClip};use concat_media::{DecodeOptions, Decoder, EncodeOptions, Encoder, FrameSink, FrameSource};-use concat_project::model::AppliedFilter;+use concat_project::model::{AppliedFilter, Cutout};use concat_render::{Compositor, CpuCompositor, Layer, Placement, plan_frame};+use concat_vision::{Mapping, MaskStore};use serde::Deserialize;/// What a flattened clip is. Typed, so a kind check the compiler has not@@ -185,6 +186,15 @@ pub struct ExportClip {/// older caller still exports correctly - just slower to start.#[serde(default)]pub has_audio: Option<bool>,+ /// The background taken away by a mask, as the document holds it.+ /// Rendered only with `mask_dir`: a cutout whose masks are nowhere yet+ /// leaves the picture whole.+ #[serde(default)]+ pub cutout: Option<Cutout>,+ /// Where this clip's media has its masks - see `concat_vision::mask_dir`+ /// - or empty when the flattener had no project folder to name it by.+ #[serde(default)]+ pub mask_dir: String,}/// One animation key, as the flattener hands it over.@@ -578,6 +588,56 @@ struct Source<F> {passes: Vec<ShaderPass>,}+/// A cutout as the frame loop runs it: the masks, what to paint on them,+/// and how a decoded pixel finds its place in the source.+struct CutoutJob {+ store: MaskStore,+ cutout: Cutout,+ mapping: Mapping,+ /// The source's width over its height, for round brushes.+ aspect: f32,+}++impl CutoutJob {+ /// The job for a clip, or `None` when it has no cutout or no masks to+ /// cut with.+ fn of(clip: &ExportClip) -> Option<CutoutJob> {+ let cutout = clip.cutout.clone()?;+ if clip.mask_dir.is_empty() {+ return None;+ }+ let aspect = match (clip.media_width, clip.media_height) {+ (Some(width), Some(height)) if width > 0 && height > 0 => width as f32 / height as f32,+ _ => 1.0,+ };+ Some(CutoutJob {+ store: MaskStore::open(Path::new(&clip.mask_dir)),+ cutout,+ mapping: Mapping {+ crop: clip+ .crop+ .map(|edges| edges.map(|edge| edge as f32))+ .unwrap_or([0.0; 4]),+ flip_h: clip.flip_h,+ flip_v: clip.flip_v,+ },+ aspect,+ })+ }++ /// The frame with its background gone, when the instant has a mask.+ /// `None` leaves the picture whole: an instant not analysed yet is+ /// shown as shot rather than not at all.+ fn cut(&self, frame: &Frame, source_time: Rational) -> Option<Frame> {+ let mask = self+ .store+ .resolved(source_time.as_f64(), &self.cutout, self.aspect)?;+ let mut out = frame.clone();+ concat_vision::cut(&mut out, &mask, &self.mapping);+ Some(out)+ }+}+/// The fraction-to-pixel placement of one composited layer: fitted and/// centred is the base, the clip's transform moves it from there. The one/// definition the exporter and the preview share - these two paths must@@ -640,6 +700,7 @@ fn render_picture(treatments,pre_chains,passes,+ cutouts,} = build_timeline(request, rate, visible, gpu);let mut encoder = Encoder::create(@@ -690,8 +751,12 @@ fn render_picture(chain,pre,) {+ let frame = cutouts+ .get(&layer.clip)+ .and_then(|job| job.cut(&frame, layer.source_time))+ .unwrap_or_else(|| frame.as_ref().clone());sources.push(Source {- frame: frame.as_ref().clone(),+ frame,opacity: layer.opacity,transform: layer.transform,track: tracks.get(&layer.clip).copied().unwrap_or(0),@@ -754,6 +819,15 @@ fn render_picture(// aborting the export - a clip trimmed past its media's end is a// mistake in the edit, not a failure of the renderer.if let Some(frame) = decoder.next_frame().map_err(|error| error.to_string())? {+ // The cutout, on the decoded picture before it is placed:+ // the same frame both compositors then draw.+ let frame = match cutouts+ .get(&layer.clip)+ .and_then(|job| job.cut(&frame, layer.source_time))+ {+ Some(cut) => cut,+ None => frame,+ };sources.push(Source {frame,opacity: layer.opacity,@@ -828,6 +902,8 @@ struct BuiltTimeline {passes: HashMap<ClipId, Vec<ShaderPass>>,/// The layers: treatments over the stack, by span.treatments: Vec<Treatment>,+ /// The clips whose background a mask takes away.+ cutouts: HashMap<ClipId, CutoutJob>,}/// A layer clip, as the compositor needs it: when, over which tracks, what@@ -962,6 +1038,7 @@ fn build_timeline(let mut treatments: Vec<Treatment> = Vec::new();let mut pre_chains: HashMap<ClipId, String> = HashMap::new();let mut passes: HashMap<ClipId, Vec<ShaderPass>> = HashMap::new();+ let mut cutouts: HashMap<ClipId, CutoutJob> = HashMap::new();let lanes = visible.iter().map(|clip| clip.track).max().unwrap_or(0) + 1;let tracks: Vec<_> = (0..lanes)@@ -1045,6 +1122,9 @@ fn build_timeline(passes.insert(id, clip_passes);}}+ if let Some(job) = CutoutJob::of(clip) {+ cutouts.insert(id, job);+ }}}@@ -1057,6 +1137,7 @@ fn build_timeline(treatments,pre_chains,passes,+ cutouts,}}@@ -1320,6 +1401,7 @@ pub fn preview_sources(treatments,pre_chains,passes,+ cutouts,} = preview_timeline(request, rate, gpu);let time = quantise(request.time, rate);let plan = plan_frame(&timeline, time);@@ -1344,14 +1426,23 @@ pub fn preview_sources(chain,pre,) {- Ok(frame) => sources.push(Source {- frame,- opacity: layer.opacity,- transform: layer.transform,- track: tracks.get(&layer.clip).copied().unwrap_or(0),- blend: layer.blend,- passes: passes.get(&layer.clip).cloned().unwrap_or_default(),- }),+ Ok(frame) => {+ let frame = match cutouts+ .get(&layer.clip)+ .and_then(|job| job.cut(&frame, layer.source_time))+ {+ Some(cut) => std::sync::Arc::new(cut),+ None => frame,+ };+ sources.push(Source {+ frame,+ opacity: layer.opacity,+ transform: layer.transform,+ track: tracks.get(&layer.clip).copied().unwrap_or(0),+ blend: layer.blend,+ passes: passes.get(&layer.clip).cloned().unwrap_or_default(),+ })+ }Err(error) => failures.push(format!("{}: {error}", layer.media.display())),}}@@ -1571,6 +1662,8 @@ mod tests {media_width: None,media_height: None,has_audio: None,+ cutout: None,+ mask_dir: String::new(),}}diff --git a/engine/crates/concat-host/Cargo.toml b/engine/crates/concat-host/Cargo.tomlindex e470dbc1..767be51c 100644--- a/engine/crates/concat-host/Cargo.toml+++ b/engine/crates/concat-host/Cargo.toml@@ -18,6 +18,8 @@ concat-project.workspace = trueconcat-export.workspace = true# Titles become pictures here; see titles.rs.concat-text.workspace = true+# The cutout model and its masks; `infer` because the host is what runs it.+concat-vision = { workspace = true, features = ["infer"] }concat-render = { workspace = true, optional = true }serde = { version = "1", features = ["derive"] }serde_json.workspace = truediff --git a/engine/crates/concat-host/src/cutout.rs b/engine/crates/concat-host/src/cutout.rsnew file mode 100644index 00000000..04199a20--- /dev/null+++ b/engine/crates/concat-host/src/cutout.rs@@ -0,0 +1,234 @@+// SPDX-License-Identifier: AGPL-3.0-or-later+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors++//! Finding the masks a cutout is made of.+//!+//! A clip with a cutout needs a mask for every source instant it shows, on+//! the analysis grid `concat-vision` reads by. This is the job that fills+//! the gaps: it decodes the media at the model's size along each missing+//! stretch, runs the model on several frames at once, and writes what it+//! finds into the media's mask store in the project folder. A still is one+//! frame; footage is [`concat_vision::MASK_RATE`] frames a second of source.+//!+//! One job at a time through a [`SingleFlight`], like every long job the+//! host runs; the window queues the next media behind it. The model loads+//! on first use and stays loaded.++use std::path::PathBuf;+use std::sync::{Arc, OnceLock};++use concat_core::frame::Frame;+use concat_core::time::{FrameRate, Rational};+use concat_media::{DecodeOptions, Decoder, FrameSource};+use concat_vision::{MASK_RATE, MODEL_SIZE, Mask, MaskStore, Segmenter, mask_dir};++use crate::jobs::{Job, SingleFlight};++/// What one analysis covers.+#[derive(Clone, Debug)]+pub struct AnalyseRequest {+ /// The project folder the masks are cached under.+ pub project: PathBuf,+ /// The media file to analyse.+ pub media_path: String,+ /// A still: one frame answers for every instant.+ pub still: bool,+ /// The stretches of source, in seconds, that clips show. Instants+ /// already analysed inside them are skipped.+ pub ranges: Vec<(f64, f64)>,+}++/// The analysis service: the model, loaded once, and the one-job slot.+pub struct Cutouts {+ gate: Arc<SingleFlight>,+ segmenter: OnceLock<Result<Arc<Segmenter>, String>>,+}++impl Default for Cutouts {+ fn default() -> Self {+ Self::new()+ }+}++impl Cutouts {+ /// A service with nothing loaded yet.+ pub fn new() -> Cutouts {+ Cutouts {+ gate: Arc::new(SingleFlight::new()),+ segmenter: OnceLock::new(),+ }+ }++ /// Whether an analysis is running.+ pub fn is_busy(&self) -> bool {+ self.gate.is_busy()+ }++ /// Asks the running analysis to stop after the frames in hand.+ pub fn cancel(&self) {+ self.gate.cancel();+ }++ fn segmenter(&self) -> Result<Arc<Segmenter>, String> {+ self.segmenter+ .get_or_init(|| Segmenter::load().map(Arc::new))+ .clone()+ }++ /// How many instants `request` still needs, without doing anything.+ pub fn outstanding(request: &AnalyseRequest) -> usize {+ let store = MaskStore::open(&mask_dir(&request.project, &request.media_path));+ if request.still {+ return usize::from(store.is_empty());+ }+ missing(&store, &request.ranges).len()+ }++ /// Fills the masks `request` is missing. Blocks for the whole run, so+ /// run it on its own thread; `progress` is called with `0..=1` as it+ /// goes. Returns how many masks were written.+ pub fn analyse(+ &self,+ request: &AnalyseRequest,+ progress: &mut dyn FnMut(f32),+ ) -> Result<usize, String> {+ let job = self.gate.begin("cutout analysis")?;+ let segmenter = self.segmenter()?;+ let mut store = MaskStore::open(&mask_dir(&request.project, &request.media_path));++ if request.still {+ if !store.is_empty() {+ return Ok(0);+ }+ let options = DecodeOptions::default()+ .scaled_to(MODEL_SIZE, MODEL_SIZE)+ .limited_to(1);+ let mut decoder =+ Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())?;+ let frame = decoder+ .next_frame()+ .map_err(|error| error.to_string())?+ .ok_or_else(|| format!("{}: no picture to analyse", request.media_path))?;+ store.put(0, &segmenter.mask(&frame)?)?;+ progress(1.0);+ return Ok(1);+ }++ let wanted = missing(&store, &request.ranges);+ let total = wanted.len();+ if total == 0 {+ return Ok(0);+ }+ let step_ms = 1000 / u64::from(MASK_RATE);+ let workers = std::thread::available_parallelism()+ .map(|count| count.get())+ .unwrap_or(1)+ .clamp(1, 8);+ let mut done = 0usize;++ // Each unbroken run of missing instants is one pass of a decoder+ // paced to the analysis grid, so no frame is sought twice.+ for run in runs(&wanted, step_ms) {+ let start = Rational::new(run[0] as i64, 1000);+ let options = DecodeOptions::default()+ .starting_at(start)+ .scaled_to(MODEL_SIZE, MODEL_SIZE)+ .at_rate(FrameRate::new(Rational::new(i64::from(MASK_RATE), 1)))+ .limited_to(run.len() as u64);+ let mut decoder =+ Decoder::open(&request.media_path, &options).map_err(|error| error.to_string())?;+ let mut next = 0usize;+ while next < run.len() {+ if job.cancelled() {+ return Err("cutout analysis cancelled".to_owned());+ }+ // A batch of frames, one per core, found together.+ let mut batch: Vec<(u64, Frame)> = Vec::with_capacity(workers);+ while batch.len() < workers && next < run.len() {+ match decoder.next_frame().map_err(|error| error.to_string())? {+ Some(frame) => batch.push((run[next], frame)),+ // The file ran out before its stated length: what+ // was found is what there is.+ None => break,+ }+ next += 1;+ }+ if batch.is_empty() {+ break;+ }+ for (millis, mask) in find_masks(&segmenter, &batch, &job)? {+ store.put(millis, &mask)?;+ done += 1;+ }+ progress(done as f32 / total as f32);+ }+ }+ progress(1.0);+ Ok(done)+ }+}++/// Every instant the ranges need that the store lacks, ascending, once.+fn missing(store: &MaskStore, ranges: &[(f64, f64)]) -> Vec<u64> {+ let mut wanted: Vec<u64> = ranges+ .iter()+ .flat_map(|&(from, to)| store.missing(from, to))+ .collect();+ wanted.sort_unstable();+ wanted.dedup();+ wanted+}++/// The instants split into runs a step apart.+fn runs(instants: &[u64], step: u64) -> Vec<Vec<u64>> {+ let mut out: Vec<Vec<u64>> = Vec::new();+ for &at in instants {+ match out.last_mut() {+ Some(run) if run.last().is_some_and(|&last| last + step == at) => run.push(at),+ _ => out.push(vec![at]),+ }+ }+ out+}++/// The masks of a batch of frames, one thread each.+fn find_masks(+ segmenter: &Arc<Segmenter>,+ batch: &[(u64, Frame)],+ job: &Job,+) -> Result<Vec<(u64, Mask)>, String> {+ if job.cancelled() {+ return Err("cutout analysis cancelled".to_owned());+ }+ std::thread::scope(|scope| {+ let handles: Vec<_> = batch+ .iter()+ .map(|(millis, frame)| {+ let segmenter = Arc::clone(segmenter);+ scope.spawn(move || segmenter.mask(frame).map(|mask| (*millis, mask)))+ })+ .collect();+ handles+ .into_iter()+ .map(|handle| {+ handle+ .join()+ .map_err(|_| "cutout analysis panicked".to_owned())?+ })+ .collect()+ })+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn instants_group_into_runs_a_step_apart() {+ assert_eq!(+ runs(&[0, 100, 200, 500, 600, 900], 100),+ vec![vec![0, 100, 200], vec![500, 600], vec![900]]+ );+ assert!(runs(&[], 100).is_empty());+ }+}diff --git a/engine/crates/concat-host/src/lib.rs b/engine/crates/concat-host/src/lib.rsindex 801d3bee..5e59c15f 100644--- a/engine/crates/concat-host/src/lib.rs+++ b/engine/crates/concat-host/src/lib.rs@@ -8,7 +8,8 @@//! window talks to: it opens a project folder as a [`session::Session`],//! keeps the recents list, caches waveforms and filmstrips beside the//! project, composites the paused monitor's true frame, plays the audible-//! clips, packs templates, and holds the one-at-a-time slots for long jobs.+//! clips, finds the masks behind cutouts, packs templates, and holds the+//! one-at-a-time slots for long jobs.//!//! The window calls these functions in-process, and the doctrine is that//! no editing decision is made here. If a function starts deciding what an@@ -17,6 +18,7 @@//! Nothing here knows about a window. Long work reports through callbacks//! and cancels through flags, and the caller decides which thread it runs on.+pub mod cutout;pub mod dirs;pub mod export;pub mod jobs;@@ -28,6 +30,7 @@ pub mod session;pub mod templates;pub mod titles;+pub use cutout::{AnalyseRequest, Cutouts};pub use dirs::AppDirs;pub use jobs::{Job, SingleFlight};pub use projects::ProjectInfo;diff --git a/engine/crates/concat-host/src/session.rs b/engine/crates/concat-host/src/session.rsindex 42e4506b..c780b4dc 100644--- a/engine/crates/concat-host/src/session.rs+++ b/engine/crates/concat-host/src/session.rs@@ -241,7 +241,11 @@ impl Session {/// preview consume: the engine flattens its own session, so the pixels/// rendered are the model's, never a copy of it.pub fn flattened_clips(&self) -> Vec<concat_export::ExportClip> {- concat_export::flatten::flatten_timeline(self.editor.project(), None)+ concat_export::flatten::flatten_timeline_in(+ self.editor.project(),+ None,+ Some(std::path::Path::new(&self.path)),+ )}}diff --git a/engine/crates/concat-host/src/titles.rs b/engine/crates/concat-host/src/titles.rsindex 5cf1f73b..3c7bd670 100644--- a/engine/crates/concat-host/src/titles.rs+++ b/engine/crates/concat-host/src/titles.rs@@ -134,6 +134,8 @@ impl Titles {media_width: Some(width),media_height: Some(height),has_audio: Some(false),+ cutout: None,+ mask_dir: String::new(),},block,});diff --git a/engine/crates/concat-project/src/commands.rs b/engine/crates/concat-project/src/commands.rsindex c112b2cd..61443435 100644--- a/engine/crates/concat-project/src/commands.rs+++ b/engine/crates/concat-project/src/commands.rs@@ -12,8 +12,9 @@ use std::collections::HashSet;use serde::{Deserialize, Serialize};use crate::model::{- AnimationSlot, AppliedFilter, Clip, ClipAnimation, ClipKind, Crop, CustomFont, MediaItem,- MediaKind, Project, SpeedPoint, TextStyle, Timeline, Track, Transition,+ AnimationSlot, AppliedFilter, Clip, ClipAnimation, ClipKind, Crop, CustomFont, Cutout,+ CutoutMode, MediaItem, MediaKind, Project, SpeedPoint, Stroke, TextStyle, Timeline, Track,+ Transition,};/// Fallback length for media whose container reports no duration.@@ -303,6 +304,23 @@ pub enum Command {/// The shape and its seconds, or None to take it off.animation: Option<ClipAnimation>,},+ /// Sets or clears a picture's cutout: the mask that takes its+ /// background away. Tidied on the way in; see [`Cutout::tidy`].+ SetClipCutout {+ /// The clip. An unknown id is a no-op.+ clip_id: String,+ /// The cutout, or None to take it off.+ cutout: Option<Cutout>,+ },+ /// Paints one stroke onto a clip's cutout. A clip with no cutout gets a+ /// custom one; an automatic cutout becomes custom, since a stroke is a+ /// correction to it. A stroke with no points is a no-op.+ AddCutoutStroke {+ /// The clip. An unknown id is a no-op.+ clip_id: String,+ /// The stroke, in source fractions.+ stroke: Stroke,+ },/// Repositions any number of clips in one edit - one undo step for a/// whole multi-selection drag. Unknown clips and tracks are tolerated/// per [`ClipMove`].@@ -613,6 +631,7 @@ fn default_clip(id: String, track_id: String, media: &MediaItem, start: f64) ->flip_v: false,blend: String::new(),crop: None,+ cutout: None,filters: Vec::new(),video_effects: Vec::new(),muted: None,@@ -956,6 +975,7 @@ pub fn apply(flip_v: false,blend: String::new(),crop: None,+ cutout: None,filters: Vec::new(),video_effects: Vec::new(),muted: None,@@ -1022,6 +1042,7 @@ pub fn apply(flip_v: false,blend: String::new(),crop: None,+ cutout: None,filters: Vec::new(),video_effects: vec![AppliedFilter {id: effect_id,@@ -1277,6 +1298,35 @@ pub fn apply(})}+ Command::SetClipCutout { clip_id, cutout } => {+ let timeline = project.active_mut();+ let Some(clip) = timeline.clip_mut(&clip_id) else {+ return Ok(Outcome::default());+ };+ let applied = assign(&mut clip.cutout, cutout.map(Cutout::tidy));+ Ok(Outcome {+ created_id: None,+ applied,+ })+ }++ Command::AddCutoutStroke { clip_id, stroke } => {+ let timeline = project.active_mut();+ let Some(clip) = timeline.clip_mut(&clip_id) else {+ return Ok(Outcome::default());+ };+ let Some(stroke) = stroke.tidy() else {+ return Ok(Outcome::default());+ };+ let cutout = clip.cutout.get_or_insert_with(Cutout::auto);+ cutout.mode = CutoutMode::Custom;+ cutout.strokes.push(stroke);+ Ok(Outcome {+ created_id: None,+ applied: true,+ })+ }+Command::SetClipAnimation {clip_id,slot,diff --git a/engine/crates/concat-project/src/doc.rs b/engine/crates/concat-project/src/doc.rsindex 7f8287bb..8f35cbea 100644--- a/engine/crates/concat-project/src/doc.rs+++ b/engine/crates/concat-project/src/doc.rs@@ -22,8 +22,8 @@ use serde_json::{Map, Value, json};use crate::commands::{MAX_STRETCH, MIN_STRETCH};use crate::model::{- AppliedFilter, Clip, ClipAnimation, ClipKind, Crop, CustomFont, MediaItem, MediaKind, Project,- SpeedPoint, TextAlign, TextStyle, Timeline, Track, Transition,+ AppliedFilter, Clip, ClipAnimation, ClipKind, Crop, CustomFont, Cutout, MediaItem, MediaKind,+ Project, SpeedPoint, TextAlign, TextStyle, Timeline, Track, Transition,};/// Bumped only when a change cannot be absorbed by defaulting.@@ -248,6 +248,12 @@ fn read_clips(raw: Option<&Value>, tracks: &[Track], media: &[MediaItem]) -> Vec}.tidy()}),+ // Tolerated like everything else: a cutout the reader+ // cannot make sense of is no cutout.+ cutout: entry+ .get("cutout")+ .and_then(|value| serde_json::from_value::<Cutout>(value.clone()).ok())+ .map(Cutout::tidy),preserve_pitch: flag(entry.get("preservePitch"), true),filters: read_filters(entry.get("filters")),video_effects: read_filters(entry.get("videoEffects")),diff --git a/engine/crates/concat-project/src/model.rs b/engine/crates/concat-project/src/model.rsindex 59e7ae3a..53be8192 100644--- a/engine/crates/concat-project/src/model.rs+++ b/engine/crates/concat-project/src/model.rs@@ -147,6 +147,114 @@ fn yes() -> bool {true}+/// How a picture's background is taken away when there is no key colour+/// to remove: a person mask found by the cutout model, alone or corrected+/// by hand.+#[derive(Clone, PartialEq, Debug, Serialize, Deserialize)]+#[serde(rename_all = "camelCase")]+pub struct Cutout {+ /// Automatic keeps what the model finds; custom adds the strokes.+ pub mode: CutoutMode,+ /// How far the edge is softened, as a fraction of the picture's width.+ #[serde(default = "default_feather")]+ pub feather: f64,+ /// Corrections painted on the monitor, in the order they were made.+ #[serde(default, skip_serializing_if = "Vec::is_empty")]+ pub strokes: Vec<Stroke>,+}++/// The two ways a cutout decides what stays.+#[derive(Clone, Copy, PartialEq, Eq, Debug, Serialize, Deserialize)]+#[serde(rename_all = "lowercase")]+pub enum CutoutMode {+ /// The model's mask as it is.+ Auto,+ /// The model's mask, then the strokes over it.+ Custom,+}++/// One brush stroke over a cutout: which tool, how wide, and where it went.+/// Points are fractions of the source picture, `(0, 0)` its top-left, so a+/// stroke survives a change of output size, a crop or a flip.+#[derive(Clone, PartialEq, Debug, Serialize, Deserialize)]+#[serde(rename_all = "camelCase")]+pub struct Stroke {+ /// What the stroke does to the mask beneath it.+ pub tool: BrushTool,+ /// The brush's diameter as a fraction of the picture's width.+ pub size: f64,+ /// The path, as `[x, y]` fractions.+ pub points: Vec<[f64; 2]>,+}++/// The four brushes of a custom cutout.+#[derive(Clone, Copy, PartialEq, Eq, Debug, Serialize, Deserialize)]+#[serde(rename_all = "camelCase")]+pub enum BrushTool {+ /// Keeps what the model thought might be the subject under the stroke.+ SmartBrush,+ /// Keeps everything under the stroke.+ Brush,+ /// Removes what the model was unsure of under the stroke.+ SmartEraser,+ /// Removes everything under the stroke.+ Eraser,+}++/// The edge softness a cutout starts with.+pub const DEFAULT_FEATHER: f64 = 0.01;+/// The widest an edge may be softened.+pub const MAX_FEATHER: f64 = 0.1;+/// The brush's narrowest and widest, as fractions of the picture's width.+pub const MIN_BRUSH: f64 = 0.005;+/// See [`MIN_BRUSH`].+pub const MAX_BRUSH: f64 = 0.5;++fn default_feather() -> f64 {+ DEFAULT_FEATHER+}++impl Cutout {+ /// An automatic cutout with the default edge.+ pub fn auto() -> Cutout {+ Cutout {+ mode: CutoutMode::Auto,+ feather: DEFAULT_FEATHER,+ strokes: Vec::new(),+ }+ }++ /// The feather held to its range, every stroke to its own, and strokes+ /// with nowhere to go dropped.+ pub fn tidy(mut self) -> Cutout {+ self.feather = if self.feather.is_finite() {+ self.feather.clamp(0.0, MAX_FEATHER)+ } else {+ DEFAULT_FEATHER+ };+ self.strokes = self.strokes.into_iter().filter_map(Stroke::tidy).collect();+ self+ }+}++impl Stroke {+ /// The size held to its range and the points to the picture; `None`+ /// for a stroke with no points left.+ pub fn tidy(mut self) -> Option<Stroke> {+ self.size = if self.size.is_finite() {+ self.size.clamp(MIN_BRUSH, MAX_BRUSH)+ } else {+ MIN_BRUSH+ };+ self.points.retain(|[x, y]| x.is_finite() && y.is_finite());+ for point in &mut self.points {+ point[0] = point[0].clamp(-0.5, 1.5);+ point[1] = point[1].clamp(-0.5, 1.5);+ }+ (!self.points.is_empty()).then_some(self)+ }+}+/// A crop: what is taken off each edge, as fractions of the source.#[derive(Clone, Copy, PartialEq, Debug, Default, Serialize, Deserialize)]#[serde(rename_all = "camelCase")]@@ -390,6 +498,10 @@ pub struct Clip {/// fractions of the source's width and height.#[serde(default, skip_serializing_if = "Option::is_none")]pub crop: Option<Crop>,+ /// The background taken away by a mask rather than a key colour; see+ /// [`Cutout`]. Keying by colour is a package on `video_effects`.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub cutout: Option<Cutout>,/// Keep voices at their natural pitch when `speed` is not 1. On by/// default; off gives the tape-machine chipmunk/slow-motion sound.pub preserve_pitch: bool,diff --git a/engine/crates/concat-vision/Cargo.toml b/engine/crates/concat-vision/Cargo.tomlnew file mode 100644index 00000000..55c257aa--- /dev/null+++ b/engine/crates/concat-vision/Cargo.toml@@ -0,0 +1,27 @@+[package]+name = "concat-vision"+version.workspace = true+edition.workspace = true+rust-version.workspace = true+license.workspace = true+description = "What the engine sees in a picture: the person mask behind a cutout, the brush strokes that correct it, and the frame with its background taken away."++# The renderer needs only the masks: reading them from the project's cache,+# painting strokes over them and cutting a frame with the result. Finding+# them is inference, and inference is a dependency tree the export path+# should not carry, so it sits behind `infer` and the host turns it on.+[features]+infer = ["dep:tract-onnx"]++[dependencies]+concat-core.workspace = true+concat-project.workspace = true+# Masks on disk are eight-bit grey PNGs: a tenth the size of raw bytes, and+# a file anyone can open to see what the model saw.+png = "0.17"+# ONNX inference in pure Rust. No runtime library to ship, the same+# binary on every platform, and the model travels inside the crate.+tract-onnx = { version = "0.23", optional = true }++[lints]+workspace = truediff --git a/engine/crates/concat-vision/models/NOTICE.md b/engine/crates/concat-vision/models/NOTICE.mdnew file mode 100644index 00000000..ffbf636c--- /dev/null+++ b/engine/crates/concat-vision/models/NOTICE.md@@ -0,0 +1,13 @@+# The cutout model++`selfie-segmentation.onnx` is Google's MediaPipe Selfie Segmentation model+(the general, 256 × 256 variant), converted to ONNX and published by the+ONNX Community at+<https://huggingface.co/onnx-community/mediapipe_selfie_segmentation>.+Both the model and the conversion are licensed under the Apache License,+Version 2.0; see THIRD_PARTY_NOTICES.md at the repository root.++The file is compiled into `concat-vision` with `include_bytes!`, so a+cutout needs no download and works the same on a desk and on a phone. The+model takes one RGB picture as `[1, 3, 256, 256]` floats in `0..=1` and+answers with `[1, 1, 256, 256]` probabilities that each pixel is a person.diff --git a/engine/crates/concat-vision/models/selfie-segmentation.onnx b/engine/crates/concat-vision/models/selfie-segmentation.onnxnew file mode 100644index 00000000..f926ab52Binary files /dev/null and b/engine/crates/concat-vision/models/selfie-segmentation.onnx differdiff --git a/engine/crates/concat-vision/src/apply.rs b/engine/crates/concat-vision/src/apply.rsnew file mode 100644index 00000000..d97f7a2e--- /dev/null+++ b/engine/crates/concat-vision/src/apply.rs@@ -0,0 +1,187 @@+// SPDX-License-Identifier: AGPL-3.0-or-later+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors++//! The frame with its background taken away.++use concat_core::frame::{BYTES_PER_PIXEL, Frame};++use crate::Mask;++/// How a decoded pixel finds its place in the source picture.+///+/// A decoded frame is the source after its crop and its flips, fitted to+/// some size. The mask covers the source as it was shot, so a pixel of the+/// frame has to be walked back through both before it can be looked up.+#[derive(Clone, Copy, Debug, PartialEq)]+pub struct Mapping {+ /// Fractions cut off the source's left, top, right and bottom before+ /// the fit, as the document stores them.+ pub crop: [f32; 4],+ /// The frame is the source mirrored left to right.+ pub flip_h: bool,+ /// The frame is the source mirrored top to bottom.+ pub flip_v: bool,+}++impl Mapping {+ /// The frame is the source as it was shot.+ pub const IDENTITY: Mapping = Mapping {+ crop: [0.0; 4],+ flip_h: false,+ flip_v: false,+ };++ /// The source fraction under a frame fraction.+ pub fn source_of(&self, x: f32, y: f32) -> (f32, f32) {+ let x = if self.flip_h { 1.0 - x } else { x };+ let y = if self.flip_v { 1.0 - y } else { y };+ let [left, top, right, bottom] = self.crop;+ (+ left + x * (1.0 - left - right).max(0.0),+ top + y * (1.0 - top - bottom).max(0.0),+ )+ }++ /// The frame fraction a source fraction lands on, or `None` when the+ /// crop cut it away. The inverse of [`Mapping::source_of`], for a brush+ /// that paints in the source's terms and shows on the frame.+ pub fn frame_of(&self, u: f32, v: f32) -> Option<(f32, f32)> {+ let [left, top, right, bottom] = self.crop;+ let width = (1.0 - left - right).max(1e-6);+ let height = (1.0 - top - bottom).max(1e-6);+ let x = (u - left) / width;+ let y = (v - top) / height;+ if !(0.0..=1.0).contains(&x) || !(0.0..=1.0).contains(&y) {+ return None;+ }+ Some((+ if self.flip_h { 1.0 - x } else { x },+ if self.flip_v { 1.0 - y } else { y },+ ))+ }+}++impl Default for Mapping {+ fn default() -> Self {+ Self::IDENTITY+ }+}++/// Multiplies the frame's alpha by the mask, sampled through `mapping`.+/// The frame keeps its straight alpha; a pixel the mask calls background+/// becomes transparent and its colour is left for the compositor to weigh+/// by that alpha.+pub fn cut(frame: &mut Frame, mask: &Mask, mapping: &Mapping) {+ let width = frame.width() as usize;+ let height = frame.height() as usize;+ if width == 0 || height == 0 {+ return;+ }+ let pixels = frame.pixels_mut();+ let row_bytes = width * BYTES_PER_PIXEL;+ // Rows split across the machine's cores for a frame worth the threads;+ // a small preview frame is faster done on the one it arrived on.+ let workers = if width * height >= 400_000 {+ std::thread::available_parallelism()+ .map(|count| count.get())+ .unwrap_or(1)+ .clamp(1, 8)+ } else {+ 1+ };+ let rows_per = height.div_ceil(workers);+ std::thread::scope(|scope| {+ for (chunk, band) in pixels.chunks_mut(rows_per * row_bytes).enumerate() {+ let first_row = chunk * rows_per;+ scope.spawn(move || {+ for (row_index, row) in band.chunks_exact_mut(row_bytes).enumerate() {+ let y = (first_row + row_index) as f32 + 0.5;+ let fy = y / height as f32;+ for (x, pixel) in row.chunks_exact_mut(BYTES_PER_PIXEL).enumerate() {+ let fx = (x as f32 + 0.5) / width as f32;+ let (u, v) = mapping.source_of(fx, fy);+ let keep = mask.sample(u, v);+ pixel[3] = (f32::from(pixel[3]) * keep + 0.5) as u8;+ }+ }+ });+ }+ });+}++#[cfg(test)]+mod tests {+ use super::*;++ fn left_half_kept() -> Mask {+ let mut mask = Mask::filled(8, 8, 0);+ for y in 0..8 {+ for x in 0..4 {+ mask.bytes_mut()[y * 8 + x] = 255;+ }+ }+ mask+ }++ fn alpha(frame: &Frame, x: usize, y: usize) -> u8 {+ frame.pixels()[(y * frame.width() as usize + x) * BYTES_PER_PIXEL + 3]+ }++ #[test]+ fn the_background_half_goes_transparent() {+ let mut frame = Frame::black(16, 8);+ cut(&mut frame, &left_half_kept(), &Mapping::IDENTITY);+ assert_eq!(alpha(&frame, 2, 4), 255);+ assert_eq!(alpha(&frame, 13, 4), 0);+ // The colour is untouched: only the alpha is the mask's business.+ assert_eq!(frame.pixels()[(4 * 16 + 13) * 4], 0);+ }++ #[test]+ fn a_flip_looks_the_mask_up_mirrored() {+ let mut frame = Frame::black(16, 8);+ let mapping = Mapping {+ flip_h: true,+ ..Mapping::IDENTITY+ };+ cut(&mut frame, &left_half_kept(), &mapping);+ assert_eq!(alpha(&frame, 2, 4), 0);+ assert_eq!(alpha(&frame, 13, 4), 255);+ }++ #[test]+ fn a_crop_looks_the_mask_up_inside_the_cropped_source() {+ // The frame is the right two fifths of the source, all of which the+ // mask calls background.+ let mut frame = Frame::black(16, 8);+ let mapping = Mapping {+ crop: [0.6, 0.0, 0.0, 0.0],+ ..Mapping::IDENTITY+ };+ cut(&mut frame, &left_half_kept(), &mapping);+ assert_eq!(alpha(&frame, 1, 4), 0);+ assert_eq!(alpha(&frame, 14, 4), 0);+ }++ #[test]+ fn frame_of_inverts_source_of() {+ let mapping = Mapping {+ crop: [0.1, 0.2, 0.3, 0.0],+ flip_h: true,+ flip_v: false,+ };+ let (u, v) = mapping.source_of(0.25, 0.75);+ let (x, y) = mapping.frame_of(u, v).expect("inside the crop");+ assert!((x - 0.25).abs() < 1e-5 && (y - 0.75).abs() < 1e-5);+ assert!(mapping.frame_of(0.05, 0.5).is_none());+ }++ #[test]+ fn a_large_frame_is_cut_the_same_on_many_threads() {+ let mut big = Frame::black(1000, 500);+ cut(&mut big, &left_half_kept(), &Mapping::IDENTITY);+ assert_eq!(alpha(&big, 10, 250), 255);+ assert_eq!(alpha(&big, 990, 250), 0);+ assert_eq!(alpha(&big, 990, 499), 0);+ }+}diff --git a/engine/crates/concat-vision/src/lib.rs b/engine/crates/concat-vision/src/lib.rsnew file mode 100644index 00000000..8639744e--- /dev/null+++ b/engine/crates/concat-vision/src/lib.rs@@ -0,0 +1,58 @@+// SPDX-License-Identifier: AGPL-3.0-or-later+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors++//! What the engine sees in a picture.+//!+//! A cutout takes a picture's background away without a key colour: a+//! model says, pixel by pixel, how likely each one is to be the person, and+//! the picture's alpha is multiplied by the answer. This crate is the whole+//! of that, in four parts:+//!+//! - [`mask`]: the answer itself, an eight-bit probability picture, with+//! the sampling, softening and PNG form everything else uses.+//! - [`strokes`]: the corrections. A custom cutout paints brush strokes+//! over the model's mask; this is where a stroke becomes pixels.+//! - [`apply`]: the frame with its background gone. One function, called+//! by the exporter and the monitor alike, so the file and the screen+//! agree by construction.+//! - [`store`]: where masks live between runs. They are keyed by the media+//! file and the source instant, and cached in the project folder like+//! its waveforms, so a cutout is found once and travels with the edit.+//!+//! Finding the mask is [`segment`], behind the `infer` feature: the model+//! is compiled in and run by tract, in pure Rust, so a cutout needs no+//! download and no runtime library, on a desk or a phone. The renderer+//! reads masks and never infers; the host infers and writes them.+//!+//! Masks are square at the model's resolution whatever the picture's+//! shape, and every position in them is a fraction of the source picture:+//! `(0, 0)` its top-left, `(1, 1)` its bottom-right. Strokes are stored in+//! the same fractions. A crop, a flip or a change of output size therefore+//! changes nothing about a mask - the mapping from a decoded pixel back to+//! a source fraction is [`apply::Mapping`], and it is the one place those+//! are undone.++pub mod apply;+pub mod mask;+#[cfg(feature = "infer")]+pub mod segment;+pub mod store;+pub mod strokes;++pub use apply::{Mapping, cut};+pub use mask::Mask;+#[cfg(feature = "infer")]+pub use segment::Segmenter;+pub use store::{MaskStore, mask_dir};++/// Masks are found this many times a second of source. Ten is where a+/// person's outline stops visibly lagging their movement, and where a+/// minute of footage is six hundred inferences rather than eighteen hundred.+pub const MASK_RATE: u32 = 10;++/// The model's input and output edge, in pixels.+pub const MODEL_SIZE: u32 = 256;++/// Names the model a mask came from. Part of the cache directory's name,+/// so a different model never serves another's masks.+pub const MODEL_ID: &str = "selfie-256";diff --git a/engine/crates/concat-vision/src/mask.rs b/engine/crates/concat-vision/src/mask.rsnew file mode 100644index 00000000..97b7e098--- /dev/null+++ b/engine/crates/concat-vision/src/mask.rs@@ -0,0 +1,217 @@+// SPDX-License-Identifier: AGPL-3.0-or-later+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors++//! The probability picture: how much of each pixel is the subject.++use std::io::Cursor;++/// An eight-bit picture of probabilities, `0` background and `255` subject,+/// covering the whole source picture whatever its shape.+#[derive(Clone, PartialEq, Eq, Debug)]+pub struct Mask {+ width: u32,+ height: u32,+ data: Vec<u8>,+}++impl Mask {+ /// A mask of one value throughout.+ pub fn filled(width: u32, height: u32, value: u8) -> Mask {+ Mask {+ width: width.max(1),+ height: height.max(1),+ data: vec![value; width.max(1) as usize * height.max(1) as usize],+ }+ }++ /// Wraps a buffer of exactly `width * height` bytes.+ pub fn from_bytes(width: u32, height: u32, data: Vec<u8>) -> Option<Mask> {+ (width > 0 && height > 0 && data.len() == width as usize * height as usize).then_some(+ Mask {+ width,+ height,+ data,+ },+ )+ }++ /// Pixels across.+ pub fn width(&self) -> u32 {+ self.width+ }++ /// Pixels down.+ pub fn height(&self) -> u32 {+ self.height+ }++ /// The bytes, row-major from the top-left.+ pub fn bytes(&self) -> &[u8] {+ &self.data+ }++ /// The bytes, to write into.+ pub fn bytes_mut(&mut self) -> &mut [u8] {+ &mut self.data+ }++ /// The value at a pixel, clamped to the edge.+ pub fn at(&self, x: i64, y: i64) -> u8 {+ let x = x.clamp(0, i64::from(self.width) - 1) as usize;+ let y = y.clamp(0, i64::from(self.height) - 1) as usize;+ self.data[y * self.width as usize + x]+ }++ /// The value at a fraction of the picture, bilinear between the four+ /// pixels around it, in `0..=1`. Outside the picture is the edge.+ pub fn sample(&self, u: f32, v: f32) -> f32 {+ let x = u.clamp(0.0, 1.0) * self.width as f32 - 0.5;+ let y = v.clamp(0.0, 1.0) * self.height as f32 - 0.5;+ let x0 = x.floor();+ let y0 = y.floor();+ let fx = x - x0;+ let fy = y - y0;+ let (x0, y0) = (x0 as i64, y0 as i64);+ let top = f32::from(self.at(x0, y0)) * (1.0 - fx) + f32::from(self.at(x0 + 1, y0)) * fx;+ let bottom =+ f32::from(self.at(x0, y0 + 1)) * (1.0 - fx) + f32::from(self.at(x0 + 1, y0 + 1)) * fx;+ (top * (1.0 - fy) + bottom * fy) / 255.0+ }++ /// The mask softened: three box blurs of `radius` pixels, which is a+ /// gaussian to the eye at a fraction of the cost. Zero is the mask+ /// itself.+ pub fn blurred(&self, radius: f32) -> Mask {+ let radius = radius.round().max(0.0) as usize;+ if radius == 0 {+ return self.clone();+ }+ let mut out = self.clone();+ let mut scratch = vec![0u8; self.data.len()];+ for _ in 0..3 {+ blur_rows(&out.data, &mut scratch, self.width as usize, radius);+ blur_columns(+ &scratch,+ &mut out.data,+ self.width as usize,+ self.height as usize,+ radius,+ );+ }+ out+ }++ /// The mask as an eight-bit greyscale PNG.+ pub fn to_png(&self) -> Vec<u8> {+ let mut bytes = Vec::new();+ {+ let mut encoder = png::Encoder::new(&mut bytes, self.width, self.height);+ encoder.set_color(png::ColorType::Grayscale);+ encoder.set_depth(png::BitDepth::Eight);+ // A failed encode into a Vec is out of memory, and an empty+ // file is what the reader treats as a miss.+ if let Ok(mut writer) = encoder.write_header() {+ let _ = writer.write_image_data(&self.data);+ }+ }+ bytes+ }++ /// A mask read back from [`Mask::to_png`], or from any PNG whose first+ /// channel is the mask. `None` for anything that is not a PNG.+ pub fn from_png(bytes: &[u8]) -> Option<Mask> {+ let mut decoder = png::Decoder::new(Cursor::new(bytes));+ decoder.set_transformations(png::Transformations::normalize_to_color8());+ let mut reader = decoder.read_info().ok()?;+ let mut buffer = vec![0u8; reader.output_buffer_size()];+ let info = reader.next_frame(&mut buffer).ok()?;+ let channels = info.color_type.samples();+ let count = info.width as usize * info.height as usize;+ let data: Vec<u8> = buffer+ .chunks_exact(channels)+ .take(count)+ .map(|pixel| pixel[0])+ .collect();+ Mask::from_bytes(info.width, info.height, data)+ }+}++/// One horizontal box blur, every row.+fn blur_rows(source: &[u8], out: &mut [u8], width: usize, radius: usize) {+ let window = (2 * radius + 1) as u32;+ for (row_in, row_out) in source.chunks_exact(width).zip(out.chunks_exact_mut(width)) {+ let mut sum: u32 = 0;+ // The window starts centred on the first pixel, with the pixels+ // before the edge reading as the edge.+ for offset in 0..=radius {+ sum += u32::from(row_in[offset.min(width - 1)]);+ }+ sum += u32::from(row_in[0]) * radius as u32;+ for x in 0..width {+ row_out[x] = ((sum + window / 2) / window) as u8;+ let leaving = if x >= radius {+ row_in[x - radius]+ } else {+ row_in[0]+ };+ let arriving = row_in[(x + radius + 1).min(width - 1)];+ sum = sum + u32::from(arriving) - u32::from(leaving);+ }+ }+}++/// One vertical box blur, every column.+fn blur_columns(source: &[u8], out: &mut [u8], width: usize, height: usize, radius: usize) {+ let window = (2 * radius + 1) as u32;+ for x in 0..width {+ let at = |y: usize| source[y.min(height - 1) * width + x];+ let mut sum: u32 = 0;+ for offset in 0..=radius {+ sum += u32::from(at(offset));+ }+ sum += u32::from(at(0)) * radius as u32;+ for y in 0..height {+ out[y * width + x] = ((sum + window / 2) / window) as u8;+ let leaving = if y >= radius { at(y - radius) } else { at(0) };+ let arriving = at(y + radius + 1);+ sum = sum + u32::from(arriving) - u32::from(leaving);+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn sampling_reads_the_pixel_under_a_fraction() {+ let mut mask = Mask::filled(4, 2, 0);+ mask.bytes_mut()[3] = 255; // top-right+ assert!((mask.sample(0.875, 0.25) - 1.0).abs() < 1e-6);+ assert!(mask.sample(0.125, 0.25).abs() < 1e-6);+ // Halfway between two pixels is halfway between their values.+ assert!((mask.sample(0.75, 0.25) - 0.5).abs() < 1e-3);+ }++ #[test]+ fn a_blur_spreads_an_edge_and_keeps_the_mean() {+ let mut mask = Mask::filled(16, 1, 0);+ for x in 8..16 {+ mask.bytes_mut()[x] = 255;+ }+ let soft = mask.blurred(2.0);+ assert!(soft.at(7, 0) > 0 && soft.at(8, 0) < 255);+ assert!(soft.at(0, 0) == 0 && soft.at(15, 0) == 255);+ let sum: u32 = soft.bytes().iter().map(|&v| u32::from(v)).sum();+ assert!((sum as i64 - 8 * 255).abs() < 16 * 3);+ }++ #[test]+ fn a_mask_survives_the_png_round_trip() {+ let mut mask = Mask::filled(5, 3, 7);+ mask.bytes_mut()[6] = 200;+ let back = Mask::from_png(&mask.to_png()).expect("decodes");+ assert_eq!(back, mask);+ assert!(Mask::from_png(b"not a png").is_none());+ }+}diff --git a/engine/crates/concat-vision/src/segment.rs b/engine/crates/concat-vision/src/segment.rsnew file mode 100644index 00000000..95490db0--- /dev/null+++ b/engine/crates/concat-vision/src/segment.rs@@ -0,0 +1,108 @@+// SPDX-License-Identifier: AGPL-3.0-or-later+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors++//! Finding the person in a picture.+//!+//! The model is MediaPipe's selfie segmentation, compiled into the binary+//! (see `models/NOTICE.md`) and run by tract. It takes a 256 × 256 RGB+//! picture and answers with a probability per pixel; a source of any shape+//! is squashed to the square on the way in and the mask is read as covering+//! the whole source on the way out, which is the convention every other+//! part of this crate keeps.+//!+//! One inference is about twenty-five milliseconds on a laptop core, and a+//! loaded [`Segmenter`] is shared between threads, so a host that has+//! several cores analyses several frames at once against one model.++use concat_core::frame::{BYTES_PER_PIXEL, Frame};+use tract_onnx::prelude::*;++use crate::{MODEL_SIZE, Mask};++/// The model as the crate ships it.+const MODEL: &[u8] = include_bytes!("../models/selfie-segmentation.onnx");++type Plan = std::sync::Arc<TypedRunnableModel>;++/// A loaded model, ready to run. `Send + Sync`: one per process is enough.+pub struct Segmenter {+ plan: Plan,+}++impl Segmenter {+ /// Loads and optimises the compiled-in model. A few tens of+ /// milliseconds, once.+ pub fn load() -> Result<Segmenter, String> {+ let onnx = tract_onnx::onnx();+ let mut proto = onnx+ .proto_model_for_read(&mut std::io::Cursor::new(MODEL))+ .map_err(|error| format!("cutout model: {error}"))?;+ // The conversion annotates every tensor with a symbolic batch, and+ // the batch here is always one; the annotations would only argue.+ if let Some(graph) = proto.graph.as_mut() {+ graph.value_info.clear();+ }+ let size = MODEL_SIZE as usize;+ let plan = onnx+ .model_for_proto_model(&proto)+ .and_then(|model| model.with_input_fact(0, f32::fact([1, 3, size, size]).into()))+ .and_then(|model| model.into_optimized())+ .and_then(|model| model.into_runnable())+ .map_err(|error| format!("cutout model: {error}"))?;+ Ok(Segmenter { plan })+ }++ /// The person mask of `frame`, at the model's resolution. A frame of+ /// any size: it is resampled to the square first.+ pub fn mask(&self, frame: &Frame) -> Result<Mask, String> {+ let size = MODEL_SIZE as usize;+ let width = frame.width() as usize;+ let height = frame.height() as usize;+ if width == 0 || height == 0 {+ return Err("cutout: an empty frame".to_owned());+ }+ let pixels = frame.pixels();+ let input = tract_ndarray::Array4::from_shape_fn((1, 3, size, size), |(_, c, y, x)| {+ // Nearest sample: the decoder already scaled to the square in+ // the analysis path, and a still is fine at this resolution.+ let sx = (x * width / size).min(width - 1);+ let sy = (y * height / size).min(height - 1);+ f32::from(pixels[(sy * width + sx) * BYTES_PER_PIXEL + c]) / 255.0+ });+ let outputs = self+ .plan+ .run(tvec!(input.into_tensor().into()))+ .map_err(|error| format!("cutout: {error}"))?;+ let view = outputs+ .first()+ .ok_or_else(|| "cutout: the model answered nothing".to_owned())?+ .to_plain_array_view::<f32>()+ .map_err(|error| format!("cutout: {error}"))?;+ let data: Vec<u8> = view+ .iter()+ .take(size * size)+ .map(|&p: &f32| (p.clamp(0.0, 1.0) * 255.0 + 0.5) as u8)+ .collect();+ Mask::from_bytes(MODEL_SIZE, MODEL_SIZE, data)+ .ok_or_else(|| "cutout: the model's answer is the wrong size".to_owned())+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn the_model_loads_and_answers_a_mask_of_its_size() {+ let segmenter = Segmenter::load().expect("loads");+ // A flat grey picture is nobody: the mask should lean background.+ let mut frame = Frame::black(64, 48);+ for pixel in frame.pixels_mut().chunks_exact_mut(4) {+ pixel[..3].copy_from_slice(&[128, 128, 128]);+ }+ let mask = segmenter.mask(&frame).expect("runs");+ assert_eq!((mask.width(), mask.height()), (MODEL_SIZE, MODEL_SIZE));+ let mean = mask.bytes().iter().map(|&v| u32::from(v)).sum::<u32>() / (256 * 256);+ assert!(mean < 128, "a blank picture read as mostly person: {mean}");+ }+}diff --git a/engine/crates/concat-vision/src/store.rs b/engine/crates/concat-vision/src/store.rsnew file mode 100644index 00000000..2aa52eca--- /dev/null+++ b/engine/crates/concat-vision/src/store.rs@@ -0,0 +1,419 @@+// SPDX-License-Identifier: AGPL-3.0-or-later+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors++//! Where masks live between runs.+//!+//! One directory per media file per model, inside the project's `cache/`+//! folder so it travels with the project and vanishes with it, holding one+//! PNG per analysed source instant, named by that instant in milliseconds.+//! The host writes them as it analyses; the renderer opens the directory,+//! reads the names, and asks for the mask nearest a source time. Nothing+//! is ever indexed: the file names are the index.+//!+//! Decoded masks are kept in a process-wide cache, since a preview asks for+//! the same handful over and over while the playhead sits still, and an+//! export asks for each one several frames running.++use std::collections::{HashMap, VecDeque};+use std::path::{Path, PathBuf};+use std::sync::{Arc, Mutex, OnceLock};++use concat_project::model::{Cutout, CutoutMode};++use crate::{MASK_RATE, MODEL_ID, Mask, strokes};++/// A mask further than this from the instant asked for is nobody's answer:+/// two and a half analysis steps, so a gap in the cache reads as "not yet"+/// rather than as the wrong frame.+const REACH_MS: u64 = 2500 / MASK_RATE as u64;++/// How many decoded masks and how many resolved ones stay in memory. A+/// mask is 64 KB; both together stay under forty megabytes.+const CACHED: usize = 300;++/// The directory a media file's masks live in under `project`.+///+/// Named by a hash of the path rather than the path, like the peaks cache+/// and for the same reason: a flat, portable name that cannot escape the+/// folder. The model's id is in it too, so a different model never+/// answers for this one.+pub fn mask_dir(project: &Path, media_path: &str) -> PathBuf {+ project+ .join("cache")+ .join("masks")+ .join(format!("{:016x}-{MODEL_ID}", fnv1a(media_path.as_bytes())))+}++/// The file for the mask at `millis` of source.+pub fn mask_file(dir: &Path, millis: u64) -> PathBuf {+ dir.join(format!("{millis:09}.png"))+}++/// FNV-1a 64, for names that outlive the process.+fn fnv1a(bytes: &[u8]) -> u64 {+ let mut hash: u64 = 0xcbf2_9ce4_8422_2325;+ for &byte in bytes {+ hash ^= u64::from(byte);+ hash = hash.wrapping_mul(0x0000_0100_0000_01b3);+ }+ hash+}++/// One media file's masks, as found on disk when it was opened.+#[derive(Clone, Debug)]+pub struct MaskStore {+ dir: PathBuf,+ /// The analysed instants, ascending, in milliseconds of source.+ times: Vec<u64>,+}++impl MaskStore {+ /// Reads the directory's names. A directory that is not there is a+ /// store with nothing in it, not an error.+ pub fn open(dir: &Path) -> MaskStore {+ let mut times: Vec<u64> = std::fs::read_dir(dir)+ .into_iter()+ .flatten()+ .flatten()+ .filter_map(|entry| {+ let name = entry.file_name();+ let name = name.to_str()?;+ name.strip_suffix(".png")?.parse::<u64>().ok()+ })+ .collect();+ times.sort_unstable();+ times.dedup();+ MaskStore {+ dir: dir.to_path_buf(),+ times,+ }+ }++ /// Where the store is.+ pub fn dir(&self) -> &Path {+ &self.dir+ }++ /// How many instants are analysed.+ pub fn len(&self) -> usize {+ self.times.len()+ }++ /// True with nothing analysed.+ pub fn is_empty(&self) -> bool {+ self.times.is_empty()+ }++ /// The analysed instant nearest `seconds` of source, when one is within+ /// reach. A store of one mask answers for every instant: a still has+ /// one frame and that frame is the answer everywhere.+ pub fn nearest_millis(&self, seconds: f64) -> Option<u64> {+ if self.times.len() == 1 {+ return self.times.first().copied();+ }+ let wanted = (seconds.max(0.0) * 1000.0).round() as u64;+ let index = self.times.partition_point(|&at| at < wanted);+ let after = self.times.get(index).copied();+ let before = index+ .checked_sub(1)+ .and_then(|i| self.times.get(i).copied());+ [before, after]+ .into_iter()+ .flatten()+ .min_by_key(|&at| at.abs_diff(wanted))+ .filter(|&at| at.abs_diff(wanted) <= REACH_MS)+ }++ /// The model's mask nearest `seconds`, decoded, or `None` when that+ /// instant is not analysed yet or its file cannot be read.+ pub fn mask_at(&self, seconds: f64) -> Option<Arc<Mask>> {+ let millis = self.nearest_millis(seconds)?;+ load(&mask_file(&self.dir, millis))+ }++ /// The mask to cut with at `seconds`: the model's, with the cutout's+ /// strokes painted on when it is custom, and softened by its feather.+ /// `aspect` is the source's width over its height, for round brushes.+ pub fn resolved(&self, seconds: f64, cutout: &Cutout, aspect: f32) -> Option<Arc<Mask>> {+ let millis = self.nearest_millis(seconds)?;+ let file = mask_file(&self.dir, millis);+ let key = ResolvedKey {+ file: file.clone(),+ settings: settings_key(cutout, aspect),+ };+ if let Some(hit) = resolved_cache()+ .lock()+ .ok()+ .and_then(|cache| cache.get(&key))+ {+ return Some(hit);+ }+ let auto = load(&file)?;+ let painted = match cutout.mode {+ CutoutMode::Custom if !cutout.strokes.is_empty() => {+ strokes::paint(&auto, &cutout.strokes, aspect)+ }+ _ => (*auto).clone(),+ };+ let radius = cutout.feather as f32 * painted.width() as f32;+ let mask = Arc::new(painted.blurred(radius));+ if let Ok(mut cache) = resolved_cache().lock() {+ cache.put(key, Arc::clone(&mask));+ }+ Some(mask)+ }++ /// The source instants between `from` and `to` seconds, on the analysis+ /// grid, that have no mask yet. Empty when the range is covered.+ pub fn missing(&self, from: f64, to: f64) -> Vec<u64> {+ let step = 1000 / u64::from(MASK_RATE);+ let first = ((from.max(0.0) * 1000.0) as u64) / step * step;+ let last = (to.max(from).max(0.0) * 1000.0).ceil() as u64;+ let mut out = Vec::new();+ let mut at = first;+ while at <= last {+ let index = self.times.partition_point(|&t| t < at);+ let hit = self.times.get(index).copied().is_some_and(|t| t == at);+ if !hit {+ out.push(at);+ }+ at += step;+ }+ out+ }++ /// Writes a mask for `millis` and remembers it. Best effort on disk;+ /// in memory it is there either way, for the run that made it.+ pub fn put(&mut self, millis: u64, mask: &Mask) -> Result<(), String> {+ std::fs::create_dir_all(&self.dir)+ .map_err(|error| format!("could not create {}: {error}", self.dir.display()))?;+ let file = mask_file(&self.dir, millis);+ std::fs::write(&file, mask.to_png())+ .map_err(|error| format!("could not write {}: {error}", file.display()))?;+ if let Err(index) = self.times.binary_search(&millis) {+ self.times.insert(index, millis);+ }+ if let Ok(mut cache) = mask_cache().lock() {+ cache.put(file, Arc::new(mask.clone()));+ }+ Ok(())+ }++ /// Forgets every mask on disk and in memory, for a media file whose+ /// masks should be found again.+ pub fn clear(&mut self) {+ for &millis in &self.times {+ let file = mask_file(&self.dir, millis);+ let _ = std::fs::remove_file(&file);+ if let Ok(mut cache) = mask_cache().lock() {+ cache.remove(&file);+ }+ }+ if let Ok(mut cache) = resolved_cache().lock() {+ cache.retain(|key| !key.file.starts_with(&self.dir));+ }+ self.times.clear();+ let _ = std::fs::remove_dir(&self.dir);+ }+}++/// What besides the file decides a resolved mask: the mode, the feather+/// and every stroke, hashed, with the aspect the brushes were sized by.+fn settings_key(cutout: &Cutout, aspect: f32) -> u64 {+ use std::hash::{Hash, Hasher};+ let mut hasher = std::collections::hash_map::DefaultHasher::new();+ (cutout.mode == CutoutMode::Custom).hash(&mut hasher);+ cutout.feather.to_bits().hash(&mut hasher);+ aspect.to_bits().hash(&mut hasher);+ if cutout.mode == CutoutMode::Custom {+ for stroke in &cutout.strokes {+ (stroke.tool as u8).hash(&mut hasher);+ stroke.size.to_bits().hash(&mut hasher);+ for [x, y] in &stroke.points {+ x.to_bits().hash(&mut hasher);+ y.to_bits().hash(&mut hasher);+ }+ }+ }+ hasher.finish()+}++#[derive(Clone, PartialEq, Eq, Hash)]+struct ResolvedKey {+ file: PathBuf,+ settings: u64,+}++/// A bounded map that forgets its oldest entry first.+struct Recent<K, V> {+ map: HashMap<K, V>,+ order: VecDeque<K>,+}++impl<K: Clone + Eq + std::hash::Hash, V: Clone> Recent<K, V> {+ fn new() -> Self {+ Self {+ map: HashMap::new(),+ order: VecDeque::new(),+ }+ }++ fn get(&self, key: &K) -> Option<V> {+ self.map.get(key).cloned()+ }++ fn put(&mut self, key: K, value: V) {+ if self.map.insert(key.clone(), value).is_none() {+ self.order.push_back(key);+ }+ while self.order.len() > CACHED {+ if let Some(oldest) = self.order.pop_front() {+ self.map.remove(&oldest);+ }+ }+ }++ fn remove(&mut self, key: &K) {+ if self.map.remove(key).is_some() {+ self.order.retain(|held| held != key);+ }+ }++ fn retain(&mut self, keep: impl Fn(&K) -> bool) {+ self.map.retain(|key, _| keep(key));+ self.order.retain(|key| keep(key));+ }+}++fn mask_cache() -> &'static Mutex<Recent<PathBuf, Arc<Mask>>> {+ static CACHE: OnceLock<Mutex<Recent<PathBuf, Arc<Mask>>>> = OnceLock::new();+ CACHE.get_or_init(|| Mutex::new(Recent::new()))+}++fn resolved_cache() -> &'static Mutex<Recent<ResolvedKey, Arc<Mask>>> {+ static CACHE: OnceLock<Mutex<Recent<ResolvedKey, Arc<Mask>>>> = OnceLock::new();+ CACHE.get_or_init(|| Mutex::new(Recent::new()))+}++/// The mask in `file`, from memory when it has been read before.+fn load(file: &Path) -> Option<Arc<Mask>> {+ if let Some(hit) = mask_cache()+ .lock()+ .ok()+ .and_then(|cache| cache.get(&file.to_path_buf()))+ {+ return Some(hit);+ }+ let bytes = std::fs::read(file).ok()?;+ let mask = Arc::new(Mask::from_png(&bytes)?);+ if let Ok(mut cache) = mask_cache().lock() {+ cache.put(file.to_path_buf(), Arc::clone(&mask));+ }+ Some(mask)+}++#[cfg(test)]+mod tests {+ use super::*;+ use concat_project::model::{BrushTool, Stroke};++ fn scratch(name: &str) -> PathBuf {+ let dir = std::env::temp_dir().join(format!(+ "concat-vision-{name}-{}-{}",+ std::process::id(),+ std::time::SystemTime::now()+ .duration_since(std::time::UNIX_EPOCH)+ .map(|d| d.as_nanos())+ .unwrap_or(0)+ ));+ let _ = std::fs::remove_dir_all(&dir);+ dir+ }++ #[test]+ fn the_directory_is_named_by_the_file_and_the_model() {+ let a = mask_dir(Path::new("/p"), "/footage/a.mp4");+ let b = mask_dir(Path::new("/p"), "/footage/b.mp4");+ assert_ne!(a, b);+ assert!(a.starts_with("/p/cache/masks"));+ assert!(a.to_string_lossy().ends_with(MODEL_ID));+ }++ #[test]+ fn masks_are_written_listed_and_found_by_nearest_instant() {+ let dir = scratch("store");+ let mut store = MaskStore::open(&dir);+ assert!(store.is_empty());+ store.put(0, &Mask::filled(4, 4, 10)).expect("writes");+ store.put(100, &Mask::filled(4, 4, 20)).expect("writes");+ store.put(200, &Mask::filled(4, 4, 30)).expect("writes");++ let reopened = MaskStore::open(&dir);+ assert_eq!(reopened.len(), 3);+ assert_eq!(reopened.nearest_millis(0.14), Some(100));+ assert_eq!(reopened.nearest_millis(0.16), Some(200));+ // Beyond reach of every analysed instant: not yet.+ assert_eq!(reopened.nearest_millis(9.0), None);+ assert_eq!(reopened.mask_at(0.21).map(|m| m.at(0, 0)), Some(30));++ assert_eq!(reopened.missing(0.0, 0.2), Vec::<u64>::new());+ assert_eq!(reopened.missing(0.0, 0.45), vec![300, 400]);+ let _ = std::fs::remove_dir_all(&dir);+ }++ #[test]+ fn a_single_mask_answers_for_every_instant() {+ let dir = scratch("still");+ let mut store = MaskStore::open(&dir);+ store.put(0, &Mask::filled(4, 4, 200)).expect("writes");+ assert_eq!(store.nearest_millis(37.5), Some(0));+ let _ = std::fs::remove_dir_all(&dir);+ }++ #[test]+ fn a_resolved_mask_carries_the_strokes_and_the_feather() {+ let dir = scratch("resolved");+ let mut store = MaskStore::open(&dir);+ store.put(0, &Mask::filled(16, 16, 0)).expect("writes");+ let plain = Cutout::auto();+ assert_eq!(+ store.resolved(0.0, &plain, 1.0).map(|m| m.at(8, 8)),+ Some(0)+ );+ let custom = Cutout {+ mode: CutoutMode::Custom,+ feather: 0.0,+ strokes: vec![Stroke {+ tool: BrushTool::Brush,+ size: 0.5,+ points: vec![[0.5, 0.5]],+ }],+ };+ assert_eq!(+ store.resolved(0.0, &custom, 1.0).map(|m| m.at(8, 8)),+ Some(255)+ );+ // Automatic ignores the strokes even when they are still stored.+ let back_to_auto = Cutout {+ mode: CutoutMode::Auto,+ ..custom.clone()+ };+ assert_eq!(+ store.resolved(0.0, &back_to_auto, 1.0).map(|m| m.at(8, 8)),+ Some(0)+ );+ let _ = std::fs::remove_dir_all(&dir);+ }++ #[test]+ fn clearing_forgets_disk_and_memory() {+ let dir = scratch("clear");+ let mut store = MaskStore::open(&dir);+ store.put(0, &Mask::filled(4, 4, 1)).expect("writes");+ assert!(store.mask_at(0.0).is_some());+ store.clear();+ assert!(store.is_empty());+ assert!(!dir.exists());+ assert!(MaskStore::open(&dir).mask_at(0.0).is_none());+ }+}diff --git a/engine/crates/concat-vision/src/strokes.rs b/engine/crates/concat-vision/src/strokes.rsnew file mode 100644index 00000000..679c045b--- /dev/null+++ b/engine/crates/concat-vision/src/strokes.rs@@ -0,0 +1,175 @@+// SPDX-License-Identifier: AGPL-3.0-or-later+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors++//! Brush strokes over a mask.+//!+//! A custom cutout is the model's mask with corrections painted on it.+//! Each stroke is a path of points in source fractions and a diameter as a+//! fraction of the source's width; here it becomes discs along the path,+//! and each disc changes the pixels beneath it the way its tool says:+//!+//! - **Brush** keeps everything under it, **Eraser** removes everything.+//! - **Smart brush** keeps only what the model gave some chance of being+//! the subject - a sleeve it was unsure of comes back, the wall behind+//! does not. **Smart eraser** removes only what the model was not sure+//! of - a bit of chair it half-kept goes, the shoulder beside it stays.+//!+//! The mask is square whatever the picture's shape, so a round brush is an+//! ellipse in mask pixels; `aspect` (width over height of the source) is+//! what makes it round again on screen.++use concat_project::model::{BrushTool, Stroke};++use crate::Mask;++/// Below this much confidence a smart brush leaves a pixel alone.+const SMART_KEEP: u8 = 38;+/// Above this much confidence a smart eraser leaves a pixel alone.+const SMART_DROP: u8 = 217;++/// The model's mask with `strokes` painted over it, in order. `aspect` is+/// the source's width over its height.+pub fn paint(auto: &Mask, strokes: &[Stroke], aspect: f32) -> Mask {+ let mut out = auto.clone();+ for stroke in strokes {+ paint_one(&mut out, auto, stroke, aspect);+ }+ out+}++fn paint_one(out: &mut Mask, auto: &Mask, stroke: &Stroke, aspect: f32) {+ let width = out.width() as f32;+ let height = out.height() as f32;+ let aspect = if aspect.is_finite() && aspect > 0.0 {+ aspect+ } else {+ 1.0+ };+ // The brush is `size` of the source's width across, and as tall as it+ // is wide on screen: in mask pixels that is an ellipse.+ let rx = (stroke.size as f32 * width / 2.0).max(0.5);+ let ry = (stroke.size as f32 * aspect * height / 2.0).max(0.5);+ let points: Vec<(f32, f32)> = stroke+ .points+ .iter()+ .map(|[x, y]| (*x as f32 * width, *y as f32 * height))+ .collect();+ let Some(&first) = points.first() else {+ return;+ };+ disc(out, auto, stroke.tool, first, rx, ry);+ for pair in points.windows(2) {+ let (from, to) = (pair[0], pair[1]);+ let distance = ((to.0 - from.0).powi(2) + (to.1 - from.1).powi(2)).sqrt();+ // A disc every quarter radius, so the path has no scallops.+ let steps = (distance / (rx.min(ry) * 0.25)).ceil().max(1.0) as usize;+ for step in 1..=steps {+ let t = step as f32 / steps as f32;+ let centre = (from.0 + (to.0 - from.0) * t, from.1 + (to.1 - from.1) * t);+ disc(out, auto, stroke.tool, centre, rx, ry);+ }+ }+}++fn disc(out: &mut Mask, auto: &Mask, tool: BrushTool, centre: (f32, f32), rx: f32, ry: f32) {+ let width = out.width() as i64;+ let height = out.height() as i64;+ let x0 = ((centre.0 - rx).floor() as i64).max(0);+ let x1 = ((centre.0 + rx).ceil() as i64).min(width - 1);+ let y0 = ((centre.1 - ry).floor() as i64).max(0);+ let y1 = ((centre.1 + ry).ceil() as i64).min(height - 1);+ if x0 > x1 || y0 > y1 {+ return;+ }+ let stride = out.width() as usize;+ let data = out.bytes_mut();+ for y in y0..=y1 {+ let dy = (y as f32 + 0.5 - centre.1) / ry;+ for x in x0..=x1 {+ let dx = (x as f32 + 0.5 - centre.0) / rx;+ if dx * dx + dy * dy > 1.0 {+ continue;+ }+ let index = y as usize * stride + x as usize;+ let was = auto.bytes()[index];+ match tool {+ BrushTool::Brush => data[index] = 255,+ BrushTool::Eraser => data[index] = 0,+ BrushTool::SmartBrush if was >= SMART_KEEP => data[index] = 255,+ BrushTool::SmartEraser if was <= SMART_DROP => data[index] = 0,+ BrushTool::SmartBrush | BrushTool::SmartEraser => {}+ }+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn stroke(tool: BrushTool, size: f64, points: &[[f64; 2]]) -> Stroke {+ Stroke {+ tool,+ size,+ points: points.to_vec(),+ }+ }++ #[test]+ fn a_brush_keeps_and_an_eraser_removes_under_the_path() {+ let auto = Mask::filled(32, 32, 0);+ let kept = paint(+ &auto,+ &[stroke(BrushTool::Brush, 0.25, &[[0.25, 0.5], [0.75, 0.5]])],+ 1.0,+ );+ assert_eq!(kept.at(16, 16), 255);+ assert_eq!(kept.at(16, 2), 0);+ assert_eq!(kept.at(2, 2), 0);++ let gone = paint(+ &kept,+ &[stroke(BrushTool::Eraser, 0.25, &[[0.5, 0.5]])],+ 1.0,+ );+ assert_eq!(gone.at(16, 16), 0);+ assert_eq!(gone.at(8, 16), 255);+ }++ #[test]+ fn smart_tools_follow_the_models_confidence() {+ let mut auto = Mask::filled(8, 8, 0);+ auto.bytes_mut()[0] = 100; // unsure: rescued by a smart brush+ auto.bytes_mut()[1] = 0; // background: left alone+ auto.bytes_mut()[2] = 250; // sure: kept by a smart eraser+ let whole = [stroke(BrushTool::SmartBrush, 2.0, &[[0.5, 0.5]])];+ let brushed = paint(&auto, &whole, 1.0);+ assert_eq!(brushed.at(0, 0), 255);+ assert_eq!(brushed.at(1, 0), 0);+ let erased = paint(+ &auto,+ &[+ whole[0].clone(),+ stroke(BrushTool::SmartEraser, 2.0, &[[0.5, 0.5]]),+ ],+ 1.0,+ );+ // The eraser judges by the model's confidence, not by what an+ // earlier stroke did: the rescued pixel was unsure, so it goes; the+ // sure one the brush also covered stays as the brush left it.+ assert_eq!(erased.at(0, 0), 0);+ assert_eq!(erased.at(2, 0), 255);+ }++ #[test]+ fn the_brush_is_round_on_a_wide_picture() {+ // A 2:1 source squashed into a square mask: a round brush must reach+ // twice as far down the mask as across it.+ let auto = Mask::filled(64, 64, 0);+ let out = paint(&auto, &[stroke(BrushTool::Brush, 0.25, &[[0.5, 0.5]])], 2.0);+ assert_eq!(out.at(32 + 7, 32), 255);+ assert_eq!(out.at(32 + 9, 32), 0);+ assert_eq!(out.at(32, 32 + 15), 255);+ assert_eq!(out.at(32, 32 + 17), 0);+ }+}diff --git a/engine/crates/concat/Cargo.toml b/engine/crates/concat/Cargo.tomlindex 6e3499f7..114bfb7f 100644--- a/engine/crates/concat/Cargo.toml+++ b/engine/crates/concat/Cargo.toml@@ -93,6 +93,9 @@ concat-host = { workspace = true, features = ["gpu"] }concat-speech.workspace = true# The effect catalogue: the inspector's shelves and each link's knobs.concat-effects.workspace = true+# The cutout's mapping from a stage press to a source fraction; the model+# itself runs in the host.+concat-vision.workspace = true# Remembered preferences, as a small JSON file in the config directory.serde = { version = "1", features = ["derive"] }diff --git a/engine/crates/concat/src/host.rs b/engine/crates/concat/src/host.rsindex a893bf28..c463355d 100644--- a/engine/crates/concat/src/host.rs+++ b/engine/crates/concat/src/host.rs@@ -41,6 +41,8 @@ pub struct Host {pub speech: Arc<Speech>,/// Titles painted to pictures, and the cache of them.pub titles: concat_host::Titles,+ /// The cutout model, and the masks it finds for the project's media.+ pub cutouts: Arc<concat_host::Cutouts>,}impl Host {@@ -62,6 +64,7 @@ impl Host {exporter: Exporter::new(),transcriber: Arc::new(Transcriber::new()),speech: Arc::new(Speech::new()),+ cutouts: Arc::new(concat_host::Cutouts::new()),})}}diff --git a/engine/crates/concat/src/lib.rs b/engine/crates/concat/src/lib.rsindex 7ec53797..dae85875 100644--- a/engine/crates/concat/src/lib.rs+++ b/engine/crates/concat/src/lib.rs@@ -678,6 +678,23 @@ pub fn run() -> Result<(), slint::PlatformError> {state.stage_released();}));+ // ── the cutout ──+ editor.on_cutout_mode(on_window!(|state, mode: i32| {+ state.cutout_mode(mode);+ }));+ editor.on_cutout_tool(on_window!(|state, index: i32| {+ state.cutout_tool(index);+ }));+ editor.on_cutout_size(on_lanes!(|state, size: f32| {+ state.cutout_size(size);+ }));+ editor.on_cutout_painting(on_window!(|state, on: bool| {+ state.cutout_painting(on);+ }));+ editor.on_cutout_clear(on_window!(|state| {+ state.cutout_clear();+ }));+// ── the context menu ──editor.on_clip_context(on_window!(|state, id: SharedString| {state.menu_token += 1;diff --git a/engine/crates/concat/src/studio.rs b/engine/crates/concat/src/studio.rsindex 186d2f49..065a0651 100644--- a/engine/crates/concat/src/studio.rs+++ b/engine/crates/concat/src/studio.rs@@ -28,7 +28,7 @@ use concat_effects::manifest::Kind as PackageKind;use concat_host::export::{self, ExportSpec};use concat_host::playback::ClipSpec;use concat_host::preview::FrameSpec;-use concat_host::{ProjectInfo, Session, media, projects, templates};+use concat_host::{AnalyseRequest, Cutouts, ProjectInfo, Session, media, projects, templates};use concat_media::Peaks;use concat_project::commands::{ClipMove, ClipPatch, TrackFlag, TrimEdge};use concat_project::model::{@@ -396,8 +396,26 @@ pub enum Gesture {centre: (f64, f64),from: f64,},+ /// A brush on the stage: the stroke so far, in source fractions for+ /// the command it becomes and in stage fractions for the line drawn+ /// under the pointer meanwhile.+ Paint {+ clip: String,+ tool: model::BrushTool,+ size: f64,+ points: Vec<[f64; 2]>,+ screen: Vec<(f32, f32)>,+ },}+/// The custom cutout's brushes, in the inspector's order.+pub const BRUSHES: [model::BrushTool; 4] = [+ model::BrushTool::SmartBrush,+ model::BrushTool::Brush,+ model::BrushTool::SmartEraser,+ model::BrushTool::Eraser,+];+/// Where a picture was when a stage move began.pub struct StageOrigin {pub clip: String,@@ -666,6 +684,17 @@ pub struct Studio {pub speakers: Vec<concat_speech::tts::VoiceInfo>,/// The looks the Text page offers; see `presets`.pub text_presets: Vec<TextPreset>,++ // ── cutouts ──+ /// The brush a custom cutout paints with: a row of `BRUSHES`.+ pub brush: usize,+ /// The brush's diameter, as a fraction of the picture's width.+ pub brush_size: f64,+ /// Whether a press on the stage paints the selected custom cutout+ /// rather than moving the picture.+ pub painting: bool,+ /// The mask analyses running, by media id, and how far each has got.+ cutout_jobs: HashMap<String, f32>,}// ── conversions between the document and the window ─────────────────────@@ -770,6 +799,7 @@ fn commit_key(commands: &[Command]) -> String {.iter().map(|command| match command {Command::SetClipTransform { .. } => "transform".to_owned(),+ Command::SetClipCutout { .. } => "cutout".to_owned(),Command::SetClipSpeed { .. } => "speed".to_owned(),Command::SetClipSpeedCurve { .. } => "curve".to_owned(),Command::SetClipAnimation { slot, .. } => format!("animation:{slot:?}"),@@ -1031,6 +1061,10 @@ impl Studio {speech: SpeechSheet::default(),speakers: Vec::new(),text_presets,+ brush: 0,+ brush_size: 0.06,+ painting: true,+ cutout_jobs: HashMap::new(),host,};studio.settings.language = studio.prefs.language.unwrap_or(0);@@ -1222,6 +1256,7 @@ impl Studio {self.sync_audio();self.request_media_art();self.request_preview();+ self.ensure_cutouts();}pub fn undo(&mut self) {@@ -2306,7 +2341,8 @@ impl Studio {| Gesture::StageMove { .. }| Gesture::StageScale { .. }| Gesture::StageRotate { .. }- | Gesture::StageStretch { .. } => {}+ | Gesture::StageStretch { .. }+ | Gesture::Paint { .. } => {}}self.gesture = gesture;}@@ -2367,7 +2403,8 @@ impl Studio {other @ (Gesture::StageMove { .. }| Gesture::StageScale { .. }| Gesture::StageRotate { .. }- | Gesture::StageStretch { .. }) => {+ | Gesture::StageStretch { .. }+ | Gesture::Paint { .. }) => {self.gesture = other;}}@@ -2395,6 +2432,10 @@ impl Studio {ClipField::OffsetY => clip.offset_y = value.clamp(-1.0, 1.0),ClipField::Rotation => clip.rotation = value.clamp(-180.0, 180.0),ClipField::Opacity => clip.opacity = value.clamp(0.0, 1.0),+ ClipField::CutoutFeather => {+ clip.cutout.get_or_insert_with(model::Cutout::auto).feather =+ value.clamp(0.0, model::MAX_FEATHER);+ }ClipField::Volume => clip.volume = value.max(0.0),ClipField::Speed => {let speed = value.clamp(0.0625, 16.0);@@ -2586,6 +2627,12 @@ impl Studio {stretch_y: Some(after.stretch_y),});}+ if after.cutout != before.cutout {+ commands.push(Command::SetClipCutout {+ clip_id: id.clone(),+ cutout: after.cutout.clone(),+ });+ }if after.speed_curve != before.speed_curve {commands.push(Command::SetClipSpeedCurve {clip_id: id.clone(),@@ -2865,6 +2912,17 @@ impl Studio {/// of everything selected that is under the playhead. Empty stage/// clears the selection, as an empty lane does.pub fn stage_pressed(&mut self, x: f32, y: f32, additive: bool) {+ if let Some(clip) = self.paint_target() {+ let point = self.stage_to_source(&clip, f64::from(x), f64::from(y));+ self.gesture = Gesture::Paint {+ clip: clip.id.clone(),+ tool: BRUSHES[self.brush.min(BRUSHES.len() - 1)],+ size: self.brush_size,+ points: vec![point],+ screen: vec![(x, y)],+ };+ return;+ }let (x, y) = (f64::from(x), f64::from(y));let Some(id) = self.stage_hit(x, y) else {if !additive {@@ -2988,9 +3046,23 @@ impl Studio {/// The pointer moved with a stage gesture live: the echo follows, and/// the monitor composites it.pub fn stage_dragged(&mut self, x: f32, y: f32, snap: bool) {+ let mut gesture = std::mem::replace(&mut self.gesture, Gesture::None);+ if let Gesture::Paint {+ clip,+ points,+ screen,+ ..+ } = &mut gesture+ {+ if let Some(clip) = self.clip(clip).cloned() {+ points.push(self.stage_to_source(&clip, f64::from(x), f64::from(y)));+ screen.push((x, y));+ }+ self.gesture = gesture;+ return;+ }let (x, y) = (f64::from(x), f64::from(y));let (width, height) = self.output_size();- let gesture = std::mem::replace(&mut self.gesture, Gesture::None);match &gesture {Gesture::StageMove {primary,@@ -3201,6 +3273,20 @@ impl Studio {Gesture::StageScale { clip, .. }| Gesture::StageRotate { clip, .. }| Gesture::StageStretch { clip, .. } => vec![clip],+ Gesture::Paint {+ clip,+ tool,+ size,+ points,+ ..+ } => {+ // The stroke becomes one command, and one undo step.+ self.apply(Command::AddCutoutStroke {+ clip_id: clip,+ stroke: model::Stroke { tool, size, points },+ });+ return;+ }other => {// Not ours to end; a lane gesture is still live.self.gesture = other;@@ -3253,6 +3339,250 @@ impl Studio {}}+ // ── cutouts ──+ //+ // A cutout is a mask per source instant, found by the host's model and+ // cached in the project folder. The window's part is to notice which+ // media need masks they do not have, run one analysis at a time, and+ // turn a brush on the stage into strokes on the document.++ /// Starts the analysis for the first media whose cutout wants masks+ /// that are not there, unless one is running. Called after every+ /// change; the finished job calls it again for whatever is next.+ pub fn ensure_cutouts(&mut self) {+ if !self.cutout_jobs.is_empty() || self.host.cutouts.is_busy() {+ return;+ }+ let Some(session) = self.session.as_ref() else {+ return;+ };+ let project = std::path::PathBuf::from(session.path());+ let mut wanted: Vec<(String, AnalyseRequest)> = Vec::new();+ for clip in &self.timeline().clips {+ if clip.cutout.is_none() || !clip.kind.is_visual() {+ continue;+ }+ let Some(media) = self.project().media_by_id(&clip.media_id) else {+ continue;+ };+ // The source the clip shows: its in-point, for as long as it+ // runs at its speed. A curve's mean is its speed, so this+ // covers a curved clip too.+ let range = (+ clip.source_start,+ clip.source_start + clip.duration * clip.speed.max(0.0625),+ );+ match wanted.iter_mut().find(|(id, _)| *id == media.id) {+ Some((_, request)) => request.ranges.push(range),+ None => wanted.push((+ media.id.clone(),+ AnalyseRequest {+ project: project.clone(),+ media_path: media.path.clone(),+ still: media.kind == model::MediaKind::Image,+ ranges: vec![range],+ },+ )),+ }+ }+ let Some((id, request)) = wanted+ .into_iter()+ .find(|(_, request)| Cutouts::outstanding(request) > 0)+ else {+ return;+ };+ self.cutout_jobs.insert(id.clone(), 0.0);+ let cutouts = Arc::clone(&self.host.cutouts);+ spawn(+ move || {+ let mut last = -1.0f32;+ let reporting = id.clone();+ let result = cutouts.analyse(&request, &mut |progress| {+ // Every percent, not every frame: the readout cannot+ // use more and the event loop has other work.+ if progress - last >= 0.01 {+ last = progress;+ let id = reporting.clone();+ on_ui(move |studio, _, _| {+ if let Some(held) = studio.cutout_jobs.get_mut(&id) {+ *held = progress;+ }+ });+ }+ });+ (id, result)+ },+ |studio, _, _, (id, result)| {+ studio.cutout_jobs.remove(&id);+ match result {+ Ok(_) => {+ // The monitor shows the cut, and whatever else is+ // waiting gets its turn.+ studio.request_preview();+ studio.ensure_cutouts();+ }+ Err(error) if error.contains("cancelled") => {}+ Err(error) => studio.notify(&format!("Remove background: {error}"), true),+ }+ },+ );+ }++ /// The Mode row: 0 off, 1 automatic, 2 custom. The chroma rows are the+ /// inspector's own business, on the chain.+ pub fn cutout_mode(&mut self, mode: i32) {+ let Some(id) = self.sole_selection() else {+ return;+ };+ let Some(clip) = self.clip(&id).cloned() else {+ return;+ };+ let held = clip.cutout.clone().unwrap_or_else(model::Cutout::auto);+ let cutout = match mode {+ 1 => Some(model::Cutout {+ mode: model::CutoutMode::Auto,+ ..held+ }),+ 2 => Some(model::Cutout {+ mode: model::CutoutMode::Custom,+ ..held+ }),+ _ => None,+ };+ if mode == 2 {+ self.painting = true;+ }+ if cutout != clip.cutout {+ self.apply(Command::SetClipCutout {+ clip_id: id,+ cutout,+ });+ }+ }++ /// Takes every stroke off the selected clip's cutout, keeping it custom.+ pub fn cutout_clear(&mut self) {+ let Some(id) = self.sole_selection() else {+ return;+ };+ let Some(cutout) = self.clip(&id).and_then(|clip| clip.cutout.clone()) else {+ return;+ };+ if cutout.strokes.is_empty() {+ return;+ }+ self.apply(Command::SetClipCutout {+ clip_id: id,+ cutout: Some(model::Cutout {+ strokes: Vec::new(),+ ..cutout+ }),+ });+ }++ pub fn cutout_tool(&mut self, index: i32) {+ self.brush = index.clamp(0, BRUSHES.len() as i32 - 1) as usize;+ }++ pub fn cutout_size(&mut self, size: f32) {+ self.brush_size = f64::from(size).clamp(model::MIN_BRUSH, model::MAX_BRUSH);+ }++ pub fn cutout_painting(&mut self, on: bool) {+ self.painting = on;+ }++ /// The picture a press on the stage would paint: the one selected clip,+ /// under the playhead, with a custom cutout, while painting is on.+ fn paint_target(&self) -> Option<Clip> {+ if !self.painting {+ return None;+ }+ let id = self.sole_selection()?;+ let clip = self.clip(&id)?;+ let custom = clip+ .cutout+ .as_ref()+ .is_some_and(|cutout| cutout.mode == model::CutoutMode::Custom);+ if !custom || !clip.kind.is_visual() || self.locked(&clip.track_id) {+ return None;+ }+ self.stage_clips()+ .into_iter()+ .any(|shown| shown.id == clip.id)+ .then(|| clip.clone())+ }++ /// Where a stage point lands in the source picture, in the fractions a+ /// stroke is stored in: the footprint's turn undone, then the crop and+ /// the flips, the same way a decoded pixel finds its mask.+ fn stage_to_source(&self, clip: &Clip, x: f64, y: f64) -> [f64; 2] {+ let footprint = self.footprint(clip);+ let (width, height) = self.output_size();+ let (width, height) = (f64::from(width.max(1)), f64::from(height.max(1)));+ let dx = (x - footprint.cx) * width;+ let dy = (y - footprint.cy) * height;+ let (sin, cos) = footprint.rotation.to_radians().sin_cos();+ let along = dx * cos + dy * sin;+ let down = -dx * sin + dy * cos;+ let px = along / (footprint.w * width).max(1e-6) + 0.5;+ let py = down / (footprint.h * height).max(1e-6) + 0.5;+ let mapping = concat_vision::Mapping {+ crop: clip+ .crop+ .map(|crop| {+ [+ crop.left as f32,+ crop.top as f32,+ crop.right as f32,+ crop.bottom as f32,+ ]+ })+ .unwrap_or([0.0; 4]),+ flip_h: clip.flip_h,+ flip_v: clip.flip_v,+ };+ let (u, v) = mapping.source_of(px as f32, py as f32);+ [f64::from(u), f64::from(v)]+ }++ /// The stroke in flight as the stage draws it: path commands over a+ /// 1000 × 1000 viewbox, the line's width as a fraction of the stage,+ /// and whether it is taking away. Empty between strokes.+ fn stroke_overlay(&self) -> (String, f32, bool) {+ let Gesture::Paint {+ clip,+ tool,+ size,+ screen,+ ..+ } = &self.gesture+ else {+ return (String::new(), 0.0, false);+ };+ let Some((first, rest)) = screen.split_first() else {+ return (String::new(), 0.0, false);+ };+ let mut path = format!("M {:.1} {:.1}", first.0 * 1000.0, first.1 * 1000.0);+ // A single press still draws a dot: a line to where it already is,+ // with round caps.+ for (x, y) in rest.iter().chain(rest.is_empty().then_some(first)) {+ path.push_str(&format!(" L {:.1} {:.1}", x * 1000.0, y * 1000.0));+ }+ // The brush is `size` of the picture's width; on the stage that is+ // `size` of the picture's footprint.+ let width = self+ .clip(clip)+ .map(|clip| self.footprint(clip).w)+ .unwrap_or(1.0) as f32+ * *size as f32;+ let erase = matches!(+ tool,+ model::BrushTool::Eraser | model::BrushTool::SmartEraser+ );+ (path, width, erase)+ }+// ── edits from menus and the tray ──/// Split at `at`: every selected clip the instant runs through, or every@@ -3430,6 +3760,7 @@ impl Studio {self.sync_audio();self.request_media_art();self.request_preview();+ self.ensure_cutouts();}Err(error) => {self.start.busy = false;@@ -3474,6 +3805,8 @@ impl Studio {/// Saves, then closes the session and returns to the launch screen.pub fn close_project(&mut self) {self.pause();+ self.host.cutouts.cancel();+ self.cutout_jobs.clear();if let Some(session) = self.session.as_mut() {let (path, document) = session.prepare_save(None, None, None);if let Err(error) = projects::save(&path, &document) {@@ -4304,6 +4637,13 @@ impl Studio {editor.set_preview_frame(self.preview.clone());sync(&models.stage, self.stage_items());sync(&models.guides, self.stage_guides.clone());+ let (path, width, erase) = self.stroke_overlay();+ editor.set_stroke_path(path.into());+ editor.set_stroke_width(width);+ editor.set_stroke_erase(erase);+ editor.set_brush_tool(self.brush as i32);+ editor.set_brush_size(self.brush_size as f32);+ editor.set_painting(self.painting);editor.set_drop(match &self.drop {Some(plan) => DropData {@@ -4477,6 +4817,26 @@ impl Studio {plated: plate.alpha() > 0,line_height: text.line_height as f32,tracking: text.tracking as f32,+ cutout: match &clip.cutout {+ None => 0,+ Some(cutout) if cutout.mode == model::CutoutMode::Auto => 1,+ Some(_) => 2,+ },+ cutout_feather: clip+ .cutout+ .as_ref()+ .map(|cutout| cutout.feather as f32)+ .unwrap_or(model::DEFAULT_FEATHER as f32),+ cutout_strokes: clip+ .cutout+ .as_ref()+ .map(|cutout| cutout.strokes.len() as i32)+ .unwrap_or(0),+ cutout_progress: self+ .cutout_jobs+ .get(&clip.media_id)+ .copied()+ .unwrap_or(-1.0),}}diff --git a/engine/crates/concat/ui/editor.slint b/engine/crates/concat/ui/editor.slintindex 57779aa8..9224075a 100644--- a/engine/crates/concat/ui/editor.slint+++ b/engine/crates/concat/ui/editor.slint@@ -182,6 +182,27 @@ export global Editor {callback stage-dragged(x: float, y: float, snap: bool);callback stage-released();+ /// A custom cutout's brush, as the inspector set it: which of the four+ /// tools, how wide as a fraction of the picture, and whether a press on+ /// the stage paints at all rather than moving the picture.+ in-out property <int> brush-tool;+ in-out property <float> brush-size;+ in-out property <bool> painting;+ /// The stroke under the pointer, as path commands over a 1000 × 1000+ /// viewbox of the stage; empty between strokes. The width is a fraction+ /// of the stage's width; `erase` colours the line for a tool that takes+ /// away.+ in-out property <string> stroke-path;+ in-out property <float> stroke-width;+ in-out property <bool> stroke-erase;+ /// The Remove background section's controls; see VideoPanel. `mode` is+ /// 0 off, 1 automatic, 2 custom - the two screens are chain links.+ callback cutout-mode(mode: int);+ callback cutout-tool(index: int);+ callback cutout-size(size: float);+ callback cutout-painting(on: bool);+ callback cutout-clear();+// ── the timeline ──in-out property <[TimelineTabData]> timeline-tabs;in-out property <int> timeline-current-tab;diff --git a/engine/crates/concat/ui/inspector/clip-inspector.slint b/engine/crates/concat/ui/inspector/clip-inspector.slintindex 0a5a5da0..1a76004b 100644--- a/engine/crates/concat/ui/inspector/clip-inspector.slint+++ b/engine/crates/concat/ui/inspector/clip-inspector.slint@@ -4,6 +4,7 @@import { Theme } from "../theme/theme.slint";import { Glyph, Icon } from "../icons.slint";import { IconButton, Row, Section, SectionBody, SectionHeader } from "../primitives/panel.slint";+import { Button } from "../primitives/modal.slint";import { SegmentedControl } from "../primitives/segmented-control.slint";import { Select } from "../primitives/select.slint";import {@@ -328,6 +329,10 @@ export component VideoPanel inherits VerticalLayout {/// it a switch and a name.in property <[AppliedEntryData]> applied;in property <[AppliedParamData]> params;+ /// The custom cutout's brush; see Editor.+ in property <int> brush-tool;+ in property <float> brush-size;+ in property <bool> painting;callback set(field: ClipField, value: float);/// A control gesture finished: the accumulated change becomes one command.@@ -335,6 +340,12 @@ export component VideoPanel inherits VerticalLayout {callback chain-add(id: string);callback chain-remove(index: int);callback chain-set-param(index: int, key: string, value: float);+ /// The cutout: its mode (0 off, 1 automatic, 2 custom), and the brush.+ callback cutout-mode(mode: int);+ callback cutout-tool(index: int);+ callback cutout-size(size: float);+ callback cutout-painting(on: bool);+ callback cutout-clear();/// Which keyer is on the chain, and where: -1 for none. Scans the first/// eight links, as PresetStack does, for want of a model lookup.@@ -351,9 +362,14 @@ export component VideoPanel inherits VerticalLayout {}property <int> green: root.keyer-at("concat.green-screen");property <int> blue: root.keyer-at("concat.blue-screen");- /// 0 off, 1 green, 2 blue - the Select's rows.+ /// 0 off, 1 green, 2 blue: which chroma package is on.property <int> keyer: root.green >= 0 ? 1 : root.blue >= 0 ? 2 : 0;property <int> keyer-index: root.green >= 0 ? root.green : root.blue;+ /// The Mode row: 0 off, 1 automatic, 2 custom, 3 green screen, 4 blue+ /// screen. A keyer on the chain wins, since a cutout under a key would+ /// be two removals at once.+ property <int> removal: root.keyer == 1 ? 3 : root.keyer == 2 ? 4 : root.selected.cutout;+ property <bool> masked: root.removal == 1 || root.removal == 2;property <float> scale;property <float> offset-x;@@ -366,8 +382,10 @@ export component VideoPanel inherits VerticalLayout {property <float> crop-top;property <float> crop-right;property <float> crop-bottom;+ property <float> feather;function seed() {+ root.feather = root.selected.cutout-feather;root.scale = root.selected.scale;root.offset-x = root.selected.offset-x;root.offset-y = root.selected.offset-y;@@ -514,14 +532,19 @@ export component VideoPanel inherits VerticalLayout {}}- // The background keyed out. A picture shot on green or blue drops it- // here; the keyer is the chroma package on the chain, which is why its- // two knobs read from the chain's own rows.+ // The background taken away. Automatic and custom cut by the mask the+ // cutout model finds for the person in the picture - custom with+ // strokes painted on the monitor over it - and the two screens key a+ // colour out. A screen is the chroma package on the chain, which is why+ // its knobs read from the chain's own rows.Section {cutout := SectionHeader {title: "Remove background";collapsible: true;- summary: root.keyer == 0 ? "off" : root.keyer == 1 ? "Green screen" : "Blue screen";+ summary: root.removal == 0 ? "off"+ : root.removal == 1 ? "Automatic"+ : root.removal == 2 ? "Custom"+ : root.removal == 3 ? "Green screen" : "Blue screen";HelpButton {open: root.cutout-help;toggled => { root.cutout-help = self.open; }@@ -530,25 +553,32 @@ export component VideoPanel inherits VerticalLayout {if cutout.open: SectionBody {if root.cutout-help: Note {- text: "Keys one colour out of the picture so what is beneath"- + " shows through. Similarity is how far from the colour"- + " still counts; softness feathers the edge.";+ text: "Automatic keeps the person the cutout model finds and"+ + " drops the rest. Custom starts from the same mask and"+ + " takes corrections painted on the monitor: the smart"+ + " brush and smart eraser follow the model's confidence,"+ + " the plain ones do exactly what you paint. Green and"+ + " blue screen key one colour out instead; similarity is"+ + " how far from the colour still counts, softness feathers"+ + " the edge.";}Row {label: "Mode";Select {- options: ["Off", "Green screen", "Blue screen"];- current: root.keyer;+ options: ["Off", "Automatic", "Custom", "Green screen", "Blue screen"];+ current: root.removal;changed(index) => {- // One keyer at a time: the old one comes off before the- // new one goes on, and off is only the removal.+ // One removal at a time: the keyer comes off the chain+ // before anything else goes on, and the cutout is set+ // or cleared to match.if (root.keyer-index >= 0) {root.chain-remove(root.keyer-index);}- if (index == 1) {+ root.cutout-mode(index == 1 ? 1 : index == 2 ? 2 : 0);+ if (index == 3) {root.chain-add("concat.green-screen");- } else if (index == 2) {+ } else if (index == 4) {root.chain-add("concat.blue-screen");}root.commit();@@ -556,6 +586,63 @@ export component VideoPanel inherits VerticalLayout {}}+ if root.masked: Param {+ label: "Feather";+ value <=> root.feather;+ minimum: 0;+ maximum: 0.1;+ step: 0.002;+ default-value: 0.01;+ fmt: ParamFormat.percent;+ changed(v) => { root.set(ClipField.cutout-feather, v); root.commit(); }+ }++ // The analysis, while it runs: the masks are found in the+ // background and the picture is shown as shot until they are.+ if root.masked && root.selected.cutout-progress >= 0: Row {+ label: "Analysis";+ Text {+ text: "Finding the subject… \{Math.round(root.selected.cutout-progress * 100)}%";+ color: Theme.fg-muted;+ font-size: Theme.fs;+ vertical-alignment: center;+ }+ }++ if root.removal == 2: Row {+ label: "Tool";+ Select {+ options: ["Smart brush", "Brush", "Smart eraser", "Eraser"];+ current: root.brush-tool;+ changed(index) => { root.cutout-tool(index); }+ }+ }+ if root.removal == 2: Param {+ label: "Brush size";+ value: root.brush-size;+ minimum: 0.005;+ maximum: 0.5;+ step: 0.005;+ default-value: 0.06;+ fmt: ParamFormat.percent;+ changed(v) => { root.cutout-size(v); }+ }+ if root.removal == 2: SwitchRow {+ label: "Paint on the monitor";+ checked: root.painting;+ toggled(on) => { root.cutout-painting(on); }+ }+ if root.removal == 2: Row {+ label: root.selected.cutout-strokes == 1 ? "1 stroke"+ : "\{root.selected.cutout-strokes} strokes";+ Button {+ label: "Clear strokes";+ ghost: true;+ enabled: root.selected.cutout-strokes > 0;+ clicked => { root.cutout-clear(); }+ }+ }+// The keyer's own knobs, re-seeded from the chain the way the// stack's rows are; see PresetStack.for p in root.params: Rectangle {diff --git a/engine/crates/concat/ui/timeline/model.slint b/engine/crates/concat/ui/timeline/model.slintindex 11ed4038..0923090d 100644--- a/engine/crates/concat/ui/timeline/model.slint+++ b/engine/crates/concat/ui/timeline/model.slint@@ -156,6 +156,7 @@ export enum ClipField {flip-h, flip-v, blend, crop-left, crop-top, crop-right, crop-bottom,font-size, font-weight, italic, text-opacity, align,stroke-width, shadow, line-height, tracking,+ cutout-feather,}/// The same again for the properties that are text rather than numbers.@@ -218,6 +219,14 @@ export struct SelectedClipData {fade-in: float,fade-out: float,+ // the cutout: 0 none, 1 automatic, 2 custom; how soft its edge is; how+ // many strokes it carries; and the analysis finding its masks, -1 when+ // none is running, else how far along it is+ cutout: int,+ cutout-feather: float,+ cutout-strokes: int,+ cutout-progress: float,+// the overlay, when this is a text clipcontent: string,font-family: string,diff --git a/engine/crates/concat/ui/workspace/config-pane.slint b/engine/crates/concat/ui/workspace/config-pane.slintindex c1184e1a..bc7236a9 100644--- a/engine/crates/concat/ui/workspace/config-pane.slint+++ b/engine/crates/concat/ui/workspace/config-pane.slint@@ -159,6 +159,16 @@ export component ConfigPane inherits Rectangle {callback clip-set-colour(field: ClipTextField, value: color);callback clip-commit();+ /// The cutout's brush and controls; see Editor.+ in property <int> brush-tool;+ in property <float> brush-size;+ in property <bool> painting;+ callback cutout-mode(mode: int);+ callback cutout-tool(index: int);+ callback cutout-size(size: float);+ callback cutout-painting(on: bool);+ callback cutout-clear();+// The tab strip follows the selection, and stays short: three tabs at// most, each with a dropdown of its sections, because seven tabs on a@@ -473,6 +483,9 @@ export component ConfigPane inherits Rectangle {selected: root.selected;applied: root.applied-visual;params: root.visual-params;+ brush-tool: root.brush-tool;+ brush-size: root.brush-size;+ painting: root.painting;set(field, value) => { root.clip-set(field, value); }commit => { root.clip-commit(); }chain-add(id) => { root.chain-add(false, id); }@@ -480,6 +493,11 @@ export component ConfigPane inherits Rectangle {chain-set-param(index, key, value) => {root.chain-set-param(false, index, key, value);}+ cutout-mode(mode) => { root.cutout-mode(mode); }+ cutout-tool(index) => { root.cutout-tool(index); }+ cutout-size(size) => { root.cutout-size(size); }+ cutout-painting(on) => { root.cutout-painting(on); }+ cutout-clear => { root.cutout-clear(); }}if root.showing == "Animation": AnimationPanel {diff --git a/engine/crates/concat/ui/workspace/preview-pane.slint b/engine/crates/concat/ui/workspace/preview-pane.slintindex 1b928f4a..82cdc1ed 100644--- a/engine/crates/concat/ui/workspace/preview-pane.slint+++ b/engine/crates/concat/ui/workspace/preview-pane.slint@@ -487,6 +487,11 @@ component Guide inherits Rectangle {component StageOverlay inherits Rectangle {in property <[StageItemData]> items;in property <[StageGuideData]> guides;+ /// The brush stroke in flight, and whether a press paints; see Editor.+ in property <string> stroke-path;+ in property <float> stroke-width;+ in property <bool> stroke-erase;+ in property <bool> painting;callback pressed(x: float, y: float, additive: bool);callback grip-pressed(id: string, grip: int, x: float, y: float);@@ -500,6 +505,7 @@ component StageOverlay inherits Rectangle {floor := TouchArea {property <bool> live;property <bool> snap;+ mouse-cursor: root.painting ? MouseCursor.crosshair : MouseCursor.default;pointer-event(event) => {self.snap = event.modifiers.shift;@@ -535,6 +541,22 @@ component StageOverlay inherits Rectangle {guide: guide;}+ // The stroke being painted, over everything else: the line the pointer+ // is drawing, translucent so the picture shows through it, in the+ // accent for a brush and in red for a tool that takes away.+ if root.stroke-path != "": Path {+ x: 0;+ y: 0;+ width: root.width;+ height: root.height;+ viewbox-width: 1000;+ viewbox-height: 1000;+ commands: root.stroke-path;+ stroke: root.stroke-erase ? #ff5a5a99 : Theme.accent.with-alpha(0.6);+ stroke-width: Math.max(2px, root.stroke-width * root.width);+ stroke-line-cap: round;+ }+for entry in root.items: Rectangle {x: 0;y: 0;@@ -575,6 +597,11 @@ export component PreviewPane inherits Rectangle {/// Every picture under the playhead, for the selection drawn over the/// frame. See StageOverlay.in property <[StageItemData]> stage-items;+ /// The cutout brush's stroke in flight, and whether the stage paints.+ in property <string> stroke-path;+ in property <float> stroke-width;+ in property <bool> stroke-erase;+ in property <bool> painting;/// The snap lines while one of them is being moved.in property <[StageGuideData]> stage-guides;@@ -733,6 +760,10 @@ export component PreviewPane inherits Rectangle {height: stage.height;items: root.stage-items;guides: root.stage-guides;+ stroke-path: root.stroke-path;+ stroke-width: root.stroke-width;+ stroke-erase: root.stroke-erase;+ painting: root.painting;pressed(x, y, additive) => { root.stage-pressed(x, y, additive); }grip-pressed(id, grip, x, y) => { root.stage-grip-pressed(id, grip, x, y); }dragged(x, y, snap) => { root.stage-dragged(x, y, snap); }diff --git a/engine/crates/concat/ui/workspace/seat.slint b/engine/crates/concat/ui/workspace/seat.slintindex 119ebd59..50499975 100644--- a/engine/crates/concat/ui/workspace/seat.slint+++ b/engine/crates/concat/ui/workspace/seat.slint@@ -148,6 +148,10 @@ export component Seat inherits Rectangle {frame: Editor.preview-frame;stage-items: Editor.stage-items;stage-guides: Editor.stage-guides;+ stroke-path: Editor.stroke-path;+ stroke-width: Editor.stroke-width;+ stroke-erase: Editor.stroke-erase;+ painting: Editor.painting && Editor.selected-clip.cutout == 2;play-toggled => { Editor.play-toggled(); }seek(seconds) => { Editor.seek(seconds); }@@ -206,6 +210,14 @@ export component Seat inherits Rectangle {chain-set-param(audio, index, key, value) => {Editor.chain-set-param(audio, index, key, value);}+ brush-tool: Editor.brush-tool;+ brush-size: Editor.brush-size;+ painting: Editor.painting;+ cutout-mode(mode) => { Editor.cutout-mode(mode); }+ cutout-tool(index) => { Editor.cutout-tool(index); }+ cutout-size(size) => { Editor.cutout-size(size); }+ cutout-painting(on) => { Editor.cutout-painting(on); }+ cutout-clear => { Editor.cutout-clear(); }project-name: Editor.project-name;project-folder: Editor.project-folder;