Skip to content

EffectCraft: An open-source motion graphics and compositing tool written by scratch in pure Rust

#783: Object Matte fidelity plan; gaps and parity notes updated

vangel76a8de199acac3
diff --git a/docs/fidelity/README.md b/docs/fidelity/README.md
index 025997d47..5afafee41 100644
--- a/docs/fidelity/README.md
+++ b/docs/fidelity/README.md
@@ -3,6 +3,9 @@
The [AE 2026 replacement acceptance gates](ae-2026-acceptance.md) define the broader
target across workflows, rendered output, work-project compatibility, performance and platforms.
+[Object Matte](object-matte.md) has a comparison plan (clips, prompt, per-frame IoU and the other
+metrics) but no measurement yet.
+
The first G1 comparison measures eight original scalar Rotation cases against a live After Effects
2026 session through AEsync 2.0.4: Easy Ease, overlapping influences, asymmetric ease, overshoot,
descending values, equal endpoints, 100%/100% influence and 0.1%/100% influence. Each case has
diff --git a/docs/fidelity/object-matte.md b/docs/fidelity/object-matte.md
new file mode 100644
index 000000000..da4e3bd3d
--- /dev/null
+++ b/docs/fidelity/object-matte.md
@@ -0,0 +1,73 @@
+# Object Matte: comparison plan (not measured yet)
+
+**Status: untested.** There is no After Effects 26.2+ comparison of Object Matte yet. Nothing on
+this page is a measurement. It lists what to compare once an After Effects reference session
+is available, so the first run can be repeated and scored without guesswork.
+
+EffectCraft side, as of #783: the Object Matte effect (`ec.matte.objectmatte`), the Object Matte
+tool in the Composition viewer, `objectMatte.apply` / `.propagate` / `.freeze` /
+`.status`, and segmentations kept in the disk cache. The point (and an optional box) is
+segmented by Roto Brush's engine: MobileSAM when it is installed (Settings ▸ Roto Brush, with
+the box as the model's box prompt), otherwise the classical graph cut. The result is tracked
+frame by frame with optical flow, and the model re-prompts each frame from the carried matte.
+Record which engine was active, because the two are separate cases.
+
+## Test clips
+
+All original, generated or openly licensed; none of them goes into the repository.
+
+1. **Synthetic disk**: the moving textured disk over a textured background from
+ `crates/engine/src/tests_roto.rs` (`draw(center(f))`, 320×240, 25 fps, 25 frames, radius
+ 40 px, about 4 px a frame to the right). Export it as a PNG sequence. Its ground truth is
+ analytic (`gt(center(f))`).
+2. **Synthetic disk with a jump**: the same disk with `jumping(f)` (it jumps at frame 8). This
+ checks how each tool handles losing the object.
+3. **Two objects**: the disk plus a second still, differently coloured square that touches it
+ on some frames, prompted once with a point only and once with a box around the disk. This
+ checks the box prompt.
+4. **Real footage**: one openly licensed clip (CC0 or CC BY) with a person or object moving
+ against clutter, at least 1080p and 5 s long. Record its URL, licence and SHA-256 next to the
+ results, and keep the clip itself out of the repository.
+
+## Prompt and settings
+
+- Reference frame 0. The point at the object's centre, in layer pixels (clip 1: `[90, 110]`).
+ The box, where used, is `[centre − 55, centre + 55]` on each axis.
+- Track forward to the last frame (Frames After = clip length − 1, Frames Before = 0).
+- Matte refinement at its defaults on both sides: Feather 0, Contrast 0, Shift Edge 0, no
+ Invert. Output the matte itself (View ▸ Matte in EffectCraft, the alpha channel in After
+ Effects) as a full-resolution PNG sequence.
+- Record the After Effects build, the OS, the GPU, and the exact click position in After
+ Effects. It is a viewer click, so convert it to layer pixels with the comp's zoom.
+
+Keep the After Effects mattes in gitignored `plan/aftereffects/ref/object-matte/`. Only an
+aggregate results file (like `keyframe-ease.json`) and the comparison script may be committed.
+
+## Metrics (per frame, then mean / worst)
+
+- **IoU against After Effects**: binarize both mattes at alpha 0.5, then intersection over
+ union. This is the headline number. `effectcraft_track::roto::iou` computes it, and
+ `roto.status` already reports it for Roto Brush through `compareTo`; Object Matte needs the
+ same for exported mattes.
+- **IoU against ground truth** for the synthetic clips, for both applications. It tells
+ "differs from After Effects" apart from "worse than After Effects".
+- **Boundary error**: the mean and 95th-percentile distance (px) between the two mattes' 0.5
+ contours, which shows edge disagreement that IoU dilutes on large objects.
+- **Soft-edge error**: the mean absolute alpha difference inside a 3 px band around the After
+ Effects contour.
+- **Temporal stability**: the frame-to-frame IoU change of each matte on clip 1, where the
+ object moves rigidly, so jitter shows.
+- **Drift**: the frame where IoU against ground truth first drops below 0.9 (clips 1 and 2).
+- **Time**: seconds per frame to track the clip, warm and cold, on the same machine, with the
+ disk cache on and off.
+
+Proposed pass bar (a maintainer decision, not yet agreed): mean IoU against After Effects
+≥ 0.95 and worst ≥ 0.90 on clips 1 and 3, with MobileSAM. With the classical engine the numbers
+are reported but not gated.
+
+## Reproduce (to write)
+
+The script does not exist yet. It should: generate the synthetic clips; run EffectCraft
+headless (`effectcraft-cli run … objectMatte.apply … objectMatte.propagate {"wait":true}`, then
+render the matte); drive After Effects through ExtendScript or AEsync to apply Object Matte with
+the same prompt and render its alpha; then compute the metrics above into a JSON results file.
diff --git a/docs/gaps.md b/docs/gaps.md
index 577f7bd6f..e9366c739 100644
--- a/docs/gaps.md
+++ b/docs/gaps.md
@@ -42,7 +42,7 @@ because 179 issues were closed with regression tests between 5 and 10 October.
| Real-user experience | ≈ 60%, uneven by platform | (in G2) | 179 issues closed 5–10 October, nearly all with regression tests ([log below](#real-user-report-log-510-october)); 134 open, many from Linux and Windows. Most checking still happens on macOS |
| Performance | unknown against After Effects, ≈ 35% | 50–100 | Internal numbers only (e.g. Advanced 3D 290 ms/frame at 1080p on the GPU on an M4 Pro). Users report stutter on cached frames (#595, #304), slow scrubbing (#329), late audio (#554). No 4K / hundreds-of-layers project tested |
| Media formats | ≈ 75% | (in file formats) | H.264, ProRes, HEVC, AV1, VP9, image sequences and audio exist. The HEVC / AV1 encoders have no B-frames, multi-reference or SAO / CDEF, so files are larger than from mature encoders. Camera formats (BRAW, R3D, ProRes RAW, variable-frame-rate phone video) are unverified |
-| AI-assisted tools | ≈ 35% | 40–80 | Roto Brush 2.0 / 3.0 with MobileSAM (M13.35) scores IoU 0.989 on the base frame of our synthetic moving-disc test and ≥ 0.980 over 20 propagated frames (classic: 0.973). Face tracking with MediaPipe Face Landmarker (M13.36) matches Google's own pipeline to 0.85 px on a test portrait. Neither is compared with After Effects; Object Matte (26.2) is missing |
+| AI-assisted tools | ≈ 35% | 40–80 | Roto Brush 2.0 / 3.0 with MobileSAM (M13.35) scores IoU 0.989 on the base frame of our synthetic moving-disc test and ≥ 0.980 over 20 propagated frames (classic: 0.973). Face tracking with MediaPipe Face Landmarker (M13.36) matches Google's own pipeline to 0.85 px on a test portrait. Neither is compared with After Effects; Object Matte (26.2) has a first slice, not compared either |
| Maturity | early | — | First commit 1 October 2026. ≈ 329,000 lines of almost entirely agent-written Rust, 2,704 tests, 313 GitHub issues. After Effects has around 30 years of edge cases behind it |
## Every known gap, ranked
@@ -70,7 +70,7 @@ are grouped by area; the issue tracker has the rest.
| 11 | **Text rendering and layout** | #453 CJK text in comps, #316 paragraph alignment inverted, #483 squeezed at 4:3, #417 soft at Fit, #583 font search, #550 toolbar font options | Text is in almost every motion graphic | 10–20 h | G2 · [ui-parity.md](ui-parity.md) |
| 12 | **3D interaction and depth** | No 3D transform gizmos (#392, #347); Z position (#489); camera aperture (#260); reflections (#261); bevel options (#315); 3D issue reopened (#307) | Advanced 3D is After Effects' growth area since 2023 | 20–35 h | G2 · [ui-parity.md](ui-parity.md) |
| 13 | **3D model formats and materials** | After Effects 26 imports FBX, USD/USDZ, STL besides glTF and OBJ (`usd_plugins`), and Substance 3D materials with mesh displacement (26.0–26.2); we read glTF and OBJ | Product shots and 3D scenes come as FBX/USDZ | 25–45 h | — · [file-format-parity.md](file-format-parity.md) |
-| 14 | **Object Matte** (26.2, disk cache in 26.5) | First slice (#783): the Object Matte effect (Effect ▸ Matte) and `objectMatte.apply` turn a point (and optional box) on a reference frame into a matte and follow the object up to 600 frames each way, or segment every frame on its own; Roto Brush's engine, cache and model (MobileSAM from Settings ▸ Roto Brush, classical graph cut without it). Missing: the click / hover tool in the viewer, a background propagation job, the disk cache, comparison with After Effects | The new one-click roto | 10–20 h | G7 · [target-app-parity.md](target-app-parity.md) |
+| 14 | **Object Matte** (26.2, disk cache in 26.5) | First slice (#783): the Object Matte effect (Effect ▸ Matte) and `objectMatte.apply` turn a point (and optional box) on a reference frame into a matte and follow the object up to 600 frames each way, or segment every frame on its own; Roto Brush's engine, cache and model (MobileSAM from Settings ▸ Roto Brush, classical graph cut without it). Since then: the Object Matte tool in the viewer (click / drag a box, hover crosshair), the box as the model's box prompt, background tracking with progress and cancel, Freeze, and segmentations kept in the disk cache (26.5). Missing: a comparison with After Effects ([plan](fidelity/object-matte.md)) | The new one-click roto | 10–20 h | G7 · [target-app-parity.md](target-app-parity.md) |
| 15 | **Hardware video decode and encode** | No VideoToolbox, NVENC, Quick Sync or VA-API; #498 asks for it; FilmCraft has VideoToolbox / NVENC / VA-API paths to borrow | 4K playback and export speed | 25–45 h | G5/G6 · [hardware-parity.md](hardware-parity.md) |
| 16 | **Localization depth** | Menus in 6 languages; panels in 4; effect names and ~1,650 effect parameter names untranslated; French, German, Korean, Italian, Russian (After Effects ships them) absent; #599 Hangul boxes on macOS | After Effects is used heavily in Japan, China, Korea, Brazil, Europe | 60–110 h + native review | G9 · [localization-parity.md](localization-parity.md) |
| 17 | **Media Browser lists files import refuses** | `media_browser.rs` lists TGA, DPX, HDR, AVI, MXF; `media::is_importable` has none of them; #431 any image imports as a sequence; #324 MP3 goes silent | Confusing failures on common VFX formats (DPX, TGA) | 15–30 h | G6 · [file-format-parity.md](file-format-parity.md) |
diff --git a/docs/target-app-parity.md b/docs/target-app-parity.md
index f0ed90b6e..c8ef95a02 100644
--- a/docs/target-app-parity.md
+++ b/docs/target-app-parity.md
@@ -122,7 +122,7 @@ compositing and effects dominate; audio is light). Hours are to full parity for
| Text and typography | 7% | 94% | 50% | 20–35 | CJK text in comps renders wrongly (#453); paragraph alignment inverted (#316); squeezed text at 4:3 (#483); font search (#583); soft text at Fit (#417); variable-font filter (26.3) |
| Effects and animation presets | 14% | 85% | 40% | 70–130 | All 306 effects exist, but rendered output is unmeasured against After Effects; 8 built-in presets against After Effects' 621; open effect bugs (#580 crash, #514, #526, #463, #369, #473); effect colour labels (26.5); see [effects-parity.md](effects-parity.md) |
| 3D (Classic and Advanced) | 6% | 82% | 35% | 35–65 | No FBX/USD/STL models, no Substance materials, mesh displacement unverified; no 3D transform gizmos (#392, #347); Z position (#489); reflections (#261); camera aperture (#260); bevel options (#315) |
-| Tracking and AI-assisted tools | 5% | 80% | 35% | 35–70 | Object Matte (26.2) is a first slice: an effect and command from a point / box prompt, no viewer tool or disk cache; Roto Brush (MobileSAM) and face tracking (MediaPipe) unmeasured against After Effects; camera-tracker view (#505) |
+| Tracking and AI-assisted tools | 5% | 80% | 35% | 35–70 | Object Matte (26.2) has the effect, the viewer tool (click / box), background tracking, Freeze and the 26.5 disk cache, not yet compared with After Effects; Roto Brush (MobileSAM) and face tracking (MediaPipe) unmeasured against After Effects; camera-tracker view (#505) |
| Preview and cache | 7% | 92% | 40% | 25–50 | Playback stutters on cached frames (#595, #304); slow scrubbing (#329); late audio (#554, #504, #335); no benchmark against After Effects |
| Render queue and export | 7% | 85% | 55% | 25–45 | No hardware encoding; ProRes 4444 colour tag (#340); rendering over an existing file (#325); default macOS output location (#446); uncompressed single-file output (#372); HEVC/AV1 without B-frames |
| Audio | 2% | 85% | 45% | 6–12 | Audio sync and crackle (#335); audio to keyframes depth |
@@ -144,7 +144,7 @@ Feature depth (≈ 50%) feeds the overall *ready* number with the other dimensio
| Stability | 45% | 40–80 | Open crash reports #580, #591, #501, #426, #341, #306, #234, #511; no CI runs the tests on pull requests | [gaps.md](gaps.md#g3-stability-and-a-green-main) |
| Platforms | 65% | 30–60 | Releases for macOS, Windows x64/x86/ARM64, Linux, FreeBSD and the web (more than After Effects), but Linux and Windows users hit basic bugs | [gaps.md](gaps.md#g2-real-user-reliability-on-every-platform) |
| Ecosystem and plug-ins | 20% | 40–80 | After Effects scripts and expressions mostly carry over; no After Effects SDK plug-ins, `.ffx` presets or `.mogrt`; our WebAssembly plug-in API has no third-party plug-ins yet | [plugins.md](plugins.md) |
-| AI features | 35% | 40–80 (overlaps Tracking) | Roto Brush (MobileSAM) and face tracking (MediaPipe) as optional downloads; Object Matte (point / box prompt, same model) without its viewer tool or disk cache; the After Effects bundle ships Whisper, BERT and Marian tokenizers and a shot-cut model we have no equivalent of | [gaps.md](gaps.md) |
+| AI features | 35% | 40–80 (overlaps Tracking) | Roto Brush (MobileSAM) and face tracking (MediaPipe) as optional downloads; Object Matte (point / box prompt, same model; viewer tool, background tracking, Freeze, disk cache) unmeasured against After Effects; the After Effects bundle ships Whisper, BERT and Marian tokenizers and a shot-cut model we have no equivalent of | [gaps.md](gaps.md) |
| Agent control (beyond After Effects) | ahead | — | Every command is drivable over MCP, the control channel and the CLI | [agents.md](agents.md) |
**Totals.** Summing the rows double-counts: AI overlaps Tracking, performance overlaps Preview,