Skip to content

AI media harness for planning, generating, and editing locally

TypeScript

Add GPU backdrop keying controls

Bitti61bfad7f4916
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 4eb9d75..13aafb8 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -4,6 +4,7 @@ This alpha update adds LAN generation workers, Linux support, expanded color gra
## What's new
+- Added Backdrop scenes to generate subjects and action against a selected green, blue, black or white background. Chroma Key now offers matching backdrop modes and automatic border detection, with matte levels, background color removal, despill, shadow removal, edge softness, choke/grow and small-speck/hole cleanup in GPU preview and export.
- Timeline clips now offer Speed in Timing, from 25% to 400%. Speed changes preserve the source range, adjust duration and later clips on the same track, and apply to preview and export, including audio pitch.
- Open the agent from the Settings top bar in a resizable right-side panel to configure generators, weight paths and LoRAs, even without an open project.
- Scene Generation settings now offer Latent upscale: generate at half resolution and upscale to the selected output size, locally or on a LAN worker. Download its model or choose a local file in Other weights settings.
diff --git a/docs/scene-types.md b/docs/scene-types.md
index 4f05a7a..1543ad8 100644
--- a/docs/scene-types.md
+++ b/docs/scene-types.md
@@ -3,6 +3,7 @@
The Scene panel saves a type on each scene:
- **First & Last Frame** keeps the existing shot descriptions, speech, Look, sound, music and optional frame anchors. Older scenes without a type use this type.
+- **Backdrop** generates the described subject and action against a selected green, blue, black or white background (green by default). The compiled prompt requests a uniform color across all shots, with no scenery, floor, horizon, texture, gradients or backdrop shadows. Prompt references guide the subject; their original settings are replaced. Stored frame anchors and continuation links are inactive. Apply Chroma Key to the generated timeline clip and select the matching backdrop or Auto to remove it. Generation still produces ordinary video: transparency is applied by the effect. Choose a color absent from the subject and describe action that fits inside the frame.
- **Continue** uses another scene's saved latents. Choose the source scene, beginning or end (default: end), and overlap in frames (default: 22; increments of 17 at 24 fps). Generate the source first, or use Generate All to render dependencies in order. Scene length controls the new segment. The saved video keeps Slopfab's full joined video and audio; adjacent source and continuation clips use that shared decode in preview and export, while trims remain relative to each scene. Regenerate older Continue scenes to preserve both sides of the decoded boundary. Old Previous scene selections become Continue scenes linked to their original source; reordering scenes keeps that link.
- **Animate** uses an Animate generator, one driving video and one repainted scene frame. It does not send a text prompt.
- **Pose** uses a selected video for body pose, movement and timing. The Scene panel places the video selector above the shot prompts, where optional references can define the target subject and setting. It uses the video's saved clip range without its soundtrack, attached stills or refmods. Only the selected pose video and references cited in the prompts are sent; frame anchors and previous-scene continuation are inactive. Pose requires a nonempty prompt and reference-capable MiniMax weights (References or Singularity), with the existing limits of three videos, nine images and 15 seconds of reference video.
diff --git a/docs/video-effects.md b/docs/video-effects.md
index 6e7cc7a..45e7ca5 100644
--- a/docs/video-effects.md
+++ b/docs/video-effects.md
@@ -6,6 +6,7 @@ Each effect can be adjusted or removed independently. Locked tracks cannot be ed
| Effect | Controls |
| --- | --- |
| Opacity | Clip opacity, 0–100%. Temperature is available in Basic Corrections. |
+| Chroma Key | Custom RGB-distance keying or Auto, Green, Blue, Black and White backdrop modes. Backdrop modes add screen gain/balance, matte black/white levels, background unmix, despill, shadow removal, edge softness, choke/grow, isolated-speck removal and tiny-hole filling. |
| Sharpen | Edge enhancement, 0–200%; zero bypasses sharpening. |
| Gaussian Blur | Radius, 0–24 source pixels; zero bypasses blur. |
| Basic Corrections | Temperature, tint, highlights, shadows, whites and blacks −100 to +100%; exposure −4 to +4 stops; contrast −100 to +100%; saturation 0–200%. The existing brightness control remains available. |
@@ -15,6 +16,12 @@ Each effect can be adjusted or removed independently. Locked tracks cannot be ed
| Vignette | Amount −100 to +100% (positive darkens, negative lightens), midpoint 0–100%, roundness −100 to +100%, and feather 0–100%. |
| 3D LUT | Import or replace a `.cube` file and blend it at 0–100% intensity. |
+For a consistent backdrop, choose **Green** or **Blue** for color-difference keying, **Black** for light, fire and glows, or **White** for dark smoke and ink. Black and white modes derive partial transparency from brightness; they are not general segmentation tools for solid subjects. The color swatch can be adjusted to the actual screen color. **Auto** measures 32 border samples on the GPU each frame, chooses the dominant green/blue/black/white group, and estimates its color. Keep the subject away from the border, or use a fixed mode and color. Existing effects retain **Custom color (RGB distance)** and their original tolerance.
+
+Screen gain changes removal strength; screen balance mixes the maximum and average of the other two color channels. Clip black removes faint matte residue and clip white makes the subject opaque; the controls always maintain black below white. Background unmix removes the backdrop contribution from soft edges, while despill suppresses excess green or blue. Remove backdrop shadows normalizes the color key for darker screen pixels. Edge softness and choke/grow use source pixels (0–4 and −4–4 respectively; positive choke shrinks the matte). Speck and hole cleanup affect isolated one-pixel defects. Cleanup/choke run before softness and before the other video effects.
+
+Backdrop keying uses the same WebGPU passes for the program monitor and export. Border analysis, matte cleanup and intermediate pictures remain on the GPU. Each frame is processed independently so seeking and scrubbing have no previous-frame state to reset. This assumes a consistent background; it does not reconstruct a spatially varying clean plate or apply temporal smoothing. If WebGPU is unavailable, the new modes report an error rather than exporting with a different key. Existing RGB-distance effects retain their fallback.
+
The render order is Chroma Key → Gaussian Blur → Sharpen (standalone + Creative) → Basic Corrections → Creative look / faded film / vibrance → Curves → Color Wheels → LUT → Vignette. Transform, opacity, and transitions composite the result into the timeline. This order is fixed; the inspector is not a reorderable effect stack. Blur radii refer to source resolution, so preview and export use the same radius even at different output sizes.
Creative includes **Teal & Orange**, **Warm Film**, **Cool Blue**, **Bleach Bypass**, **Faded Matte**, **Monochrome**, **Golden Hour**, and **Night**. These are shader colour transforms, not LUT files. None and zero look intensity leave the look unchanged; faded film, sharpening and vibrance remain independently adjustable. Vibrance gives less saturated colours more weight.
diff --git a/scripts/check-backdrop-key.html b/scripts/check-backdrop-key.html
new file mode 100644
index 0000000..753d668
--- /dev/null
+++ b/scripts/check-backdrop-key.html
@@ -0,0 +1,93 @@
+<!doctype html>
+<!-- Run npm run dev, then open /scripts/check-backdrop-key.html in a WebGPU browser.
+ Pixel readback is confined to this test; production frames stay on the GPU. -->
+<meta charset="utf-8"><title>Backdrop key GPU check</title><pre id="result">RUNNING</pre>
+<script type="module">
+import { createVideoEffectsProcessor } from '/src/lib/videoEffectsGpu.ts';
+import { createCaptureCompositor } from '/src/lib/exportPipeline.ts';
+const result = document.getElementById('result');
+try {
+ const adapter = await navigator.gpu.requestAdapter();
+ if (!adapter) throw new Error('No WebGPU adapter');
+ const device = await adapter.requestDevice();
+ const errors = [];
+ device.addEventListener('uncapturederror', event => errors.push(event.error.message));
+ // Capture compilation errors, which are asynchronous in WebGPU.
+ device.pushErrorScope('validation');
+ const processor = createVideoEffectsProcessor(device);
+ const source = new OffscreenCanvas(32, 32);
+ const ctx = source.getContext('2d');
+ const target = device.createTexture({size:[32,32], format:'rgba8unorm', usage:GPUTextureUsage.RENDER_ATTACHMENT|GPUTextureUsage.COPY_SRC});
+ const module = device.createShaderModule({code:`
+ @group(0) @binding(0) var input: texture_2d<f32>;
+ @vertex fn vs(@builtin(vertex_index) i:u32) -> @builtin(position) vec4f {
+ let p = array<vec2f,3>(vec2f(-1,-1),vec2f(3,-1),vec2f(-1,3)); return vec4f(p[i],0,1);
+ }
+ @fragment fn fs(@builtin(position) p:vec4f) -> @location(0) vec4f { return textureLoad(input,vec2i(p.xy),0); }
+ `});
+ const pipeline=device.createRenderPipeline({layout:'auto',vertex:{module,entryPoint:'vs'},fragment:{module,entryPoint:'fs',targets:[{format:'rgba8unorm'}]}});
+ async function render(backdrop, background, subject, extra={}, paint=()=>{}, effects={}) {
+ ctx.fillStyle=background;ctx.fillRect(0,0,32,32);ctx.fillStyle=subject;ctx.fillRect(8,8,16,16);
+ paint(ctx);
+ const frame=new VideoFrame(source,{timestamp:0});
+ const output=processor.render(frame,{...effects,chromaKey:{backdrop,color:background,tolerance:20,clipBlack:0,clipWhite:100,...extra}});
+ const buffer=device.createBuffer({size:256*32,usage:GPUBufferUsage.COPY_DST|GPUBufferUsage.MAP_READ});
+ const encoder=device.createCommandEncoder();
+ const pass=encoder.beginRenderPass({colorAttachments:[{view:target.createView(),loadOp:'clear',storeOp:'store'}]});
+ pass.setPipeline(pipeline);pass.setBindGroup(0,device.createBindGroup({layout:pipeline.getBindGroupLayout(0),entries:[{binding:0,resource:output.createView()}]}));pass.draw(3);pass.end();
+ encoder.copyTextureToBuffer({texture:target},{buffer,bytesPerRow:256},[32,32]);device.queue.submit([encoder.finish()]);
+ await buffer.mapAsync(GPUMapMode.READ);
+ const bytes=new Uint8Array(buffer.getMappedRange());
+ const at=(x,y)=>Array.from(bytes.slice(y*256+x*4,y*256+x*4+4));
+ const data={border:at(0,0),center:at(16,16),edge:at(8,16)};
+ buffer.unmap();buffer.destroy();frame.close();return data;
+ }
+ const cases=[];
+ function check(condition,message){if(!condition)throw new Error(message);}
+ for(const [mode,bg,fg] of [['green','#00ff00','#ff0000'],['blue','#0000ff','#ff0000'],['black','#000000','#800000'],['white','#ffffff','#7f7f7f']]){
+ const data=await render(mode,bg,fg);
+ check(data.border[3]===0,mode+' did not remove background: '+JSON.stringify(data));
+ check(Math.abs(data.center[3]-(mode==='black'||mode==='white'?128:255))<3,mode+' alpha: '+JSON.stringify(data));
+ const auto=await render('auto',bg,fg);
+ check(JSON.stringify(auto)===JSON.stringify(data),'auto differs from '+mode);
+ cases.push({mode,...data});
+ }
+ const edge=await render('green','#00ff00','#ff0000',{softness:1,choke:.5,fillHoles:true,despeckle:true});
+ check(edge.edge[3]>0&&edge.edge[3]<255,'refinement edge: '+JSON.stringify(edge));
+ const legacy=await render('color','#00ff00','#ff0000');
+ check(legacy.border[3]===0&&legacy.center[3]===255,'legacy regression');
+ const bypass=await render('green','#00ff00','#ff0000',{enabled:false});
+ check(bypass.border[3]===255,'bypass regression');
+ const mixed=await render('green','#00ff00','#80ff80');
+ check(mixed.center.every(value=>Math.abs(value-128)<3),'unmix green edge: '+JSON.stringify(mixed));
+ const shadow=await render('green','#00ff00','#008000',{removeShadows:true});
+ check(shadow.center[3]===0,'shadow removal');
+ const speck=ctx=>{ctx.fillStyle='#00ff00';ctx.fillRect(0,0,32,32);ctx.fillStyle='#33cc00';ctx.fillRect(16,16,1,1);};
+ check((await render('green','#00ff00','#ff0000',{},speck)).center[3]>0,'speck fixture');
+ check((await render('green','#00ff00','#ff0000',{despeckle:true},speck)).center[3]===0,'speck cleanup');
+ const hole=ctx=>{ctx.fillStyle='#00ff00';ctx.fillRect(16,16,1,1);};
+ const filled=await render('green','#00ff00','#ff0000',{fillHoles:true},hole);
+ check(filled.center[0]===255 && filled.center[3]===255,'hole cleanup color: '+JSON.stringify(filled));
+ const blurred=await render('green','#00ff00','#ff0000',{softness:1,choke:.5},()=>{},{blur:{radius:2},sharpen:{amount:10}});
+ check(blurred.border[3]===0&&blurred.center[0]===255,'effect stack regression');
+ // Exercise the actual monitor/export compositor over an opaque blue layer.
+ ctx.fillStyle='#00ff00';ctx.fillRect(0,0,32,32);ctx.fillStyle='#ff0000';ctx.fillRect(8,8,16,16);
+ const compositor=await createCaptureCompositor(32,32,'#0000ff');
+ check(compositor.kind==='webgpu','expected GPU compositor');
+ compositor.clear();
+ const input=new VideoFrame(source,{timestamp:0});
+ compositor.draw(input,{transform:{scale:100,rotation:0,positionX:0,positionY:0},look:{opacity:100},opacity:1,revealStart:0,revealEnd:1,
+ transition:{type:'cut',durationMs:500},chromaKey:{color:'#00ff00',tolerance:20,backdrop:'green'}});
+ const composed=compositor.take(0,41667);
+ const pixels=new Uint8Array(composed.allocationSize({format:'RGBA'}));
+ const layout=await composed.copyTo(pixels,{format:'RGBA'});
+ const c=16*layout[0].stride+16*4;
+ check(pixels[2]>250&&pixels[0]<3&&pixels[c]>250&&pixels[c+2]<3,'compositor did not layer keyed picture');
+ input.close();composed.close();compositor.dispose();
+ const validation=await device.popErrorScope();
+ if(validation)throw new Error(validation.message);
+ if(errors.length)throw new Error(errors.join('\n'));
+ processor.dispose();target.destroy();device.destroy();
+ result.textContent='PASS '+JSON.stringify({cases,edge});
+} catch(error) {result.textContent='FAIL '+error.stack;}
+</script>
diff --git a/src-tauri/src/project/timeline.rs b/src-tauri/src/project/timeline.rs
index e6ee1a6..f27afa6 100644
--- a/src-tauri/src/project/timeline.rs
+++ b/src-tauri/src/project/timeline.rs
@@ -79,10 +79,35 @@ pub(crate) struct ClipLook {
pub(crate) enabled: Option<bool>,
}
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
pub(crate) struct ClipChromaKey {
pub(crate) color: String,
pub(crate) tolerance: f64,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) backdrop: Option<String>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) screen_gain: Option<f64>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) screen_balance: Option<f64>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) clip_black: Option<f64>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) clip_white: Option<f64>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) despill: Option<f64>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) unmix: Option<f64>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) softness: Option<f64>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) choke: Option<f64>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) remove_shadows: Option<bool>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) despeckle: Option<bool>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) fill_holes: Option<bool>,
/// Effect bypass: `Some(false)` keeps the settings but renders without
/// them. Absent in every project written before bypass existed.
#[serde(default, skip_serializing_if = "Option::is_none")]
diff --git a/src-tauri/src/project/validation/timeline.rs b/src-tauri/src/project/validation/timeline.rs
index cd04302..f37d58b 100644
--- a/src-tauri/src/project/validation/timeline.rs
+++ b/src-tauri/src/project/validation/timeline.rs
@@ -49,6 +49,13 @@ pub(super) fn validate(config: &mut ProjectConfig) -> Result<(), String> {
|| !key.color.as_bytes()[1..].iter().all(u8::is_ascii_hexdigit)
|| !key.tolerance.is_finite()
|| !(0.0..=100.0).contains(&key.tolerance)
+ || key.backdrop.as_deref().is_some_and(|value| !matches!(value, "color" | "auto" | "green" | "blue" | "black" | "white"))
+ || [(key.screen_gain, 0.0, 200.0), (key.screen_balance, 0.0, 100.0),
+ (key.clip_black, 0.0, 99.0), (key.clip_white, 1.0, 100.0),
+ (key.despill, 0.0, 100.0), (key.unmix, 0.0, 100.0),
+ (key.softness, 0.0, 4.0), (key.choke, -4.0, 4.0)]
+ .iter().any(|(value, min, max)| value.is_some_and(|v| !v.is_finite() || v < *min || v > *max))
+ || key.clip_black.unwrap_or(3.0) >= key.clip_white.unwrap_or(95.0)
{
return Err(format!("Clip '{}' has an invalid chroma key.", clip.id));
}
diff --git a/src-tauri/src/tests/persistence.rs b/src-tauri/src/tests/persistence.rs
index f068bb5..fb25871 100644
--- a/src-tauri/src/tests/persistence.rs
+++ b/src-tauri/src/tests/persistence.rs
@@ -114,6 +114,7 @@ fn effect_bypass_round_trips_and_stays_absent_when_unset() {
color: "#00ff00".into(),
tolerance: 20.0,
enabled: Some(false),
+ ..Default::default()
});
clip.look = Some(ClipLook {
opacity: 80.0,
@@ -145,6 +146,7 @@ fn chroma_key_survives_save_and_reopen_and_validates_settings() {
color: "#12ABef".into(),
tolerance: 27.0,
enabled: None,
+ ..Default::default()
});
let config = without_derived_project_state(validate_and_normalize_config(config).unwrap());
let root = tempfile::tempdir().unwrap();
@@ -169,6 +171,7 @@ fn chroma_key_survives_save_and_reopen_and_validates_settings() {
color: color.into(),
tolerance,
enabled: None,
+ ..Default::default()
});
assert!(validate_and_normalize_config(invalid)
.unwrap_err()
@@ -184,6 +187,35 @@ fn chroma_key_survives_save_and_reopen_and_validates_settings() {
);
}
+#[test]
+fn backdrop_key_controls_round_trip_and_reject_invalid_ranges() {
+ let mut config = fixture();
+ let key = serde_json::json!({"color":"#0000ff", "tolerance":20, "backdrop":"blue",
+ "screenGain":110, "screenBalance":50, "clipBlack":5, "clipWhite":90,
+ "despill":70, "unmix":80, "softness":1.5, "choke":-0.5,
+ "removeShadows":true, "despeckle":true, "fillHoles":true});
+ config.timeline.tracks[0].clips[0].chroma_key = Some(serde_json::from_value(key.clone()).unwrap());
+ let config = without_derived_project_state(validate_and_normalize_config(config).unwrap());
+ let root = tempfile::tempdir().unwrap();
+ let created = create_project_in(root.path(), &config).unwrap();
+ let folder = PathBuf::from(&created.folder_path);
+ write_project(&folder, &config).unwrap();
+ assert_eq!(read_project(&folder).unwrap().config, config);
+ let restored = serde_json::to_value(&config.timeline.tracks[0].clips[0].chroma_key).unwrap();
+ assert_eq!(restored["backdrop"], "blue");
+ assert_eq!(restored["softness"], 1.5);
+ assert_eq!(restored["fillHoles"], true);
+ for (name, value) in [("backdrop", serde_json::json!("red")), ("clipBlack", serde_json::json!(90)),
+ ("clipWhite", serde_json::json!(5)), ("screenGain", serde_json::json!(201)),
+ ("despill", serde_json::json!(-1)), ("softness", serde_json::json!(5)), ("choke", serde_json::json!(-5))] {
+ let mut invalid = config.clone();
+ let mut invalid_key = key.clone();
+ invalid_key[name] = value;
+ invalid.timeline.tracks[0].clips[0].chroma_key = Some(serde_json::from_value(invalid_key).unwrap());
+ assert!(validate_and_normalize_config(invalid).unwrap_err().contains("chroma key"));
+ }
+}
+
#[test]
fn deleting_a_project_removes_its_folder_and_every_file_inside_it() {
let parent = tempfile::tempdir().unwrap();
diff --git a/src/components/workspace/ClipEffects.test.tsx b/src/components/workspace/ClipEffects.test.tsx
index ff598d5..f4f412f 100644
--- a/src/components/workspace/ClipEffects.test.tsx
+++ b/src/components/workspace/ClipEffects.test.tsx
@@ -5,6 +5,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
import { timelineClipSchema, type TimelineClip } from "../../lib/project";
import { DEFAULT_CREATIVE, DEFAULT_WHEELS, parseCube } from "../../lib/effectSettings";
import { ClipEffects } from "./ClipEffects";
+import { choose } from "./comboTestUtils";
vi.mock("@tauri-apps/api/core", () => ({ invoke: vi.fn() }));
@@ -42,6 +43,26 @@ async function inTauri(answers: Record<string, (args: unknown) => unknown>, body
const CUBE = "LUT_3D_SIZE 2\n0 0 0\n1 0 0\n0 1 0\n1 1 0\n0 0 1\n1 0 1\n0 1 1\n1 1 1";
describe("clip effects", () => {
+ it("selects backdrop presets, keeps matte levels valid, and bypasses without losing controls", () => {
+ render(<Harness />);
+ add("Chroma key");
+ choose("Chroma key backdrop", "Blue");
+ expect(saved().chromaKey).toMatchObject({ backdrop: "blue", color: "#0000ff" });
+ expect(screen.queryByRole("slider", { name: "Chroma key tolerance" })).toBeNull();
+ fireEvent.change(screen.getByRole("slider", { name: "Chroma key clip black" }), { target: { value: "98" } });
+ expect(saved().chromaKey).toMatchObject({ clipBlack: 98, clipWhite: 99 });
+ fireEvent.click(screen.getByRole("checkbox", { name: "Fill tiny holes" }));
+ expect(saved().chromaKey.fillHoles).toBe(true);
+ fireEvent.click(screen.getByRole("checkbox", { name: "Chroma key on" }));
+ expect(saved().chromaKey).toMatchObject({ enabled: false, backdrop: "blue", fillHoles: true });
+ expect(screen.getByRole("slider", { name: "Chroma key screen gain" })).toBeTruthy();
+ fireEvent.click(screen.getByRole("checkbox", { name: "Chroma key on" }));
+ choose("Chroma key backdrop", "Black");
+ expect(saved().chromaKey.color).toBe("#000000");
+ expect(screen.queryByRole("slider", { name: "Chroma key despill" })).toBeNull();
+ choose("Chroma key backdrop", "Custom color");
+ expect(screen.getByRole("slider", { name: "Chroma key tolerance" })).toBeTruthy();
+ });
it("disables Paste settings until settings have been copied", () => {
render(<ClipEffects clip={{ ...original, sharpen: { amount: 50 } }} disabled={false} onChange={vi.fn()} />);
fireEvent.click(screen.getByRole("button", { name: "Sharpen options" }));
diff --git a/src/components/workspace/ClipEffects.tsx b/src/components/workspace/ClipEffects.tsx
index 5bcb536..9fbb4e3 100644
--- a/src/components/workspace/ClipEffects.tsx
+++ b/src/components/workspace/ClipEffects.tsx
@@ -8,6 +8,8 @@ import { Checkbox, ComboBox, Flyout, PropRow, Slider, tooltipProps, useContextMe
import { CurveEditor } from "./CurveEditor";
import { GradingWheel } from "./GradingWheel";
import { ColorSwatch } from "./ColorPicker";
+import { BACKDROP_COLORS, BACKDROP_OPTIONS, type BackdropColor } from "../../lib/backdrop";
+import { BACKDROP_KEY_DEFAULTS, usesBackdropKey } from "../../lib/chromaKey";
type EffectId = "look" | "transition" | "chromaKey" | "sharpen" | "blur" | "colorCorrection" | "vignette" | "lut" | "creative" | "curves" | "colorWheels";
// Keep copied settings across clip selections and inspector remounts, like the
@@ -127,13 +129,40 @@ const EFFECTS: readonly EffectDefinition[] = [
defaults: { chromaKey: DEFAULT_CLIP_CHROMA_KEY },
editor: ({ clip, update, disabled }) => {
const key = clip.chromaKey!;
+ const advanced = usesBackdropKey({ ...key, enabled: true });
+ const presetColor = BACKDROP_COLORS[key.backdrop as BackdropColor] ?? DEFAULT_CLIP_CHROMA_KEY.color;
+ const chromatic = key.backdrop !== "black" && key.backdrop !== "white";
return <>
- <PropRow label="Colour" value={key.color.toLowerCase()} defaultValue={DEFAULT_CLIP_CHROMA_KEY.color.toLowerCase()}
- onReset={disabled ? undefined : () => update({ chromaKey: { ...key, color: DEFAULT_CLIP_CHROMA_KEY.color } })} resetLabel="Reset colour">
+ <PropRow label="Backdrop">
+ <ComboBox aria-label="Chroma key backdrop" value={key.backdrop ?? "color"} disabled={disabled}
+ options={[{ value: "color", label: "Custom color" }, { value: "auto", label: "Auto (frame border)" }, ...BACKDROP_OPTIONS]}
+ onChange={(backdrop) => update({ chromaKey: { ...key, backdrop: backdrop as NonNullable<typeof key.backdrop>,
+ ...(backdrop in BACKDROP_COLORS ? { color: BACKDROP_COLORS[backdrop as BackdropColor] } : {}) } })} />
+ </PropRow>
+ {key.backdrop !== "auto" && <PropRow label="Colour" value={key.color.toLowerCase()} defaultValue={presetColor}
+ onReset={disabled ? undefined : () => update({ chromaKey: { ...key, color: presetColor } })} resetLabel="Reset colour">
<ColorSwatch label="Chroma key colour" value={key.color} disabled={disabled} onChange={(color) => update({ chromaKey: { ...key, color } })} />
<code className="clip-effect__value">{key.color.toUpperCase()}</code>
- </PropRow>
- <Param label="Tolerance" ariaLabel="Chroma key tolerance" value={key.tolerance} defaultValue={DEFAULT_CLIP_CHROMA_KEY.tolerance} disabled={disabled} onChange={(tolerance) => update({ chromaKey: { ...key, tolerance } })} />
+ </PropRow>}
+ {!advanced && <Param label="Tolerance" ariaLabel="Chroma key tolerance" value={key.tolerance} defaultValue={DEFAULT_CLIP_CHROMA_KEY.tolerance} disabled={disabled} onChange={(tolerance) => update({ chromaKey: { ...key, tolerance } })} />}
+ {advanced && <>
+ <p className="prop-caption">Use a consistent backdrop. Black preserves glows; white preserves dark smoke or ink. Matching colors inside the subject also become transparent.</p>
+ {([
+ ["screenGain", "Screen gain", 0, 200], ["screenBalance", "Screen balance", 0, 100],
+ ["clipBlack", "Clip black", 0, 99], ["clipWhite", "Clip white", 1, 100],
+ ["despill", "Despill", 0, 100], ["unmix", "Background unmix", 0, 100],
+ ["softness", "Edge softness", 0, 4], ["choke", "Choke / grow", -4, 4],
+ ] as const).filter(([field]) => chromatic || (field !== "screenBalance" && field !== "despill")).map(([field, label, min, max]) => <Param key={field} label={label} ariaLabel={`Chroma key ${label.toLowerCase()}`}
+ value={key[field] ?? BACKDROP_KEY_DEFAULTS[field]} defaultValue={BACKDROP_KEY_DEFAULTS[field]} min={min} max={max}
+ step={field === "softness" || field === "choke" ? .25 : 1} suffix={field === "softness" || field === "choke" ? " px" : "%"}
+ disabled={disabled} onChange={(value) => update({ chromaKey: { ...key, [field]: value,
+ ...(field === "clipBlack" && value >= (key.clipWhite ?? 95) ? { clipWhite: value + 1 } : {}),
+ ...(field === "clipWhite" && value <= (key.clipBlack ?? 3) ? { clipBlack: value - 1 } : {}),
+ } })} />)}
+ {([["removeShadows", "Remove backdrop shadows"], ["despeckle", "Remove isolated specks"], ["fillHoles", "Fill tiny holes"]] as const).filter(([field]) => chromatic || field !== "removeShadows").map(([field, label]) =>
+ <PropRow key={field} label={label}><Checkbox aria-label={label} checked={key[field] ?? false} disabled={disabled}
+ onChange={(checked) => update({ chromaKey: { ...key, [field]: checked } })} /></PropRow>)}
+ </>}
</>;
},
},
diff --git a/src/components/workspace/GeneratorView.test.tsx b/src/components/workspace/GeneratorView.test.tsx
index 27355af..36bb765 100644
--- a/src/components/workspace/GeneratorView.test.tsx
+++ b/src/components/workspace/GeneratorView.test.tsx
@@ -38,6 +38,20 @@ it("hides unfinished templates from the generator picker", () => {
expect(screen.queryByRole("option", { name: "First/Last Frame" })).toBeNull();
});
+it("generates the selected backdrop with the scene action and restores it on reopen", () => {
+ const state = setup();
+ fireEvent.click(screen.getByRole("button", { name: "Select scene First scene" }));
+ choose("Scene type", "Backdrop");
+ choose("Scene backdrop color", "White");
+ expect(state.latest().generationJobs[0]).toMatchObject({ sceneType: "backdrop", backdropColor: "white" });
+ expect(screen.queryByLabelText("Start frame for this scene")).toBeNull();
+ fireEvent.click(within(screen.getByRole("region", { name: "First scene" })).getByRole("button", { name: "Generate" }));
+ const saved = parseProjectConfig(JSON.parse(JSON.stringify(state.latest())));
+ const snapshot = JSON.parse(saved.generationJobs[0].generationSnapshot!);
+ expect(snapshot.prompt).toContain("solid white (#ffffff)");
+ expect(saved.generationJobs[0].backdropColor).toBe("white");
+});
+
it("unbinds a reference after its last prompt chip is removed and excludes it from generation and Used by", () => {
const initial = project();
initial.references = ["hero", "opening"].map((id) => ({ id, kind: "image", name: id, description: "", intendedUse: [],
diff --git a/src/components/workspace/ProgramLayer.tsx b/src/components/workspace/ProgramLayer.tsx
index fa273fd..4f6dac1 100644
--- a/src/components/workspace/ProgramLayer.tsx
+++ b/src/components/workspace/ProgramLayer.tsx
@@ -6,6 +6,7 @@ import { clipChromaKey, type ProjectAsset, type TimelineClip } from "../../lib/p
import { ChromaKeyPreview } from "./ChromaKeyPreview";
import { VideoEffectsPreview } from "./VideoEffectsPreview";
import { activeVideoEffects, hasVideoEffects } from "../../lib/effectSettings";
+import { usesBackdropKey } from "../../lib/chromaKey";
import type { GpuEffects } from "../../lib/videoEffectsGpu";
import { isTauri } from "../../lib/persistence";
import type { PreviewMedia } from "../../lib/previewPresentation";
@@ -99,7 +100,7 @@ export function ProgramLayer({ clip, asset, sources, playheadMs, playing, rate =
per clip object so the effect canvases only redraw when the clip does. */
const gpuEffects = useMemo<GpuEffects>(() => ({ ...activeVideoEffects(clip), chromaKey: clipChromaKey(clip) }), [clip]);
const chromaKey = gpuEffects.chromaKey;
- const effects = hasVideoEffects(gpuEffects);
+ const effects = hasVideoEffects(gpuEffects) || usesBackdropKey(chromaKey);
const sourceStyle: CSSProperties = composited || chromaKey || effects ? { ...style, visibility: "hidden", position: "absolute" } : style;
const showStatus = active && foreground && isTauri() && Boolean(asset?.relativePath || asset?.sourcePath);
return <div className="program-layer" data-clip-id={clip.id} data-active={active} aria-hidden={!active}
diff --git a/src/lib/backdropKeyShader.ts b/src/lib/backdropKeyShader.ts
new file mode 100644
index 0000000..e32699a
--- /dev/null
+++ b/src/lib/backdropKeyShader.ts
@@ -0,0 +1,126 @@
+/** Flat-backdrop keying. Only the 1x1 plate pass examines the frame border;
+ * all subsequent passes sample that GPU texture, without CPU readback.
+ * Mode: 0 legacy distance, 1 green, 2 blue, 3 black, 4 white, 5 automatic. */
+export const BACKDROP_PLATE_WGSL = /* wgsl */ `
+@group(0) @binding(1) var picture: texture_external;
+fn classifyBackdrop(rgb: vec3f) -> u32 {
+ if (rgb.g - max(rgb.r, rgb.b) > .1) { return 1u; }
+ if (rgb.b - max(rgb.r, rgb.g) > .1) { return 2u; }
+ if (dot(rgb, vec3f(.333333)) < .5) { return 3u; }
+ return 4u;
+}
+@fragment fn fs() -> @location(0) vec4f {
+ if (params.keyControls.x != 5.) { return vec4f(params.key.rgb, params.keyControls.x); }
+ var sums: array<vec3f, 4>;
+ var counts: array<f32, 4>;
+ // Eight samples on each edge, inset to avoid codec padding.
+ for (var i = 0u; i < 32u; i++) {
+ let along = (f32(i % 8u) + .5) / 8.;
+ let side = i / 8u;
+ var uv = vec2f(along, .02);
+ if (side == 1u) { uv = vec2f(along, .98); }
+ if (side == 2u) { uv = vec2f(.02, along); }
+ if (side == 3u) { uv = vec2f(.98, along); }
+ let rgb = textureSampleBaseClampToEdge(picture, linearSampler, uv).rgb;
+ let bucket = classifyBackdrop(rgb) - 1u;
+ sums[bucket] += rgb;
+ counts[bucket] += 1.;
+ }
+ var selected = 0u;
+ for (var i = 1u; i < 4u; i++) {
+ if (counts[i] > counts[selected]) { selected = i; }
+ }
+ return vec4f(sums[selected] / max(counts[selected], 1.), f32(selected + 1u));
+}
+`;
+
+export const BACKDROP_KEY_WGSL = /* wgsl */ `
+@group(0) @binding(3) var backdropPlate: texture_2d<f32>;
+fn screenSignal(rgb: vec3f, mode: f32) -> f32 {
+ var other = rgb.rb;
+ var key = rgb.g;
+ if (mode == 2.) { other = rgb.rg; key = rgb.b; }
+ return key - mix(max(other.x, other.y), (other.x + other.y) * .5, params.keyControls.z);
+}
+fn backdropKey(color: vec4f) -> vec4f {
+ let plate = textureLoad(backdropPlate, vec2i(0), 0);
+ let rgb = color.rgb;
+ let gain = params.keyControls.y;
+ var a = 1.;
+ if (plate.a <= 2.) {
+ a = 1. - gain * screenSignal(rgb, plate.a) / max(screenSignal(plate.rgb, plate.a), .04);
+ if (params.keyMatte.w > 0.) {
+ let normalized = rgb / max(max(rgb.r, max(rgb.g, rgb.b)), .001);
+ let normalizedPlate = plate.rgb / max(max(plate.r, max(plate.g, plate.b)), .001);
+ let shadowAlpha = 1. - gain * screenSignal(normalized, plate.a) / max(screenSignal(normalizedPlate, plate.a), .04);
+ a = min(a, shadowAlpha);
+ }
+ } else if (plate.a == 3.) {
+ let delta = max(rgb - plate.rgb, vec3f(0.));
+ a = gain * max(delta.r, max(delta.g, delta.b)) / max(1. - max(plate.r, max(plate.g, plate.b)), .2);
+ } else {
+ let delta = max(vec3f(1.) - rgb / max(plate.rgb, vec3f(.2)), vec3f(0.));
+ a = gain * max(delta.r, max(delta.g, delta.b));
+ }
+ a = clamp(a, 0., 1.);
+ // Recover straight foreground before clipping the matte; clipping is an
+ // artistic adjustment and must not change the color solve underneath it.
+ var foreground = mix(rgb, (rgb - (1. - a) * plate.rgb) / max(a, .04), params.keyMatte.z);
+ if (plate.a == 1.) {
+ let limit = mix(max(foreground.r, foreground.b), (foreground.r + foreground.b) * .5, params.keyControls.z);
+ foreground.g -= params.keyControls.w * max(foreground.g - limit, 0.);
+ } else if (plate.a == 2.) {
+ let limit = mix(max(foreground.r, foreground.g), (foreground.r + foreground.g) * .5, params.keyControls.z);
+ foreground.b -= params.keyControls.w * max(foreground.b - limit, 0.);
+ }
+ let alpha = color.a * clamp((a - params.keyMatte.x) / max(params.keyMatte.y - params.keyMatte.x, .001), 0., 1.);
+ return vec4f(clamp(foreground, vec3f(0.), vec3f(1.)) * alpha, alpha);
+}
+`;
+
+export const BACKDROP_REFINE_WGSL = /* wgsl */ `
+@group(0) @binding(1) var picture: texture_2d<f32>;
+@fragment fn fs(@location(0) uv: vec2f) -> @location(0) vec4f {
+ let center = textureSampleLevel(picture, linearSampler, uv, 0.);
+ let pixel = 1. / vec2f(textureDimensions(picture));
+ var lowest = center.a;
+ var highest = center.a;
+ var neighborLow = 1.;
+ var neighborHigh = 0.;
+ var total = center * 4.;
+ var weight = 4.;
+ var edgeColor = center;
+ for (var y = -1; y <= 1; y++) {
+ for (var x = -1; x <= 1; x++) {
+ if (x == 0 && y == 0) { continue; }
+ let offset = vec2f(f32(x), f32(y)) * pixel;
+ let nearby = textureSampleLevel(picture, linearSampler, uv + offset, 0.);
+ neighborLow = min(neighborLow, nearby.a);
+ neighborHigh = max(neighborHigh, nearby.a);
+ edgeColor += nearby;
+ if (params.keyEdge.y != 0.) {
+ let moved = textureSampleLevel(picture, linearSampler, uv + offset * abs(params.keyEdge.y), 0.);
+ lowest = min(lowest, moved.a);
+ highest = max(highest, moved.a);
+ edgeColor += moved;
+ }
+ if (params.keyEdge.x > 0.) {
+ let softened = textureSampleLevel(picture, linearSampler, uv + offset * params.keyEdge.x, 0.);
+ let w = select(1., 2., x == 0 || y == 0);
+ total += softened * w;
+ weight += w;
+ }
+ }
+ }
+ var alpha = center.a;
+ if (params.keyEdge.z > 0. && neighborHigh < .05 && alpha < .5) { alpha = 0.; }
+ if (params.keyEdge.w > 0. && neighborLow > .95) { alpha = 1.; }
+ if (params.keyEdge.y > 0.) { alpha = min(alpha, lowest); }
+ if (params.keyEdge.y < 0.) { alpha = max(alpha, highest); }
+ if (params.keyEdge.x > 0.) { alpha = mix(alpha, total.a / weight, min(params.keyEdge.x, 1.)); }
+ // Extend neighboring foreground colors into newly grown transparent pixels.
+ var rgb = edgeColor.rgb / max(edgeColor.a, .00001);
+ if (center.a > .04) { rgb = center.rgb / center.a; }
+ return vec4f(rgb * alpha, alpha);
+}
+`;
diff --git a/src/lib/chromaKey.test.ts b/src/lib/chromaKey.test.ts
index aaa4065..1c44fae 100644
--- a/src/lib/chromaKey.test.ts
+++ b/src/lib/chromaKey.test.ts
@@ -1,8 +1,23 @@
import { describe, expect, it } from "vitest";
-import { applyChromaKey, keyAlpha } from "./chromaKey";
+import { applyChromaKey, backdropKeyUniforms, keyAlpha, usesBackdropKey } from "./chromaKey";
import { clipChromaKeySchema, timelineClipSchema } from "./project";
describe("Chroma Key", () => {
+ it("validates backdrop controls without changing legacy settings", () => {
+ const base = { color: "#00ff00", tolerance: 20 };
+ const key = { ...base, backdrop: "green" as const, screenGain: 110, screenBalance: 50, clipBlack: 5, clipWhite: 90,
+ despill: 70, unmix: 80, softness: 1.5, choke: -.5, removeShadows: true, despeckle: true, fillHoles: true };
+ expect(clipChromaKeySchema.parse(JSON.parse(JSON.stringify(key)))).toEqual(key);
+ expect(usesBackdropKey(key)).toBe(true);
+ expect(usesBackdropKey({ ...key, enabled: false })).toBe(false);
+ expect(usesBackdropKey(base)).toBe(false);
+ expect(backdropKeyUniforms(key)).toEqual([1, 1.1, .5, .7, .05, .9, .8, 1, 1.5, -.5, 1, 1]);
+ expect(backdropKeyUniforms({ ...base, backdrop: "auto", screenGain: null })[1]).toBe(1);
+ for (const patch of [{ backdrop: "red" }, { clipBlack: 95 }, { clipWhite: 3 }, { screenGain: 201 }, { despill: -1 },
+ { softness: 5 }, { choke: -5 }, { unmix: NaN }]) {
+ expect(clipChromaKeySchema.safeParse({ ...base, ...patch }).success).toBe(false);
+ }
+ });
it("removes the chosen color, keeps other colors and preserves existing alpha", () => {
const pixels = new Uint8ClampedArray([0, 255, 0, 255, 255, 0, 0, 128, 0, 0, 255, 255]);
applyChromaKey(pixels, { color: "#00ff00", tolerance: 0 });
diff --git a/src/lib/chromaKey.ts b/src/lib/chromaKey.ts
index fdee206..60a85e0 100644
--- a/src/lib/chromaKey.ts
+++ b/src/lib/chromaKey.ts
@@ -1,5 +1,18 @@
import type { ClipChromaKey } from "./project";
+export const BACKDROP_KEY_DEFAULTS = { screenGain: 100, screenBalance: 0, clipBlack: 3, clipWhite: 95,
+ despill: 100, unmix: 100, softness: 0, choke: 0, removeShadows: false, despeckle: false, fillHoles: false } as const;
+export const usesBackdropKey = (key?: ClipChromaKey | null): boolean => !!key && key.enabled !== false && !!key.backdrop && key.backdrop !== "color";
+export function backdropKeyUniforms(key?: ClipChromaKey | null): number[] {
+ const defaults = BACKDROP_KEY_DEFAULTS;
+ return [
+ usesBackdropKey(key) ? ["color", "green", "blue", "black", "white", "auto"].indexOf(key!.backdrop!) : 0,
+ (key?.screenGain ?? defaults.screenGain) / 100, (key?.screenBalance ?? defaults.screenBalance) / 100, (key?.despill ?? defaults.despill) / 100,
+ (key?.clipBlack ?? defaults.clipBlack) / 100, (key?.clipWhite ?? defaults.clipWhite) / 100, (key?.unmix ?? defaults.unmix) / 100, key?.removeShadows ? 1 : 0,
+ key?.softness ?? 0, key?.choke ?? 0, key?.despeckle ? 1 : 0, key?.fillHoles ? 1 : 0,
+ ];
+}
+
export function keyColor(color: string): [number, number, number] {
const value = Number.parseInt(color.slice(1), 16);
return [((value >> 16) & 255) / 255, ((value >> 8) & 255) / 255, (value & 255) / 255];
diff --git a/src/lib/exportPipeline.test.ts b/src/lib/exportPipeline.test.ts
index 9bdc154..b51c7fa 100644
--- a/src/lib/exportPipeline.test.ts
+++ b/src/lib/exportPipeline.test.ts
@@ -730,13 +730,14 @@ describe("running an export of external media", () => {
}
}, 10_000);
- it("refuses to silently omit GPU effects when only canvas compositing is available", async () => {
+ it.each(["sharpen", "backdrop"])("refuses to silently omit %s effects when only canvas compositing is available", async (effect) => {
const { restore, recorded } = installWebCodecs({ decodes: true, audio: false });
try {
invoked.mockResolvedValue(await sampleFile(5, 30));
const config = externalProject();
const video = config.timeline.tracks.find((track) => track.kind === "video" && track.clips.length > 0)!;
- video.clips[0].sharpen = { amount: 50 };
+ if (effect === "sharpen") video.clips[0].sharpen = { amount: 50 };
+ else video.clips[0].chromaKey = { color: "#00ff00", tolerance: 20, backdrop: "green" };
const settings = defaultExportSettings(config);
await expect(runExport({
folderPath: FOLDER, config, settings, plan: buildExportPlan(config, settings),
diff --git a/src/lib/exportPipeline.ts b/src/lib/exportPipeline.ts
index eaf10e7..ef92bdc 100644
--- a/src/lib/exportPipeline.ts
+++ b/src/lib/exportPipeline.ts
@@ -35,7 +35,7 @@ import {
type OutputCodecId,
} from "./export";
import type { ProjectAsset, ProjectConfig } from "./project";
-import { applyChromaKey, keyColor, KEY_FEATHER, RGB_DISTANCE_SCALE } from "./chromaKey";
+import { applyChromaKey, keyColor, KEY_FEATHER, RGB_DISTANCE_SCALE, usesBackdropKey } from "./chromaKey";
import { hasVideoEffects } from "./effectSettings";
import { createVideoEffectsProcessor } from "./videoEffectsGpu";
import { previewEngine } from "./previewEngine";
@@ -309,7 +309,7 @@ async function createWebGpuCompositor(width: number, height: number, background:
pass(context.getCurrentTexture().createView(), null);
},
draw(frame, style) {
- const processed = hasVideoEffects(style);
+ const processed = hasVideoEffects(style) || usesBackdropKey(style.chromaKey);
const renderPipeline = processed ? processedPipeline : pipeline;
const fitted = fitRect(frame.displayWidth, frame.displayHeight, width, height);
const scale = style.transform.scale / 100;
@@ -397,7 +397,7 @@ function createCanvasCompositor(width: number, height: number, background: strin
clear: paintBackground,
draw(frame, style) {
let picture: VideoFrame | OffscreenCanvas = frame;
- if (hasVideoEffects(style)) throw new Error("This clip uses video effects that require WebGPU. A GPU is unavailable; effects cannot be included in this export.");
+ if (hasVideoEffects(style) || usesBackdropKey(style.chromaKey)) throw new Error("This clip uses video effects that require WebGPU. A GPU is unavailable; effects cannot be included in this export.");
if (style.chromaKey) {
keyCanvas ??= new OffscreenCanvas(frame.displayWidth, frame.displayHeight);
if (keyCanvas.width !== frame.displayWidth) keyCanvas.width = frame.displayWidth;
diff --git a/src/lib/project.backdrop.test.ts b/src/lib/project.backdrop.test.ts
index 5678f4a..0153766 100644
--- a/src/lib/project.backdrop.test.ts
+++ b/src/lib/project.backdrop.test.ts
@@ -21,5 +21,6 @@ it("uses green by default and retains explicitly cited subject references", () =
const references: ProjectReference[] = [{ id: "subject", name: "Dancer", kind: "text", description: "A red robot", intendedUse: [], createdAt: "2026-10-09T00:00:00Z" }];
expect(sceneGenerationReferences(job, references)).toEqual(references);
expect(compileGenerationJobPrompt(job, references)).toContain("solid green");
+ expect(compileGenerationJobPrompt(job, references)).toContain("partially_preserved");
expect(generationJobSchema.safeParse({ ...job, backdropColor: "red" }).success).toBe(false);
});
diff --git a/src/lib/project.ts b/src/lib/project.ts
index 90ca205..fc0cc58 100644
--- a/src/lib/project.ts
+++ b/src/lib/project.ts
@@ -257,8 +257,20 @@ export const clipTransformSchema = z.object({
export const clipChromaKeySchema = z.object({
color: z.string().regex(/^#[0-9a-fA-F]{6}$/),
tolerance: z.number().min(0).max(100),
+ backdrop: z.enum(["color", "auto", "green", "blue", "black", "white"]).nullish(),
+ screenGain: z.number().finite().min(0).max(200).nullish(),
+ screenBalance: z.number().finite().min(0).max(100).nullish(),
+ clipBlack: z.number().finite().min(0).max(99).nullish(),
+ clipWhite: z.number().finite().min(1).max(100).nullish(),
+ despill: z.number().finite().min(0).max(100).nullish(),
+ unmix: z.number().finite().min(0).max(100).nullish(),
+ softness: z.number().finite().min(0).max(4).nullish(),
+ choke: z.number().finite().min(-4).max(4).nullish(),
+ removeShadows: z.boolean().nullish(),
+ despeckle: z.boolean().nullish(),
+ fillHoles: z.boolean().nullish(),
enabled: effectEnabledSchema,
-});
+}).refine((key) => (key.clipBlack ?? 3) < (key.clipWhite ?? 95), "Clip white must exceed clip black.");
// Keep the legacy `look` key for opacity; retired temperature values are stripped on load.
export const clipLookSchema = z.object({
@@ -1605,6 +1617,7 @@ export function compileScenePromptSegments(scene: ScenePrompt, references: Proje
const appearances = compiled.filter((shot) => featured[shot.index].includes(reference.id))
.map((shot) => `[Shot ${shot.index + 1}]`);
const where = appearances.length > 0 ? ` (appears in ${appearances.join(", ")})` : "";
+ if (scene.backdropColor) return `${referenceLabel(reference, index)}${where}: partially_preserved - retain the subject's appearance and identity; replace the setting with the uniform ${scene.backdropColor} backdrop.`;
return `${referenceLabel(reference, index)}${where}: fully_preserved - the referenced characteristics are retained.`;
});
// §4.2: `reference`, not a copy marker. The clip guides the target sound;
diff --git a/src/lib/videoEffectsGpu.ts b/src/lib/videoEffectsGpu.ts
index 51dfb4c..72a379b 100644
--- a/src/lib/videoEffectsGpu.ts
+++ b/src/lib/videoEffectsGpu.ts
@@ -1,5 +1,6 @@
/// <reference types="@webgpu/types" />
-import { keyColor, KEY_FEATHER, RGB_DISTANCE_SCALE } from "./chromaKey";
+import { backdropKeyUniforms, keyColor, KEY_FEATHER, RGB_DISTANCE_SCALE, usesBackdropKey } from "./chromaKey";
+import { BACKDROP_KEY_WGSL, BACKDROP_PLATE_WGSL, BACKDROP_REFINE_WGSL } from "./backdropKeyShader";
import { activeVideoEffects, CREATIVE_LOOKS, type VideoEffects, type LutTable } from "./effectSettings";
import { compileCurves, CURVE_SAMPLES } from "./colorCurves";
import { COLOR_GRADING_WGSL } from "./colorGradingShader";
@@ -30,14 +31,18 @@ struct Params {
shadows: vec4f,
midtones: vec4f,
highlights: vec4f,
+ keyControls: vec4f,
+ keyMatte: vec4f,
+ keyEdge: vec4f,
};
@group(0) @binding(0) var<uniform> params: Params;
@group(0) @binding(2) var linearSampler: sampler;
`;
-const IMPORT = VERTEX + PARAMS + /* wgsl */ `
+const IMPORT = VERTEX + PARAMS + BACKDROP_KEY_WGSL + /* wgsl */ `
@group(0) @binding(1) var picture: texture_external;
@fragment fn fs(@location(0) uv: vec2f) -> @location(0) vec4f {
let color = textureSampleBaseClampToEdge(picture, linearSampler, uv);
+ if (params.detail.w > 0. && params.keyControls.x > 0.) { return backdropKey(color); }
var alpha = color.a;
if (params.detail.w > 0.) {
let difference = distance(color.rgb, params.key.rgb) / ${RGB_DISTANCE_SCALE};
@@ -149,10 +154,13 @@ function pipeline(device: GPUDevice, code: string, format: GPUTextureFormat): GP
/** Shared by monitor and export. All pictures stay in GPU textures; output is premultiplied. */
export function createVideoEffectsProcessor(device: GPUDevice) {
const importPipeline = pipeline(device, IMPORT, "rgba16float");
+ const platePipeline = pipeline(device, VERTEX + PARAMS + BACKDROP_PLATE_WGSL, "rgba16float");
+ const refinePipeline = pipeline(device, VERTEX + PARAMS + BACKDROP_REFINE_WGSL, "rgba16float");
+ const plate = device.createTexture({ size: [1, 1], format: "rgba16float", usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING });
const blurPipeline = pipeline(device, BLUR, "rgba16float");
const finishPipeline = pipeline(device, FINISH, "rgba16float");
const sampler = device.createSampler({ minFilter: "linear", magFilter: "linear" });
- const uniforms = Array.from({ length: 3 }, () => device.createBuffer({ size: 208, usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST }));
+ const uniforms = Array.from({ length: 5 }, () => device.createBuffer({ size: 256, usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST }));
const curveBuffer = device.createBuffer({ size: 9 * CURVE_SAMPLES * 4, usage: GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST });
let curveKey: string | undefined;
let curveMask = 0;
@@ -163,6 +171,7 @@ export function createVideoEffectsProcessor(device: GPUDevice) {
return {
render(frame: VideoFrame, effects: GpuEffects): GPUTexture {
effects = { ...effects, ...activeVideoEffects(effects) };
+ if (effects.chromaKey?.enabled === false) effects = { ...effects, chromaKey: null };
const nextCurveKey = JSON.stringify(effects.curves ?? null);
if (nextCurveKey !== curveKey) {
const compiled = compileCurves(effects.curves);
@@ -206,6 +215,7 @@ export function createVideoEffectsProcessor(device: GPUDevice) {
const wheel = effects.colorWheels?.[name];
return [wheel?.x ?? 0, wheel?.y ?? 0, (wheel?.lightness ?? 0) / 100, 0];
}),
+ ...backdropKeyUniforms(effects.chromaKey),
]);
device.queue.writeBuffer(uniforms[0], 0, data);
const encoder = device.createCommandEncoder();
@@ -214,20 +224,41 @@ export function createVideoEffectsProcessor(device: GPUDevice) {
{ binding: 0, resource: { buffer: uniform } }, { binding: 1, resource: source }, { binding: 2, resource: sampler },
];
if (lut) entries.push({ binding: 3, resource: { buffer: lut } }, { binding: 4, resource: { buffer: curveBuffer } });
+ if (renderPipeline === importPipeline) entries.push({ binding: 3, resource: plate.createView() });
const group = device.createBindGroup({ layout: renderPipeline.getBindGroupLayout(0), entries });
const render = encoder.beginRenderPass({ colorAttachments: [{ view: target.createView(), loadOp: "clear", storeOp: "store", clearValue: [0, 0, 0, 0] }] });
render.setPipeline(renderPipeline); render.setBindGroup(0, group); render.draw(3); render.end();
};
- pass(importPipeline, device.importExternalTexture({ source: frame }), textures[0], uniforms[0]);
+ const external = device.importExternalTexture({ source: frame });
+ const backdrop = usesBackdropKey(effects.chromaKey);
+ if (backdrop) pass(platePipeline, external, plate, uniforms[0]);
+ pass(importPipeline, external, textures[0], uniforms[0]);
+ let source = textures[0];
+ let spare = textures[1];
+ const key = effects.chromaKey;
+ if (backdrop && ((key?.choke ?? 0) !== 0 || key?.despeckle || key?.fillHoles)) {
+ const refinement = data.slice();
+ refinement[60] = 0;
+ device.queue.writeBuffer(uniforms[3], 0, refinement);
+ pass(refinePipeline, source.createView(), spare, uniforms[3]);
+ [source, spare] = [spare, source];
+ }
+ if (backdrop && (key?.softness ?? 0) > 0) {
+ const refinement = data.slice();
+ refinement.fill(0, 61, 64);
+ device.queue.writeBuffer(uniforms[4], 0, refinement);
+ pass(refinePipeline, source.createView(), spare, uniforms[4]);
+ [source, spare] = [spare, source];
+ }
if ((effects.blur?.radius ?? 0) > 0) {
data[21] = 1;
device.queue.writeBuffer(uniforms[1], 0, data);
- pass(blurPipeline, textures[0].createView(), textures[1], uniforms[1]);
+ pass(blurPipeline, source.createView(), spare, uniforms[1]);
data[21] = 0; data[22] = 1;
device.queue.writeBuffer(uniforms[2], 0, data);
- pass(blurPipeline, textures[1].createView(), textures[0], uniforms[2]);
+ pass(blurPipeline, spare.createView(), source, uniforms[2]);
}
- pass(finishPipeline, textures[0].createView(), textures[2], uniforms[0], lutBuffer);
+ pass(finishPipeline, source.createView(), textures[2], uniforms[0], lutBuffer);
device.queue.submit([encoder.finish()]);
return textures[2];
},
@@ -237,6 +268,7 @@ export function createVideoEffectsProcessor(device: GPUDevice) {
luts.forEach(({ buffer }) => buffer.destroy());
emptyLut.destroy();
curveBuffer.destroy();
+ plate.destroy();
},
};
}