Skip to content

Write HTML. Render video. Built for agents.

feat: keep a video's sound on the clip by default (1/8) (#4813)

Vance Ingalls37f30b1d06c4
diff --git a/docs/guides/claude-design-hyperframes.md b/docs/guides/claude-design-hyperframes.md
index 172b4b1bb5..b16df8953e 100644
--- a/docs/guides/claude-design-hyperframes.md
+++ b/docs/guides/claude-design-hyperframes.md
@@ -383,8 +383,9 @@ The skeleton handles most structural rules. These are the runtime rules the skel
| Never | Use instead |
| ------------------------------- | --------------------------- |
| `video.play()`, `audio.play()` | Framework owns playback |
-| `<video>` without `muted` | Always `muted playsinline` |
-| Audio on `<video>` | Separate `<audio>` element |
+| `<video>` with sound | `data-has-audio="true"`, no `muted` |
+| Music, voiceover, or replacement audio | separate `<audio>` element |
+| Silent footage and b-roll | `muted` |
| Base64 media | File reference or HTTPS URL |
| Placeholder URLs (placehold.co) | Real assets |
diff --git a/docs/guides/claude-design-send-to-hyperframes.md b/docs/guides/claude-design-send-to-hyperframes.md
index c5a2e7a40b..bfb3aa8077 100644
--- a/docs/guides/claude-design-send-to-hyperframes.md
+++ b/docs/guides/claude-design-send-to-hyperframes.md
@@ -375,7 +375,9 @@ The cloud renderer seeks the timeline frame-by-frame. Non-deterministic or self-
| `stagger: { from: "random" }` | `from: "start"`, `"center"`, or `"end"` |
| async timeline construction | build synchronously at page load |
| `video.play()` / `audio.play()` | the framework owns playback |
-| `<video>` without `muted playsinline` | always `muted playsinline`; audio on a separate `<audio>` |
+| `<video>` with sound | `data-has-audio="true"`, no `muted` |
+| Music, voiceover, or replacement audio | separate `<audio>` |
+| Silent footage and b-roll | `muted` |
| exit tweens before a shader | the shader IS the exit — content stays visible |
| `tl.set`/`tl.to` on an anchor container (beyond the required first-anchor `opacity:1` set) | HyperShader owns anchor opacity; only `tl.set(firstAnchor, { opacity: 1 }, start)` is allowed |
| `autoAlpha` on an anchor scene | anchors use `opacity` (HyperShader-managed); `autoAlpha` is for non-anchor scenes |
diff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdx
index 74abd1a044..8b2d3d2b8d 100644
--- a/docs/packages/cli.mdx
+++ b/docs/packages/cli.mdx
@@ -641,7 +641,7 @@ npx hyperframes lint [dir] --json
◆ Linting my-project/index.html
✗ missing_gsap_script: Composition uses GSAP but no GSAP script is loaded.
- ⚠ unmuted-video [clip-1]: Video should have the 'muted' attribute for reliable autoplay.
+ ⚠ video_missing_muted [clip-1]: <video id="clip-1"> has data-start but declares neither muted nor data-has-audio. If the file has sound, add data-has-audio="true" (the sound stays on this clip). If it is silent, add muted.
◇ 1 error(s), 1 warning(s)
```
diff --git a/docs/packages/lint.mdx b/docs/packages/lint.mdx
index daa461275c..2a4e10fc68 100644
--- a/docs/packages/lint.mdx
+++ b/docs/packages/lint.mdx
@@ -103,7 +103,7 @@ The browser entry exposes `lintHyperframeHtml`, `lintMediaUrls`, and `shouldBloc
Detected issues include:
- Missing timeline registration (`window.__timelines`)
-- Unmuted video elements (causes autoplay failures)
+- A timed `<video>` that declares neither `muted` (silent footage) nor `data-has-audio="true"` (has sound) — `video_missing_muted`
- Missing `class="clip"` on timed visible elements
- Deprecated attribute names
- Missing composition dimensions (`data-width`, `data-height`)
diff --git a/docs/prompting/anatomy.mdx b/docs/prompting/anatomy.mdx
index b043d0c45e..32b852dfc1 100644
--- a/docs/prompting/anatomy.mdx
+++ b/docs/prompting/anatomy.mdx
@@ -95,7 +95,8 @@ The skeleton above is how you ask. This is what the agent writes, and it's worth
| `class="clip"` | Marks a timed section. The framework owns its visibility — never animate a clip's own opacity or `display` to hide it. |
| `data-start` / `data-duration` / `data-track-index` | When the clip begins, how long it runs, and which track it sits on. Two clips on the same track index must not overlap in time. |
| `window.__timelines["<id>"]` | Every GSAP-driven composition registers exactly one paused timeline here, built synchronously at load. The renderer seeks that timeline frame by frame — which is why nothing may depend on wall-clock time or unseeded randomness. Other adapters, including Lottie, CSS animations, and WAAPI, register through their own seekable runtime contracts. |
-| `<video muted>` + a separate `<audio>` | Video elements render muted and their sound rides a sibling audio element, so the mixer can duck and balance it independently. |
+| Footage with sound | `<video playsinline data-has-audio="true">` — the sound stays on the clip |
+| Music / voiceover | separate `<audio>` |
You rarely write these by hand, but naming them in a prompt is legitimate and sometimes necessary — "put the captions on their own track index," "keep it one composition file, one timeline" — and it's the vocabulary every error message uses. The full contract lives in the `/hyperframes-core` skill; the [appendix](/prompting/rules-and-anti-patterns) carries the rules that most often bite.
diff --git a/docs/prompting/rules-and-anti-patterns.mdx b/docs/prompting/rules-and-anti-patterns.mdx
index 0801c6b865..6e5d69e32c 100644
--- a/docs/prompting/rules-and-anti-patterns.mdx
+++ b/docs/prompting/rules-and-anti-patterns.mdx
@@ -10,7 +10,7 @@ Almost every rule below is taught in context somewhere earlier in this guide; a
The skills enforce these automatically, but if you hand-edit compositions or debug issues, these are the rules that matter:
1. **Register all timelines** on `window.__timelines` — the renderer can't seek animations it doesn't know about.
-2. **Video elements must be `muted`** — audio goes in separate `<audio>` elements so the renderer can mix it.
+2. **A video with sound keeps it on the `<video>`** — `data-has-audio="true"`, no `muted`. Use a separate `<audio>` for music, voiceover, replacement audio, J/L cuts, or audio detached in Studio. Silent footage and b-roll: `muted`.
3. **No `Math.random()`** — random values produce different frames on each render, breaking determinism. Use a seeded PRNG (e.g. mulberry32) if you need pseudo-random values. And when you want a stepped, stop-motion hold, quantize it on the **integer frame index**, not on elapsed seconds — seek times don't land on exact 1/fps doubles, so second-domain arithmetic drifts and the hold length stutters. See [Handmade, still deterministic](/prompting/motion#handmade-still-deterministic).
4. **Synchronous timeline construction** — no `async`/`await` or `fetch()` during GSAP timeline setup.
5. **Timed elements need `class="clip"`** — plus `data-start`, `data-duration`, and `data-track-index`.
diff --git a/docs/reference/html-schema.mdx b/docs/reference/html-schema.mdx
index 2c445aabcb..7c12049073 100644
--- a/docs/reference/html-schema.mdx
+++ b/docs/reference/html-schema.mdx
@@ -133,8 +133,7 @@ Video and audio may omit `data-duration` when their intrinsic duration is known
and the whole remaining source should play.
Do not imperatively call `play()`, `pause()`, or set `currentTime`.
-HyperFrames owns media playback and seeking. A video should be `muted` unless it
-is intentionally audible and declares `data-has-audio="true"`.
+HyperFrames owns media playback and seeking. A `<video>` with sound declares `data-has-audio="true"` and is not muted. Silent footage is `muted`. Every timed `<video>` must say one or the other (lint `video_missing_muted`).
Media can live inside nested composition markup. Keep element IDs unique across
the assembled project because render-time frame injection targets those IDs.
diff --git a/packages/cli/src/commands/__fixtures__/silent.mp4 b/packages/cli/src/commands/__fixtures__/silent.mp4
new file mode 100644
index 0000000000..a5e4423739
Binary files /dev/null and b/packages/cli/src/commands/__fixtures__/silent.mp4 differ
diff --git a/packages/cli/src/commands/init.test.ts b/packages/cli/src/commands/init.test.ts
index e9f0e1b75a..fe0f99220b 100644
--- a/packages/cli/src/commands/init.test.ts
+++ b/packages/cli/src/commands/init.test.ts
@@ -220,14 +220,63 @@ describe("hyperframes init flag rename", () => {
const res = runInit([target, "--non-interactive", "--skip-transcribe", "--video", clip]);
expect(res.status).toBe(0);
const html = readFileSync(join(target, "index.html"), "utf-8");
- expect(html).toContain('id="a-roll"');
- expect(html).toContain('src="clip.mp4"');
+ const video = html.match(/<video\b[^>]*>/)?.[0] ?? "";
+ expect(video).toContain('id="a-roll"');
+ expect(video).toContain('src="clip.mp4"');
+ expect(video).toContain('data-has-audio="true"');
+ expect(video).not.toMatch(/\bmuted\b/);
+ expect(html).not.toMatch(/<audio\b/);
expect(existsSync(join(target, "clip.mp4"))).toBe(true);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
+ it("scaffolds a silent --video as a muted clip with no audio element", () => {
+ const dir = mkdtempSync(join(tmpdir(), "hf-init-silent-"));
+ const target = join(dir, "proj");
+ const clip = join(dir, "silent.mp4");
+ copyFileSync(resolve(fileURLToPath(import.meta.url), "../__fixtures__/silent.mp4"), clip);
+ try {
+ const res = runInit([target, "--non-interactive", "--skip-transcribe", "--video", clip]);
+ expect(res.status).toBe(0);
+ const html = readFileSync(join(target, "index.html"), "utf-8");
+ const video = html.match(/<video\b[^>]*>/)?.[0] ?? "";
+ expect(video).toMatch(/\bmuted\b/);
+ expect(video).not.toContain("data-has-audio");
+ expect(html).not.toMatch(/<audio\b/);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ it("wires --audio into the composition without creating a video clip", () => {
+ const dir = mkdtempSync(join(tmpdir(), "hf-init-audio-test-"));
+ const target = join(dir, "proj");
+ const audio = join(dir, "track.wav");
+ copyFileSync(
+ resolve(
+ fileURLToPath(import.meta.url),
+ "../../../../producer/tests/audio-mux-parity/src/assets/tone.wav",
+ ),
+ audio,
+ );
+ try {
+ const res = runInit([target, "--non-interactive", "--skip-transcribe", "--audio", audio]);
+ expect(res.status).toBe(0);
+ const html = readFileSync(join(target, "index.html"), "utf-8");
+ expect(html).toMatch(/<audio\b[^>]*src="track\.wav"/);
+ expect(html).not.toMatch(/<video\b/);
+ // tone.wav is 3 s; the root and the <audio> both carry the probed length, not the 10 s default.
+ expect(html).toMatch(/<audio\b[^>]*data-duration="3"/);
+ expect(html).toMatch(/<div[^>]*id="root"[^>]*data-duration="3"/);
+ expect(html).not.toContain("__VIDEO_SRC__");
+ expect(html).not.toContain("__AUDIO_SRC__");
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
it("-v works as the short alias for --video", () => {
const dir = mkdtempSync(join(tmpdir(), "hf-init-test-"));
const target = join(dir, "proj");
diff --git a/packages/cli/src/commands/init.ts b/packages/cli/src/commands/init.ts
index 8168c0f598..22b82baa7d 100644
--- a/packages/cli/src/commands/init.ts
+++ b/packages/cli/src/commands/init.ts
@@ -7,6 +7,7 @@ import { failCommand, finishCommand } from "../utils/commandResult.js";
import { writeNewFileSync } from "../utils/writeNewFile.js";
import { defineCommand, runCommand } from "citty";
import type { Example } from "./_examples.js";
+import { patchMediaPlaceholders, type InitMediaOptions } from "./initMedia.js";
export const examples: Example[] = [
["Create a project with the interactive wizard", "hyperframes init my-video"],
@@ -82,7 +83,8 @@ const DEFAULT_META: VideoMeta = {
width: 1920,
height: 1080,
fps: 30,
- hasAudio: false,
+ // ffprobe unavailable: assume sound. A wrong guess fails loudly at RENDER (lint catches it only once #3638 lands); assuming silence would mute real audio.
+ hasAudio: true,
videoCodec: "h264",
};
@@ -348,31 +350,12 @@ function writeTailwindSupport(destDir: string): void {
}
}
-function patchVideoSrc(
- dir: string,
- videoFilename: string | undefined,
- durationSeconds?: number,
-): void {
+function patchVideoSrc(dir: string, media: InitMediaOptions): void {
const htmlFiles = readdirSync(dir, { withFileTypes: true, recursive: true })
.filter((e) => e.isFile() && e.name.endsWith(".html"))
.map((e) => join(e.parentPath, e.name));
-
for (const file of htmlFiles) {
- let content = readFileSync(file, "utf-8");
- if (videoFilename) {
- content = content.replaceAll("__VIDEO_SRC__", videoFilename);
- } else {
- // Remove video elements with placeholder src
- content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/video>/g, "");
- content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
- // Remove audio elements with placeholder src
- content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/audio>/g, "");
- content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
- }
- // Patch duration — use probed duration or default
- const dur = durationSeconds ? String(Math.round(durationSeconds * 100) / 100) : "10";
- content = content.replaceAll("__VIDEO_DURATION__", dur);
- writeFileSync(file, content, "utf-8");
+ writeFileSync(file, patchMediaPlaceholders(readFileSync(file, "utf-8"), media), "utf-8");
}
}
@@ -404,7 +387,7 @@ async function handleVideoFile(
);
}
} else {
- const msg = `ffprobe not found — using defaults (1920x1080, 5s, 30fps). Install: ${getFFmpegInstallHint()}`;
+ const msg = `ffprobe not found — using defaults (1920x1080, 5s, 30fps) and assuming the video has sound (data-has-audio="true"); if it is silent, replace that with muted. Install: ${getFFmpegInstallHint()}`;
if (interactive) {
clack.log.warn(msg);
} else {
@@ -559,8 +542,7 @@ async function scaffoldProject(
destDir: string,
name: string,
templateId: string,
- localVideoName: string | undefined,
- durationSeconds?: number,
+ media: InitMediaOptions,
tailwind = false,
resolution?: CanvasResolution,
authoringSkill?: string,
@@ -576,7 +558,7 @@ async function scaffoldProject(
} else {
await fetchRemoteTemplate(templateId, destDir);
}
- patchVideoSrc(destDir, localVideoName, durationSeconds);
+ patchVideoSrc(destDir, media);
if (tailwind) writeTailwindSupport(destDir);
if (resolution) applyResolutionPreset(destDir, resolution);
@@ -863,6 +845,7 @@ export default defineCommand({
mkdirSync(destDir, { recursive: true });
let localVideoName: string | undefined;
+ let videoHasAudio = true;
let videoDuration: number | undefined;
let sourceFilePath: string | undefined;
@@ -871,6 +854,7 @@ export default defineCommand({
sourceFilePath = videoPath;
const result = await handleVideoFile(videoPath, destDir, false);
localVideoName = result.localVideoName;
+ videoHasAudio = result.meta.hasAudio;
videoDuration = result.meta.durationSeconds;
console.log(
`Video: ${result.meta.width}x${result.meta.height}, ${result.meta.durationSeconds.toFixed(1)}s`,
@@ -881,6 +865,8 @@ export default defineCommand({
if (audioPath) {
sourceFilePath = audioPath;
copyFileSync(audioPath, resolve(destDir, basename(audioPath)));
+ const { getMediaDurationSeconds } = await import("../whisper/transcribe.js");
+ videoDuration = getMediaDurationSeconds(audioPath) ?? undefined;
console.log(`Audio: ${basename(audioPath)}`);
}
@@ -907,12 +893,16 @@ export default defineCommand({
// Scaffold
try {
+ const media: InitMediaOptions = {
+ video: localVideoName ? { filename: localVideoName, hasAudio: videoHasAudio } : undefined,
+ audio: audioPath ? { filename: basename(audioPath) } : undefined,
+ durationSeconds: videoDuration,
+ };
await scaffoldProject(
destDir,
basename(destDir),
templateId,
- localVideoName,
- videoDuration,
+ media,
tailwind,
resolutionPreset,
args.skill,
@@ -1018,6 +1008,7 @@ export default defineCommand({
// 2. Video/audio file handling (only via --video/--audio flags, no interactive prompt)
let localVideoName: string | undefined;
+ let videoHasAudio = true;
let sourceFilePath: string | undefined;
let videoDuration: number | undefined;
@@ -1032,6 +1023,7 @@ export default defineCommand({
sourceFilePath = videoPath;
const result = await handleVideoFile(videoPath, destDir, true);
localVideoName = result.localVideoName;
+ videoHasAudio = result.meta.hasAudio;
videoDuration = result.meta.durationSeconds;
} else if (audioFlag) {
const audioPath = resolve(audioFlag);
@@ -1043,6 +1035,8 @@ export default defineCommand({
mkdirSync(destDir, { recursive: true });
sourceFilePath = audioPath;
copyFileSync(audioPath, resolve(destDir, basename(audioPath)));
+ const { getMediaDurationSeconds } = await import("../whisper/transcribe.js");
+ videoDuration = getMediaDurationSeconds(audioPath) ?? undefined;
clack.log.info(`Audio copied to ${c.accent(basename(audioPath))}`);
}
@@ -1124,12 +1118,16 @@ export default defineCommand({
spin.start(`Downloading example ${c.accent(templateId)}...`);
}
try {
+ const media: InitMediaOptions = {
+ video: localVideoName ? { filename: localVideoName, hasAudio: videoHasAudio } : undefined,
+ audio: audioFlag ? { filename: basename(audioFlag) } : undefined,
+ durationSeconds: videoDuration,
+ };
await scaffoldProject(
destDir,
name,
templateId,
- localVideoName,
- videoDuration,
+ media,
tailwind,
resolutionPreset,
args.skill,
diff --git a/packages/cli/src/commands/initMedia.test.ts b/packages/cli/src/commands/initMedia.test.ts
new file mode 100644
index 0000000000..6484c78158
--- /dev/null
+++ b/packages/cli/src/commands/initMedia.test.ts
@@ -0,0 +1,171 @@
+import { describe, expect, it } from "vitest";
+import { patchMediaPlaceholders } from "./initMedia.js";
+import { readdirSync, readFileSync, existsSync } from "node:fs";
+import { join, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const NEW_TEMPLATE = `<div id="root">
+ <video
+ id="a-roll"
+ class="clip"
+ src="__VIDEO_SRC__"
+ playsinline
+ data-has-audio="true"
+ data-duration="__VIDEO_DURATION__"
+ data-track-index="0"
+ ></video>
+ <audio
+ id="a-roll-audio"
+ src="__AUDIO_SRC__"
+ data-start="0"
+ data-duration="__VIDEO_DURATION__"
+ data-track-index="2"
+ data-volume="1"
+ ></audio>
+</div>`;
+
+const LEGACY_TEMPLATE = `<div id="root">
+ <video id="a-roll" class="clip" src="__VIDEO_SRC__" muted playsinline data-duration="__VIDEO_DURATION__"></video>
+ <audio id="a-roll-audio" src="__VIDEO_SRC__" data-start="0" data-duration="__VIDEO_DURATION__"></audio>
+</div>`;
+
+const videoTag = (html: string) => html.match(/<video\b[^>]*>/)?.[0] ?? "";
+
+describe("patchMediaPlaceholders — new template", () => {
+ it("video with audio: one audible <video>, no <audio>", () => {
+ const out = patchMediaPlaceholders(NEW_TEMPLATE, {
+ video: { filename: "talk.mp4", hasAudio: true },
+ durationSeconds: 6,
+ });
+ expect(videoTag(out)).toContain('data-has-audio="true"');
+ expect(videoTag(out)).not.toMatch(/\bmuted\b/);
+ expect(videoTag(out)).toContain('src="talk.mp4"');
+ expect(out).not.toMatch(/<audio\b/);
+ expect(out).toContain('data-duration="6"');
+ expect(out).not.toContain("__");
+ });
+
+ it("silent video: muted <video>, no data-has-audio, no <audio>", () => {
+ const out = patchMediaPlaceholders(NEW_TEMPLATE, {
+ video: { filename: "silent.mp4", hasAudio: false },
+ durationSeconds: 3,
+ });
+ expect(videoTag(out)).toMatch(/\bmuted\b/);
+ expect(videoTag(out)).not.toContain("data-has-audio");
+ expect(out).not.toMatch(/<audio\b/);
+ });
+
+ it("keeps every other attribute and the closing > of a multi-line tag", () => {
+ const out = patchMediaPlaceholders(NEW_TEMPLATE, {
+ video: { filename: "silent.mp4", hasAudio: false },
+ });
+ const tag = videoTag(out);
+ for (const attr of ['id="a-roll"', 'class="clip"', "playsinline", 'data-track-index="0"']) {
+ expect(tag).toContain(attr);
+ }
+ expect(out).toMatch(/<video\b[^>]*>\s*<\/video>/);
+ });
+
+ it("audio only: one <audio> on the file, no <video>", () => {
+ const out = patchMediaPlaceholders(NEW_TEMPLATE, { audio: { filename: "track.wav" } });
+ expect(out).toMatch(/<audio\b[^>]*src="track\.wav"/);
+ expect(out).not.toMatch(/<video\b/);
+ expect(out).not.toContain("__");
+ });
+
+ it("no media: strips both placeholders and defaults duration to 10", () => {
+ const out = patchMediaPlaceholders(
+ NEW_TEMPLATE.replace('<div id="root">', '<div id="root" data-duration="__VIDEO_DURATION__">'),
+ {},
+ );
+ expect(out).not.toMatch(/<video\b|<audio\b/);
+ expect(out).not.toContain("__");
+ expect(out).toContain('<div id="root" data-duration="10">');
+ });
+
+ it('normalises muted="" and a stray data-has-audio on the placeholder', () => {
+ const html = `<video src="__VIDEO_SRC__" muted="" data-has-audio="false" data-x="1"></video>`;
+ const out = patchMediaPlaceholders(html, { video: { filename: "a.mp4", hasAudio: true } });
+ expect(out).toBe(`<video src="a.mp4" data-x="1" data-has-audio="true"></video>`);
+ });
+});
+
+describe("patchMediaPlaceholders — muted token matching", () => {
+ it("does not strip the word muted from another attribute's value", () => {
+ const html = `<video src="__VIDEO_SRC__" class="a muted b" muted></video>`;
+ const out = patchMediaPlaceholders(html, { video: { filename: "a.mp4", hasAudio: true } });
+ expect(out).toBe(`<video src="a.mp4" class="a muted b" data-has-audio="true"></video>`);
+ });
+});
+
+describe("patchMediaPlaceholders — legacy remote templates", () => {
+ it("video with audio: keeps the legacy split untouched", () => {
+ const out = patchMediaPlaceholders(LEGACY_TEMPLATE, {
+ video: { filename: "talk.mp4", hasAudio: true },
+ });
+ expect(videoTag(out)).toMatch(/\bmuted\b/);
+ expect(out).toMatch(/<audio\b[^>]*src="talk\.mp4"/);
+ });
+
+ it("silent video: strips the legacy <audio> (a silent file would fail the render)", () => {
+ const out = patchMediaPlaceholders(LEGACY_TEMPLATE, {
+ video: { filename: "silent.mp4", hasAudio: false },
+ });
+ expect(out).not.toMatch(/<audio\b/);
+ expect(videoTag(out)).toMatch(/\bmuted\b/);
+ });
+
+ it("audio only: fills the legacy audio slot and drops the video", () => {
+ const out = patchMediaPlaceholders(LEGACY_TEMPLATE, { audio: { filename: "track.wav" } });
+ expect(out).toMatch(/<audio\b[^>]*src="track\.wav"/);
+ expect(out).not.toMatch(/<video\b/);
+ });
+});
+
+describe("registry examples use the audio-on-video form", () => {
+ const examplesDir = resolve(fileURLToPath(import.meta.url), "../../../../../registry/examples");
+ const withVideoSlot = readdirSync(examplesDir)
+ .map((name) => join(examplesDir, name, "index.html"))
+ .filter((file) => existsSync(file) && readFileSync(file, "utf-8").includes("__VIDEO_SRC__"));
+
+ it("finds the examples that carry a video slot", () => {
+ expect(withVideoSlot.length).toBeGreaterThan(0);
+ });
+
+ it.each(withVideoSlot)("%s has no muted placeholder video and no __VIDEO_SRC__ audio", (file) => {
+ const html = readFileSync(file, "utf-8");
+ const video = html.match(/<video\b[^>]*src="__VIDEO_SRC__"[^>]*>/)?.[0] ?? "";
+ expect(video).toContain('data-has-audio="true"');
+ expect(video).not.toMatch(/\bmuted\b/);
+ expect(html).not.toMatch(/<audio\b[^>]*src="__VIDEO_SRC__"/);
+ });
+});
+
+describe("patchMediaPlaceholders — hostile filenames", () => {
+ const cases: Array<[string, string]> = [
+ ["beat$$.wav", "beat$$.wav"],
+ ["cut$'s.mp4", "cut$'s.mp4"],
+ ["Track #1.wav", "Track%20%231.wav"],
+ ['a"b.mp4', "a%22b.mp4"],
+ ["q?x&y.mp4", "q%3Fx&y.mp4"],
+ ["p$&q.mp4", "p$&q.mp4"],
+ ];
+ it.each(cases)("video %s is inserted literally as an encoded src", (name, encoded) => {
+ const out = patchMediaPlaceholders(NEW_TEMPLATE, {
+ video: { filename: name, hasAudio: true },
+ durationSeconds: 6,
+ });
+ expect(out).toContain(`src="${encoded}"`);
+ expect(out).toContain('data-duration="6"');
+ expect(out).toContain("</div>");
+ expect(out).not.toContain("__");
+ });
+ it.each(cases)("audio %s is inserted literally as an encoded src", (name, encoded) => {
+ const out = patchMediaPlaceholders(NEW_TEMPLATE, {
+ audio: { filename: name },
+ durationSeconds: 6,
+ });
+ expect(out).toContain(`src="${encoded}"`);
+ expect(out).not.toContain("__");
+ });
+});
diff --git a/packages/cli/src/commands/initMedia.ts b/packages/cli/src/commands/initMedia.ts
new file mode 100644
index 0000000000..ee2e0b216a
--- /dev/null
+++ b/packages/cli/src/commands/initMedia.ts
@@ -0,0 +1,72 @@
+// The one place `hyperframes init` (and the template-preview script) rewrites media placeholders.
+// New templates: an audible <video data-has-audio="true"> on __VIDEO_SRC__ plus an audio-only slot on __AUDIO_SRC__.
+// Legacy templates (muted <video> + <audio> both on __VIDEO_SRC__) still come from older remote registry examples.
+
+export interface InitMediaOptions {
+ video?: { filename: string; hasAudio: boolean };
+ audio?: { filename: string };
+ durationSeconds?: number;
+}
+
+const VIDEO_SRC = "__VIDEO_SRC__";
+const AUDIO_SRC = "__AUDIO_SRC__";
+const DURATION = "__VIDEO_DURATION__";
+
+// encodeURI leaves `#` and `?` alone, which would truncate the path at the URL fragment/query.
+function toSrcValue(filename: string): string {
+ return encodeURI(filename).replace(/#/g, "%23").replace(/\?/g, "%3F");
+}
+
+function openTag(tag: "video" | "audio", placeholder: string): string {
+ return `<${tag}\\b[^>]*src="${placeholder}"[^>]*>`;
+}
+
+// Element with a closing tag first, then a bare open tag — two passes so a bare tag never swallows a later element.
+function stripPlaceholder(html: string, tag: "video" | "audio", placeholder: string): string {
+ const open = openTag(tag, placeholder);
+ return html
+ .replace(new RegExp(`${open}[\\s\\S]*?</${tag}>`, "g"), "")
+ .replace(new RegExp(open, "g"), "");
+}
+
+function setVideoAudio(html: string, hasAudio: boolean): string {
+ return html.replace(new RegExp(openTag("video", VIDEO_SRC), "g"), (tag) => {
+ // Attribute-by-attribute so a quoted value (class="a muted b") is consumed whole and never matched.
+ const cleaned = tag.replace(
+ /\s+([^\s=>"']+)(?:\s*=\s*(?:"[^"]*"|'[^']*'|[^\s>"']+))?/g,
+ (attr, name: string) => (name === "muted" || name === "data-has-audio" ? "" : attr),
+ );
+ const attr = hasAudio ? ' data-has-audio="true"' : " muted";
+ return cleaned.replace(/\s*>$/, `${attr}>`);
+ });
+}
+
+export function patchMediaPlaceholders(html: string, opts: InitMediaOptions): string {
+ const legacyAudioSlot = new RegExp(openTag("audio", VIDEO_SRC)).test(html);
+ let out = html;
+ if (opts.video) {
+ out = stripPlaceholder(out, "audio", AUDIO_SRC);
+ if (legacyAudioSlot) {
+ // Legacy split: keep it for a file with sound; a silent file's <audio> fails the render preflight.
+ if (!opts.video.hasAudio) out = stripPlaceholder(out, "audio", VIDEO_SRC);
+ } else {
+ out = setVideoAudio(out, opts.video.hasAudio);
+ }
+ const videoSrc = toSrcValue(opts.video.filename);
+ out = out.replaceAll(VIDEO_SRC, () => videoSrc);
+ } else if (opts.audio) {
+ out = stripPlaceholder(out, "video", VIDEO_SRC);
+ const audioSrc = toSrcValue(opts.audio.filename);
+ out = out.replaceAll(AUDIO_SRC, () => audioSrc);
+ // Legacy templates carry the audio slot on __VIDEO_SRC__.
+ out = out.replaceAll(VIDEO_SRC, () => audioSrc);
+ } else {
+ out = stripPlaceholder(out, "video", VIDEO_SRC);
+ out = stripPlaceholder(out, "audio", VIDEO_SRC);
+ out = stripPlaceholder(out, "audio", AUDIO_SRC);
+ }
+ const duration = opts.durationSeconds
+ ? String(Math.round(opts.durationSeconds * 100) / 100)
+ : "10";
+ return out.replaceAll(DURATION, duration);
+}
diff --git a/packages/cli/src/docs/data-attributes.md b/packages/cli/src/docs/data-attributes.md
index 90526c8890..50712d6285 100644
--- a/packages/cli/src/docs/data-attributes.md
+++ b/packages/cli/src/docs/data-attributes.md
@@ -13,7 +13,7 @@ Core attributes for controlling element timing and behavior.
- `data-media-start="2"` — Media playback offset / trim point (seconds)
- `data-volume="0.8"` — Audio/video gain. `1` is 0 dB, `0` is silence, and values above `1` boost up to `3.98` (+12 dB). For ducking and shaped envelopes use the `data-automation` volume lane (`hyperframes-core` skill, `creator-editing-recipes.md`)
- `data-fade-in="0.5"` / `data-fade-out="1"` — Clip-edge fades (seconds): linear gain ramps anchored to the clip's start and end, multiplied on top of `data-volume` and any volume lane
-- `data-has-audio="true"` — Indicates video has an audio track
+- `data-has-audio="true"` — Keeps the file's sound on this clip (the default for footage with sound). Silent footage uses `muted` instead.
## Composition
diff --git a/packages/cli/src/templates/_shared/AGENTS.md b/packages/cli/src/templates/_shared/AGENTS.md
index ff89fb99e5..30a6da67e5 100644
--- a/packages/cli/src/templates/_shared/AGENTS.md
+++ b/packages/cli/src/templates/_shared/AGENTS.md
@@ -102,6 +102,6 @@ Fix all errors before presenting the result. Warnings should be reviewed before
Scene timelines manually added to this root must not be paused. A paused
child does not advance when the root is seeked. The runtime activates
registered composition siblings, not arbitrary nested scene timelines.
-4. Videos use `muted` with a separate `<audio>` element for the audio track
+4. A video with sound keeps it on the `<video>` (`data-has-audio="true"`, no `muted`). Use a separate `<audio>` for music, voiceover, replacement audio, J/L cuts, or audio detached in Studio. Silent footage and b-roll: `muted`.
5. Sub-compositions use `data-composition-src="compositions/file.html"` to reference other HTML files
6. Only deterministic logic — no `Date.now()`, no `Math.random()`, no network fetches
diff --git a/packages/cli/src/templates/_shared/CLAUDE.md b/packages/cli/src/templates/_shared/CLAUDE.md
index ff89fb99e5..30a6da67e5 100644
--- a/packages/cli/src/templates/_shared/CLAUDE.md
+++ b/packages/cli/src/templates/_shared/CLAUDE.md
@@ -102,6 +102,6 @@ Fix all errors before presenting the result. Warnings should be reviewed before
Scene timelines manually added to this root must not be paused. A paused
child does not advance when the root is seeked. The runtime activates
registered composition siblings, not arbitrary nested scene timelines.
-4. Videos use `muted` with a separate `<audio>` element for the audio track
+4. A video with sound keeps it on the `<video>` (`data-has-audio="true"`, no `muted`). Use a separate `<audio>` for music, voiceover, replacement audio, J/L cuts, or audio detached in Studio. Silent footage and b-roll: `muted`.
5. Sub-compositions use `data-composition-src="compositions/file.html"` to reference other HTML files
6. Only deterministic logic — no `Date.now()`, no `Math.random()`, no network fetches
diff --git a/packages/cli/src/templates/from-file/index.html b/packages/cli/src/templates/from-file/index.html
index 04c92deb98..2f3c251d4d 100644
--- a/packages/cli/src/templates/from-file/index.html
+++ b/packages/cli/src/templates/from-file/index.html
@@ -41,15 +41,16 @@
id="a-roll"
class="clip"
src="__VIDEO_SRC__"
- muted
playsinline
+ data-has-audio="true"
+ data-start="0"
data-duration="__VIDEO_DURATION__"
data-track-index="0"
style="position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover"
></video>
<audio
id="a-roll-audio"
- src="__VIDEO_SRC__"
+ src="__AUDIO_SRC__"
data-start="0"
data-duration="__VIDEO_DURATION__"
data-track-index="2"
diff --git a/packages/cli/src/whisper/transcribe.ts b/packages/cli/src/whisper/transcribe.ts
index f22df4e7e5..1c890a0292 100644
--- a/packages/cli/src/whisper/transcribe.ts
+++ b/packages/cli/src/whisper/transcribe.ts
@@ -159,7 +159,7 @@ export function resolveAudioPreparationTimeoutMs(durationSeconds: number | null)
);
}
-function getMediaDurationSeconds(filePath: string): number | null {
+export function getMediaDurationSeconds(filePath: string): number | null {
try {
const ffprobePath = findFFprobe();
if (!ffprobePath) return null;
diff --git a/packages/core/docs/quickstart-template.html b/packages/core/docs/quickstart-template.html
index 86dea54abb..c70edcc068 100644
--- a/packages/core/docs/quickstart-template.html
+++ b/packages/core/docs/quickstart-template.html
@@ -73,7 +73,7 @@
(higher = in front, clips on same track cannot overlap)
REQUIRED on <video>:
- muted — framework manages audio via separate <audio> element
+ data-has-audio="true" — keep the clip's own sound (no muted); silent footage: muted
playsinline — prevents fullscreen on mobile
REQUIRED:
@@ -82,7 +82,7 @@
DO NOT:
- Call video.play(), video.pause(), or set video.currentTime in scripts
- Toggle visibility in scripts
- - Use the video element for audio (use a separate <audio> element)
+ - Add muted to a video whose sound should play
-->
<video
id="el-video"
@@ -91,7 +91,7 @@
data-duration="10"
data-track-index="0"
src="my-video.mp4"
- muted
+ data-has-audio="true"
playsinline
></video>
@@ -117,7 +117,8 @@
════════════════════════════════════════════════════
Audio is invisible — no class="clip" needed (no visual to hide).
- Use a separate <audio> element even if the source is the same .mp4 file.
+ Use <audio> for music, voiceover, or replacement sound. A video with sound
+ keeps it on the <video> (data-has-audio="true").
The framework syncs audio playback to the timeline position.
Optional: data-volume="0.5" (0-1, default 1)
@@ -128,7 +129,7 @@
data-duration="15"
data-track-index="1"
data-volume="0.8"
- src="my-video.mp4"
+ src="my-music.mp3"
></audio>
<!--
diff --git a/packages/engine/src/services/audioMixer.ts b/packages/engine/src/services/audioMixer.ts
index 85f60f1853..6f58cf75ad 100644
--- a/packages/engine/src/services/audioMixer.ts
+++ b/packages/engine/src/services/audioMixer.ts
@@ -5,6 +5,7 @@
* Processes and mixes audio tracks using FFmpeg.
*/
+import { isSelfOrAncestorHidden } from "./mediaHidden.js";
import { closeSync, existsSync, mkdirSync, mkdtempSync, openSync, rmSync, writeFileSync } from "fs";
import { join, dirname, isAbsolute, relative } from "path";
import { parseHTML } from "linkedom";
@@ -588,12 +589,7 @@ export function parseAudioElements(html: string): AudioElement[] {
const parseEnd = (raw: string | null): number => {
return parseStrictFiniteTimingNumber(raw) ?? 0;
};
- const isHidden = (el: AudioMediaElement): boolean => {
- for (let current: AudioMediaElement | null = el; current; current = current.parentElement) {
- if (current.hasAttribute("data-hidden")) return true;
- }
- return false;
- };
+ const isHidden = isSelfOrAncestorHidden;
// Resolved once per parse. A group element carrying `data-hidden` drops
// every member from the render (RULES: mute-by-drop, never
diff --git a/packages/engine/src/services/mediaHidden.ts b/packages/engine/src/services/mediaHidden.ts
new file mode 100644
index 0000000000..0fe24a38e3
--- /dev/null
+++ b/packages/engine/src/services/mediaHidden.ts
@@ -0,0 +1,11 @@
+interface HiddenCheckEl {
+ hasAttribute(name: string): boolean;
+ parentElement: HiddenCheckEl | null;
+}
+
+export function isSelfOrAncestorHidden(el: HiddenCheckEl): boolean {
+ for (let current: HiddenCheckEl | null = el; current; current = current.parentElement) {
+ if (current.hasAttribute("data-hidden")) return true;
+ }
+ return false;
+}
diff --git a/packages/engine/src/services/videoFrameExtractor.test.ts b/packages/engine/src/services/videoFrameExtractor.test.ts
index e4c8635ded..925da5e65e 100644
--- a/packages/engine/src/services/videoFrameExtractor.test.ts
+++ b/packages/engine/src/services/videoFrameExtractor.test.ts
@@ -3700,3 +3700,18 @@ describe("getFrameAtTime — IEEE 754 boundary precision", () => {
expect(getFrameAtTime(extracted, 3, 0)).toBeNull();
});
});
+
+describe("parseVideoElements hidden flag", () => {
+ it("marks a video hidden on itself or through a hidden ancestor", () => {
+ const videos = parseVideoElements(
+ `<div data-hidden><video id="a" src="a.mp4" data-has-audio="true"></video></div>` +
+ `<video id="b" src="b.mp4" data-has-audio="true" data-hidden></video>` +
+ `<video id="c" src="c.mp4" data-has-audio="true"></video>`,
+ );
+ expect(videos.map((v) => [v.id, v.hidden])).toEqual([
+ ["a", true],
+ ["b", true],
+ ["c", undefined],
+ ]);
+ });
+});
diff --git a/packages/engine/src/services/videoFrameExtractor.ts b/packages/engine/src/services/videoFrameExtractor.ts
index 32d182ed51..56d077dd4e 100644
--- a/packages/engine/src/services/videoFrameExtractor.ts
+++ b/packages/engine/src/services/videoFrameExtractor.ts
@@ -6,6 +6,7 @@
* Videos are replaced with <img> elements during capture.
*/
+import { isSelfOrAncestorHidden } from "./mediaHidden.js";
import { copyFileSync, existsSync, linkSync, mkdirSync, rmSync } from "fs";
import { join } from "path";
import { parseHTML } from "linkedom";
@@ -81,6 +82,7 @@ export interface VideoElement {
playbackRate?: RateSpec;
loop: boolean;
hasAudio: boolean;
+ hidden?: boolean;
}
export interface ExtractedFrames {
@@ -720,6 +722,7 @@ export function parseVideoElements(html: string): VideoElement[] {
playbackRate: readElementRateSpec(el),
loop: el.hasAttribute("loop"),
hasAudio: hasAudioAttr === "true",
+ ...(isSelfOrAncestorHidden(el) ? { hidden: true } : {}),
});
}
diff --git a/packages/lint/src/project.test.ts b/packages/lint/src/project.test.ts
index 079c076416..4dc8669eb1 100644
--- a/packages/lint/src/project.test.ts
+++ b/packages/lint/src/project.test.ts
@@ -834,3 +834,51 @@ describe("templating tokens are checked on the raw src, before cleanAssetUrl", (
expect(c.has("missing_local_asset")).toBe(true);
});
});
+
+describe("audio-aware project rules", () => {
+ const composition = (body: string) => validHtml().replace("></div>", `>${body}</div>`);
+ const codesFor = async (html: string, files: string[] = []) => {
+ const project = makeProject(html);
+ for (const file of files) writeFileSync(join(project, file), "x");
+ const { results } = await lintProject(project);
+ return results.flatMap((result) => result.result.findings.map((f) => f.code));
+ };
+ const video = (attrs: string, start = 0) =>
+ `<video id="v${start}${attrs.length}" src="a.mp4" data-start="${start}" data-duration="5" data-track-index="0" ${attrs}></video>`;
+
+ describe("duplicate_audio_track", () => {
+ it("warns when audible videos overlap on one track", async () => {
+ const codes = await codesFor(
+ composition(video('data-has-audio="true"', 0) + video('data-has-audio="true"', 2)),
+ );
+ expect(codes).toContain("duplicate_audio_track");
+ });
+
+ it("stays quiet when one of the overlapping videos is muted", async () => {
+ const codes = await codesFor(
+ composition(video('data-has-audio="true"', 0) + video("muted", 2)),
+ );
+ expect(codes).not.toContain("duplicate_audio_track");
+ });
+
+ it("stays quiet when an overlapping video is explicitly non-audible", async () => {
+ const codes = await codesFor(
+ composition(video('data-has-audio="true"', 0) + video('data-has-audio="false"', 2)),
+ );
+ expect(codes).not.toContain("duplicate_audio_track");
+ });
+ });
+
+ describe("audio_file_without_element", () => {
+ const audibleRoll = video('data-has-audio="true"');
+ it("does not claim silence when an audible video carries the sound", async () => {
+ const codes = await codesFor(composition(audibleRoll), ["audio.mp3"]);
+ expect(codes).not.toContain("audio_file_without_element");
+ });
+
+ it("still warns when the only video is muted", async () => {
+ const codes = await codesFor(composition(video("muted")), ["audio.mp3"]);
+ expect(codes).toContain("audio_file_without_element");
+ });
+ });
+});
diff --git a/packages/lint/src/project.ts b/packages/lint/src/project.ts
index 72a9981dea..4d5eccdd48 100644
--- a/packages/lint/src/project.ts
+++ b/packages/lint/src/project.ts
@@ -29,7 +29,7 @@ import type {
HyperframeLinterOptions,
} from "./types.js";
import type { ParsableDocumentLike } from "@hyperframes/parsers/sub-composition-validity";
-import { mediaSrcTagRe } from "./utils";
+import { isAudibleVideoTag, mediaSrcTagRe } from "./utils";
/** Adapts linkedom's `parseHTML` to the `checkSubCompositionUsability` contract. */
function parseSubCompHtml(html: string): ParsableDocumentLike {
@@ -339,8 +339,11 @@ function lintProjectAudioFiles(
if (audioFiles.length === 0) return findings;
const hasAudioElement = htmlSources.some(({ html }) => /<audio\b/i.test(html));
+ const hasAudibleVideo = htmlSources.some(({ html }) =>
+ (html.match(/<video\b[^>]*>/gi) ?? []).some(isAudibleVideoTag),
+ );
- if (!hasAudioElement) {
+ if (!hasAudioElement && !hasAudibleVideo) {
findings.push({
code: "audio_file_without_element",
severity: "warning",
@@ -549,10 +552,11 @@ function lintDuplicateAudioTracks(htmlSources: HtmlSource[]): HyperframeLintFind
const seen = new Set<string>();
for (const { html } of htmlSources) {
- const audioTagRe = /<audio\b[^>]*>/gi;
+ const soundTagRe = /<(?:audio|video)\b[^>]*>/gi;
let match: RegExpExecArray | null;
- while ((match = audioTagRe.exec(html)) !== null) {
+ while ((match = soundTagRe.exec(html)) !== null) {
const tag = match[0];
+ if (/^<video/i.test(tag) && !isAudibleVideoTag(tag)) continue;
const trackStr = extractAttr(tag, "data-track-index");
const startStr = extractAttr(tag, "data-start");
const durStr = extractAttr(tag, "data-duration");
@@ -579,7 +583,7 @@ function lintDuplicateAudioTracks(htmlSources: HtmlSource[]): HyperframeLintFind
findings.push({
code: "duplicate_audio_track",
severity: "warning",
- message: `Multiple <audio> elements on track ${a.trackIndex} overlap (${a.src} at ${a.start}-${Number.isFinite(a.end) ? a.end.toFixed(1) : "end"}s, ${b.src} at ${b.start}-${Number.isFinite(b.end) ? b.end.toFixed(1) : "end"}s). This causes layered audio playback.`,
+ message: `Multiple audible <audio>/<video> elements on track ${a.trackIndex} overlap (${a.src} at ${a.start}-${Number.isFinite(a.end) ? a.end.toFixed(1) : "end"}s, ${b.src} at ${b.start}-${Number.isFinite(b.end) ? b.end.toFixed(1) : "end"}s). This causes layered audio playback.`,
fixHint: "Use non-overlapping time windows or different track indices.",
});
}
diff --git a/packages/lint/src/rules/media.test.ts b/packages/lint/src/rules/media.test.ts
index 6d27a56043..0f55557ad5 100644
--- a/packages/lint/src/rules/media.test.ts
+++ b/packages/lint/src/rules/media.test.ts
@@ -293,6 +293,93 @@ describe("media rules", () => {
expect(finding?.elementId).toBe("demo-video");
});
+ it("video_missing_muted recommends data-has-audio first", async () => {
+ const html = `
+<html><body>
+ <div id="root" data-composition-id="c1" data-width="1920" data-height="1080">
+ <video id="v" data-start="0" data-duration="5" src="clip.mp4" playsinline></video>
+ </div>
+ <script>window.__timelines = window.__timelines || {}; window.__timelines["c1"] = gsap.timeline({ paused: true });</script>
+</body></html>`;
+ const result = await lintHyperframeHtml(html);
+ const finding = result.findings.find((f) => f.code === "video_missing_muted");
+ expect(finding?.severity).toBe("error");
+ expect(finding?.message).toContain("declares neither muted nor data-has-audio");
+ expect(finding?.message).not.toContain("separate <audio>");
+ expect(finding?.fixHint?.indexOf("data-has-audio")).toBeLessThan(
+ finding?.fixHint?.indexOf("muted") ?? -1,
+ );
+ });
+
+ it("video_audio_double_source suggests removing the <audio> first", async () => {
+ const html = `
+<html><body>
+ <div id="root" data-composition-id="c1" data-width="1920" data-height="1080">
+ <video id="v" data-start="0" data-duration="5" data-has-audio="true" src="clip.mp4" playsinline></video>
+ <audio id="a" data-start="0" data-duration="5" src="clip.mp4"></audio>
+ </div>
+ <script>window.__timelines = window.__timelines || {}; window.__timelines["c1"] = gsap.timeline({ paused: true });</script>
+</body></html>`;
+ const result = await lintHyperframeHtml(html);
+ const finding = result.findings.find((f) => f.code === "video_audio_double_source");
+ expect(finding?.fixHint?.startsWith("Remove the <audio>")).toBe(true);
+ });
+
+ const wrap = (body: string) => `
+<html><body>
+ <div id="root" data-composition-id="c1" data-width="1920" data-height="1080">
+ ${body}
+ </div>
+ <script>window.__timelines = window.__timelines || {}; window.__timelines["c1"] = gsap.timeline({ paused: true });</script>
+</body></html>`;
+ const codes = async (html: string) =>
+ (await lintHyperframeHtml(html)).findings.map((f) => f.code);
+
+ it.each(['data-has-audio="false"', 'data-has-audio=""', "data-has-audio"])(
+ "video_missing_muted stays quiet when the video declares its audio state (%s)",
+ async (attr) => {
+ const found = await codes(
+ wrap(`<video id="v" data-start="0" data-duration="5" ${attr} src="clip.mp4"></video>`),
+ );
+ expect(found).not.toContain("video_missing_muted");
+ },
+ );
+
+ it.each(['data-has-audio="false"', 'data-has-audio=""', "muted"])(
+ "video_audio_double_source ignores an explicitly silent video (%s)",
+ async (attr) => {
+ const found = await codes(
+ wrap(`<video id="v" data-start="0" data-duration="5" ${attr} src="clip.mp4"></video>
+ <audio id="a" data-start="0" data-duration="5" src="clip.mp4"></audio>`),
+ );
+ expect(found).not.toContain("video_audio_double_source");
+ },
+ );
+
+ it("video_audio_double_source ignores same-src clips that do not overlap in time", async () => {
+ const found = await codes(
+ wrap(`<video id="v" data-start="0" data-duration="4" data-has-audio="true" src="clip.mp4"></video>
+ <audio id="a" data-start="4" data-duration="4" src="clip.mp4"></audio>`),
+ );
+ expect(found).not.toContain("video_audio_double_source");
+ });
+
+ it("video_audio_double_source flags same-src clips that overlap in time", async () => {
+ const found = await codes(
+ wrap(`<video id="v" data-start="0" data-duration="4" data-has-audio="true" src="clip.mp4"></video>
+ <audio id="a" data-start="3" data-duration="4" src="clip.mp4"></audio>`),
+ );
+ expect(found).toContain("video_audio_double_source");
+ });
+
+ it("video_audio_double_source keeps flagging when a duration is not numeric", async () => {
+ const found = await codes(
+ wrap(`<video id="v" data-start="0" data-has-audio="true" src="clip.mp4"></video>
+ <audio id="a" data-start="9" data-duration="4" src="clip.mp4"></audio>`),
+ );
+ expect(found).toContain("video_audio_double_source");
+ });
+
it("does NOT flag <video> as nested in a void element with data-start (regression)", async () => {
// Regression: void elements like <img> have no closing tag, so the previous
// implementation kept them on the parent stack indefinitely and flagged any
diff --git a/packages/lint/src/rules/media.ts b/packages/lint/src/rules/media.ts
index abfd1a6419..6d492c7c14 100644
--- a/packages/lint/src/rules/media.ts
+++ b/packages/lint/src/rules/media.ts
@@ -1,5 +1,15 @@
import type { LintContext, HyperframeLintFinding, OpenTag } from "../context";
-import { readAttr, readDecodedAttr, stripJsComments, truncateSnippet, isMediaTag } from "../utils";
+import {
+ readAttr,
+ readDecodedAttr,
+ stripJsComments,
+ truncateSnippet,
+ isMediaTag,
+ hasAttrName,
+ isAudibleVideoTag,
+ mediaTimeWindow,
+ mediaWindowsOverlap,
+} from "../utils";
import { validateColorGradingContract } from "@hyperframes/parsers/color-grading-contract";
import { extractMediaSrcMutations } from "@hyperframes/parsers/composition";
import { parseHTML } from "linkedom";
@@ -37,12 +47,6 @@ function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
-function hasAttrName(tagSource: string, attr: string): boolean {
- const escaped = escapeRegExp(attr);
- const attrs = tagSource.replace(/^<\s*[a-z][\w:-]*/i, "");
- return new RegExp(`(?:^|\\s)${escaped}(?:\\s*=|\\s|/?>)`, "i").test(attrs);
-}
-
const IMAGE_SRC_EXT = new Set([
"jpg",
"jpeg",
@@ -513,15 +517,16 @@ export const mediaRules: Array<(ctx: LintContext) => HyperframeLintFinding[]> =
if (tag.name !== "video") continue;
const hasMuted = hasAttrName(tag.raw, "muted");
const hasDeclaredAudio = readAttr(tag.raw, "data-has-audio") === "true";
- if (!hasMuted && !hasDeclaredAudio && readAttr(tag.raw, "data-start")) {
+ const declaresAudioState = hasAttrName(tag.raw, "data-has-audio");
+ if (!hasMuted && !declaresAudioState && readAttr(tag.raw, "data-start")) {
const elementId = readAttr(tag.raw, "id") || undefined;
findings.push({
code: "video_missing_muted",
severity: "error",
- message: `<video${elementId ? ` id="${elementId}"` : ""}> has data-start but is not muted. Mark audible videos with data-has-audio="true"; otherwise keep video muted and use a separate <audio> element for sound.`,
+ message: `<video${elementId ? ` id="${elementId}"` : ""}> has data-start but declares neither muted nor data-has-audio. If the file has sound, add data-has-audio="true" (the sound stays on this clip). If it is silent, add muted.`,
elementId,
fixHint:
- 'Add the `muted` attribute for silent video, or add data-has-audio="true" when the video track should contribute audio.',
+ 'Add data-has-audio="true" when the file has sound (recommended — keeps picture and sound on one clip), or add `muted` for silent footage.',
snippet: truncateSnippet(tag.raw),
});
}
@@ -769,34 +774,37 @@ export const mediaRules: Array<(ctx: LintContext) => HyperframeLintFinding[]> =
// <audio> pointing to the same file, which causes double playback at runtime
({ tags }) => {
const findings: HyperframeLintFinding[] = [];
- const videoSources = new Map<string, { id?: string; raw: string }>();
- const audioSources = new Map<string, { id?: string; raw: string }>();
+ type Source = { id?: string; raw: string };
+ const videos: Array<[string, Source]> = [];
+ const audios: Array<[string, Source]> = [];
for (const tag of tags) {
if (!readAttr(tag.raw, "data-start")) continue;
const src = readAttr(tag.raw, "src");
if (!src) continue;
- const elementId = readAttr(tag.raw, "id") || undefined;
+ const source = { id: readAttr(tag.raw, "id") || undefined, raw: tag.raw };
if (tag.name === "video") {
- const isMuted = hasAttrName(tag.raw, "muted");
- if (!isMuted) {
- videoSources.set(src, { id: elementId, raw: tag.raw });
- }
+ if (isAudibleVideoTag(tag.raw)) videos.push([src, source]);
} else if (tag.name === "audio") {
- audioSources.set(src, { id: elementId, raw: tag.raw });
+ audios.push([src, source]);
}
}
- for (const [src, audioInfo] of audioSources) {
- const videoInfo = videoSources.get(src);
- if (!videoInfo) continue;
+ for (const [src, audioInfo] of audios) {
+ const match = videos.find(
+ ([videoSrc, video]) =>
+ videoSrc === src &&
+ mediaWindowsOverlap(mediaTimeWindow(video.raw), mediaTimeWindow(audioInfo.raw)),
+ );
+ if (!match) continue;
+ const videoInfo = match[1];
findings.push({
code: "video_audio_double_source",
severity: "error",
- message: `<audio${audioInfo.id ? ` id="${audioInfo.id}"` : ""}> and <video${videoInfo.id ? ` id="${videoInfo.id}"` : ""}> both point to the same source. The unmuted video already provides audio — the duplicate <audio> will cause double playback and echo.`,
+ message: `<audio${audioInfo.id ? ` id="${audioInfo.id}"` : ""}> and <video${videoInfo.id ? ` id="${videoInfo.id}"` : ""}> both point to the same source at the same time. The unmuted video already provides audio — the duplicate <audio> will cause double playback and echo.`,
elementId: audioInfo.id,
fixHint:
- "Either mute the video (add `muted` attribute) and keep the separate <audio>, or remove the <audio> element and let the video provide its own audio track.",
+ "Remove the <audio> element and let the video carry its own sound (recommended), or mute the video (add `muted`) and keep the separate <audio> when picture and sound must be cut independently.",
snippet: truncateSnippet(audioInfo.raw),
});
}
diff --git a/packages/lint/src/utils.ts b/packages/lint/src/utils.ts
index cf4712a7e2..265bc9110c 100644
--- a/packages/lint/src/utils.ts
+++ b/packages/lint/src/utils.ts
@@ -233,6 +233,36 @@ export function readAttr(tagSource: string, attr: string): string | null {
return match?.[1] || null;
}
+export function hasAttrName(tagSource: string, attr: string): boolean {
+ const escaped = attr.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+ const attrs = tagSource.replace(/^<\s*[a-z][\w:-]*/i, "");
+ return new RegExp(`(?:^|\\s)${escaped}(?:\\s*=|\\s|/?>)`, "i").test(attrs);
+}
+
+// An explicit data-has-audio is authoritative for the compiler; "false" and empty mean silent.
+export function isAudibleVideoTag(tagSource: string): boolean {
+ if (hasAttrName(tagSource, "muted")) return false;
+ if (!hasAttrName(tagSource, "data-has-audio")) return true;
+ const declared = readAttr(tagSource, "data-has-audio");
+ return declared !== null && declared.trim().toLowerCase() !== "false";
+}
+
+export function mediaTimeWindow(tagSource: string): { start: number; end: number } | null {
+ const start = Number(readAttr(tagSource, "data-start"));
+ const duration = Number(readAttr(tagSource, "data-duration"));
+ if (!readAttr(tagSource, "data-start") || !readAttr(tagSource, "data-duration")) return null;
+ if (!Number.isFinite(start) || !Number.isFinite(duration)) return null;
+ return { start, end: start + duration };
+}
+
+export function mediaWindowsOverlap(
+ a: { start: number; end: number } | null,
+ b: { start: number; end: number } | null,
+): boolean {
+ if (!a || !b) return true;
+ return a.start < b.end && b.start < a.end;
+}
+
/** Read an HTML attribute using browser-equivalent character-reference decoding. */
export function readDecodedAttr(tagSource: string, attr: string): string | null {
if (!tagSource) return null;
diff --git a/packages/parsers/src/htmlParser.test.ts b/packages/parsers/src/htmlParser.test.ts
index 1d6129bb03..ce70b8a35d 100644
--- a/packages/parsers/src/htmlParser.test.ts
+++ b/packages/parsers/src/htmlParser.test.ts
@@ -908,3 +908,65 @@ describe("extractCompositionMetadata", () => {
expect(meta.variables.find((v) => v.id === "ok")).toBeDefined();
});
});
+
+describe("video audio attributes", () => {
+ const DOC = `<!DOCTYPE html>
+<html><body>
+ <div id="stage">
+ <div id="stage-zoom-container"><video id="v" src="a.mp4" muted data-start="0" data-duration="2"></video></div>
+ </div>
+</body></html>`;
+
+ const videoTag = (html: string, id: string): string =>
+ html.match(new RegExp(`<video[^>]*id="${id}"[^>]*>`))?.[0] ?? "";
+
+ it("addElementToHtml: an audible video gets data-has-audio and no muted", () => {
+ const { html, id } = addElementToHtml(DOC, {
+ type: "video",
+ name: "Audible",
+ src: "b.mp4",
+ startTime: 0,
+ duration: 2,
+ zIndex: 0,
+ hasAudio: true,
+ });
+ const tag = videoTag(html, id);
+ expect(tag).toContain('data-has-audio="true"');
+ expect(tag).not.toMatch(/\bmuted\b/);
+ });
+
+ it("addElementToHtml: a silent video is muted with no data-has-audio", () => {
+ const { html, id } = addElementToHtml(DOC, {
+ type: "video",
+ name: "Silent",
+ src: "b.mp4",
+ startTime: 0,
+ duration: 2,
+ zIndex: 0,
+ });
+ const tag = videoTag(html, id);
+ expect(tag).toMatch(/\bmuted\b/);
+ expect(tag).not.toContain("data-has-audio");
+ });
+
+ it("updateElementInHtml hasAudio:true clears muted; false restores it", () => {
+ const on = updateElementInHtml(DOC, "v", { hasAudio: true });
+ expect(videoTag(on, "v")).not.toMatch(/\bmuted\b/);
+ expect(videoTag(on, "v")).toContain('data-has-audio="true"');
+ const off = updateElementInHtml(on, "v", { hasAudio: false });
+ const tag = videoTag(off, "v");
+ expect(tag).toMatch(/\bmuted\b/);
+ expect(tag).not.toContain("data-has-audio");
+ });
+
+ it("updateElementInHtml hasAudio only touches videos, and only true/false", () => {
+ const html =
+ '<html><body><video id="v" src="a.mp4" data-has-audio="true"></video><audio id="a" src="m.wav"></audio></body></html>';
+ const audio = updateElementInHtml(html, "a", { hasAudio: false });
+ expect(audio).not.toMatch(/<audio\b[^>]*\bmuted\b/);
+ expect(audio).not.toMatch(/<audio\b[^>]*data-has-audio/);
+ const untouched = updateElementInHtml(html, "v", { hasAudio: undefined });
+ expect(videoTag(untouched, "v")).toContain('data-has-audio="true"');
+ expect(videoTag(untouched, "v")).not.toMatch(/\bmuted\b/);
+ });
+});
diff --git a/packages/parsers/src/htmlParser.ts b/packages/parsers/src/htmlParser.ts
index b571f0d2cc..9387f992ee 100644
--- a/packages/parsers/src/htmlParser.ts
+++ b/packages/parsers/src/htmlParser.ts
@@ -617,11 +617,13 @@ export function updateElementInHtml(
}
// Handle hasAudio property for videos
- if ("hasAudio" in updates) {
+ if (el.tagName.toLowerCase() === "video" && "hasAudio" in updates) {
if (updates.hasAudio === true) {
el.setAttribute("data-has-audio", "true");
- } else {
+ el.removeAttribute("muted");
+ } else if (updates.hasAudio === false) {
el.removeAttribute("data-has-audio");
+ el.setAttribute("muted", "");
}
}
@@ -672,12 +674,10 @@ export function addElementToHtml(
case "video": {
const mediaEl = element as TimelineMediaElement;
newEl = doc.createElement("video");
- newEl.setAttribute("muted", "");
newEl.setAttribute("playsinline", "");
applyMediaAttrs(newEl, mediaEl);
- if (mediaEl.hasAudio) {
- newEl.setAttribute("data-has-audio", "true");
- }
+ if (mediaEl.hasAudio) newEl.setAttribute("data-has-audio", "true");
+ else newEl.setAttribute("muted", "");
break;
}
case "image": {
diff --git a/packages/producer/src/services/render/stages/extractVideosStage.test.ts b/packages/producer/src/services/render/stages/extractVideosStage.test.ts
index dc6e7ff0c8..dc4d91598d 100644
--- a/packages/producer/src/services/render/stages/extractVideosStage.test.ts
+++ b/packages/producer/src/services/render/stages/extractVideosStage.test.ts
@@ -114,6 +114,15 @@ describe("appendAutoDetectedVideoAudio", () => {
expect(composition.audios).toHaveLength(0);
});
+ it("skips a hidden video so a track hidden in Studio stays silent in the render", () => {
+ const composition = {
+ videos: [makeVideo({ hidden: true })],
+ audios: [] as never[],
+ };
+ appendAutoDetectedVideoAudio(composition, [makeExtracted("v1", true)]);
+ expect(composition.audios).toHaveLength(0);
+ });
+
it("skips when the source file has no audio track", () => {
const composition = { videos: [makeVideo()], audios: [] as never[] };
appendAutoDetectedVideoAudio(composition, [makeExtracted("v1", false)]);
diff --git a/packages/producer/src/services/render/stages/extractVideosStage.ts b/packages/producer/src/services/render/stages/extractVideosStage.ts
index fb48f1f571..a2c27774d8 100644
--- a/packages/producer/src/services/render/stages/extractVideosStage.ts
+++ b/packages/producer/src/services/render/stages/extractVideosStage.ts
@@ -562,7 +562,7 @@ export function appendAutoDetectedVideoAudio(
for (const ext of extracted) {
if (!ext.metadata.hasAudio) continue;
const video = composition.videos.find((v) => v.id === ext.videoId);
- if (!video || !video.hasAudio || existingAudioSrcs.has(video.src)) continue;
+ if (!video || !video.hasAudio || video.hidden || existingAudioSrcs.has(video.src)) continue;
composition.audios.push({
id: `${video.id}-audio`,
src: video.src,
diff --git a/packages/studio/src/components/StudioRightPanels.tsx b/packages/studio/src/components/StudioRightPanels.tsx
index f803dddc74..1461f42257 100644
--- a/packages/studio/src/components/StudioRightPanels.tsx
+++ b/packages/studio/src/components/StudioRightPanels.tsx
@@ -62,6 +62,7 @@ export function StudioRightPanels({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomPathOffsetCommit,
handleDomBoxSizeCommit,
handleDomRotationCommit,
@@ -210,6 +211,7 @@ export function StudioRightPanels({
onSetStyle={handleDomStyleCommit}
onSetAttribute={handleDomAttributeCommit}
onSetAttributes={handleDomAttributesCommit}
+ onSetAttributeBatch={handleDomAttributeBatchCommit}
onSetAttributeLive={setAttributeWhileDragging}
onSetAttributeQuiet={handleDomAttributeQuietCommit}
onApplyColorGradingScope={handleApplyColorGradingScope}
diff --git a/packages/studio/src/components/editor/PropertyPanel.tsx b/packages/studio/src/components/editor/PropertyPanel.tsx
index 8c160a45ca..487c22dc55 100644
--- a/packages/studio/src/components/editor/PropertyPanel.tsx
+++ b/packages/studio/src/components/editor/PropertyPanel.tsx
@@ -58,7 +58,6 @@ export {
export const PropertyPanel = memo(function PropertyPanel(props: PropertyPanelProps) {
const {
projectId,
- projectDir,
assets,
element,
multiSelectCount = 0,
@@ -72,8 +71,6 @@ export const PropertyPanel = memo(function PropertyPanel(props: PropertyPanelPro
onSetAttribute,
onSetAttributeLive,
onApplyColorGradingScope,
- onSetHtmlAttribute,
- onRemoveBackground,
onSetManualOffset,
onSetManualSize,
onSetManualRotation,
@@ -374,17 +371,7 @@ export const PropertyPanel = memo(function PropertyPanel(props: PropertyPanelPro
/>
)}
- {sections.media && (
- <MediaSection
- projectDir={projectDir}
- element={element}
- styles={styles}
- onSetStyle={onSetStyle}
- onSetAttribute={onSetAttribute}
- onSetHtmlAttribute={onSetHtmlAttribute}
- onRemoveBackground={onRemoveBackground}
- />
- )}
+ {sections.media && <MediaSection {...props} element={element} styles={styles} />}
{sections.layout && (
<Section title="Layout" icon={<Move size={15} />}>
diff --git a/packages/studio/src/components/editor/PropertyPanelFlat.tsx b/packages/studio/src/components/editor/PropertyPanelFlat.tsx
index 7ceb80a1e0..a33deaacd8 100644
--- a/packages/studio/src/components/editor/PropertyPanelFlat.tsx
+++ b/packages/studio/src/components/editor/PropertyPanelFlat.tsx
@@ -61,7 +61,6 @@ export function PropertyPanelFlat({
clipboardCopied,
onCopyElementInfo,
projectId,
- projectDir,
assets,
previewIframeRef,
onClearSelection,
@@ -73,8 +72,6 @@ export function PropertyPanelFlat({
onSetAttributeLive,
onSetAttributeQuiet,
onApplyColorGradingScope,
- onSetHtmlAttribute,
- onRemoveBackground,
onSetText,
onSetTextFieldStyle,
onPreviewTextFieldStyle,
@@ -130,6 +127,7 @@ export function PropertyPanelFlat({
onUpdateKeyframeEase,
onUpdateSegmentEase,
onSetAllKeyframeEases,
+ ...forwardedProps
}: PropertyPanelFlatProps) {
// PropertyPanel keys this component by selection, so the default is per element.
const [openGroupId, setOpenGroupId] = useState<string>(() =>
@@ -502,13 +500,12 @@ export function PropertyPanelFlat({
summary: element.tagName,
content: (
<FlatMediaSection
- projectDir={projectDir}
+ {...forwardedProps}
+ projectId={projectId}
element={element}
styles={styles}
onSetStyle={onSetStyle}
onSetAttribute={onSetAttribute}
- onSetHtmlAttribute={onSetHtmlAttribute}
- onRemoveBackground={onRemoveBackground}
{...volumeAutomation}
/>
),
diff --git a/packages/studio/src/components/editor/mediaAudioEdits.test.ts b/packages/studio/src/components/editor/mediaAudioEdits.test.ts
new file mode 100644
index 0000000000..df089f55e7
--- /dev/null
+++ b/packages/studio/src/components/editor/mediaAudioEdits.test.ts
@@ -0,0 +1,212 @@
+// @vitest-environment happy-dom
+import { describe, expect, it } from "vitest";
+import { applyPatchByTarget } from "../../utils/sourcePatcher";
+import {
+ buildKeepSoundCutoutEdit,
+ cutoutOps,
+ hasAudioToggleOps,
+ mintLinkId,
+ mutedToggleOps,
+} from "./mediaAudioEdits";
+
+const attr = (property: string, value: string | null) => ({
+ type: "attribute" as const,
+ property,
+ value,
+});
+const html = (property: string, value: string | null) => ({
+ type: "html-attribute" as const,
+ property,
+ value,
+});
+
+describe("mutedToggleOps", () => {
+ it("mutes a video and removes has-audio instead of writing an empty marker", () => {
+ expect(mutedToggleOps({ isVideo: true, nextMuted: true, probedHasAudio: null })).toEqual([
+ html("muted", "true"),
+ attr("has-audio", null),
+ ]);
+ });
+
+ it("unmutes a video the probe hears as audible", () => {
+ expect(mutedToggleOps({ isVideo: true, nextMuted: false, probedHasAudio: true })).toEqual([
+ html("muted", null),
+ attr("has-audio", "true"),
+ ]);
+ });
+
+ it("unmutes a silent-probe video with an explicit has-audio=false", () => {
+ expect(mutedToggleOps({ isVideo: true, nextMuted: false, probedHasAudio: false })).toEqual([
+ html("muted", null),
+ attr("has-audio", "false"),
+ ]);
+ });
+
+ it("leaves has-audio absent when the probe fails", () => {
+ expect(mutedToggleOps({ isVideo: true, nextMuted: false, probedHasAudio: null })).toEqual([
+ html("muted", null),
+ attr("has-audio", null),
+ ]);
+ });
+
+ it("toggles only muted on an audio element", () => {
+ expect(mutedToggleOps({ isVideo: false, nextMuted: true, probedHasAudio: null })).toEqual([
+ html("muted", "true"),
+ ]);
+ expect(mutedToggleOps({ isVideo: false, nextMuted: false, probedHasAudio: null })).toEqual([
+ html("muted", null),
+ ]);
+ });
+});
+
+describe("hasAudioToggleOps", () => {
+ it("turns sound on by stamping has-audio and clearing muted", () => {
+ expect(hasAudioToggleOps(true)).toEqual([attr("has-audio", "true"), html("muted", null)]);
+ });
+
+ it("turns sound off by removing has-audio rather than writing an empty value", () => {
+ expect(hasAudioToggleOps(false)).toEqual([attr("has-audio", null), html("muted", "true")]);
+ });
+});
+
+describe("cutoutOps", () => {
+ it("swaps the src of an image and nothing else", () => {
+ expect(cutoutOps({ isVideo: false, cutoutSrc: "assets/cut.png" })).toEqual([
+ html("src", "assets/cut.png"),
+ ]);
+ });
+
+ it("swaps a silent video's src and keeps it muted without a has-audio marker", () => {
+ expect(cutoutOps({ isVideo: true, cutoutSrc: "assets/cut.webm" })).toEqual([
+ html("src", "assets/cut.webm"),
+ html("muted", "true"),
+ attr("has-audio", null),
+ ]);
+ });
+});
+
+describe("mintLinkId", () => {
+ it("dedupes against element ids and existing links", () => {
+ const doc = document.implementation.createHTMLDocument("t");
+ expect(mintLinkId(doc)).toBe("link");
+ doc.body.innerHTML = '<div id="link"></div><video data-link="link-2"></video>';
+ expect(mintLinkId(doc)).toBe("link-3");
+ });
+});
+
+function makeVideo(attrs: Record<string, string>): HTMLVideoElement {
+ const doc = document.implementation.createHTMLDocument("t");
+ const video = doc.createElement("video");
+ for (const [name, value] of Object.entries(attrs)) video.setAttribute(name, value);
+ doc.body.append(video);
+ return video;
+}
+
+function applyEdit(source: string, edit: ReturnType<typeof buildKeepSoundCutoutEdit>): string {
+ const target = { id: "clip" };
+ const patched = edit.ops.reduce((acc, op) => applyPatchByTarget(acc, target, op), source);
+ return edit.prepareContent(patched);
+}
+
+function parse(source: string): Document {
+ return new DOMParser().parseFromString(source, "text/html");
+}
+
+describe("buildKeepSoundCutoutEdit", () => {
+ const automation = JSON.stringify({
+ version: 1,
+ lanes: [
+ { target: "volume", points: [{ t: 0, v: 1 }] },
+ { target: "rate", points: [{ t: 0, v: 2 }] },
+ ],
+ });
+
+ const source = [
+ '<div data-composition-id="main" data-duration="10">',
+ ' <video id="clip" class="clip" src="assets/talk.mp4" data-start="1" data-duration="4" data-media-start="2" data-track-index="0" data-has-audio="true" data-volume="0.5" data-fade-in="0.3" data-audio-group="voiceover"></video>',
+ ' <audio id="music" class="clip" src="assets/bgm.mp3" data-start="0" data-duration="10" data-track-index="1"></audio>',
+ "</div>",
+ ].join("\n");
+
+ const liveVideo = () =>
+ makeVideo({
+ id: "clip",
+ src: "assets/talk.mp4",
+ "data-has-audio": "true",
+ "data-volume": "0.5",
+ "data-fade-in": "0.3",
+ "data-audio-group": "voiceover",
+ "data-automation": automation,
+ });
+
+ it("mutes the video onto the cutout and moves its sound to a linked sibling audio", () => {
+ const edit = buildKeepSoundCutoutEdit({
+ video: liveVideo(),
+ videoId: "clip",
+ target: { id: "clip" },
+ cutoutSrc: "assets/talk-cutout.webm",
+ });
+ const doc = parse(applyEdit(source, edit));
+ const video = doc.getElementById("clip");
+ const audio = doc.getElementById("clip-audio");
+ if (!video || !audio) throw new Error("expected both clips");
+
+ expect(video.getAttribute("src")).toBe("assets/talk-cutout.webm");
+ expect(video.hasAttribute("muted")).toBe(true);
+ expect(video.hasAttribute("data-has-audio")).toBe(false);
+ expect(video.hasAttribute("data-volume")).toBe(false);
+ expect(video.hasAttribute("data-fade-in")).toBe(false);
+ expect(video.hasAttribute("data-audio-group")).toBe(false);
+ expect(JSON.parse(video.getAttribute("data-automation") ?? "").lanes).toEqual([
+ { target: "rate", points: [{ t: 0, v: 2 }] },
+ ]);
+
+ expect(audio.tagName).toBe("AUDIO");
+ expect(audio.getAttribute("src")).toBe("assets/talk.mp4");
+ expect(audio.getAttribute("data-start")).toBe("1");
+ expect(audio.getAttribute("data-duration")).toBe("4");
+ expect(audio.getAttribute("data-media-start")).toBe("2");
+ expect(audio.hasAttribute("data-playback-rate")).toBe(false);
+ expect(audio.getAttribute("data-track-index")).toBe("2");
+ expect(audio.getAttribute("data-volume")).toBe("0.5");
+ expect(audio.getAttribute("data-fade-in")).toBe("0.3");
+ expect(audio.getAttribute("data-audio-group")).toBe("voiceover");
+ expect(JSON.parse(audio.getAttribute("data-automation") ?? "").lanes).toEqual([
+ { target: "volume", points: [{ t: 0, v: 1 }] },
+ { target: "rate", points: [{ t: 0, v: 2 }] },
+ ]);
+ expect(audio.getAttribute("data-link")).toBe(video.getAttribute("data-link"));
+ expect(video.getAttribute("data-link")).toBe("link");
+ expect(audio.parentElement).toBe(video.parentElement);
+ });
+
+ it("drops the video's automation attribute when it held no rate lane", () => {
+ const video = makeVideo({
+ id: "clip",
+ src: "assets/talk.mp4",
+ "data-automation": JSON.stringify({
+ version: 1,
+ lanes: [{ target: "volume", points: [{ t: 0, v: 1 }] }],
+ }),
+ });
+ const edit = buildKeepSoundCutoutEdit({
+ video,
+ videoId: "clip",
+ target: { id: "clip" },
+ cutoutSrc: "assets/cut.webm",
+ });
+ expect(edit.ops).toContainEqual(attr("automation", null));
+ });
+
+ it("suffixes the audio id when <video-id>-audio is taken", () => {
+ const edit = buildKeepSoundCutoutEdit({
+ video: liveVideo(),
+ videoId: "clip",
+ target: { id: "clip" },
+ cutoutSrc: "assets/cut.webm",
+ });
+ const taken = source.replace('id="music"', 'id="clip-audio"');
+ const doc = parse(applyEdit(taken, edit));
+ expect(doc.getElementById("clip-audio-2")?.getAttribute("src")).toBe("assets/talk.mp4");
+ });
+});
diff --git a/packages/studio/src/components/editor/mediaAudioEdits.ts b/packages/studio/src/components/editor/mediaAudioEdits.ts
new file mode 100644
index 0000000000..3ab2317fa0
--- /dev/null
+++ b/packages/studio/src/components/editor/mediaAudioEdits.ts
@@ -0,0 +1,253 @@
+import {
+ HF_AUDIO_AUTOMATION_ATTR,
+ RATE_TARGET,
+ parseAutomation,
+ serializeAutomation,
+ type HfAutomationLane,
+} from "@hyperframes/core/audio-automation";
+import { HF_AUDIO_CARVE_ATTR } from "@hyperframes/core/audio-carve";
+import { HF_AUDIO_FADE_IN_ATTR, HF_AUDIO_FADE_OUT_ATTR } from "@hyperframes/core/audio-fade";
+import { HF_AUDIO_FX_ATTR } from "@hyperframes/core/audio-fx";
+import { HF_AUDIO_GROUP_ATTR } from "@hyperframes/core/audio-groups";
+import {
+ escapeHtmlAttribute,
+ findTagByTarget,
+ readAttributeByTarget,
+ type PatchOperation,
+ type PatchTarget,
+} from "../../utils/sourcePatcher";
+import { collectHtmlIds, resolveAssetHasAudio } from "../../utils/studioHelpers";
+import type { CommitDomAttributeBatch } from "../../hooks/domEditCommitTypes";
+import { buildDomEditPatchTarget, type DomEditSelection } from "./domEditing";
+import { generateId } from "../../utils/generateId";
+
+const MEDIA_LINK_ATTR = "data-link";
+const SOUND_KEPT_ON_LINKED_AUDIO_STAGE = "Background removed. Sound kept on a linked audio track.";
+
+const MOVED_SOUND_ATTRS = [
+ "data-volume",
+ HF_AUDIO_FADE_IN_ATTR,
+ HF_AUDIO_FADE_OUT_ATTR,
+ HF_AUDIO_FX_ATTR,
+ HF_AUDIO_CARVE_ATTR,
+ HF_AUDIO_GROUP_ATTR,
+];
+
+const COPIED_TIMING_ATTRS = [
+ "start",
+ "end",
+ "duration",
+ "media-start",
+ "playback-start",
+ "playback-rate",
+ "source-duration",
+];
+
+const htmlAttr = (property: string, value: string | null): PatchOperation => ({
+ type: "html-attribute",
+ property,
+ value,
+});
+
+const dataAttr = (property: string, value: string | null): PatchOperation => ({
+ type: "attribute",
+ property,
+ value,
+});
+
+function stripDataPrefix(name: string): string {
+ return name.startsWith("data-") ? name.slice("data-".length) : name;
+}
+
+function probedHasAudioValue(probedHasAudio: boolean | null): string | null {
+ if (probedHasAudio === null) return null;
+ return probedHasAudio ? "true" : "false";
+}
+
+export function mutedToggleOps(input: {
+ isVideo: boolean;
+ nextMuted: boolean;
+ probedHasAudio: boolean | null;
+}): PatchOperation[] {
+ const muted = htmlAttr("muted", input.nextMuted ? "true" : null);
+ if (!input.isVideo) return [muted];
+ const hasAudio = input.nextMuted ? null : probedHasAudioValue(input.probedHasAudio);
+ return [muted, dataAttr("has-audio", hasAudio)];
+}
+
+export function hasAudioToggleOps(next: boolean): PatchOperation[] {
+ return next
+ ? [dataAttr("has-audio", "true"), htmlAttr("muted", null)]
+ : [dataAttr("has-audio", null), htmlAttr("muted", "true")];
+}
+
+export function cutoutOps(input: { isVideo: boolean; cutoutSrc: string }): PatchOperation[] {
+ const src = htmlAttr("src", input.cutoutSrc);
+ return input.isVideo ? [src, htmlAttr("muted", "true"), dataAttr("has-audio", null)] : [src];
+}
+
+function firstFreeName(base: string, taken: Set<string>): string {
+ if (!taken.has(base)) return base;
+ let n = 2;
+ while (taken.has(`${base}-${n}`)) n += 1;
+ return `${base}-${n}`;
+}
+
+export function mintLinkId(doc: Document): string {
+ const taken = new Set<string>();
+ for (const el of Array.from(doc.querySelectorAll(`[id], [${MEDIA_LINK_ATTR}]`))) {
+ if (el.id) taken.add(el.id);
+ const link = el.getAttribute(MEDIA_LINK_ATTR);
+ if (link) taken.add(link);
+ }
+ return firstFreeName("link", taken);
+}
+
+function firstFreeTrackIndex(source: string): number {
+ const used = new Set(
+ Array.from(source.matchAll(/\bdata-track-index=["'](-?\d+)["']/g), (m) => Number(m[1])),
+ );
+ let track = 0;
+ while (used.has(track)) track += 1;
+ return track;
+}
+
+function splitAutomation(raw: string): { videoKeeps: string | null; audioTakes: string } {
+ let lanes: HfAutomationLane[];
+ try {
+ lanes = parseAutomation(raw).lanes;
+ } catch {
+ return { videoKeeps: null, audioTakes: raw };
+ }
+ const rateLanes = lanes.filter((lane) => lane.target === RATE_TARGET);
+ const soundLanes = lanes.filter((lane) => lane.target !== RATE_TARGET);
+ return {
+ videoKeeps: rateLanes.length > 0 ? serializeAutomation({ version: 1, lanes: rateLanes }) : null,
+ audioTakes: serializeAutomation({ version: 1, lanes: [...soundLanes, ...rateLanes] }),
+ };
+}
+
+function formatAttrs(attrs: Array<[string, string]>): string {
+ return attrs.map(([name, value]) => `${name}="${escapeHtmlAttribute(value)}"`).join(" ");
+}
+
+function insertBeforeTarget(source: string, target: PatchTarget, markup: string): string {
+ const match = findTagByTarget(source, target);
+ if (!match) return source;
+ const lineStart = source.lastIndexOf("\n", match.start) + 1;
+ const indent = /^[ \t]*$/.test(source.slice(lineStart, match.start))
+ ? source.slice(lineStart, match.start)
+ : "";
+ const separator = indent ? `\n${indent}` : "";
+ return `${source.slice(0, match.start)}${markup}${separator}${source.slice(match.start)}`;
+}
+
+export interface KeepSoundCutoutEdit {
+ ops: PatchOperation[];
+ prepareContent: (source: string) => string;
+}
+
+export function buildKeepSoundCutoutEdit(input: {
+ video: Element;
+ videoId: string | null;
+ target: PatchTarget;
+ cutoutSrc: string;
+}): KeepSoundCutoutEdit {
+ const { video, target } = input;
+ const linkId = mintLinkId(video.ownerDocument);
+ const originalSrc = video.getAttribute("src") ?? "";
+
+ const movedSound: Array<[string, string]> = [];
+ const ops: PatchOperation[] = [...cutoutOps({ isVideo: true, cutoutSrc: input.cutoutSrc })];
+ for (const name of MOVED_SOUND_ATTRS) {
+ const value = video.getAttribute(name);
+ if (value === null) continue;
+ movedSound.push([name, value]);
+ ops.push(dataAttr(stripDataPrefix(name), null));
+ }
+ const automation = video.getAttribute(HF_AUDIO_AUTOMATION_ATTR);
+ if (automation !== null) {
+ const { videoKeeps, audioTakes } = splitAutomation(automation);
+ movedSound.push([HF_AUDIO_AUTOMATION_ATTR, audioTakes]);
+ ops.push(dataAttr(stripDataPrefix(HF_AUDIO_AUTOMATION_ATTR), videoKeeps));
+ }
+ ops.push(dataAttr(stripDataPrefix(MEDIA_LINK_ATTR), linkId));
+
+ const prepareContent = (source: string): string => {
+ const audioId = firstFreeName(
+ `${input.videoId || "video"}-audio`,
+ new Set(collectHtmlIds(source)),
+ );
+ const timing: Array<[string, string]> = [];
+ for (const name of COPIED_TIMING_ATTRS) {
+ const value = readAttributeByTarget(source, target, name);
+ if (value !== undefined) timing.push([`data-${name}`, value]);
+ }
+ const attrs: Array<[string, string]> = [
+ ["id", audioId],
+ ["data-hf-id", `hf-${generateId()}`],
+ ["class", "clip"],
+ ["src", originalSrc],
+ ...timing,
+ ["data-track-index", String(firstFreeTrackIndex(source))],
+ [MEDIA_LINK_ATTR, linkId],
+ ...movedSound,
+ ];
+ return insertBeforeTarget(source, target, `<audio ${formatAttrs(attrs)}></audio>`);
+ };
+
+ return { ops, prepareContent };
+}
+
+export interface MediaEditContext {
+ element: DomEditSelection;
+ projectId: string | null;
+ projectSrc: string;
+ commit: CommitDomAttributeBatch;
+}
+
+const isVideoSelection = (element: DomEditSelection) => element.tagName === "video";
+
+export async function commitMutedToggle(ctx: MediaEditContext, nextMuted: boolean): Promise<void> {
+ const isVideo = isVideoSelection(ctx.element);
+ const probedHasAudio =
+ isVideo && !nextMuted && ctx.projectId && ctx.projectSrc
+ ? await resolveAssetHasAudio(ctx.projectId, ctx.projectSrc)
+ : null;
+ await ctx.commit(ctx.element, mutedToggleOps({ isVideo, nextMuted, probedHasAudio }), {
+ label: "Edit muted",
+ });
+}
+
+export async function commitHasAudioToggle(ctx: MediaEditContext, next: boolean): Promise<void> {
+ await ctx.commit(ctx.element, hasAudioToggleOps(next), { label: "Edit has audio" });
+}
+
+export async function commitCutout(
+ ctx: MediaEditContext,
+ cutoutSrc: string,
+ keepSound: boolean,
+): Promise<string> {
+ const isVideo = isVideoSelection(ctx.element);
+ const label = "Remove background";
+ const keep = isVideo && keepSound;
+ const landed = keep
+ ? await commitKeepSoundCutout(ctx, cutoutSrc, label)
+ : await ctx.commit(ctx.element, cutoutOps({ isVideo, cutoutSrc }), { label });
+ if (!landed) throw new Error("Couldn't apply the cutout");
+ return keep ? SOUND_KEPT_ON_LINKED_AUDIO_STAGE : "Applied cutout";
+}
+
+function commitKeepSoundCutout(
+ ctx: MediaEditContext,
+ cutoutSrc: string,
+ label: string,
+): Promise<boolean> {
+ const edit = buildKeepSoundCutoutEdit({
+ video: ctx.element.element,
+ videoId: ctx.element.id ?? null,
+ target: buildDomEditPatchTarget(ctx.element),
+ cutoutSrc,
+ });
+ return ctx.commit(ctx.element, edit.ops, { label, prepareContent: edit.prepareContent });
+}
diff --git a/packages/studio/src/components/editor/propertyPanelFlatMediaSection.test.tsx b/packages/studio/src/components/editor/propertyPanelFlatMediaSection.test.tsx
index 297d98027f..4186ae24b0 100644
--- a/packages/studio/src/components/editor/propertyPanelFlatMediaSection.test.tsx
+++ b/packages/studio/src/components/editor/propertyPanelFlatMediaSection.test.tsx
@@ -6,6 +6,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
import { FlatMediaSection } from "./propertyPanelFlatMediaSection";
import { MediaSection } from "./propertyPanelMediaSection";
import type { DomEditSelection } from "./domEditing";
+import type { PatchOperation } from "../../utils/sourcePatcher";
import { formatTimingValue } from "./propertyPanelHelpers";
import { readMediaOffsetSeconds } from "@hyperframes/parsers/media-duration";
@@ -62,6 +63,7 @@ function renderSection(overrides: Partial<DomEditSelection> = {}) {
onSetStyle={vi.fn()}
onSetAttribute={vi.fn()}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -102,6 +104,7 @@ describe("FlatMediaSection — cutout", () => {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={onSetHtmlAttribute}
+ onSetAttributeBatch={vi.fn()}
onRemoveBackground={onRemoveBackground}
/>,
);
@@ -149,6 +152,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -174,6 +178,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -208,6 +213,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
volumeAutomated
onCommitVolumeAt={onCommitVolumeAt}
/>,
@@ -242,6 +248,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -273,6 +280,7 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -308,6 +316,7 @@ describe("FlatMediaSection — loop/muted/has-audio", () => {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={onSetHtmlAttribute}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -323,92 +332,6 @@ describe("FlatMediaSection — loop/muted/has-audio", () => {
expect(hasAudioToggle?.getAttribute("aria-checked")).toBe("true");
act(() => root.unmount());
});
-
- it("toggles muted via onSetHtmlAttribute", () => {
- const onSetHtmlAttribute = vi.fn();
- const onSetAttribute = vi.fn();
- const element = makeVideoElement();
- const host = document.createElement("div");
- document.body.append(host);
- const root = createRoot(host);
- act(() => {
- root.render(
- <FlatMediaSection
- projectDir={null}
- element={element}
- styles={{}}
- onSetStyle={vi.fn()}
- onSetAttribute={onSetAttribute}
- onSetHtmlAttribute={onSetHtmlAttribute}
- />,
- );
- });
- const mutedToggle = host.querySelector<HTMLButtonElement>(
- '[data-flat-toggle="true"][aria-label="Muted"]',
- );
- expect(mutedToggle?.getAttribute("aria-checked")).toBe("false");
- act(() => mutedToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true })));
- expect(onSetHtmlAttribute).toHaveBeenCalledWith("muted", "true");
- act(() => root.unmount());
- });
-
- it("enables has-audio-track and clears muted on click", () => {
- const onSetHtmlAttribute = vi.fn();
- const onSetAttribute = vi.fn();
- const element = makeVideoElement();
- const host = document.createElement("div");
- document.body.append(host);
- const root = createRoot(host);
- act(() => {
- root.render(
- <FlatMediaSection
- projectDir={null}
- element={element}
- styles={{}}
- onSetStyle={vi.fn()}
- onSetAttribute={onSetAttribute}
- onSetHtmlAttribute={onSetHtmlAttribute}
- />,
- );
- });
- const hasAudioToggle = host.querySelector<HTMLButtonElement>(
- '[data-flat-toggle="true"][aria-label="Has audio track"]',
- );
- expect(hasAudioToggle?.getAttribute("aria-checked")).toBe("false");
- act(() => hasAudioToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true })));
- expect(onSetAttribute).toHaveBeenCalledWith("has-audio", "true");
- expect(onSetHtmlAttribute).toHaveBeenCalledWith("muted", null);
- act(() => root.unmount());
- });
-
- it("disables has-audio-track and sets muted on click", () => {
- const onSetHtmlAttribute = vi.fn();
- const onSetAttribute = vi.fn();
- const element = makeVideoElement({ dataAttributes: { "has-audio": "true" } });
- const host = document.createElement("div");
- document.body.append(host);
- const root = createRoot(host);
- act(() => {
- root.render(
- <FlatMediaSection
- projectDir={null}
- element={element}
- styles={{}}
- onSetStyle={vi.fn()}
- onSetAttribute={onSetAttribute}
- onSetHtmlAttribute={onSetHtmlAttribute}
- />,
- );
- });
- const hasAudioToggle = host.querySelector<HTMLButtonElement>(
- '[data-flat-toggle="true"][aria-label="Has audio track"]',
- );
- expect(hasAudioToggle?.getAttribute("aria-checked")).toBe("true");
- act(() => hasAudioToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true })));
- expect(onSetAttribute).toHaveBeenCalledWith("has-audio", "");
- expect(onSetHtmlAttribute).toHaveBeenCalledWith("muted", "true");
- act(() => root.unmount());
- });
});
describe("FlatMediaSection — fit/position", () => {
@@ -428,6 +351,7 @@ describe("FlatMediaSection — fit/position", () => {
onSetStyle={onSetStyle}
onSetAttribute={vi.fn()}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -461,6 +385,7 @@ describe("FlatMediaSection — fit/position", () => {
onSetStyle={onSetStyle}
onSetAttribute={vi.fn()}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -504,6 +429,7 @@ function renderWithRate(element: DomEditSelection, onSetAttribute = vi.fn()) {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
rate={{
automated: false,
automatedValue: undefined,
@@ -543,7 +469,9 @@ describe("FlatMediaSection — audio clips", () => {
);
act(() => audio.root.unmount());
- const silentVideo = renderWithRate(makeVideoElement({ dataAttributes: { duration: "10" } }));
+ const mutedSelection = makeVideoElement({ dataAttributes: { duration: "10" } });
+ mutedSelection.element.setAttribute("muted", "");
+ const silentVideo = renderWithRate(mutedSelection);
expect(labelsOf(silentVideo.host)).not.toContain("Fade in");
act(() => silentVideo.root.unmount());
@@ -659,6 +587,7 @@ describe("FlatMediaSection — audio clips", () => {
onSetStyle={vi.fn()}
onSetAttribute={onSetAttribute}
onSetHtmlAttribute={vi.fn()}
+ onSetAttributeBatch={vi.fn()}
/>,
);
});
@@ -708,3 +637,273 @@ function typeInto(input: HTMLInputElement, value: string) {
setter?.call(input, value);
input.dispatchEvent(new Event("input", { bubbles: true }));
}
+
+type PanelKind = "flat" | "design";
+
+type BatchOptions = { label: string; prepareContent?: (html: string) => string };
+
+function makeHandlers() {
+ return {
+ onSetAttribute: vi.fn(),
+ onSetHtmlAttribute: vi.fn(),
+ onSetAttributeBatch: vi.fn<
+ (
+ selection: DomEditSelection,
+ ops: PatchOperation[],
+ options: BatchOptions,
+ ) => Promise<boolean>
+ >(async () => true),
+ };
+}
+
+function mountPanel(
+ kind: PanelKind,
+ element: DomEditSelection,
+ handlers: ReturnType<typeof makeHandlers> & {
+ onRemoveBackground?: () => Promise<{ outputPath: string }>;
+ },
+) {
+ const host = document.createElement("div");
+ document.body.append(host);
+ const root = createRoot(host);
+ const Panel = kind === "flat" ? FlatMediaSection : MediaSection;
+ act(() => {
+ root.render(
+ <Panel
+ projectId="p1"
+ projectDir={null}
+ element={element}
+ styles={{}}
+ onSetStyle={vi.fn()}
+ {...handlers}
+ />,
+ );
+ });
+ return { host, root };
+}
+
+async function flush() {
+ for (let i = 0; i < 8; i++) await Promise.resolve();
+}
+
+async function clickToggle(kind: PanelKind, host: HTMLElement, label: string, option: string) {
+ if (kind === "flat") {
+ const toggle = host.querySelector<HTMLButtonElement>(
+ `[data-flat-toggle="true"][aria-label="${label}"]`,
+ );
+ if (!toggle) throw new Error(`expected ${label} toggle`);
+ await act(async () => {
+ toggle.dispatchEvent(new MouseEvent("click", { bubbles: true }));
+ await flush();
+ });
+ return;
+ }
+ const row = Array.from(host.querySelectorAll("span")).find(
+ (n) => n.textContent === label,
+ )?.parentElement;
+ const button = Array.from(row?.querySelectorAll("button") ?? []).find(
+ (b) => b.textContent === option,
+ );
+ if (!button) throw new Error(`expected ${label} segmented button`);
+ await act(async () => {
+ button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
+ await flush();
+ });
+}
+
+function stubHasAudioProbe(hasAudio: boolean | "fail") {
+ const fetchMock = vi.fn(async (_input: Parameters<typeof fetch>[0]) =>
+ hasAudio === "fail"
+ ? new Response("nope", { status: 500 })
+ : new Response(JSON.stringify({ metadata: { hasAudio } })),
+ );
+ vi.stubGlobal("fetch", fetchMock);
+ return fetchMock;
+}
+
+const htmlOp = (property: string, value: string | null) => ({
+ type: "html-attribute",
+ property,
+ value,
+});
+const dataOp = (property: string, value: string | null) => ({
+ type: "attribute",
+ property,
+ value,
+});
+
+describe.each<PanelKind>(["flat", "design"])("%s panel — Muted is one atomic edit", (kind) => {
+ afterEach(() => vi.unstubAllGlobals());
+
+ it("muting an audible video sets muted and removes has-audio in ONE write", async () => {
+ const handlers = makeHandlers();
+ const element = makeVideoElement({ dataAttributes: { "has-audio": "true" } });
+ const { host, root } = mountPanel(kind, element, handlers);
+ await clickToggle(kind, host, "Muted", "On");
+ expect(handlers.onSetAttributeBatch).toHaveBeenCalledTimes(1);
+ expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
+ htmlOp("muted", "true"),
+ dataOp("has-audio", null),
+ ]);
+ expect(handlers.onSetAttribute).not.toHaveBeenCalled();
+ expect(handlers.onSetHtmlAttribute).not.toHaveBeenCalled();
+ act(() => root.unmount());
+ });
+
+ it("unmuting a video the probe hears stamps has-audio=true in the same write", async () => {
+ const probe = stubHasAudioProbe(true);
+ const handlers = makeHandlers();
+ const element = makeVideoElement();
+ element.element.setAttribute("muted", "");
+ const { host, root } = mountPanel(kind, element, handlers);
+ await clickToggle(kind, host, "Muted", "Off");
+ expect(String(probe.mock.calls[0]?.[0])).toContain("assets%2Fintro-loop.mp4");
+ expect(handlers.onSetAttributeBatch).toHaveBeenCalledTimes(1);
+ expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
+ htmlOp("muted", null),
+ dataOp("has-audio", "true"),
+ ]);
+ act(() => root.unmount());
+ });
+
+ it("unmuting a silent-probe video writes has-audio=false", async () => {
+ stubHasAudioProbe(false);
+ const handlers = makeHandlers();
+ const element = makeVideoElement();
+ element.element.setAttribute("muted", "");
+ const { host, root } = mountPanel(kind, element, handlers);
+ await clickToggle(kind, host, "Muted", "Off");
+ expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
+ htmlOp("muted", null),
+ dataOp("has-audio", "false"),
+ ]);
+ act(() => root.unmount());
+ });
+
+ it("unmuting when the probe fails leaves has-audio absent", async () => {
+ stubHasAudioProbe("fail");
+ const handlers = makeHandlers();
+ const element = makeVideoElement();
+ element.element.setAttribute("muted", "");
+ const { host, root } = mountPanel(kind, element, handlers);
+ await clickToggle(kind, host, "Muted", "Off");
+ expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
+ htmlOp("muted", null),
+ dataOp("has-audio", null),
+ ]);
+ act(() => root.unmount());
+ });
+
+ it("audio Muted toggles only the muted attribute", async () => {
+ const handlers = makeHandlers();
+ const { host, root } = mountPanel(kind, makeAudioElement({}), handlers);
+ await clickToggle(kind, host, "Muted", "On");
+ expect(handlers.onSetAttributeBatch).toHaveBeenCalledTimes(1);
+ expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([htmlOp("muted", "true")]);
+ act(() => root.unmount());
+ });
+
+ it("Has audio track No removes has-audio and mutes in ONE write", async () => {
+ const handlers = makeHandlers();
+ const element = makeVideoElement({ dataAttributes: { "has-audio": "true" } });
+ const { host, root } = mountPanel(kind, element, handlers);
+ await clickToggle(kind, host, "Has audio track", "No");
+ expect(handlers.onSetAttributeBatch).toHaveBeenCalledTimes(1);
+ expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
+ dataOp("has-audio", null),
+ htmlOp("muted", "true"),
+ ]);
+ act(() => root.unmount());
+ });
+
+ it("Has audio track Yes stamps has-audio and clears muted in ONE write", async () => {
+ const handlers = makeHandlers();
+ const element = makeVideoElement();
+ element.element.setAttribute("muted", "");
+ const { host, root } = mountPanel(kind, element, handlers);
+ await clickToggle(kind, host, "Has audio track", "Yes");
+ expect(handlers.onSetAttributeBatch.mock.calls[0]?.[1]).toEqual([
+ dataOp("has-audio", "true"),
+ htmlOp("muted", null),
+ ]);
+ act(() => root.unmount());
+ });
+});
+
+describe.each<PanelKind>(["flat", "design"])("%s panel — cutout keeps the sound", (kind) => {
+ async function runCutout(element: DomEditSelection) {
+ const handlers = makeHandlers();
+ const onRemoveBackground = vi.fn().mockResolvedValue({ outputPath: "assets/cut.webm" });
+ const confirm = vi.spyOn(window, "confirm");
+ const { host, root } = mountPanel(kind, element, { ...handlers, onRemoveBackground });
+ const button =
+ kind === "flat"
+ ? host.querySelector<HTMLButtonElement>('[data-flat-media-remove-bg="true"]')
+ : Array.from(host.querySelectorAll("button")).find((b) =>
+ /remove bg/i.test(b.textContent ?? ""),
+ );
+ if (!button) throw new Error("expected remove background button");
+ await act(async () => {
+ button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
+ await flush();
+ });
+ const text = host.textContent ?? "";
+ act(() => root.unmount());
+ return { ...handlers, confirm, text };
+ }
+
+ afterEach(() => vi.restoreAllMocks());
+
+ it("audible video: one write mutes onto the cutout and inserts a linked sibling audio", async () => {
+ const element = makeVideoElement({
+ dataAttributes: { "has-audio": "true", volume: "0.5" },
+ });
+ element.element.setAttribute("id", "s1-bg");
+ element.element.setAttribute("data-has-audio", "true");
+ element.element.setAttribute("data-volume", "0.5");
+ const { onSetAttributeBatch, onSetAttribute, onSetHtmlAttribute, confirm, text } =
+ await runCutout(element);
+
+ expect(confirm).not.toHaveBeenCalled();
+ expect(onSetAttribute).not.toHaveBeenCalled();
+ expect(onSetHtmlAttribute).not.toHaveBeenCalled();
+ expect(onSetAttributeBatch).toHaveBeenCalledTimes(1);
+ const [selection, ops, options] = onSetAttributeBatch.mock.calls[0] ?? [];
+ expect(selection).toBe(element);
+ expect(ops).toEqual([
+ htmlOp("src", "assets/cut.webm"),
+ htmlOp("muted", "true"),
+ dataOp("has-audio", null),
+ dataOp("volume", null),
+ dataOp("link", "link"),
+ ]);
+ const source =
+ '<div data-composition-id="main"><video id="s1-bg" src="assets/cut.webm" muted data-start="2" data-duration="3" data-track-index="0" data-link="link"></video></div>';
+ const prepared = options?.prepareContent?.(source) ?? "";
+ const audio = new DOMParser()
+ .parseFromString(prepared, "text/html")
+ .getElementById("s1-bg-audio");
+ expect(audio?.getAttribute("src")).toBe("assets/intro-loop.mp4");
+ expect(audio?.getAttribute("data-start")).toBe("2");
+ expect(audio?.getAttribute("data-duration")).toBe("3");
+ expect(audio?.getAttribute("data-link")).toBe("link");
+ expect(audio?.getAttribute("data-volume")).toBe("0.5");
+ expect(text).toContain("Background removed. Sound kept on a linked audio track.");
+ });
+
+ it("silent video: one write swaps the src and inserts no audio", async () => {
+ const silent = makeVideoElement();
+ silent.element.setAttribute("muted", "");
+ const { onSetAttributeBatch, confirm, text } = await runCutout(silent);
+ expect(confirm).not.toHaveBeenCalled();
+ expect(onSetAttributeBatch).toHaveBeenCalledTimes(1);
+ const [, ops, options] = onSetAttributeBatch.mock.calls[0] ?? [];
+ expect(ops).toEqual([
+ htmlOp("src", "assets/cut.webm"),
+ htmlOp("muted", "true"),
+ dataOp("has-audio", null),
+ ]);
+ expect(options?.prepareContent).toBeUndefined();
+ expect(text).toContain("Applied cutout");
+ });
+});
diff --git a/packages/studio/src/components/editor/propertyPanelFlatMediaSection.tsx b/packages/studio/src/components/editor/propertyPanelFlatMediaSection.tsx
index cd38f9aaae..273728a2c8 100644
--- a/packages/studio/src/components/editor/propertyPanelFlatMediaSection.tsx
+++ b/packages/studio/src/components/editor/propertyPanelFlatMediaSection.tsx
@@ -1,4 +1,5 @@
import { useEffect, useState } from "react";
+import { isVideoAudible } from "../../player/lib/timelineElementHelpers";
import { useTrackDesignInput } from "../../contexts/DesignPanelInputContext";
import { Check, ClipboardList } from "../../icons/SystemIcons";
import type { DomEditSelection } from "./domEditing";
@@ -34,15 +35,19 @@ import {
readFadeSeconds,
} from "@hyperframes/core/audio-fade";
import { parseGainInput, parseRateInput, parseSecondsInput } from "./audioInspectorInput";
+import type { CommitDomAttributeBatch } from "../../hooks/domEditCommitTypes";
+import { commitCutout, commitHasAudioToggle, commitMutedToggle } from "./mediaAudioEdits";
// fallow-ignore-next-line complexity
export function FlatMediaSection({
+ projectId = null,
projectDir,
element,
styles,
onSetStyle,
onSetAttribute,
onSetHtmlAttribute,
+ onSetAttributeBatch,
onRemoveBackground,
volumeAutomated,
onAutomateVolume,
@@ -51,12 +56,14 @@ export function FlatMediaSection({
automatedVolumeValue,
rate,
}: {
+ projectId?: string | null;
projectDir: string | null;
element: DomEditSelection;
styles: Record<string, string>;
onSetStyle: (prop: string, value: string) => void | Promise<unknown>;
onSetAttribute: (attr: string, value: string) => void | Promise<void>;
onSetHtmlAttribute: (attr: string, value: string | null) => void | Promise<void>;
+ onSetAttributeBatch: CommitDomAttributeBatch;
/** A volume lane in the timeline drives the level; the slider writes a keyframe instead. */
volumeAutomated?: boolean;
onAutomateVolume?: () => void;
@@ -104,7 +111,11 @@ export function FlatMediaSection({
const fadeMax = clipDuration > 0 ? clipDuration : 10;
const hasLoop = el.hasAttribute("loop");
const hasMuted = el.hasAttribute("muted");
- const hasAudio = element.dataAttributes["has-audio"] === "true";
+ const hasAudio = isVideoAudible({
+ tag: el.tagName,
+ hasAudioAttr: element.dataAttributes["has-audio"],
+ muted: el.hasAttribute("muted"),
+ });
const objectFit = styles["object-fit"] || "contain";
const objectPosition = styles["object-position"] || "center";
@@ -128,13 +139,7 @@ export function FlatMediaSection({
setCreatePlate(false);
}, [srcAttr]);
- const applyCutoutResult = async (result: BackgroundRemovalResult) => {
- await onSetHtmlAttribute("src", result.outputPath);
- if (isVideo) {
- await onSetAttribute("has-audio", "");
- await onSetHtmlAttribute("muted", "true");
- }
- };
+ const mediaEdit = { element, projectId, projectSrc, commit: onSetAttributeBatch };
const runBackgroundRemoval = async () => {
if (!onRemoveBackground || !projectSrc || removeBusy) return;
@@ -147,8 +152,8 @@ export function FlatMediaSection({
quality,
onProgress: setRemoveProgress,
});
- await applyCutoutResult(result);
- setRemoveProgress({ status: "complete", progress: 100, stage: "Applied cutout", ...result });
+ const stage = await commitCutout(mediaEdit, result.outputPath, hasAudio);
+ setRemoveProgress({ status: "complete", progress: 100, stage, ...result });
} catch (error) {
setRemoveProgress({
status: "failed",
@@ -362,21 +367,13 @@ export function FlatMediaSection({
<FlatToggle
label="Muted"
checked={hasMuted}
- onChange={(next) => void onSetHtmlAttribute("muted", next ? "true" : null)}
+ onChange={(next) => void commitMutedToggle(mediaEdit, next)}
/>
{isVideo && (
<FlatToggle
label="Has audio track"
checked={hasAudio}
- onChange={(next) => {
- if (next) {
- void onSetAttribute("has-audio", "true");
- void onSetHtmlAttribute("muted", null);
- } else {
- void onSetAttribute("has-audio", "");
- void onSetHtmlAttribute("muted", "true");
- }
- }}
+ onChange={(next) => void commitHasAudioToggle(mediaEdit, next)}
/>
)}
</>
diff --git a/packages/studio/src/components/editor/propertyPanelFlatProps.ts b/packages/studio/src/components/editor/propertyPanelFlatProps.ts
index 6fb7a93270..c0d10b9bda 100644
--- a/packages/studio/src/components/editor/propertyPanelFlatProps.ts
+++ b/packages/studio/src/components/editor/propertyPanelFlatProps.ts
@@ -19,6 +19,7 @@ export type PropertyPanelFlatProps = Pick<
| "onSetAttributeQuiet"
| "onApplyColorGradingScope"
| "onSetHtmlAttribute"
+ | "onSetAttributeBatch"
| "onRemoveBackground"
| "onSetText"
| "onSetTextFieldStyle"
diff --git a/packages/studio/src/components/editor/propertyPanelMediaSection.tsx b/packages/studio/src/components/editor/propertyPanelMediaSection.tsx
index de1822fdbe..c86db6461d 100644
--- a/packages/studio/src/components/editor/propertyPanelMediaSection.tsx
+++ b/packages/studio/src/components/editor/propertyPanelMediaSection.tsx
@@ -1,4 +1,5 @@
import { useEffect, useState } from "react";
+import { isVideoAudible } from "../../player/lib/timelineElementHelpers";
import { Check, ClipboardList, Film, Music, Scissors } from "../../icons/SystemIcons";
import type { DomEditSelection } from "./domEditing";
import {
@@ -22,23 +23,29 @@ import {
audioGainToFaderPosition,
audioGainToText,
} from "@hyperframes/core/audio-gain";
+import type { CommitDomAttributeBatch } from "../../hooks/domEditCommitTypes";
+import { commitCutout, commitHasAudioToggle, commitMutedToggle } from "./mediaAudioEdits";
// fallow-ignore-next-line complexity
export function MediaSection({
+ projectId = null,
projectDir,
element,
styles,
onSetStyle,
onSetAttribute,
onSetHtmlAttribute,
+ onSetAttributeBatch,
onRemoveBackground,
}: {
+ projectId?: string | null;
projectDir: string | null;
element: DomEditSelection;
styles: Record<string, string>;
onSetStyle: (prop: string, value: string) => void | Promise<unknown>;
onSetAttribute: (attr: string, value: string) => void | Promise<void>;
onSetHtmlAttribute: (attr: string, value: string | null) => void | Promise<void>;
+ onSetAttributeBatch: CommitDomAttributeBatch;
onRemoveBackground?: (
inputPath: string,
options: {
@@ -62,7 +69,11 @@ export function MediaSection({
const hasLoop = el.hasAttribute("loop");
const hasMuted = el.hasAttribute("muted");
- const hasAudio = element.dataAttributes["has-audio"] === "true";
+ const hasAudio = isVideoAudible({
+ tag: el.tagName,
+ hasAudioAttr: element.dataAttributes["has-audio"],
+ muted: el.hasAttribute("muted"),
+ });
const playbackRate = Number.parseFloat(element.dataAttributes["playback-rate"] ?? "1") || 1;
@@ -96,13 +107,7 @@ export function MediaSection({
setCreatePlate(false);
}, [srcAttr]);
- const applyCutoutResult = async (result: BackgroundRemovalResult) => {
- await onSetHtmlAttribute("src", result.outputPath);
- if (isVideo) {
- await onSetAttribute("has-audio", "");
- await onSetHtmlAttribute("muted", "true");
- }
- };
+ const mediaEdit = { element, projectId, projectSrc, commit: onSetAttributeBatch };
const runBackgroundRemoval = async () => {
if (!onRemoveBackground || !projectSrc || removeBusy) return;
@@ -115,13 +120,8 @@ export function MediaSection({
quality,
onProgress: setRemoveProgress,
});
- await applyCutoutResult(result);
- setRemoveProgress({
- status: "complete",
- progress: 100,
- stage: "Applied cutout",
- ...result,
- });
+ const stage = await commitCutout(mediaEdit, result.outputPath, hasAudio);
+ setRemoveProgress({ status: "complete", progress: 100, stage, ...result });
} catch (error) {
setRemoveProgress({
status: "failed",
@@ -320,9 +320,7 @@ export function MediaSection({
<SegmentedControl
trackName="Muted"
value={hasMuted ? "on" : "off"}
- onChange={(next) => {
- void onSetHtmlAttribute("muted", next === "on" ? "true" : null);
- }}
+ onChange={(next) => void commitMutedToggle(mediaEdit, next === "on")}
options={[
{ label: "On", value: "on" },
{ label: "Off", value: "off" },
@@ -337,15 +335,7 @@ export function MediaSection({
<SegmentedControl
trackName="Has audio track"
value={hasAudio ? "yes" : "no"}
- onChange={(next) => {
- if (next === "yes") {
- void onSetAttribute("has-audio", "true");
- void onSetHtmlAttribute("muted", null);
- } else {
- void onSetAttribute("has-audio", "");
- void onSetHtmlAttribute("muted", "true");
- }
- }}
+ onChange={(next) => void commitHasAudioToggle(mediaEdit, next === "yes")}
options={[
{ label: "Yes", value: "yes" },
{ label: "No", value: "no" },
diff --git a/packages/studio/src/components/editor/propertyPanelTypes.ts b/packages/studio/src/components/editor/propertyPanelTypes.ts
index e79727496f..3b58a02ebc 100644
--- a/packages/studio/src/components/editor/propertyPanelTypes.ts
+++ b/packages/studio/src/components/editor/propertyPanelTypes.ts
@@ -3,6 +3,7 @@ import type { ArcPathSegment, GsapAnimation } from "@hyperframes/parsers/gsap-pa
import type { DomEditSelection } from "./domEditing";
import type { ImportedFontAsset } from "./fontAssets";
import type { GsapAnimationEditCallbacks } from "./gsapAnimationCallbacks";
+import type { CommitDomAttributeBatch } from "../../hooks/domEditCommitTypes";
export interface BackgroundRemovalProgress {
status: "processing" | "complete" | "failed";
@@ -68,6 +69,7 @@ export interface PropertyPanelProps {
value: string | null,
) => Promise<{ changedFiles: number; changedElements: number }>;
onSetHtmlAttribute: (attr: string, value: string | null) => void | Promise<void>;
+ onSetAttributeBatch: CommitDomAttributeBatch;
onRemoveBackground?: (
inputPath: string,
options: {
diff --git a/packages/studio/src/contexts/DomEditContext.tsx b/packages/studio/src/contexts/DomEditContext.tsx
index 4781410626..d84f19ca54 100644
--- a/packages/studio/src/contexts/DomEditContext.tsx
+++ b/packages/studio/src/contexts/DomEditContext.tsx
@@ -21,6 +21,7 @@ const SAVES_AN_EDIT = {
handleDomAttributeQuietCommit: true,
handleDomHtmlAttributeCommit: true,
handleDomAttributesCommit: true,
+ handleDomAttributeBatchCommit: true,
handleDomPathOffsetCommit: true,
handleDomGroupPathOffsetCommit: true,
handleDomZIndexReorderCommit: true,
@@ -175,6 +176,7 @@ export function DomEditProvider({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomPathOffsetCommit,
handleDomGroupPathOffsetCommit,
handleDomZIndexReorderCommit,
@@ -269,6 +271,7 @@ export function DomEditProvider({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomPathOffsetCommit,
handleDomGroupPathOffsetCommit,
handleDomZIndexReorderCommit,
@@ -344,6 +347,7 @@ export function DomEditProvider({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomPathOffsetCommit,
handleDomGroupPathOffsetCommit,
handleDomZIndexReorderCommit,
diff --git a/packages/studio/src/hooks/domEditCommitTypes.ts b/packages/studio/src/hooks/domEditCommitTypes.ts
index 882edf5e4b..714470e0dd 100644
--- a/packages/studio/src/hooks/domEditCommitTypes.ts
+++ b/packages/studio/src/hooks/domEditCommitTypes.ts
@@ -57,3 +57,10 @@ export type PersistDomEditOperations = (
shouldSave?: () => boolean;
},
) => Promise<DomEditPersistOutcome | undefined>;
+
+/** Several data-* and HTML attribute ops on one element as ONE persist and ONE undo entry. */
+export type CommitDomAttributeBatch = (
+ selection: DomEditSelection,
+ operations: PatchOperation[],
+ options: { label: string; prepareContent?: (html: string) => string },
+) => Promise<boolean>;
diff --git a/packages/studio/src/hooks/useDomEditAttributeCommits.ts b/packages/studio/src/hooks/useDomEditAttributeCommits.ts
index cf1ea19cab..062e4a4b81 100644
--- a/packages/studio/src/hooks/useDomEditAttributeCommits.ts
+++ b/packages/studio/src/hooks/useDomEditAttributeCommits.ts
@@ -5,7 +5,7 @@ import {
getDomEditTargetKey,
type DomEditSelection,
} from "../components/editor/domEditing";
-import type { PersistDomEditOperations } from "./domEditCommitTypes";
+import type { CommitDomAttributeBatch, PersistDomEditOperations } from "./domEditCommitTypes";
import { reportDomEditPersistFailure } from "./domEditPersistFailure";
import { bumpDomEditCommitMapVersion, runDomEditCommit } from "./domEditCommitRunner";
import { syncStoredAutomationFromPreview } from "../player/lib/automationStoreSync";
@@ -54,6 +54,15 @@ export function resolveOptimisticAttributeValue(attr: string, value: string | nu
return value === "false" && HTML_BOOLEAN_ATTRIBUTES.has(attr) ? null : value;
}
+function previewAttributeName(op: PatchOperation): string {
+ return op.type === "attribute" ? resolveFullAttrName(op.property, true) : op.property;
+}
+
+function previewAttributeValue(op: PatchOperation): string | null {
+ if (op.type === "html-attribute") return resolveOptimisticAttributeValue(op.property, op.value);
+ return op.value === "" ? null : op.value;
+}
+
function setOrRemovePreviewAttribute(
el: HTMLElement,
fullAttr: string,
@@ -209,7 +218,7 @@ export function useDomEditAttributeCommits({
],
);
- // Commits several data-* attributes on the SAME element in ONE persist call
+ // Commits several data-* and HTML attributes on the SAME element in ONE persist call
// — needed when two attributes together describe a single logical value
// (e.g. a pinned timing range's start+duration): committing them through two
// separate sequential `commitDataAttribute` calls leaves a window where the
@@ -221,32 +230,30 @@ export function useDomEditAttributeCommits({
// explicit, caller-supplied `selection` (not the "current" one) closes both
// gaps — matching `onCommitAnimatedProperties`'s same-shaped fix for GSAP
// property batches.
- const commitDataAttributes = useCallback(
+ const commitAttributeOps = useCallback(
async (
selection: DomEditSelection,
- attrs: Record<string, string | null>,
- options: DataAttributeCommitOptions,
- ) => {
+ ops: PatchOperation[],
+ options: DataAttributeCommitOptions & {
+ prepareContent?: (html: string) => string;
+ syncAutomation?: boolean;
+ },
+ ): Promise<boolean> => {
const iframe = previewIframeRef.current;
- const entries = Object.entries(attrs).map(([attr, value]) => ({
- attr,
- fullAttr: resolveFullAttrName(attr, true),
- value,
+ const entries = ops.map((op) => ({
+ fullAttr: previewAttributeName(op),
+ value: previewAttributeValue(op),
}));
- const commitKey = `${options.coalescePrefix}:${entries
- .map((entry) => entry.attr)
+ const commitKey = `${options.coalescePrefix}:${ops
+ .map((op) => op.property)
.sort()
.join(",")}:${getDomEditTargetKey(selection)}`;
const isLatestCommit = bumpDomEditCommitMapVersion(
domAttributeCommitVersionRef.current,
commitKey,
);
- const ops: PatchOperation[] = entries.map((entry) => ({
- type: "attribute",
- property: entry.attr,
- value: entry.value,
- }));
let captured: CapturedMultiAttributeElement | null = null;
+ let landed = false;
await runDomEditCommit({
capture: () => {
@@ -260,8 +267,7 @@ export function useDomEditAttributeCommits({
apply: () => {
if (!captured) return;
for (const entry of entries) {
- const nextValue = entry.value === null || entry.value === "" ? null : entry.value;
- setOrRemovePreviewAttribute(captured.element, entry.fullAttr, nextValue);
+ setOrRemovePreviewAttribute(captured.element, entry.fullAttr, entry.value);
}
},
persist: () =>
@@ -269,6 +275,7 @@ export function useDomEditAttributeCommits({
label: options.label,
coalesceKey: commitKey,
skipRefresh: options.skipRefresh,
+ prepareContent: options.prepareContent,
}),
shouldRevert: () => isLatestCommit(),
revert: () => {
@@ -283,10 +290,19 @@ export function useDomEditAttributeCommits({
},
onError: (error) => reportDomEditPersistFailure(selection, ops, error, showToast),
shouldResync: () => isLatestCommit() && !!options.refreshAfter,
- resync: () => refreshDomEditSelectionFromPreview(selection),
- onSettled: options.onSettled,
+ resync: () => {
+ refreshDomEditSelectionFromPreview(selection);
+ if (options.syncAutomation) {
+ syncStoredAutomationFromPreview(previewIframeRef.current?.contentDocument ?? null);
+ }
+ },
+ onSettled: (ok) => {
+ landed = ok;
+ options.onSettled?.(ok);
+ },
onFinally: isLatestCommit.release,
});
+ return landed;
},
[
activeCompPath,
@@ -299,14 +315,35 @@ export function useDomEditAttributeCommits({
const handleDomAttributesCommit = useCallback(
async (selection: DomEditSelection, attrs: Record<string, string>) => {
- await commitDataAttributes(selection, attrs, {
- label: "Edit timing",
- coalescePrefix: "attrs",
+ await commitAttributeOps(
+ selection,
+ Object.entries(attrs).map(([property, value]) => ({
+ type: "attribute",
+ property,
+ value,
+ })),
+ {
+ label: "Edit timing",
+ coalescePrefix: "attrs",
+ skipRefresh: false,
+ refreshAfter: true,
+ },
+ );
+ },
+ [commitAttributeOps],
+ );
+
+ const handleDomAttributeBatchCommit: CommitDomAttributeBatch = useCallback(
+ (selection, ops, options) =>
+ commitAttributeOps(selection, ops, {
+ label: options.label,
+ coalescePrefix: "attr-batch",
skipRefresh: false,
refreshAfter: true,
- });
- },
- [commitDataAttributes],
+ syncAutomation: true,
+ prepareContent: options.prepareContent,
+ }),
+ [commitAttributeOps],
);
const handleDomAttributeCommit = useCallback(
@@ -438,5 +475,6 @@ export function useDomEditAttributeCommits({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
};
}
diff --git a/packages/studio/src/hooks/useDomEditCommits.test.tsx b/packages/studio/src/hooks/useDomEditCommits.test.tsx
index cd6cba745f..b7ef890460 100644
--- a/packages/studio/src/hooks/useDomEditCommits.test.tsx
+++ b/packages/studio/src/hooks/useDomEditCommits.test.tsx
@@ -1736,3 +1736,128 @@ describe("useDomEditCommits attribute persist handling", () => {
}
});
});
+
+describe("useDomEditCommits attribute batch commit", () => {
+ beforeEach(() => {
+ ensureCssEscape();
+ vi.clearAllMocks();
+ vi.unstubAllGlobals();
+ document.body.replaceChildren();
+ });
+
+ const original = '<video data-hf-id="hf-card" data-has-audio="true"></video>';
+ const patched = '<video data-hf-id="hf-card" muted></video>';
+ const mute = [
+ { type: "html-attribute" as const, property: "muted", value: "true" },
+ { type: "attribute" as const, property: "has-audio", value: null },
+ ];
+
+ it("sends mixed html and data attribute ops in ONE patch and ONE undo entry", async () => {
+ const fetchMock = stubPatchFetch(
+ {
+ ok: true,
+ changed: true,
+ matched: true,
+ content: patched,
+ path: "index.html",
+ version: "v2",
+ },
+ original,
+ );
+ const { iframe, element } = createPreviewElement(original);
+ const rendered = renderDomEditCommits(createSelection(element, { tagName: "video" }), iframe);
+
+ try {
+ let ok = false;
+ await act(async () => {
+ ok = await rendered.hook.handleDomAttributeBatchCommit(
+ createSelection(element, { tagName: "video" }),
+ mute,
+ { label: "Edit muted" },
+ );
+ });
+
+ const patches = fetchMock.mock.calls.filter(([input]) =>
+ requestUrl(input).includes("/file-mutations/patch-element/"),
+ );
+ expect(ok).toBe(true);
+ expect(patches).toHaveLength(1);
+ const body = JSON.parse(String(patches[0]?.[1]?.body));
+ expect(body.operations).toEqual(mute);
+ expect(rendered.recordEdit).toHaveBeenCalledTimes(1);
+ expect(rendered.recordEdit).toHaveBeenCalledWith(
+ expect.objectContaining({
+ files: { "index.html": { before: original, after: patched } },
+ }),
+ );
+ expect(rendered.reloadPreview).toHaveBeenCalledTimes(1);
+ expect(element.hasAttribute("muted")).toBe(true);
+ expect(element.hasAttribute("data-has-audio")).toBe(false);
+ } finally {
+ rendered.cleanup();
+ }
+ });
+
+ it("folds a document-level prepareContent into the same single undo entry", async () => {
+ stubPatchFetch(
+ {
+ ok: true,
+ changed: true,
+ matched: true,
+ content: patched,
+ path: "index.html",
+ version: "v2",
+ },
+ original,
+ );
+ const writes: string[] = [];
+ const { iframe, element } = createPreviewElement(original);
+ const rendered = renderDomEditCommits(createSelection(element), iframe, {
+ writeProjectFile: async (_path, content) => {
+ writes.push(content);
+ },
+ });
+ const withSibling = `<audio id="a"></audio>${patched}`;
+
+ try {
+ await act(async () => {
+ await rendered.hook.handleDomAttributeBatchCommit(createSelection(element), mute, {
+ label: "Remove background",
+ prepareContent: () => withSibling,
+ });
+ });
+
+ expect(writes).toEqual([withSibling]);
+ expect(rendered.recordEdit).toHaveBeenCalledTimes(1);
+ expect(rendered.recordEdit).toHaveBeenCalledWith(
+ expect.objectContaining({
+ label: "Remove background",
+ files: { "index.html": { before: original, after: withSibling } },
+ }),
+ );
+ } finally {
+ rendered.cleanup();
+ }
+ });
+
+ it("reverts every op in the batch and reports failure when the patch rejects", async () => {
+ stubPatchFetch(new Error("network down"), original);
+ const { iframe, element } = createPreviewElement(original);
+ const rendered = renderDomEditCommits(createSelection(element), iframe);
+
+ try {
+ let ok = true;
+ await act(async () => {
+ ok = await rendered.hook.handleDomAttributeBatchCommit(createSelection(element), mute, {
+ label: "Edit muted",
+ });
+ });
+
+ expect(ok).toBe(false);
+ expect(element.hasAttribute("muted")).toBe(false);
+ expect(element.getAttribute("data-has-audio")).toBe("true");
+ } finally {
+ rendered.cleanup();
+ }
+ });
+});
diff --git a/packages/studio/src/hooks/useDomEditCommits.ts b/packages/studio/src/hooks/useDomEditCommits.ts
index 2c95cb5002..5051b62cde 100644
--- a/packages/studio/src/hooks/useDomEditCommits.ts
+++ b/packages/studio/src/hooks/useDomEditCommits.ts
@@ -239,6 +239,7 @@ export function useDomEditCommits({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomTextCommit,
handleDomTextCommitForSelection,
handleDomRichTextCommit,
@@ -305,6 +306,7 @@ export function useDomEditCommits({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomTextCommit,
handleDomTextCommitForSelection,
handleDomRichTextCommit,
diff --git a/packages/studio/src/hooks/useDomEditSession.ts b/packages/studio/src/hooks/useDomEditSession.ts
index 3830d1e28a..24791bd37a 100644
--- a/packages/studio/src/hooks/useDomEditSession.ts
+++ b/packages/studio/src/hooks/useDomEditSession.ts
@@ -170,7 +170,6 @@ export function useDomEditSession({
refreshDomEditSelectionFromPreview,
});
// ── GSAP cache (hoisted so both useGsapScriptCommits and useDomEditWiring share the same instance) ──
-
const { version: gsapCacheVersion, bump: bumpGsapCache } = useGsapCacheVersion();
const {
@@ -202,6 +201,7 @@ export function useDomEditSession({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomTextCommit,
handleDomTextCommitForSelection,
handleDomRichTextCommit,
@@ -288,7 +288,6 @@ export function useDomEditSession({
});
// ── Element groups (wrap selected elements in a data-hf-group div) ──
-
const { groupSelection, ungroupSelection } = useGroupCommits({
activeCompPath,
showToast,
@@ -363,7 +362,6 @@ export function useDomEditSession({
}, [domEditSelectionRef, ungroupSelection, setActiveGroupElement, showToast]);
// ── Wiring: selection sync, GSAP cache, preview sync, selection handlers ──
-
const {
onClickToSource,
selectedGsapAnimations,
@@ -496,6 +494,7 @@ export function useDomEditSession({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomPathOffsetCommit: handleGsapAwarePathOffsetCommit,
handleDomGroupPathOffsetCommit: handleGsapAwareGroupPathOffsetCommit,
handleDomZIndexReorderCommit,
diff --git a/packages/studio/src/hooks/useDomEditTextCommits.ts b/packages/studio/src/hooks/useDomEditTextCommits.ts
index 4ec9055cd3..c747478b75 100644
--- a/packages/studio/src/hooks/useDomEditTextCommits.ts
+++ b/packages/studio/src/hooks/useDomEditTextCommits.ts
@@ -99,6 +99,7 @@ export function useDomEditTextCommits({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
} = useDomEditAttributeCommits({
activeCompPath,
previewIframeRef,
@@ -485,6 +486,7 @@ export function useDomEditTextCommits({
handleDomAttributeQuietCommit,
handleDomHtmlAttributeCommit,
handleDomAttributesCommit,
+ handleDomAttributeBatchCommit,
handleDomTextCommit,
handleDomTextCommitForSelection,
handleDomRichTextCommit,
diff --git a/packages/studio/src/player/lib/timelineElementHelpers.test.ts b/packages/studio/src/player/lib/timelineElementHelpers.test.ts
new file mode 100644
index 0000000000..b8753fd15d
--- /dev/null
+++ b/packages/studio/src/player/lib/timelineElementHelpers.test.ts
@@ -0,0 +1,16 @@
+import { describe, expect, it } from "vitest";
+import { isVideoAudible } from "./timelineElementHelpers";
+
+describe("isVideoAudible — the compiler's data-has-audio rule", () => {
+ it("explicit data-has-audio wins", () => {
+ expect(isVideoAudible({ tag: "video", hasAudioAttr: "true", muted: false })).toBe(true);
+ expect(isVideoAudible({ tag: "video", hasAudioAttr: "false", muted: false })).toBe(false);
+ });
+ it("no attribute: an unmuted video is audible, a muted one is not", () => {
+ expect(isVideoAudible({ tag: "video", hasAudioAttr: null, muted: false })).toBe(true);
+ expect(isVideoAudible({ tag: "video", hasAudioAttr: undefined, muted: true })).toBe(false);
+ });
+ it("never true for non-video without the attribute", () => {
+ expect(isVideoAudible({ tag: "img", hasAudioAttr: null, muted: false })).toBe(false);
+ });
+});
diff --git a/packages/studio/src/player/lib/timelineElementHelpers.ts b/packages/studio/src/player/lib/timelineElementHelpers.ts
index 3ee3101cbf..c5e3b944de 100644
--- a/packages/studio/src/player/lib/timelineElementHelpers.ts
+++ b/packages/studio/src/player/lib/timelineElementHelpers.ts
@@ -220,12 +220,26 @@ function readVolume(el: Element, media: Element): number | undefined {
return Number.isFinite(volume) ? volume : undefined;
}
+/** The compiler's rule (timingCompiler): explicit data-has-audio wins; otherwise an unmuted <video> is audible. */
+export function isVideoAudible(opts: {
+ tag: string;
+ hasAudioAttr: string | null | undefined;
+ muted: boolean;
+}): boolean {
+ if (opts.hasAudioAttr === "true") return true;
+ if (opts.hasAudioAttr === "false" || opts.hasAudioAttr === "") return false;
+ return opts.tag.toLowerCase() === "video" && !opts.muted;
+}
+
/** What the mixer gets: the compiler's `data-has-audio` rule, muted and volume. */
function applyAudioMetadataFromElement(entry: TimelineElement, el: Element): void {
const media = resolveMediaElement(el) ?? el;
const muted = el.hasAttribute("muted") || media.hasAttribute("muted");
- const hasAudio = el.getAttribute("data-has-audio");
- const sound = hasAudio === null ? el.tagName === "VIDEO" && !muted : hasAudio === "true";
+ const sound = isVideoAudible({
+ tag: el.tagName,
+ hasAudioAttr: el.getAttribute("data-has-audio"),
+ muted,
+ });
setOptional(entry, "hasAudio", sound ? true : undefined);
setOptional(entry, "muted", muted ? true : undefined);
setOptional(entry, "volume", readVolume(el, media));
diff --git a/packages/studio/src/utils/sourcePatcher.ts b/packages/studio/src/utils/sourcePatcher.ts
index 44c9843a69..47ef37cc70 100644
--- a/packages/studio/src/utils/sourcePatcher.ts
+++ b/packages/studio/src/utils/sourcePatcher.ts
@@ -12,7 +12,7 @@ function escapeStyleAttributeValue(value: string, quote: string): string {
}
/** Escape a string for safe use inside a double-quoted HTML attribute. */
-function escapeHtmlAttribute(value: string): string {
+export function escapeHtmlAttribute(value: string): string {
return value
.replace(/&/g, "&amp;")
.replace(/"/g, "&quot;")
diff --git a/packages/studio/src/utils/studioHelpers.test.ts b/packages/studio/src/utils/studioHelpers.test.ts
index 31b7bfb925..3581415dab 100644
--- a/packages/studio/src/utils/studioHelpers.test.ts
+++ b/packages/studio/src/utils/studioHelpers.test.ts
@@ -5,6 +5,7 @@ import {
historyTooltipLabel,
findMatchingTimelineElementId,
findTimelineIdByAncestor,
+ resolveAssetHasAudio,
resolveDroppedAssetDimensions,
resolveElementTrack,
resolveTimelineIdForSelection,
@@ -235,3 +236,32 @@ describe("historyTooltipLabel", () => {
expect(historyTooltipLabel("redo", null)).toMatch(/^Redo \(.+\)$/);
});
});
+
+describe("resolveAssetHasAudio", () => {
+ afterEach(() => vi.unstubAllGlobals());
+
+ const stubMetadata = (response: Response | Error) =>
+ vi.stubGlobal(
+ "fetch",
+ vi.fn(async () => {
+ if (response instanceof Error) throw response;
+ return response;
+ }),
+ );
+
+ it("answers the metadata endpoint's audio flag", async () => {
+ stubMetadata(new Response(JSON.stringify({ metadata: { hasAudio: true } })));
+ expect(await resolveAssetHasAudio("p1", "assets/talk.mp4")).toBe(true);
+ stubMetadata(new Response(JSON.stringify({ metadata: { hasAudio: false } })));
+ expect(await resolveAssetHasAudio("p1", "assets/cutout.webm")).toBe(false);
+ });
+
+ it("answers null when the probe cannot tell", async () => {
+ stubMetadata(new Response("nope", { status: 500 }));
+ expect(await resolveAssetHasAudio("p1", "assets/talk.mp4")).toBeNull();
+ stubMetadata(new Error("offline"));
+ expect(await resolveAssetHasAudio("p1", "assets/talk.mp4")).toBeNull();
+ stubMetadata(new Response(JSON.stringify({ metadata: {} })));
+ expect(await resolveAssetHasAudio("p1", "assets/talk.mp4")).toBeNull();
+ });
+});
diff --git a/packages/studio/src/utils/studioHelpers.ts b/packages/studio/src/utils/studioHelpers.ts
index e9746ac7b2..adfb3bd77b 100644
--- a/packages/studio/src/utils/studioHelpers.ts
+++ b/packages/studio/src/utils/studioHelpers.ts
@@ -388,13 +388,27 @@ export async function resolveDroppedAssetHasAudio(
kind: TimelineAssetKind,
): Promise<boolean> {
if (kind !== "video") return false;
+ return (await resolveAssetHasAudio(projectId, assetPath)) === true;
+}
+
+function isPlainRecord(value: unknown): value is Record<string, unknown> {
+ return typeof value === "object" && value !== null;
+}
+
+/** An asset's audio stream from the metadata endpoint: null when the probe can't tell. */
+export async function resolveAssetHasAudio(
+ projectId: string,
+ assetPath: string,
+): Promise<boolean | null> {
try {
const response = await fetch(mediaMetadataUrl(projectId, assetPath));
- if (!response.ok) return false;
- const data = (await response.json()) as { metadata?: { hasAudio?: boolean } } | null;
- return data?.metadata?.hasAudio === true;
+ if (!response.ok) return null;
+ const data: unknown = await response.json();
+ const metadata = isPlainRecord(data) ? data.metadata : undefined;
+ const hasAudio = isPlainRecord(metadata) ? metadata.hasAudio : undefined;
+ return typeof hasAudio === "boolean" ? hasAudio : null;
} catch {
- return false;
+ return null;
}
}
diff --git a/registry/examples/play-mode/index.html b/registry/examples/play-mode/index.html
index aaf9ecb860..7eb041d181 100644
--- a/registry/examples/play-mode/index.html
+++ b/registry/examples/play-mode/index.html
@@ -142,22 +142,14 @@
<video
id="short_mag_cut"
src="__VIDEO_SRC__"
- muted
playsinline
+ data-has-audio="true"
data-start="0"
data-duration="__VIDEO_DURATION__"
data-track-index="1"
style="width: 100%; height: auto"
></video>
</div>
- <audio
- id="a-roll-audio"
- src="__VIDEO_SRC__"
- data-start="0"
- data-duration="__VIDEO_DURATION__"
- data-track-index="5"
- data-volume="1"
- ></audio>
<style>
#aroll-container {
position: absolute;
diff --git a/registry/examples/swiss-grid/index.html b/registry/examples/swiss-grid/index.html
index 84911190e3..15b9d19f99 100644
--- a/registry/examples/swiss-grid/index.html
+++ b/registry/examples/swiss-grid/index.html
@@ -202,20 +202,12 @@
<video
id="short_mag_cut"
src="__VIDEO_SRC__"
- muted
playsinline
+ data-has-audio="true"
data-start="0"
data-duration="__VIDEO_DURATION__"
data-track-index="1"
></video>
</div>
- <audio
- id="a-roll-audio"
- src="__VIDEO_SRC__"
- data-start="0"
- data-duration="__VIDEO_DURATION__"
- data-track-index="5"
- data-volume="1"
- ></audio>
</body>
</html>
diff --git a/registry/examples/vignelli/index.html b/registry/examples/vignelli/index.html
index bfc989904a..05b110d821 100644
--- a/registry/examples/vignelli/index.html
+++ b/registry/examples/vignelli/index.html
@@ -176,20 +176,12 @@
<video
id="a-roll"
src="__VIDEO_SRC__"
- muted
playsinline
+ data-has-audio="true"
data-start="0"
data-duration="__VIDEO_DURATION__"
data-track-index="0"
></video>
</div>
- <audio
- id="a-roll-audio"
- src="__VIDEO_SRC__"
- data-start="0"
- data-duration="__VIDEO_DURATION__"
- data-track-index="5"
- data-volume="1"
- ></audio>
</body>
</html>
diff --git a/registry/examples/warm-grain/index.html b/registry/examples/warm-grain/index.html
index b3433877f2..6b7a2775b8 100644
--- a/registry/examples/warm-grain/index.html
+++ b/registry/examples/warm-grain/index.html
@@ -125,24 +125,16 @@
</script>
</div>
- <!-- A-Roll Video (muted — audio is separate) -->
+ <!-- A-roll keeps its own sound -->
<video
id="a-roll"
src="__VIDEO_SRC__"
- muted
playsinline
+ data-has-audio="true"
data-start="0"
data-duration="__VIDEO_DURATION__"
data-track-index="0"
></video>
- <audio
- id="a-roll-audio"
- src="__VIDEO_SRC__"
- data-start="0"
- data-duration="__VIDEO_DURATION__"
- data-track-index="5"
- data-volume="1"
- ></audio>
<!-- Compositions -->
<div
diff --git a/scripts/creator-editing-capabilities.test.mjs b/scripts/creator-editing-capabilities.test.mjs
index d9c829fc1c..f24a7e4eb1 100644
--- a/scripts/creator-editing-capabilities.test.mjs
+++ b/scripts/creator-editing-capabilities.test.mjs
@@ -180,7 +180,7 @@ test("core and general-video author temporal edits as duplicated source-range cl
/data-media-start[\s\S]{0,120}data-duration/i,
/data-start/,
/hard cut[\s\S]{0,100}trim[\s\S]{0,100}splice[\s\S]{0,100}reorder/i,
- /separate(?:ly)? authored audio[\s\S]{0,180}(identical|same)[\s\S]{0,100}(range|timing)/i,
+ /sound stays on the clip[\s\S]{0,160}data-has-audio/i,
],
surface,
);
@@ -239,7 +239,7 @@ test("creator editing recipes are copyable, owned, mathematical, and limitation-
/data-playback-rate/,
/consumed source\s*=\s*timeline duration\s*[×*]\s*rate/i,
/natural timeline duration\s*=\s*remaining source\s*\/\s*rate/i,
- /separate audio track/i,
+ /separate `?<audio>`?[\s\S]{0,160}(J\/L cut|replacement audio|voiceover|music)/i,
/final-source[\s\S]{0,100}subcomp[\s\S]{0,100}visual pose/i,
/arbitrary mid-source[\s\S]{0,120}preprocess/i,
/distinct tracks[\s\S]{0,120}overlap[\s\S]{0,120}opposing/i,
@@ -288,7 +288,7 @@ test("crossfade and volume recipes contain executable opposing envelopes", async
/gsap\.timeline\(\{\s*paused:\s*true\s*\}\)/,
/window\.__timelines/,
/autoAlpha|opacity/,
- /<audio[\s\S]*<audio/,
+ /data-has-audio="true"[\s\S]*data-has-audio="true"/,
/data-automation='/,
],
"crossfade recipe",
@@ -318,8 +318,7 @@ test("natural media duration is rate-scaled across every preview surface", async
}
});
-// fallow-ignore-next-line complexity
-test("every temporal source-range recipe includes matching audio markup", async () => {
+test("every temporal source-range recipe keeps sound on the video", async () => {
const recipes = await read(files.editingRecipes);
for (const name of [
"Hard cut",
@@ -331,21 +330,17 @@ test("every temporal source-range recipe includes matching audio markup", async
const start = recipes.indexOf(`## ${name}`);
const next = recipes.indexOf("## ", start + 3);
const section = recipes.slice(start, next < 0 ? recipes.length : next);
- const videos = [...section.matchAll(/<video[\s\S]*?<\/video>/g)].map((m) => m[0]);
- const audios = [...section.matchAll(/<audio[\s\S]*?<\/audio>/g)].map((m) => m[0]);
- assert.ok(
- videos.length > 0 && audios.length >= videos.length,
- `${name}: matching audio missing`,
- );
+ const videos = [...section.matchAll(/<video\b[^>]*>/g)].map((m) => m[0]);
+ assert.ok(videos.length > 0, `${name}: no video markup`);
for (const video of videos) {
- for (const attr of ["src", "data-start", "data-duration", "data-media-start"]) {
- const value = video.match(new RegExp(`${attr}="([^"]+)"`))?.[1];
- assert.ok(
- value && audios.some((audio) => audio.includes(`${attr}="${value}"`)),
- `${name}: audio mismatch ${attr}`,
- );
- }
+ assert.match(video, /data-has-audio="true"/, `${name}: video must keep its sound`);
+ assert.doesNotMatch(video, /\bmuted\b/, `${name}: video must not be muted`);
}
+ assert.doesNotMatch(
+ section,
+ /<audio\b/,
+ `${name}: no duplicate <audio> for the video's own sound`,
+ );
}
});
diff --git a/scripts/generate-template-previews.ts b/scripts/generate-template-previews.ts
index a63642a892..7abf15f850 100644
--- a/scripts/generate-template-previews.ts
+++ b/scripts/generate-template-previews.ts
@@ -37,6 +37,7 @@ import {
executeRenderJob,
} from "../packages/producer/src/index.js";
import { openOpaqueCapture } from "./preview-capture.js";
+import { patchMediaPlaceholders } from "../packages/cli/src/commands/initMedia.ts";
const scriptDir = dirname(fileURLToPath(import.meta.url));
const repoRoot = resolve(scriptDir, "..");
@@ -64,13 +65,7 @@ function patchTemplateHtml(dir: string, durationSeconds: number): void {
for (const entry of htmlEntries) {
const file = join(entry.parentPath, entry.name);
- let content = readFileSync(file, "utf-8");
- content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/video>/g, "");
- content = content.replace(/<video[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
- content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>[\s\S]*?<\/audio>/g, "");
- content = content.replace(/<audio[^>]*src="__VIDEO_SRC__"[^>]*>/g, "");
- const dur = String(Math.round(durationSeconds * 100) / 100);
- content = content.replaceAll("__VIDEO_DURATION__", dur);
+ const content = patchMediaPlaceholders(readFileSync(file, "utf-8"), { durationSeconds });
writeFileSync(file, content, "utf-8");
}
}
diff --git a/skills-manifest.json b/skills-manifest.json
index da195b39f1..b1490ddb7a 100644
--- a/skills-manifest.json
+++ b/skills-manifest.json
@@ -2,7 +2,7 @@
"source": "heygen-com/hyperframes",
"skills": {
"embedded-captions": {
- "hash": "7c6f147f722cc4c4",
+ "hash": "9ae9e4e06bf4b61e",
"files": 144
},
"faceless-explainer": {
@@ -14,7 +14,7 @@
"files": 2
},
"general-video": {
- "hash": "77938376e7cc61fc",
+ "hash": "a4f4532304ff7ec5",
"files": 5
},
"hyperframes": {
@@ -22,7 +22,7 @@
"files": 29
},
"hyperframes-animation": {
- "hash": "3055d5038b7fe96b",
+ "hash": "2a567656b9291f5a",
"files": 122
},
"hyperframes-audio": {
@@ -34,11 +34,11 @@
"files": 11
},
"hyperframes-core": {
- "hash": "b632d12d11eb54e8",
+ "hash": "db99256dd85d0460",
"files": 11
},
"hyperframes-creative": {
- "hash": "922bce76a69b97d9",
+ "hash": "7835cfaad29f1fcb",
"files": 79
},
"hyperframes-keyframes": {
@@ -62,7 +62,7 @@
"files": 23
},
"music-to-video": {
- "hash": "3c982c2ddeac43f2",
+ "hash": "887a71e8575e0342",
"files": 169
},
"pr-to-video": {
@@ -74,7 +74,7 @@
"files": 34
},
"remotion-to-hyperframes": {
- "hash": "4fcef874801e58f7",
+ "hash": "e13e3681d9b0dd75",
"files": 77
},
"slideshow": {
@@ -82,7 +82,7 @@
"files": 2
},
"talking-head-recut": {
- "hash": "eac2082ea7096802",
+ "hash": "865d98665f0011da",
"files": 28
}
}
diff --git a/skills/embedded-captions/modes/cinematic/cinematic-cream/template.html b/skills/embedded-captions/modes/cinematic/cinematic-cream/template.html
index 7833cce4a8..318130adbf 100644
--- a/skills/embedded-captions/modes/cinematic/cinematic-cream/template.html
+++ b/skills/embedded-captions/modes/cinematic/cinematic-cream/template.html
@@ -153,23 +153,14 @@
<video
id="a-roll"
src="source.mp4"
- muted
playsinline
+ data-has-audio="true"
data-duration="{{DURATION}}"
data-track-index="0"
style="z-index: 1"
></video>
<div id="stage">{{GROUPS_HTML}}</div>
-
- <audio
- id="a-roll-audio"
- src="source.mp4"
- data-start="0"
- data-duration="{{DURATION}}"
- data-track-index="3"
- data-volume="1"
- ></audio>
</div>
<script>
diff --git a/skills/embedded-captions/modes/cinematic/engine.html b/skills/embedded-captions/modes/cinematic/engine.html
index 2773bed113..3928c8532a 100644
--- a/skills/embedded-captions/modes/cinematic/engine.html
+++ b/skills/embedded-captions/modes/cinematic/engine.html
@@ -7,7 +7,7 @@
typography. Compiled by make-composition.cjs.
Contract (unchanged from cinematic-cream — gates depend on it):
- #root[data-composition-id="main"] · #a-roll · #stage · #a-roll-audio
+ #root[data-composition-id="main"] · #a-roll · #stage
.plane > .cap > .w spans · paused seek-safe GSAP on window.__timelines["main"]
container fade via tl.set at g.in−0.01 · exit fade + display:none at g.out
no Math.random / Date.now / CSS keyframes / repeat:-1
@@ -102,23 +102,14 @@
<video
id="a-roll"
src="source.mp4"
- muted
playsinline
+ data-has-audio="true"
data-duration="{{DURATION}}"
data-track-index="0"
style="z-index: 1"
></video>
<div id="stage">{{GROUPS_HTML}}</div>
-
- <audio
- id="a-roll-audio"
- src="source.mp4"
- data-start="0"
- data-duration="{{DURATION}}"
- data-track-index="3"
- data-volume="1"
- ></audio>
</div>
<script>
diff --git a/skills/embedded-captions/scripts/make-theme.cjs b/skills/embedded-captions/scripts/make-theme.cjs
index 4e2f0ea31d..3ae65f1505 100644
--- a/skills/embedded-captions/scripts/make-theme.cjs
+++ b/skills/embedded-captions/scripts/make-theme.cjs
@@ -580,13 +580,11 @@ ${css}
<body>
<div id="root" data-composition-id="main" data-start="0" data-duration="${DUR}"
data-width="${W}" data-height="${H}">
- <video id="a-roll" src="source.mp4" muted playsinline
+ <video id="a-roll" src="source.mp4" playsinline data-has-audio="true"
data-duration="${DUR}" data-track-index="0"></video>
<div id="stage">
${stageHtml}
</div>
- <audio id="a-roll-audio" src="source.mp4" data-start="0" data-duration="${DUR}"
- data-track-index="3" data-volume="1"></audio>
</div>
<script>
window.__timelines = window.__timelines || {};
diff --git a/skills/embedded-captions/scripts/transcribe.cjs b/skills/embedded-captions/scripts/transcribe.cjs
index 5be62262ee..0ceff74c5e 100644
--- a/skills/embedded-captions/scripts/transcribe.cjs
+++ b/skills/embedded-captions/scripts/transcribe.cjs
@@ -143,7 +143,9 @@ function main() {
process.exit(2);
}
const audio = path.join(project, "audio.mp3");
- if (!fs.existsSync(audio))
+ // A stray audio.mp3 in the project makes lint call the render silent; only remove one this run made.
+ const audioIsScratch = !fs.existsSync(audio);
+ if (audioIsScratch)
cp.execFileSync(
"ffmpeg",
["-y", "-i", src, "-vn", "-acodec", "libmp3lame", "-q:a", "2", audio],
@@ -281,6 +283,8 @@ function main() {
}
}
+ if (audioIsScratch) fs.rmSync(audio, { force: true });
+
const text = words
.map((w) => w.text)
.join(" ")
diff --git a/skills/embedded-captions/scripts/transcribe.test.mjs b/skills/embedded-captions/scripts/transcribe.test.mjs
index 5882a52854..9b5bea3761 100644
--- a/skills/embedded-captions/scripts/transcribe.test.mjs
+++ b/skills/embedded-captions/scripts/transcribe.test.mjs
@@ -1,6 +1,6 @@
import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";
-import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
@@ -39,3 +39,31 @@ test("the CLI fallback labels the transcript with the engine the CLI ran", () =>
rmSync(root, { recursive: true, force: true });
}
});
+
+test("the extracted audio scratch file does not outlive the run", () => {
+ const root = mkdtempSync(join(tmpdir(), "embedded-captions-transcribe-"));
+ try {
+ const cliDir = join(root, "hf", "packages", "cli", "dist");
+ mkdirSync(cliDir, { recursive: true });
+ writeFileSync(join(cliDir, "cli.js"), STUB_CLI);
+ const project = join(root, "project");
+ mkdirSync(project);
+ execFileSync(
+ "ffmpeg",
+ ["-y", "-f", "lavfi", "-i", "sine=frequency=440:duration=1", join(project, "source.mp4")],
+ { stdio: "ignore" },
+ );
+ execFileSync(
+ process.execPath,
+ [fileURLToPath(new URL("./transcribe.cjs", import.meta.url)), project],
+ {
+ stdio: "ignore",
+ env: { ...process.env, HYPERFRAMES_ROOT: join(root, "hf"), TRANSCRIBE_ENGINE: "whisper" },
+ },
+ );
+ assert.equal(existsSync(join(project, "audio.mp3")), false);
+ assert.equal(existsSync(join(project, "transcript.json")), true);
+ } finally {
+ rmSync(root, { recursive: true, force: true });
+ }
+});
diff --git a/skills/general-video/SKILL.md b/skills/general-video/SKILL.md
index d8d62368b3..95e32dafb9 100644
--- a/skills/general-video/SKILL.md
+++ b/skills/general-video/SKILL.md
@@ -64,8 +64,7 @@ Do not invent synonyms for these states. An ongoing “just build it” signal i
For a hard cut, trim, splice, or reorder of existing footage, duplicate the same
video source into multiple clip elements. On each copy, set the source range
with `data-media-start` plus `data-duration`, then set authored placement/order
-with `data-start`. Separately authored audio follows the identical clip ranges
-and timing on matching `<audio>` elements. `/hyperframes-core` owns this temporal
+with `data-start`. Each video segment keeps its sound: the sound stays on the clip (`data-has-audio="true"`), so cutting the video cuts its sound. `/hyperframes-core` owns this temporal
edit; use `/hyperframes-keyframes` only for visual-property animation such as
zoom, punch, pan, crop, mask, or `clip-path` on an inner wrapper.
Copy the full contracts from `../hyperframes-core/references/creator-editing-recipes.md`.
diff --git a/skills/hyperframes-animation/examples/workflow-approve-press.html b/skills/hyperframes-animation/examples/workflow-approve-press.html
index a5de4d62ab..1333d27747 100644
--- a/skills/hyperframes-animation/examples/workflow-approve-press.html
+++ b/skills/hyperframes-animation/examples/workflow-approve-press.html
@@ -393,7 +393,7 @@
<!--
Real asset path (replace for production).
If missing, the CSS mockup below shows through the transparent
- poster region. The video stays muted per HyperFrames rules.
+ poster region. Muted: decorative footage.
-->
<!-- Decorative CSS-mockup of an editor — paused screenshot feel. -->
<div class="demo-mockup">
diff --git a/skills/hyperframes-animation/techniques.md b/skills/hyperframes-animation/techniques.md
index afdb596525..d851a75093 100644
--- a/skills/hyperframes-animation/techniques.md
+++ b/skills/hyperframes-animation/techniques.md
@@ -203,7 +203,7 @@ Vector animations that play inside a composition. Use for logos, character anima
## 6. Video Compositing
-Embed real video footage inside compositions. Videos must be `muted` with `playsinline`.
+Embed real video footage inside compositions. Videos are `playsinline`; add `muted` for silent footage or b-roll, or `data-has-audio="true"` when the clip's own sound should play.
```html
<div class="video-frame" style="width:680px;height:840px;border-radius:16px;overflow:hidden;">
diff --git a/skills/hyperframes-core/references/creator-editing-recipes.md b/skills/hyperframes-core/references/creator-editing-recipes.md
index b0c150980f..bedf6ee6e5 100644
--- a/skills/hyperframes-core/references/creator-editing-recipes.md
+++ b/skills/hyperframes-core/references/creator-editing-recipes.md
@@ -4,7 +4,7 @@ Use these copyable contracts after `tracks-and-clips.md`. Global math: **consume
Before any edit, run `npx hyperframes timeline` (add `--json` for a machine-readable list) to see the project's tracks and clips instead of reading the HTML.
-These recipes keep sound on a separate `<audio>` element with the `<video>` muted, which is the pattern to reach for when picture and sound are cut independently. An unmuted `<video>` that declares `data-has-audio="true"` is also mixed, so a separate track is a choice, not a requirement.
+These recipes keep a video's sound on the `<video>` (`data-has-audio="true"`, no `muted`), so cutting the video cuts its sound. Use a separate `<audio>` only when picture and sound must be cut independently (J/L cuts, replacement audio) or for other sound (music, voiceover) — the recipes that do say so. Silent footage or b-roll: replace `data-has-audio="true"` with `muted` in these blocks.
**Every `<video>` and `<audio>` below carries an `id`, and that is not cosmetic**: `lint` errors with `media_missing_id` on timed media without one, and an id-less `<audio>` is never picked up by the mixer, so the render comes out silent. Keep the ids when you copy a recipe.
@@ -18,8 +18,8 @@ These recipes keep sound on a separate `<audio>` element with the `<video>` mute
data-duration="2"
data-media-start="4"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
<video
id="b"
@@ -28,28 +28,12 @@ These recipes keep sound on a separate `<audio>` element with the `<video>` mute
data-duration="3"
data-media-start="10"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
-<audio
- id="a-audio"
- src="take.mp4"
- data-start="0"
- data-duration="2"
- data-media-start="4"
- data-track-index="10"
-></audio>
-<audio
- id="b-audio"
- src="take.mp4"
- data-start="2"
- data-duration="3"
- data-media-start="10"
- data-track-index="10"
-></audio>
```
-Timeline math: B starts at A start + duration. Source math: each range starts at `data-media-start`; consumed source = timeline duration × rate. Audio follows: duplicate matching `<audio>` ranges/timing. Owner: `/hyperframes-core`. Limit: adjacent windows only; author the two windows edge to edge. Same-track overlap is valid; both clips paint in CSS order.
+Timeline math: B starts at A start + duration. Source math: each range starts at `data-media-start`; consumed source = timeline duration × rate. Audio follows: the sound moves with each video clip. Owner: `/hyperframes-core`. Limit: adjacent windows only; author the two windows edge to edge. Same-track overlap is valid; both clips paint in CSS order.
## Trim in/out
@@ -61,20 +45,12 @@ Timeline math: B starts at A start + duration. Source math: each range starts at
data-duration="3"
data-media-start="6"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
-<audio
- id="shot-1-audio"
- src="take.mp4"
- data-start="1"
- data-duration="3"
- data-media-start="6"
- data-track-index="10"
-></audio>
```
-Timeline math: visible window is `[1,4]`. Source math: in=6, out=6+3 at 1x; never invent source-end syntax. Audio follows: matching separate audio track uses the same three attributes. Owner: `/hyperframes-core`. Limit: use another clip for another range.
+Timeline math: visible window is `[1,4]`. Source math: in=6, out=6+3 at 1x; never invent source-end syntax. Audio follows: the sound moves with the video clip, using the same three attributes. Owner: `/hyperframes-core`. Limit: use another clip for another range.
## Split / splice
@@ -86,8 +62,8 @@ Timeline math: visible window is `[1,4]`. Source math: in=6, out=6+3 at 1x; neve
data-duration="2"
data-media-start="0"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
<video
id="shot-2"
@@ -96,28 +72,12 @@ Timeline math: visible window is `[1,4]`. Source math: in=6, out=6+3 at 1x; neve
data-duration="2"
data-media-start="8"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
-<audio
- id="shot-1-audio"
- src="take.mp4"
- data-start="0"
- data-duration="2"
- data-media-start="0"
- data-track-index="10"
-></audio>
-<audio
- id="shot-2-audio"
- src="take.mp4"
- data-start="2"
- data-duration="2"
- data-media-start="8"
- data-track-index="10"
-></audio>
```
-Timeline math: splice at t=2. Source math: independent source offsets select kept pieces. Audio follows: split matching audio identically. Owner: `/hyperframes-core`. Limit: source cuts are core, never keyframes.
+Timeline math: splice at t=2. Source math: independent source offsets select kept pieces. Audio follows: the sound moves with each video clip, so it splits identically. Owner: `/hyperframes-core`. Limit: source cuts are core, never keyframes.
## Duplicate / reuse same source
@@ -129,8 +89,8 @@ Timeline math: splice at t=2. Source math: independent source offsets select kep
data-duration="1"
data-media-start="2"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
<video
id="shot-2"
@@ -139,28 +99,12 @@ Timeline math: splice at t=2. Source math: independent source offsets select kep
data-duration="1"
data-media-start="2"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
-<audio
- id="shot-1-audio"
- src="take.mp4"
- data-start="0"
- data-duration="1"
- data-media-start="2"
- data-track-index="10"
-></audio>
-<audio
- id="shot-2-audio"
- src="take.mp4"
- data-start="4"
- data-duration="1"
- data-media-start="2"
- data-track-index="10"
-></audio>
```
-Timeline math: copies may occupy different starts. Source math: identical offsets reuse identical source. Audio follows: duplicate the separate audio track too. Owner: `/hyperframes-core`. Limit: every element needs a unique id when ids are present.
+Timeline math: copies may occupy different starts. Source math: identical offsets reuse identical source. Audio follows: the sound moves with each video clip, so each copy carries its own. Owner: `/hyperframes-core`. Limit: every element needs a unique id when ids are present.
## Reorder
@@ -172,8 +116,8 @@ Timeline math: copies may occupy different starts. Source math: identical offset
data-duration="2"
data-media-start="10"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
<video
id="shot-2"
@@ -182,28 +126,12 @@ Timeline math: copies may occupy different starts. Source math: identical offset
data-duration="2"
data-media-start="2"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
-<audio
- id="shot-1-audio"
- src="take.mp4"
- data-start="0"
- data-duration="2"
- data-media-start="10"
- data-track-index="10"
-></audio>
-<audio
- id="shot-2-audio"
- src="take.mp4"
- data-start="2"
- data-duration="2"
- data-media-start="2"
- data-track-index="10"
-></audio>
```
-Timeline math: `data-start` defines authored order. Source math: source offsets need not be chronological. Audio follows: reorder identical matching audio windows. Owner: `/hyperframes-core`. Limit: reordering changes placement only, not source ranges.
+Timeline math: `data-start` defines authored order. Source math: source offsets need not be chronological. Audio follows: the sound moves with the video clip, so reordering clips reorders their sound. Owner: `/hyperframes-core`. Limit: reordering changes placement only, not source ranges.
## Freeze / hold
@@ -224,12 +152,12 @@ Timeline math: the still owns its hold duration. Source math: final-source frame
data-media-start="4"
data-playback-rate="0.5"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
```
-Timeline math: duration is authored timeline time. Source math: consumed source = timeline duration × rate; natural timeline duration = remaining source / rate. Audio follows: matching separate audio track uses the same constant rate. Owner: `/hyperframes-core`. Limit: normalized 0.1..10. For a speed ramp put a `rate` lane in `data-automation`, e.g. `{"version":1,"lanes":[{"target":"rate","points":[{"t":0,"v":1},{"t":2,"v":4}]}]}`; it wins over the constant.
+Timeline math: duration is authored timeline time. Source math: consumed source = timeline duration × rate; natural timeline duration = remaining source / rate. Audio follows: the sound moves with the video clip and plays at the same constant rate. Owner: `/hyperframes-core`. Limit: normalized 0.1..10. For a speed ramp put a `rate` lane in `data-automation`, e.g. `{"version":1,"lanes":[{"target":"rate","points":[{"t":0,"v":1},{"t":2,"v":4}]}]}`; it wins over the constant.
## Zoom / punch
@@ -250,7 +178,7 @@ tl.fromTo(
);
```
-Timeline math: move spans four authored seconds. Source math: unchanged. Audio follows: matching clip timing remains separate. Owner: `/hyperframes-keyframes`. Limit: authored geometry, not automatic face tracking.
+Timeline math: move spans four authored seconds. Source math: unchanged. Audio follows: the sound stays on the video clip; the tween does not touch it. Owner: `/hyperframes-keyframes`. Limit: authored geometry, not automatic face tracking.
## Crop / reframe
@@ -271,7 +199,7 @@ tl.fromTo(
);
```
-Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip keeps its own range. Audio follows: place matching audio on its own tracks. Owner: `/hyperframes-keyframes` + `/hyperframes-animation`. Limit: visual mask/polygon/split-screen only; source cuts stay `/hyperframes-core`.
+Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip keeps its own range. Audio follows: the sound stays on each video clip. Owner: `/hyperframes-keyframes` + `/hyperframes-animation`. Limit: visual mask/polygon/split-screen only; source cuts stay `/hyperframes-core`.
## Crossfade
@@ -283,8 +211,9 @@ Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip
data-duration="3"
data-track-index="0"
src="a.mp4"
- muted
+ data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":1},{"t":2.5,"v":1},{"t":3,"v":0}]}]}'
playsinline
+ data-has-audio="true"
></video>
</div>
<div id="b-visual" class="inner">
@@ -294,26 +223,11 @@ Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip
data-duration="3"
data-track-index="1"
src="b.mp4"
- muted
+ data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":0},{"t":0.5,"v":1},{"t":3,"v":1}]}]}'
playsinline
+ data-has-audio="true"
></video>
</div>
-<audio
- id="a-audio"
- src="a.mp4"
- data-start="0"
- data-duration="3"
- data-track-index="10"
- data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":1},{"t":2.5,"v":1},{"t":3,"v":0}]}]}'
-></audio>
-<audio
- id="b-audio"
- src="b.mp4"
- data-start="2.5"
- data-duration="3"
- data-track-index="11"
- data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":0},{"t":0.5,"v":1},{"t":3,"v":1}]}]}'
-></audio>
<script>
const tl = gsap.timeline({ paused: true });
tl.set("#b-visual", { opacity: 0 }, 0)
@@ -323,7 +237,7 @@ Timeline math: overlap placed clips for the 0.5s handoff. Source math: each clip
</script>
```
-Timeline math: distinct tracks overlap by 0.5s with opposing opacity envelopes. Source math: each source range remains independent. Audio follows: opposing volume envelopes on distinct audio tracks. Owner: `/hyperframes-core` + `/hyperframes-keyframes` + `/hyperframes-audio`. Limit: the crossfade is the opacity/volume envelopes, not a source-level dissolve.
+Timeline math: distinct tracks overlap by 0.5s with opposing opacity envelopes. Source math: each source range remains independent. Audio follows: opposing volume envelopes on each video's own `data-automation`, because the sound stays on the video. Owner: `/hyperframes-core` + `/hyperframes-keyframes` + `/hyperframes-audio`. Limit: the crossfade is the opacity/volume envelopes, not a source-level dissolve.
## Volume fades / ducking
@@ -338,7 +252,7 @@ Timeline math: distinct tracks overlap by 0.5s with opposing opacity envelopes.
></audio>
```
-Timeline math: lane `t` is clip-local authored time: fade-in 0–1, duck down 2–2.2, hold 2.2–3, duck up 3–3.2, fade-out 4–5. Source math: source selection still uses core attributes. Audio follows: the explicit down-hold-up envelope affects this separate audio track. Owner: `/hyperframes-audio`. Limit: automation is not source retiming.
+Timeline math: lane `t` is clip-local authored time: fade-in 0–1, duck down 2–2.2, hold 2.2–3, duck up 3–3.2, fade-out 4–5. Source math: source selection still uses core attributes. Audio follows: the explicit down-hold-up envelope affects this `<audio>` (music is separate sound). The same lane works on a `<video data-has-audio="true">`. Owner: `/hyperframes-audio`. Limit: automation is not source retiming.
**One rule for volume over time: use the lane.** `lint` accepts a timeline tween on `volume` too, but when a track has both, the lane wins and the tween is ignored (`audio_volume_double_automation`). Never add a lane to a track that already has a `volume` tween, and never add a tween to a track that has a lane; edit the one that exists. To ramp 0.1 to 0.5 over ten seconds, write `{"t":0,"v":0.1},{"t":10,"v":0.5}`. `t` is seconds from the clip's own start, so a ramp past `data-duration` never finishes: check the clip's length before choosing the times. `data-volume` stays as the static level of the clip and combines with nothing else you author here.
@@ -353,21 +267,55 @@ Timeline math: lane `t` is clip-local authored time: fade-in 0–1, duck down 2
data-media-start="8"
data-playback-rate="2"
data-track-index="0"
+ playsinline
+ data-has-audio="true"
+></video>
+```
+
+Timeline math: picture and sound share start/duration because the sound stays on the clip (`data-has-audio="true"`). Source math: both consume four source seconds. Audio follows: identical timing, range, and rate, with nothing to keep in sync. Owner: `/hyperframes-core` + `/hyperframes-audio`. Limit: no waveform auto-sync or drift correction.
+
+A J cut or L cut is the case that needs a separate `<audio>`: picture and sound are cut independently, so the sound gets its own element (the same goes for replacement audio, a voiceover, or music). Mute the video whose sound you are replacing.
+
+```html
+<!-- Outgoing shot: picture runs 0-5, its own sound is a separate clip that ends at the audio cut (4). -->
+<video
+ id="shot-1"
+ src="intro.mp4"
+ data-start="0"
+ data-duration="5"
+ data-track-index="0"
muted
playsinline
></video>
<audio
id="shot-1-audio"
+ src="intro.mp4"
+ data-start="0"
+ data-duration="4"
+ data-track-index="10"
+></audio>
+<!-- Incoming shot: picture starts at 5, its sound leads it by one second. -->
+<video
+ id="shot-2"
src="take.mp4"
- data-start="3"
- data-duration="2"
- data-media-start="8"
- data-playback-rate="2"
+ data-start="5"
+ data-duration="3"
+ data-media-start="12"
+ data-track-index="0"
+ muted
+ playsinline
+></video>
+<audio
+ id="shot-2-audio"
+ src="take.mp4"
+ data-start="4"
+ data-duration="4"
+ data-media-start="11"
data-track-index="10"
></audio>
```
-Timeline math: picture and sound share start/duration. Source math: both consume four source seconds. Audio follows: identical timing, range, and rate on the separate audio track. Owner: `/hyperframes-core` + `/hyperframes-audio`. Limit: no waveform auto-sync or drift correction.
+The sound leads the picture by one second (a J cut): `shot-2-audio` starts at 4 and reads from source 11, while the picture starts at 5 and reads from 12. Both stay on the same source clock. Every video in a J or L cut is `muted` and its sound is its own `<audio>`: the outgoing shot's `<audio>` ends at the audio cut (4) while its picture carries on to 5, so two sounds never overlap on the same source. An audible `<video>` and an `<audio>` on the same file are only flagged when their time windows overlap.
## Align a sound to an on-screen event
@@ -426,7 +374,7 @@ Write what Studio writes when a person drops a file on the timeline, so an agent
- **Image: `data-duration` is optional and defaults to 3 seconds**, the same as a dropped image, because a still has no length of its own. Write it only for another length. A test keeps the 3 equal to the default in code.
- **Video and audio: `data-start` is enough.** The length comes from the media itself. An authored `data-duration` shorter than the file is a trim, never a requirement; leave it out unless the request asks for a shorter clip.
- **Start: the playhead or the requested time, never a silent `0`.** Studio's asset-panel Add uses the playhead time on track `0`; a drop uses the drop point.
-- Give every clip `id`, `class="clip"`, `data-start` and `data-track-index`. Video is `muted playsinline`; audio carries `data-volume="1"`.
+- Give every clip `id`, `class="clip"`, `data-start` and `data-track-index`. A video with sound is `playsinline data-has-audio="true"`; silent footage and b-roll is `muted playsinline`. Audio carries `data-volume="1"`.
- Then make sure the root composition's `data-duration` is at least the clip's end (`data-start` plus its length: 3 for an image unless you set another, the media's length for video and audio): Studio raises a declared root duration to cover the new clip, so an agent must too, or the clip lies past the end and never plays.
- **Images and video fill the whole frame**: absolutely positioned at `left: 0; top: 0`, `width` and `height` equal to the composition's `data-width` and `data-height`, `object-fit: contain`. Studio does not know a dropped file's natural size, so it does not centre a smaller one.
- `z-index` is the number of top-level clips already in that file plus one (at least `1`); later clips stack above earlier ones.
@@ -479,13 +427,13 @@ Inside a sub-composition file, `data-start` is scene-local (see `## Align a soun
data-duration="4"
data-media-start="0"
data-track-index="0"
- muted
playsinline
+ data-has-audio="true"
></video>
```
-Timeline math: change only `src`. Source math: reset `data-media-start` to the offset you want in the NEW file, and set `data-duration` no longer than the new file's remaining length (probe it with `ffprobe`). Audio follows: a separate `<audio>` that pointed at the old file needs the same `src` swap. Keep `id`, `data-start`, `data-track-index` and any `data-automation` so nothing else moves. Run `lint`: `audio_src_not_found` and `media_src_kind_mismatch` catch a wrong path or kind. Owner: `/hyperframes-core`. Limit: a still swapped for a video (or the reverse) is a tag change, not a swap.
+Timeline math: change only `src`. Source math: reset `data-media-start` to the offset you want in the NEW file, and set `data-duration` no longer than the new file's remaining length (probe it with `ffprobe`). Audio follows: the sound moves with the video clip; a separate `<audio>` that pointed at the old file (music, voiceover) needs its own `src` swap. Keep `id`, `data-start`, `data-track-index` and any `data-automation` so nothing else moves. Run `lint`: `audio_src_not_found` and `media_src_kind_mismatch` catch a wrong path or kind. Owner: `/hyperframes-core`. Limit: a still swapped for a video (or the reverse) is a tag change, not a swap.
## Split a section and change its speed
-Timeline math: a section that is a sub-composition or a group of clips has no `data-playback-rate` of its own to set; split it by giving each half its own host or clips and shift everything after the cut by the length change. New length of a part = old length / rate. Every later `data-start` (clips, audio, root-timeline tweens) moves by the same delta. Source math: `<video>` and `<audio>` parts use `data-playback-rate` (0.1 to 10, constant) per the constant-speed recipe above, with matching audio. A speed ramp (a rate that changes within one clip) is a `rate` lane in `data-automation` on the `<video>`/`<audio>`; see `docs/reference/speed-ramps`. Say which you did.
+Timeline math: a section that is a sub-composition or a group of clips has no `data-playback-rate` of its own to set; split it by giving each half its own host or clips and shift everything after the cut by the length change. New length of a part = old length / rate. Every later `data-start` (clips, audio, root-timeline tweens) moves by the same delta. Source math: `<video>` and `<audio>` parts use `data-playback-rate` (0.1 to 10, constant) per the constant-speed recipe above, the sound moves with the video. A speed ramp (a rate that changes within one clip) is a `rate` lane in `data-automation` on the `<video>`/`<audio>`; see `docs/reference/speed-ramps`. Say which you did.
diff --git a/skills/hyperframes-core/references/data-attributes.md b/skills/hyperframes-core/references/data-attributes.md
index 4f410a8656..8f98dddd51 100644
--- a/skills/hyperframes-core/references/data-attributes.md
+++ b/skills/hyperframes-core/references/data-attributes.md
@@ -26,15 +26,15 @@ The root should be `position: relative`, have explicit pixel dimensions, and hid
**Nesting is allowed.** A timed element inside a wrapper is still timed, and a timed ancestor clamps its descendants: a child cannot be visible while its timed ancestor is hidden. Direct children of the root get automatic layout (see "Root-level clips get automatic layout" below); nested ones do not, so give them their own positioning.
-| Attribute | Required | Meaning |
-| ------------------ | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `id` | Yes on `<video>`/`<audio>`, else recommended | `lint` errors with `media_missing_id` on media without one, and an id-less `<audio>` is never mixed, so the render is **silent**. Elsewhere it is a warning (`studio_missing_editable_id`): Studio needs a stable edit target, and timeline targets reference it. |
-| `data-start` | Yes | Start time in seconds, or a supported clip-time reference. This attribute is what marks the element as timed. |
-| `data-duration` | Required for `div` and sub-compositions | Duration in seconds. An `img` with `data-start` defaults to 3 s; video/audio default to the source length (less the playback offset, over the rate) once known; an authored value trims. Without any resolvable duration the element has no end and stays visible for the rest of the composition. |
-| `data-track-index` | No | Studio timeline lane, display only. The render never reads it, and clips on one track may overlap in time. Absent, the parser defaults it and Studio lays out one lane per clip. Two `<audio>` elements on the same index that overlap in time raise a `lint` warning. |
-| `data-media-start` | No | Offset into the media source, in seconds. |
-| `data-volume` | No | Static audio gain, default `1` (0 dB). `0` is silence and values above `1` boost, up to `3.98` (+12 dB) — Studio's fader writes this. For fades and ducking, use the `data-automation` volume lane (see `creator-editing-recipes.md`). |
-| `data-has-audio` | No (`<video>` only) | `"true"` to declare the video carries an audio track when auto-detection would miss it. |
+| Attribute | Required | Meaning |
+| ------------------ | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `id` | Yes on `<video>`/`<audio>`, else recommended | `lint` errors with `media_missing_id` on media without one, and an id-less `<audio>` is never mixed, so the render is **silent**. Elsewhere it is a warning (`studio_missing_editable_id`): Studio needs a stable edit target, and timeline targets reference it. |
+| `data-start` | Yes | Start time in seconds, or a supported clip-time reference. This attribute is what marks the element as timed. |
+| `data-duration` | Required for `div` and sub-compositions | Duration in seconds. An `img` with `data-start` defaults to 3 s; video/audio default to the source length (less the playback offset, over the rate) once known; an authored value trims. Without any resolvable duration the element has no end and stays visible for the rest of the composition. |
+| `data-track-index` | No | Studio timeline lane, display only. The render never reads it, and clips on one track may overlap in time. Absent, the parser defaults it and Studio lays out one lane per clip. Two `<audio>` elements on the same index that overlap in time raise a `lint` warning. |
+| `data-media-start` | No | Offset into the media source, in seconds. |
+| `data-volume` | No | Static audio gain, default `1` (0 dB). `0` is silence and values above `1` boost, up to `3.98` (+12 dB) — Studio's fader writes this. For fades and ducking, use the `data-automation` volume lane (see `creator-editing-recipes.md`). |
+| `data-has-audio` | Required on a timed `<video>` unless it is `muted` | `"true"` keeps the file's sound on this clip (the default for footage with sound). Silent footage uses `muted` instead; a video with neither fails lint (`video_missing_muted`). |
**The visibility window is half-open: `[start, start + duration)`.** A clip shows while `start ≤ t < start + duration` and is hidden at exactly `t = start + duration`. Land an animation's resolved end state slightly **before** `data-duration`, not on it, or its last frame is never rendered. Two clips can therefore be authored back to back (`b.start === a.start + a.duration`) with no overlapping frame.
diff --git a/skills/hyperframes-core/references/tracks-and-clips.md b/skills/hyperframes-core/references/tracks-and-clips.md
index e9231d0131..d979bc8c60 100644
--- a/skills/hyperframes-core/references/tracks-and-clips.md
+++ b/skills/hyperframes-core/references/tracks-and-clips.md
@@ -8,7 +8,7 @@ A clip is any DOM element with `data-start` and, where required, `data-duration`
- **Visual `<div>` clips** — scenes, cards, overlays. Always require `data-duration`.
- **Sub-composition hosts** — `<div>` with `data-composition-src`. Always require `data-duration`.
-- **Video clips** — `<video>` with `muted` and `playsinline`. Duration can default to media length.
+- **Video clips** — `<video playsinline>`. With sound: `data-has-audio="true"` (the sound stays on the clip). Silent: `muted`. Duration can default to media length.
- **Audio clips** — `<audio>`. Duration can default to media length.
- **Image clips** — `<img>`. `data-duration` is optional and defaults to 3 seconds; write it only for another length.
@@ -46,9 +46,7 @@ multiple clip elements. Each copy selects its source range with
timeline with `data-start`. Change the source offsets and placement order; do
not try to keyframe source cutting.
-Separately authored audio gives each audio copy the identical source range and
-timing as its matching video clip (`data-media-start`, `data-duration`, and
-`data-start`). Video stays muted; the separate audio elements carry sound.
+Each video segment keeps its sound: the sound stays on the clip (`data-has-audio="true"`), so cutting the video cuts its sound. A separate `<audio>` is for other sound (music, voiceover, replacement audio, J/L cuts).
## Relative Timing
diff --git a/skills/hyperframes-core/references/variables-and-media.md b/skills/hyperframes-core/references/variables-and-media.md
index 36952bb494..32c790c133 100644
--- a/skills/hyperframes-core/references/variables-and-media.md
+++ b/skills/hyperframes-core/references/variables-and-media.md
@@ -66,27 +66,29 @@ document.getElementById("title").textContent = title;
The one real constraint is about **timelines, not media placement**: a sub-composition timeline **cannot reach or animate host elements** — neither `document.querySelector("#host-id")` nor a gsap selector string (`tl.to("#host-id", …)`) resolves across the boundary; a sub-comp timeline only drives its own subtree. So if a media element lives at the host root, **its per-scene motion (scale/opacity/morph/tilt/breathing) must be authored on the MAIN timeline in `index.html`, at GLOBAL time** (scene-local time + the scene slot's `data-start`). Keeping the media inside the scene sub-comp instead lets that sub-comp's own timeline animate it with scene-local time. For 3D tilt without a perspective parent, use gsap `transformPerspective` on the element. See `composition-patterns.md` archetype B.
-Video elements must be muted and inline. Audio must be a separate `<audio>` element, even when it uses the same source file.
+A video with sound keeps it on the `<video>` (`data-has-audio="true"`, no `muted`). Use a separate `<audio>` for music, voiceover, replacement audio, J/L cuts, or audio detached in Studio. Silent footage and b-roll: `muted`.
```html
<video
id="a-roll"
class="clip"
src="assets/demo.mp4"
+ playsinline
+ data-has-audio="true"
data-start="0"
data-duration="12"
data-track-index="0"
- muted
- playsinline
+ data-volume="1"
></video>
+<!-- Separate <audio> only for other sound: music, voiceover, replacement audio. -->
<audio
- id="a-roll-audio"
- src="assets/demo.mp4"
+ id="music"
+ src="assets/bed.mp3"
data-start="0"
data-duration="12"
- data-track-index="10"
- data-volume="1"
+ data-track-index="2"
+ data-volume="0.4"
></audio>
```
@@ -99,7 +101,7 @@ Video elements must be muted and inline. Audio must be a separate `<audio>` elem
- **Sub-compositions are exempt and work.** A `<video>`/`<audio>` inside a sub-composition renders identically to one at the host root, because a composition host propagates its offset. Only plain timed wrappers (a `<section data-start>` around a `<video data-start>`) break.
- **Never** add `crossorigin` to `<video>`/`<audio>`. `lint` rejects it unconditionally (`media_crossorigin_breaks_preview`, error) because a media host without `Access-Control-Allow-Origin` then fails silently in preview while renders still work, hiding the bug. There is no suppression, so this holds even for the canvas/WebGL/WebAudio readback case.
- **Every `<audio>` needs an `id`.** The mixer selects `audio[id][src]`, so an id-less `<audio>` is never mixed and the render is **silent**. `lint` catches it as `media_missing_id`.
-- Audio always lives on a separate `<audio>` element — even if its source file is the same as a `<video>`. The `<video>` is muted; the `<audio>` carries sound.
+- A video's own sound stays on the `<video>` (`data-has-audio="true"`, no `muted`). Add a separate `<audio>` only for other sound (music, voiceover, replacement audio, J/L cuts) and mute the video it replaces.
- For volume fades and ducking, use the `data-automation` volume lane; the exact form is in `creator-editing-recipes.md`. `data-volume` is the static baseline. A timeline `volume` tween is ignored when a lane is present.
For media duration: `<video>` and `<audio>` can omit `data-duration` if the media's intrinsic length is known and you want the full clip. Otherwise provide `data-duration` explicitly.
diff --git a/skills/hyperframes-creative/references/composition-patterns.md b/skills/hyperframes-creative/references/composition-patterns.md
index e2ddb391c0..ef999f91ee 100644
--- a/skills/hyperframes-creative/references/composition-patterns.md
+++ b/skills/hyperframes-creative/references/composition-patterns.md
@@ -23,7 +23,7 @@ Animate a wrapper div for position/size. The video fills the wrapper. The wrappe
data-duration="60"
data-track-index="0"
src="talking-head.mp4"
- muted
+ data-has-audio="true"
playsinline
></video>
</div>
@@ -53,7 +53,7 @@ Three layers, plus one critical rule:
data-media-start="0"
data-track-index="0"
src="presenter.mp4"
- muted
+ data-has-audio="true"
playsinline
></video>
diff --git a/skills/music-to-video/references/montage.md b/skills/music-to-video/references/montage.md
index 46bc15c714..4d1db85e44 100644
--- a/skills/music-to-video/references/montage.md
+++ b/skills/music-to-video/references/montage.md
@@ -15,7 +15,7 @@ realizes it inside the frame file (HOW). **Obey the frame's `pacing`.**
The asset-driven analogue of a per-onset typographic group: cut to a new clip on each anchor
(the frame's beats/onsets from the audiomap). Each clip is a `class="clip"` element
(`<img>` for a photo, **muted** `<video>` for a motion clip) placed at its anchor with
-`data-start`/`data-duration`/`data-track-index` per the core clip contract. Between clips,
+`data-start`/`data-duration`/`data-track-index` per the core clip contract. (Muted on purpose: the music track drives the sound.) Between clips,
crossfade the outgoing content to `opacity:0` ending **at** the next anchor.
Cut on the **strong** anchors; land a hero clip on a `key_moment`/downbeat.
diff --git a/skills/remotion-to-hyperframes/references/api-map.md b/skills/remotion-to-hyperframes/references/api-map.md
index c6639889d9..3eba671a11 100644
--- a/skills/remotion-to-hyperframes/references/api-map.md
+++ b/skills/remotion-to-hyperframes/references/api-map.md
@@ -53,16 +53,16 @@ See [timing.md](timing.md) — this is the highest-leverage section.
See [media.md](media.md) for trim, volume ramps, and decoder notes.
-| Remotion | HyperFrames |
-| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `<Audio src volume>` | `<audio data-start data-duration data-track-index data-volume src>` |
-| `<Audio playbackRate startFrom endAt>` | `data-playback-rate`, `data-trim-start`, `data-trim-end` |
-| `<Video src>` | `<video muted playsinline data-start data-duration data-track-index src>` |
-| `<OffthreadVideo>` | `<video>` — HF doesn't need the off-thread variant (uses headless Chrome) |
-| `<Img src>` | `<img>` |
-| `<IFrame src>` | `<iframe>` — HF auto-falls back to screenshot mode for nested iframes |
-| `staticFile("x.png")` | `"assets/x.png"` in the root, `"../assets/x.png"` inside a `compositions/` scene file — copy the file into `hf-src/assets/` next to `index.html` |
-| `delayRender()` / `continueRender()` | drop — HF waits on asset readiness via the Frame Adapter pattern |
+| Remotion | HyperFrames |
+| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `<Audio src volume>` | `<audio data-start data-duration data-track-index data-volume src>` |
+| `<Audio playbackRate startFrom endAt>` | `data-playback-rate`, `data-trim-start`, `data-trim-end` |
+| `<Video src>` | `<video playsinline data-has-audio="true" data-start data-duration data-track-index src>` only when the file has an audio stream (check with `ffprobe -v error -select_streams a -show_entries stream=codec_type -of csv=p=0 <file>`: non-empty means it has audio) and the source is not `muted` or `volume={0}`; otherwise `muted` instead |
+| `<OffthreadVideo>` | `<video>` — HF doesn't need the off-thread variant (uses headless Chrome) |
+| `<Img src>` | `<img>` |
+| `<IFrame src>` | `<iframe>` — HF auto-falls back to screenshot mode for nested iframes |
+| `staticFile("x.png")` | `"assets/x.png"` in the root, `"../assets/x.png"` inside a `compositions/` scene file — copy the file into `hf-src/assets/` next to `index.html` |
+| `delayRender()` / `continueRender()` | drop — HF waits on asset readiness via the Frame Adapter pattern |
## Transitions
diff --git a/skills/remotion-to-hyperframes/references/media.md b/skills/remotion-to-hyperframes/references/media.md
index f44f00acd0..50d013f550 100644
--- a/skills/remotion-to-hyperframes/references/media.md
+++ b/skills/remotion-to-hyperframes/references/media.md
@@ -70,14 +70,25 @@ is the static baseline.
## `<Video>` and `<OffthreadVideo>`
```tsx
-<Video src={staticFile("intro.mp4")} muted playsInline />
-<OffthreadVideo src={staticFile("intro.mp4")} muted />
+<Video src={staticFile("intro.mp4")} playsInline />
+<OffthreadVideo src={staticFile("intro.mp4")} />
```
```html
+<!-- intro.mp4 has an audio stream -->
+<video
+ playsinline
+ data-has-audio="true"
+ data-start="0"
+ data-duration="5"
+ data-track-index="0"
+ src="assets/intro.mp4"
+></video>
+
+<!-- intro.mp4 is silent (or the Remotion element is muted) -->
<video
- muted
playsinline
+ muted
data-start="0"
data-duration="5"
data-track-index="0"
@@ -89,8 +100,14 @@ is the static baseline.
rendering. HF runs in headless Chrome already, so the off-thread variant
collapses to a regular `<video>`.
-`muted` and `playsinline` are required for the runtime to autoplay
-(browser policy). Always emit them.
+`playsinline` is required for the runtime to autoplay (browser policy); always
+emit it. Check the source file for an audio stream first:
+`ffprobe -v error -select_streams a -show_entries stream=codec_type -of csv=p=0 <file>`
+(non-empty output means it has audio). Remotion `<Video>` maps to
+`<video playsinline>` + `data-has-audio="true"` only when the file has an audio
+stream and the Remotion source does not set `muted` or `volume={0}`; otherwise
+emit `muted`. Silent b-roll with `data-has-audio="true"` aborts the render
+(`ASSET_MEDIA_TYPE_MISMATCH`).
## `<Img>`
diff --git a/skills/talking-head-recut/SKILL.md b/skills/talking-head-recut/SKILL.md
index 7784122d9a..fe87c925a6 100644
--- a/skills/talking-head-recut/SKILL.md
+++ b/skills/talking-head-recut/SKILL.md
@@ -939,22 +939,13 @@ ffmpeg -y -i "$VIDEO_PATH" -c:v libx264 -crf 18 -g 30 -keyint_min 30 \
<video
id="bg-video"
src="input-video.mp4"
- muted
playsinline
+ data-has-audio="true"
data-start="0"
data-duration="121.2"
data-track-index="1"
></video>
</div>
- <!-- Preserve the source program audio while the visual video stays muted. -->
- <audio
- id="source-audio"
- src="input-video.mp4"
- data-start="0"
- data-duration="121.2"
- data-track-index="10"
- data-volume="1"
- ></audio>
<!-- Layer 2: each card-host sits at the bounds dictated by its layout. -->
<!-- IMPORTANT: every card-host MUST carry BOTH "card-host" and "clip" classes. -->
@@ -1182,11 +1173,8 @@ PRODUCER_BROWSER_GPU_MODE=hardware npx hyperframes render public \
```
`hyperframes render <dir>` reads `<dir>/index.html` and produces the MP4.
-The canonical composition keeps the visual `<video>` muted and mounts the same
-source as the root `#source-audio` track, so the rendered MP4 preserves the
-talking-head audio without a manual remux. This uses a separate audio track
-rather than `data-has-audio="true"` so its volume and ducking remain independently
-controllable on the timeline.
+The source program audio stays on `#bg-video`, so the rendered MP4 preserves the
+talking-head audio without a manual remux.
The flag `PRODUCER_BROWSER_GPU_MODE=hardware` (or `--browser-gpu`) is
strongly recommended on macOS — software-only Chrome rendering times out
on most laptops.
diff --git a/skills/talking-head-recut/media-contract.test.mjs b/skills/talking-head-recut/media-contract.test.mjs
index 55d6ded371..2ce726ca29 100644
--- a/skills/talking-head-recut/media-contract.test.mjs
+++ b/skills/talking-head-recut/media-contract.test.mjs
@@ -16,20 +16,9 @@ function readAttribute(tag, attribute) {
return match[1];
}
-test("canonical composition preserves source audio as a root media track", () => {
+test("canonical composition keeps the source program audio on the video", () => {
const video = findTag("video", "bg-video");
- const audio = findTag("audio", "source-audio");
-
- assert.match(video, /\bmuted\b/);
- assert.match(
- skill,
- /<\/div>\s*<!-- Preserve the source program audio[\s\S]*?<audio\b[^>]*\bid="source-audio"[^>]*>[\s\S]*?<\/audio>/,
- );
- assert.equal(readAttribute(audio, "src"), readAttribute(video, "src"));
- assert.equal(readAttribute(audio, "data-start"), readAttribute(video, "data-start"));
- assert.equal(readAttribute(audio, "data-duration"), readAttribute(video, "data-duration"));
- assert.notEqual(
- readAttribute(audio, "data-track-index"),
- readAttribute(video, "data-track-index"),
- );
+ assert.doesNotMatch(video, /\bmuted\b/);
+ assert.equal(readAttribute(video, "data-has-audio"), "true");
+ assert.doesNotMatch(skill, /<audio\b[^>]*\bid="source-audio"/);
});
diff --git a/skills/talking-head-recut/references/frames/hairline.html b/skills/talking-head-recut/references/frames/hairline.html
index db98a1142c..ee831d2d6f 100644
--- a/skills/talking-head-recut/references/frames/hairline.html
+++ b/skills/talking-head-recut/references/frames/hairline.html
@@ -9,8 +9,8 @@
How to apply in the composition
Add ONE decorative div as a sibling of #video-wrap (not a child — the
- video element is muted/playsinline and the decorative div should sit
- above it). The accent color is configurable per scene.
+ video element keeps its sound (data-has-audio="true") and plays inline,
+ and the decorative div should sit above it). The accent color is configurable per scene.
<div class="video-wrapper" id="video-wrap">
<video src="input-video.mp4" ...></video>