Skip to content

Write HTML. Render video. Built for agents.

feat(cli): judge text behind a transparent video by its decoded alpha in check (#5657)

WaterrrForevercea8aa8a7dee
diff --git a/docs/guides/remove-background.mdx b/docs/guides/remove-background.mdx
index deb8d91236..f763737cac 100644
--- a/docs/guides/remove-background.mdx
+++ b/docs/guides/remove-background.mdx
@@ -84,6 +84,11 @@ The plate keeps the original surroundings and leaves a transparent
person-shaped hole. It is not an inpainted empty background. Place an opaque
graphic or scene beneath the hole, then place `subject.webm` above it.
+`hyperframes check` judges text behind the cutout by the cutout's own alpha at
+each sample time. A word partly behind the subject is reported as info with the
+share it hides; a word 80% hidden, or two neighbouring words each half hidden,
+is an error. You do not need `data-layout-allow-occlusion` for this layering.
+
## Know when it will work well
The included model is designed for people. It works best with a clear human
diff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdx
index 42026315d1..dd97ff7a41 100644
--- a/docs/packages/cli.mdx
+++ b/docs/packages/cli.mdx
@@ -857,6 +857,21 @@ cover every finding from the full check.
configured seek samples falls inside `--range`; a requested gate never silently
passes without running.
+Text under a `<video>` is judged by the frame the video shows at that sample.
+An opaque video covers its whole box, as an opaque image does. A video with
+transparency, such as a `remove-background` cutout, covers only where its
+alpha is: a word partly behind the subject is an info finding whose
+`coveredFraction` is the most covered word's share, and a word 80% hidden or two
+neighbouring words each half hidden are an error. A CSS background on the video
+still covers its whole box, and a mirrored video (`scaleX(-1)`) is read on the
+side it is painted. When a video's frame cannot be read, the audit treats the
+video as opaque and adds a `video_alpha_unread` info finding saying why: no frame
+decoded yet, the seek still landing, a cross-origin source, no frame size, a
+rotated, skewed or 3D-transformed video, or a frame that reads back fully
+transparent. A fully transparent frame is treated as unread on purpose, since a
+failed draw looks the same; mark text that sits under a cutout before its subject
+enters with `data-layout-allow-occlusion`.
+
Contrast failures are errors, and each one reports the sampled foreground and
background colours, the measured ratio against the required one, and a
compliant colour you could use instead.
diff --git a/docs/prompting/rules-and-anti-patterns.mdx b/docs/prompting/rules-and-anti-patterns.mdx
index 6e5d69e32c..1d4a7a26f8 100644
--- a/docs/prompting/rules-and-anti-patterns.mdx
+++ b/docs/prompting/rules-and-anti-patterns.mdx
@@ -64,7 +64,10 @@ Two more ways an SVG "line draws itself" effect (animated `strokeDasharray` / `s
`hyperframes check` flags a text block covered by another element as `text_occluded`, and
two attributes waive it: `data-layout-allow-overlap` for intentional layering, and
`data-layout-allow-occlusion` for something deliberately painted over type. Both are
-legitimate — a caption designed to sit behind a matted subject needs one.
+legitimate. A caption behind a transparent `<video>` cutout needs neither: `check` reads the
+cutout's decoded alpha at each sample, reports how much of each word the subject covers as
+info, and fails only a word that is 80% hidden or two neighbouring words that are each half
+hidden.
Two things about them are worth knowing before you reach for either:
diff --git a/packages/cli/src/commands/layout-audit.browser.js b/packages/cli/src/commands/layout-audit.browser.js
index bb8d583a64..9ba3269eb3 100644
--- a/packages/cli/src/commands/layout-audit.browser.js
+++ b/packages/cli/src/commands/layout-audit.browser.js
@@ -1026,14 +1026,46 @@
return { x: objectPositionOffset(x, freeX), y: objectPositionOffset(y, freeY) };
}
- // Return the alpha painted by an <img> at a viewport point. `null` means the
- // browser would not let us inspect the image (not loaded or cross-origin), in
- // which case callers preserve the conservative opaque fallback.
- function imageAlphaAt(element, x, y) {
- const sourceWidth = element.naturalWidth;
- const sourceHeight = element.naturalHeight;
+ const IDENTITY_FLIP = { flipX: false, flipY: false };
+
+ // 2D CSS matrix components [a, b, c, d], or null when the transform is 3D or unparsed.
+ function transformMatrix(value) {
+ if (!value || value === "none") return [1, 0, 0, 1];
+ const match = /^matrix\(([^)]*)\)$/.exec(value.trim());
+ if (!match) return null;
+ const parts = match[1].split(",").map(Number);
+ return parts.length === 6 && parts.every(Number.isFinite) ? parts.slice(0, 4) : null;
+ }
+
+ // Which axes the element's paint is mirrored on, or null when a rotation, skew or 3D transform
+ // (on it or an ancestor) means its bounding box no longer maps axis-aligned onto its pixels.
+ function paintFlips(element) {
+ let flipX = false;
+ let flipY = false;
+ for (let current = element; current; current = current.parentElement) {
+ const style = getComputedStyle(current);
+ const matrix = transformMatrix(style.transform);
+ if (!matrix) return null;
+ const [a, b, c, d] = matrix;
+ if (Math.abs(b) > 1e-6 || Math.abs(c) > 1e-6) return null;
+ if (style.rotate && style.rotate !== "none" && parseFloat(style.rotate) !== 0) return null;
+ const scale = String(style.scale || "none")
+ .trim()
+ .split(/\s+/)
+ .map(Number);
+ const scaleX = Number.isFinite(scale[0]) ? scale[0] : 1;
+ const scaleY = Number.isFinite(scale[1]) ? scale[1] : scaleX;
+ if (a * scaleX < 0) flipX = !flipX;
+ if (d * scaleY < 0) flipY = !flipY;
+ }
+ return flipX || flipY ? { flipX, flipY } : IDENTITY_FLIP;
+ }
+
+ function paintedSourceMapping(element, sourceWidth, sourceHeight) {
const rect = element.getBoundingClientRect();
if (!sourceWidth || !sourceHeight || !rect.width || !rect.height) return null;
+ const flips = paintFlips(element);
+ if (!flips) return null;
const style = getComputedStyle(element);
const fit = style.objectFit || "fill";
@@ -1061,9 +1093,48 @@
rect.width - paintedWidth,
rect.height - paintedHeight,
);
- const localX = x - rect.left - offsets.x;
- const localY = y - rect.top - offsets.y;
- if (localX < 0 || localY < 0 || localX >= paintedWidth || localY >= paintedHeight) return 0;
+ return {
+ rect: { left: rect.left, top: rect.top, right: rect.right, bottom: rect.bottom },
+ offsets,
+ ...flips,
+ paintedWidth,
+ paintedHeight,
+ scaleX,
+ scaleY,
+ sourceWidth,
+ sourceHeight,
+ };
+ }
+
+ // The source pixel painted at a viewport point, or null outside the painted content box.
+ function sourcePixelAt(mapping, x, y) {
+ const { rect, offsets } = mapping;
+ const localX = (mapping.flipX ? rect.right - x : x - rect.left) - offsets.x;
+ const localY = (mapping.flipY ? rect.bottom - y : y - rect.top) - offsets.y;
+ if (
+ localX < 0 ||
+ localY < 0 ||
+ localX >= mapping.paintedWidth ||
+ localY >= mapping.paintedHeight
+ ) {
+ return null;
+ }
+ return {
+ x: Math.min(mapping.sourceWidth - 1, Math.max(0, Math.floor(localX / mapping.scaleX))),
+ y: Math.min(mapping.sourceHeight - 1, Math.max(0, Math.floor(localY / mapping.scaleY))),
+ };
+ }
+
+ // Return the alpha painted by an <img> at a viewport point. `null` means the
+ // browser would not let us inspect the image (not loaded or cross-origin), in
+ // which case callers preserve the conservative opaque fallback.
+ function imageAlphaAt(element, x, y) {
+ const sourceWidth = element.naturalWidth;
+ const sourceHeight = element.naturalHeight;
+ const mapping = paintedSourceMapping(element, sourceWidth, sourceHeight);
+ if (!mapping) return null;
+ const pixel = sourcePixelAt(mapping, x, y);
+ if (!pixel) return 0;
try {
let cached = imageAlphaCanvases.get(element);
@@ -1083,25 +1154,129 @@
cached = { context, width: sourceWidth, height: sourceHeight, source };
imageAlphaCanvases.set(element, cached);
}
- const sourceX = Math.min(sourceWidth - 1, Math.max(0, Math.floor(localX / scaleX)));
- const sourceY = Math.min(sourceHeight - 1, Math.max(0, Math.floor(localY / scaleY)));
- return cached.context.getImageData(sourceX, sourceY, 1, 1).data[3] / 255;
+ return cached.context.getImageData(pixel.x, pixel.y, 1, 1).data[3] / 255;
} catch {
return null;
}
}
+ const OCCLUDING_ALPHA = 0.6;
+ const VIDEO_ALPHA_MAX_SIDE = 480;
+ let videoAlphaFrames = new Map();
+ let videoAlphaScratch = null;
+
+ function videoAlphaContext(width, height) {
+ if (!videoAlphaScratch) {
+ const canvas = document.createElement("canvas");
+ const context = canvas.getContext("2d", { willReadFrequently: true });
+ if (!context) return null;
+ videoAlphaScratch = { canvas, context };
+ }
+ videoAlphaScratch.canvas.width = width;
+ videoAlphaScratch.canvas.height = height;
+ return videoAlphaScratch.context;
+ }
+
+ // Read once per audit call, so per sample time, and only for a video some text probe lands under.
+ function videoAlphaFrame(video) {
+ let frame = videoAlphaFrames.get(video);
+ if (!frame) {
+ frame = readVideoAlphaFrame(video);
+ videoAlphaFrames.set(video, frame);
+ }
+ return frame;
+ }
+
+ function unreadVideoFrame(reason) {
+ return { kind: "unread", reason };
+ }
+
+ // "opaque" keeps a video's whole box occluding; "matte" judges each point by the decoded
+ // frame's alpha; "unread" falls back to opaque and is reported, never guessed.
+ function readVideoAlphaFrame(video) {
+ if (video.seeking) return unreadVideoFrame("the seek had not landed");
+ if (video.readyState < 2) return unreadVideoFrame("no frame was decoded at this time");
+ if (!video.videoWidth || !video.videoHeight)
+ return unreadVideoFrame("the video has no frame size");
+ const mapping = paintedSourceMapping(video, video.videoWidth, video.videoHeight);
+ if (!mapping) return unreadVideoFrame("it is rotated, skewed or transformed in 3D");
+ const downscale = Math.min(
+ 1,
+ VIDEO_ALPHA_MAX_SIDE / Math.max(mapping.sourceWidth, mapping.sourceHeight),
+ );
+ const width = Math.max(1, Math.round(mapping.sourceWidth * downscale));
+ const height = Math.max(1, Math.round(mapping.sourceHeight * downscale));
+ let rgba;
+ try {
+ const context = videoAlphaContext(width, height);
+ if (!context) return unreadVideoFrame("no 2D canvas was available");
+ context.clearRect(0, 0, width, height);
+ context.drawImage(video, 0, 0, width, height);
+ rgba = context.getImageData(0, 0, width, height).data;
+ } catch (error) {
+ return unreadVideoFrame(
+ error && error.name === "SecurityError"
+ ? "its source is cross-origin"
+ : "its frame could not be drawn",
+ );
+ }
+ return classifyVideoAlpha(rgba, width, height, mapping, opacityChain(video));
+ }
+
+ function classifyVideoAlpha(rgba, width, height, mapping, opacity) {
+ const alpha = new Uint8Array(width * height);
+ let clear = 0;
+ let painted = 0;
+ for (let index = 0; index < alpha.length; index++) {
+ alpha[index] = rgba[index * 4 + 3];
+ if (alpha[index] < OCCLUDING_ALPHA * 255) clear += 1;
+ if (alpha[index] > 0) painted += 1;
+ }
+ if (clear === 0) return { kind: "opaque" };
+ if (painted === 0) return unreadVideoFrame("its frame read back fully transparent");
+ return {
+ kind: "matte",
+ alpha,
+ width,
+ height,
+ opacity,
+ mapping: {
+ ...mapping,
+ sourceWidth: width,
+ sourceHeight: height,
+ scaleX: mapping.paintedWidth / width,
+ scaleY: mapping.paintedHeight / height,
+ },
+ };
+ }
+
+ function matteOccludesAt(frame, x, y) {
+ const pixel = sourcePixelAt(frame.mapping, x, y);
+ if (!pixel) return false;
+ return (frame.alpha[pixel.y * frame.width + pixel.x] / 255) * frame.opacity >= OCCLUDING_ALPHA;
+ }
+
+ // A video with an opaque CSS background paints that background behind its clear pixels.
+ function matteFrameFor(element) {
+ if (element.tagName !== "VIDEO") return null;
+ if (hasOpaqueBackground(getComputedStyle(element))) return null;
+ const frame = videoAlphaFrame(element);
+ return frame.kind === "matte" ? frame : null;
+ }
+
// An element hides text beneath it when it paints opaque pixels at near-full
// opacity: raster content (img/video/canvas), a background image, or a solid
// background colour. Low-opacity overlays (grain, scrims) do not occlude.
function isOpaqueOccluder(element, x, y) {
const opacity = opacityChain(element);
- if (opacity < 0.6) return false;
+ if (opacity < OCCLUDING_ALPHA) return false;
if (IGNORE_TAGS.has(element.tagName)) return false;
if (element.tagName === "IMG") {
const alpha = imageAlphaAt(element, x, y);
- if (alpha !== null) return alpha * opacity >= 0.6;
+ if (alpha !== null) return alpha * opacity >= OCCLUDING_ALPHA;
}
+ const matte = matteFrameFor(element);
+ if (matte) return matteOccludesAt(matte, x, y);
if (RASTER_TAGS.has(element.tagName)) return true;
return hasOpaqueBackground(getComputedStyle(element));
}
@@ -1150,7 +1325,7 @@
// there is related to the text, non-opaque, sharing a 3D context with it, or
// part of a transient crossfade overlap.
// fallow-ignore-next-line complexity
- function occluderAt(element, x, y) {
+ function occluderAt(element, x, y, seenMattes) {
// Walk the paint-ordered stack: a transparent layer on top must not mask an opaque one below it.
const stack =
typeof document.elementsFromPoint === "function"
@@ -1163,6 +1338,8 @@
// Pair-specific exemptions excuse this hit only; keep walking for deeper occluders.
if (sharedPreserve3d(element, hit)) continue;
if (isCrossSceneTransitionOverlap(element, hit)) continue;
+ const matte = seenMattes && matteFrameFor(hit);
+ if (matte && !seenMattes.has(matte)) seenMattes.set(matte, hit);
if (isOpaqueOccluder(hit, x, y)) return hit;
}
return null;
@@ -1196,21 +1373,162 @@
function occlusionCoverage(element, textRects) {
let occluder = null;
let hits = 0;
+ let opaqueHits = 0;
+ // Every matte above the text, including where its alpha is clear: a feature narrower than the
+ // probe grid (an arm, a mic stand) can hide a word between two probe columns.
+ const seenMattes = new Map();
for (const textRect of textRects) {
for (const yFraction of OCCLUSION_PROBE_Y_FRACTIONS) {
const y = textRect.top + textRect.height * yFraction;
for (const xFraction of OCCLUSION_PROBE_X_FRACTIONS) {
- const hit = occluderAt(element, textRect.left + textRect.width * xFraction, y);
+ const x = textRect.left + textRect.width * xFraction;
+ const hit = occluderAt(element, x, y, seenMattes);
if (!hit) continue;
hits += 1;
if (!occluder) occluder = hit;
+ if (!matteFrameFor(hit)) opaqueHits += 1;
}
}
}
+ const judgeByMatte = opaqueHits === 0 && seenMattes.size > 0;
return {
- occluder,
+ occluder: occluder || (judgeByMatte ? seenMattes.values().next().value : null),
coveredFraction: round(hits / (OCCLUSION_GRID_POINTS * textRects.length)),
+ mattes: judgeByMatte ? [...seenMattes.keys()] : [],
+ };
+ }
+
+ // Thresholds from measuring rendered words against real subject mattes: a word reads as
+ // deliberately tucked behind the subject at 30-55% covered, and is lost at 80%, or when it
+ // and its neighbour are both half covered.
+ const MATTE_WORD_LOST_COVERAGE = 0.8;
+ const MATTE_PAIR_LOST_COVERAGE = 0.5;
+ const MATTE_MAX_SAMPLES_PER_AXIS = 48;
+
+ function wordClientRects(element) {
+ const words = [];
+ for (const node of directTextNodes(element)) {
+ for (const match of (node.textContent || "").matchAll(/\S+/g)) {
+ const range = document.createRange();
+ range.setStart(node, match.index);
+ range.setEnd(node, match.index + match[0].length);
+ const rects = Array.from(range.getClientRects()).filter(
+ (rect) => rect.width > 0.5 && rect.height > 0.5,
+ );
+ range.detach();
+ if (rects.length > 0) words.push({ word: match[0], rects });
+ }
+ }
+ return words;
+ }
+
+ function matteSampleCount(length, mattes) {
+ const pitch = Math.max(1, Math.min(...mattes.map((matte) => matte.mapping.scaleX)));
+ return Math.max(1, Math.min(MATTE_MAX_SAMPLES_PER_AXIS, Math.ceil(length / pitch)));
+ }
+
+ // Area-weighted share of the rects under any matte's occluding alpha, sampled at the matte's resolution.
+ function matteCoverage(rects, mattes) {
+ let coveredArea = 0;
+ let totalArea = 0;
+ for (const rect of rects) {
+ const cols = matteSampleCount(rect.width, mattes);
+ const rows = matteSampleCount(rect.height, mattes);
+ let covered = 0;
+ for (let row = 0; row < rows; row++) {
+ const y = rect.top + ((row + 0.5) * rect.height) / rows;
+ for (let col = 0; col < cols; col++) {
+ const x = rect.left + ((col + 0.5) * rect.width) / cols;
+ if (mattes.some((matte) => matteOccludesAt(matte, x, y))) covered += 1;
+ }
+ }
+ const area = rect.width * rect.height;
+ coveredArea += (area * covered) / (rows * cols);
+ totalArea += area;
+ }
+ return totalArea > 0 ? round(coveredArea / totalArea) : 0;
+ }
+
+ function lostNeighbourPair(words) {
+ for (let index = 1; index < words.length; index++) {
+ const pair = [words[index - 1], words[index]];
+ if (pair.every((word) => word.coverage >= MATTE_PAIR_LOST_COVERAGE)) return pair;
+ }
+ return null;
+ }
+
+ function percent(fraction) {
+ return `${Math.round(fraction * 100)}%`;
+ }
+
+ function lostWordsMessage(lostWord, pair) {
+ if (lostWord) {
+ return `"${lostWord.word}" is ${percent(lostWord.coverage)} behind a transparent video's matte.`;
+ }
+ return `Neighbouring words "${pair[0].word}" and "${pair[1].word}" are ${percent(pair[0].coverage)} and ${percent(pair[1].coverage)} behind a transparent video's matte.`;
+ }
+
+ function markMatteWordsLost(issue, lostWord, pair) {
+ issue.severity = "error";
+ issue.message = lostWordsMessage(lostWord, pair);
+ issue.fixHint =
+ "Move or resize the text so no word is 80% behind the matte and no two neighbouring words are each half behind it, or mark intentional layering with data-layout-allow-occlusion.";
+ }
+
+ // Text behind a cutout is judged per rendered word against the decoded alpha: partial cover is
+ // reported as info with its share, a lost word is an error.
+ function matteOcclusionIssue(element, time, textRect, text, occluder, mattes) {
+ const words = wordClientRects(element).map(({ word, rects }) => ({
+ word,
+ coverage: matteCoverage(rects, mattes),
+ }));
+ if (words.length === 0) return null;
+ const peak = words.reduce((top, word) => (word.coverage > top.coverage ? word : top));
+ if (peak.coverage <= 0) return null;
+ const issue = {
+ code: "text_occluded",
+ severity: "info",
+ time,
+ selector: uniqueSelectorFor(element),
+ containerSelector: uniqueSelectorFor(occluder),
+ text,
+ message: `Text sits partly behind a transparent video's matte; its most covered word "${peak.word}" is ${percent(peak.coverage)} hidden.`,
+ rect: textRect,
+ coveredFraction: peak.coverage,
};
+ const lostWord = peak.coverage >= MATTE_WORD_LOST_COVERAGE ? peak : null;
+ const pair = lostNeighbourPair(words);
+ if (lostWord || pair) markMatteWordsLost(issue, lostWord, pair);
+ return { issue, first: words[0], last: words[words.length - 1] };
+ }
+
+ // Captions often render one word per element, so the neighbour rule also spans sibling elements.
+ function markLostSiblingWords(matteResults) {
+ for (const [element, result] of matteResults) {
+ const next = element.nextElementSibling && matteResults.get(element.nextElementSibling);
+ if (!next) continue;
+ const pair = [result.last, next.first];
+ if (!pair.every((word) => word.coverage >= MATTE_PAIR_LOST_COVERAGE)) continue;
+ for (const { issue } of [result, next]) {
+ if (issue.severity !== "error") markMatteWordsLost(issue, null, pair);
+ }
+ }
+ }
+
+ function unreadVideoIssues(time) {
+ const issues = [];
+ for (const [video, frame] of videoAlphaFrames) {
+ if (frame.kind !== "unread") continue;
+ issues.push({
+ code: "video_alpha_unread",
+ severity: "info",
+ time,
+ selector: uniqueSelectorFor(video),
+ message: `Could not read this video's frame alpha (${frame.reason}), so text under it was judged as if the video were opaque.`,
+ rect: toRect(video.getBoundingClientRect()),
+ });
+ }
+ return issues;
}
// pointer-events:none hides elements from elementFromPoint — both probed text AND occluders.
@@ -1240,18 +1558,23 @@
}
// text_occluded: atomic labels flag at any hit; prose needs coveredFraction >= proseCoverageFloor (default 0.15).
- function occludedTextIssue(element, time, proseCoverageFloor) {
+ function occludedTextIssue(element, time, proseCoverageFloor, matteResults) {
if (hasAllowOcclusionFlag(element)) return null;
if (!hasVisibleTextInk(element)) return null;
const textRect = textRectFor(element, true);
if (!textRect) return null;
const textRects = textClientRects(element, true);
const text = textContentFor(element, true);
- const { occluder, coveredFraction } = occlusionCoverage(
+ const { occluder, coveredFraction, mattes } = occlusionCoverage(
element,
textRects.length > 0 ? textRects : [textRect],
);
if (!occluder) return null;
+ if (mattes.length > 0) {
+ const result = matteOcclusionIssue(element, time, textRect, text, occluder, mattes);
+ if (result) matteResults.set(element, result);
+ return result ? result.issue : null;
+ }
if (!isAtomicLabel(text) && coveredFraction < proseCoverageFloor) return null;
return {
code: "text_occluded",
@@ -2015,6 +2338,8 @@
isVisibleElement(element, 0.05),
);
const issues = [];
+ const matteResults = new Map();
+ videoAlphaFrames = new Map();
const restoreHits = restoreHitTesting(root);
try {
@@ -2023,7 +2348,7 @@
const clipped = clippedTextIssue(element, time, tolerance);
if (clipped) issues.push(clipped);
issues.push(...textOverflowIssues(element, root, rootRect, time, tolerance, clipped));
- const occluded = occludedTextIssue(element, time, proseCoverageFloor);
+ const occluded = occludedTextIssue(element, time, proseCoverageFloor, matteResults);
if (occluded) issues.push(occluded);
const invisible = invisibleTextIssue(element, time);
if (invisible) issues.push(invisible);
@@ -2031,6 +2356,8 @@
} finally {
restoreHits();
}
+ markLostSiblingWords(matteResults);
+ issues.push(...unreadVideoIssues(time));
issues.push(...containerOverflowIssues(root, time, tolerance));
issues.push(...contentOverlapIssues(root, time));
diff --git a/packages/cli/src/commands/layout-audit.video-alpha.browser.test.ts b/packages/cli/src/commands/layout-audit.video-alpha.browser.test.ts
new file mode 100644
index 0000000000..16a601b76f
--- /dev/null
+++ b/packages/cli/src/commands/layout-audit.video-alpha.browser.test.ts
@@ -0,0 +1,427 @@
+// @vitest-environment happy-dom
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const script = readFileSync(
+ join(dirname(fileURLToPath(import.meta.url)), "layout-audit.browser.js"),
+ "utf-8",
+);
+
+interface AuditIssue {
+ code: string;
+ severity: string;
+ selector: string;
+ containerSelector?: string;
+ message: string;
+ coveredFraction?: number;
+}
+
+interface Box {
+ left: number;
+ top: number;
+ width: number;
+ height: number;
+}
+
+// The <video> fills a 1920x1080 frame from a 192x108 source, so one source pixel is 10x10 px.
+const SOURCE_WIDTH = 192;
+const SOURCE_HEIGHT = 108;
+const FRAME: Box = { left: 0, top: 0, width: 1920, height: 1080 };
+const WORD_TOP = 500;
+const WORD_HEIGHT = 80;
+
+interface VideoScene {
+ /** Text elements: id -> words with their left edge and width, all on one line. */
+ text: Record<string, Array<{ word: string; left: number; width: number }>>;
+ /** Alpha (0-255) the decoded frame paints at a composition point. */
+ alphaAt?: (x: number, y: number) => number;
+ video?: { readyState?: number; seeking?: boolean; readError?: Error; overlapsText?: boolean };
+ /** Computed style the <video> adds to the base, e.g. a transform or a background. */
+ videoStyle?: Record<string, string>;
+ /** Report each element's text as one line box, as Chromium does, rather than one per word. */
+ wholeLine?: boolean;
+ textAttrs?: string;
+}
+
+function domRect({ left, top, width, height }: Box): DOMRect {
+ return {
+ left,
+ top,
+ width,
+ height,
+ right: left + width,
+ bottom: top + height,
+ x: left,
+ y: top,
+ toJSON() {
+ return this;
+ },
+ } as DOMRect;
+}
+
+const BASE_STYLE: Record<string, string> = {
+ display: "block",
+ visibility: "visible",
+ opacity: "1",
+ overflow: "visible",
+ overflowX: "visible",
+ overflowY: "visible",
+ backgroundColor: "rgba(0, 0, 0, 0)",
+ backgroundImage: "none",
+ objectFit: "fill",
+ objectPosition: "50% 50%",
+ fontSize: "36px",
+};
+
+function wordBox(left: number, width: number): Box {
+ return { left, top: WORD_TOP, width, height: WORD_HEIGHT };
+}
+
+function installVideoScene(scene: VideoScene): { getImageData: ReturnType<typeof vi.fn> } {
+ const spans = Object.entries(scene.text)
+ .map(
+ ([id, words]) =>
+ `<span id="${id}" ${scene.textAttrs ?? ""}>${words.map(({ word }) => word).join(" ")}</span>`,
+ )
+ .join("");
+ document.body.innerHTML = `
+ <div id="root" data-composition-id="main" data-width="1920" data-height="1080">
+ <div id="line">${spans}</div>
+ <video id="clip"></video>
+ </div>`;
+
+ const video = document.getElementById("clip") as HTMLVideoElement;
+ Object.defineProperties(video, {
+ videoWidth: { configurable: true, value: SOURCE_WIDTH },
+ videoHeight: { configurable: true, value: SOURCE_HEIGHT },
+ readyState: { configurable: true, value: scene.video?.readyState ?? 4 },
+ seeking: { configurable: true, value: scene.video?.seeking ?? false },
+ });
+
+ vi.spyOn(window, "getComputedStyle").mockImplementation(
+ (element) =>
+ ({
+ ...BASE_STYLE,
+ ...(element === video ? scene.videoStyle : {}),
+ }) as unknown as CSSStyleDeclaration,
+ );
+ for (const element of Array.from(document.querySelectorAll("*"))) {
+ vi.spyOn(element, "getBoundingClientRect").mockReturnValue(domRect(FRAME));
+ }
+
+ const wordRect = (node: Node | null, start: number, end: number): DOMRect[] => {
+ const owner = node?.parentElement?.id ?? "";
+ const word = (node?.textContent ?? "").slice(start, end);
+ const entry = scene.text[owner]?.find((candidate) => candidate.word === word);
+ return entry ? [domRect(wordBox(entry.left, entry.width))] : [];
+ };
+ vi.spyOn(document, "createRange").mockImplementation(() => {
+ let node: Node | null = null;
+ let start = 0;
+ let end: number | null = null;
+ return {
+ selectNodeContents(selected: Node) {
+ node = selected;
+ },
+ setStart(selected: Node, offset: number) {
+ node = selected;
+ start = offset;
+ },
+ setEnd(_selected: Node, offset: number) {
+ end = offset;
+ },
+ getClientRects() {
+ const owner = node?.parentElement?.id ?? "";
+ if (end === null) {
+ const boxes = (scene.text[owner] ?? []).map(({ left, width }) => wordBox(left, width));
+ if (scene.wholeLine && boxes.length > 0) {
+ const left = Math.min(...boxes.map((box) => box.left));
+ const right = Math.max(...boxes.map((box) => box.left + box.width));
+ return [domRect(wordBox(left, right - left))] as unknown as DOMRectList;
+ }
+ return boxes.map(domRect) as unknown as DOMRectList;
+ }
+ return wordRect(node, start, end) as unknown as DOMRectList;
+ },
+ detach() {},
+ } as unknown as Range;
+ });
+
+ const overlapsText = scene.video?.overlapsText ?? true;
+ Object.defineProperty(document, "elementsFromPoint", {
+ configurable: true,
+ value: (x: number) => {
+ const textElement = Object.keys(scene.text)
+ .map((id) => document.getElementById(id))
+ .find((element) => {
+ const words = scene.text[element?.id ?? ""] ?? [];
+ return words.some(({ left, width }) => x >= left && x < left + width);
+ });
+ return [...(overlapsText ? [video] : []), textElement, video].filter(Boolean);
+ },
+ });
+
+ const alphaAt = scene.alphaAt ?? (() => 255);
+ const getImageData = vi.fn((_x: number, _y: number, width: number, height: number) => {
+ if (scene.video?.readError) throw scene.video.readError;
+ const data = new Uint8ClampedArray(width * height * 4);
+ for (let row = 0; row < height; row++) {
+ for (let col = 0; col < width; col++) {
+ const x = ((col + 0.5) * FRAME.width) / width;
+ const y = ((row + 0.5) * FRAME.height) / height;
+ data[(row * width + col) * 4 + 3] = alphaAt(x, y);
+ }
+ }
+ return { data };
+ });
+ // getContext's overloads resolve to the WebGPU one; the audit only asks for "2d".
+ const getContextSpy = vi.spyOn(HTMLCanvasElement.prototype, "getContext") as unknown as {
+ mockReturnValue(value: CanvasRenderingContext2D): void;
+ };
+ getContextSpy.mockReturnValue({
+ clearRect() {},
+ drawImage() {},
+ getImageData,
+ } as unknown as CanvasRenderingContext2D);
+
+ window.eval(script);
+ return { getImageData };
+}
+
+function runAudit(): AuditIssue[] {
+ const audit = (
+ window as unknown as {
+ __hyperframesLayoutAudit: (options: { time: number; tolerance: number }) => AuditIssue[];
+ }
+ ).__hyperframesLayoutAudit;
+ return audit({ time: 1, tolerance: 2 });
+}
+
+const occlusionOf = (issues: AuditIssue[], id: string) =>
+ issues.find((issue) => issue.code === "text_occluded" && issue.selector === `#${id}`);
+
+const SHIP = { ship: [{ word: "SHIP", left: 800, width: 200 }] };
+const subjectLeftOf = (edge: number) => (x: number) => (x < edge ? 255 : 0);
+
+describe("layout-audit.browser text behind a <video>", () => {
+ afterEach(() => {
+ vi.restoreAllMocks();
+ document.body.innerHTML = "";
+ Reflect.deleteProperty(document, "elementsFromPoint");
+ Reflect.deleteProperty(window, "__hyperframesLayoutAudit");
+ });
+
+ it("keeps a fully opaque frame occluding like any raster", () => {
+ installVideoScene({ text: SHIP });
+ expect(occlusionOf(runAudit(), "ship")).toMatchObject({
+ severity: "error",
+ containerSelector: "#clip",
+ coveredFraction: 1,
+ message: "Text is hidden beneath an opaque element.",
+ });
+ });
+
+ it("does not flag text under the transparent part of a cutout", () => {
+ installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(400) });
+ expect(occlusionOf(runAudit(), "ship")).toBeUndefined();
+ });
+
+ it("reports a partly covered word as info with its share under the matte", () => {
+ installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(860) });
+ expect(occlusionOf(runAudit(), "ship")).toMatchObject({
+ severity: "info",
+ coveredFraction: 0.3,
+ message: `Text sits partly behind a transparent video's matte; its most covered word "SHIP" is 30% hidden.`,
+ });
+ });
+
+ it("fails a word that is 80% or more behind the matte", () => {
+ installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(970) });
+ expect(occlusionOf(runAudit(), "ship")).toMatchObject({
+ severity: "error",
+ coveredFraction: 0.85,
+ message: `"SHIP" is 85% behind a transparent video's matte.`,
+ });
+ });
+
+ it("judges each rendered word, so one lost word fails a mostly clear line", () => {
+ installVideoScene({
+ text: {
+ line1: [
+ { word: "a", left: 200, width: 40 },
+ { word: "big", left: 260, width: 100 },
+ { word: "launch", left: 380, width: 240 },
+ ],
+ },
+ alphaAt: (x) => (x >= 370 ? 255 : 0),
+ });
+ expect(occlusionOf(runAudit(), "line1")).toMatchObject({
+ severity: "error",
+ coveredFraction: 1,
+ message: `"launch" is 100% behind a transparent video's matte.`,
+ });
+ });
+
+ it("fails two neighbouring words that are each half covered", () => {
+ installVideoScene({
+ text: {
+ pair: [
+ { word: "IN", left: 200, width: 100 },
+ { word: "SHIP", left: 320, width: 200 },
+ ],
+ },
+ alphaAt: (_x, y) => (y >= 530 ? 255 : 0),
+ });
+ expect(occlusionOf(runAudit(), "pair")).toMatchObject({
+ severity: "error",
+ coveredFraction: 0.63,
+ message: `Neighbouring words "IN" and "SHIP" are 63% and 63% behind a transparent video's matte.`,
+ });
+ });
+
+ it("applies the neighbour rule across one-word sibling elements", () => {
+ installVideoScene({
+ text: {
+ first: [{ word: "IN", left: 200, width: 100 }],
+ second: [{ word: "SHIP", left: 320, width: 200 }],
+ },
+ alphaAt: (_x, y) => (y >= 530 ? 255 : 0),
+ });
+ const issues = runAudit();
+ expect(occlusionOf(issues, "first")?.severity).toBe("error");
+ expect(occlusionOf(issues, "second")?.severity).toBe("error");
+ });
+
+ it("keeps a single half-covered word at info", () => {
+ installVideoScene({
+ text: {
+ first: [{ word: "IN", left: 200, width: 100 }],
+ second: [{ word: "SHIP", left: 320, width: 200 }],
+ },
+ alphaAt: (x, y) => (x < 310 && y >= 530 ? 255 : 0),
+ });
+ const issues = runAudit();
+ expect(occlusionOf(issues, "first")?.severity).toBe("info");
+ expect(occlusionOf(issues, "second")).toBeUndefined();
+ });
+
+ it("fails a word hidden by a matte feature that falls between two probe columns", () => {
+ // One 1800px line box: its 20% and 35% probe columns sit at x 360 and 630, the strip at 440-540.
+ installVideoScene({
+ text: {
+ caption: [
+ { word: "a", left: 0, width: 40 },
+ { word: "gh", left: 460, width: 60 },
+ { word: "end", left: 1700, width: 100 },
+ ],
+ },
+ wholeLine: true,
+ alphaAt: (x) => (x >= 440 && x < 540 ? 255 : 0),
+ });
+ expect(occlusionOf(runAudit(), "caption")).toMatchObject({
+ severity: "error",
+ containerSelector: "#clip",
+ message: `"gh" is 100% behind a transparent video's matte.`,
+ });
+ });
+
+ it("reads a mirrored video's alpha on the side it is painted", () => {
+ installVideoScene({
+ text: {
+ shown: [{ word: "SHIP", left: 200, width: 200 }],
+ hidden: [{ word: "SHIP", left: 1500, width: 200 }],
+ },
+ videoStyle: { transform: "matrix(-1, 0, 0, 1, 0, 0)" },
+ // Source alpha: the subject fills the source's left half, so it paints on the right.
+ alphaAt: subjectLeftOf(960),
+ });
+ const issues = runAudit();
+ expect(occlusionOf(issues, "shown")).toBeUndefined();
+ expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });
+ });
+
+ it("keeps a video's opaque CSS background occluding behind its clear pixels", () => {
+ installVideoScene({
+ text: SHIP,
+ videoStyle: { backgroundColor: "rgb(18, 52, 86)" },
+ alphaAt: subjectLeftOf(400),
+ });
+ expect(occlusionOf(runAudit(), "ship")).toMatchObject({
+ severity: "error",
+ message: "Text is hidden beneath an opaque element.",
+ });
+ });
+
+ it("respects data-layout-allow-occlusion", () => {
+ installVideoScene({
+ text: SHIP,
+ alphaAt: subjectLeftOf(970),
+ textAttrs: "data-layout-allow-occlusion",
+ });
+ expect(occlusionOf(runAudit(), "ship")).toBeUndefined();
+ });
+
+ it("reads each probed video's frame once per sample and leaves videos under no text alone", () => {
+ const covering = installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(860) });
+ runAudit();
+ expect(covering.getImageData).toHaveBeenCalledTimes(1);
+ runAudit();
+ expect(covering.getImageData).toHaveBeenCalledTimes(2);
+
+ vi.restoreAllMocks();
+ const beneath = installVideoScene({ text: SHIP, video: { overlapsText: false } });
+ runAudit();
+ expect(beneath.getImageData).not.toHaveBeenCalled();
+ });
+
+ it.each([
+ {
+ name: "a video with no decoded frame",
+ video: { readyState: 1 },
+ alphaAt: undefined,
+ reason: "no frame was decoded at this time",
+ },
+ {
+ name: "a video still seeking",
+ video: { seeking: true },
+ alphaAt: undefined,
+ reason: "the seek had not landed",
+ },
+ {
+ name: "a cross-origin video",
+ video: { readError: Object.assign(new Error("tainted"), { name: "SecurityError" }) },
+ alphaAt: undefined,
+ reason: "its source is cross-origin",
+ },
+ {
+ name: "a rotated video",
+ video: {},
+ videoStyle: { transform: "matrix(0, 1, -1, 0, 0, 0)" },
+ alphaAt: subjectLeftOf(400),
+ reason: "it is rotated, skewed or transformed in 3D",
+ },
+ {
+ name: "a frame that reads back fully transparent",
+ video: {},
+ alphaAt: () => 0,
+ reason: "its frame read back fully transparent",
+ },
+ ])(
+ "falls back to opaque with an info note for $name",
+ ({ video, videoStyle, alphaAt, reason }) => {
+ installVideoScene({ text: SHIP, video, videoStyle, alphaAt });
+ const issues = runAudit();
+ expect(occlusionOf(issues, "ship")).toMatchObject({ severity: "error", coveredFraction: 1 });
+ expect(issues).toContainEqual(
+ expect.objectContaining({
+ code: "video_alpha_unread",
+ severity: "info",
+ selector: "#clip",
+ message: `Could not read this video's frame alpha (${reason}), so text under it was judged as if the video were opaque.`,
+ }),
+ );
+ },
+ );
+});
diff --git a/packages/cli/src/commands/layout-audit.video-alpha.chromium.test.ts b/packages/cli/src/commands/layout-audit.video-alpha.chromium.test.ts
new file mode 100644
index 0000000000..bbf5ec4d34
--- /dev/null
+++ b/packages/cli/src/commands/layout-audit.video-alpha.chromium.test.ts
@@ -0,0 +1,291 @@
+import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import { spawnSync } from "node:child_process";
+import { mkdtempSync, readFileSync, rmSync } from "node:fs";
+import { createServer, type Server, type ServerResponse } from "node:http";
+import type { AddressInfo } from "node:net";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import puppeteer, { type Browser } from "puppeteer-core";
+
+interface AuditIssue {
+ code: string;
+ severity: string;
+ selector: string;
+ message: string;
+ coveredFraction?: number;
+}
+
+declare global {
+ interface Window {
+ __hyperframesLayoutAudit(options: { time: number; tolerance: number }): AuditIssue[];
+ }
+}
+
+const executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
+const hasFfmpeg = spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0;
+const script = readFileSync(new URL("./layout-audit.browser.js", import.meta.url), "utf8");
+
+// 320x180 clips shown at 640x360. The cutout's subject fills the left half until 0.5s and the
+// right half after it, the horizon's fills the bottom half, the strip's is opaque only at source
+// x 76-104 (page x 152-208), and the opaque clip has no alpha plane.
+const CLIPS: Record<string, string[]> = {
+ "cutout.webm": [
+ "-f",
+ "lavfi",
+ "-i",
+ "color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(lt(T,0.5),if(lt(X,W/2),255,0),if(gte(X,W/2),255,0))'",
+ "-c:v",
+ "libvpx-vp9",
+ "-pix_fmt",
+ "yuva420p",
+ "-auto-alt-ref",
+ "0",
+ ],
+ "horizon.webm": [
+ "-f",
+ "lavfi",
+ "-i",
+ "color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(gte(Y,H/2),255,0)'",
+ "-c:v",
+ "libvpx-vp9",
+ "-pix_fmt",
+ "yuva420p",
+ "-auto-alt-ref",
+ "0",
+ ],
+ "strip.webm": [
+ "-f",
+ "lavfi",
+ "-i",
+ "color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(between(X,76,104),255,0)'",
+ "-c:v",
+ "libvpx-vp9",
+ "-pix_fmt",
+ "yuva420p",
+ "-auto-alt-ref",
+ "0",
+ ],
+ "opaque.webm": [
+ "-f",
+ "lavfi",
+ "-i",
+ "color=c=blue:s=320x180:d=1:r=10",
+ "-c:v",
+ "libvpx-vp9",
+ "-pix_fmt",
+ "yuv420p",
+ ],
+};
+
+const word = (id: string, left: number, text: string, extra = "") =>
+ `<span id="${id}" ${extra} style="position:absolute;left:${left}px;top:150px;transform:translateX(-50%);margin:0;font:700 48px/1 Arial;color:#fff;white-space:nowrap">${text}</span>`;
+
+function page(video: string, words: string): string {
+ return `<!doctype html><body style="margin:0">
+ <div data-composition-id="main" data-width="640" data-height="360" style="position:relative;width:640px;height:360px;background:#111">
+ <div style="position:absolute;inset:0;z-index:1">${words}</div>
+ ${video}
+ </div></body>`;
+}
+
+const videoTag = (src: string, extra = "", style = "") =>
+ `<video id="clip" src="/${src}" muted playsinline ${extra} style="position:absolute;left:0;top:0;width:640px;height:360px;z-index:2;${style}"></video>`;
+
+// A ~590px caption whose word "gh" is moved onto the strip's centre (page x 180). The coarse probe
+// columns at 20% and 35% of the line then fall either side of the strip, never on it.
+const STRIP_CAPTION = `<span id="line" style="position:absolute;left:0;top:150px;margin:0;font:24px/1 monospace;color:#fff;white-space:nowrap">abcdefghi gh ij kl mn op qr st uv wx yz ab cd</span>
+ <script>
+ const line = document.getElementById("line");
+ const text = line.firstChild;
+ const range = document.createRange();
+ range.setStart(text, 10);
+ range.setEnd(text, 12);
+ const word = range.getBoundingClientRect();
+ line.style.left = 180 - (word.left + word.width / 2) + "px";
+ </script>`;
+
+// Chromium only seeks a media resource its server serves by byte range.
+function serveByteRange(body: Buffer, rangeHeader: string | undefined, response: ServerResponse) {
+ const range = /bytes=(\d+)-(\d*)/.exec(rangeHeader ?? "");
+ const start = range ? Number(range[1]) : 0;
+ const end = range?.[2] ? Number(range[2]) : body.length - 1;
+ response.writeHead(range ? 206 : 200, {
+ "content-type": "video/webm",
+ "accept-ranges": "bytes",
+ "content-length": end - start + 1,
+ ...(range ? { "content-range": `bytes ${start}-${end}/${body.length}` } : {}),
+ });
+ response.end(body.subarray(start, end + 1));
+}
+
+describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in Chromium", () => {
+ let browser: Browser;
+ let server: Server;
+ let origin: string;
+ let clipDir: string;
+ const pages = new Map<string, string>();
+ const clips = new Map<string, Buffer>();
+
+ beforeAll(async () => {
+ clipDir = mkdtempSync(join(tmpdir(), "hf-video-alpha-"));
+ for (const [name, args] of Object.entries(CLIPS)) {
+ const result = spawnSync("ffmpeg", [
+ "-y",
+ "-loglevel",
+ "error",
+ ...args,
+ join(clipDir, name),
+ ]);
+ if (result.status !== 0) throw new Error(`ffmpeg ${name}: ${result.stderr}`);
+ clips.set(`/${name}`, readFileSync(join(clipDir, name)));
+ }
+ server = createServer((request, response) => {
+ const path = (request.url ?? "/").split("?")[0] ?? "/";
+ const html = pages.get(path);
+ if (html) {
+ response.writeHead(200, { "content-type": "text/html" });
+ response.end(html);
+ return;
+ }
+ const body = clips.get(path);
+ if (body) serveByteRange(body, request.headers.range, response);
+ else response.writeHead(404).end();
+ });
+ await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
+ origin = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
+ browser = await puppeteer.launch({ executablePath, args: ["--no-sandbox"] });
+ });
+
+ afterAll(async () => {
+ await browser?.close();
+ await new Promise((resolve) => server?.close(resolve));
+ if (clipDir) rmSync(clipDir, { recursive: true, force: true });
+ });
+
+ async function audit(html: string, seekTo: number | null = 0.25): Promise<AuditIssue[]> {
+ const path = `/page-${pages.size}.html`;
+ pages.set(path, html);
+ const tab = await browser.newPage();
+ try {
+ await tab.setViewport({ width: 640, height: 360, deviceScaleFactor: 1 });
+ await tab.goto(`${origin}${path}`);
+ if (seekTo !== null) {
+ await tab.evaluate(async (time) => {
+ const video = document.querySelector("video");
+ if (!video) return;
+ if (video.readyState < 2) {
+ await new Promise((resolve) => video.addEventListener("loadeddata", resolve));
+ }
+ video.currentTime = time;
+ await new Promise((resolve) => video.addEventListener("seeked", resolve));
+ }, seekTo);
+ }
+ await tab.addScriptTag({ content: script });
+ return await tab.evaluate(() => window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 }));
+ } finally {
+ await tab.close();
+ }
+ }
+
+ const occlusionOf = (issues: AuditIssue[], id: string) =>
+ issues.find((issue) => issue.code === "text_occluded" && issue.selector === `#${id}`);
+
+ it("keeps an opaque video occluding its whole box", async () => {
+ const issues = await audit(page(videoTag("opaque.webm"), word("hidden", 480, "SHIP")));
+ expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });
+ expect(issues.some((issue) => issue.code === "video_alpha_unread")).toBe(false);
+ });
+
+ it("does not flag text behind the transparent part of a cutout", async () => {
+ const issues = await audit(page(videoTag("cutout.webm"), word("clear", 480, "SHIP")));
+ expect(occlusionOf(issues, "clear")).toBeUndefined();
+ });
+
+ it("reports a word half under the subject's matte as info with its coverage", async () => {
+ const issues = await audit(page(videoTag("cutout.webm"), word("tucked", 320, "SHIP")));
+ const finding = occlusionOf(issues, "tucked");
+ expect(finding).toMatchObject({ severity: "info", containerSelector: "#clip" });
+ expect(finding?.coveredFraction).toBeGreaterThan(0.4);
+ expect(finding?.coveredFraction).toBeLessThan(0.6);
+ expect(finding?.message).toMatch(/"SHIP" is \d+% hidden/);
+ });
+
+ it("fails a word that the matte covers entirely", async () => {
+ const issues = await audit(page(videoTag("cutout.webm"), word("eaten", 160, "SHIP")));
+ expect(occlusionOf(issues, "eaten")).toMatchObject({ severity: "error", coveredFraction: 1 });
+ });
+
+ it("fails two neighbouring words that are each at least half covered", async () => {
+ const line = `<span id="pair" style="position:absolute;left:200px;top:160px;margin:0;font:700 48px/1 Arial;color:#fff;white-space:nowrap">IN SHIP</span>`;
+ const issues = await audit(page(videoTag("horizon.webm"), line));
+ const finding = occlusionOf(issues, "pair");
+ expect(finding?.severity).toBe("error");
+ expect(finding?.message).toMatch(/Neighbouring words "IN" and "SHIP"/);
+ });
+
+ it("reads the frame the video was seeked to", async () => {
+ const html = page(
+ videoTag("cutout.webm"),
+ word("left", 160, "SHIP") + word("right", 480, "SHIP"),
+ );
+ const early = await audit(html, 0.25);
+ const late = await audit(html, 0.75);
+ expect(occlusionOf(early, "left")?.severity).toBe("error");
+ expect(occlusionOf(early, "right")).toBeUndefined();
+ expect(occlusionOf(late, "left")).toBeUndefined();
+ expect(occlusionOf(late, "right")?.severity).toBe("error");
+ });
+
+ it("fails a word hidden by a matte feature narrower than the probe grid", async () => {
+ const issues = await audit(page(videoTag("strip.webm"), STRIP_CAPTION));
+ const finding = occlusionOf(issues, "line");
+ expect(finding).toMatchObject({ severity: "error", containerSelector: "#clip" });
+ expect(finding?.message).toMatch(/"gh" is 100% behind/);
+ });
+
+ it("reads a mirrored cutout's alpha on the side it is painted", async () => {
+ const issues = await audit(
+ page(
+ videoTag("cutout.webm", "", "transform:scaleX(-1)"),
+ word("shown", 160, "SHIP") + word("hidden", 480, "SHIP"),
+ ),
+ );
+ expect(occlusionOf(issues, "shown")).toBeUndefined();
+ expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });
+ });
+
+ it("keeps a cutout's opaque CSS background occluding behind its clear pixels", async () => {
+ const issues = await audit(
+ page(videoTag("cutout.webm", "", "background:#123456"), word("boxed", 480, "SHIP")),
+ );
+ expect(occlusionOf(issues, "boxed")).toMatchObject({ severity: "error" });
+ });
+
+ it("judges a rotated cutout as opaque and says why", async () => {
+ const issues = await audit(
+ page(videoTag("cutout.webm", "", "transform:rotate(90deg)"), word("turned", 480, "SHIP")),
+ );
+ expect(occlusionOf(issues, "turned")).toMatchObject({ severity: "error" });
+ expect(issues).toContainEqual(
+ expect.objectContaining({ code: "video_alpha_unread", selector: "#clip" }),
+ );
+ });
+
+ it("respects data-layout-allow-occlusion on text behind a cutout", async () => {
+ const issues = await audit(
+ page(videoTag("cutout.webm"), word("waived", 160, "SHIP", "data-layout-allow-occlusion")),
+ );
+ expect(occlusionOf(issues, "waived")).toBeUndefined();
+ });
+
+ it("falls back to opaque with an info note when no frame has been decoded", async () => {
+ const issues = await audit(
+ page(videoTag("cutout.webm", 'preload="none"'), word("unknown", 480, "SHIP")),
+ null,
+ );
+ expect(occlusionOf(issues, "unknown")?.severity).toBe("error");
+ expect(issues).toContainEqual(
+ expect.objectContaining({ code: "video_alpha_unread", severity: "info", selector: "#clip" }),
+ );
+ });
+});
diff --git a/packages/cli/src/utils/checkBrowser.ts b/packages/cli/src/utils/checkBrowser.ts
index 28fb693eda..d3d7702a75 100644
--- a/packages/cli/src/utils/checkBrowser.ts
+++ b/packages/cli/src/utils/checkBrowser.ts
@@ -1441,6 +1441,7 @@ const LAYOUT_ISSUE_CODES: readonly LayoutIssueCode[] = [
"content_overlap",
"visibility_escapes_segment",
"text_occluded",
+ "video_alpha_unread",
"text_not_painted",
"caption_zone_collision",
"frame_out_of_frame",
diff --git a/packages/cli/src/utils/layoutAudit.ts b/packages/cli/src/utils/layoutAudit.ts
index a3cf312475..353cef06df 100644
--- a/packages/cli/src/utils/layoutAudit.ts
+++ b/packages/cli/src/utils/layoutAudit.ts
@@ -18,6 +18,7 @@ export type LayoutIssueCode =
| "content_overlap"
| "visibility_escapes_segment"
| "text_occluded"
+ | "video_alpha_unread"
| "text_not_painted"
| "caption_zone_collision"
| "frame_out_of_frame"
@@ -62,8 +63,8 @@ export interface LayoutIssue {
rect: LayoutRect;
containerRect?: LayoutRect;
overflow?: LayoutOverflow;
- /** `text_occluded` only: approximate fraction (0-1) of the occlusion probe
- * grid that hit an opaque occluder — see layout-audit.browser.js. */
+ /** `text_occluded` only (0-1): probe-grid share hit by an opaque occluder, or behind a
+ * transparent video the most covered word's share under its alpha (layout-audit.browser.js). */
coveredFraction?: number;
fixHint?: string;
}
diff --git a/skills-manifest.json b/skills-manifest.json
index 3c12e2af1f..a877996429 100644
--- a/skills-manifest.json
+++ b/skills-manifest.json
@@ -30,7 +30,7 @@
"files": 8
},
"hyperframes-cli": {
- "hash": "f08c429fb0f8b82c",
+ "hash": "3c8d1fac9e0a02fe",
"files": 11
},
"hyperframes-core": {
diff --git a/skills/hyperframes-cli/references/lint-validate-inspect.md b/skills/hyperframes-cli/references/lint-validate-inspect.md
index ed2e0472fe..469a120a9f 100644
--- a/skills/hyperframes-cli/references/lint-validate-inspect.md
+++ b/skills/hyperframes-cli/references/lint-validate-inspect.md
@@ -44,7 +44,7 @@ npx hyperframes check --strict # exit non-zero on warnings too (defaul
One command, one Chrome boot. `check` runs the linter first and skips the browser entirely when lint reports errors. When the browser never ran (lint errors, a linter crash, or a browser launch failure), `browserSkipped` is `true` and the `layout`, `motion` and `contrast` sections are empty, not clean. Otherwise it loads the bundled composition once, wires runtime listeners before navigation, and sweeps one seek grid running every audit per sample:
- **Runtime**: JavaScript console errors, unhandled exceptions, failed network requests (media-file `ERR_ABORTED` filtered out), HTTP 4xx/5xx.
-- **Layout**: text extending outside its container or the canvas, text clipped by its own box, held text overlaps and occlusion (with an approximate covered fraction), children escaping clipping containers.
+- **Layout**: text extending outside its container or the canvas, text clipped by its own box, held text overlaps and occlusion (with an approximate covered fraction), children escaping clipping containers. Text under a transparent `<video>` (a `remove-background` cutout) is measured per word against the frame's alpha: partial cover is info with the share hidden, a word 80% hidden or two neighbouring words each half hidden is an error, and a frame that cannot be read falls back to opaque with a `video_alpha_unread` note.
- **Motion**: `*.motion.json` sidecar assertions against the same seeked timeline (see below).
- **Contrast**: WCAG AA on visible text, sampled at 5 grid points. Failures are **errors** and each finding carries the sampled fg/bg colors, measured vs required ratio, and a suggested compliant color in the same palette direction, so most contrast fixes need no screenshot at all.