Write HTML. Render video. Built for agents.
JavaScript aiai-agentsanimationclaude-codeffmpegframeworkgsaphtmlhtml-to-videomcpmotion-graphicspuppeteerrenderingtypescriptvideo-editor
feat(cli): judge text behind a transparent video by its decoded alpha in check (#5657)
WaterrrForevercea8aa8a7dee
diff --git a/docs/guides/remove-background.mdx b/docs/guides/remove-background.mdxindex deb8d91236..f763737cac 100644--- a/docs/guides/remove-background.mdx+++ b/docs/guides/remove-background.mdx@@ -84,6 +84,11 @@ The plate keeps the original surroundings and leaves a transparentperson-shaped hole. It is not an inpainted empty background. Place an opaquegraphic or scene beneath the hole, then place `subject.webm` above it.+`hyperframes check` judges text behind the cutout by the cutout's own alpha at+each sample time. A word partly behind the subject is reported as info with the+share it hides; a word 80% hidden, or two neighbouring words each half hidden,+is an error. You do not need `data-layout-allow-occlusion` for this layering.+## Know when it will work wellThe included model is designed for people. It works best with a clear humandiff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdxindex 42026315d1..dd97ff7a41 100644--- a/docs/packages/cli.mdx+++ b/docs/packages/cli.mdx@@ -857,6 +857,21 @@ cover every finding from the full check.configured seek samples falls inside `--range`; a requested gate never silentlypasses without running.+Text under a `<video>` is judged by the frame the video shows at that sample.+An opaque video covers its whole box, as an opaque image does. A video with+transparency, such as a `remove-background` cutout, covers only where its+alpha is: a word partly behind the subject is an info finding whose+`coveredFraction` is the most covered word's share, and a word 80% hidden or two+neighbouring words each half hidden are an error. A CSS background on the video+still covers its whole box, and a mirrored video (`scaleX(-1)`) is read on the+side it is painted. When a video's frame cannot be read, the audit treats the+video as opaque and adds a `video_alpha_unread` info finding saying why: no frame+decoded yet, the seek still landing, a cross-origin source, no frame size, a+rotated, skewed or 3D-transformed video, or a frame that reads back fully+transparent. A fully transparent frame is treated as unread on purpose, since a+failed draw looks the same; mark text that sits under a cutout before its subject+enters with `data-layout-allow-occlusion`.+Contrast failures are errors, and each one reports the sampled foreground andbackground colours, the measured ratio against the required one, and acompliant colour you could use instead.diff --git a/docs/prompting/rules-and-anti-patterns.mdx b/docs/prompting/rules-and-anti-patterns.mdxindex 6e5d69e32c..1d4a7a26f8 100644--- a/docs/prompting/rules-and-anti-patterns.mdx+++ b/docs/prompting/rules-and-anti-patterns.mdx@@ -64,7 +64,10 @@ Two more ways an SVG "line draws itself" effect (animated `strokeDasharray` / `s`hyperframes check` flags a text block covered by another element as `text_occluded`, andtwo attributes waive it: `data-layout-allow-overlap` for intentional layering, and`data-layout-allow-occlusion` for something deliberately painted over type. Both are-legitimate — a caption designed to sit behind a matted subject needs one.+legitimate. A caption behind a transparent `<video>` cutout needs neither: `check` reads the+cutout's decoded alpha at each sample, reports how much of each word the subject covers as+info, and fails only a word that is 80% hidden or two neighbouring words that are each half+hidden.Two things about them are worth knowing before you reach for either:diff --git a/packages/cli/src/commands/layout-audit.browser.js b/packages/cli/src/commands/layout-audit.browser.jsindex bb8d583a64..9ba3269eb3 100644--- a/packages/cli/src/commands/layout-audit.browser.js+++ b/packages/cli/src/commands/layout-audit.browser.js@@ -1026,14 +1026,46 @@return { x: objectPositionOffset(x, freeX), y: objectPositionOffset(y, freeY) };}- // Return the alpha painted by an <img> at a viewport point. `null` means the- // browser would not let us inspect the image (not loaded or cross-origin), in- // which case callers preserve the conservative opaque fallback.- function imageAlphaAt(element, x, y) {- const sourceWidth = element.naturalWidth;- const sourceHeight = element.naturalHeight;+ const IDENTITY_FLIP = { flipX: false, flipY: false };++ // 2D CSS matrix components [a, b, c, d], or null when the transform is 3D or unparsed.+ function transformMatrix(value) {+ if (!value || value === "none") return [1, 0, 0, 1];+ const match = /^matrix\(([^)]*)\)$/.exec(value.trim());+ if (!match) return null;+ const parts = match[1].split(",").map(Number);+ return parts.length === 6 && parts.every(Number.isFinite) ? parts.slice(0, 4) : null;+ }++ // Which axes the element's paint is mirrored on, or null when a rotation, skew or 3D transform+ // (on it or an ancestor) means its bounding box no longer maps axis-aligned onto its pixels.+ function paintFlips(element) {+ let flipX = false;+ let flipY = false;+ for (let current = element; current; current = current.parentElement) {+ const style = getComputedStyle(current);+ const matrix = transformMatrix(style.transform);+ if (!matrix) return null;+ const [a, b, c, d] = matrix;+ if (Math.abs(b) > 1e-6 || Math.abs(c) > 1e-6) return null;+ if (style.rotate && style.rotate !== "none" && parseFloat(style.rotate) !== 0) return null;+ const scale = String(style.scale || "none")+ .trim()+ .split(/\s+/)+ .map(Number);+ const scaleX = Number.isFinite(scale[0]) ? scale[0] : 1;+ const scaleY = Number.isFinite(scale[1]) ? scale[1] : scaleX;+ if (a * scaleX < 0) flipX = !flipX;+ if (d * scaleY < 0) flipY = !flipY;+ }+ return flipX || flipY ? { flipX, flipY } : IDENTITY_FLIP;+ }++ function paintedSourceMapping(element, sourceWidth, sourceHeight) {const rect = element.getBoundingClientRect();if (!sourceWidth || !sourceHeight || !rect.width || !rect.height) return null;+ const flips = paintFlips(element);+ if (!flips) return null;const style = getComputedStyle(element);const fit = style.objectFit || "fill";@@ -1061,9 +1093,48 @@rect.width - paintedWidth,rect.height - paintedHeight,);- const localX = x - rect.left - offsets.x;- const localY = y - rect.top - offsets.y;- if (localX < 0 || localY < 0 || localX >= paintedWidth || localY >= paintedHeight) return 0;+ return {+ rect: { left: rect.left, top: rect.top, right: rect.right, bottom: rect.bottom },+ offsets,+ ...flips,+ paintedWidth,+ paintedHeight,+ scaleX,+ scaleY,+ sourceWidth,+ sourceHeight,+ };+ }++ // The source pixel painted at a viewport point, or null outside the painted content box.+ function sourcePixelAt(mapping, x, y) {+ const { rect, offsets } = mapping;+ const localX = (mapping.flipX ? rect.right - x : x - rect.left) - offsets.x;+ const localY = (mapping.flipY ? rect.bottom - y : y - rect.top) - offsets.y;+ if (+ localX < 0 ||+ localY < 0 ||+ localX >= mapping.paintedWidth ||+ localY >= mapping.paintedHeight+ ) {+ return null;+ }+ return {+ x: Math.min(mapping.sourceWidth - 1, Math.max(0, Math.floor(localX / mapping.scaleX))),+ y: Math.min(mapping.sourceHeight - 1, Math.max(0, Math.floor(localY / mapping.scaleY))),+ };+ }++ // Return the alpha painted by an <img> at a viewport point. `null` means the+ // browser would not let us inspect the image (not loaded or cross-origin), in+ // which case callers preserve the conservative opaque fallback.+ function imageAlphaAt(element, x, y) {+ const sourceWidth = element.naturalWidth;+ const sourceHeight = element.naturalHeight;+ const mapping = paintedSourceMapping(element, sourceWidth, sourceHeight);+ if (!mapping) return null;+ const pixel = sourcePixelAt(mapping, x, y);+ if (!pixel) return 0;try {let cached = imageAlphaCanvases.get(element);@@ -1083,25 +1154,129 @@cached = { context, width: sourceWidth, height: sourceHeight, source };imageAlphaCanvases.set(element, cached);}- const sourceX = Math.min(sourceWidth - 1, Math.max(0, Math.floor(localX / scaleX)));- const sourceY = Math.min(sourceHeight - 1, Math.max(0, Math.floor(localY / scaleY)));- return cached.context.getImageData(sourceX, sourceY, 1, 1).data[3] / 255;+ return cached.context.getImageData(pixel.x, pixel.y, 1, 1).data[3] / 255;} catch {return null;}}+ const OCCLUDING_ALPHA = 0.6;+ const VIDEO_ALPHA_MAX_SIDE = 480;+ let videoAlphaFrames = new Map();+ let videoAlphaScratch = null;++ function videoAlphaContext(width, height) {+ if (!videoAlphaScratch) {+ const canvas = document.createElement("canvas");+ const context = canvas.getContext("2d", { willReadFrequently: true });+ if (!context) return null;+ videoAlphaScratch = { canvas, context };+ }+ videoAlphaScratch.canvas.width = width;+ videoAlphaScratch.canvas.height = height;+ return videoAlphaScratch.context;+ }++ // Read once per audit call, so per sample time, and only for a video some text probe lands under.+ function videoAlphaFrame(video) {+ let frame = videoAlphaFrames.get(video);+ if (!frame) {+ frame = readVideoAlphaFrame(video);+ videoAlphaFrames.set(video, frame);+ }+ return frame;+ }++ function unreadVideoFrame(reason) {+ return { kind: "unread", reason };+ }++ // "opaque" keeps a video's whole box occluding; "matte" judges each point by the decoded+ // frame's alpha; "unread" falls back to opaque and is reported, never guessed.+ function readVideoAlphaFrame(video) {+ if (video.seeking) return unreadVideoFrame("the seek had not landed");+ if (video.readyState < 2) return unreadVideoFrame("no frame was decoded at this time");+ if (!video.videoWidth || !video.videoHeight)+ return unreadVideoFrame("the video has no frame size");+ const mapping = paintedSourceMapping(video, video.videoWidth, video.videoHeight);+ if (!mapping) return unreadVideoFrame("it is rotated, skewed or transformed in 3D");+ const downscale = Math.min(+ 1,+ VIDEO_ALPHA_MAX_SIDE / Math.max(mapping.sourceWidth, mapping.sourceHeight),+ );+ const width = Math.max(1, Math.round(mapping.sourceWidth * downscale));+ const height = Math.max(1, Math.round(mapping.sourceHeight * downscale));+ let rgba;+ try {+ const context = videoAlphaContext(width, height);+ if (!context) return unreadVideoFrame("no 2D canvas was available");+ context.clearRect(0, 0, width, height);+ context.drawImage(video, 0, 0, width, height);+ rgba = context.getImageData(0, 0, width, height).data;+ } catch (error) {+ return unreadVideoFrame(+ error && error.name === "SecurityError"+ ? "its source is cross-origin"+ : "its frame could not be drawn",+ );+ }+ return classifyVideoAlpha(rgba, width, height, mapping, opacityChain(video));+ }++ function classifyVideoAlpha(rgba, width, height, mapping, opacity) {+ const alpha = new Uint8Array(width * height);+ let clear = 0;+ let painted = 0;+ for (let index = 0; index < alpha.length; index++) {+ alpha[index] = rgba[index * 4 + 3];+ if (alpha[index] < OCCLUDING_ALPHA * 255) clear += 1;+ if (alpha[index] > 0) painted += 1;+ }+ if (clear === 0) return { kind: "opaque" };+ if (painted === 0) return unreadVideoFrame("its frame read back fully transparent");+ return {+ kind: "matte",+ alpha,+ width,+ height,+ opacity,+ mapping: {+ ...mapping,+ sourceWidth: width,+ sourceHeight: height,+ scaleX: mapping.paintedWidth / width,+ scaleY: mapping.paintedHeight / height,+ },+ };+ }++ function matteOccludesAt(frame, x, y) {+ const pixel = sourcePixelAt(frame.mapping, x, y);+ if (!pixel) return false;+ return (frame.alpha[pixel.y * frame.width + pixel.x] / 255) * frame.opacity >= OCCLUDING_ALPHA;+ }++ // A video with an opaque CSS background paints that background behind its clear pixels.+ function matteFrameFor(element) {+ if (element.tagName !== "VIDEO") return null;+ if (hasOpaqueBackground(getComputedStyle(element))) return null;+ const frame = videoAlphaFrame(element);+ return frame.kind === "matte" ? frame : null;+ }+// An element hides text beneath it when it paints opaque pixels at near-full// opacity: raster content (img/video/canvas), a background image, or a solid// background colour. Low-opacity overlays (grain, scrims) do not occlude.function isOpaqueOccluder(element, x, y) {const opacity = opacityChain(element);- if (opacity < 0.6) return false;+ if (opacity < OCCLUDING_ALPHA) return false;if (IGNORE_TAGS.has(element.tagName)) return false;if (element.tagName === "IMG") {const alpha = imageAlphaAt(element, x, y);- if (alpha !== null) return alpha * opacity >= 0.6;+ if (alpha !== null) return alpha * opacity >= OCCLUDING_ALPHA;}+ const matte = matteFrameFor(element);+ if (matte) return matteOccludesAt(matte, x, y);if (RASTER_TAGS.has(element.tagName)) return true;return hasOpaqueBackground(getComputedStyle(element));}@@ -1150,7 +1325,7 @@// there is related to the text, non-opaque, sharing a 3D context with it, or// part of a transient crossfade overlap.// fallow-ignore-next-line complexity- function occluderAt(element, x, y) {+ function occluderAt(element, x, y, seenMattes) {// Walk the paint-ordered stack: a transparent layer on top must not mask an opaque one below it.const stack =typeof document.elementsFromPoint === "function"@@ -1163,6 +1338,8 @@// Pair-specific exemptions excuse this hit only; keep walking for deeper occluders.if (sharedPreserve3d(element, hit)) continue;if (isCrossSceneTransitionOverlap(element, hit)) continue;+ const matte = seenMattes && matteFrameFor(hit);+ if (matte && !seenMattes.has(matte)) seenMattes.set(matte, hit);if (isOpaqueOccluder(hit, x, y)) return hit;}return null;@@ -1196,21 +1373,162 @@function occlusionCoverage(element, textRects) {let occluder = null;let hits = 0;+ let opaqueHits = 0;+ // Every matte above the text, including where its alpha is clear: a feature narrower than the+ // probe grid (an arm, a mic stand) can hide a word between two probe columns.+ const seenMattes = new Map();for (const textRect of textRects) {for (const yFraction of OCCLUSION_PROBE_Y_FRACTIONS) {const y = textRect.top + textRect.height * yFraction;for (const xFraction of OCCLUSION_PROBE_X_FRACTIONS) {- const hit = occluderAt(element, textRect.left + textRect.width * xFraction, y);+ const x = textRect.left + textRect.width * xFraction;+ const hit = occluderAt(element, x, y, seenMattes);if (!hit) continue;hits += 1;if (!occluder) occluder = hit;+ if (!matteFrameFor(hit)) opaqueHits += 1;}}}+ const judgeByMatte = opaqueHits === 0 && seenMattes.size > 0;return {- occluder,+ occluder: occluder || (judgeByMatte ? seenMattes.values().next().value : null),coveredFraction: round(hits / (OCCLUSION_GRID_POINTS * textRects.length)),+ mattes: judgeByMatte ? [...seenMattes.keys()] : [],+ };+ }++ // Thresholds from measuring rendered words against real subject mattes: a word reads as+ // deliberately tucked behind the subject at 30-55% covered, and is lost at 80%, or when it+ // and its neighbour are both half covered.+ const MATTE_WORD_LOST_COVERAGE = 0.8;+ const MATTE_PAIR_LOST_COVERAGE = 0.5;+ const MATTE_MAX_SAMPLES_PER_AXIS = 48;++ function wordClientRects(element) {+ const words = [];+ for (const node of directTextNodes(element)) {+ for (const match of (node.textContent || "").matchAll(/\S+/g)) {+ const range = document.createRange();+ range.setStart(node, match.index);+ range.setEnd(node, match.index + match[0].length);+ const rects = Array.from(range.getClientRects()).filter(+ (rect) => rect.width > 0.5 && rect.height > 0.5,+ );+ range.detach();+ if (rects.length > 0) words.push({ word: match[0], rects });+ }+ }+ return words;+ }++ function matteSampleCount(length, mattes) {+ const pitch = Math.max(1, Math.min(...mattes.map((matte) => matte.mapping.scaleX)));+ return Math.max(1, Math.min(MATTE_MAX_SAMPLES_PER_AXIS, Math.ceil(length / pitch)));+ }++ // Area-weighted share of the rects under any matte's occluding alpha, sampled at the matte's resolution.+ function matteCoverage(rects, mattes) {+ let coveredArea = 0;+ let totalArea = 0;+ for (const rect of rects) {+ const cols = matteSampleCount(rect.width, mattes);+ const rows = matteSampleCount(rect.height, mattes);+ let covered = 0;+ for (let row = 0; row < rows; row++) {+ const y = rect.top + ((row + 0.5) * rect.height) / rows;+ for (let col = 0; col < cols; col++) {+ const x = rect.left + ((col + 0.5) * rect.width) / cols;+ if (mattes.some((matte) => matteOccludesAt(matte, x, y))) covered += 1;+ }+ }+ const area = rect.width * rect.height;+ coveredArea += (area * covered) / (rows * cols);+ totalArea += area;+ }+ return totalArea > 0 ? round(coveredArea / totalArea) : 0;+ }++ function lostNeighbourPair(words) {+ for (let index = 1; index < words.length; index++) {+ const pair = [words[index - 1], words[index]];+ if (pair.every((word) => word.coverage >= MATTE_PAIR_LOST_COVERAGE)) return pair;+ }+ return null;+ }++ function percent(fraction) {+ return `${Math.round(fraction * 100)}%`;+ }++ function lostWordsMessage(lostWord, pair) {+ if (lostWord) {+ return `"${lostWord.word}" is ${percent(lostWord.coverage)} behind a transparent video's matte.`;+ }+ return `Neighbouring words "${pair[0].word}" and "${pair[1].word}" are ${percent(pair[0].coverage)} and ${percent(pair[1].coverage)} behind a transparent video's matte.`;+ }++ function markMatteWordsLost(issue, lostWord, pair) {+ issue.severity = "error";+ issue.message = lostWordsMessage(lostWord, pair);+ issue.fixHint =+ "Move or resize the text so no word is 80% behind the matte and no two neighbouring words are each half behind it, or mark intentional layering with data-layout-allow-occlusion.";+ }++ // Text behind a cutout is judged per rendered word against the decoded alpha: partial cover is+ // reported as info with its share, a lost word is an error.+ function matteOcclusionIssue(element, time, textRect, text, occluder, mattes) {+ const words = wordClientRects(element).map(({ word, rects }) => ({+ word,+ coverage: matteCoverage(rects, mattes),+ }));+ if (words.length === 0) return null;+ const peak = words.reduce((top, word) => (word.coverage > top.coverage ? word : top));+ if (peak.coverage <= 0) return null;+ const issue = {+ code: "text_occluded",+ severity: "info",+ time,+ selector: uniqueSelectorFor(element),+ containerSelector: uniqueSelectorFor(occluder),+ text,+ message: `Text sits partly behind a transparent video's matte; its most covered word "${peak.word}" is ${percent(peak.coverage)} hidden.`,+ rect: textRect,+ coveredFraction: peak.coverage,};+ const lostWord = peak.coverage >= MATTE_WORD_LOST_COVERAGE ? peak : null;+ const pair = lostNeighbourPair(words);+ if (lostWord || pair) markMatteWordsLost(issue, lostWord, pair);+ return { issue, first: words[0], last: words[words.length - 1] };+ }++ // Captions often render one word per element, so the neighbour rule also spans sibling elements.+ function markLostSiblingWords(matteResults) {+ for (const [element, result] of matteResults) {+ const next = element.nextElementSibling && matteResults.get(element.nextElementSibling);+ if (!next) continue;+ const pair = [result.last, next.first];+ if (!pair.every((word) => word.coverage >= MATTE_PAIR_LOST_COVERAGE)) continue;+ for (const { issue } of [result, next]) {+ if (issue.severity !== "error") markMatteWordsLost(issue, null, pair);+ }+ }+ }++ function unreadVideoIssues(time) {+ const issues = [];+ for (const [video, frame] of videoAlphaFrames) {+ if (frame.kind !== "unread") continue;+ issues.push({+ code: "video_alpha_unread",+ severity: "info",+ time,+ selector: uniqueSelectorFor(video),+ message: `Could not read this video's frame alpha (${frame.reason}), so text under it was judged as if the video were opaque.`,+ rect: toRect(video.getBoundingClientRect()),+ });+ }+ return issues;}// pointer-events:none hides elements from elementFromPoint — both probed text AND occluders.@@ -1240,18 +1558,23 @@}// text_occluded: atomic labels flag at any hit; prose needs coveredFraction >= proseCoverageFloor (default 0.15).- function occludedTextIssue(element, time, proseCoverageFloor) {+ function occludedTextIssue(element, time, proseCoverageFloor, matteResults) {if (hasAllowOcclusionFlag(element)) return null;if (!hasVisibleTextInk(element)) return null;const textRect = textRectFor(element, true);if (!textRect) return null;const textRects = textClientRects(element, true);const text = textContentFor(element, true);- const { occluder, coveredFraction } = occlusionCoverage(+ const { occluder, coveredFraction, mattes } = occlusionCoverage(element,textRects.length > 0 ? textRects : [textRect],);if (!occluder) return null;+ if (mattes.length > 0) {+ const result = matteOcclusionIssue(element, time, textRect, text, occluder, mattes);+ if (result) matteResults.set(element, result);+ return result ? result.issue : null;+ }if (!isAtomicLabel(text) && coveredFraction < proseCoverageFloor) return null;return {code: "text_occluded",@@ -2015,6 +2338,8 @@isVisibleElement(element, 0.05),);const issues = [];+ const matteResults = new Map();+ videoAlphaFrames = new Map();const restoreHits = restoreHitTesting(root);try {@@ -2023,7 +2348,7 @@const clipped = clippedTextIssue(element, time, tolerance);if (clipped) issues.push(clipped);issues.push(...textOverflowIssues(element, root, rootRect, time, tolerance, clipped));- const occluded = occludedTextIssue(element, time, proseCoverageFloor);+ const occluded = occludedTextIssue(element, time, proseCoverageFloor, matteResults);if (occluded) issues.push(occluded);const invisible = invisibleTextIssue(element, time);if (invisible) issues.push(invisible);@@ -2031,6 +2356,8 @@} finally {restoreHits();}+ markLostSiblingWords(matteResults);+ issues.push(...unreadVideoIssues(time));issues.push(...containerOverflowIssues(root, time, tolerance));issues.push(...contentOverlapIssues(root, time));diff --git a/packages/cli/src/commands/layout-audit.video-alpha.browser.test.ts b/packages/cli/src/commands/layout-audit.video-alpha.browser.test.tsnew file mode 100644index 0000000000..16a601b76f--- /dev/null+++ b/packages/cli/src/commands/layout-audit.video-alpha.browser.test.ts@@ -0,0 +1,427 @@+// @vitest-environment happy-dom+import { afterEach, describe, expect, it, vi } from "vitest";+import { readFileSync } from "node:fs";+import { dirname, join } from "node:path";+import { fileURLToPath } from "node:url";++const script = readFileSync(+ join(dirname(fileURLToPath(import.meta.url)), "layout-audit.browser.js"),+ "utf-8",+);++interface AuditIssue {+ code: string;+ severity: string;+ selector: string;+ containerSelector?: string;+ message: string;+ coveredFraction?: number;+}++interface Box {+ left: number;+ top: number;+ width: number;+ height: number;+}++// The <video> fills a 1920x1080 frame from a 192x108 source, so one source pixel is 10x10 px.+const SOURCE_WIDTH = 192;+const SOURCE_HEIGHT = 108;+const FRAME: Box = { left: 0, top: 0, width: 1920, height: 1080 };+const WORD_TOP = 500;+const WORD_HEIGHT = 80;++interface VideoScene {+ /** Text elements: id -> words with their left edge and width, all on one line. */+ text: Record<string, Array<{ word: string; left: number; width: number }>>;+ /** Alpha (0-255) the decoded frame paints at a composition point. */+ alphaAt?: (x: number, y: number) => number;+ video?: { readyState?: number; seeking?: boolean; readError?: Error; overlapsText?: boolean };+ /** Computed style the <video> adds to the base, e.g. a transform or a background. */+ videoStyle?: Record<string, string>;+ /** Report each element's text as one line box, as Chromium does, rather than one per word. */+ wholeLine?: boolean;+ textAttrs?: string;+}++function domRect({ left, top, width, height }: Box): DOMRect {+ return {+ left,+ top,+ width,+ height,+ right: left + width,+ bottom: top + height,+ x: left,+ y: top,+ toJSON() {+ return this;+ },+ } as DOMRect;+}++const BASE_STYLE: Record<string, string> = {+ display: "block",+ visibility: "visible",+ opacity: "1",+ overflow: "visible",+ overflowX: "visible",+ overflowY: "visible",+ backgroundColor: "rgba(0, 0, 0, 0)",+ backgroundImage: "none",+ objectFit: "fill",+ objectPosition: "50% 50%",+ fontSize: "36px",+};++function wordBox(left: number, width: number): Box {+ return { left, top: WORD_TOP, width, height: WORD_HEIGHT };+}++function installVideoScene(scene: VideoScene): { getImageData: ReturnType<typeof vi.fn> } {+ const spans = Object.entries(scene.text)+ .map(+ ([id, words]) =>+ `<span id="${id}" ${scene.textAttrs ?? ""}>${words.map(({ word }) => word).join(" ")}</span>`,+ )+ .join("");+ document.body.innerHTML = `+ <div id="root" data-composition-id="main" data-width="1920" data-height="1080">+ <div id="line">${spans}</div>+ <video id="clip"></video>+ </div>`;++ const video = document.getElementById("clip") as HTMLVideoElement;+ Object.defineProperties(video, {+ videoWidth: { configurable: true, value: SOURCE_WIDTH },+ videoHeight: { configurable: true, value: SOURCE_HEIGHT },+ readyState: { configurable: true, value: scene.video?.readyState ?? 4 },+ seeking: { configurable: true, value: scene.video?.seeking ?? false },+ });++ vi.spyOn(window, "getComputedStyle").mockImplementation(+ (element) =>+ ({+ ...BASE_STYLE,+ ...(element === video ? scene.videoStyle : {}),+ }) as unknown as CSSStyleDeclaration,+ );+ for (const element of Array.from(document.querySelectorAll("*"))) {+ vi.spyOn(element, "getBoundingClientRect").mockReturnValue(domRect(FRAME));+ }++ const wordRect = (node: Node | null, start: number, end: number): DOMRect[] => {+ const owner = node?.parentElement?.id ?? "";+ const word = (node?.textContent ?? "").slice(start, end);+ const entry = scene.text[owner]?.find((candidate) => candidate.word === word);+ return entry ? [domRect(wordBox(entry.left, entry.width))] : [];+ };+ vi.spyOn(document, "createRange").mockImplementation(() => {+ let node: Node | null = null;+ let start = 0;+ let end: number | null = null;+ return {+ selectNodeContents(selected: Node) {+ node = selected;+ },+ setStart(selected: Node, offset: number) {+ node = selected;+ start = offset;+ },+ setEnd(_selected: Node, offset: number) {+ end = offset;+ },+ getClientRects() {+ const owner = node?.parentElement?.id ?? "";+ if (end === null) {+ const boxes = (scene.text[owner] ?? []).map(({ left, width }) => wordBox(left, width));+ if (scene.wholeLine && boxes.length > 0) {+ const left = Math.min(...boxes.map((box) => box.left));+ const right = Math.max(...boxes.map((box) => box.left + box.width));+ return [domRect(wordBox(left, right - left))] as unknown as DOMRectList;+ }+ return boxes.map(domRect) as unknown as DOMRectList;+ }+ return wordRect(node, start, end) as unknown as DOMRectList;+ },+ detach() {},+ } as unknown as Range;+ });++ const overlapsText = scene.video?.overlapsText ?? true;+ Object.defineProperty(document, "elementsFromPoint", {+ configurable: true,+ value: (x: number) => {+ const textElement = Object.keys(scene.text)+ .map((id) => document.getElementById(id))+ .find((element) => {+ const words = scene.text[element?.id ?? ""] ?? [];+ return words.some(({ left, width }) => x >= left && x < left + width);+ });+ return [...(overlapsText ? [video] : []), textElement, video].filter(Boolean);+ },+ });++ const alphaAt = scene.alphaAt ?? (() => 255);+ const getImageData = vi.fn((_x: number, _y: number, width: number, height: number) => {+ if (scene.video?.readError) throw scene.video.readError;+ const data = new Uint8ClampedArray(width * height * 4);+ for (let row = 0; row < height; row++) {+ for (let col = 0; col < width; col++) {+ const x = ((col + 0.5) * FRAME.width) / width;+ const y = ((row + 0.5) * FRAME.height) / height;+ data[(row * width + col) * 4 + 3] = alphaAt(x, y);+ }+ }+ return { data };+ });+ // getContext's overloads resolve to the WebGPU one; the audit only asks for "2d".+ const getContextSpy = vi.spyOn(HTMLCanvasElement.prototype, "getContext") as unknown as {+ mockReturnValue(value: CanvasRenderingContext2D): void;+ };+ getContextSpy.mockReturnValue({+ clearRect() {},+ drawImage() {},+ getImageData,+ } as unknown as CanvasRenderingContext2D);++ window.eval(script);+ return { getImageData };+}++function runAudit(): AuditIssue[] {+ const audit = (+ window as unknown as {+ __hyperframesLayoutAudit: (options: { time: number; tolerance: number }) => AuditIssue[];+ }+ ).__hyperframesLayoutAudit;+ return audit({ time: 1, tolerance: 2 });+}++const occlusionOf = (issues: AuditIssue[], id: string) =>+ issues.find((issue) => issue.code === "text_occluded" && issue.selector === `#${id}`);++const SHIP = { ship: [{ word: "SHIP", left: 800, width: 200 }] };+const subjectLeftOf = (edge: number) => (x: number) => (x < edge ? 255 : 0);++describe("layout-audit.browser text behind a <video>", () => {+ afterEach(() => {+ vi.restoreAllMocks();+ document.body.innerHTML = "";+ Reflect.deleteProperty(document, "elementsFromPoint");+ Reflect.deleteProperty(window, "__hyperframesLayoutAudit");+ });++ it("keeps a fully opaque frame occluding like any raster", () => {+ installVideoScene({ text: SHIP });+ expect(occlusionOf(runAudit(), "ship")).toMatchObject({+ severity: "error",+ containerSelector: "#clip",+ coveredFraction: 1,+ message: "Text is hidden beneath an opaque element.",+ });+ });++ it("does not flag text under the transparent part of a cutout", () => {+ installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(400) });+ expect(occlusionOf(runAudit(), "ship")).toBeUndefined();+ });++ it("reports a partly covered word as info with its share under the matte", () => {+ installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(860) });+ expect(occlusionOf(runAudit(), "ship")).toMatchObject({+ severity: "info",+ coveredFraction: 0.3,+ message: `Text sits partly behind a transparent video's matte; its most covered word "SHIP" is 30% hidden.`,+ });+ });++ it("fails a word that is 80% or more behind the matte", () => {+ installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(970) });+ expect(occlusionOf(runAudit(), "ship")).toMatchObject({+ severity: "error",+ coveredFraction: 0.85,+ message: `"SHIP" is 85% behind a transparent video's matte.`,+ });+ });++ it("judges each rendered word, so one lost word fails a mostly clear line", () => {+ installVideoScene({+ text: {+ line1: [+ { word: "a", left: 200, width: 40 },+ { word: "big", left: 260, width: 100 },+ { word: "launch", left: 380, width: 240 },+ ],+ },+ alphaAt: (x) => (x >= 370 ? 255 : 0),+ });+ expect(occlusionOf(runAudit(), "line1")).toMatchObject({+ severity: "error",+ coveredFraction: 1,+ message: `"launch" is 100% behind a transparent video's matte.`,+ });+ });++ it("fails two neighbouring words that are each half covered", () => {+ installVideoScene({+ text: {+ pair: [+ { word: "IN", left: 200, width: 100 },+ { word: "SHIP", left: 320, width: 200 },+ ],+ },+ alphaAt: (_x, y) => (y >= 530 ? 255 : 0),+ });+ expect(occlusionOf(runAudit(), "pair")).toMatchObject({+ severity: "error",+ coveredFraction: 0.63,+ message: `Neighbouring words "IN" and "SHIP" are 63% and 63% behind a transparent video's matte.`,+ });+ });++ it("applies the neighbour rule across one-word sibling elements", () => {+ installVideoScene({+ text: {+ first: [{ word: "IN", left: 200, width: 100 }],+ second: [{ word: "SHIP", left: 320, width: 200 }],+ },+ alphaAt: (_x, y) => (y >= 530 ? 255 : 0),+ });+ const issues = runAudit();+ expect(occlusionOf(issues, "first")?.severity).toBe("error");+ expect(occlusionOf(issues, "second")?.severity).toBe("error");+ });++ it("keeps a single half-covered word at info", () => {+ installVideoScene({+ text: {+ first: [{ word: "IN", left: 200, width: 100 }],+ second: [{ word: "SHIP", left: 320, width: 200 }],+ },+ alphaAt: (x, y) => (x < 310 && y >= 530 ? 255 : 0),+ });+ const issues = runAudit();+ expect(occlusionOf(issues, "first")?.severity).toBe("info");+ expect(occlusionOf(issues, "second")).toBeUndefined();+ });++ it("fails a word hidden by a matte feature that falls between two probe columns", () => {+ // One 1800px line box: its 20% and 35% probe columns sit at x 360 and 630, the strip at 440-540.+ installVideoScene({+ text: {+ caption: [+ { word: "a", left: 0, width: 40 },+ { word: "gh", left: 460, width: 60 },+ { word: "end", left: 1700, width: 100 },+ ],+ },+ wholeLine: true,+ alphaAt: (x) => (x >= 440 && x < 540 ? 255 : 0),+ });+ expect(occlusionOf(runAudit(), "caption")).toMatchObject({+ severity: "error",+ containerSelector: "#clip",+ message: `"gh" is 100% behind a transparent video's matte.`,+ });+ });++ it("reads a mirrored video's alpha on the side it is painted", () => {+ installVideoScene({+ text: {+ shown: [{ word: "SHIP", left: 200, width: 200 }],+ hidden: [{ word: "SHIP", left: 1500, width: 200 }],+ },+ videoStyle: { transform: "matrix(-1, 0, 0, 1, 0, 0)" },+ // Source alpha: the subject fills the source's left half, so it paints on the right.+ alphaAt: subjectLeftOf(960),+ });+ const issues = runAudit();+ expect(occlusionOf(issues, "shown")).toBeUndefined();+ expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });+ });++ it("keeps a video's opaque CSS background occluding behind its clear pixels", () => {+ installVideoScene({+ text: SHIP,+ videoStyle: { backgroundColor: "rgb(18, 52, 86)" },+ alphaAt: subjectLeftOf(400),+ });+ expect(occlusionOf(runAudit(), "ship")).toMatchObject({+ severity: "error",+ message: "Text is hidden beneath an opaque element.",+ });+ });++ it("respects data-layout-allow-occlusion", () => {+ installVideoScene({+ text: SHIP,+ alphaAt: subjectLeftOf(970),+ textAttrs: "data-layout-allow-occlusion",+ });+ expect(occlusionOf(runAudit(), "ship")).toBeUndefined();+ });++ it("reads each probed video's frame once per sample and leaves videos under no text alone", () => {+ const covering = installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(860) });+ runAudit();+ expect(covering.getImageData).toHaveBeenCalledTimes(1);+ runAudit();+ expect(covering.getImageData).toHaveBeenCalledTimes(2);++ vi.restoreAllMocks();+ const beneath = installVideoScene({ text: SHIP, video: { overlapsText: false } });+ runAudit();+ expect(beneath.getImageData).not.toHaveBeenCalled();+ });++ it.each([+ {+ name: "a video with no decoded frame",+ video: { readyState: 1 },+ alphaAt: undefined,+ reason: "no frame was decoded at this time",+ },+ {+ name: "a video still seeking",+ video: { seeking: true },+ alphaAt: undefined,+ reason: "the seek had not landed",+ },+ {+ name: "a cross-origin video",+ video: { readError: Object.assign(new Error("tainted"), { name: "SecurityError" }) },+ alphaAt: undefined,+ reason: "its source is cross-origin",+ },+ {+ name: "a rotated video",+ video: {},+ videoStyle: { transform: "matrix(0, 1, -1, 0, 0, 0)" },+ alphaAt: subjectLeftOf(400),+ reason: "it is rotated, skewed or transformed in 3D",+ },+ {+ name: "a frame that reads back fully transparent",+ video: {},+ alphaAt: () => 0,+ reason: "its frame read back fully transparent",+ },+ ])(+ "falls back to opaque with an info note for $name",+ ({ video, videoStyle, alphaAt, reason }) => {+ installVideoScene({ text: SHIP, video, videoStyle, alphaAt });+ const issues = runAudit();+ expect(occlusionOf(issues, "ship")).toMatchObject({ severity: "error", coveredFraction: 1 });+ expect(issues).toContainEqual(+ expect.objectContaining({+ code: "video_alpha_unread",+ severity: "info",+ selector: "#clip",+ message: `Could not read this video's frame alpha (${reason}), so text under it was judged as if the video were opaque.`,+ }),+ );+ },+ );+});diff --git a/packages/cli/src/commands/layout-audit.video-alpha.chromium.test.ts b/packages/cli/src/commands/layout-audit.video-alpha.chromium.test.tsnew file mode 100644index 0000000000..bbf5ec4d34--- /dev/null+++ b/packages/cli/src/commands/layout-audit.video-alpha.chromium.test.ts@@ -0,0 +1,291 @@+import { afterAll, beforeAll, describe, expect, it } from "vitest";+import { spawnSync } from "node:child_process";+import { mkdtempSync, readFileSync, rmSync } from "node:fs";+import { createServer, type Server, type ServerResponse } from "node:http";+import type { AddressInfo } from "node:net";+import { tmpdir } from "node:os";+import { join } from "node:path";+import puppeteer, { type Browser } from "puppeteer-core";++interface AuditIssue {+ code: string;+ severity: string;+ selector: string;+ message: string;+ coveredFraction?: number;+}++declare global {+ interface Window {+ __hyperframesLayoutAudit(options: { time: number; tolerance: number }): AuditIssue[];+ }+}++const executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;+const hasFfmpeg = spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0;+const script = readFileSync(new URL("./layout-audit.browser.js", import.meta.url), "utf8");++// 320x180 clips shown at 640x360. The cutout's subject fills the left half until 0.5s and the+// right half after it, the horizon's fills the bottom half, the strip's is opaque only at source+// x 76-104 (page x 152-208), and the opaque clip has no alpha plane.+const CLIPS: Record<string, string[]> = {+ "cutout.webm": [+ "-f",+ "lavfi",+ "-i",+ "color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(lt(T,0.5),if(lt(X,W/2),255,0),if(gte(X,W/2),255,0))'",+ "-c:v",+ "libvpx-vp9",+ "-pix_fmt",+ "yuva420p",+ "-auto-alt-ref",+ "0",+ ],+ "horizon.webm": [+ "-f",+ "lavfi",+ "-i",+ "color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(gte(Y,H/2),255,0)'",+ "-c:v",+ "libvpx-vp9",+ "-pix_fmt",+ "yuva420p",+ "-auto-alt-ref",+ "0",+ ],+ "strip.webm": [+ "-f",+ "lavfi",+ "-i",+ "color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(between(X,76,104),255,0)'",+ "-c:v",+ "libvpx-vp9",+ "-pix_fmt",+ "yuva420p",+ "-auto-alt-ref",+ "0",+ ],+ "opaque.webm": [+ "-f",+ "lavfi",+ "-i",+ "color=c=blue:s=320x180:d=1:r=10",+ "-c:v",+ "libvpx-vp9",+ "-pix_fmt",+ "yuv420p",+ ],+};++const word = (id: string, left: number, text: string, extra = "") =>+ `<span id="${id}" ${extra} style="position:absolute;left:${left}px;top:150px;transform:translateX(-50%);margin:0;font:700 48px/1 Arial;color:#fff;white-space:nowrap">${text}</span>`;++function page(video: string, words: string): string {+ return `<!doctype html><body style="margin:0">+ <div data-composition-id="main" data-width="640" data-height="360" style="position:relative;width:640px;height:360px;background:#111">+ <div style="position:absolute;inset:0;z-index:1">${words}</div>+ ${video}+ </div></body>`;+}++const videoTag = (src: string, extra = "", style = "") =>+ `<video id="clip" src="/${src}" muted playsinline ${extra} style="position:absolute;left:0;top:0;width:640px;height:360px;z-index:2;${style}"></video>`;++// A ~590px caption whose word "gh" is moved onto the strip's centre (page x 180). The coarse probe+// columns at 20% and 35% of the line then fall either side of the strip, never on it.+const STRIP_CAPTION = `<span id="line" style="position:absolute;left:0;top:150px;margin:0;font:24px/1 monospace;color:#fff;white-space:nowrap">abcdefghi gh ij kl mn op qr st uv wx yz ab cd</span>+ <script>+ const line = document.getElementById("line");+ const text = line.firstChild;+ const range = document.createRange();+ range.setStart(text, 10);+ range.setEnd(text, 12);+ const word = range.getBoundingClientRect();+ line.style.left = 180 - (word.left + word.width / 2) + "px";+ </script>`;++// Chromium only seeks a media resource its server serves by byte range.+function serveByteRange(body: Buffer, rangeHeader: string | undefined, response: ServerResponse) {+ const range = /bytes=(\d+)-(\d*)/.exec(rangeHeader ?? "");+ const start = range ? Number(range[1]) : 0;+ const end = range?.[2] ? Number(range[2]) : body.length - 1;+ response.writeHead(range ? 206 : 200, {+ "content-type": "video/webm",+ "accept-ranges": "bytes",+ "content-length": end - start + 1,+ ...(range ? { "content-range": `bytes ${start}-${end}/${body.length}` } : {}),+ });+ response.end(body.subarray(start, end + 1));+}++describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in Chromium", () => {+ let browser: Browser;+ let server: Server;+ let origin: string;+ let clipDir: string;+ const pages = new Map<string, string>();+ const clips = new Map<string, Buffer>();++ beforeAll(async () => {+ clipDir = mkdtempSync(join(tmpdir(), "hf-video-alpha-"));+ for (const [name, args] of Object.entries(CLIPS)) {+ const result = spawnSync("ffmpeg", [+ "-y",+ "-loglevel",+ "error",+ ...args,+ join(clipDir, name),+ ]);+ if (result.status !== 0) throw new Error(`ffmpeg ${name}: ${result.stderr}`);+ clips.set(`/${name}`, readFileSync(join(clipDir, name)));+ }+ server = createServer((request, response) => {+ const path = (request.url ?? "/").split("?")[0] ?? "/";+ const html = pages.get(path);+ if (html) {+ response.writeHead(200, { "content-type": "text/html" });+ response.end(html);+ return;+ }+ const body = clips.get(path);+ if (body) serveByteRange(body, request.headers.range, response);+ else response.writeHead(404).end();+ });+ await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));+ origin = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;+ browser = await puppeteer.launch({ executablePath, args: ["--no-sandbox"] });+ });++ afterAll(async () => {+ await browser?.close();+ await new Promise((resolve) => server?.close(resolve));+ if (clipDir) rmSync(clipDir, { recursive: true, force: true });+ });++ async function audit(html: string, seekTo: number | null = 0.25): Promise<AuditIssue[]> {+ const path = `/page-${pages.size}.html`;+ pages.set(path, html);+ const tab = await browser.newPage();+ try {+ await tab.setViewport({ width: 640, height: 360, deviceScaleFactor: 1 });+ await tab.goto(`${origin}${path}`);+ if (seekTo !== null) {+ await tab.evaluate(async (time) => {+ const video = document.querySelector("video");+ if (!video) return;+ if (video.readyState < 2) {+ await new Promise((resolve) => video.addEventListener("loadeddata", resolve));+ }+ video.currentTime = time;+ await new Promise((resolve) => video.addEventListener("seeked", resolve));+ }, seekTo);+ }+ await tab.addScriptTag({ content: script });+ return await tab.evaluate(() => window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 }));+ } finally {+ await tab.close();+ }+ }++ const occlusionOf = (issues: AuditIssue[], id: string) =>+ issues.find((issue) => issue.code === "text_occluded" && issue.selector === `#${id}`);++ it("keeps an opaque video occluding its whole box", async () => {+ const issues = await audit(page(videoTag("opaque.webm"), word("hidden", 480, "SHIP")));+ expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });+ expect(issues.some((issue) => issue.code === "video_alpha_unread")).toBe(false);+ });++ it("does not flag text behind the transparent part of a cutout", async () => {+ const issues = await audit(page(videoTag("cutout.webm"), word("clear", 480, "SHIP")));+ expect(occlusionOf(issues, "clear")).toBeUndefined();+ });++ it("reports a word half under the subject's matte as info with its coverage", async () => {+ const issues = await audit(page(videoTag("cutout.webm"), word("tucked", 320, "SHIP")));+ const finding = occlusionOf(issues, "tucked");+ expect(finding).toMatchObject({ severity: "info", containerSelector: "#clip" });+ expect(finding?.coveredFraction).toBeGreaterThan(0.4);+ expect(finding?.coveredFraction).toBeLessThan(0.6);+ expect(finding?.message).toMatch(/"SHIP" is \d+% hidden/);+ });++ it("fails a word that the matte covers entirely", async () => {+ const issues = await audit(page(videoTag("cutout.webm"), word("eaten", 160, "SHIP")));+ expect(occlusionOf(issues, "eaten")).toMatchObject({ severity: "error", coveredFraction: 1 });+ });++ it("fails two neighbouring words that are each at least half covered", async () => {+ const line = `<span id="pair" style="position:absolute;left:200px;top:160px;margin:0;font:700 48px/1 Arial;color:#fff;white-space:nowrap">IN SHIP</span>`;+ const issues = await audit(page(videoTag("horizon.webm"), line));+ const finding = occlusionOf(issues, "pair");+ expect(finding?.severity).toBe("error");+ expect(finding?.message).toMatch(/Neighbouring words "IN" and "SHIP"/);+ });++ it("reads the frame the video was seeked to", async () => {+ const html = page(+ videoTag("cutout.webm"),+ word("left", 160, "SHIP") + word("right", 480, "SHIP"),+ );+ const early = await audit(html, 0.25);+ const late = await audit(html, 0.75);+ expect(occlusionOf(early, "left")?.severity).toBe("error");+ expect(occlusionOf(early, "right")).toBeUndefined();+ expect(occlusionOf(late, "left")).toBeUndefined();+ expect(occlusionOf(late, "right")?.severity).toBe("error");+ });++ it("fails a word hidden by a matte feature narrower than the probe grid", async () => {+ const issues = await audit(page(videoTag("strip.webm"), STRIP_CAPTION));+ const finding = occlusionOf(issues, "line");+ expect(finding).toMatchObject({ severity: "error", containerSelector: "#clip" });+ expect(finding?.message).toMatch(/"gh" is 100% behind/);+ });++ it("reads a mirrored cutout's alpha on the side it is painted", async () => {+ const issues = await audit(+ page(+ videoTag("cutout.webm", "", "transform:scaleX(-1)"),+ word("shown", 160, "SHIP") + word("hidden", 480, "SHIP"),+ ),+ );+ expect(occlusionOf(issues, "shown")).toBeUndefined();+ expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });+ });++ it("keeps a cutout's opaque CSS background occluding behind its clear pixels", async () => {+ const issues = await audit(+ page(videoTag("cutout.webm", "", "background:#123456"), word("boxed", 480, "SHIP")),+ );+ expect(occlusionOf(issues, "boxed")).toMatchObject({ severity: "error" });+ });++ it("judges a rotated cutout as opaque and says why", async () => {+ const issues = await audit(+ page(videoTag("cutout.webm", "", "transform:rotate(90deg)"), word("turned", 480, "SHIP")),+ );+ expect(occlusionOf(issues, "turned")).toMatchObject({ severity: "error" });+ expect(issues).toContainEqual(+ expect.objectContaining({ code: "video_alpha_unread", selector: "#clip" }),+ );+ });++ it("respects data-layout-allow-occlusion on text behind a cutout", async () => {+ const issues = await audit(+ page(videoTag("cutout.webm"), word("waived", 160, "SHIP", "data-layout-allow-occlusion")),+ );+ expect(occlusionOf(issues, "waived")).toBeUndefined();+ });++ it("falls back to opaque with an info note when no frame has been decoded", async () => {+ const issues = await audit(+ page(videoTag("cutout.webm", 'preload="none"'), word("unknown", 480, "SHIP")),+ null,+ );+ expect(occlusionOf(issues, "unknown")?.severity).toBe("error");+ expect(issues).toContainEqual(+ expect.objectContaining({ code: "video_alpha_unread", severity: "info", selector: "#clip" }),+ );+ });+});diff --git a/packages/cli/src/utils/checkBrowser.ts b/packages/cli/src/utils/checkBrowser.tsindex 28fb693eda..d3d7702a75 100644--- a/packages/cli/src/utils/checkBrowser.ts+++ b/packages/cli/src/utils/checkBrowser.ts@@ -1441,6 +1441,7 @@ const LAYOUT_ISSUE_CODES: readonly LayoutIssueCode[] = ["content_overlap","visibility_escapes_segment","text_occluded",+ "video_alpha_unread","text_not_painted","caption_zone_collision","frame_out_of_frame",diff --git a/packages/cli/src/utils/layoutAudit.ts b/packages/cli/src/utils/layoutAudit.tsindex a3cf312475..353cef06df 100644--- a/packages/cli/src/utils/layoutAudit.ts+++ b/packages/cli/src/utils/layoutAudit.ts@@ -18,6 +18,7 @@ export type LayoutIssueCode =| "content_overlap"| "visibility_escapes_segment"| "text_occluded"+ | "video_alpha_unread"| "text_not_painted"| "caption_zone_collision"| "frame_out_of_frame"@@ -62,8 +63,8 @@ export interface LayoutIssue {rect: LayoutRect;containerRect?: LayoutRect;overflow?: LayoutOverflow;- /** `text_occluded` only: approximate fraction (0-1) of the occlusion probe- * grid that hit an opaque occluder — see layout-audit.browser.js. */+ /** `text_occluded` only (0-1): probe-grid share hit by an opaque occluder, or behind a+ * transparent video the most covered word's share under its alpha (layout-audit.browser.js). */coveredFraction?: number;fixHint?: string;}diff --git a/skills-manifest.json b/skills-manifest.jsonindex 3c12e2af1f..a877996429 100644--- a/skills-manifest.json+++ b/skills-manifest.json@@ -30,7 +30,7 @@"files": 8},"hyperframes-cli": {- "hash": "f08c429fb0f8b82c",+ "hash": "3c8d1fac9e0a02fe","files": 11},"hyperframes-core": {diff --git a/skills/hyperframes-cli/references/lint-validate-inspect.md b/skills/hyperframes-cli/references/lint-validate-inspect.mdindex ed2e0472fe..469a120a9f 100644--- a/skills/hyperframes-cli/references/lint-validate-inspect.md+++ b/skills/hyperframes-cli/references/lint-validate-inspect.md@@ -44,7 +44,7 @@ npx hyperframes check --strict # exit non-zero on warnings too (defaulOne command, one Chrome boot. `check` runs the linter first and skips the browser entirely when lint reports errors. When the browser never ran (lint errors, a linter crash, or a browser launch failure), `browserSkipped` is `true` and the `layout`, `motion` and `contrast` sections are empty, not clean. Otherwise it loads the bundled composition once, wires runtime listeners before navigation, and sweeps one seek grid running every audit per sample:- **Runtime**: JavaScript console errors, unhandled exceptions, failed network requests (media-file `ERR_ABORTED` filtered out), HTTP 4xx/5xx.-- **Layout**: text extending outside its container or the canvas, text clipped by its own box, held text overlaps and occlusion (with an approximate covered fraction), children escaping clipping containers.+- **Layout**: text extending outside its container or the canvas, text clipped by its own box, held text overlaps and occlusion (with an approximate covered fraction), children escaping clipping containers. Text under a transparent `<video>` (a `remove-background` cutout) is measured per word against the frame's alpha: partial cover is info with the share hidden, a word 80% hidden or two neighbouring words each half hidden is an error, and a frame that cannot be read falls back to opaque with a `video_alpha_unread` note.- **Motion**: `*.motion.json` sidecar assertions against the same seeked timeline (see below).- **Contrast**: WCAG AA on visible text, sampled at 5 grid points. Failures are **errors** and each finding carries the sampled fg/bg colors, measured vs required ratio, and a suggested compliant color in the same palette direction, so most contrast fixes need no screenshot at all.