Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

feat(interaction): cast filmstrips, rect sampling, a Storybook highlight default and the WebKit shader cache (#8549)

Cindy Zhang64ed81235b3a
diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.md
index 4ab9aaa260..91e418f342 100644
--- a/internal/interaction-harness/README.md
+++ b/internal/interaction-harness/README.md
@@ -38,6 +38,12 @@ under a rootful docker daemon the container runs as your user so its
node internal/interaction-harness/scripts/webkit-container.mjs [playwright test args…]
```
+WebKit composites through Mesa's software GL there, which compiles each shader
+the first time a page draws it (450–700 ms for the first frame of a blurred or
+newly animated layer). The script keeps Mesa's shader cache on the host
+(`~/.cache/astryx-webkit-shaders/<image>`, or `ASTRYX_WEBKIT_SHADER_CACHE`), so
+only the first run on a machine pays for them.
+
Evidence (storyboards, curve traces) is written to
`test-results/interaction-evidence/` (override with
`ASTRYX_INTERACTION_EVIDENCE`) and attached to the test report. It is never
@@ -49,13 +55,13 @@ committed: a pull request links or attaches it.
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scripts.ts` | Touch scripts as data: `tap`, `drag` (hold, rest), `flick` (exact final-move speed, velocity profiles), `holdThenDrag`, `pinch`, `rotate`, `path`, `merge` (a second finger mid-gesture), `releaseVelocity`, `toTouchEvents`. |
| `touch.ts` | `touchDriver(page).play(script)`: the script as real engine input. `installInputLog` / `readInputLog`: what the page received. |
-| `frames.ts` | `startScreencast` (timestamped composited frames), `markElement` (a marker chip or outline), `trackMarker` (marker box per frame), `frameStats`. |
+| `frames.ts` | `startScreencast` (timestamped composited frames), `markElement` (a marker chip or outline), `trackMarker` (marker box per frame), `sampleRects` (per-rAF rects), `frameStats`. |
| `busy.ts` | `blockMainThread(page, ms)` and `busyWindowProof(tracked, window)`: frames presented, frames that moved, longest stall, a verdict; `presented` adds the compositor's own frames. |
| `webInput.ts` | `webInputDriver(page)`: `wheel({x, y, dx, dy, ctrl})` and `pen(points)` as real engine input, for proofs across pointers (`spec:AST-100` FR4); `wheel` resolves once the page has the step; read back through the input log. |
| `trace.ts` | `traceRendering(page, body)`: main-thread counts of UpdateLayoutTree / Layout / PrePaint / Paint / Layerize / UpdateLayer / ScrollLayer / HitTest, `countBetween`, long tasks; `installLongTaskLog` / `readLongTasks`; `startCompositorTrace`: when the compositor presents. |
| `curves.ts` | `holdAnimations`, `armHold`, `seekAnimations`, `sampleCurve` (presented transform / opacity at any t), `waitForAnimation`, `watchAnimations`, `waitForRest`, `releaseAnimations`, `releaseAnimationLog` (frees what the recorder kept, before a heap or listener count). |
| `math.ts` | Oracles independent of the code under test: `cubicBezier`, `timingFunction`, `springStep` (SwiftUI response / dampingFraction, initial velocity), `projectDecay`, `rubberBand`, `parseTransform`, `compareCurve`, `timeline`. |
-| `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event): PNG contact sheet + JSON trace. |
+| `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event), `castStoryboard` (real screencast frames): PNG contact sheet + JSON trace. |
| `output.ts` | `evidenceDir`, `writeEvidence`: where evidence goes and how it is attached to the report. |
| `story.ts` | `phone`, `withStorybook`, `openStory`, `frames`, and the timer gate (`installTimerGate`, `freezeTimers`, `thawTimers`). |
@@ -71,6 +77,14 @@ dispatch latency, and the events go through the compositor's gesture detection
the limits below. Read positions a frame after a move: Chromium delivers
`pointermove` aligned to animation frames.
+**Stories, not Storybook.** `openStory` turns Storybook's highlight module off
+on every page it opens, and checks that its root element is absent. Left on,
+the module observes `#storybook-root` and, on every insertion, reads the
+computed style of every element on the page inside the inserting task (117
+reads and two forced style recalculations for one exit copy in a Button story),
+so a proof would measure Storybook as well as the component.
+`openStory(page, origin, id, args, {storybookHighlight: true})` keeps it on.
+
**Curve sampling.** Pause every running animation (CSS transitions and
animations are Web Animations too), seek to a time, read `getComputedStyle`:
the presented value at that exact time, independent of frame timing. Finished
@@ -152,7 +166,8 @@ every frame, so one late frame is not "1 of 1 moved"). Pass
`maxMedianFrameMs` (20 at 60 Hz) only in a proof that claims display rate.
`verdict.failures` names each miss with its numbers, and `summary` carries the
raw numbers for the report. Frames are judged in presentation order: a loaded
-machine can deliver them out of order.
+machine can deliver them out of order. `castStoryboard` turns the frames of the
+block into a filmstrip.
A just-started animation can reach the screen after the block began (50–100 ms
later in WebKit on a loaded machine). When the cast saw the element at rest
@@ -255,14 +270,14 @@ backstop, a cleanup render) would do so behind a held storyboard's back:
`installTimerGate` before the animation starts and `freezeTimers` while
photographing. The page can still cancel a held callback, a held frame callback
gets the time it actually runs, and the harness's own waits (its frame waits,
-`blockMainThread` with `during`) are not held.
+`blockMainThread` with `during`, `sampleRects`) are not held.
An animation that repeats forever does not set a storyboard's default length;
with nothing else to film, pass `times`. A touch filmstrip keeps the scripted
event `timeStamp`s (Chromium), but a handler that reads `performance.now()` or
`Date.now()` sees one event per photograph, about 100 ms apart, so a
clock-timed release flings slower when filmed than when played: a storyboard
cannot show such a gesture; record it in real time instead, with
-`startScreencast`. Chromium can return an empty copy of
+`startScreencast` and `castStoryboard`. Chromium can return an empty copy of
the page's surface right after it changes ("Unable to capture screenshot"); a
storyboard takes that photograph again two frames later, up to three attempts
in all, and lists each retake under `recaptures` in the trace.
diff --git a/internal/interaction-harness/scripts/webkit-container.mjs b/internal/interaction-harness/scripts/webkit-container.mjs
index 3f15bb2b0c..21cb7b4eb1 100644
--- a/internal/interaction-harness/scripts/webkit-container.mjs
+++ b/internal/interaction-harness/scripts/webkit-container.mjs
@@ -16,10 +16,19 @@
* docker daemon the container runs as the invoking user (rootless podman and
* rootless docker already map it), so the `test-results/` it writes into the
* mounted checkout stay the host user's and the next host run can clear them.
+ *
+ * WebKit composites through Mesa's software GL in the image, which compiles
+ * each shader variant the first time a page draws it: 450–700 ms for the
+ * first frame of a blurred or newly animated layer, long enough for a short
+ * animation to end before it is ever presented. Mesa caches what it compiles,
+ * and the cache is kept on the host (`~/.cache/astryx-webkit-shaders/<image>`,
+ * or `ASTRYX_WEBKIT_SHADER_CACHE`), so a run starts with every shader an
+ * earlier run compiled instead of paying for them in its first tests.
*/
import {spawnSync} from 'node:child_process';
import fs from 'node:fs';
import {createRequire} from 'node:module';
+import os from 'node:os';
import path from 'node:path';
import {fileURLToPath} from 'node:url';
@@ -42,6 +51,21 @@ if (engine == null) {
process.exit(1);
}
const args = process.argv.slice(2);
+// Where the container sees the host's shader cache (see the header).
+const containerShaderCache = '/astryx-shader-cache';
+// Resolved to an absolute path: a bare relative value would be mounted as a
+// named volume rather than a host directory. An empty value means the default.
+const shaderCache = process.env.ASTRYX_WEBKIT_SHADER_CACHE
+ ? path.resolve(process.env.ASTRYX_WEBKIT_SHADER_CACHE)
+ : path.join(
+ os.homedir(),
+ '.cache',
+ 'astryx-webkit-shaders',
+ image.replace(/[^\w.-]+/g, '_'),
+ );
+fs.mkdirSync(shaderCache, {recursive: true});
+// The harness self-test reads this marker inside the container.
+fs.writeFileSync(path.join(shaderCache, 'ASTRYX-SHADER-CACHE'), `${image}\n`);
const env = [
'ASTRYX_STORYBOOK_DIR',
'ASTRYX_INTERACTION_EVIDENCE',
@@ -84,6 +108,10 @@ const command = [
'--pids-limit=-1',
'-v',
`${root}:${root}`,
+ '-v',
+ `${shaderCache}:${containerShaderCache}`,
+ '-e',
+ `MESA_SHADER_CACHE_DIR=${containerShaderCache}`,
'-w',
root,
...env,
diff --git a/internal/interaction-harness/src/frames.ts b/internal/interaction-harness/src/frames.ts
index c61ea7e171..83841d259c 100644
--- a/internal/interaction-harness/src/frames.ts
+++ b/internal/interaction-harness/src/frames.ts
@@ -7,11 +7,12 @@
* `Screencast` domain)
* @output `startScreencast` (timestamped composited frames), `markElement`
* (a marker outline the frames can find), `trackMarker` (per-frame marker
- * box), `frameStats`.
+ * box), `sampleRects` (per-rAF rect sampling), `frameStats`.
* @position Frame evidence for motion proofs. The screencast is captured off
* the page's main thread, from what the compositor presents, so it keeps
* delivering frames while a script blocks the main thread: that is what makes
- * the busy-main-thread proof possible.
+ * the busy-main-thread proof possible. Rect sampling is the cheap path when
+ * the main thread is free.
*
* Marker tracking needs no image library: frames are decoded and scanned in a
* scratch page of the same browser (createImageBitmap + canvas), so the
@@ -399,6 +400,71 @@ export async function trackMarker(
}
}
+export interface RectSample {
+ /** Epoch ms of the rAF callback. */
+ readonly t: number;
+ readonly box: Box | null;
+ readonly transform: string;
+ readonly opacity: number;
+}
+
+/**
+ * Samples the element's rect, transform and opacity on every animation frame
+ * for `durationMs`. Main-thread sampling: it stops while the main thread is
+ * blocked, which is the point of the screencast path above.
+ */
+export async function sampleRects(
+ locator: Locator,
+ durationMs: number,
+): Promise<RectSample[]> {
+ return locator.evaluate(
+ (element, duration) =>
+ new Promise<
+ {
+ t: number;
+ box: {x: number; y: number; width: number; height: number} | null;
+ transform: string;
+ opacity: number;
+ }[]
+ >(resolve => {
+ // The harness's own waits run on the real timers: a frozen timer gate
+ // (story.ts) holds only the page's.
+ const gate = (
+ window as unknown as {
+ __harnessTimerGate?: {
+ frame?: typeof requestAnimationFrame;
+ };
+ }
+ ).__harnessTimerGate;
+ const frame = gate?.frame ?? window.requestAnimationFrame.bind(window);
+ const samples: {
+ t: number;
+ box: {x: number; y: number; width: number; height: number} | null;
+ transform: string;
+ opacity: number;
+ }[] = [];
+ const end = performance.now() + duration;
+ const tick = (now: number) => {
+ const rect = element.getBoundingClientRect();
+ const style = getComputedStyle(element);
+ samples.push({
+ t: performance.timeOrigin + now,
+ box: {x: rect.x, y: rect.y, width: rect.width, height: rect.height},
+ transform: style.transform,
+ opacity: Number(style.opacity),
+ });
+ if (now < end) {
+ frame(tick);
+ } else {
+ resolve(samples);
+ }
+ };
+ frame(tick);
+ }),
+ durationMs,
+ );
+}
+
export interface FrameStats {
readonly frames: number;
/** Frames whose tracked value differs from the frame before. */
diff --git a/internal/interaction-harness/src/harness.interaction.browser.spec.ts b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
index 775c631c06..4b981b9079 100644
--- a/internal/interaction-harness/src/harness.interaction.browser.spec.ts
+++ b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
@@ -29,18 +29,22 @@
* two frames later; a touch filmstrip keeps scripted event times while a
* handler clock sees its pacing; the timer gate holds a component timer
* until the filmstrip is done, drops callbacks the page cancels, stamps held
- * frames with the time they run, and leaves the harness its own frames.
+ * frames with the time they run, and leaves the harness its own frames; a
+ * cast filmstrip keeps exactly the presented frames of its window, labelled
+ * from its zero, as PNG.
* @position The harness's own test. A component spec trusts these
* instruments; this file is why it may.
*/
import fs from 'node:fs';
+import path from 'node:path';
import {expect, test, type Page} from '@playwright/test';
import type {CastFrame, TrackedFrame} from './frames';
import {
armHold,
animationStoryboard,
+ castStoryboard,
blockMainThread,
busyWindowProof,
compareCurve,
@@ -68,6 +72,7 @@ import {
releaseAnimations,
retakeOnWheelPacing,
sampleCurve,
+ sampleRects,
seekAnimations,
springStep,
startCompositorTrace,
@@ -1116,6 +1121,83 @@ test.describe('frame evidence', () => {
expect(proof.monotonic).toBe(true);
});
+ test('in the WebKit container, compiled shaders outlive the run', async ({
+ browser,
+ }, testInfo) => {
+ // The official Playwright image keeps its browsers here.
+ test.skip(
+ testInfo.project.name !== 'webkit' ||
+ process.env.PLAYWRIGHT_BROWSERS_PATH !== '/ms-playwright',
+ 'runs inside webkit-container.mjs',
+ );
+ // A cache inside the container dies with it: every run then compiled each
+ // shader again, 450–700 ms in the first frame of a blurred layer.
+ const dir = process.env.MESA_SHADER_CACHE_DIR;
+ expect(dir, 'MESA_SHADER_CACHE_DIR names the host-kept cache').toBeTruthy();
+ // Written on the host by the script before the container starts.
+ expect(
+ fs.readFileSync(path.join(dir ?? '', 'ASTRYX-SHADER-CACHE'), 'utf8'),
+ ).toContain('playwright');
+ // Mesa writes what it compiles there once a page draws a blurred layer.
+ // Earlier runs have filled the cache already, so this run's browser gets
+ // an empty directory beside them, in the same host-kept mount, and the
+ // test watches its own writes arrive.
+ const fresh = fs.mkdtempSync(path.join(dir ?? '', 'self-test-'));
+ try {
+ const own = await browser.browserType().launch({
+ env: {...process.env, MESA_SHADER_CACHE_DIR: fresh},
+ });
+ try {
+ const page = await own.newPage();
+ await page.setContent(
+ '<div id="glass" style="width:120px;height:80px;backdrop-filter:blur(16px) saturate(180%)"></div>',
+ );
+ await page.evaluate(
+ () =>
+ document
+ .getElementById('glass')
+ ?.animate([{scale: '1'}, {scale: '0.9'}], {duration: 200})
+ .finished,
+ );
+ } finally {
+ await own.close();
+ }
+ expect(fs.readdirSync(fresh, {recursive: true})).not.toEqual([]);
+ } finally {
+ fs.rmSync(fresh, {recursive: true, force: true});
+ }
+ });
+
+ test('rect sampling reads one rect per animation frame while the main thread is free', async ({
+ page,
+ browserName,
+ }) => {
+ await fixture(page);
+ const box = page.locator('#box');
+ await box.evaluate(el => {
+ el.animate(
+ [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}],
+ {duration: 600, easing: 'linear'},
+ );
+ });
+ const samples = await sampleRects(box, 400);
+ const xs = samples.map(s => s.box?.x ?? NaN);
+ const gaps = samples
+ .slice(1)
+ .map((s, i) => s.t - samples[i].t)
+ .sort((a, b) => a - b);
+ const cadence = gaps[Math.floor(gaps.length / 2)];
+ expect(cadence).toBeLessThan(34);
+ expect(samples.length).toBeGreaterThanOrEqual(10);
+ if (browserName === 'chromium') {
+ // One sample per presented frame at 60 Hz. (Headless WebKit's animation
+ // frames are irregular, so only the cadence is checked there.)
+ expect(samples.length).toBeGreaterThanOrEqual(20);
+ }
+ expect(xs.every((x, i) => i === 0 || x >= xs[i - 1])).toBe(true);
+ expect(xs.at(-1) ?? 0).toBeGreaterThan(xs[0] + 80);
+ });
+
test('the trace counts no Layout for a transform animation and one per frame for `top`', async ({
page,
browserName,
@@ -3775,7 +3857,7 @@ test.describe('storyboards', {tag: '@storyboard'}, () => {
}
});
- test("the harness's own waits run on the real timers: a frozen timer gate does not hold blockMainThread's block", async ({
+ test("the harness's own waits run on the real timers: a frozen timer gate holds neither blockMainThread's block nor sampleRects", async ({
page,
}) => {
await fixture(page);
@@ -3799,6 +3881,7 @@ test.describe('storyboards', {tag: '@storyboard'}, () => {
expect(
await within(blockMainThread(page, 100, {during: box, timeout: 2_000})),
).toBe('settled');
+ expect(await within(sampleRects(box, 200))).toBe('settled');
await thawTimers(page);
});
@@ -4005,4 +4088,58 @@ test.describe('storyboards', {tag: '@storyboard'}, () => {
expect(state).toMatchObject({timer: false, frame: false});
expect(state.heldFor).toBeGreaterThanOrEqual(140);
});
+
+ test('a cast filmstrip keeps exactly the presented frames of its window, labelled from its zero', async ({
+ page,
+ }, testInfo) => {
+ await fixture(page);
+ const box = page.locator('#box');
+ // JPEG frames (WebKit's are always JPEG) take the conversion path.
+ const cast = await startScreencast(page, {format: 'jpeg'});
+ await box.evaluate(el =>
+ el.animate(
+ [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}],
+ {duration: 600},
+ ),
+ );
+ await page.waitForTimeout(500);
+ const presented = await cast.stop();
+ expect(presented.length).toBeGreaterThan(8);
+ // A window of the middle frames, zeroed on one of them.
+ const zero = presented[2].t;
+ const from = presented[3].t;
+ const to = presented[presented.length - 3].t;
+ const inside = presented.filter(f => f.t >= from && f.t <= to);
+ const seen: number[] = [];
+ const film = await castStoryboard(page, presented, {
+ name: 'harness-self-test-cast',
+ zero,
+ from,
+ to,
+ data: t => {
+ seen.push(t);
+ return {t};
+ },
+ });
+ expect(film.frames.map(f => f.t)).toEqual(inside.map(f => f.t - zero));
+ expect(film.frames.map(f => f.label)).toEqual(
+ inside.map(f => `t=${(f.t - zero).toFixed(1)} ms`),
+ );
+ expect(seen).toEqual(inside.map(f => f.t));
+ // Every cell is a PNG, whatever the screencast delivered.
+ for (const frame of film.frames) {
+ expect([...frame.png.subarray(0, 4)]).toEqual([0x89, 0x50, 0x4e, 0x47]);
+ }
+ const files = await film.write(testInfo);
+ expect(files['sheet.png']).toMatch(/sheet\.png$/);
+ // A window with no presented frame in it films nothing.
+ const empty = await castStoryboard(page, presented, {
+ name: 'harness-self-test-cast-empty',
+ zero,
+ from: to + 1,
+ to: to + 2,
+ });
+ expect(empty.frames).toEqual([]);
+ await expect(empty.write(testInfo)).rejects.toThrow(/no frames/);
+ });
});
diff --git a/internal/interaction-harness/src/story.ts b/internal/interaction-harness/src/story.ts
index 5a1f2237c6..627969a663 100644
--- a/internal/interaction-harness/src/story.ts
+++ b/internal/interaction-harness/src/story.ts
@@ -6,7 +6,8 @@
* static server over a built Storybook
* @output `phone` (the touch-phone browser context every interaction proof
* runs in), `withStorybook` (serve the built Storybook for a spec file), and
- * `openStory` (load one story and wait until it has rendered).
+ * `openStory` (load one story, with Storybook's highlight module off, and
+ * wait until it has rendered).
* @position Stories are the reproduction path, as for every other browser
* check in this repository: a scenario used for verification is checked in,
* not built in a one-off page.
@@ -59,13 +60,47 @@ export function withStorybook(test: TestType<any, any>): () => string {
};
}
-/** Loads a story in the page and waits for its first render, its fonts and two frames. */
+/** Pages whose Storybook highlight module `openStory` turned off. */
+const highlightOff = new WeakSet<Page>();
+
+export interface OpenStoryOptions {
+ /**
+ * Keeps Storybook's highlight module on. It is off by default: when on, it
+ * observes `#storybook-root` and, on every insertion, reads the computed
+ * style of every element on the page inside the inserting task, so a proof
+ * would measure Storybook as well as the component.
+ */
+ readonly storybookHighlight?: boolean;
+}
+
+/**
+ * Loads a story in the page and waits for its first render, its fonts and two
+ * frames. Storybook's highlight module is off (see `OpenStoryOptions`), and
+ * that is checked: its root element must be absent once the story renders.
+ */
export async function openStory(
page: Page,
origin: string,
storyId: string,
args?: Readonly<Record<string, string>>,
+ {storybookHighlight = false}: OpenStoryOptions = {},
): Promise<void> {
+ if (storybookHighlight) {
+ if (highlightOff.has(page)) {
+ throw new Error(
+ "openStory: this page already turned Storybook's highlight module off; open the story in a new page to keep it on",
+ );
+ }
+ } else if (!highlightOff.has(page)) {
+ highlightOff.add(page);
+ // Storybook's highlight module returns at once when this is set before
+ // its preview runs.
+ await page.addInitScript(() => {
+ (
+ globalThis as unknown as {__STORYBOOK_HIGHLIGHT_INITIALIZED?: boolean}
+ ).__STORYBOOK_HIGHLIGHT_INITIALIZED = true;
+ });
+ }
const params = new URLSearchParams({id: storyId, viewMode: 'story'});
if (args != null) {
params.set(
@@ -86,6 +121,14 @@ export async function openStory(
requestAnimationFrame(() => requestAnimationFrame(() => resolve())),
);
});
+ if (
+ !storybookHighlight &&
+ (await page.locator('#storybook-highlights-root').count()) > 0
+ ) {
+ throw new Error(
+ "openStory: Storybook's highlight module is on (#storybook-highlights-root exists), so every insertion would also read the style of every element on the page; this Storybook may no longer read __STORYBOOK_HIGHLIGHT_INITIALIZED",
+ );
+ }
}
/** Resolves after `count` animation frames have been presented in the page. */
diff --git a/internal/interaction-harness/src/storyboard.ts b/internal/interaction-harness/src/storyboard.ts
index 5b7208d309..936c427138 100644
--- a/internal/interaction-harness/src/storyboard.ts
+++ b/internal/interaction-harness/src/storyboard.ts
@@ -346,7 +346,8 @@ export interface TouchStoryboardOptions extends StoryboardOptions {
* photograph per event, about 100 ms between moves instead of the script's
* 16.7. Its release velocity is that much lower, so a flick that dismisses
* when played can settle back when filmed: a storyboard cannot show such a
- * gesture; record it in real time instead.
+ * gesture; record it in real time instead (`startScreencast`,
+ * `castStoryboard`).
*/
export async function touchStoryboard(
page: Page,
@@ -374,3 +375,65 @@ export async function touchStoryboard(
});
return storyboard;
}
+
+/**
+ * A filmstrip of screencast frames (real presented frames, e.g. the frames of
+ * a busy-main-thread window), labelled with their time relative to `zero`.
+ * Evidence for what the compositor presented, not a deterministic storyboard.
+ */
+export async function castStoryboard(
+ page: Page,
+ frames: readonly {
+ readonly t: number;
+ readonly data: string;
+ readonly format: 'png' | 'jpeg';
+ }[],
+ // A cast is already the screen's frames: no clip is applied to them.
+ options: Omit<StoryboardOptions, 'clip'> & {
+ readonly zero: number;
+ readonly from?: number;
+ readonly to?: number;
+ readonly data?: (t: number) => Readonly<Record<string, unknown>>;
+ },
+): Promise<Storyboard> {
+ const storyboard = new Storyboard(page, options);
+ for (const frame of frames) {
+ if (options.from != null && frame.t < options.from) {
+ continue;
+ }
+ if (options.to != null && frame.t > options.to) {
+ continue;
+ }
+ const png =
+ frame.format === 'png'
+ ? Buffer.from(frame.data, 'base64')
+ : await toPng(page, frame.data);
+ const t = frame.t - options.zero;
+ storyboard.frames.push({
+ label: `t=${t.toFixed(1)} ms`,
+ t,
+ data: options.data?.(frame.t) ?? {},
+ png,
+ });
+ }
+ return storyboard;
+}
+
+async function toPng(page: Page, jpegBase64: string): Promise<Buffer> {
+ const scratch = await page.context().newPage();
+ try {
+ const url = await scratch.evaluate(async data => {
+ const bitmap = await createImageBitmap(
+ await (await fetch(`data:image/jpeg;base64,${data}`)).blob(),
+ );
+ const canvas = document.createElement('canvas');
+ canvas.width = bitmap.width;
+ canvas.height = bitmap.height;
+ canvas.getContext('2d')?.drawImage(bitmap, 0, 0);
+ return canvas.toDataURL('image/png');
+ }, jpegBase64);
+ return Buffer.from(url.slice(url.indexOf(',') + 1), 'base64');
+ } finally {
+ await scratch.close();
+ }
+}
diff --git a/internal/interaction-harness/src/storybook.interaction.browser.spec.ts b/internal/interaction-harness/src/storybook.interaction.browser.spec.ts
new file mode 100644
index 0000000000..a711feb62c
--- /dev/null
+++ b/internal/interaction-harness/src/storybook.interaction.browser.spec.ts
@@ -0,0 +1,104 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file storybook.interaction.browser.spec.ts
+ * @input The built Storybook (`withStorybook`) and one of its stories
+ * @output `openStory` turns Storybook's highlight module off, so an insertion
+ * in a story carries no `getComputedStyle` scan of the page; with the module
+ * on (the control), every insertion carries one. Both engines; Chromium's
+ * trace also counts the style recalculations the scan forces
+ * @position Self-test for story.ts; it needs the built Storybook, as every
+ * component proof does
+ */
+
+import {expect, test, type Page} from '@playwright/test';
+import {openStory, withStorybook} from './story';
+import {countBetween, traceRendering} from './trace';
+
+const storybook = withStorybook(test);
+
+/** Counts the page's `getComputedStyle` calls, from before Storybook loads. */
+function countStyleReads(): void {
+ const w = window as unknown as {__styleReads: number};
+ w.__styleReads = 0;
+ const read = window.getComputedStyle.bind(window);
+ window.getComputedStyle = ((element: Element, pseudo?: string | null) => {
+ w.__styleReads += 1;
+ return read(element, pseudo);
+ }) as typeof window.getComputedStyle;
+}
+
+/**
+ * Inserts an exit copy of the story's first element into `#storybook-root`,
+ * as an exit animation's copy is, lets the insertion's mutation microtasks run,
+ * and reports what ran between its marks.
+ */
+async function insertCopy(page: Page) {
+ return page.evaluate(async () => {
+ const w = window as unknown as {__styleReads: number};
+ const root = document.getElementById('storybook-root') as HTMLElement;
+ const reads = w.__styleReads;
+ const from =
+ performance.timeOrigin + performance.mark('copy:start').startTime;
+ const copy = (root.firstElementChild as HTMLElement).cloneNode(
+ true,
+ ) as HTMLElement;
+ copy.setAttribute('aria-hidden', 'true');
+ root.appendChild(copy);
+ // A MutationObserver's callback is a microtask the insertion queued:
+ // these run after it.
+ await Promise.resolve();
+ await Promise.resolve();
+ const to = performance.timeOrigin + performance.mark('copy:end').startTime;
+ const scanned = w.__styleReads - reads;
+ const elements = document.body.querySelectorAll('*').length;
+ copy.remove();
+ return {from, to, scanned, elements};
+ });
+}
+
+test("openStory turns Storybook's highlight module off: an insertion in a story carries no getComputedStyle scan of the page", async ({
+ page,
+ browserName,
+}) => {
+ // The insertion's reads in every engine, and in Chromium (whose trace the
+ // harness reads) the style recalculations between its marks.
+ const measure = async (p: Page) => {
+ if (browserName !== 'chromium') {
+ return {copy: await insertCopy(p), styleRecalc: null};
+ }
+ let copy = {from: 0, to: 0, scanned: 0, elements: 0};
+ const trace = await traceRendering(p, async () => {
+ copy = await insertCopy(p);
+ });
+ return {
+ copy,
+ styleRecalc:
+ countBetween(trace, copy.from, copy.to).UpdateLayoutTree ?? 0,
+ };
+ };
+ // The harness default.
+ await page.addInitScript(countStyleReads);
+ await openStory(page, storybook(), 'core-button--primary');
+ expect(await page.locator('#storybook-highlights-root').count()).toBe(0);
+ const off = await measure(page);
+ expect(off.copy.scanned).toBe(0);
+ if (off.styleRecalc != null) {
+ expect(off.styleRecalc).toBe(0);
+ }
+
+ // Control: Storybook's module kept on reads the style of every element on
+ // the page inside the inserting task.
+ const control = await page.context().newPage();
+ await control.addInitScript(countStyleReads);
+ await openStory(control, storybook(), 'core-button--primary', undefined, {
+ storybookHighlight: true,
+ });
+ expect(await control.locator('#storybook-highlights-root').count()).toBe(1);
+ const on = await measure(control);
+ expect(on.copy.scanned).toBeGreaterThanOrEqual(on.copy.elements);
+ if (on.styleRecalc != null) {
+ expect(on.styleRecalc).toBeGreaterThan(0);
+ }
+ await control.close();
+});