Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

test(interaction): deterministic storyboards (#8348)

Christopher Chedeaud07518f92b53
diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.md
index 95931b9147..31c5149c2d 100644
--- a/internal/interaction-harness/README.md
+++ b/internal/interaction-harness/README.md
@@ -1,7 +1,7 @@
# @astryxdesign/interaction-harness
Playwright instruments for touch interaction and animation proofs: real touch
-input and curve sampling. **Internal and unpublished.** Specs live
+input, curve sampling and storyboards. **Internal and unpublished.** Specs live
next to their component as `*.interaction.browser.spec.ts` and run under
[`playwright.interaction.config.ts`](../../playwright.interaction.config.ts) on
a phone-sized touch screen (402 × 714 CSS px page, 402 × 874 screen, 3 device
@@ -18,17 +18,35 @@ pnpm storybook:build # component specs read stories from th
pnpm exec playwright install chromium webkit
pnpm test:interaction --project chromium # everything, Chromium
pnpm test:interaction <spec file> # one spec, both engines
+pnpm storyboards # only the tests tagged @storyboard
```
+On Linux distributions whose system libraries are older than Playwright's
+WebKit build needs, run the WebKit project inside the official Playwright
+image (podman or docker; the repository is mounted at the same path, and
+under a rootful docker daemon the container runs as your user so its
+`test-results/` stay yours):
+
+```sh
+node internal/interaction-harness/scripts/webkit-container.mjs [playwright test args…]
+```
+
+Evidence (storyboards, curve traces) is written to
+`test-results/interaction-evidence/` (override with
+`ASTRYX_INTERACTION_EVIDENCE`) and attached to the test report. It is never
+committed: a pull request links or attaches it.
+
## Instruments
-| Module | What it gives a spec |
-| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `scripts.ts` | Touch scripts as data: `tap`, `drag` (hold, rest), `flick` (exact final-move speed, velocity profiles), `holdThenDrag`, `pinch`, `rotate`, `path`, `merge` (a second finger mid-gesture), `releaseVelocity`, `toTouchEvents`. |
-| `touch.ts` | `touchDriver(page).play(script)`: the script as real engine input. `installInputLog` / `readInputLog`: what the page received. |
-| `curves.ts` | `holdAnimations`, `seekAnimations`, `sampleCurve` (presented transform / opacity at any t), `waitForAnimation`, `watchAnimations`, `waitForRest`, `releaseAnimations`. |
-| `math.ts` | Oracles independent of the code under test: `cubicBezier`, `timingFunction`, `springStep` (SwiftUI response / dampingFraction, initial velocity), `projectDecay`, `rubberBand`, `parseTransform`, `compareCurve`, `timeline`. |
-| `story.ts` | `phone`, `withStorybook`, `openStory`, `frames`. |
+| Module | What it gives a spec |
+| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `scripts.ts` | Touch scripts as data: `tap`, `drag` (hold, rest), `flick` (exact final-move speed, velocity profiles), `holdThenDrag`, `pinch`, `rotate`, `path`, `merge` (a second finger mid-gesture), `releaseVelocity`, `toTouchEvents`. |
+| `touch.ts` | `touchDriver(page).play(script)`: the script as real engine input. `installInputLog` / `readInputLog`: what the page received. |
+| `curves.ts` | `holdAnimations`, `seekAnimations`, `sampleCurve` (presented transform / opacity at any t), `waitForAnimation`, `watchAnimations`, `waitForRest`, `releaseAnimations`. |
+| `math.ts` | Oracles independent of the code under test: `cubicBezier`, `timingFunction`, `springStep` (SwiftUI response / dampingFraction, initial velocity), `projectDecay`, `rubberBand`, `parseTransform`, `compareCurve`, `timeline`. |
+| `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event): PNG contact sheet + JSON trace. |
+| `output.ts` | `evidenceDir`, `writeEvidence`: where evidence goes and how it is attached to the report. |
+| `story.ts` | `phone`, `withStorybook`, `openStory`, `frames`, and the timer gate (`installTimerGate`, `freezeTimers`, `thawTimers`). |
## The proofs
@@ -54,8 +72,8 @@ budget (for a spring baked into `linear()` stops of `h` ms, chords err by up to
inside it, the last frame it presents, and the sample says so (`presentedAt`,
where oracles are read). An animation whose keyframes end in a run of
identical values (an end value held so a later interruption can read it)
-ends where that run starts: `curveEndMs` and `waitForAnimation`'s interval
-stop there, and `waitForRest`, which judges the
+ends where that run starts: `curveEndMs`, `waitForAnimation`'s interval and
+the default storyboard times stop there, and `waitForRest`, which judges the
presented `transform`, `translate`, `rotate`, `scale` and `opacity` rather
than `playState`, does not wait the hold out. That holds for an animation that
plays forward under a linear or cubic-bézier effect easing (the time the run
@@ -79,6 +97,24 @@ set): a short entrance no longer races the first look on a cold page. When the
action may be the page's first contact with the harness, take
`watchAnimations(locator)` before it and read `watch.started()` after.
+**Storyboards.** Deterministic: frames come from seeking held animations or
+from stepping scripted input, never from wall-clock capture. The JSON trace
+carries each frame's measurements and the spec asserts on it; the sheet is
+evidence. A component that finishes an animation from a timer (a transitionend
+backstop, a cleanup render) would do so behind a held storyboard's back:
+`installTimerGate` before the animation starts and `freezeTimers` while
+photographing. The page can still cancel a held callback, a held frame callback
+gets the time it actually runs, and the harness's own frame waits are not held.
+An animation that repeats forever does not set a storyboard's default length;
+with nothing else to film, pass `times`. A touch filmstrip keeps the scripted
+event `timeStamp`s (Chromium), but a handler that reads `performance.now()` or
+`Date.now()` sees one event per photograph, about 100 ms apart, so a
+clock-timed release flings slower when filmed than when played: a storyboard
+cannot show such a gesture; record it in real time instead. Chromium can return
+an empty copy of the page's surface right after it changes ("Unable to capture
+screenshot"); a storyboard takes that photograph again two frames later, up to
+three attempts in all, and lists each retake under `recaptures` in the trace.
+
## Engine limits (measured with Playwright 1.61)
| | Chromium 149 | WebKit 26.5 (Linux build) |
@@ -89,7 +125,7 @@ action may be the page's first contact with the harness, take
| Native scroll, fling, pinch-zoom from injected touches | yes | no |
| Lifting one finger while others stay down | yes | pointer events right, but that `touchend` lists no remaining `touches`: skip with `PARTIAL_LIFT_SKIP` |
| Click synthesised from a touch | yes | only from `driver.tap` (WebKit's tap gesture), not from a raw touch sequence |
-| Curve sampling, input log | yes | yes |
+| Curve sampling, storyboards, input log | yes | yes |
What WebKit cannot show here — native scroll hand-off, flings, a finger
lifting while another stays down, and anything iOS-only — needs Mobile Safari
diff --git a/internal/interaction-harness/package.json b/internal/interaction-harness/package.json
index de2f8e87aa..40cccb26ca 100644
--- a/internal/interaction-harness/package.json
+++ b/internal/interaction-harness/package.json
@@ -3,7 +3,7 @@
"version": "0.0.0",
"private": true,
"type": "module",
- "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, curve sampling). Never published.",
+ "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, curve sampling, storyboards). Never published.",
"main": "./src/index.ts",
"exports": {
".": "./src/index.ts",
diff --git a/internal/interaction-harness/scripts/webkit-container.mjs b/internal/interaction-harness/scripts/webkit-container.mjs
new file mode 100644
index 0000000000..cb20fd44aa
--- /dev/null
+++ b/internal/interaction-harness/scripts/webkit-container.mjs
@@ -0,0 +1,97 @@
+#!/usr/bin/env node
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * Runs the interaction proofs' WebKit project inside the official Playwright
+ * container image, for hosts where WebKit cannot launch (Playwright's Linux
+ * WebKit build needs newer system libraries than some distributions ship).
+ *
+ * node internal/interaction-harness/scripts/webkit-container.mjs [playwright test args…]
+ *
+ * The repository is mounted at the same path, so node_modules, the built
+ * Storybook and the evidence directory are the host's own. The image tag
+ * matches the installed playwright, so its browsers match the client.
+ * Uses podman when present, else docker. `--network=host` lets the specs'
+ * local Storybook server and the browser reach each other. Under a rootful
+ * docker daemon the container runs as the invoking user (rootless podman and
+ * rootless docker already map it), so the `test-results/` it writes into the
+ * mounted checkout stay the host user's and the next host run can clear them.
+ */
+import {spawnSync} from 'node:child_process';
+import fs from 'node:fs';
+import {createRequire} from 'node:module';
+import path from 'node:path';
+import {fileURLToPath} from 'node:url';
+
+const root = path.resolve(
+ path.dirname(fileURLToPath(import.meta.url)),
+ '../../..',
+);
+const require = createRequire(path.join(root, 'package.json'));
+const {version} = JSON.parse(
+ fs.readFileSync(require.resolve('playwright/package.json'), 'utf8'),
+);
+const image =
+ process.env.PLAYWRIGHT_IMAGE ??
+ `mcr.microsoft.com/playwright:v${version}-noble`;
+const engine = ['podman', 'docker'].find(
+ cmd => spawnSync(cmd, ['--version'], {stdio: 'ignore'}).status === 0,
+);
+if (engine == null) {
+ console.error('webkit-container: neither podman nor docker is installed');
+ process.exit(1);
+}
+const args = process.argv.slice(2);
+const env = [
+ 'ASTRYX_STORYBOOK_DIR',
+ 'ASTRYX_INTERACTION_EVIDENCE',
+ 'INTERACTION_WORKERS',
+ 'CI',
+]
+ .filter(name => process.env[name] != null)
+ .flatMap(name => ['-e', `${name}=${process.env[name]}`]);
+// A rootful Docker daemon runs the image as root unless told otherwise; the
+// files it writes into the bind mount would then be root's. Rootless Docker
+// maps the container's root to the invoking user (as podman does), and any
+// other container uid to a subordinate host uid that cannot write the
+// checkout, so it gets no `--user`.
+const rootlessDocker =
+ engine === 'docker' &&
+ /name=rootless/.test(
+ spawnSync('docker', ['info', '--format', '{{json .SecurityOptions}}'], {
+ encoding: 'utf8',
+ }).stdout ?? '',
+ );
+const user =
+ engine === 'docker' && !rootlessDocker && typeof process.getuid === 'function'
+ ? [
+ '--user',
+ `${process.getuid()}:${process.getgid?.() ?? process.getuid()}`,
+ '-e',
+ 'HOME=/tmp',
+ ]
+ : [];
+const command = [
+ 'run',
+ '--rm',
+ ...user,
+ '--network=host',
+ '--ipc=host',
+ '-v',
+ `${root}:${root}`,
+ '-w',
+ root,
+ ...env,
+ image,
+ 'node',
+ path.join(root, 'node_modules/@playwright/test/cli.js'),
+ 'test',
+ '--config',
+ 'playwright.interaction.config.ts',
+ '--project',
+ 'webkit',
+ ...args,
+];
+console.log(`${engine} ${command.join(' ')}`);
+const result = spawnSync(engine, command, {stdio: 'inherit'});
+process.exit(result.status ?? 1);
diff --git a/internal/interaction-harness/src/curves.ts b/internal/interaction-harness/src/curves.ts
index f3b03ffe45..69b9908066 100644
--- a/internal/interaction-harness/src/curves.ts
+++ b/internal/interaction-harness/src/curves.ts
@@ -17,6 +17,8 @@
* fixed time, independent of frame timing, so a curve can be compared with
* the UIKit reference to a fraction of a pixel. A paused animation's effect
* is applied at its current time and `getComputedStyle` reads it.
+ *
+ * SYNC: Keep ./storyboard.ts using these for its animation filmstrips.
*/
import type {Locator, Page} from '@playwright/test';
@@ -39,7 +41,14 @@ function animationRecorder(): void {
if (w.__harnessAnimations) {
return;
}
- const frame = window.requestAnimationFrame.bind(window);
+ // The page's real frames, even when a timer gate already holds its callbacks.
+ const frame = (
+ (
+ window as unknown as {
+ __harnessTimerGate?: {frame: typeof requestAnimationFrame};
+ }
+ ).__harnessTimerGate?.frame ?? window.requestAnimationFrame
+ ).bind(window);
const seen = new WeakSet<Animation>();
const log: {
animation: Animation;
@@ -1148,7 +1157,13 @@ export async function waitForRest(
backdrop,
].join(' | ');
};
- const frame = requestAnimationFrame;
+ // The page's real frames, even while a timer gate holds its callbacks.
+ const frame =
+ (
+ window as unknown as {
+ __harnessTimerGate?: {frame: typeof requestAnimationFrame};
+ }
+ ).__harnessTimerGate?.frame ?? requestAnimationFrame;
let last = read();
let still = 0;
const tick = () => {
diff --git a/internal/interaction-harness/src/harness.interaction.browser.spec.ts b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
index 7a0df1c845..7e47af95b2 100644
--- a/internal/interaction-harness/src/harness.interaction.browser.spec.ts
+++ b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
@@ -13,20 +13,31 @@
* an end reports the time it was read at; holding takes only running
* document-timeline animations, each at its own offset, holds each where it
* was read, adds a second hold to the held set without moving the first
- * hold's times, drops one the page cancels, and releases only them.
+ * hold's times, drops one the page cancels, and releases only them;
+ * storyboards write a sheet and a trace that follows the curve, an endless
+ * animation does not set their length, and a photograph Chromium returns
+ * empty is taken again two frames later; a touch filmstrip keeps scripted
+ * event times while a handler clock sees its pacing; the timer gate holds a
+ * component timer until the filmstrip is done, drops callbacks the page
+ * cancels, stamps held frames with the time they run, and leaves the
+ * harness its own frames.
* @position The harness's own test. A component spec trusts these
* instruments; this file is why it may.
*/
import {expect, test, type Page} from '@playwright/test';
import {
+ animationStoryboard,
compareCurve,
cubicBezier,
curveOf,
drag,
flick,
+ frames,
+ freezeTimers,
holdAnimations,
installInputLog,
+ installTimerGate,
merge,
parseTransform,
PARTIAL_LIFT_SKIP,
@@ -36,9 +47,12 @@ import {
sampleCurve,
seekAnimations,
springStep,
+ Storyboard,
tap,
+ thawTimers,
timeline,
touchDriver,
+ touchStoryboard,
waitForAnimation,
waitForRest,
watchAnimations,
@@ -1253,3 +1267,351 @@ test.describe('animation state', () => {
expect(end.transform.scaleY).toBeCloseTo(1.5, 3);
});
});
+
+test.describe('storyboards', {tag: '@storyboard'}, () => {
+ test('an animation filmstrip records every frame and its trace follows the curve', async ({
+ page,
+ }, testInfo) => {
+ await fixture(page);
+ const box = page.locator('#box');
+ await box.evaluate(el =>
+ el.animate(
+ [{transform: 'translateX(0)'}, {transform: 'translateX(240px)'}],
+ {duration: 200, easing: 'ease-out'},
+ ),
+ );
+ const {storyboard} = await animationStoryboard(page, {
+ name: 'harness-self-test-animation',
+ clip: {x: 0, y: 340, width: 402, height: 120},
+ measure: async () => ({
+ x: await box.evaluate(el => el.getBoundingClientRect().x),
+ }),
+ columns: 4,
+ });
+ const files = await storyboard.write(testInfo);
+ const xs = storyboard.frames.map(f => f.data.x as number);
+ expect(xs).toHaveLength(13); // t = 0, 16.7 … 200
+ expect(xs[0]).toBeCloseTo(20, 1);
+ expect(xs.at(-1)).toBeCloseTo(260, 1);
+ expect(xs.every((x, i) => i === 0 || x >= xs[i - 1])).toBe(true);
+ expect(files['sheet.png']).toMatch(/sheet\.png$/);
+ });
+
+ test('a capture Chromium returns empty is taken again two frames later', async ({
+ page,
+ }) => {
+ await fixture(page);
+ // Counts the page's frames, so the wait before the second capture shows.
+ await page.evaluate(() => {
+ const w = window as unknown as {__frames: number};
+ w.__frames = 0;
+ const tick = () => {
+ w.__frames += 1;
+ requestAnimationFrame(tick);
+ };
+ requestAnimationFrame(tick);
+ });
+ const screenshot = page.screenshot.bind(page);
+ const calledAt: number[] = [];
+ let failures = 1;
+ // What Chromium answers when its surface copy comes back empty.
+ page.screenshot = (async (options?: Parameters<Page['screenshot']>[0]) => {
+ calledAt.push(
+ await page.evaluate(
+ () => (window as unknown as {__frames: number}).__frames,
+ ),
+ );
+ if (failures-- > 0) {
+ throw new Error(
+ 'page.screenshot: Protocol error (Page.captureScreenshot): Unable to capture screenshot',
+ );
+ }
+ return screenshot(options);
+ }) as Page['screenshot'];
+ const storyboard = new Storyboard(page, {
+ name: 'harness-self-test-recapture',
+ clip: {x: 0, y: 340, width: 402, height: 120},
+ });
+ await storyboard.capture('t=0 ms', 0);
+ expect(storyboard.frames).toHaveLength(1);
+ expect(calledAt).toHaveLength(2);
+ expect(calledAt[1] - calledAt[0]).toBeGreaterThanOrEqual(2);
+ expect(storyboard.trace().recaptures).toEqual([
+ {label: 't=0 ms', error: expect.stringContaining('Unable to capture')},
+ ]);
+ // Captures that keep failing still fail, with Chromium's error.
+ failures = Infinity;
+ await expect(storyboard.capture('t=1 ms', 1)).rejects.toThrow(
+ /Unable to capture screenshot/,
+ );
+ expect(storyboard.frames).toHaveLength(1);
+ });
+
+ test('a touch filmstrip photographs every scripted event', async ({
+ page,
+ }, testInfo) => {
+ await fixture(page);
+ await page.evaluate(() => {
+ const pad = document.getElementById('pad');
+ if (pad == null) {
+ throw new Error('no pad');
+ }
+ const dot = document.createElement('div');
+ dot.id = 'dot';
+ dot.style.cssText =
+ 'position:absolute;width:24px;height:24px;border-radius:12px;background:#c33;left:-30px;top:0';
+ pad.append(dot);
+ pad.addEventListener('pointermove', e => {
+ dot.style.transform = `translate(${e.clientX - 12 + 30}px, ${e.clientY - 12}px)`;
+ });
+ });
+ const driver = await touchDriver(page);
+ const storyboard = await touchStoryboard(
+ page,
+ driver,
+ drag({x: 60, y: 60}, {x: 300, y: 240}, {durationMs: 100, hz: 60}),
+ {
+ name: 'harness-self-test-touch',
+ clip: {x: 0, y: 0, width: 402, height: 300},
+ measure: async () => ({
+ x: await page
+ .locator('#dot')
+ .evaluate(el => el.getBoundingClientRect().x + 12),
+ }),
+ columns: 4,
+ },
+ );
+ await storyboard.write(testInfo);
+ // before + touchStart + 6 moves + touchEnd
+ expect(storyboard.frames).toHaveLength(9);
+ expect(storyboard.frames.at(-2)?.data.x).toBeCloseTo(300, 0);
+ });
+
+ test('a touch filmstrip keeps the scripted event times, and a handler clock sees its pacing', async ({
+ page,
+ browserName,
+ }) => {
+ test.skip(
+ browserName !== 'chromium',
+ 'WebKit stamps events with their real dispatch time',
+ );
+ await fixture(page);
+ await page.evaluate(() => {
+ const w = window as unknown as {__moves: [number, number][]};
+ w.__moves = [];
+ document.getElementById('pad')?.addEventListener('pointermove', e => {
+ w.__moves.push([e.timeStamp, performance.now()]);
+ });
+ });
+ const driver = await touchDriver(page);
+ await touchStoryboard(
+ page,
+ driver,
+ drag({x: 60, y: 60}, {x: 300, y: 240}, {durationMs: 100, hz: 60}),
+ {
+ name: 'harness-self-test-touch-clock',
+ clip: {x: 0, y: 0, width: 402, height: 300},
+ },
+ );
+ const moves = await page.evaluate(
+ () => (window as unknown as {__moves: [number, number][]}).__moves,
+ );
+ const gaps = (column: 0 | 1) =>
+ moves.slice(1).map((move, i) => move[column] - moves[i][column]);
+ expect(moves.length).toBeGreaterThanOrEqual(5);
+ // The events carry the script's 60 Hz times…
+ for (const gap of gaps(0)) {
+ expect(gap).toBeCloseTo(1000 / 60, 0);
+ }
+ // …but the clock a handler reads advances by a photograph per event.
+ for (const gap of gaps(1)) {
+ expect(gap).toBeGreaterThan(2 * (1000 / 60));
+ }
+ });
+
+ test('the timer gate holds a component timer until the filmstrip is done', async ({
+ page,
+ }, testInfo) => {
+ await fixture(page);
+ await installTimerGate(page);
+ const box = page.locator('#box');
+ // A component that ends its own animation from a timer (a transitionend
+ // backstop): without the gate it would finish behind the held storyboard.
+ await box.evaluate(el => {
+ const animation = el.animate(
+ [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}],
+ {duration: 300, easing: 'linear'},
+ );
+ setTimeout(() => {
+ animation.finish();
+ (window as unknown as {timerRan?: boolean}).timerRan = true;
+ }, 40);
+ });
+ await freezeTimers(page);
+ await page.waitForTimeout(120); // the timer is due now, and held
+ const {storyboard} = await animationStoryboard(page, {
+ name: 'harness-self-test-timer-gate',
+ clip: {x: 0, y: 340, width: 402, height: 120},
+ measure: async () => ({
+ x: await box.evaluate(el => el.getBoundingClientRect().x),
+ }),
+ columns: 6,
+ });
+ const timerRan = () =>
+ page.evaluate(
+ () => (window as unknown as {timerRan?: boolean}).timerRan === true,
+ );
+ expect(await timerRan()).toBe(false);
+ await thawTimers(page);
+ expect(await timerRan()).toBe(true);
+ await storyboard.write(testInfo);
+ const xs = storyboard.frames.map(f => f.data.x as number);
+ expect(xs).toHaveLength(19); // t = 0 … 300 ms every 16.7 ms
+ expect(xs.every((x, i) => i === 0 || x > xs[i - 1])).toBe(true);
+ });
+
+ test('an animation that repeats forever does not set the filmstrip length', async ({
+ page,
+ }, testInfo) => {
+ await fixture(page);
+ await page.locator('#box').evaluate(el => {
+ el.animate([{rotate: '0deg'}, {rotate: '360deg'}], {
+ duration: 800,
+ iterations: Infinity,
+ });
+ el.animate([{opacity: 1}, {opacity: 0.4}], {duration: 200});
+ });
+ const {storyboard, held} = await animationStoryboard(page, {
+ name: 'harness-self-test-endless',
+ clip: {x: 0, y: 340, width: 402, height: 120},
+ columns: 4,
+ });
+ expect(held.map(a => Number.isFinite(a.curveEndMs))).toEqual([false, true]);
+ expect(storyboard.frames).toHaveLength(13); // t = 0 … 200 ms
+ await releaseAnimations(page);
+ await storyboard.write(testInfo);
+ });
+
+ test('a filmstrip runs from the earliest start to the latest end among the animations that end', async ({
+ page,
+ }, testInfo) => {
+ await fixture(page);
+ // A spinner that started a second earlier, a 300 ms move, and a 240 ms
+ // move that starts 83 ms after it.
+ await page.evaluate(async () => {
+ const make = (id: string, top: number) => {
+ const el = document.createElement('div');
+ el.id = id;
+ el.style.cssText = `position:absolute;left:0;top:${top}px;width:40px;height:40px;background:#36c`;
+ document.body.append(el);
+ return el;
+ };
+ document
+ .getElementById('box')
+ ?.animate([{rotate: '0deg'}, {rotate: '360deg'}], {
+ duration: 800,
+ iterations: Infinity,
+ });
+ await new Promise(resolve => setTimeout(resolve, 1000));
+ make('first', 360).animate(
+ [{transform: 'translateX(0px)'}, {transform: 'translateX(300px)'}],
+ {duration: 300, easing: 'linear'},
+ );
+ await new Promise(resolve => setTimeout(resolve, 83));
+ make('second', 410).animate(
+ [{transform: 'translateX(0px)'}, {transform: 'translateX(240px)'}],
+ {duration: 240, easing: 'linear'},
+ );
+ });
+ const x = (id: string) =>
+ page
+ .locator(`#${id}`)
+ .evaluate(el => new DOMMatrix(getComputedStyle(el).transform).m41);
+ const {storyboard, held} = await animationStoryboard(page, {
+ name: 'harness-self-test-offsets',
+ clip: {x: 0, y: 340, width: 402, height: 120},
+ columns: 6,
+ measure: async () => ({
+ first: await x('first'),
+ second: await x('second'),
+ }),
+ });
+ const second = held.find(a => a.target === 'div#second');
+ const first = held.find(a => a.target === 'div#first');
+ const lag = (second?.offsetMs ?? 0) - (first?.offsetMs ?? 0);
+ const firsts = storyboard.frames.map(f => Number(f.data.first));
+ const seconds = storyboard.frames.map(f => Number(f.data.second));
+ // The film starts with the first move at its start, the spinner aside...
+ expect(firsts[0]).toBeCloseTo(0, 1);
+ expect(firsts[6]).toBeCloseTo(100, 0);
+ // ...and ends with the second move at its end, not cut off.
+ expect(seconds.at(-1)).toBeGreaterThan(239);
+ expect(storyboard.frames.at(-1)?.t).toBeCloseTo(lag + 240, 0);
+ await releaseAnimations(page);
+ await storyboard.write(testInfo);
+ });
+
+ test('a filmstrip of endless animations alone asks for its times', async ({
+ page,
+ }) => {
+ await fixture(page);
+ await page.locator('#box').evaluate(el =>
+ el.animate([{rotate: '0deg'}, {rotate: '360deg'}], {
+ duration: 800,
+ iterations: Infinity,
+ }),
+ );
+ await expect(
+ animationStoryboard(page, {
+ name: 'harness-self-test-endless-only',
+ clip: {x: 0, y: 340, width: 402, height: 120},
+ }),
+ ).rejects.toThrow(/repeats forever/);
+ await releaseAnimations(page);
+ });
+
+ test('the timer gate drops callbacks the page cancels and stamps held frames with the time they run', async ({
+ page,
+ }) => {
+ await fixture(page);
+ await installTimerGate(page);
+ await freezeTimers(page);
+ await page.evaluate(() => {
+ const w = window as unknown as Record<string, unknown>;
+ w.frozenAt = performance.now();
+ w.timer = setTimeout(() => {
+ w.cancelledTimerRan = true;
+ }, 20);
+ w.frame = requestAnimationFrame(() => {
+ w.cancelledFrameRan = true;
+ });
+ requestAnimationFrame(time => {
+ w.heldFrameTime = time;
+ });
+ });
+ await page.waitForTimeout(150); // both are due now, and held
+ // The harness's own frame waits still resolve while the page's are held.
+ await Promise.race([
+ frames(page, 2),
+ page.waitForTimeout(5000).then(() => {
+ throw new Error('frames() did not resolve while the gate was frozen');
+ }),
+ ]);
+ await page.evaluate(() => {
+ const w = window as unknown as {timer: number; frame: number};
+ clearTimeout(w.timer);
+ cancelAnimationFrame(w.frame);
+ });
+ await thawTimers(page);
+ const state = await page.evaluate(() => {
+ const w = window as unknown as Record<string, unknown>;
+ return {
+ timer: w.cancelledTimerRan === true,
+ frame: w.cancelledFrameRan === true,
+ heldFor: Number(w.heldFrameTime) - Number(w.frozenAt),
+ };
+ });
+ expect(state).toMatchObject({timer: false, frame: false});
+ expect(state.heldFor).toBeGreaterThanOrEqual(140);
+ });
+});
diff --git a/internal/interaction-harness/src/index.ts b/internal/interaction-harness/src/index.ts
index 32c238e4a1..537bfc4d4b 100644
--- a/internal/interaction-harness/src/index.ts
+++ b/internal/interaction-harness/src/index.ts
@@ -14,6 +14,8 @@
export * from './curves';
export * from './math';
+export * from './output';
export * from './scripts';
export * from './story';
+export * from './storyboard';
export * from './touch';
diff --git a/internal/interaction-harness/src/output.ts b/internal/interaction-harness/src/output.ts
new file mode 100644
index 0000000000..59926b82c9
--- /dev/null
+++ b/internal/interaction-harness/src/output.ts
@@ -0,0 +1,66 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file output.ts
+ * @input Uses node:fs, node:path and the Playwright `TestInfo`
+ * @output `evidenceDir` / `writeEvidence` — where storyboards and curve traces
+ * go, and the one way they are written and attached to the test report.
+ * @position Every artifact the harness produces lands under one gitignored
+ * directory, `test-results/interaction-evidence/` by default
+ * (`ASTRYX_INTERACTION_EVIDENCE` overrides it). Playwright wipes its own
+ * output directory per run; this one is separate, so a storyboard run never
+ * deletes another's evidence. Generated evidence is never committed: a pull
+ * request links or attaches it.
+ */
+
+import fs from 'node:fs';
+import path from 'node:path';
+import type {TestInfo} from '@playwright/test';
+
+export function evidenceDir(): string {
+ return path.resolve(
+ process.env.ASTRYX_INTERACTION_EVIDENCE ??
+ 'test-results/interaction-evidence',
+ );
+}
+
+/** A filesystem-safe name: letters, digits, dot, dash and underscore. */
+export function safeName(name: string): string {
+ return (
+ name
+ .replace(/[^A-Za-z0-9._-]+/g, '-')
+ .replace(/^-+|-+$/g, '')
+ .slice(0, 120) || 'evidence'
+ );
+}
+
+/**
+ * Writes `files` (name → contents) under `<evidenceDir>/<group>/` and attaches
+ * them to the test's report when a `TestInfo` is given. Returns the paths.
+ */
+export async function writeEvidence(
+ group: string,
+ files: Readonly<Record<string, string | Buffer>>,
+ testInfo?: TestInfo,
+): Promise<Record<string, string>> {
+ // `group` may nest ("storyboards/<name>"): each segment is made safe on its own.
+ const dir = path.join(evidenceDir(), ...group.split('/').map(safeName));
+ fs.mkdirSync(dir, {recursive: true});
+ const written: Record<string, string> = {};
+ for (const [name, contents] of Object.entries(files)) {
+ const file = path.join(dir, safeName(name));
+ fs.writeFileSync(file, contents);
+ written[name] = file;
+ if (testInfo != null) {
+ await testInfo.attach(`${group}/${safeName(name)}`, {
+ path: file,
+ contentType: name.endsWith('.png')
+ ? 'image/png'
+ : name.endsWith('.json')
+ ? 'application/json'
+ : 'text/plain',
+ });
+ }
+ }
+ return written;
+}
diff --git a/internal/interaction-harness/src/story.ts b/internal/interaction-harness/src/story.ts
index 9e4323436f..598f75d852 100644
--- a/internal/interaction-harness/src/story.ts
+++ b/internal/interaction-harness/src/story.ts
@@ -93,11 +93,101 @@ export async function frames(page: Page, count = 2): Promise<void> {
await page.evaluate(
n =>
new Promise<void>(resolve => {
+ // The page's real frames, even while a timer gate holds its callbacks.
+ const frame =
+ (
+ window as unknown as {
+ __harnessTimerGate?: {frame: typeof requestAnimationFrame};
+ }
+ ).__harnessTimerGate?.frame ?? requestAnimationFrame;
let left = n;
- const tick = () =>
- --left <= 0 ? resolve() : requestAnimationFrame(tick);
- requestAnimationFrame(tick);
+ const tick = () => (--left <= 0 ? resolve() : frame(tick));
+ frame(tick);
}),
count,
);
}
+
+// A held callback is queued under its timer or frame id, so the page's own
+// clearTimeout and cancelAnimationFrame still cancel it while frozen; a held
+// frame callback gets the time it actually runs at. `frame` stays the real
+// requestAnimationFrame, for the harness's own waits.
+const TIMER_GATE = `(() => {
+ if (window.__harnessTimerGate) return;
+ const realTimeout = window.setTimeout.bind(window);
+ const realClearTimeout = window.clearTimeout.bind(window);
+ const realInterval = window.setInterval.bind(window);
+ const realFrame = window.requestAnimationFrame.bind(window);
+ const realCancelFrame = window.cancelAnimationFrame.bind(window);
+ const gate = {frozen: false, queue: [], frame: realFrame};
+ window.__harnessTimerGate = gate;
+ const drop = key => { gate.queue = gate.queue.filter(entry => entry.key !== key); };
+ window.setTimeout = (fn, ms, ...args) => {
+ const id = realTimeout(() => {
+ const run = () => (typeof fn === 'function' ? fn(...args) : 0);
+ if (gate.frozen) gate.queue.push({key: 't' + id, run}); else run();
+ }, ms);
+ return id;
+ };
+ window.clearTimeout = id => { realClearTimeout(id); drop('t' + id); };
+ window.setInterval = (fn, ms, ...args) => realInterval(() => { if (!gate.frozen && typeof fn === 'function') fn(...args); }, ms);
+ window.requestAnimationFrame = fn => {
+ const id = realFrame(time => {
+ if (gate.frozen) gate.queue.push({key: 'f' + id, run: () => fn(performance.now())}); else fn(time);
+ });
+ return id;
+ };
+ window.cancelAnimationFrame = id => { realCancelFrame(id); drop('f' + id); };
+})()`;
+
+/**
+ * Installs a gate on the page's setTimeout / setInterval /
+ * requestAnimationFrame. Install it BEFORE the animation under test starts:
+ * only callbacks scheduled through the gate can be held. Event timestamps,
+ * Date, performance.now and the Web Animations timeline are untouched.
+ */
+export async function installTimerGate(page: Page): Promise<void> {
+ await page.evaluate(TIMER_GATE);
+}
+
+/**
+ * Holds every gated timer and frame callback until `thawTimers`, so a
+ * component cannot finish an animation behind a held storyboard's back (a
+ * transitionend backstop timer, a cleanup render). Callbacks that come due
+ * meanwhile run, in order, on thaw, unless the page cancels them first
+ * (clearTimeout, cancelAnimationFrame); intervals skip their frozen ticks.
+ */
+export async function freezeTimers(page: Page): Promise<void> {
+ await page.evaluate(() => {
+ const gate = (window as unknown as {__harnessTimerGate?: {frozen: boolean}})
+ .__harnessTimerGate;
+ if (gate == null) {
+ throw new Error(
+ 'freezeTimers: call installTimerGate before the animation starts',
+ );
+ }
+ gate.frozen = true;
+ });
+}
+
+/** Releases the gate: held callbacks run now, in the order they came due. */
+export async function thawTimers(page: Page): Promise<void> {
+ await page.evaluate(() => {
+ const gate = (
+ window as unknown as {
+ __harnessTimerGate?: {
+ frozen: boolean;
+ queue: {key: string; run: () => void}[];
+ };
+ }
+ ).__harnessTimerGate;
+ if (gate == null) {
+ return;
+ }
+ gate.frozen = false;
+ // One at a time from the live queue: a callback may cancel a later one.
+ while (gate.queue.length > 0) {
+ gate.queue.shift()?.run();
+ }
+ });
+}
diff --git a/internal/interaction-harness/src/storyboard.ts b/internal/interaction-harness/src/storyboard.ts
new file mode 100644
index 0000000000..2dc22fdc8c
--- /dev/null
+++ b/internal/interaction-harness/src/storyboard.ts
@@ -0,0 +1,366 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file storyboard.ts
+ * @input Uses a Playwright `Page`, a clip (`Locator` or box), ./curves to seek
+ * animations, ./touch to play scripted touches, and ./output to write
+ * @output `Storyboard` (collect frames, write a PNG contact sheet + JSON trace),
+ * `animationStoryboard` (seek every held animation to t = 0, 16.7, 33.3 … ms
+ * and photograph each), `touchStoryboard` (photograph the page after every
+ * scripted touch event).
+ * @position Deterministic filmstrips: frames come from seeking paused
+ * animations or from stepping scripted input, never from wall-clock capture,
+ * so the same code produces the same sheet. The JSON trace carries the
+ * measured value of every frame and is what the spec asserts; the sheet is
+ * the human evidence attached to the report (and linked from a pull
+ * request, never committed).
+ *
+ * The contact sheet is composed in a scratch page of the same browser (a
+ * canvas grid with a label per cell), so the harness needs no image library.
+ *
+ * SYNC: Keep the README's storyboard section and `pnpm storyboards` in step.
+ */
+
+import type {Locator, Page, TestInfo} from '@playwright/test';
+import {holdAnimations, seekAnimations, type HeldAnimation} from './curves';
+import {timeline} from './math';
+import {writeEvidence} from './output';
+import type {TouchScript} from './scripts';
+import {frames} from './story';
+import type {TouchDriver} from './touch';
+
+export interface ClipBox {
+ readonly x: number;
+ readonly y: number;
+ readonly width: number;
+ readonly height: number;
+}
+
+export interface StoryboardFrame {
+ readonly label: string;
+ /** Milliseconds on the storyboard's own timeline. */
+ readonly t: number;
+ /** Measurements taken at this frame; the spec asserts these. */
+ readonly data: Readonly<Record<string, unknown>>;
+ readonly png: Buffer;
+}
+
+export interface StoryboardOptions {
+ readonly name: string;
+ /** What to photograph: an element, a box, or the whole viewport (default). */
+ readonly clip?: Locator | ClipBox;
+ /** Grid columns in the sheet (default: up to 8). */
+ readonly columns?: number;
+ /** Cell scale relative to CSS px (default 0.5). */
+ readonly cellScale?: number;
+}
+
+/**
+ * Chromium photographs the page by copying its compositor surface. Right after
+ * the page's surface changes (a navigation, a resize) the copy can come back
+ * empty, and Chromium answers "Unable to capture screenshot" once its own five
+ * retries, issued without waiting for a new frame, have all found no surface.
+ * A capture that fails that way is taken again after the page has presented
+ * two more frames, at most this many attempts in all.
+ */
+const CAPTURE_ATTEMPTS = 3;
+const EMPTY_SURFACE_COPY = /Unable to capture screenshot/;
+
+export class Storyboard {
+ readonly frames: StoryboardFrame[] = [];
+ /** Captures Chromium returned empty and that were taken again two frames later. */
+ readonly recaptures: {label: string; error: string}[] = [];
+ private clipBox: ClipBox | undefined;
+
+ constructor(
+ private readonly page: Page,
+ readonly options: StoryboardOptions,
+ ) {}
+
+ /**
+ * Photographs the clip now and records `data` with it. An empty surface
+ * copy (see CAPTURE_ATTEMPTS) is taken again once the page has presented two
+ * more frames; any other failure, or a third empty copy, throws.
+ */
+ async capture(
+ label: string,
+ t: number,
+ data: Readonly<Record<string, unknown>> = {},
+ ): Promise<void> {
+ const clip = await this.resolveClip();
+ for (let attempt = 1; ; attempt += 1) {
+ try {
+ const png = await this.page.screenshot({
+ clip,
+ scale: 'css',
+ animations: 'allow',
+ caret: 'hide',
+ });
+ this.frames.push({label, t, data, png});
+ return;
+ } catch (error) {
+ const message = error instanceof Error ? error.message : String(error);
+ if (attempt >= CAPTURE_ATTEMPTS || !EMPTY_SURFACE_COPY.test(message)) {
+ throw error;
+ }
+ this.recaptures.push({label, error: message.split('\n')[0]});
+ await frames(this.page, 2);
+ }
+ }
+ }
+
+ private async resolveClip(): Promise<ClipBox | undefined> {
+ if (this.clipBox != null) {
+ return this.clipBox;
+ }
+ const {clip} = this.options;
+ if (clip == null) {
+ return undefined;
+ }
+ if ('boundingBox' in clip) {
+ // A locator is measured once, at the first frame, so every cell shows the same region.
+ const box = await clip.boundingBox();
+ if (box == null) {
+ throw new Error(
+ `storyboard ${this.options.name}: clip element is not visible`,
+ );
+ }
+ const viewport = this.page.viewportSize();
+ const x = Math.max(0, Math.floor(box.x));
+ const y = Math.max(0, Math.floor(box.y));
+ const right = Math.min(
+ viewport?.width ?? Infinity,
+ Math.ceil(box.x + box.width),
+ );
+ const bottom = Math.min(
+ viewport?.height ?? Infinity,
+ Math.ceil(box.y + box.height),
+ );
+ this.clipBox = {x, y, width: right - x, height: bottom - y};
+ } else {
+ this.clipBox = clip;
+ }
+ return this.clipBox;
+ }
+
+ /** The JSON trace: every frame's label, time and data. */
+ trace(
+ extra: Readonly<Record<string, unknown>> = {},
+ ): Record<string, unknown> {
+ return {
+ name: this.options.name,
+ browser: this.page.context().browser()?.browserType().name() ?? null,
+ viewport: this.page.viewportSize(),
+ clip: this.clipBox ?? null,
+ ...(this.recaptures.length > 0 ? {recaptures: this.recaptures} : {}),
+ ...extra,
+ frames: this.frames.map(({label, t, data}) => ({label, t, data})),
+ };
+ }
+
+ /** Composes the sheet, writes `sheet.png` + `trace.json`, attaches both. */
+ async write(
+ testInfo?: TestInfo,
+ extra: Readonly<Record<string, unknown>> = {},
+ ): Promise<Record<string, string>> {
+ if (this.frames.length === 0) {
+ throw new Error(`storyboard ${this.options.name}: no frames`);
+ }
+ const sheet = await composeSheet(this.page, this.frames, this.options);
+ const browser =
+ this.page.context().browser()?.browserType().name() ?? 'browser';
+ return writeEvidence(
+ `storyboards/${this.options.name}.${browser}`,
+ {
+ 'sheet.png': sheet,
+ 'trace.json': JSON.stringify(this.trace(extra), null, 2),
+ },
+ testInfo,
+ );
+ }
+}
+
+async function composeSheet(
+ page: Page,
+ frames: readonly StoryboardFrame[],
+ options: StoryboardOptions,
+): Promise<Buffer> {
+ const scratch = await page.context().newPage();
+ try {
+ const dataUrl = await scratch.evaluate(
+ async ({cells, columns, scale, title}) => {
+ const images = await Promise.all(
+ cells.map(async cell =>
+ createImageBitmap(
+ await (await fetch(`data:image/png;base64,${cell.png}`)).blob(),
+ ),
+ ),
+ );
+ const cellW = Math.ceil(images[0].width * scale);
+ const cellH = Math.ceil(images[0].height * scale);
+ const labelH = 30;
+ const pad = 8;
+ const cols = Math.min(columns, cells.length);
+ const rows = Math.ceil(cells.length / cols);
+ const canvas = document.createElement('canvas');
+ canvas.width = pad + cols * (cellW + pad);
+ canvas.height = 28 + pad + rows * (cellH + labelH + pad);
+ const context = canvas.getContext('2d');
+ if (context == null) {
+ throw new Error('no 2d context');
+ }
+ context.fillStyle = '#ffffff';
+ context.fillRect(0, 0, canvas.width, canvas.height);
+ context.fillStyle = '#111111';
+ context.font = '600 14px sans-serif';
+ context.fillText(title, pad, 19);
+ cells.forEach((cell, i) => {
+ const x = pad + (i % cols) * (cellW + pad);
+ const y = 28 + pad + Math.floor(i / cols) * (cellH + labelH + pad);
+ context.drawImage(images[i], x, y, cellW, cellH);
+ context.strokeStyle = '#999999';
+ context.lineWidth = 1;
+ context.strokeRect(x - 0.5, y - 0.5, cellW + 1, cellH + 1);
+ context.fillStyle = '#111111';
+ context.font = '600 11px sans-serif';
+ context.fillText(cell.label, x, y + cellH + 13, cellW);
+ context.fillStyle = '#555555';
+ context.font = '10px monospace';
+ context.fillText(cell.caption, x, y + cellH + 26, cellW);
+ });
+ return canvas.toDataURL('image/png');
+ },
+ {
+ cells: frames.map(f => ({
+ png: f.png.toString('base64'),
+ label: f.label,
+ caption: Object.entries(f.data)
+ .filter(
+ ([, v]) =>
+ typeof v === 'number' ||
+ typeof v === 'string' ||
+ typeof v === 'boolean',
+ )
+ .map(
+ ([k, v]) =>
+ `${k}=${typeof v === 'number' ? +v.toFixed(1) : String(v)}`,
+ )
+ .join(' '),
+ })),
+ columns: options.columns ?? 8,
+ scale: options.cellScale ?? 0.5,
+ title: options.name,
+ },
+ );
+ return Buffer.from(dataUrl.slice(dataUrl.indexOf(',') + 1), 'base64');
+ } finally {
+ await scratch.close();
+ }
+}
+
+export interface AnimationStoryboardOptions extends StoryboardOptions {
+ /**
+ * Frame times, ms from the film's start: the earliest start among the held
+ * animations that end (an animation that repeats forever, such as a
+ * spinner that started earlier, neither starts nor ends the film). Default:
+ * every 16.7 ms to the latest curve end among them, and that end itself,
+ * each animation at its own offset (a held end value is not filmed).
+ */
+ readonly times?: readonly number[];
+ /** Restrict to animations inside this element. */
+ readonly scope?: Locator;
+ /** Measurements per frame (runs after the seek). */
+ readonly measure?: () => Promise<Readonly<Record<string, unknown>>>;
+}
+
+/**
+ * Holds every running animation, then seeks them together to each time and
+ * photographs the clip. Call it right after starting the animation under test.
+ */
+export async function animationStoryboard(
+ page: Page,
+ options: AnimationStoryboardOptions,
+): Promise<{storyboard: Storyboard; held: HeldAnimation[]}> {
+ const held = await holdAnimations(page, options.scope);
+ if (held.length === 0) {
+ throw new Error(`storyboard ${options.name}: no running animation to hold`);
+ }
+ // An animation that repeats forever (a spinner, a pulsing skeleton) has no
+ // end to film to: the film runs over the animations that end, from the
+ // earliest of their starts to the latest of their curve ends. Seeks are on
+ // the held timeline (from the earliest held start), so both carry offsets.
+ const ending = held.filter(a => Number.isFinite(a.curveEndMs));
+ if (options.times == null && ending.length === 0) {
+ throw new Error(
+ `storyboard ${options.name}: every held animation repeats forever ` +
+ `(${held.map(a => a.name ?? a.kind).join(', ')}); pass \`times\``,
+ );
+ }
+ const origin = ending.length ? Math.min(...ending.map(a => a.offsetMs)) : 0;
+ const length =
+ Math.max(...ending.map(a => a.offsetMs + a.curveEndMs)) - origin;
+ // The 16.7 ms grid, plus the end itself when the grid misses it: the last
+ // frame is the one the page presents at the end.
+ const grid = options.times == null ? timeline(length) : [];
+ if (options.times == null && length - (grid.at(-1) ?? 0) > 1e-3) {
+ grid.push(+length.toFixed(3));
+ }
+ const times = options.times ?? grid;
+ const storyboard = new Storyboard(page, options);
+ for (const t of times) {
+ // Labelled with the time the page presents (just inside an end), from
+ // the film's start.
+ const presentedAt = (await seekAnimations(page, origin + t)) - origin;
+ const data = (await options.measure?.()) ?? {};
+ await storyboard.capture(
+ `t=${+presentedAt.toFixed(1)} ms`,
+ presentedAt,
+ data,
+ );
+ }
+ return {storyboard, held};
+}
+
+export interface TouchStoryboardOptions extends StoryboardOptions {
+ readonly measure?: () => Promise<Readonly<Record<string, unknown>>>;
+}
+
+/**
+ * Plays a touch script without wall-clock pacing and photographs the page
+ * after every event. In Chromium each event's `timeStamp` is still its
+ * scripted time, so velocity read from event timestamps is unchanged; in
+ * WebKit it is the real (slower) dispatch time.
+ *
+ * A handler that reads the clock instead (`performance.now()`, `Date.now()`)
+ * sees the filmstrip's pacing in both engines: two frames, `measure` and a
+ * photograph per event, about 100 ms between moves instead of the script's
+ * 16.7. Its release velocity is that much lower, so a flick that dismisses
+ * when played can settle back when filmed: a storyboard cannot show such a
+ * gesture; record it in real time instead.
+ */
+export async function touchStoryboard(
+ page: Page,
+ driver: TouchDriver,
+ script: TouchScript,
+ options: TouchStoryboardOptions,
+): Promise<Storyboard> {
+ const storyboard = new Storyboard(page, options);
+ await storyboard.capture('before', 0, (await options.measure?.()) ?? {});
+ await driver.play(script, {
+ realtime: false,
+ onEvent: async step => {
+ // Let the page run its handlers and present the frame before the photo.
+ await frames(page, 2);
+ const data = (await options.measure?.()) ?? {};
+ const where = step.changed
+ .map(f => `${Math.round(f.x)},${Math.round(f.y)}`)
+ .join(' ');
+ await storyboard.capture(
+ `${step.type} t=${+step.t.toFixed(1)} @${where}`,
+ step.t,
+ data,
+ );
+ },
+ });
+ return storyboard;
+}
diff --git a/package.json b/package.json
index 51b433c77d..ff3547ccab 100644
--- a/package.json
+++ b/package.json
@@ -32,6 +32,7 @@
"test:a11y-contract:pr": "node .github/scripts/a11y-pr-contracts.mjs",
"test:docsite-browser": "playwright test --config apps/docsite/playwright.config.ts",
"test:interaction": "playwright test --config playwright.interaction.config.ts",
+ "storyboards": "playwright test --config playwright.interaction.config.ts --grep @storyboard",
"typecheck": "pnpm -r --no-bail --workspace-concurrency=1 --if-present --filter \"!@astryxdesign/docsite\" --filter \"!@astryxdesign/sandbox\" run typecheck",
"check:package-boundaries": "node scripts/check-package-boundaries.js",
"check:repo": "pnpm check:visual-owner && pnpm check:sync && pnpm check:neutral-palette-refs && pnpm check:stone-palette-refs && pnpm check:gothic-palette-refs && pnpm check:matcha-palette-refs && pnpm check:butter-palette-refs && pnpm check:y2k-palette-refs && pnpm check:data-tokens && pnpm check:token-docs && pnpm check:knowledge && pnpm check:package-boundaries && pnpm check:changesets && pnpm check:demo-media && pnpm check:executable-bits && pnpm check:cli-structure && pnpm check:use-client && pnpm check:portable-scripts && pnpm check:i18n-runtime && pnpm check:i18n-catalog && pnpm check:cldr-weekdays && pnpm check:fixtures && pnpm check:setup-contract",
diff --git a/playwright.interaction.config.ts b/playwright.interaction.config.ts
index e4bfed2f51..f0c61e2e6e 100644
--- a/playwright.interaction.config.ts
+++ b/playwright.interaction.config.ts
@@ -5,9 +5,9 @@
* @input Uses @playwright/test
* @output Touch interaction and animation proofs on a phone-sized touch
* screen, in Chromium and WebKit.
- * @position Run with `pnpm test:interaction`. Separate from `pnpm test` (no
- * browser needed there) and from playwright.config.ts (accessibility
- * contracts on desktop profiles).
+ * @position Run with `pnpm test:interaction`; filmstrips only with
+ * `pnpm storyboards`. Separate from `pnpm test` (no browser needed there) and
+ * from playwright.config.ts (accessibility contracts on desktop profiles).
*
* Specs are named `*.interaction.browser.spec.ts` and read stories from a
* built Storybook, like the accessibility specs:
@@ -19,6 +19,9 @@
*
* A spec that needs Chromium-only evidence (CDP tracing, scripted touch
* timestamps) skips itself in WebKit with the reason; see the harness README.
+ * Where WebKit cannot launch on the host (missing system libraries on some
+ * Linux distributions), internal/interaction-harness/scripts/webkit-container.mjs
+ * runs the WebKit project inside the official Playwright image.
*
* SYNC: Keep the phone profile here equal to `phone` in
* internal/interaction-harness/src/story.ts.