Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

test(interaction): frame evidence (#8045)

Christopher Chedeau6ecbf415d4e9
diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.md
index 31c5149c2d..4c67256a26 100644
--- a/internal/interaction-harness/README.md
+++ b/internal/interaction-harness/README.md
@@ -1,20 +1,23 @@
# @astryxdesign/interaction-harness
Playwright instruments for touch interaction and animation proofs: real touch
-input, curve sampling and storyboards. **Internal and unpublished.** Specs live
-next to their component as `*.interaction.browser.spec.ts` and run under
+input, frame evidence, curve sampling and storyboards. **Internal and
+unpublished.** Specs live next to their component as
+`*.interaction.browser.spec.ts` and run under
[`playwright.interaction.config.ts`](../../playwright.interaction.config.ts) on
a phone-sized touch screen (402 × 714 CSS px page, 402 × 874 screen, 3 device
pixels per CSS px, touch, mobile viewport rules), in Chromium and WebKit.
The harness proves its own instruments in
-[`src/harness.interaction.browser.spec.ts`](src/harness.interaction.browser.spec.ts),
-on fixture pages written inline.
+[`src/harness.interaction.browser.spec.ts`](src/harness.interaction.browser.spec.ts).
+The busy-main-thread proof has a negative control beside its positive check
+(a rAF animation that freezes under a blocked main thread); other self-tests
+check an instrument against a known answer.
## Running
```sh
-pnpm storybook:build # component specs read stories from the built Storybook
+pnpm storybook:build # the specs read stories from the built Storybook
pnpm exec playwright install chromium webkit
pnpm test:interaction --project chromium # everything, Chromium
pnpm test:interaction <spec file> # one spec, both engines
@@ -42,6 +45,8 @@ committed: a pull request links or attaches it.
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scripts.ts` | Touch scripts as data: `tap`, `drag` (hold, rest), `flick` (exact final-move speed, velocity profiles), `holdThenDrag`, `pinch`, `rotate`, `path`, `merge` (a second finger mid-gesture), `releaseVelocity`, `toTouchEvents`. |
| `touch.ts` | `touchDriver(page).play(script)`: the script as real engine input. `installInputLog` / `readInputLog`: what the page received. |
+| `frames.ts` | `startScreencast` (timestamped composited frames), `markElement` (a marker chip or outline), `trackMarker` (marker box per frame), `frameStats`. |
+| `busy.ts` | `blockMainThread(page, ms)` and `busyWindowProof(tracked, window)`: frames presented, frames that moved, longest stall, a verdict; `presented` adds the compositor's own frames. |
| `curves.ts` | `holdAnimations`, `seekAnimations`, `sampleCurve` (presented transform / opacity at any t), `waitForAnimation`, `watchAnimations`, `waitForRest`, `releaseAnimations`. |
| `math.ts` | Oracles independent of the code under test: `cubicBezier`, `timingFunction`, `springStep` (SwiftUI response / dampingFraction, initial velocity), `projectDecay`, `rubberBand`, `parseTransform`, `compareCurve`, `timeline`. |
| `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event): PNG contact sheet + JSON trace. |
@@ -97,6 +102,72 @@ set): a short entrance no longer races the first look on a cold page. When the
action may be the page's first contact with the harness, take
`watchAnimations(locator)` before it and read `watch.started()` after.
+**Busy main thread.** Mark the element, start the screencast, start the
+animation, wait until it runs (`waitForAnimation`: an animation cannot start
+while the thread is held), block the main thread for 1 s (`spec:AST-094`
+FR5's verification row), stop. Always
+start the gesture or animation first and block mid-flight, never before:
+in headless Chromium a touch that starts while the main thread is busy does
+not scroll or start a gesture even with only passive listeners (the input
+waits for the main thread). Tracking a finger under a busy main thread is
+therefore proven on the iOS simulator, not here. The
+screencast is captured off the main thread, so a compositor animation shows a
+new position every frame of the block, and a main-thread animation shows none
+(nothing changed, so no frame is sent at all: `busyWindowProof` counts the
+whole window as one stall). `settledWithin` (2 px) stops judging at the easing
+tail: the screencast rounds to whole pixels, so a tail that still moves shows
+plateaus within a pixel or two of rest. A judged part shorter than four of the
+screencast's intervals fails: the animation under test was not moving during
+the block. With
+`during`, `blockMainThread` starts the loop from a task queued in the frame
+that saw the animation, after that frame's rendering update: WebKit hands a
+newly started animation to its compositor there, and a loop inside the rAF
+callback would freeze even a compositor animation. Call it before the release
+that starts a settle and await it after, so the page is already waiting when
+the settle begins. An animation counts once it is no longer pending and has
+run at least one frame: a `minProgressMs` under 16 is raised to 16, since one
+created in the frame the page looks is not on the compositor yet. The `verdict` judges the
+frames the screencast delivered, whatever its rate (a loaded machine's can
+deliver one frame in five or six), and tolerates nothing a frozen animation
+does: at least one frame in the judged part, no stall longer than three of the
+screencast's intervals (two dropped frames), and at least 90% of the frames it
+delivered in the judged part presenting a new position (one miss is always
+allowed; a frame it did not deliver is a gap, for the stall check, unless
+it delivered fewer than four frames there: then the judged part's span counts
+every frame, so one late frame is not "1 of 1 moved"). Pass
+`maxMedianFrameMs` (20 at 60 Hz) only in a proof that claims display rate.
+`verdict.failures` names each miss with its numbers, and `summary` carries the
+raw numbers for the report. Frames are judged in presentation order: a loaded
+machine can deliver them out of order.
+
+A just-started animation can reach the screen after the block began (50–100 ms
+later in WebKit on a loaded machine). When the cast saw the element at rest
+before the block (wait for `cast.frameCount()` to pass 0 before starting the
+animation), judging starts at its first presented change: the judged part still
+ends with the block and still needs four intervals, and `onScreenFromMs` and the
+summary report when it came. An element already moving is judged from the
+block's start. WebKit's casts are 2 px quantized: the frames come at the
+viewport's size, but the page in them is captured at half resolution, so
+positions move in 2 CSS px steps (Chromium's in 1). With under two steps of
+motion per frame, a frame delivered a few ms early shows no step and reads as
+not moved. The proof reports `stepPx` and `pxPerFrame` and notes motion under
+two steps per frame, which never fails it. Move a proof's element at least two
+steps a frame: the self-tests move 560 px in 1,600 ms, about 4.8 px a frame.
+
+On a loaded machine Chromium's screencast can skip frames the compositor
+still presents. Pass when the compositor presented each frame (epoch ms on the
+page clock, from a trace of `Display::FrameDisplayed`) as `presented`. A gap between two moved
+frames with nothing delivered inside it is then measured between the
+compositor's own frames: the frames it presented there count as presented
+(`recorderSkipped`, in the summary), and a gap it did not present through is
+a stall as before. A gap that opens or closes the judged part, or holds a
+delivered frame that did not move, is judged from the screencast, so a
+frozen element fails whatever else the compositor draws (a rAF animation's
+block presents no compositor frame at all). With the record, the 90% counts the
+compositor's frames from the first to the last moved frame; the gaps before and
+after are the stall check's. Without a record (WebKit has no such trace), the
+screencast is judged alone.
+
**Storyboards.** Deterministic: frames come from seeking held animations or
from stepping scripted input, never from wall-clock capture. The JSON trace
carries each frame's measurements and the spec asserts on it; the sheet is
@@ -104,16 +175,18 @@ evidence. A component that finishes an animation from a timer (a transitionend
backstop, a cleanup render) would do so behind a held storyboard's back:
`installTimerGate` before the animation starts and `freezeTimers` while
photographing. The page can still cancel a held callback, a held frame callback
-gets the time it actually runs, and the harness's own frame waits are not held.
+gets the time it actually runs, and the harness's own waits (its frame waits,
+`blockMainThread` with `during`) are not held.
An animation that repeats forever does not set a storyboard's default length;
with nothing else to film, pass `times`. A touch filmstrip keeps the scripted
event `timeStamp`s (Chromium), but a handler that reads `performance.now()` or
`Date.now()` sees one event per photograph, about 100 ms apart, so a
clock-timed release flings slower when filmed than when played: a storyboard
-cannot show such a gesture; record it in real time instead. Chromium can return
-an empty copy of the page's surface right after it changes ("Unable to capture
-screenshot"); a storyboard takes that photograph again two frames later, up to
-three attempts in all, and lists each retake under `recaptures` in the trace.
+cannot show such a gesture; record it in real time instead, with
+`startScreencast`. Chromium can return an empty copy of
+the page's surface right after it changes ("Unable to capture screenshot"); a
+storyboard takes that photograph again two frames later, up to three attempts
+in all, and lists each retake under `recaptures` in the trace.
## Engine limits (measured with Playwright 1.61)
@@ -126,11 +199,13 @@ three attempts in all, and lists each retake under `recaptures` in the trace.
| Lifting one finger while others stay down | yes | pointer events right, but that `touchend` lists no remaining `touches`: skip with `PARTIAL_LIFT_SKIP` |
| Click synthesised from a touch | yes | only from `driver.tap` (WebKit's tap gesture), not from a raw touch sequence |
| Curve sampling, storyboards, input log | yes | yes |
+| Screencast frame evidence (busy main thread) | yes | yes (WebKit's own screencast; frames keep coming while the main thread is blocked) |
What WebKit cannot show here — native scroll hand-off, flings, a finger
-lifting while another stays down, and anything iOS-only — needs Mobile Safari
-itself, on a device or the iOS simulator. Skip such a proof on the driver's
-flag, citing the reason: `test.skip(!driver.partialLifts, PARTIAL_LIFT_SKIP)`.
+lifting while another stays down, finger tracking under a busy main thread, and
+anything iOS-only — needs Mobile Safari itself, on a device or the iOS
+simulator. Skip such a proof on the driver's flag, citing the reason:
+`test.skip(!driver.partialLifts, PARTIAL_LIFT_SKIP)`.
The WebKit input command is reached through Playwright's in-process server
object, so it needs a locally launched browser (the default), not `connect()`.
diff --git a/internal/interaction-harness/package.json b/internal/interaction-harness/package.json
index 40cccb26ca..942eccb9ed 100644
--- a/internal/interaction-harness/package.json
+++ b/internal/interaction-harness/package.json
@@ -3,7 +3,7 @@
"version": "0.0.0",
"private": true,
"type": "module",
- "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, curve sampling, storyboards). Never published.",
+ "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, frame evidence, curve sampling, storyboards). Never published.",
"main": "./src/index.ts",
"exports": {
".": "./src/index.ts",
diff --git a/internal/interaction-harness/src/busy.test.ts b/internal/interaction-harness/src/busy.test.ts
new file mode 100644
index 0000000000..372860125f
--- /dev/null
+++ b/internal/interaction-harness/src/busy.test.ts
@@ -0,0 +1,503 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file busy.test.ts
+ * @input Synthetic tracked frames (no browser)
+ * @output The busy-main-thread verdict's limits: dropped frames a loaded
+ * machine produces pass, a frozen animation fails every criterion, a window
+ * with nothing moving to judge fails, and out-of-order frames are judged in
+ * presentation order.
+ * @position Unit test of ./busy and ./frames' ordering; the browser proof
+ * with a real compositor animation and a rAF control is in
+ * harness.interaction.browser.spec.ts.
+ */
+
+import {describe, expect, it} from 'vitest';
+import {busyWindowProof} from './busy';
+import {inPresentationOrder, type TrackedFrame} from './frames';
+
+const FRAME = 1000 / 60;
+const BLOCK = {from: 1000, to: 1700};
+
+/** A marker moving 3 px per frame from t = 800 to 1900, minus `dropped` frame indices. */
+function moving(dropped: readonly number[] = []): TrackedFrame[] {
+ const frames: TrackedFrame[] = [];
+ for (let i = 0; 800 + i * FRAME <= 1900; i += 1) {
+ if (dropped.includes(i)) {
+ continue;
+ }
+ frames.push({
+ t: 800 + i * FRAME,
+ box: {x: 0, y: 3 * i, width: 10, height: 10},
+ } as TrackedFrame);
+ }
+ return frames;
+}
+
+/** Index of the first frame at or after `t`. */
+const at = (t: number) => Math.ceil((t - 800) / FRAME);
+
+/** `moving()` with each frame in `repeats` presenting the previous frame's position. */
+function repeating(repeats: readonly number[]): TrackedFrame[] {
+ return moving().map((f, i) =>
+ repeats.includes(i)
+ ? ({...f, box: {...f.box!, y: 3 * (i - 1)}} as TrackedFrame)
+ : f,
+ );
+}
+
+/** A marker moving 3 px per frame from t = 800 to 1900, frames `intervals` apart in turn. */
+function cadenced(intervals: readonly number[]): TrackedFrame[] {
+ const frames: TrackedFrame[] = [];
+ for (let i = 0, t = 800; t <= 1900; i += 1) {
+ frames.push({
+ t,
+ box: {x: 0, y: 3 * i, width: 10, height: 10},
+ } as TrackedFrame);
+ t += intervals[i % intervals.length];
+ }
+ return frames;
+}
+
+describe('busyWindowProof verdict', () => {
+ it('passes a compositor animation with every frame', () => {
+ const proof = busyWindowProof(moving(), BLOCK);
+ expect(proof.verdict.failures).toEqual([]);
+ expect(proof.longestStallFrames).toBe(1);
+ });
+
+ it('passes one and two consecutive dropped frames (3 intervals)', () => {
+ const k = at(1300);
+ expect(busyWindowProof(moving([k]), BLOCK).verdict.failures).toEqual([]);
+ const two = busyWindowProof(moving([k, k + 1]), BLOCK);
+ expect(two.longestStallFrames).toBe(3);
+ expect(two.verdict.failures).toEqual([]);
+ });
+
+ it('fails three consecutive dropped frames', () => {
+ const k = at(1300);
+ const proof = busyWindowProof(moving([k, k + 1, k + 2]), BLOCK);
+ expect(proof.longestStallFrames).toBe(4);
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ });
+
+ it('fails when more than 10% of the delivered frames do not move', () => {
+ const k = at(1050);
+ // 39 frames delivered: up to 3 may show no new position (10% = 3.9).
+ const scattered = [0, 12, 24].map(d => k + d);
+ expect(
+ busyWindowProof(repeating(scattered), BLOCK).verdict.failures,
+ ).toEqual([]);
+ const many = [0, 6, 12, 18, 24].map(d => k + d);
+ expect(
+ busyWindowProof(repeating(many), BLOCK).verdict.failures.join('\n'),
+ ).toMatch(/expected frames moved/);
+ });
+
+ it('judges a frame the screencast did not deliver as a gap for the stall check, not a miss', () => {
+ // The same five frames, not delivered: two-interval gaps, and every
+ // delivered frame moved.
+ const k = at(1050);
+ const many = [0, 6, 12, 18, 24].map(d => k + d);
+ const proof = busyWindowProof(moving(many), BLOCK);
+ expect(proof.verdict.failures).toEqual([]);
+ expect(proof.deliveredFrames).toBe(proof.moved);
+ });
+
+ it('judges an irregular cast on the frames it delivered', () => {
+ // A cast cycling 11.7, 11.7 and 17 ms: its median interval would expect
+ // 56 frames where it delivered 49, every one moved.
+ const proof = busyWindowProof(cadenced([11.7, 11.7, 17]), BLOCK);
+ expect(proof.verdict.failures).toEqual([]);
+ expect(proof.deliveredFrames).toBe(49);
+ expect(proof.moved).toBe(49);
+ // Still a stall where it delivers nothing for four frames, and still a
+ // miss where five delivered frames show no new position.
+ const gap = cadenced([11.7, 11.7, 17]).filter(
+ f => f.t < 1300 || f.t > 1360,
+ );
+ expect(busyWindowProof(gap, BLOCK).verdict.failures.join('\n')).toMatch(
+ /longest stall/,
+ );
+ const unmoved = cadenced([11.7, 11.7, 17]).map((f, i, all) =>
+ [20, 26, 32, 38, 44].includes(i)
+ ? ({...f, box: {...all[i - 1].box!}} as TrackedFrame)
+ : f,
+ );
+ expect(busyWindowProof(unmoved, BLOCK).verdict.failures.join('\n')).toMatch(
+ /expected frames moved/,
+ );
+ });
+
+ it('judges a window that delivered fewer than four frames against its span', () => {
+ // A rAF animation frozen by the block, in WebKit: one frame from before
+ // the block arrives 45 ms into it, then nothing until after it.
+ const late = moving().filter(f => f.t <= 1000 || f.t >= 1750);
+ late.push({
+ t: 1045,
+ box: {x: 0, y: 3 * at(1001), width: 10, height: 10},
+ } as TrackedFrame);
+ const proof = busyWindowProof(inPresentationOrder(late), BLOCK);
+ expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/);
+ });
+
+ it('lets a short window miss a frame, not three', () => {
+ // 150 ms judged: nine frames delivered, one miss allowed (10% would be 0.9).
+ const short = {from: 1000, to: 1190};
+ const k = at(1040);
+ expect(busyWindowProof(repeating([k]), short).verdict.failures).toEqual([]);
+ expect(
+ busyWindowProof(
+ repeating([k, k + 3, k + 6]),
+ short,
+ ).verdict.failures.join('\n'),
+ ).toMatch(/expected frames moved/);
+ });
+
+ it('fails a frozen animation on every criterion', () => {
+ // Frames before and after the block only: nothing changed during it.
+ const frozen = moving().filter(f => f.t < BLOCK.from || f.t > BLOCK.to);
+ const proof = busyWindowProof(frozen, BLOCK);
+ expect(proof.changed).toBe(0);
+ expect(proof.verdict.pass).toBe(false);
+ expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/);
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/);
+ });
+
+ it('fails a window in which the animation under test was not moving', () => {
+ // Settled before the block: the judgment ends where it starts.
+ const settled = moving().map(f =>
+ f.t < 950
+ ? f
+ : ({...f, box: {...f.box!, y: 3 * at(950)}} as TrackedFrame),
+ );
+ const before = busyWindowProof(settled, BLOCK, {settledWithin: 2});
+ expect(before.verdict.pass).toBe(false);
+ expect(before.verdict.failures.join('\n')).toMatch(
+ /judged = -?\d+ frames < 4: the animation under test was not moving/,
+ );
+ // Settled 40 ms into the block: two frames, too few to judge.
+ const early = moving().map(f =>
+ f.t < 1040
+ ? f
+ : ({...f, box: {...f.box!, y: 3 * at(1040)}} as TrackedFrame),
+ );
+ expect(
+ busyWindowProof(early, BLOCK, {settledWithin: 2}).verdict.failures.join(
+ '\n',
+ ),
+ ).toMatch(/frames < 4: the animation under test was not moving/);
+ });
+
+ it('judges the frames a slower screencast delivers by default, and display rate only when asked', () => {
+ // One frame in six, as a loaded machine's screencast delivers: every
+ // delivered frame moved, so the default verdict passes.
+ const sixth = moving().filter((_, i) => i % 6 === 0);
+ expect(busyWindowProof(sixth, BLOCK).verdict.failures).toEqual([]);
+ // Half rate fails only a proof that claims display rate.
+ const odd = moving().filter((_, i) => i % 2 === 1);
+ expect(busyWindowProof(odd, BLOCK).verdict.pass).toBe(true);
+ expect(
+ busyWindowProof(odd, BLOCK, {maxMedianFrameMs: 20}).verdict.failures.join(
+ '\n',
+ ),
+ ).toMatch(/median frame interval/);
+ });
+});
+
+describe('busyWindowProof settledWithin', () => {
+ it('ends the judgment at a rounded easing tail, which would read as a stall', () => {
+ // An ease-out to y = 24 as the screencast reports it: whole pixels, so the
+ // tail repeats values while it still moves.
+ const ys = [
+ 120, 100, 82, 67, 55, 46, 39, 34, 30, 28, 27, 26, 26, 26, 25, 25, 24, 24,
+ 24, 24,
+ ];
+ const frames = ys.map(
+ (y, i) =>
+ ({
+ t: 1000 + i * FRAME,
+ box: {x: 0, y, width: 10, height: 10},
+ }) as TrackedFrame,
+ );
+ const window = {from: 1000, to: 1000 + ys.length * FRAME};
+ const loose = busyWindowProof(frames, window, {
+ settledWithin: 2,
+ marginMs: 0,
+ });
+ expect(loose.verdict.failures).toEqual([]);
+ const strict = busyWindowProof(frames, window, {
+ settledWithin: 1,
+ marginMs: 0,
+ });
+ expect(strict.longestStallFrames).toBeGreaterThanOrEqual(3);
+ });
+
+ it('takes the end value from settledAt over a stale last frame', () => {
+ // The tail reaches y = 24, then the last frame shows a stale 100 (WebKit
+ // presents the last committed position once an accelerated animation
+ // ends while the main thread is still blocked).
+ const ys = [
+ 120, 100, 82, 67, 55, 46, 39, 34, 30, 28, 27, 26, 25, 24, 24, 24, 100,
+ ];
+ const frames = ys.map(
+ (y, i) =>
+ ({
+ t: 1000 + i * FRAME,
+ box: {x: 0, y, width: 10, height: 10},
+ }) as TrackedFrame,
+ );
+ const window = {from: 1000, to: 1000 + ys.length * FRAME};
+ const stale = busyWindowProof(frames, window, {
+ settledWithin: 2,
+ marginMs: 0,
+ });
+ expect(stale.verdict.pass).toBe(false);
+ const fromPage = busyWindowProof(frames, window, {
+ settledWithin: 2,
+ settledAt: 24,
+ marginMs: 0,
+ });
+ expect(fromPage.verdict.failures).toEqual([]);
+ expect(fromPage.judged.to).toBe(frames[11].t);
+ });
+});
+
+/** Seen at rest at t = 900, then on screen from `start`, moving 3 px a frame. */
+function startingAt(start: number): TrackedFrame[] {
+ const frames = [
+ {t: 900, box: {x: 0, y: 0, width: 10, height: 10}} as TrackedFrame,
+ ];
+ for (let i = 1; start + (i - 1) * FRAME <= 1900; i += 1) {
+ frames.push({
+ t: start + (i - 1) * FRAME,
+ box: {x: 0, y: 3 * i, width: 10, height: 10},
+ } as TrackedFrame);
+ }
+ return frames;
+}
+
+describe('busyWindowProof on-screen start', () => {
+ it('judges a just-started animation from its first presented change when it was seen at rest before the block, and reports when', () => {
+ // On screen 87 ms into the block: four intervals after the judged part's start.
+ const proof = busyWindowProof(startingAt(1087), BLOCK);
+ expect(proof.verdict.failures).toEqual([]);
+ expect(proof.onScreenFromMs).toBeCloseTo(87, 5);
+ expect(proof.judged.from).toBeCloseTo(1087, 5);
+ expect(proof.summary).toMatch(
+ /on screen from \+87 ms \(nothing moved before the block\)/,
+ );
+ });
+
+ it('still fails an animation that was already moving and stalls at the block start', () => {
+ const k = at(1020);
+ const proof = busyWindowProof(moving([k, k + 1, k + 2, k + 3]), BLOCK);
+ expect(proof.onScreenFromMs).toBeNull();
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ });
+
+ it('still fails a stall at the start when no frame before the block shows it at rest', () => {
+ const unseen = startingAt(1087).filter(f => f.t >= BLOCK.from);
+ const proof = busyWindowProof(unseen, BLOCK);
+ expect(proof.onScreenFromMs).toBeNull();
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ });
+
+ it('still fails a start the block itself delays until it ends', () => {
+ const proof = busyWindowProof(startingAt(1720), BLOCK);
+ expect(proof.onScreenFromMs).toBeNull();
+ expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/);
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ });
+
+ it('still fails a start that leaves fewer than minJudgedFrames inside the block', () => {
+ const proof = busyWindowProof(startingAt(1640), BLOCK);
+ expect(proof.onScreenFromMs).toBeCloseTo(640, 5);
+ expect(proof.verdict.failures.join('\n')).toMatch(
+ /frames < 4: the animation under test was not moving/,
+ );
+ });
+});
+
+describe('busyWindowProof position step', () => {
+ it("reports the cast's position step and motion per frame, and notes motion under two steps without failing", () => {
+ // 2.4 px a frame on a 2 px grid, as WebKit's casts report positions.
+ const grid = moving().map(
+ (f, i) =>
+ ({
+ ...f,
+ box: {...f.box!, y: 2 * Math.round((2.4 * i) / 2)},
+ }) as TrackedFrame,
+ );
+ const coarse = busyWindowProof(grid, BLOCK);
+ expect(coarse.stepPx).toBe(2);
+ expect(coarse.pxPerFrame).toBeCloseTo(2.4, 1);
+ expect(coarse.notes.join('\n')).toMatch(
+ /under two of the cast's 2 px position steps/,
+ );
+ expect(coarse.summary).toMatch(/2\.4 px\/frame in 2 px steps; note: /);
+ expect(coarse.verdict.failures).toEqual([]);
+ // 3 px a frame in whole pixels: resolvable, nothing to note.
+ const fine = busyWindowProof(moving(), BLOCK);
+ expect(fine.stepPx).toBe(1);
+ expect(fine.pxPerFrame).toBeCloseTo(3, 1);
+ expect(fine.notes).toEqual([]);
+ });
+});
+
+/** The compositor presenting every frame from `from` to `to`, minus `gaps` ([from, to] spans). */
+function drawn(
+ gaps: ReadonlyArray<readonly [number, number]> = [],
+ from = 700,
+ to = 2000,
+): number[] {
+ const draws: number[] = [];
+ for (let t = from; t <= to; t += FRAME) {
+ if (!gaps.some(([a, b]) => t > a && t < b)) {
+ draws.push(t);
+ }
+ }
+ return draws;
+}
+
+describe('busyWindowProof with the compositor record', () => {
+ it('with a compositor record, judges a sparse cast from its first to its last moved frame', () => {
+ // One screencast frame in six, as on a loaded machine, with the compositor
+ // presenting every frame: the frames before the first and after the last
+ // moved one are the stall check's, not misses.
+ const sparse = moving().filter((_, i) => i % 6 === 0);
+ expect(
+ busyWindowProof(sparse, BLOCK, {presented: drawn()}).verdict.failures,
+ ).toEqual([]);
+ });
+
+ it('with a compositor record and an on-screen start, counts the frames it credits as skipped in what it judges', () => {
+ // On screen 87 ms into the block, then four delivered frames that show
+ // no new position: that fails on a full cast.
+ const misses = [15, 20, 25, 30];
+ const missing = startingAt(1087).map((f, i, all) =>
+ misses.includes(i)
+ ? ({...f, box: {...all[i - 1].box!}} as TrackedFrame)
+ : f,
+ );
+ const full = busyWindowProof(missing, BLOCK, {presented: drawn()});
+ expect(full.verdict.failures.join('\n')).toMatch(/expected frames moved/);
+ // The screencast skipping the five frames after the on-screen frame
+ // changes nothing: the compositor presented them, and they are judged.
+ const skip = (frames: TrackedFrame[]) =>
+ frames.filter((_, i) => i < 2 || i > 6);
+ const skipped = busyWindowProof(skip(missing), BLOCK, {
+ presented: drawn(),
+ });
+ expect(skipped.recorderSkipped).toBe(5);
+ expect(skipped.verdict.failures.join('\n')).toMatch(
+ /expected frames moved/,
+ );
+ // And a run with no misses never reads as more than every frame moved.
+ const clean = busyWindowProof(skip(startingAt(1087)), BLOCK, {
+ presented: drawn(),
+ });
+ expect(clean.verdict.failures).toEqual([]);
+ expect(clean.movedFraction).toBeLessThanOrEqual(1);
+ });
+
+ it('counts a gap the compositor drew through as skipped, and one it did not draw through as a stall', () => {
+ const k = at(1300);
+ const gap: [number, number] = [
+ 800 + (k - 1) * FRAME,
+ 800 + (k + 4) * FRAME,
+ ];
+ const frames = moving([k, k + 1, k + 2, k + 3]);
+ expect(busyWindowProof(frames, BLOCK).verdict.pass).toBe(false);
+ const skipped = busyWindowProof(frames, BLOCK, {presented: drawn()});
+ expect(skipped.verdict.failures).toEqual([]);
+ expect(skipped.recorderSkipped).toBe(4);
+ const stalled = busyWindowProof(frames, BLOCK, {presented: drawn([gap])});
+ expect(stalled.recorderSkipped).toBe(0);
+ expect(stalled.verdict.failures.join('\n')).toMatch(/longest stall/);
+ });
+
+ it('still fails an element that does not move while the compositor presents for something else', () => {
+ const frozen = moving().map(f =>
+ f.t >= BLOCK.from && f.t <= BLOCK.to
+ ? ({...f, box: {...f.box!, y: 3 * at(BLOCK.from)}} as TrackedFrame)
+ : f,
+ );
+ const proof = busyWindowProof(frozen, BLOCK, {presented: drawn()});
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/);
+ });
+
+ it('still fails a frozen animation, whether or not the compositor presents during the block', () => {
+ const frozen = moving().filter(f => f.t < BLOCK.from || f.t > BLOCK.to);
+ for (const presented of [drawn([[BLOCK.from, BLOCK.to]]), drawn()]) {
+ const proof = busyWindowProof(frozen, BLOCK, {presented});
+ expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/);
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ }
+ });
+
+ it('keeps a gap a stall when the screencast delivered a frame inside it that did not move', () => {
+ const k = at(1300);
+ const frames = moving([k, k + 1, k + 3]).map(f =>
+ f.t > 800 + (k + 1.5) * FRAME && f.t < 800 + (k + 2.5) * FRAME
+ ? ({...f, box: {...f.box!, y: 3 * (k - 1)}} as TrackedFrame)
+ : f,
+ );
+ const proof = busyWindowProof(frames, BLOCK, {presented: drawn()});
+ expect(proof.recorderSkipped).toBe(0);
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ });
+
+ it('still counts single dropped frames the compositor did not present as missed', () => {
+ const k = at(1050);
+ const drops = [0, 6, 12, 18, 24, 30].map(d => k + d);
+ const frames = moving(drops);
+ const gaps = drops.map(
+ i =>
+ [800 + (i - 0.5) * FRAME, 800 + (i + 0.5) * FRAME] as [number, number],
+ );
+ expect(
+ busyWindowProof(frames, BLOCK, {
+ presented: drawn(gaps),
+ }).verdict.failures.join('\n'),
+ ).toMatch(/expected frames moved/);
+ expect(
+ busyWindowProof(frames, BLOCK, {presented: drawn()}).verdict.failures,
+ ).toEqual([]);
+ });
+
+ it('judges a gap at the edge of the judged part from the screencast alone', () => {
+ // Nothing delivered for the block's first 90 ms: a stall at its start,
+ // whatever the compositor drew, since no moved frame opens the gap.
+ const frames = moving().filter(
+ f => f.t < BLOCK.from || f.t > BLOCK.from + 90,
+ );
+ const proof = busyWindowProof(frames, BLOCK, {presented: drawn()});
+ expect(proof.recorderSkipped).toBe(0);
+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);
+ });
+
+ it('treats an empty record as no record, and says so', () => {
+ const frames = moving([at(1300), at(1300) + 1, at(1300) + 2, at(1300) + 3]);
+ const proof = busyWindowProof(frames, BLOCK, {presented: []});
+ expect(proof.verdict.failures).toEqual(
+ busyWindowProof(frames, BLOCK).verdict.failures,
+ );
+ expect(proof.summary).toMatch(/no compositor frames recorded/);
+ });
+});
+
+describe('inPresentationOrder', () => {
+ it('sorts frames delivered out of order, so motion stays monotonic', () => {
+ const frames = moving();
+ const k = at(1040);
+ const shuffled = [...frames];
+ [shuffled[k], shuffled[k + 1]] = [shuffled[k + 1], shuffled[k]];
+ expect(busyWindowProof(shuffled, BLOCK).monotonic).toBe(false);
+ const proof = busyWindowProof(inPresentationOrder(shuffled), BLOCK);
+ expect(proof.monotonic).toBe(true);
+ expect(proof.verdict.failures).toEqual([]);
+ });
+});
diff --git a/internal/interaction-harness/src/busy.ts b/internal/interaction-harness/src/busy.ts
new file mode 100644
index 0000000000..20c3d86a11
--- /dev/null
+++ b/internal/interaction-harness/src/busy.ts
@@ -0,0 +1,577 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file busy.ts
+ * @input Uses a Playwright `Page`
+ * @output `blockMainThread` — a synchronous busy loop on the page's main thread
+ * that reports exactly when it ran — and `busyWindowProof`, which reads frame
+ * evidence for that window.
+ * @position The busy-main-thread proof: an animation that runs on the
+ * compositor keeps advancing at display rate while JavaScript holds the main
+ * thread; one driven from the main thread (rAF, style writes, layout
+ * properties) freezes for the whole block. Pair it with ./frames
+ * (`startScreencast` + `trackMarker`), which observes off the main thread.
+ *
+ * SYNC: Keep the README's busy-main-thread section equal to this contract.
+ */
+
+import type {Locator, Page} from '@playwright/test';
+import {frameStats, type FrameStats, type TrackedFrame} from './frames';
+
+export interface BusyWindow {
+ /** Epoch ms when the loop started and ended, on the page's own clock. */
+ readonly from: number;
+ readonly to: number;
+}
+
+/**
+ * Holds the page's main thread for `ms` (a synchronous loop: no task, frame,
+ * rAF, event or microtask runs meanwhile). Resolves with the window it held.
+ *
+ * Block MID-FLIGHT only: start the gesture or animation first. A touch that
+ * starts while the main thread is busy does not scroll or begin a gesture in
+ * headless Chromium, even with only passive listeners, and an animation that
+ * has not started cannot start (see `waitForAnimation`). Finger tracking
+ * under a busy main thread is a simulator proof, not a Playwright one.
+ *
+ * With `during`, the page itself waits until an animation on that element (or
+ * inside it) has run `minProgressMs`, then blocks from a task it queues in
+ * that frame. A round trip from the test would start the block later the
+ * busier the machine is, leaving less of a short animation to judge. The task
+ * runs after the frame's rendering update, which is where WebKit hands a newly
+ * started animation to its compositor: a loop inside the rAF callback itself
+ * holds that hand-off, and even a compositor animation freezes. To catch a
+ * settle near its start, call it before the release that starts the settle and
+ * await it after: the page is already waiting when the animation begins. Any
+ * animation inside `during` counts, so pick an element that runs nothing
+ * else meanwhile.
+ *
+ * An animation counts once it is no longer pending and has run at least one
+ * frame (`minProgressMs` below 16 ms is raised to 16). One created in the
+ * frame the page looks is still pending, not yet on the compositor
+ * (Chromium), and in WebKit its start time resolves before its layer is
+ * flushed: a block that started on it froze it.
+ */
+/** One display frame, ms: the least progress `blockMainThread` waits for. */
+const ONE_FRAME_MS = 16;
+
+export async function blockMainThread(
+ page: Page,
+ ms: number,
+ {
+ during,
+ minProgressMs = 16,
+ timeout = 5_000,
+ }: {during?: Locator; minProgressMs?: number; timeout?: number} = {},
+): Promise<BusyWindow> {
+ const minProgress = Math.max(minProgressMs, ONE_FRAME_MS);
+ if (during == null) {
+ return page.evaluate(duration => {
+ const from = performance.timeOrigin + performance.now();
+ const end = performance.now() + duration;
+ let spins = 0;
+ while (performance.now() < end) {
+ spins += 1;
+ }
+ return {from, to: performance.timeOrigin + performance.now(), spins};
+ }, ms);
+ }
+ return during.evaluate(
+ (element, {duration, minProgress, wait}) =>
+ new Promise<BusyWindow>((resolve, reject) => {
+ // The harness's own waits run on the real timers: a frozen timer gate
+ // (story.ts) holds only the page's.
+ const gate = (
+ window as unknown as {
+ __harnessTimerGate?: {
+ frame?: typeof requestAnimationFrame;
+ timeout?: typeof setTimeout;
+ };
+ }
+ ).__harnessTimerGate;
+ const frame = gate?.frame ?? window.requestAnimationFrame.bind(window);
+ const later = gate?.timeout ?? window.setTimeout.bind(window);
+ const deadline = performance.now() + wait;
+ const tick = () => {
+ const running = element
+ .getAnimations({subtree: true})
+ .some(
+ a =>
+ a.playState === 'running' &&
+ !a.pending &&
+ Number(a.currentTime ?? 0) >= minProgress,
+ );
+ if (running) {
+ // The same loop as above, from a task queued in the frame that saw
+ // the animation: it runs once that frame's rendering update has
+ // handed the animation to the compositor.
+ later(() => {
+ const from = performance.timeOrigin + performance.now();
+ const end = performance.now() + duration;
+ while (performance.now() < end) {
+ // Busy: nothing else runs.
+ }
+ resolve({from, to: performance.timeOrigin + performance.now()});
+ }, 0);
+ } else if (performance.now() > deadline) {
+ reject(new Error('blockMainThread: no animation started'));
+ } else {
+ frame(tick);
+ }
+ };
+ frame(tick);
+ }),
+ {duration: ms, minProgress, wait: timeout},
+ );
+}
+
+export interface BusyVerdict {
+ readonly pass: boolean;
+ /** Each criterion the window missed, with its measured value and limit. */
+ readonly failures: readonly string[];
+}
+
+export interface BusyProof extends FrameStats {
+ readonly window: BusyWindow;
+ /**
+ * The part of the window judged: inside the margins, up to `settledWithin`,
+ * and from `onScreenFromMs` when that is set.
+ */
+ readonly judged: {readonly from: number; readonly to: number};
+ /**
+ * When the tracked value was seen at rest before the judged part (a frame
+ * there, no change in the 250 ms before it: a just-started animation not on
+ * screen yet), ms after the window's start of its first presented change
+ * inside the block, where judging starts; null otherwise.
+ */
+ readonly onScreenFromMs: number | null;
+ /**
+ * The cast's position step, CSS px: the largest of 0.5, 1 and 2 that every
+ * tracked position is a multiple of (WebKit's casts are 2: see the README).
+ */
+ readonly stepPx: number;
+ /** The tracked value's mean motion per screencast interval in the judged part, px. */
+ readonly pxPerFrame: number;
+ /** Facts worth reading beside the verdict, which never fail it. */
+ readonly notes: readonly string[];
+ /**
+ * With a compositor record (`presented`): frames the screencast skipped
+ * between two moved frames while the compositor presented them, which are
+ * neither stalls nor misses; 0 without one.
+ */
+ readonly recorderSkipped: number;
+ /**
+ * The compositor's frame interval around the window, ms, from `presented`;
+ * null without a usable record (the screencast alone is judged).
+ */
+ readonly compositorFrameMs: number | null;
+ /**
+ * The screencast's frame interval, from the frames around the window as well
+ * as inside it (a short window has few intervals; a frozen one has none).
+ */
+ readonly cadenceMs: number;
+ /**
+ * Whole frame intervals in the judged part at the measured cadence (the
+ * compositor's, when recorded): the `minJudgedFrames` floor reads it.
+ */
+ readonly expectedFrames: number;
+ /** Frames the screencast delivered in the judged part, with the marker found. */
+ readonly deliveredFrames: number;
+ /**
+ * Longest time inside the window with no presented change, the window's own
+ * edges included. A frozen animation presents no new frame at all (nothing
+ * on screen changed, so the compositor sends none): its stall is the whole
+ * window.
+ */
+ readonly longestStallMs: number;
+ /** Where the longest stall starts, ms after the window's start. */
+ readonly longestStallAtMs: number;
+ /**
+ * `longestStallMs` in screencast intervals, rounded: 1 is no frame missed,
+ * 2 is one dropped frame. Rounding absorbs timestamp jitter.
+ */
+ readonly longestStallFrames: number;
+ /**
+ * Frames in the window that presented a new position (the first compared
+ * with the last frame before the window).
+ */
+ readonly moved: number;
+ /**
+ * `moved` (and `recorderSkipped`) over the frames judged: the frames the
+ * screencast delivered in the judged part, or with a compositor record the
+ * compositor's frames from the first to the last moved frame;
+ * `expectedFrames` when nothing moved, or fewer than `minJudgedFrames`
+ * frames were delivered.
+ */
+ readonly movedFraction: number;
+ /** The judgment against the limits (see `busyWindowProof`). */
+ readonly verdict: BusyVerdict;
+ /** One line with the raw numbers, for an annotation or a failure message. */
+ readonly summary: string;
+}
+
+/**
+ * Frame evidence for a busy window: how many frames the screencast presented
+ * inside it, how many of them moved the tracked axis, and the longest stall.
+ * The window is trimmed by `marginMs` at both ends so the frame the block
+ * started on and the one it ended on are not counted.
+ *
+ * The verdict judges the frames the screencast delivered, whatever its rate
+ * (a loaded machine's screencast can deliver one frame in five or six), and
+ * tolerates nothing a main-thread animation does (it presents no frame for
+ * the whole block). It passes when:
+ * - the judged part presented at least one frame, and, only when a proof
+ * claims display rate and passes `maxMedianFrameMs`, their median interval
+ * is at most that;
+ * - no stall spans more than `maxStallFrames` screencast intervals;
+ * - at least `minMovedFraction` of the frames judged moved the tracked axis
+ * (a short window may still miss one): the frames the screencast delivered
+ * in the judged part, so an irregular cast is judged on what it delivered
+ * and a frame it did not deliver is a gap for the stall check, not a miss
+ * (unless it delivered fewer than `minJudgedFrames` there: then every frame
+ * of the judged part's span counts, so one late frame is not 1/1 moved);
+ * with a compositor record, the compositor's frames from the first to the
+ * last moved frame (the gaps before and after are the stall check's);
+ * - the judged part spans at least `minJudgedFrames` frames. Fewer means the
+ * animation under test was not moving during the block (it settled before
+ * the block, or the block caught another animation), so there is nothing to
+ * judge, and that fails rather than passes.
+ *
+ * A just-started animation can reach the screen after the block began (its
+ * hand-off to the compositor lags under load: 50–100 ms in WebKit). When the
+ * tracked value was seen at rest before the judged part (a frame there, and no
+ * change in the 250 ms before it), judging starts at its first presented
+ * change inside the block, which the
+ * proof reports as `onScreenFromMs`. The judged part still ends with the
+ * block and the `minJudgedFrames` floor applies to it, so a start the block
+ * itself delays fails; an animation that was already moving is judged from
+ * the block's start, and a stall there fails as before.
+ *
+ * `stepPx` and `pxPerFrame` say whether a frame's motion is resolvable at
+ * all: under two position steps per frame, a frame delivered early can show
+ * no step and read as not moved, which the proof notes (never a failure).
+ *
+ * A loaded machine's screencast can skip frames the compositor still presents
+ * (Chromium). Given the compositor's record (`presented`: when it presented
+ * each frame), a gap between two moved frames with no frame
+ * delivered inside it is measured between the compositor's own frames: it is
+ * a stall only where the compositor presented nothing, and the frames it did
+ * present there are neither stalls nor misses (`recorderSkipped`). A gap
+ * that opens or closes the judged part, or holds a delivered frame that did
+ * not move, is judged from the screencast as before, so a frozen element
+ * fails whatever else the compositor draws.
+ */
+export function busyWindowProof(
+ tracked: readonly TrackedFrame[],
+ window: BusyWindow,
+ {
+ axis = 'y',
+ marginMs = 20,
+ settledWithin,
+ settledAt,
+ maxMedianFrameMs = Infinity,
+ maxStallFrames = 3,
+ minMovedFraction = 0.9,
+ minJudgedFrames = 4,
+ presented,
+ }: {
+ axis?: 'x' | 'y' | 'width' | 'height';
+ marginMs?: number;
+ /**
+ * Opt-in, for a proof that claims display rate: the most the median frame
+ * interval may be, ms (20 at 60 Hz). Off by default, since it measures the
+ * screencast's own rate as much as the animation.
+ */
+ maxMedianFrameMs?: number;
+ /** The most screencast intervals one stall may span (3 = two dropped frames). */
+ maxStallFrames?: number;
+ /**
+ * The least fraction of the expected frames that must move. One missed
+ * frame is always allowed, so a window of under ten frames is not judged
+ * stricter than a long one.
+ */
+ minMovedFraction?: number;
+ /** The fewest whole frame intervals the judged part of the window may span. */
+ minJudgedFrames?: number;
+ /**
+ * Stop judging once the value is within this many px of where it ends.
+ * The screencast reports whole CSS pixels, so an easing tail that still
+ * moves shows plateaus (26, 26, 26 against a final 24) that would read
+ * as a stall: 2 is the usual value.
+ */
+ settledWithin?: number;
+ /**
+ * Where the value ends, for `settledWithin`: read it from the page once the
+ * animation has ended (`waitForRest`). Defaults to the last frame's value,
+ * which can be stale: once an accelerated animation ends while the main
+ * thread is still blocked, WebKit presents the layer's last committed
+ * position until the main thread commits the end state.
+ */
+ settledAt?: number;
+ /**
+ * When the compositor presented each frame, epoch ms on the page clock
+ * (Chromium's `Display::FrameDisplayed`). Empty or absent, the screencast
+ * is judged alone.
+ */
+ presented?: readonly number[];
+ } = {},
+): BusyProof {
+ let end = window.to - marginMs;
+ if (settledWithin != null) {
+ const valued = tracked.flatMap(f =>
+ f.box == null ? [] : [{t: f.t, value: f.box[axis]}],
+ );
+ const final =
+ settledAt ?? (valued.length > 0 ? valued[valued.length - 1].value : null);
+ const arrived = valued.find(
+ f =>
+ f.t >= window.from &&
+ final != null &&
+ Math.abs(f.value - final) <= settledWithin,
+ );
+ if (arrived != null) {
+ end = Math.min(end, arrived.t);
+ }
+ }
+ const inner = {from: window.from + marginMs, to: end};
+ const values = tracked
+ .filter(f => f.t >= inner.from && f.t <= inner.to)
+ .map(f => ({t: f.t, value: f.box == null ? null : f.box[axis]}));
+ const before = [...tracked].reverse().find(f => f.t < inner.from);
+ let previous: number | null = before?.box == null ? null : before.box[axis];
+ const inWindow: number[] = [];
+ for (const v of values) {
+ if (
+ v.value != null &&
+ previous != null &&
+ Math.abs(v.value - previous) > 0.25
+ ) {
+ inWindow.push(v.t);
+ }
+ if (v.value != null) {
+ previous = v.value;
+ }
+ }
+ // Seen at rest before the judged part (a frame there, and no change in the
+ // 250 ms before it): the first presented change is the animation's on-screen
+ // start, and judging starts there. With no frame before, nothing is known,
+ // and the window is judged from its start.
+ const prior = tracked.filter(f => f.box != null && f.t < inner.from);
+ let leadFrom = 0;
+ prior.forEach((f, i) => {
+ if (f.t < inner.from - 250) {
+ leadFrom = i;
+ }
+ });
+ const lead = prior.flatMap(f => (f.box == null ? [] : [f.box[axis]]));
+ const movedBefore = lead.some(
+ (value, i) => i > leadFrom && Math.abs(value - lead[i - 1]) > 0.25,
+ );
+ const onScreenFrom =
+ prior.length > 0 && !movedBefore && inWindow.length > 0
+ ? inWindow[0]
+ : null;
+ const judged = {from: onScreenFrom ?? inner.from, to: inner.to};
+ const changes = inWindow.filter(t => t > judged.from || onScreenFrom == null);
+ const edges = [judged.from, ...changes, judged.to];
+ const around = tracked
+ .filter(f => f.t >= inner.from - 250 && f.t <= inner.to + 250)
+ .map(f => f.t);
+ const intervals = around
+ .slice(1)
+ .map((t, i) => t - around[i])
+ .sort((a, b) => a - b);
+ const stats = frameStats(
+ tracked.map(f => ({t: f.t, value: f.box == null ? null : f.box[axis]})),
+ judged,
+ );
+ const cadence =
+ intervals.length >= 4
+ ? intervals[Math.floor(intervals.length / 2)]
+ : stats.medianFrameMs || 1000 / 60;
+ // The compositor's own frames, when recorded: its interval, from the frames
+ // around the judged part.
+ const draws = [...(presented ?? [])]
+ .filter(t => t >= judged.from - 250 && t <= judged.to + 250)
+ .sort((a, b) => a - b);
+ const drawIntervals = draws
+ .slice(1)
+ .map((t, i) => t - draws[i])
+ .sort((a, b) => a - b);
+ const compositorFrameMs =
+ drawIntervals.length >= 4
+ ? drawIntervals[Math.floor(drawIntervals.length / 2)]
+ : null;
+ const changed = new Set(inWindow);
+ const delivered = tracked
+ .filter(f => f.box != null && f.t > judged.from && f.t < judged.to)
+ .map(f => f.t);
+ // Each gap between presented changes, in intervals of the channel that saw
+ // it: the screencast's, or, for a gap between two moved frames with no
+ // frame delivered inside it, the compositor's own longest interval there.
+ const gaps = edges.slice(1).map((to, i) => {
+ const from = edges[i];
+ const ms = to - from;
+ if (
+ compositorFrameMs == null ||
+ !changed.has(from) ||
+ !changed.has(to) ||
+ delivered.some(t => t > from && t < to)
+ ) {
+ return {from, ms, frames: Math.round(ms / cadence), skipped: 0};
+ }
+ const span = draws.filter(
+ t => t > from - compositorFrameMs / 2 && t < to + compositorFrameMs / 2,
+ );
+ const longest =
+ span.length >= 2
+ ? Math.min(ms, Math.max(...span.slice(1).map((t, j) => t - span[j])))
+ : ms;
+ const frames = Math.round(longest / compositorFrameMs);
+ return {
+ from,
+ ms: longest,
+ frames,
+ skipped: Math.max(0, Math.round(ms / compositorFrameMs) - frames),
+ };
+ });
+ const longestGap = gaps.reduce((a, b) =>
+ b.frames > a.frames || (b.frames === a.frames && b.ms > a.ms) ? b : a,
+ );
+ const longestStallMs = longestGap.ms;
+ // Where the longest stall starts, from the start of the window.
+ const longestStallAtMs = longestGap.from - window.from;
+ const longestStallFrames = longestGap.frames;
+ const skips = gaps.filter(g => g.skipped > 0);
+ const recorderSkipped = skips.reduce((sum, g) => sum + g.skipped, 0);
+ // With a compositor record, frames count in its intervals, and the frames
+ // it presented where the screencast skipped are presented frames.
+ const expectedFrames = Math.floor(
+ (judged.to - judged.from) / (compositorFrameMs ?? cadence),
+ );
+ const moved = changes.length;
+ // The frames judged for motion: the screencast's own frames in the judged
+ // part (each `changes` entry is one of them), or, with a compositor record,
+ // the compositor's frames from the first to the last moved frame (the gaps
+ // before and after them are the stall check's). After an on-screen start
+ // they count from the on-screen frame, which `changes` leaves out, so the
+ // frames credited as skipped after it are judged too. A part that moved
+ // nothing, or delivered fewer than `minJudgedFrames` frames, is judged
+ // against its span, so a frozen animation misses here too (one late frame
+ // from before the block is not 1/1 moved).
+ const deliveredFrames = tracked.filter(
+ f =>
+ f.box != null &&
+ (onScreenFrom == null ? f.t >= judged.from : f.t > judged.from) &&
+ f.t <= judged.to,
+ ).length;
+ const judgedFrames =
+ compositorFrameMs != null
+ ? changes.length > 0
+ ? Math.round(
+ (changes[changes.length - 1] - (onScreenFrom ?? changes[0])) /
+ compositorFrameMs,
+ ) + (onScreenFrom == null ? 1 : 0)
+ : expectedFrames
+ : deliveredFrames < minJudgedFrames
+ ? expectedFrames
+ : deliveredFrames;
+ const movedFraction = (moved + recorderSkipped) / Math.max(1, judgedFrames);
+ // The cast's position step, and the motion per interval it has to resolve.
+ const positions = tracked.flatMap(f => (f.box == null ? [] : [f.box[axis]]));
+ const stepPx =
+ [2, 1, 0.5].find(step =>
+ positions.every(p => Math.abs(p / step - Math.round(p / step)) < 0.01),
+ ) ?? 0.5;
+ const judgedValues = tracked.flatMap(f =>
+ f.box != null && f.t >= judged.from && f.t <= judged.to
+ ? [{t: f.t, value: f.box[axis]}]
+ : [],
+ );
+ const first = judgedValues[0];
+ const last = judgedValues[judgedValues.length - 1];
+ const pxPerFrame =
+ first == null || last == null || last.t <= first.t
+ ? 0
+ : Math.abs(last.value - first.value) / ((last.t - first.t) / cadence);
+ const notes: string[] = [];
+ if (presented != null && compositorFrameMs == null) {
+ notes.push(
+ 'no compositor frames recorded around the window: judged from the screencast alone',
+ );
+ }
+ if (recorderSkipped > 0) {
+ notes.push(
+ `the screencast skipped ${recorderSkipped} frames the compositor presented (at ${skips.map(g => `+${(g.from - window.from).toFixed(0)} ms`).join(', ')})`,
+ );
+ }
+ if (pxPerFrame > 0 && pxPerFrame < 2 * stepPx) {
+ notes.push(
+ `${pxPerFrame.toFixed(1)} px/frame is under two of the cast's ${stepPx} px position steps: a frame delivered early can show no step and read as not moved`,
+ );
+ }
+ const failures: string[] = [];
+ if (stats.frames === 0) {
+ failures.push('no frame presented in the judged part');
+ } else if (stats.medianFrameMs > maxMedianFrameMs) {
+ failures.push(
+ `median frame interval ${stats.medianFrameMs.toFixed(1)} ms > ${maxMedianFrameMs} ms (${stats.frames} frames presented)`,
+ );
+ }
+ if (longestStallFrames > maxStallFrames) {
+ failures.push(
+ `longest stall ${longestStallMs.toFixed(1)} ms = ${longestStallFrames} intervals > ${maxStallFrames}`,
+ );
+ }
+ if (expectedFrames < minJudgedFrames) {
+ failures.push(
+ `${(judged.to - judged.from).toFixed(0)} ms judged = ${expectedFrames} frames < ${minJudgedFrames}: the animation under test was not moving during the block`,
+ );
+ }
+ const allowedMisses = Math.max(
+ 1,
+ Math.floor((1 - minMovedFraction) * judgedFrames + 1e-9),
+ );
+ if (judgedFrames - moved - recorderSkipped > allowedMisses) {
+ failures.push(
+ `${moved} of ${judgedFrames} expected frames moved${recorderSkipped > 0 ? ` (and ${recorderSkipped} the screencast skipped)` : ''} (${(movedFraction * 100).toFixed(0)}% < ${(minMovedFraction * 100).toFixed(0)}%)`,
+ );
+ }
+ const onScreenFromMs =
+ onScreenFrom == null ? null : onScreenFrom - window.from;
+ const summary =
+ `${(judged.to - judged.from).toFixed(0)} ms judged at ${cadence.toFixed(1)} ms/frame: ${stats.frames} frames presented, ` +
+ `${moved}/${judgedFrames} moved${recorderSkipped > 0 ? ` and ${recorderSkipped} skipped by the screencast` : ''} (${(movedFraction * 100).toFixed(0)}%), ` +
+ `median ${stats.medianFrameMs.toFixed(1)} ms, ` +
+ `longest stall ${longestStallMs.toFixed(1)} ms (${longestStallFrames} intervals) at +${longestStallAtMs.toFixed(0)} ms, ` +
+ `${stats.monotonic ? 'monotonic' : 'not monotonic'}, ` +
+ `${pxPerFrame.toFixed(1)} px/frame in ${stepPx} px steps` +
+ (compositorFrameMs == null
+ ? ''
+ : `, compositor presenting every ${compositorFrameMs.toFixed(1)} ms`) +
+ (onScreenFromMs == null
+ ? ''
+ : `, on screen from +${onScreenFromMs.toFixed(0)} ms (nothing moved before the block)`) +
+ notes.map(note => `; note: ${note}`).join('');
+ return {
+ ...stats,
+ window,
+ judged,
+ onScreenFromMs,
+ stepPx,
+ pxPerFrame,
+ notes,
+ recorderSkipped,
+ compositorFrameMs,
+ cadenceMs: cadence,
+ longestStallMs,
+ longestStallAtMs,
+ expectedFrames,
+ deliveredFrames,
+ longestStallFrames,
+ moved,
+ movedFraction,
+ verdict: {pass: failures.length === 0, failures},
+ summary,
+ };
+}
diff --git a/internal/interaction-harness/src/frames.ts b/internal/interaction-harness/src/frames.ts
new file mode 100644
index 0000000000..c61ea7e171
--- /dev/null
+++ b/internal/interaction-harness/src/frames.ts
@@ -0,0 +1,465 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file frames.ts
+ * @input Uses a Playwright `Page`, a `Locator` for the element under test, and
+ * the engine's screencast (Chromium CDP `Page.startScreencast`; WebKit's
+ * `Screencast` domain)
+ * @output `startScreencast` (timestamped composited frames), `markElement`
+ * (a marker outline the frames can find), `trackMarker` (per-frame marker
+ * box), `frameStats`.
+ * @position Frame evidence for motion proofs. The screencast is captured off
+ * the page's main thread, from what the compositor presents, so it keeps
+ * delivering frames while a script blocks the main thread: that is what makes
+ * the busy-main-thread proof possible.
+ *
+ * Marker tracking needs no image library: frames are decoded and scanned in a
+ * scratch page of the same browser (createImageBitmap + canvas), so the
+ * harness adds no dependency.
+ *
+ * SYNC: Keep the README's frame-evidence section equal to what this measures.
+ */
+
+import type {CDPSession, Locator, Page} from '@playwright/test';
+import {webkitPageSession} from './touch';
+
+export interface CastFrame {
+ /** Presentation time, epoch ms (the compositor's frame timestamp). */
+ readonly t: number;
+ /** Base64 PNG/JPEG. */
+ readonly data: string;
+ readonly format: 'png' | 'jpeg';
+ /** CSS px of the viewport the frame shows. */
+ readonly viewportWidth: number;
+ readonly viewportHeight: number;
+}
+
+export interface Screencast {
+ /**
+ * Frames captured so far: wait for the first one to know the cast saw the
+ * page before an animation starts (`busyWindowProof` then knows its at-rest
+ * position).
+ */
+ frameCount(): number;
+ /** Stops and returns every frame captured, in order. */
+ stop(): Promise<CastFrame[]>;
+}
+
+export interface ScreencastOptions {
+ /** Lossless frames by default: marker colours survive PNG, not JPEG. */
+ readonly format?: 'png' | 'jpeg';
+ readonly quality?: number;
+ /**
+ * Keep one presented frame in this many (default 1, every frame): a slower
+ * evidence channel, like a loaded machine's screencast, for proving that a
+ * verdict does not depend on the screencast's rate.
+ */
+ readonly everyNthFrame?: number;
+}
+
+/**
+ * Starts capturing composited frames. Chromium delivers one per presented
+ * frame (16.7 ms at 60 Hz), timestamped by the compositor; WebKit's own
+ * screencast delivers one per composited update (about every 14 ms in the
+ * Linux build), stamped on CLOCK_MONOTONIC and rebased here onto the wall
+ * clock by that clock's measured offset, so both engines share the page's
+ * timeline. Both keep delivering while the page's main thread is blocked, and
+ * neither sends a frame when nothing on screen changed.
+ */
+export async function startScreencast(
+ page: Page,
+ options: ScreencastOptions = {},
+): Promise<Screencast> {
+ const name = page.context().browser()?.browserType().name();
+ const viewport = page.viewportSize() ?? {width: 0, height: 0};
+ if (name === 'chromium') {
+ return chromiumScreencast(page, viewport, options);
+ }
+ if (name === 'webkit') {
+ return webkitScreencast(page, viewport, options);
+ }
+ throw new Error(
+ `startScreencast: unsupported browser ${name ?? '(unknown)'}`,
+ );
+}
+
+async function chromiumScreencast(
+ page: Page,
+ viewport: {width: number; height: number},
+ {format = 'png', quality = 90, everyNthFrame = 1}: ScreencastOptions,
+): Promise<Screencast> {
+ const cdp: CDPSession = await page.context().newCDPSession(page);
+ const frames: CastFrame[] = [];
+ const onFrame = (frame: {
+ data: string;
+ sessionId: number;
+ metadata: {timestamp?: number; deviceWidth: number; deviceHeight: number};
+ }) => {
+ frames.push({
+ t: (frame.metadata.timestamp ?? Date.now() / 1000) * 1000,
+ data: frame.data,
+ format,
+ viewportWidth: frame.metadata.deviceWidth || viewport.width,
+ viewportHeight: frame.metadata.deviceHeight || viewport.height,
+ });
+ cdp
+ .send('Page.screencastFrameAck', {sessionId: frame.sessionId})
+ .catch(() => {});
+ };
+ cdp.on('Page.screencastFrame', onFrame);
+ // One CSS px per frame px keeps decoding cheap; everyNthFrame 1 = every presented frame.
+ await cdp.send('Page.startScreencast', {
+ format,
+ quality,
+ maxWidth: viewport.width,
+ maxHeight: viewport.height,
+ everyNthFrame,
+ });
+ return {
+ frameCount: () => frames.length,
+ async stop() {
+ await cdp.send('Page.stopScreencast');
+ cdp.off('Page.screencastFrame', onFrame);
+ await cdp.detach().catch(() => {});
+ return inPresentationOrder(frames);
+ },
+ };
+}
+
+async function webkitScreencast(
+ page: Page,
+ viewport: {width: number; height: number},
+ {quality = 90, everyNthFrame = 1}: ScreencastOptions,
+): Promise<Screencast> {
+ const session = webkitPageSession(page);
+ const frames: CastFrame[] = [];
+ let generation = 0;
+ let delivered = 0;
+ const onFrame = (payload: unknown) => {
+ const frame = payload as {data: string; timestamp: number};
+ // WebKit's screencast has no everyNthFrame: keep one frame in N here.
+ delivered += 1;
+ if ((delivered - 1) % everyNthFrame !== 0) {
+ session
+ .send('Screencast.screencastFrameAck', {generation})
+ .catch(() => {});
+ return;
+ }
+ frames.push({
+ // WebKit stamps frames in seconds on the monotonic clock; rebased below.
+ t: frame.timestamp * 1000,
+ data: frame.data,
+ format: 'jpeg',
+ viewportWidth: viewport.width,
+ viewportHeight: viewport.height,
+ });
+ session.send('Screencast.screencastFrameAck', {generation}).catch(() => {});
+ };
+ session.on('Screencast.screencastFrame', onFrame);
+ const started = (await session.send('Screencast.startScreencast', {
+ width: viewport.width,
+ height: viewport.height,
+ toolbarHeight: 0,
+ quality,
+ })) as {generation: number};
+ generation = started.generation;
+ // WebKit stamps frames in seconds on CLOCK_MONOTONIC, which this process
+ // shares: rebase them by the wall clock's offset from that clock, sampled
+ // together. (Taking the first frame as the start shifted every frame earlier
+ // by however long that frame took to come, 60–100 ms on a busy page.)
+ const wallMinusMonotonic = Date.now() - Number(process.hrtime.bigint()) / 1e6;
+ return {
+ frameCount: () => frames.length,
+ async stop() {
+ await session.send('Screencast.stopScreencast');
+ session.off?.('Screencast.screencastFrame', onFrame);
+ if (frames.length === 0) {
+ return frames;
+ }
+ return inPresentationOrder(frames).map(f => ({
+ ...f,
+ t: f.t + wallMinusMonotonic,
+ }));
+ },
+ };
+}
+
+/**
+ * Frames sorted by their presentation timestamp. A loaded machine can deliver
+ * screencast frames out of order (a frame stamped 35 ms arriving after one
+ * stamped 46 ms); judged in arrival order, a steadily moving element would
+ * appear to step backwards.
+ */
+export function inPresentationOrder<T extends {readonly t: number}>(
+ frames: readonly T[],
+): T[] {
+ return [...frames].sort((a, b) => a.t - b.t);
+}
+
+/** The marker colour: magenta never occurs in Astryx themes. */
+export const MARKER_RGB: readonly [number, number, number] = [255, 0, 255];
+
+export interface MarkerOptions {
+ readonly rgb?: readonly [number, number, number];
+ /**
+ * `chip` (default): a small square child pinned inside the element's
+ * top-left corner, above every sibling (z-index max), so nothing the
+ * component paints can cover it. The element must be its containing block:
+ * a static element is made `position: relative` while marked.
+ * `outline`: an inset outline on the element itself (no DOM insertion), for
+ * elements that cannot take a child; children with their own layers can
+ * cover parts of it.
+ */
+ readonly mode?: 'chip' | 'outline';
+ /** Chip size or outline width, CSS px. */
+ readonly size?: number;
+ /** Chip inset from the element's top-left corner, CSS px. */
+ readonly inset?: number;
+}
+
+/**
+ * Marks the element so screencast frames can find it: the mark moves, scales
+ * and fades with the element's own layer and never changes its layout.
+ * Returns an undo.
+ */
+export async function markElement(
+ locator: Locator,
+ options: MarkerOptions = {},
+): Promise<() => Promise<void>> {
+ const {
+ rgb = MARKER_RGB,
+ mode = 'chip',
+ size = mode === 'chip' ? 16 : 6,
+ inset = 8,
+ } = options;
+ const id = `m${Math.random().toString(36).slice(2, 8)}`;
+ await locator.evaluate(
+ (element, {id: markerId, color, size: px, inset: offset, mode: kind}) => {
+ const el = element as HTMLElement;
+ el.setAttribute('data-harness-marker', markerId);
+ if (kind === 'outline') {
+ const style = document.createElement('style');
+ style.setAttribute('data-harness-marker-style', markerId);
+ style.textContent = `[data-harness-marker="${markerId}"]{outline:${px}px solid rgb(${color.join(',')}) !important;outline-offset:-${px}px !important;}`;
+ document.head.append(style);
+ return;
+ }
+ if (getComputedStyle(el).position === 'static') {
+ el.setAttribute('data-harness-marker-position', el.style.position);
+ el.style.position = 'relative';
+ }
+ const chip = document.createElement('div');
+ chip.setAttribute('data-harness-marker-chip', markerId);
+ chip.setAttribute('aria-hidden', 'true');
+ chip.style.cssText = `position:absolute;top:${offset}px;left:${offset}px;width:${px}px;height:${px}px;background:rgb(${color.join(',')});z-index:2147483647;pointer-events:none;`;
+ el.append(chip);
+ },
+ {id, color: [...rgb], size, inset, mode},
+ );
+ return async () => {
+ await locator.page().evaluate(markerId => {
+ const el = document.querySelector<HTMLElement>(
+ `[data-harness-marker="${markerId}"]`,
+ );
+ if (el?.hasAttribute('data-harness-marker-position')) {
+ el.style.position =
+ el.getAttribute('data-harness-marker-position') ?? '';
+ el.removeAttribute('data-harness-marker-position');
+ }
+ el?.removeAttribute('data-harness-marker');
+ document
+ .querySelector(`[data-harness-marker-chip="${markerId}"]`)
+ ?.remove();
+ document
+ .querySelector(`[data-harness-marker-style="${markerId}"]`)
+ ?.remove();
+ }, id);
+ };
+}
+
+export interface Box {
+ readonly x: number;
+ readonly y: number;
+ readonly width: number;
+ readonly height: number;
+}
+
+export interface TrackedFrame {
+ /** Epoch ms. */
+ readonly t: number;
+ /** The marker's bounding box in CSS px, or null when it is not on screen. */
+ readonly box: Box | null;
+ /** Marker pixels found (a fading element finds fewer at the same tolerance). */
+ readonly pixels: number;
+}
+
+/**
+ * Finds the marker in every frame. Decoding runs in a scratch page of the same
+ * browser, so neither the page under test nor Node needs an image decoder.
+ */
+export async function trackMarker(
+ page: Page,
+ frames: readonly CastFrame[],
+ {
+ rgb = MARKER_RGB,
+ tolerance = 40,
+ }: {rgb?: readonly [number, number, number]; tolerance?: number} = {},
+): Promise<TrackedFrame[]> {
+ const scratch = await page.context().newPage();
+ try {
+ const out: TrackedFrame[] = [];
+ // Batches keep each evaluate payload moderate.
+ for (let i = 0; i < frames.length; i += 12) {
+ const batch = frames.slice(i, i + 12);
+ const boxes = await scratch.evaluate(
+ async ({batch: items, color, tol}) => {
+ const results: {
+ box: {x: number; y: number; width: number; height: number} | null;
+ pixels: number;
+ }[] = [];
+ for (const item of items) {
+ const blob = await (
+ await fetch(`data:image/${item.format};base64,${item.data}`)
+ ).blob();
+ const bitmap = await createImageBitmap(blob);
+ const canvas = new OffscreenCanvas(bitmap.width, bitmap.height);
+ const context = canvas.getContext('2d');
+ if (context == null) {
+ throw new Error('no 2d context');
+ }
+ context.drawImage(bitmap, 0, 0);
+ const {data, width, height} = context.getImageData(
+ 0,
+ 0,
+ bitmap.width,
+ bitmap.height,
+ );
+ let minX = Infinity;
+ let minY = Infinity;
+ let maxX = -Infinity;
+ let maxY = -Infinity;
+ let pixels = 0;
+ for (let y = 0; y < height; y += 1) {
+ for (let x = 0; x < width; x += 1) {
+ const o = (y * width + x) * 4;
+ if (
+ Math.abs(data[o] - color[0]) <= tol &&
+ Math.abs(data[o + 1] - color[1]) <= tol &&
+ Math.abs(data[o + 2] - color[2]) <= tol
+ ) {
+ pixels += 1;
+ if (x < minX) {
+ minX = x;
+ }
+ if (x > maxX) {
+ maxX = x;
+ }
+ if (y < minY) {
+ minY = y;
+ }
+ if (y > maxY) {
+ maxY = y;
+ }
+ }
+ }
+ }
+ const scale = width / item.viewportWidth;
+ results.push({
+ box:
+ pixels === 0
+ ? null
+ : {
+ x: minX / scale,
+ y: minY / scale,
+ width: (maxX - minX + 1) / scale,
+ height: (maxY - minY + 1) / scale,
+ },
+ pixels,
+ });
+ }
+ return results;
+ },
+ {
+ batch: batch.map(f => ({
+ data: f.data,
+ format: f.format,
+ viewportWidth: f.viewportWidth,
+ })),
+ color: [...rgb],
+ tol: tolerance,
+ },
+ );
+ boxes.forEach((result, k) =>
+ out.push({t: batch[k].t, box: result.box, pixels: result.pixels}),
+ );
+ }
+ return out;
+ } finally {
+ await scratch.close();
+ }
+}
+
+export interface FrameStats {
+ readonly frames: number;
+ /** Frames whose tracked value differs from the frame before. */
+ readonly changed: number;
+ /** Longest stretch without a change inside the window (an open stall at its end included), ms. */
+ readonly longestStallMs: number;
+ /** Median spacing between frames, ms. */
+ readonly medianFrameMs: number;
+ readonly monotonic: boolean;
+ readonly first: number | null;
+ readonly last: number | null;
+}
+
+/** Summarises per-frame values (e.g. the marker's y) inside a time window. */
+export function frameStats(
+ samples: readonly {readonly t: number; readonly value: number | null}[],
+ window: {readonly from: number; readonly to: number} = {
+ from: -Infinity,
+ to: Infinity,
+ },
+ epsilon = 0.25,
+): FrameStats {
+ const inside = samples.filter(
+ s => s.t >= window.from && s.t <= window.to && s.value != null,
+ ) as {t: number; value: number}[];
+ let changed = 0;
+ let increasing = 0;
+ let decreasing = 0;
+ let stallStart = inside[0]?.t ?? 0;
+ let longestStallMs = 0;
+ for (let i = 1; i < inside.length; i += 1) {
+ const delta = inside[i].value - inside[i - 1].value;
+ if (Math.abs(delta) > epsilon) {
+ changed += 1;
+ if (delta > 0) {
+ increasing += 1;
+ } else {
+ decreasing += 1;
+ }
+ longestStallMs = Math.max(longestStallMs, inside[i].t - stallStart);
+ stallStart = inside[i].t;
+ }
+ }
+ // A stall still open when the window closes counts too: a frozen animation never changes at all.
+ if (inside.length > 0) {
+ longestStallMs = Math.max(
+ longestStallMs,
+ inside[inside.length - 1].t - stallStart,
+ );
+ }
+ const gaps = inside
+ .slice(1)
+ .map((s, i) => s.t - inside[i].t)
+ .sort((a, b) => a - b);
+ return {
+ frames: inside.length,
+ changed,
+ longestStallMs,
+ medianFrameMs: gaps.length ? gaps[Math.floor(gaps.length / 2)] : 0,
+ monotonic: increasing === 0 || decreasing === 0,
+ first: inside[0]?.value ?? null,
+ last: inside[inside.length - 1]?.value ?? null,
+ };
+}
diff --git a/internal/interaction-harness/src/harness.interaction.browser.spec.ts b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
index 7e47af95b2..a6c73fdf88 100644
--- a/internal/interaction-harness/src/harness.interaction.browser.spec.ts
+++ b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
@@ -3,24 +3,28 @@
/**
* @file harness.interaction.browser.spec.ts
* @input Fixture pages written inline (no Storybook), real Chromium/WebKit
- * @output Proof that each harness instrument measures what it claims:
- * scripted timestamps reach the page; a flick flings natively (Chromium);
- * a second finger and a pinch arrive as distinct pointers; sampled curves
- * match the oracles; the animation-state waits report a running transition
- * and its end, and judge rest by presented values even while an animation
- * holds its end, and a watch taken on first contact ignores an animation
- * already running and keeps the start of one the page cancels when it ends; individual transform properties are sampled; a sample at
+ * @output Proof that each harness instrument measures what it claims (the
+ * busy-main-thread proof with a negative control beside its positive
+ * check): scripted timestamps reach the
+ * page; a flick flings natively (Chromium); a second finger and a pinch
+ * arrive as distinct pointers; the screencast shows a compositor animation
+ * advancing through a blocked main thread while a rAF animation freezes;
+ * sampled curves
+ * match the oracles, individual transform properties included; a sample at
* an end reports the time it was read at; holding takes only running
* document-timeline animations, each at its own offset, holds each where it
* was read, adds a second hold to the held set without moving the first
- * hold's times, drops one the page cancels, and releases only them;
- * storyboards write a sheet and a trace that follows the curve, an endless
- * animation does not set their length, and a photograph Chromium returns
- * empty is taken again two frames later; a touch filmstrip keeps scripted
- * event times while a handler clock sees its pacing; the timer gate holds a
- * component timer until the filmstrip is done, drops callbacks the page
- * cancels, stamps held frames with the time they run, and leaves the
- * harness its own frames.
+ * hold's times, drops one the page cancels, and releases only them; the
+ * animation-state waits report a running transition and its end, and judge
+ * rest by presented values even while an animation holds its end, and a
+ * watch taken on first contact ignores an animation already running and
+ * keeps the start of one the page cancels when it ends; storyboards write a
+ * sheet and a trace that follows the curve, an endless animation does not
+ * set their length, and a photograph Chromium returns empty is taken again
+ * two frames later; a touch filmstrip keeps scripted event times while a
+ * handler clock sees its pacing; the timer gate holds a component timer
+ * until the filmstrip is done, drops callbacks the page cancels, stamps held
+ * frames with the time they run, and leaves the harness its own frames.
* @position The harness's own test. A component spec trusts these
* instruments; this file is why it may.
*/
@@ -28,6 +32,8 @@
import {expect, test, type Page} from '@playwright/test';
import {
animationStoryboard,
+ blockMainThread,
+ busyWindowProof,
compareCurve,
cubicBezier,
curveOf,
@@ -38,6 +44,7 @@ import {
holdAnimations,
installInputLog,
installTimerGate,
+ markElement,
merge,
parseTransform,
PARTIAL_LIFT_SKIP,
@@ -47,12 +54,14 @@ import {
sampleCurve,
seekAnimations,
springStep,
+ startScreencast,
Storyboard,
tap,
thawTimers,
timeline,
touchDriver,
touchStoryboard,
+ trackMarker,
waitForAnimation,
waitForRest,
watchAnimations,
@@ -71,6 +80,24 @@ const FIXTURE = `<!doctype html><meta name=viewport content="width=device-width,
<div id=box></div>
<div id=scroller>${Array.from({length: 200}, (_, i) => `<div class=row>row ${i}</div>`).join('')}</div>`;
+// The busy-main-thread proofs' page: one box with room to move 560 px down,
+// nothing over it. 560 px in 1,600 ms is about 4.8 px a frame, more than two
+// of WebKit's 2 px position steps (README: frame evidence), so every frame's
+// motion is resolvable.
+const BUSY_FIXTURE = `<!doctype html><meta name=viewport content="width=device-width,initial-scale=1">
+<style>
+ body { margin: 0; background: #fff; }
+ #box { position: absolute; left: 20px; top: 40px; width: 80px; height: 80px; background: #36c; }
+</style>
+<div id=box></div>`;
+
+async function busyFixture(page: Page) {
+ await page.route('http://busy.test/', route =>
+ route.fulfill({contentType: 'text/html', body: BUSY_FIXTURE}),
+ );
+ await page.goto('http://busy.test/');
+}
+
// A real navigation (not setContent, whose document.open() erases the window
// listeners the input log installs).
async function fixture(page: Page) {
@@ -242,6 +269,161 @@ test.describe('touch input', () => {
});
});
+test.describe('frame evidence', () => {
+ test('a compositor animation advances every frame through a blocked main thread; a rAF one freezes, whatever the screencast rate', async ({
+ page,
+ }, testInfo) => {
+ await busyFixture(page);
+ const box = page.locator('#box');
+ await markElement(box, {mode: 'outline'});
+
+ const run = async (driver: 'compositor' | 'raf', everyNthFrame = 1) => {
+ await box.evaluate((el, how) => {
+ el.getAnimations().forEach(a => a.cancel());
+ el.style.transform = 'none';
+ if (how === 'compositor') {
+ el.animate(
+ [{transform: 'translateY(0px)'}, {transform: 'translateY(560px)'}],
+ {duration: 1600, easing: 'linear'},
+ );
+ } else {
+ const start = performance.now();
+ const step = (now: number) => {
+ const p = Math.min(1, (now - start) / 1600);
+ el.style.transform = `translateY(${560 * p}px)`;
+ if (p < 1) {
+ requestAnimationFrame(step);
+ }
+ };
+ requestAnimationFrame(step);
+ }
+ }, driver);
+ const cast = await startScreencast(page, {everyNthFrame});
+ await page.waitForTimeout(200);
+ const window = await blockMainThread(page, 700);
+ await page.waitForTimeout(200);
+ const tracked = await trackMarker(page, await cast.stop());
+ return busyWindowProof(tracked, window, {axis: 'y'});
+ };
+
+ const compositor = await run('compositor');
+ testInfo.annotations.push({
+ type: 'busy-main-thread compositor',
+ description: compositor.summary,
+ });
+ expect(compositor.verdict.failures, compositor.summary).toEqual([]);
+ expect(compositor.monotonic).toBe(true);
+ // Resolvable on the cast's grid: two position steps or more per frame.
+ expect(compositor.notes, compositor.summary).toEqual([]);
+ expect(compositor.pxPerFrame).toBeGreaterThanOrEqual(2 * compositor.stepPx);
+
+ const raf = await run('raf');
+ testInfo.annotations.push({
+ type: 'busy-main-thread rAF control',
+ description: raf.summary,
+ });
+ // Frozen for the whole block: no presented frame moved (the compositor
+ // sends none at all when nothing on screen changes), and the verdict that
+ // passes the compositor animation fails it on every criterion.
+ expect(raf.changed).toBe(0);
+ expect(raf.longestStallMs).toBeGreaterThan(600);
+ expect(raf.verdict.pass).toBe(false);
+ expect(raf.verdict.failures.join('\n')).toMatch(/longest stall/);
+ expect(raf.verdict.failures.join('\n')).toMatch(/expected frames moved/);
+
+ // The verdict judges the frames the screencast delivered, not its rate:
+ // at one frame in six (a loaded machine's ~100 ms), the compositor
+ // animation still passes and the rAF one still fails.
+ const slow = await run('compositor', 6);
+ expect(slow.cadenceMs, slow.summary).toBeGreaterThan(60);
+ expect(slow.verdict.failures, slow.summary).toEqual([]);
+ const slowRaf = await run('raf', 6);
+ expect(slowRaf.verdict.pass, slowRaf.summary).toBe(false);
+ expect(slowRaf.verdict.failures.join('\n')).toMatch(/longest stall/);
+ });
+
+ test('a block that starts during a just-started transition still lets the compositor advance it', async ({
+ page,
+ }, testInfo) => {
+ // `during` blocks in the frame the transition has run 16 ms. In WebKit
+ // the transition reaches the compositor in that frame's rendering update,
+ // so a loop inside the rAF callback froze it (2 of 29 frames moved).
+ // Linear and longer than the block, so every frame of the block moves two
+ // position steps or more and the transition is still running when it
+ // ends. The cast shows the box at rest first, so a transition that reaches
+ // the screen after the block began is judged from there (onScreenFromMs).
+ await busyFixture(page);
+ const box = page.locator('#box');
+ await markElement(box, {mode: 'outline'});
+ const cast = await startScreencast(page);
+ await expect.poll(() => cast.frameCount()).toBeGreaterThan(0);
+ await box.evaluate(el => {
+ el.style.transition = 'transform 1600ms linear';
+ el.getBoundingClientRect();
+ el.style.transform = 'translateY(560px)';
+ });
+ const window = await blockMainThread(page, 700, {
+ during: box,
+ minProgressMs: 16,
+ });
+ await waitForRest(box);
+ const tracked = await trackMarker(page, await cast.stop());
+ const proof = busyWindowProof(tracked, window, {axis: 'y'});
+ testInfo.annotations.push({
+ type: 'busy-main-thread transition',
+ description: proof.summary,
+ });
+ expect(proof.verdict.failures, proof.summary).toEqual([]);
+ expect(proof.monotonic).toBe(true);
+ });
+
+ test('a block never starts on an animation that is not on the compositor yet', async ({
+ page,
+ }, testInfo) => {
+ // An animation created in the frame the helper looks is still pending (not
+ // committed to the compositor). A block that started on it then froze it
+ // in Chromium: no presented frame moved.
+ await busyFixture(page);
+ const box = page.locator('#box');
+ await markElement(box, {mode: 'outline'});
+ await box.evaluate(el => {
+ el.style.willChange = 'transform';
+ });
+ const cast = await startScreencast(page);
+ // A rAF loop registered before the helper's tick runs ahead of it in every
+ // frame; on its 12th frame it creates the animation, and the helper's tick
+ // looks in that same frame.
+ await box.evaluate(el => {
+ let n = 0;
+ const loop = () => {
+ n += 1;
+ if (n === 12) {
+ el.animate(
+ [{transform: 'translateY(0px)'}, {transform: 'translateY(560px)'}],
+ {duration: 1600, easing: 'linear'},
+ );
+ return;
+ }
+ requestAnimationFrame(loop);
+ };
+ requestAnimationFrame(loop);
+ });
+ const window = await blockMainThread(page, 700, {
+ during: box,
+ minProgressMs: 0,
+ });
+ await waitForRest(box);
+ const tracked = await trackMarker(page, await cast.stop());
+ const proof = busyWindowProof(tracked, window, {axis: 'y'});
+ testInfo.annotations.push({
+ type: 'busy-main-thread pending animation',
+ description: proof.summary,
+ });
+ expect(proof.verdict.failures, proof.summary).toEqual([]);
+ expect(proof.monotonic).toBe(true);
+ });
+});
+
test.describe('curve sampling', () => {
test('a CSS transition samples onto its cubic-bezier to 0.05 px', async ({
page,
@@ -1429,6 +1611,33 @@ test.describe('storyboards', {tag: '@storyboard'}, () => {
}
});
+ test("the harness's own waits run on the real timers: a frozen timer gate does not hold blockMainThread's block", async ({
+ page,
+ }) => {
+ await fixture(page);
+ await installTimerGate(page);
+ const box = page.locator('#box');
+ await box.evaluate(el => {
+ el.animate(
+ [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}],
+ {duration: 1_500, easing: 'linear'},
+ );
+ });
+ await freezeTimers(page);
+ // Settled, or still held after 5 s.
+ const within = (work: Promise<unknown>) =>
+ Promise.race([
+ work.then(() => 'settled'),
+ new Promise<string>(resolve =>
+ setTimeout(() => resolve('held'), 5_000),
+ ),
+ ]);
+ expect(
+ await within(blockMainThread(page, 100, {during: box, timeout: 2_000})),
+ ).toBe('settled');
+ await thawTimers(page);
+ });
+
test('the timer gate holds a component timer until the filmstrip is done', async ({
page,
}, testInfo) => {
diff --git a/internal/interaction-harness/src/index.ts b/internal/interaction-harness/src/index.ts
index 537bfc4d4b..2bcb127f36 100644
--- a/internal/interaction-harness/src/index.ts
+++ b/internal/interaction-harness/src/index.ts
@@ -12,7 +12,9 @@
* SYNC: When a module is added, export it here and describe it in README.md.
*/
+export * from './busy';
export * from './curves';
+export * from './frames';
export * from './math';
export * from './output';
export * from './scripts';
diff --git a/internal/interaction-harness/src/story.ts b/internal/interaction-harness/src/story.ts
index 598f75d852..5a1f2237c6 100644
--- a/internal/interaction-harness/src/story.ts
+++ b/internal/interaction-harness/src/story.ts
@@ -119,7 +119,7 @@ const TIMER_GATE = `(() => {
const realInterval = window.setInterval.bind(window);
const realFrame = window.requestAnimationFrame.bind(window);
const realCancelFrame = window.cancelAnimationFrame.bind(window);
- const gate = {frozen: false, queue: [], frame: realFrame};
+ const gate = {frozen: false, queue: [], frame: realFrame, timeout: realTimeout};
window.__harnessTimerGate = gate;
const drop = key => { gate.queue = gate.queue.filter(entry => entry.key !== key); };
window.setTimeout = (fn, ms, ...args) => {