An open source design system that's fully customizable and agent ready
TypeScript
test(interaction): frame evidence (#8045)
Christopher Chedeau6ecbf415d4e9
diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.mdindex 31c5149c2d..4c67256a26 100644--- a/internal/interaction-harness/README.md+++ b/internal/interaction-harness/README.md@@ -1,20 +1,23 @@# @astryxdesign/interaction-harnessPlaywright instruments for touch interaction and animation proofs: real touch-input, curve sampling and storyboards. **Internal and unpublished.** Specs live-next to their component as `*.interaction.browser.spec.ts` and run under+input, frame evidence, curve sampling and storyboards. **Internal and+unpublished.** Specs live next to their component as+`*.interaction.browser.spec.ts` and run under[`playwright.interaction.config.ts`](../../playwright.interaction.config.ts) ona phone-sized touch screen (402 × 714 CSS px page, 402 × 874 screen, 3 devicepixels per CSS px, touch, mobile viewport rules), in Chromium and WebKit.The harness proves its own instruments in-[`src/harness.interaction.browser.spec.ts`](src/harness.interaction.browser.spec.ts),-on fixture pages written inline.+[`src/harness.interaction.browser.spec.ts`](src/harness.interaction.browser.spec.ts).+The busy-main-thread proof has a negative control beside its positive check+(a rAF animation that freezes under a blocked main thread); other self-tests+check an instrument against a known answer.## Running```sh-pnpm storybook:build # component specs read stories from the built Storybook+pnpm storybook:build # the specs read stories from the built Storybookpnpm exec playwright install chromium webkitpnpm test:interaction --project chromium # everything, Chromiumpnpm test:interaction <spec file> # one spec, both engines@@ -42,6 +45,8 @@ committed: a pull request links or attaches it.| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || `scripts.ts` | Touch scripts as data: `tap`, `drag` (hold, rest), `flick` (exact final-move speed, velocity profiles), `holdThenDrag`, `pinch`, `rotate`, `path`, `merge` (a second finger mid-gesture), `releaseVelocity`, `toTouchEvents`. || `touch.ts` | `touchDriver(page).play(script)`: the script as real engine input. `installInputLog` / `readInputLog`: what the page received. |+| `frames.ts` | `startScreencast` (timestamped composited frames), `markElement` (a marker chip or outline), `trackMarker` (marker box per frame), `frameStats`. |+| `busy.ts` | `blockMainThread(page, ms)` and `busyWindowProof(tracked, window)`: frames presented, frames that moved, longest stall, a verdict; `presented` adds the compositor's own frames. || `curves.ts` | `holdAnimations`, `seekAnimations`, `sampleCurve` (presented transform / opacity at any t), `waitForAnimation`, `watchAnimations`, `waitForRest`, `releaseAnimations`. || `math.ts` | Oracles independent of the code under test: `cubicBezier`, `timingFunction`, `springStep` (SwiftUI response / dampingFraction, initial velocity), `projectDecay`, `rubberBand`, `parseTransform`, `compareCurve`, `timeline`. || `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event): PNG contact sheet + JSON trace. |@@ -97,6 +102,72 @@ set): a short entrance no longer races the first look on a cold page. When theaction may be the page's first contact with the harness, take`watchAnimations(locator)` before it and read `watch.started()` after.+**Busy main thread.** Mark the element, start the screencast, start the+animation, wait until it runs (`waitForAnimation`: an animation cannot start+while the thread is held), block the main thread for 1 s (`spec:AST-094`+FR5's verification row), stop. Always+start the gesture or animation first and block mid-flight, never before:+in headless Chromium a touch that starts while the main thread is busy does+not scroll or start a gesture even with only passive listeners (the input+waits for the main thread). Tracking a finger under a busy main thread is+therefore proven on the iOS simulator, not here. The+screencast is captured off the main thread, so a compositor animation shows a+new position every frame of the block, and a main-thread animation shows none+(nothing changed, so no frame is sent at all: `busyWindowProof` counts the+whole window as one stall). `settledWithin` (2 px) stops judging at the easing+tail: the screencast rounds to whole pixels, so a tail that still moves shows+plateaus within a pixel or two of rest. A judged part shorter than four of the+screencast's intervals fails: the animation under test was not moving during+the block. With+`during`, `blockMainThread` starts the loop from a task queued in the frame+that saw the animation, after that frame's rendering update: WebKit hands a+newly started animation to its compositor there, and a loop inside the rAF+callback would freeze even a compositor animation. Call it before the release+that starts a settle and await it after, so the page is already waiting when+the settle begins. An animation counts once it is no longer pending and has+run at least one frame: a `minProgressMs` under 16 is raised to 16, since one+created in the frame the page looks is not on the compositor yet. The `verdict` judges the+frames the screencast delivered, whatever its rate (a loaded machine's can+deliver one frame in five or six), and tolerates nothing a frozen animation+does: at least one frame in the judged part, no stall longer than three of the+screencast's intervals (two dropped frames), and at least 90% of the frames it+delivered in the judged part presenting a new position (one miss is always+allowed; a frame it did not deliver is a gap, for the stall check, unless+it delivered fewer than four frames there: then the judged part's span counts+every frame, so one late frame is not "1 of 1 moved"). Pass+`maxMedianFrameMs` (20 at 60 Hz) only in a proof that claims display rate.+`verdict.failures` names each miss with its numbers, and `summary` carries the+raw numbers for the report. Frames are judged in presentation order: a loaded+machine can deliver them out of order.++A just-started animation can reach the screen after the block began (50–100 ms+later in WebKit on a loaded machine). When the cast saw the element at rest+before the block (wait for `cast.frameCount()` to pass 0 before starting the+animation), judging starts at its first presented change: the judged part still+ends with the block and still needs four intervals, and `onScreenFromMs` and the+summary report when it came. An element already moving is judged from the+block's start. WebKit's casts are 2 px quantized: the frames come at the+viewport's size, but the page in them is captured at half resolution, so+positions move in 2 CSS px steps (Chromium's in 1). With under two steps of+motion per frame, a frame delivered a few ms early shows no step and reads as+not moved. The proof reports `stepPx` and `pxPerFrame` and notes motion under+two steps per frame, which never fails it. Move a proof's element at least two+steps a frame: the self-tests move 560 px in 1,600 ms, about 4.8 px a frame.++On a loaded machine Chromium's screencast can skip frames the compositor+still presents. Pass when the compositor presented each frame (epoch ms on the+page clock, from a trace of `Display::FrameDisplayed`) as `presented`. A gap between two moved+frames with nothing delivered inside it is then measured between the+compositor's own frames: the frames it presented there count as presented+(`recorderSkipped`, in the summary), and a gap it did not present through is+a stall as before. A gap that opens or closes the judged part, or holds a+delivered frame that did not move, is judged from the screencast, so a+frozen element fails whatever else the compositor draws (a rAF animation's+block presents no compositor frame at all). With the record, the 90% counts the+compositor's frames from the first to the last moved frame; the gaps before and+after are the stall check's. Without a record (WebKit has no such trace), the+screencast is judged alone.+**Storyboards.** Deterministic: frames come from seeking held animations orfrom stepping scripted input, never from wall-clock capture. The JSON tracecarries each frame's measurements and the spec asserts on it; the sheet is@@ -104,16 +175,18 @@ evidence. A component that finishes an animation from a timer (a transitionendbackstop, a cleanup render) would do so behind a held storyboard's back:`installTimerGate` before the animation starts and `freezeTimers` whilephotographing. The page can still cancel a held callback, a held frame callback-gets the time it actually runs, and the harness's own frame waits are not held.+gets the time it actually runs, and the harness's own waits (its frame waits,+`blockMainThread` with `during`) are not held.An animation that repeats forever does not set a storyboard's default length;with nothing else to film, pass `times`. A touch filmstrip keeps the scriptedevent `timeStamp`s (Chromium), but a handler that reads `performance.now()` or`Date.now()` sees one event per photograph, about 100 ms apart, so aclock-timed release flings slower when filmed than when played: a storyboard-cannot show such a gesture; record it in real time instead. Chromium can return-an empty copy of the page's surface right after it changes ("Unable to capture-screenshot"); a storyboard takes that photograph again two frames later, up to-three attempts in all, and lists each retake under `recaptures` in the trace.+cannot show such a gesture; record it in real time instead, with+`startScreencast`. Chromium can return an empty copy of+the page's surface right after it changes ("Unable to capture screenshot"); a+storyboard takes that photograph again two frames later, up to three attempts+in all, and lists each retake under `recaptures` in the trace.## Engine limits (measured with Playwright 1.61)@@ -126,11 +199,13 @@ three attempts in all, and lists each retake under `recaptures` in the trace.| Lifting one finger while others stay down | yes | pointer events right, but that `touchend` lists no remaining `touches`: skip with `PARTIAL_LIFT_SKIP` || Click synthesised from a touch | yes | only from `driver.tap` (WebKit's tap gesture), not from a raw touch sequence || Curve sampling, storyboards, input log | yes | yes |+| Screencast frame evidence (busy main thread) | yes | yes (WebKit's own screencast; frames keep coming while the main thread is blocked) |What WebKit cannot show here — native scroll hand-off, flings, a finger-lifting while another stays down, and anything iOS-only — needs Mobile Safari-itself, on a device or the iOS simulator. Skip such a proof on the driver's-flag, citing the reason: `test.skip(!driver.partialLifts, PARTIAL_LIFT_SKIP)`.+lifting while another stays down, finger tracking under a busy main thread, and+anything iOS-only — needs Mobile Safari itself, on a device or the iOS+simulator. Skip such a proof on the driver's flag, citing the reason:+`test.skip(!driver.partialLifts, PARTIAL_LIFT_SKIP)`.The WebKit input command is reached through Playwright's in-process serverobject, so it needs a locally launched browser (the default), not `connect()`.diff --git a/internal/interaction-harness/package.json b/internal/interaction-harness/package.jsonindex 40cccb26ca..942eccb9ed 100644--- a/internal/interaction-harness/package.json+++ b/internal/interaction-harness/package.json@@ -3,7 +3,7 @@"version": "0.0.0","private": true,"type": "module",- "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, curve sampling, storyboards). Never published.",+ "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, frame evidence, curve sampling, storyboards). Never published.","main": "./src/index.ts","exports": {".": "./src/index.ts",diff --git a/internal/interaction-harness/src/busy.test.ts b/internal/interaction-harness/src/busy.test.tsnew file mode 100644index 0000000000..372860125f--- /dev/null+++ b/internal/interaction-harness/src/busy.test.ts@@ -0,0 +1,503 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file busy.test.ts+ * @input Synthetic tracked frames (no browser)+ * @output The busy-main-thread verdict's limits: dropped frames a loaded+ * machine produces pass, a frozen animation fails every criterion, a window+ * with nothing moving to judge fails, and out-of-order frames are judged in+ * presentation order.+ * @position Unit test of ./busy and ./frames' ordering; the browser proof+ * with a real compositor animation and a rAF control is in+ * harness.interaction.browser.spec.ts.+ */++import {describe, expect, it} from 'vitest';+import {busyWindowProof} from './busy';+import {inPresentationOrder, type TrackedFrame} from './frames';++const FRAME = 1000 / 60;+const BLOCK = {from: 1000, to: 1700};++/** A marker moving 3 px per frame from t = 800 to 1900, minus `dropped` frame indices. */+function moving(dropped: readonly number[] = []): TrackedFrame[] {+ const frames: TrackedFrame[] = [];+ for (let i = 0; 800 + i * FRAME <= 1900; i += 1) {+ if (dropped.includes(i)) {+ continue;+ }+ frames.push({+ t: 800 + i * FRAME,+ box: {x: 0, y: 3 * i, width: 10, height: 10},+ } as TrackedFrame);+ }+ return frames;+}++/** Index of the first frame at or after `t`. */+const at = (t: number) => Math.ceil((t - 800) / FRAME);++/** `moving()` with each frame in `repeats` presenting the previous frame's position. */+function repeating(repeats: readonly number[]): TrackedFrame[] {+ return moving().map((f, i) =>+ repeats.includes(i)+ ? ({...f, box: {...f.box!, y: 3 * (i - 1)}} as TrackedFrame)+ : f,+ );+}++/** A marker moving 3 px per frame from t = 800 to 1900, frames `intervals` apart in turn. */+function cadenced(intervals: readonly number[]): TrackedFrame[] {+ const frames: TrackedFrame[] = [];+ for (let i = 0, t = 800; t <= 1900; i += 1) {+ frames.push({+ t,+ box: {x: 0, y: 3 * i, width: 10, height: 10},+ } as TrackedFrame);+ t += intervals[i % intervals.length];+ }+ return frames;+}++describe('busyWindowProof verdict', () => {+ it('passes a compositor animation with every frame', () => {+ const proof = busyWindowProof(moving(), BLOCK);+ expect(proof.verdict.failures).toEqual([]);+ expect(proof.longestStallFrames).toBe(1);+ });++ it('passes one and two consecutive dropped frames (3 intervals)', () => {+ const k = at(1300);+ expect(busyWindowProof(moving([k]), BLOCK).verdict.failures).toEqual([]);+ const two = busyWindowProof(moving([k, k + 1]), BLOCK);+ expect(two.longestStallFrames).toBe(3);+ expect(two.verdict.failures).toEqual([]);+ });++ it('fails three consecutive dropped frames', () => {+ const k = at(1300);+ const proof = busyWindowProof(moving([k, k + 1, k + 2]), BLOCK);+ expect(proof.longestStallFrames).toBe(4);+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ });++ it('fails when more than 10% of the delivered frames do not move', () => {+ const k = at(1050);+ // 39 frames delivered: up to 3 may show no new position (10% = 3.9).+ const scattered = [0, 12, 24].map(d => k + d);+ expect(+ busyWindowProof(repeating(scattered), BLOCK).verdict.failures,+ ).toEqual([]);+ const many = [0, 6, 12, 18, 24].map(d => k + d);+ expect(+ busyWindowProof(repeating(many), BLOCK).verdict.failures.join('\n'),+ ).toMatch(/expected frames moved/);+ });++ it('judges a frame the screencast did not deliver as a gap for the stall check, not a miss', () => {+ // The same five frames, not delivered: two-interval gaps, and every+ // delivered frame moved.+ const k = at(1050);+ const many = [0, 6, 12, 18, 24].map(d => k + d);+ const proof = busyWindowProof(moving(many), BLOCK);+ expect(proof.verdict.failures).toEqual([]);+ expect(proof.deliveredFrames).toBe(proof.moved);+ });++ it('judges an irregular cast on the frames it delivered', () => {+ // A cast cycling 11.7, 11.7 and 17 ms: its median interval would expect+ // 56 frames where it delivered 49, every one moved.+ const proof = busyWindowProof(cadenced([11.7, 11.7, 17]), BLOCK);+ expect(proof.verdict.failures).toEqual([]);+ expect(proof.deliveredFrames).toBe(49);+ expect(proof.moved).toBe(49);+ // Still a stall where it delivers nothing for four frames, and still a+ // miss where five delivered frames show no new position.+ const gap = cadenced([11.7, 11.7, 17]).filter(+ f => f.t < 1300 || f.t > 1360,+ );+ expect(busyWindowProof(gap, BLOCK).verdict.failures.join('\n')).toMatch(+ /longest stall/,+ );+ const unmoved = cadenced([11.7, 11.7, 17]).map((f, i, all) =>+ [20, 26, 32, 38, 44].includes(i)+ ? ({...f, box: {...all[i - 1].box!}} as TrackedFrame)+ : f,+ );+ expect(busyWindowProof(unmoved, BLOCK).verdict.failures.join('\n')).toMatch(+ /expected frames moved/,+ );+ });++ it('judges a window that delivered fewer than four frames against its span', () => {+ // A rAF animation frozen by the block, in WebKit: one frame from before+ // the block arrives 45 ms into it, then nothing until after it.+ const late = moving().filter(f => f.t <= 1000 || f.t >= 1750);+ late.push({+ t: 1045,+ box: {x: 0, y: 3 * at(1001), width: 10, height: 10},+ } as TrackedFrame);+ const proof = busyWindowProof(inPresentationOrder(late), BLOCK);+ expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/);+ });++ it('lets a short window miss a frame, not three', () => {+ // 150 ms judged: nine frames delivered, one miss allowed (10% would be 0.9).+ const short = {from: 1000, to: 1190};+ const k = at(1040);+ expect(busyWindowProof(repeating([k]), short).verdict.failures).toEqual([]);+ expect(+ busyWindowProof(+ repeating([k, k + 3, k + 6]),+ short,+ ).verdict.failures.join('\n'),+ ).toMatch(/expected frames moved/);+ });++ it('fails a frozen animation on every criterion', () => {+ // Frames before and after the block only: nothing changed during it.+ const frozen = moving().filter(f => f.t < BLOCK.from || f.t > BLOCK.to);+ const proof = busyWindowProof(frozen, BLOCK);+ expect(proof.changed).toBe(0);+ expect(proof.verdict.pass).toBe(false);+ expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/);+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/);+ });++ it('fails a window in which the animation under test was not moving', () => {+ // Settled before the block: the judgment ends where it starts.+ const settled = moving().map(f =>+ f.t < 950+ ? f+ : ({...f, box: {...f.box!, y: 3 * at(950)}} as TrackedFrame),+ );+ const before = busyWindowProof(settled, BLOCK, {settledWithin: 2});+ expect(before.verdict.pass).toBe(false);+ expect(before.verdict.failures.join('\n')).toMatch(+ /judged = -?\d+ frames < 4: the animation under test was not moving/,+ );+ // Settled 40 ms into the block: two frames, too few to judge.+ const early = moving().map(f =>+ f.t < 1040+ ? f+ : ({...f, box: {...f.box!, y: 3 * at(1040)}} as TrackedFrame),+ );+ expect(+ busyWindowProof(early, BLOCK, {settledWithin: 2}).verdict.failures.join(+ '\n',+ ),+ ).toMatch(/frames < 4: the animation under test was not moving/);+ });++ it('judges the frames a slower screencast delivers by default, and display rate only when asked', () => {+ // One frame in six, as a loaded machine's screencast delivers: every+ // delivered frame moved, so the default verdict passes.+ const sixth = moving().filter((_, i) => i % 6 === 0);+ expect(busyWindowProof(sixth, BLOCK).verdict.failures).toEqual([]);+ // Half rate fails only a proof that claims display rate.+ const odd = moving().filter((_, i) => i % 2 === 1);+ expect(busyWindowProof(odd, BLOCK).verdict.pass).toBe(true);+ expect(+ busyWindowProof(odd, BLOCK, {maxMedianFrameMs: 20}).verdict.failures.join(+ '\n',+ ),+ ).toMatch(/median frame interval/);+ });+});++describe('busyWindowProof settledWithin', () => {+ it('ends the judgment at a rounded easing tail, which would read as a stall', () => {+ // An ease-out to y = 24 as the screencast reports it: whole pixels, so the+ // tail repeats values while it still moves.+ const ys = [+ 120, 100, 82, 67, 55, 46, 39, 34, 30, 28, 27, 26, 26, 26, 25, 25, 24, 24,+ 24, 24,+ ];+ const frames = ys.map(+ (y, i) =>+ ({+ t: 1000 + i * FRAME,+ box: {x: 0, y, width: 10, height: 10},+ }) as TrackedFrame,+ );+ const window = {from: 1000, to: 1000 + ys.length * FRAME};+ const loose = busyWindowProof(frames, window, {+ settledWithin: 2,+ marginMs: 0,+ });+ expect(loose.verdict.failures).toEqual([]);+ const strict = busyWindowProof(frames, window, {+ settledWithin: 1,+ marginMs: 0,+ });+ expect(strict.longestStallFrames).toBeGreaterThanOrEqual(3);+ });++ it('takes the end value from settledAt over a stale last frame', () => {+ // The tail reaches y = 24, then the last frame shows a stale 100 (WebKit+ // presents the last committed position once an accelerated animation+ // ends while the main thread is still blocked).+ const ys = [+ 120, 100, 82, 67, 55, 46, 39, 34, 30, 28, 27, 26, 25, 24, 24, 24, 100,+ ];+ const frames = ys.map(+ (y, i) =>+ ({+ t: 1000 + i * FRAME,+ box: {x: 0, y, width: 10, height: 10},+ }) as TrackedFrame,+ );+ const window = {from: 1000, to: 1000 + ys.length * FRAME};+ const stale = busyWindowProof(frames, window, {+ settledWithin: 2,+ marginMs: 0,+ });+ expect(stale.verdict.pass).toBe(false);+ const fromPage = busyWindowProof(frames, window, {+ settledWithin: 2,+ settledAt: 24,+ marginMs: 0,+ });+ expect(fromPage.verdict.failures).toEqual([]);+ expect(fromPage.judged.to).toBe(frames[11].t);+ });+});++/** Seen at rest at t = 900, then on screen from `start`, moving 3 px a frame. */+function startingAt(start: number): TrackedFrame[] {+ const frames = [+ {t: 900, box: {x: 0, y: 0, width: 10, height: 10}} as TrackedFrame,+ ];+ for (let i = 1; start + (i - 1) * FRAME <= 1900; i += 1) {+ frames.push({+ t: start + (i - 1) * FRAME,+ box: {x: 0, y: 3 * i, width: 10, height: 10},+ } as TrackedFrame);+ }+ return frames;+}++describe('busyWindowProof on-screen start', () => {+ it('judges a just-started animation from its first presented change when it was seen at rest before the block, and reports when', () => {+ // On screen 87 ms into the block: four intervals after the judged part's start.+ const proof = busyWindowProof(startingAt(1087), BLOCK);+ expect(proof.verdict.failures).toEqual([]);+ expect(proof.onScreenFromMs).toBeCloseTo(87, 5);+ expect(proof.judged.from).toBeCloseTo(1087, 5);+ expect(proof.summary).toMatch(+ /on screen from \+87 ms \(nothing moved before the block\)/,+ );+ });++ it('still fails an animation that was already moving and stalls at the block start', () => {+ const k = at(1020);+ const proof = busyWindowProof(moving([k, k + 1, k + 2, k + 3]), BLOCK);+ expect(proof.onScreenFromMs).toBeNull();+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ });++ it('still fails a stall at the start when no frame before the block shows it at rest', () => {+ const unseen = startingAt(1087).filter(f => f.t >= BLOCK.from);+ const proof = busyWindowProof(unseen, BLOCK);+ expect(proof.onScreenFromMs).toBeNull();+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ });++ it('still fails a start the block itself delays until it ends', () => {+ const proof = busyWindowProof(startingAt(1720), BLOCK);+ expect(proof.onScreenFromMs).toBeNull();+ expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/);+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ });++ it('still fails a start that leaves fewer than minJudgedFrames inside the block', () => {+ const proof = busyWindowProof(startingAt(1640), BLOCK);+ expect(proof.onScreenFromMs).toBeCloseTo(640, 5);+ expect(proof.verdict.failures.join('\n')).toMatch(+ /frames < 4: the animation under test was not moving/,+ );+ });+});++describe('busyWindowProof position step', () => {+ it("reports the cast's position step and motion per frame, and notes motion under two steps without failing", () => {+ // 2.4 px a frame on a 2 px grid, as WebKit's casts report positions.+ const grid = moving().map(+ (f, i) =>+ ({+ ...f,+ box: {...f.box!, y: 2 * Math.round((2.4 * i) / 2)},+ }) as TrackedFrame,+ );+ const coarse = busyWindowProof(grid, BLOCK);+ expect(coarse.stepPx).toBe(2);+ expect(coarse.pxPerFrame).toBeCloseTo(2.4, 1);+ expect(coarse.notes.join('\n')).toMatch(+ /under two of the cast's 2 px position steps/,+ );+ expect(coarse.summary).toMatch(/2\.4 px\/frame in 2 px steps; note: /);+ expect(coarse.verdict.failures).toEqual([]);+ // 3 px a frame in whole pixels: resolvable, nothing to note.+ const fine = busyWindowProof(moving(), BLOCK);+ expect(fine.stepPx).toBe(1);+ expect(fine.pxPerFrame).toBeCloseTo(3, 1);+ expect(fine.notes).toEqual([]);+ });+});++/** The compositor presenting every frame from `from` to `to`, minus `gaps` ([from, to] spans). */+function drawn(+ gaps: ReadonlyArray<readonly [number, number]> = [],+ from = 700,+ to = 2000,+): number[] {+ const draws: number[] = [];+ for (let t = from; t <= to; t += FRAME) {+ if (!gaps.some(([a, b]) => t > a && t < b)) {+ draws.push(t);+ }+ }+ return draws;+}++describe('busyWindowProof with the compositor record', () => {+ it('with a compositor record, judges a sparse cast from its first to its last moved frame', () => {+ // One screencast frame in six, as on a loaded machine, with the compositor+ // presenting every frame: the frames before the first and after the last+ // moved one are the stall check's, not misses.+ const sparse = moving().filter((_, i) => i % 6 === 0);+ expect(+ busyWindowProof(sparse, BLOCK, {presented: drawn()}).verdict.failures,+ ).toEqual([]);+ });++ it('with a compositor record and an on-screen start, counts the frames it credits as skipped in what it judges', () => {+ // On screen 87 ms into the block, then four delivered frames that show+ // no new position: that fails on a full cast.+ const misses = [15, 20, 25, 30];+ const missing = startingAt(1087).map((f, i, all) =>+ misses.includes(i)+ ? ({...f, box: {...all[i - 1].box!}} as TrackedFrame)+ : f,+ );+ const full = busyWindowProof(missing, BLOCK, {presented: drawn()});+ expect(full.verdict.failures.join('\n')).toMatch(/expected frames moved/);+ // The screencast skipping the five frames after the on-screen frame+ // changes nothing: the compositor presented them, and they are judged.+ const skip = (frames: TrackedFrame[]) =>+ frames.filter((_, i) => i < 2 || i > 6);+ const skipped = busyWindowProof(skip(missing), BLOCK, {+ presented: drawn(),+ });+ expect(skipped.recorderSkipped).toBe(5);+ expect(skipped.verdict.failures.join('\n')).toMatch(+ /expected frames moved/,+ );+ // And a run with no misses never reads as more than every frame moved.+ const clean = busyWindowProof(skip(startingAt(1087)), BLOCK, {+ presented: drawn(),+ });+ expect(clean.verdict.failures).toEqual([]);+ expect(clean.movedFraction).toBeLessThanOrEqual(1);+ });++ it('counts a gap the compositor drew through as skipped, and one it did not draw through as a stall', () => {+ const k = at(1300);+ const gap: [number, number] = [+ 800 + (k - 1) * FRAME,+ 800 + (k + 4) * FRAME,+ ];+ const frames = moving([k, k + 1, k + 2, k + 3]);+ expect(busyWindowProof(frames, BLOCK).verdict.pass).toBe(false);+ const skipped = busyWindowProof(frames, BLOCK, {presented: drawn()});+ expect(skipped.verdict.failures).toEqual([]);+ expect(skipped.recorderSkipped).toBe(4);+ const stalled = busyWindowProof(frames, BLOCK, {presented: drawn([gap])});+ expect(stalled.recorderSkipped).toBe(0);+ expect(stalled.verdict.failures.join('\n')).toMatch(/longest stall/);+ });++ it('still fails an element that does not move while the compositor presents for something else', () => {+ const frozen = moving().map(f =>+ f.t >= BLOCK.from && f.t <= BLOCK.to+ ? ({...f, box: {...f.box!, y: 3 * at(BLOCK.from)}} as TrackedFrame)+ : f,+ );+ const proof = busyWindowProof(frozen, BLOCK, {presented: drawn()});+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/);+ });++ it('still fails a frozen animation, whether or not the compositor presents during the block', () => {+ const frozen = moving().filter(f => f.t < BLOCK.from || f.t > BLOCK.to);+ for (const presented of [drawn([[BLOCK.from, BLOCK.to]]), drawn()]) {+ const proof = busyWindowProof(frozen, BLOCK, {presented});+ expect(proof.verdict.failures.join('\n')).toMatch(/no frame presented/);+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ }+ });++ it('keeps a gap a stall when the screencast delivered a frame inside it that did not move', () => {+ const k = at(1300);+ const frames = moving([k, k + 1, k + 3]).map(f =>+ f.t > 800 + (k + 1.5) * FRAME && f.t < 800 + (k + 2.5) * FRAME+ ? ({...f, box: {...f.box!, y: 3 * (k - 1)}} as TrackedFrame)+ : f,+ );+ const proof = busyWindowProof(frames, BLOCK, {presented: drawn()});+ expect(proof.recorderSkipped).toBe(0);+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ });++ it('still counts single dropped frames the compositor did not present as missed', () => {+ const k = at(1050);+ const drops = [0, 6, 12, 18, 24, 30].map(d => k + d);+ const frames = moving(drops);+ const gaps = drops.map(+ i =>+ [800 + (i - 0.5) * FRAME, 800 + (i + 0.5) * FRAME] as [number, number],+ );+ expect(+ busyWindowProof(frames, BLOCK, {+ presented: drawn(gaps),+ }).verdict.failures.join('\n'),+ ).toMatch(/expected frames moved/);+ expect(+ busyWindowProof(frames, BLOCK, {presented: drawn()}).verdict.failures,+ ).toEqual([]);+ });++ it('judges a gap at the edge of the judged part from the screencast alone', () => {+ // Nothing delivered for the block's first 90 ms: a stall at its start,+ // whatever the compositor drew, since no moved frame opens the gap.+ const frames = moving().filter(+ f => f.t < BLOCK.from || f.t > BLOCK.from + 90,+ );+ const proof = busyWindowProof(frames, BLOCK, {presented: drawn()});+ expect(proof.recorderSkipped).toBe(0);+ expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/);+ });++ it('treats an empty record as no record, and says so', () => {+ const frames = moving([at(1300), at(1300) + 1, at(1300) + 2, at(1300) + 3]);+ const proof = busyWindowProof(frames, BLOCK, {presented: []});+ expect(proof.verdict.failures).toEqual(+ busyWindowProof(frames, BLOCK).verdict.failures,+ );+ expect(proof.summary).toMatch(/no compositor frames recorded/);+ });+});++describe('inPresentationOrder', () => {+ it('sorts frames delivered out of order, so motion stays monotonic', () => {+ const frames = moving();+ const k = at(1040);+ const shuffled = [...frames];+ [shuffled[k], shuffled[k + 1]] = [shuffled[k + 1], shuffled[k]];+ expect(busyWindowProof(shuffled, BLOCK).monotonic).toBe(false);+ const proof = busyWindowProof(inPresentationOrder(shuffled), BLOCK);+ expect(proof.monotonic).toBe(true);+ expect(proof.verdict.failures).toEqual([]);+ });+});diff --git a/internal/interaction-harness/src/busy.ts b/internal/interaction-harness/src/busy.tsnew file mode 100644index 0000000000..20c3d86a11--- /dev/null+++ b/internal/interaction-harness/src/busy.ts@@ -0,0 +1,577 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file busy.ts+ * @input Uses a Playwright `Page`+ * @output `blockMainThread` — a synchronous busy loop on the page's main thread+ * that reports exactly when it ran — and `busyWindowProof`, which reads frame+ * evidence for that window.+ * @position The busy-main-thread proof: an animation that runs on the+ * compositor keeps advancing at display rate while JavaScript holds the main+ * thread; one driven from the main thread (rAF, style writes, layout+ * properties) freezes for the whole block. Pair it with ./frames+ * (`startScreencast` + `trackMarker`), which observes off the main thread.+ *+ * SYNC: Keep the README's busy-main-thread section equal to this contract.+ */++import type {Locator, Page} from '@playwright/test';+import {frameStats, type FrameStats, type TrackedFrame} from './frames';++export interface BusyWindow {+ /** Epoch ms when the loop started and ended, on the page's own clock. */+ readonly from: number;+ readonly to: number;+}++/**+ * Holds the page's main thread for `ms` (a synchronous loop: no task, frame,+ * rAF, event or microtask runs meanwhile). Resolves with the window it held.+ *+ * Block MID-FLIGHT only: start the gesture or animation first. A touch that+ * starts while the main thread is busy does not scroll or begin a gesture in+ * headless Chromium, even with only passive listeners, and an animation that+ * has not started cannot start (see `waitForAnimation`). Finger tracking+ * under a busy main thread is a simulator proof, not a Playwright one.+ *+ * With `during`, the page itself waits until an animation on that element (or+ * inside it) has run `minProgressMs`, then blocks from a task it queues in+ * that frame. A round trip from the test would start the block later the+ * busier the machine is, leaving less of a short animation to judge. The task+ * runs after the frame's rendering update, which is where WebKit hands a newly+ * started animation to its compositor: a loop inside the rAF callback itself+ * holds that hand-off, and even a compositor animation freezes. To catch a+ * settle near its start, call it before the release that starts the settle and+ * await it after: the page is already waiting when the animation begins. Any+ * animation inside `during` counts, so pick an element that runs nothing+ * else meanwhile.+ *+ * An animation counts once it is no longer pending and has run at least one+ * frame (`minProgressMs` below 16 ms is raised to 16). One created in the+ * frame the page looks is still pending, not yet on the compositor+ * (Chromium), and in WebKit its start time resolves before its layer is+ * flushed: a block that started on it froze it.+ */+/** One display frame, ms: the least progress `blockMainThread` waits for. */+const ONE_FRAME_MS = 16;++export async function blockMainThread(+ page: Page,+ ms: number,+ {+ during,+ minProgressMs = 16,+ timeout = 5_000,+ }: {during?: Locator; minProgressMs?: number; timeout?: number} = {},+): Promise<BusyWindow> {+ const minProgress = Math.max(minProgressMs, ONE_FRAME_MS);+ if (during == null) {+ return page.evaluate(duration => {+ const from = performance.timeOrigin + performance.now();+ const end = performance.now() + duration;+ let spins = 0;+ while (performance.now() < end) {+ spins += 1;+ }+ return {from, to: performance.timeOrigin + performance.now(), spins};+ }, ms);+ }+ return during.evaluate(+ (element, {duration, minProgress, wait}) =>+ new Promise<BusyWindow>((resolve, reject) => {+ // The harness's own waits run on the real timers: a frozen timer gate+ // (story.ts) holds only the page's.+ const gate = (+ window as unknown as {+ __harnessTimerGate?: {+ frame?: typeof requestAnimationFrame;+ timeout?: typeof setTimeout;+ };+ }+ ).__harnessTimerGate;+ const frame = gate?.frame ?? window.requestAnimationFrame.bind(window);+ const later = gate?.timeout ?? window.setTimeout.bind(window);+ const deadline = performance.now() + wait;+ const tick = () => {+ const running = element+ .getAnimations({subtree: true})+ .some(+ a =>+ a.playState === 'running' &&+ !a.pending &&+ Number(a.currentTime ?? 0) >= minProgress,+ );+ if (running) {+ // The same loop as above, from a task queued in the frame that saw+ // the animation: it runs once that frame's rendering update has+ // handed the animation to the compositor.+ later(() => {+ const from = performance.timeOrigin + performance.now();+ const end = performance.now() + duration;+ while (performance.now() < end) {+ // Busy: nothing else runs.+ }+ resolve({from, to: performance.timeOrigin + performance.now()});+ }, 0);+ } else if (performance.now() > deadline) {+ reject(new Error('blockMainThread: no animation started'));+ } else {+ frame(tick);+ }+ };+ frame(tick);+ }),+ {duration: ms, minProgress, wait: timeout},+ );+}++export interface BusyVerdict {+ readonly pass: boolean;+ /** Each criterion the window missed, with its measured value and limit. */+ readonly failures: readonly string[];+}++export interface BusyProof extends FrameStats {+ readonly window: BusyWindow;+ /**+ * The part of the window judged: inside the margins, up to `settledWithin`,+ * and from `onScreenFromMs` when that is set.+ */+ readonly judged: {readonly from: number; readonly to: number};+ /**+ * When the tracked value was seen at rest before the judged part (a frame+ * there, no change in the 250 ms before it: a just-started animation not on+ * screen yet), ms after the window's start of its first presented change+ * inside the block, where judging starts; null otherwise.+ */+ readonly onScreenFromMs: number | null;+ /**+ * The cast's position step, CSS px: the largest of 0.5, 1 and 2 that every+ * tracked position is a multiple of (WebKit's casts are 2: see the README).+ */+ readonly stepPx: number;+ /** The tracked value's mean motion per screencast interval in the judged part, px. */+ readonly pxPerFrame: number;+ /** Facts worth reading beside the verdict, which never fail it. */+ readonly notes: readonly string[];+ /**+ * With a compositor record (`presented`): frames the screencast skipped+ * between two moved frames while the compositor presented them, which are+ * neither stalls nor misses; 0 without one.+ */+ readonly recorderSkipped: number;+ /**+ * The compositor's frame interval around the window, ms, from `presented`;+ * null without a usable record (the screencast alone is judged).+ */+ readonly compositorFrameMs: number | null;+ /**+ * The screencast's frame interval, from the frames around the window as well+ * as inside it (a short window has few intervals; a frozen one has none).+ */+ readonly cadenceMs: number;+ /**+ * Whole frame intervals in the judged part at the measured cadence (the+ * compositor's, when recorded): the `minJudgedFrames` floor reads it.+ */+ readonly expectedFrames: number;+ /** Frames the screencast delivered in the judged part, with the marker found. */+ readonly deliveredFrames: number;+ /**+ * Longest time inside the window with no presented change, the window's own+ * edges included. A frozen animation presents no new frame at all (nothing+ * on screen changed, so the compositor sends none): its stall is the whole+ * window.+ */+ readonly longestStallMs: number;+ /** Where the longest stall starts, ms after the window's start. */+ readonly longestStallAtMs: number;+ /**+ * `longestStallMs` in screencast intervals, rounded: 1 is no frame missed,+ * 2 is one dropped frame. Rounding absorbs timestamp jitter.+ */+ readonly longestStallFrames: number;+ /**+ * Frames in the window that presented a new position (the first compared+ * with the last frame before the window).+ */+ readonly moved: number;+ /**+ * `moved` (and `recorderSkipped`) over the frames judged: the frames the+ * screencast delivered in the judged part, or with a compositor record the+ * compositor's frames from the first to the last moved frame;+ * `expectedFrames` when nothing moved, or fewer than `minJudgedFrames`+ * frames were delivered.+ */+ readonly movedFraction: number;+ /** The judgment against the limits (see `busyWindowProof`). */+ readonly verdict: BusyVerdict;+ /** One line with the raw numbers, for an annotation or a failure message. */+ readonly summary: string;+}++/**+ * Frame evidence for a busy window: how many frames the screencast presented+ * inside it, how many of them moved the tracked axis, and the longest stall.+ * The window is trimmed by `marginMs` at both ends so the frame the block+ * started on and the one it ended on are not counted.+ *+ * The verdict judges the frames the screencast delivered, whatever its rate+ * (a loaded machine's screencast can deliver one frame in five or six), and+ * tolerates nothing a main-thread animation does (it presents no frame for+ * the whole block). It passes when:+ * - the judged part presented at least one frame, and, only when a proof+ * claims display rate and passes `maxMedianFrameMs`, their median interval+ * is at most that;+ * - no stall spans more than `maxStallFrames` screencast intervals;+ * - at least `minMovedFraction` of the frames judged moved the tracked axis+ * (a short window may still miss one): the frames the screencast delivered+ * in the judged part, so an irregular cast is judged on what it delivered+ * and a frame it did not deliver is a gap for the stall check, not a miss+ * (unless it delivered fewer than `minJudgedFrames` there: then every frame+ * of the judged part's span counts, so one late frame is not 1/1 moved);+ * with a compositor record, the compositor's frames from the first to the+ * last moved frame (the gaps before and after are the stall check's);+ * - the judged part spans at least `minJudgedFrames` frames. Fewer means the+ * animation under test was not moving during the block (it settled before+ * the block, or the block caught another animation), so there is nothing to+ * judge, and that fails rather than passes.+ *+ * A just-started animation can reach the screen after the block began (its+ * hand-off to the compositor lags under load: 50–100 ms in WebKit). When the+ * tracked value was seen at rest before the judged part (a frame there, and no+ * change in the 250 ms before it), judging starts at its first presented+ * change inside the block, which the+ * proof reports as `onScreenFromMs`. The judged part still ends with the+ * block and the `minJudgedFrames` floor applies to it, so a start the block+ * itself delays fails; an animation that was already moving is judged from+ * the block's start, and a stall there fails as before.+ *+ * `stepPx` and `pxPerFrame` say whether a frame's motion is resolvable at+ * all: under two position steps per frame, a frame delivered early can show+ * no step and read as not moved, which the proof notes (never a failure).+ *+ * A loaded machine's screencast can skip frames the compositor still presents+ * (Chromium). Given the compositor's record (`presented`: when it presented+ * each frame), a gap between two moved frames with no frame+ * delivered inside it is measured between the compositor's own frames: it is+ * a stall only where the compositor presented nothing, and the frames it did+ * present there are neither stalls nor misses (`recorderSkipped`). A gap+ * that opens or closes the judged part, or holds a delivered frame that did+ * not move, is judged from the screencast as before, so a frozen element+ * fails whatever else the compositor draws.+ */+export function busyWindowProof(+ tracked: readonly TrackedFrame[],+ window: BusyWindow,+ {+ axis = 'y',+ marginMs = 20,+ settledWithin,+ settledAt,+ maxMedianFrameMs = Infinity,+ maxStallFrames = 3,+ minMovedFraction = 0.9,+ minJudgedFrames = 4,+ presented,+ }: {+ axis?: 'x' | 'y' | 'width' | 'height';+ marginMs?: number;+ /**+ * Opt-in, for a proof that claims display rate: the most the median frame+ * interval may be, ms (20 at 60 Hz). Off by default, since it measures the+ * screencast's own rate as much as the animation.+ */+ maxMedianFrameMs?: number;+ /** The most screencast intervals one stall may span (3 = two dropped frames). */+ maxStallFrames?: number;+ /**+ * The least fraction of the expected frames that must move. One missed+ * frame is always allowed, so a window of under ten frames is not judged+ * stricter than a long one.+ */+ minMovedFraction?: number;+ /** The fewest whole frame intervals the judged part of the window may span. */+ minJudgedFrames?: number;+ /**+ * Stop judging once the value is within this many px of where it ends.+ * The screencast reports whole CSS pixels, so an easing tail that still+ * moves shows plateaus (26, 26, 26 against a final 24) that would read+ * as a stall: 2 is the usual value.+ */+ settledWithin?: number;+ /**+ * Where the value ends, for `settledWithin`: read it from the page once the+ * animation has ended (`waitForRest`). Defaults to the last frame's value,+ * which can be stale: once an accelerated animation ends while the main+ * thread is still blocked, WebKit presents the layer's last committed+ * position until the main thread commits the end state.+ */+ settledAt?: number;+ /**+ * When the compositor presented each frame, epoch ms on the page clock+ * (Chromium's `Display::FrameDisplayed`). Empty or absent, the screencast+ * is judged alone.+ */+ presented?: readonly number[];+ } = {},+): BusyProof {+ let end = window.to - marginMs;+ if (settledWithin != null) {+ const valued = tracked.flatMap(f =>+ f.box == null ? [] : [{t: f.t, value: f.box[axis]}],+ );+ const final =+ settledAt ?? (valued.length > 0 ? valued[valued.length - 1].value : null);+ const arrived = valued.find(+ f =>+ f.t >= window.from &&+ final != null &&+ Math.abs(f.value - final) <= settledWithin,+ );+ if (arrived != null) {+ end = Math.min(end, arrived.t);+ }+ }+ const inner = {from: window.from + marginMs, to: end};+ const values = tracked+ .filter(f => f.t >= inner.from && f.t <= inner.to)+ .map(f => ({t: f.t, value: f.box == null ? null : f.box[axis]}));+ const before = [...tracked].reverse().find(f => f.t < inner.from);+ let previous: number | null = before?.box == null ? null : before.box[axis];+ const inWindow: number[] = [];+ for (const v of values) {+ if (+ v.value != null &&+ previous != null &&+ Math.abs(v.value - previous) > 0.25+ ) {+ inWindow.push(v.t);+ }+ if (v.value != null) {+ previous = v.value;+ }+ }+ // Seen at rest before the judged part (a frame there, and no change in the+ // 250 ms before it): the first presented change is the animation's on-screen+ // start, and judging starts there. With no frame before, nothing is known,+ // and the window is judged from its start.+ const prior = tracked.filter(f => f.box != null && f.t < inner.from);+ let leadFrom = 0;+ prior.forEach((f, i) => {+ if (f.t < inner.from - 250) {+ leadFrom = i;+ }+ });+ const lead = prior.flatMap(f => (f.box == null ? [] : [f.box[axis]]));+ const movedBefore = lead.some(+ (value, i) => i > leadFrom && Math.abs(value - lead[i - 1]) > 0.25,+ );+ const onScreenFrom =+ prior.length > 0 && !movedBefore && inWindow.length > 0+ ? inWindow[0]+ : null;+ const judged = {from: onScreenFrom ?? inner.from, to: inner.to};+ const changes = inWindow.filter(t => t > judged.from || onScreenFrom == null);+ const edges = [judged.from, ...changes, judged.to];+ const around = tracked+ .filter(f => f.t >= inner.from - 250 && f.t <= inner.to + 250)+ .map(f => f.t);+ const intervals = around+ .slice(1)+ .map((t, i) => t - around[i])+ .sort((a, b) => a - b);+ const stats = frameStats(+ tracked.map(f => ({t: f.t, value: f.box == null ? null : f.box[axis]})),+ judged,+ );+ const cadence =+ intervals.length >= 4+ ? intervals[Math.floor(intervals.length / 2)]+ : stats.medianFrameMs || 1000 / 60;+ // The compositor's own frames, when recorded: its interval, from the frames+ // around the judged part.+ const draws = [...(presented ?? [])]+ .filter(t => t >= judged.from - 250 && t <= judged.to + 250)+ .sort((a, b) => a - b);+ const drawIntervals = draws+ .slice(1)+ .map((t, i) => t - draws[i])+ .sort((a, b) => a - b);+ const compositorFrameMs =+ drawIntervals.length >= 4+ ? drawIntervals[Math.floor(drawIntervals.length / 2)]+ : null;+ const changed = new Set(inWindow);+ const delivered = tracked+ .filter(f => f.box != null && f.t > judged.from && f.t < judged.to)+ .map(f => f.t);+ // Each gap between presented changes, in intervals of the channel that saw+ // it: the screencast's, or, for a gap between two moved frames with no+ // frame delivered inside it, the compositor's own longest interval there.+ const gaps = edges.slice(1).map((to, i) => {+ const from = edges[i];+ const ms = to - from;+ if (+ compositorFrameMs == null ||+ !changed.has(from) ||+ !changed.has(to) ||+ delivered.some(t => t > from && t < to)+ ) {+ return {from, ms, frames: Math.round(ms / cadence), skipped: 0};+ }+ const span = draws.filter(+ t => t > from - compositorFrameMs / 2 && t < to + compositorFrameMs / 2,+ );+ const longest =+ span.length >= 2+ ? Math.min(ms, Math.max(...span.slice(1).map((t, j) => t - span[j])))+ : ms;+ const frames = Math.round(longest / compositorFrameMs);+ return {+ from,+ ms: longest,+ frames,+ skipped: Math.max(0, Math.round(ms / compositorFrameMs) - frames),+ };+ });+ const longestGap = gaps.reduce((a, b) =>+ b.frames > a.frames || (b.frames === a.frames && b.ms > a.ms) ? b : a,+ );+ const longestStallMs = longestGap.ms;+ // Where the longest stall starts, from the start of the window.+ const longestStallAtMs = longestGap.from - window.from;+ const longestStallFrames = longestGap.frames;+ const skips = gaps.filter(g => g.skipped > 0);+ const recorderSkipped = skips.reduce((sum, g) => sum + g.skipped, 0);+ // With a compositor record, frames count in its intervals, and the frames+ // it presented where the screencast skipped are presented frames.+ const expectedFrames = Math.floor(+ (judged.to - judged.from) / (compositorFrameMs ?? cadence),+ );+ const moved = changes.length;+ // The frames judged for motion: the screencast's own frames in the judged+ // part (each `changes` entry is one of them), or, with a compositor record,+ // the compositor's frames from the first to the last moved frame (the gaps+ // before and after them are the stall check's). After an on-screen start+ // they count from the on-screen frame, which `changes` leaves out, so the+ // frames credited as skipped after it are judged too. A part that moved+ // nothing, or delivered fewer than `minJudgedFrames` frames, is judged+ // against its span, so a frozen animation misses here too (one late frame+ // from before the block is not 1/1 moved).+ const deliveredFrames = tracked.filter(+ f =>+ f.box != null &&+ (onScreenFrom == null ? f.t >= judged.from : f.t > judged.from) &&+ f.t <= judged.to,+ ).length;+ const judgedFrames =+ compositorFrameMs != null+ ? changes.length > 0+ ? Math.round(+ (changes[changes.length - 1] - (onScreenFrom ?? changes[0])) /+ compositorFrameMs,+ ) + (onScreenFrom == null ? 1 : 0)+ : expectedFrames+ : deliveredFrames < minJudgedFrames+ ? expectedFrames+ : deliveredFrames;+ const movedFraction = (moved + recorderSkipped) / Math.max(1, judgedFrames);+ // The cast's position step, and the motion per interval it has to resolve.+ const positions = tracked.flatMap(f => (f.box == null ? [] : [f.box[axis]]));+ const stepPx =+ [2, 1, 0.5].find(step =>+ positions.every(p => Math.abs(p / step - Math.round(p / step)) < 0.01),+ ) ?? 0.5;+ const judgedValues = tracked.flatMap(f =>+ f.box != null && f.t >= judged.from && f.t <= judged.to+ ? [{t: f.t, value: f.box[axis]}]+ : [],+ );+ const first = judgedValues[0];+ const last = judgedValues[judgedValues.length - 1];+ const pxPerFrame =+ first == null || last == null || last.t <= first.t+ ? 0+ : Math.abs(last.value - first.value) / ((last.t - first.t) / cadence);+ const notes: string[] = [];+ if (presented != null && compositorFrameMs == null) {+ notes.push(+ 'no compositor frames recorded around the window: judged from the screencast alone',+ );+ }+ if (recorderSkipped > 0) {+ notes.push(+ `the screencast skipped ${recorderSkipped} frames the compositor presented (at ${skips.map(g => `+${(g.from - window.from).toFixed(0)} ms`).join(', ')})`,+ );+ }+ if (pxPerFrame > 0 && pxPerFrame < 2 * stepPx) {+ notes.push(+ `${pxPerFrame.toFixed(1)} px/frame is under two of the cast's ${stepPx} px position steps: a frame delivered early can show no step and read as not moved`,+ );+ }+ const failures: string[] = [];+ if (stats.frames === 0) {+ failures.push('no frame presented in the judged part');+ } else if (stats.medianFrameMs > maxMedianFrameMs) {+ failures.push(+ `median frame interval ${stats.medianFrameMs.toFixed(1)} ms > ${maxMedianFrameMs} ms (${stats.frames} frames presented)`,+ );+ }+ if (longestStallFrames > maxStallFrames) {+ failures.push(+ `longest stall ${longestStallMs.toFixed(1)} ms = ${longestStallFrames} intervals > ${maxStallFrames}`,+ );+ }+ if (expectedFrames < minJudgedFrames) {+ failures.push(+ `${(judged.to - judged.from).toFixed(0)} ms judged = ${expectedFrames} frames < ${minJudgedFrames}: the animation under test was not moving during the block`,+ );+ }+ const allowedMisses = Math.max(+ 1,+ Math.floor((1 - minMovedFraction) * judgedFrames + 1e-9),+ );+ if (judgedFrames - moved - recorderSkipped > allowedMisses) {+ failures.push(+ `${moved} of ${judgedFrames} expected frames moved${recorderSkipped > 0 ? ` (and ${recorderSkipped} the screencast skipped)` : ''} (${(movedFraction * 100).toFixed(0)}% < ${(minMovedFraction * 100).toFixed(0)}%)`,+ );+ }+ const onScreenFromMs =+ onScreenFrom == null ? null : onScreenFrom - window.from;+ const summary =+ `${(judged.to - judged.from).toFixed(0)} ms judged at ${cadence.toFixed(1)} ms/frame: ${stats.frames} frames presented, ` ++ `${moved}/${judgedFrames} moved${recorderSkipped > 0 ? ` and ${recorderSkipped} skipped by the screencast` : ''} (${(movedFraction * 100).toFixed(0)}%), ` ++ `median ${stats.medianFrameMs.toFixed(1)} ms, ` ++ `longest stall ${longestStallMs.toFixed(1)} ms (${longestStallFrames} intervals) at +${longestStallAtMs.toFixed(0)} ms, ` ++ `${stats.monotonic ? 'monotonic' : 'not monotonic'}, ` ++ `${pxPerFrame.toFixed(1)} px/frame in ${stepPx} px steps` ++ (compositorFrameMs == null+ ? ''+ : `, compositor presenting every ${compositorFrameMs.toFixed(1)} ms`) ++ (onScreenFromMs == null+ ? ''+ : `, on screen from +${onScreenFromMs.toFixed(0)} ms (nothing moved before the block)`) ++ notes.map(note => `; note: ${note}`).join('');+ return {+ ...stats,+ window,+ judged,+ onScreenFromMs,+ stepPx,+ pxPerFrame,+ notes,+ recorderSkipped,+ compositorFrameMs,+ cadenceMs: cadence,+ longestStallMs,+ longestStallAtMs,+ expectedFrames,+ deliveredFrames,+ longestStallFrames,+ moved,+ movedFraction,+ verdict: {pass: failures.length === 0, failures},+ summary,+ };+}diff --git a/internal/interaction-harness/src/frames.ts b/internal/interaction-harness/src/frames.tsnew file mode 100644index 0000000000..c61ea7e171--- /dev/null+++ b/internal/interaction-harness/src/frames.ts@@ -0,0 +1,465 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file frames.ts+ * @input Uses a Playwright `Page`, a `Locator` for the element under test, and+ * the engine's screencast (Chromium CDP `Page.startScreencast`; WebKit's+ * `Screencast` domain)+ * @output `startScreencast` (timestamped composited frames), `markElement`+ * (a marker outline the frames can find), `trackMarker` (per-frame marker+ * box), `frameStats`.+ * @position Frame evidence for motion proofs. The screencast is captured off+ * the page's main thread, from what the compositor presents, so it keeps+ * delivering frames while a script blocks the main thread: that is what makes+ * the busy-main-thread proof possible.+ *+ * Marker tracking needs no image library: frames are decoded and scanned in a+ * scratch page of the same browser (createImageBitmap + canvas), so the+ * harness adds no dependency.+ *+ * SYNC: Keep the README's frame-evidence section equal to what this measures.+ */++import type {CDPSession, Locator, Page} from '@playwright/test';+import {webkitPageSession} from './touch';++export interface CastFrame {+ /** Presentation time, epoch ms (the compositor's frame timestamp). */+ readonly t: number;+ /** Base64 PNG/JPEG. */+ readonly data: string;+ readonly format: 'png' | 'jpeg';+ /** CSS px of the viewport the frame shows. */+ readonly viewportWidth: number;+ readonly viewportHeight: number;+}++export interface Screencast {+ /**+ * Frames captured so far: wait for the first one to know the cast saw the+ * page before an animation starts (`busyWindowProof` then knows its at-rest+ * position).+ */+ frameCount(): number;+ /** Stops and returns every frame captured, in order. */+ stop(): Promise<CastFrame[]>;+}++export interface ScreencastOptions {+ /** Lossless frames by default: marker colours survive PNG, not JPEG. */+ readonly format?: 'png' | 'jpeg';+ readonly quality?: number;+ /**+ * Keep one presented frame in this many (default 1, every frame): a slower+ * evidence channel, like a loaded machine's screencast, for proving that a+ * verdict does not depend on the screencast's rate.+ */+ readonly everyNthFrame?: number;+}++/**+ * Starts capturing composited frames. Chromium delivers one per presented+ * frame (16.7 ms at 60 Hz), timestamped by the compositor; WebKit's own+ * screencast delivers one per composited update (about every 14 ms in the+ * Linux build), stamped on CLOCK_MONOTONIC and rebased here onto the wall+ * clock by that clock's measured offset, so both engines share the page's+ * timeline. Both keep delivering while the page's main thread is blocked, and+ * neither sends a frame when nothing on screen changed.+ */+export async function startScreencast(+ page: Page,+ options: ScreencastOptions = {},+): Promise<Screencast> {+ const name = page.context().browser()?.browserType().name();+ const viewport = page.viewportSize() ?? {width: 0, height: 0};+ if (name === 'chromium') {+ return chromiumScreencast(page, viewport, options);+ }+ if (name === 'webkit') {+ return webkitScreencast(page, viewport, options);+ }+ throw new Error(+ `startScreencast: unsupported browser ${name ?? '(unknown)'}`,+ );+}++async function chromiumScreencast(+ page: Page,+ viewport: {width: number; height: number},+ {format = 'png', quality = 90, everyNthFrame = 1}: ScreencastOptions,+): Promise<Screencast> {+ const cdp: CDPSession = await page.context().newCDPSession(page);+ const frames: CastFrame[] = [];+ const onFrame = (frame: {+ data: string;+ sessionId: number;+ metadata: {timestamp?: number; deviceWidth: number; deviceHeight: number};+ }) => {+ frames.push({+ t: (frame.metadata.timestamp ?? Date.now() / 1000) * 1000,+ data: frame.data,+ format,+ viewportWidth: frame.metadata.deviceWidth || viewport.width,+ viewportHeight: frame.metadata.deviceHeight || viewport.height,+ });+ cdp+ .send('Page.screencastFrameAck', {sessionId: frame.sessionId})+ .catch(() => {});+ };+ cdp.on('Page.screencastFrame', onFrame);+ // One CSS px per frame px keeps decoding cheap; everyNthFrame 1 = every presented frame.+ await cdp.send('Page.startScreencast', {+ format,+ quality,+ maxWidth: viewport.width,+ maxHeight: viewport.height,+ everyNthFrame,+ });+ return {+ frameCount: () => frames.length,+ async stop() {+ await cdp.send('Page.stopScreencast');+ cdp.off('Page.screencastFrame', onFrame);+ await cdp.detach().catch(() => {});+ return inPresentationOrder(frames);+ },+ };+}++async function webkitScreencast(+ page: Page,+ viewport: {width: number; height: number},+ {quality = 90, everyNthFrame = 1}: ScreencastOptions,+): Promise<Screencast> {+ const session = webkitPageSession(page);+ const frames: CastFrame[] = [];+ let generation = 0;+ let delivered = 0;+ const onFrame = (payload: unknown) => {+ const frame = payload as {data: string; timestamp: number};+ // WebKit's screencast has no everyNthFrame: keep one frame in N here.+ delivered += 1;+ if ((delivered - 1) % everyNthFrame !== 0) {+ session+ .send('Screencast.screencastFrameAck', {generation})+ .catch(() => {});+ return;+ }+ frames.push({+ // WebKit stamps frames in seconds on the monotonic clock; rebased below.+ t: frame.timestamp * 1000,+ data: frame.data,+ format: 'jpeg',+ viewportWidth: viewport.width,+ viewportHeight: viewport.height,+ });+ session.send('Screencast.screencastFrameAck', {generation}).catch(() => {});+ };+ session.on('Screencast.screencastFrame', onFrame);+ const started = (await session.send('Screencast.startScreencast', {+ width: viewport.width,+ height: viewport.height,+ toolbarHeight: 0,+ quality,+ })) as {generation: number};+ generation = started.generation;+ // WebKit stamps frames in seconds on CLOCK_MONOTONIC, which this process+ // shares: rebase them by the wall clock's offset from that clock, sampled+ // together. (Taking the first frame as the start shifted every frame earlier+ // by however long that frame took to come, 60–100 ms on a busy page.)+ const wallMinusMonotonic = Date.now() - Number(process.hrtime.bigint()) / 1e6;+ return {+ frameCount: () => frames.length,+ async stop() {+ await session.send('Screencast.stopScreencast');+ session.off?.('Screencast.screencastFrame', onFrame);+ if (frames.length === 0) {+ return frames;+ }+ return inPresentationOrder(frames).map(f => ({+ ...f,+ t: f.t + wallMinusMonotonic,+ }));+ },+ };+}++/**+ * Frames sorted by their presentation timestamp. A loaded machine can deliver+ * screencast frames out of order (a frame stamped 35 ms arriving after one+ * stamped 46 ms); judged in arrival order, a steadily moving element would+ * appear to step backwards.+ */+export function inPresentationOrder<T extends {readonly t: number}>(+ frames: readonly T[],+): T[] {+ return [...frames].sort((a, b) => a.t - b.t);+}++/** The marker colour: magenta never occurs in Astryx themes. */+export const MARKER_RGB: readonly [number, number, number] = [255, 0, 255];++export interface MarkerOptions {+ readonly rgb?: readonly [number, number, number];+ /**+ * `chip` (default): a small square child pinned inside the element's+ * top-left corner, above every sibling (z-index max), so nothing the+ * component paints can cover it. The element must be its containing block:+ * a static element is made `position: relative` while marked.+ * `outline`: an inset outline on the element itself (no DOM insertion), for+ * elements that cannot take a child; children with their own layers can+ * cover parts of it.+ */+ readonly mode?: 'chip' | 'outline';+ /** Chip size or outline width, CSS px. */+ readonly size?: number;+ /** Chip inset from the element's top-left corner, CSS px. */+ readonly inset?: number;+}++/**+ * Marks the element so screencast frames can find it: the mark moves, scales+ * and fades with the element's own layer and never changes its layout.+ * Returns an undo.+ */+export async function markElement(+ locator: Locator,+ options: MarkerOptions = {},+): Promise<() => Promise<void>> {+ const {+ rgb = MARKER_RGB,+ mode = 'chip',+ size = mode === 'chip' ? 16 : 6,+ inset = 8,+ } = options;+ const id = `m${Math.random().toString(36).slice(2, 8)}`;+ await locator.evaluate(+ (element, {id: markerId, color, size: px, inset: offset, mode: kind}) => {+ const el = element as HTMLElement;+ el.setAttribute('data-harness-marker', markerId);+ if (kind === 'outline') {+ const style = document.createElement('style');+ style.setAttribute('data-harness-marker-style', markerId);+ style.textContent = `[data-harness-marker="${markerId}"]{outline:${px}px solid rgb(${color.join(',')}) !important;outline-offset:-${px}px !important;}`;+ document.head.append(style);+ return;+ }+ if (getComputedStyle(el).position === 'static') {+ el.setAttribute('data-harness-marker-position', el.style.position);+ el.style.position = 'relative';+ }+ const chip = document.createElement('div');+ chip.setAttribute('data-harness-marker-chip', markerId);+ chip.setAttribute('aria-hidden', 'true');+ chip.style.cssText = `position:absolute;top:${offset}px;left:${offset}px;width:${px}px;height:${px}px;background:rgb(${color.join(',')});z-index:2147483647;pointer-events:none;`;+ el.append(chip);+ },+ {id, color: [...rgb], size, inset, mode},+ );+ return async () => {+ await locator.page().evaluate(markerId => {+ const el = document.querySelector<HTMLElement>(+ `[data-harness-marker="${markerId}"]`,+ );+ if (el?.hasAttribute('data-harness-marker-position')) {+ el.style.position =+ el.getAttribute('data-harness-marker-position') ?? '';+ el.removeAttribute('data-harness-marker-position');+ }+ el?.removeAttribute('data-harness-marker');+ document+ .querySelector(`[data-harness-marker-chip="${markerId}"]`)+ ?.remove();+ document+ .querySelector(`[data-harness-marker-style="${markerId}"]`)+ ?.remove();+ }, id);+ };+}++export interface Box {+ readonly x: number;+ readonly y: number;+ readonly width: number;+ readonly height: number;+}++export interface TrackedFrame {+ /** Epoch ms. */+ readonly t: number;+ /** The marker's bounding box in CSS px, or null when it is not on screen. */+ readonly box: Box | null;+ /** Marker pixels found (a fading element finds fewer at the same tolerance). */+ readonly pixels: number;+}++/**+ * Finds the marker in every frame. Decoding runs in a scratch page of the same+ * browser, so neither the page under test nor Node needs an image decoder.+ */+export async function trackMarker(+ page: Page,+ frames: readonly CastFrame[],+ {+ rgb = MARKER_RGB,+ tolerance = 40,+ }: {rgb?: readonly [number, number, number]; tolerance?: number} = {},+): Promise<TrackedFrame[]> {+ const scratch = await page.context().newPage();+ try {+ const out: TrackedFrame[] = [];+ // Batches keep each evaluate payload moderate.+ for (let i = 0; i < frames.length; i += 12) {+ const batch = frames.slice(i, i + 12);+ const boxes = await scratch.evaluate(+ async ({batch: items, color, tol}) => {+ const results: {+ box: {x: number; y: number; width: number; height: number} | null;+ pixels: number;+ }[] = [];+ for (const item of items) {+ const blob = await (+ await fetch(`data:image/${item.format};base64,${item.data}`)+ ).blob();+ const bitmap = await createImageBitmap(blob);+ const canvas = new OffscreenCanvas(bitmap.width, bitmap.height);+ const context = canvas.getContext('2d');+ if (context == null) {+ throw new Error('no 2d context');+ }+ context.drawImage(bitmap, 0, 0);+ const {data, width, height} = context.getImageData(+ 0,+ 0,+ bitmap.width,+ bitmap.height,+ );+ let minX = Infinity;+ let minY = Infinity;+ let maxX = -Infinity;+ let maxY = -Infinity;+ let pixels = 0;+ for (let y = 0; y < height; y += 1) {+ for (let x = 0; x < width; x += 1) {+ const o = (y * width + x) * 4;+ if (+ Math.abs(data[o] - color[0]) <= tol &&+ Math.abs(data[o + 1] - color[1]) <= tol &&+ Math.abs(data[o + 2] - color[2]) <= tol+ ) {+ pixels += 1;+ if (x < minX) {+ minX = x;+ }+ if (x > maxX) {+ maxX = x;+ }+ if (y < minY) {+ minY = y;+ }+ if (y > maxY) {+ maxY = y;+ }+ }+ }+ }+ const scale = width / item.viewportWidth;+ results.push({+ box:+ pixels === 0+ ? null+ : {+ x: minX / scale,+ y: minY / scale,+ width: (maxX - minX + 1) / scale,+ height: (maxY - minY + 1) / scale,+ },+ pixels,+ });+ }+ return results;+ },+ {+ batch: batch.map(f => ({+ data: f.data,+ format: f.format,+ viewportWidth: f.viewportWidth,+ })),+ color: [...rgb],+ tol: tolerance,+ },+ );+ boxes.forEach((result, k) =>+ out.push({t: batch[k].t, box: result.box, pixels: result.pixels}),+ );+ }+ return out;+ } finally {+ await scratch.close();+ }+}++export interface FrameStats {+ readonly frames: number;+ /** Frames whose tracked value differs from the frame before. */+ readonly changed: number;+ /** Longest stretch without a change inside the window (an open stall at its end included), ms. */+ readonly longestStallMs: number;+ /** Median spacing between frames, ms. */+ readonly medianFrameMs: number;+ readonly monotonic: boolean;+ readonly first: number | null;+ readonly last: number | null;+}++/** Summarises per-frame values (e.g. the marker's y) inside a time window. */+export function frameStats(+ samples: readonly {readonly t: number; readonly value: number | null}[],+ window: {readonly from: number; readonly to: number} = {+ from: -Infinity,+ to: Infinity,+ },+ epsilon = 0.25,+): FrameStats {+ const inside = samples.filter(+ s => s.t >= window.from && s.t <= window.to && s.value != null,+ ) as {t: number; value: number}[];+ let changed = 0;+ let increasing = 0;+ let decreasing = 0;+ let stallStart = inside[0]?.t ?? 0;+ let longestStallMs = 0;+ for (let i = 1; i < inside.length; i += 1) {+ const delta = inside[i].value - inside[i - 1].value;+ if (Math.abs(delta) > epsilon) {+ changed += 1;+ if (delta > 0) {+ increasing += 1;+ } else {+ decreasing += 1;+ }+ longestStallMs = Math.max(longestStallMs, inside[i].t - stallStart);+ stallStart = inside[i].t;+ }+ }+ // A stall still open when the window closes counts too: a frozen animation never changes at all.+ if (inside.length > 0) {+ longestStallMs = Math.max(+ longestStallMs,+ inside[inside.length - 1].t - stallStart,+ );+ }+ const gaps = inside+ .slice(1)+ .map((s, i) => s.t - inside[i].t)+ .sort((a, b) => a - b);+ return {+ frames: inside.length,+ changed,+ longestStallMs,+ medianFrameMs: gaps.length ? gaps[Math.floor(gaps.length / 2)] : 0,+ monotonic: increasing === 0 || decreasing === 0,+ first: inside[0]?.value ?? null,+ last: inside[inside.length - 1]?.value ?? null,+ };+}diff --git a/internal/interaction-harness/src/harness.interaction.browser.spec.ts b/internal/interaction-harness/src/harness.interaction.browser.spec.tsindex 7e47af95b2..a6c73fdf88 100644--- a/internal/interaction-harness/src/harness.interaction.browser.spec.ts+++ b/internal/interaction-harness/src/harness.interaction.browser.spec.ts@@ -3,24 +3,28 @@/*** @file harness.interaction.browser.spec.ts* @input Fixture pages written inline (no Storybook), real Chromium/WebKit- * @output Proof that each harness instrument measures what it claims:- * scripted timestamps reach the page; a flick flings natively (Chromium);- * a second finger and a pinch arrive as distinct pointers; sampled curves- * match the oracles; the animation-state waits report a running transition- * and its end, and judge rest by presented values even while an animation- * holds its end, and a watch taken on first contact ignores an animation- * already running and keeps the start of one the page cancels when it ends; individual transform properties are sampled; a sample at+ * @output Proof that each harness instrument measures what it claims (the+ * busy-main-thread proof with a negative control beside its positive+ * check): scripted timestamps reach the+ * page; a flick flings natively (Chromium); a second finger and a pinch+ * arrive as distinct pointers; the screencast shows a compositor animation+ * advancing through a blocked main thread while a rAF animation freezes;+ * sampled curves+ * match the oracles, individual transform properties included; a sample at* an end reports the time it was read at; holding takes only running* document-timeline animations, each at its own offset, holds each where it* was read, adds a second hold to the held set without moving the first- * hold's times, drops one the page cancels, and releases only them;- * storyboards write a sheet and a trace that follows the curve, an endless- * animation does not set their length, and a photograph Chromium returns- * empty is taken again two frames later; a touch filmstrip keeps scripted- * event times while a handler clock sees its pacing; the timer gate holds a- * component timer until the filmstrip is done, drops callbacks the page- * cancels, stamps held frames with the time they run, and leaves the- * harness its own frames.+ * hold's times, drops one the page cancels, and releases only them; the+ * animation-state waits report a running transition and its end, and judge+ * rest by presented values even while an animation holds its end, and a+ * watch taken on first contact ignores an animation already running and+ * keeps the start of one the page cancels when it ends; storyboards write a+ * sheet and a trace that follows the curve, an endless animation does not+ * set their length, and a photograph Chromium returns empty is taken again+ * two frames later; a touch filmstrip keeps scripted event times while a+ * handler clock sees its pacing; the timer gate holds a component timer+ * until the filmstrip is done, drops callbacks the page cancels, stamps held+ * frames with the time they run, and leaves the harness its own frames.* @position The harness's own test. A component spec trusts these* instruments; this file is why it may.*/@@ -28,6 +32,8 @@import {expect, test, type Page} from '@playwright/test';import {animationStoryboard,+ blockMainThread,+ busyWindowProof,compareCurve,cubicBezier,curveOf,@@ -38,6 +44,7 @@ import {holdAnimations,installInputLog,installTimerGate,+ markElement,merge,parseTransform,PARTIAL_LIFT_SKIP,@@ -47,12 +54,14 @@ import {sampleCurve,seekAnimations,springStep,+ startScreencast,Storyboard,tap,thawTimers,timeline,touchDriver,touchStoryboard,+ trackMarker,waitForAnimation,waitForRest,watchAnimations,@@ -71,6 +80,24 @@ const FIXTURE = `<!doctype html><meta name=viewport content="width=device-width,<div id=box></div><div id=scroller>${Array.from({length: 200}, (_, i) => `<div class=row>row ${i}</div>`).join('')}</div>`;+// The busy-main-thread proofs' page: one box with room to move 560 px down,+// nothing over it. 560 px in 1,600 ms is about 4.8 px a frame, more than two+// of WebKit's 2 px position steps (README: frame evidence), so every frame's+// motion is resolvable.+const BUSY_FIXTURE = `<!doctype html><meta name=viewport content="width=device-width,initial-scale=1">+<style>+ body { margin: 0; background: #fff; }+ #box { position: absolute; left: 20px; top: 40px; width: 80px; height: 80px; background: #36c; }+</style>+<div id=box></div>`;++async function busyFixture(page: Page) {+ await page.route('http://busy.test/', route =>+ route.fulfill({contentType: 'text/html', body: BUSY_FIXTURE}),+ );+ await page.goto('http://busy.test/');+}+// A real navigation (not setContent, whose document.open() erases the window// listeners the input log installs).async function fixture(page: Page) {@@ -242,6 +269,161 @@ test.describe('touch input', () => {});});+test.describe('frame evidence', () => {+ test('a compositor animation advances every frame through a blocked main thread; a rAF one freezes, whatever the screencast rate', async ({+ page,+ }, testInfo) => {+ await busyFixture(page);+ const box = page.locator('#box');+ await markElement(box, {mode: 'outline'});++ const run = async (driver: 'compositor' | 'raf', everyNthFrame = 1) => {+ await box.evaluate((el, how) => {+ el.getAnimations().forEach(a => a.cancel());+ el.style.transform = 'none';+ if (how === 'compositor') {+ el.animate(+ [{transform: 'translateY(0px)'}, {transform: 'translateY(560px)'}],+ {duration: 1600, easing: 'linear'},+ );+ } else {+ const start = performance.now();+ const step = (now: number) => {+ const p = Math.min(1, (now - start) / 1600);+ el.style.transform = `translateY(${560 * p}px)`;+ if (p < 1) {+ requestAnimationFrame(step);+ }+ };+ requestAnimationFrame(step);+ }+ }, driver);+ const cast = await startScreencast(page, {everyNthFrame});+ await page.waitForTimeout(200);+ const window = await blockMainThread(page, 700);+ await page.waitForTimeout(200);+ const tracked = await trackMarker(page, await cast.stop());+ return busyWindowProof(tracked, window, {axis: 'y'});+ };++ const compositor = await run('compositor');+ testInfo.annotations.push({+ type: 'busy-main-thread compositor',+ description: compositor.summary,+ });+ expect(compositor.verdict.failures, compositor.summary).toEqual([]);+ expect(compositor.monotonic).toBe(true);+ // Resolvable on the cast's grid: two position steps or more per frame.+ expect(compositor.notes, compositor.summary).toEqual([]);+ expect(compositor.pxPerFrame).toBeGreaterThanOrEqual(2 * compositor.stepPx);++ const raf = await run('raf');+ testInfo.annotations.push({+ type: 'busy-main-thread rAF control',+ description: raf.summary,+ });+ // Frozen for the whole block: no presented frame moved (the compositor+ // sends none at all when nothing on screen changes), and the verdict that+ // passes the compositor animation fails it on every criterion.+ expect(raf.changed).toBe(0);+ expect(raf.longestStallMs).toBeGreaterThan(600);+ expect(raf.verdict.pass).toBe(false);+ expect(raf.verdict.failures.join('\n')).toMatch(/longest stall/);+ expect(raf.verdict.failures.join('\n')).toMatch(/expected frames moved/);++ // The verdict judges the frames the screencast delivered, not its rate:+ // at one frame in six (a loaded machine's ~100 ms), the compositor+ // animation still passes and the rAF one still fails.+ const slow = await run('compositor', 6);+ expect(slow.cadenceMs, slow.summary).toBeGreaterThan(60);+ expect(slow.verdict.failures, slow.summary).toEqual([]);+ const slowRaf = await run('raf', 6);+ expect(slowRaf.verdict.pass, slowRaf.summary).toBe(false);+ expect(slowRaf.verdict.failures.join('\n')).toMatch(/longest stall/);+ });++ test('a block that starts during a just-started transition still lets the compositor advance it', async ({+ page,+ }, testInfo) => {+ // `during` blocks in the frame the transition has run 16 ms. In WebKit+ // the transition reaches the compositor in that frame's rendering update,+ // so a loop inside the rAF callback froze it (2 of 29 frames moved).+ // Linear and longer than the block, so every frame of the block moves two+ // position steps or more and the transition is still running when it+ // ends. The cast shows the box at rest first, so a transition that reaches+ // the screen after the block began is judged from there (onScreenFromMs).+ await busyFixture(page);+ const box = page.locator('#box');+ await markElement(box, {mode: 'outline'});+ const cast = await startScreencast(page);+ await expect.poll(() => cast.frameCount()).toBeGreaterThan(0);+ await box.evaluate(el => {+ el.style.transition = 'transform 1600ms linear';+ el.getBoundingClientRect();+ el.style.transform = 'translateY(560px)';+ });+ const window = await blockMainThread(page, 700, {+ during: box,+ minProgressMs: 16,+ });+ await waitForRest(box);+ const tracked = await trackMarker(page, await cast.stop());+ const proof = busyWindowProof(tracked, window, {axis: 'y'});+ testInfo.annotations.push({+ type: 'busy-main-thread transition',+ description: proof.summary,+ });+ expect(proof.verdict.failures, proof.summary).toEqual([]);+ expect(proof.monotonic).toBe(true);+ });++ test('a block never starts on an animation that is not on the compositor yet', async ({+ page,+ }, testInfo) => {+ // An animation created in the frame the helper looks is still pending (not+ // committed to the compositor). A block that started on it then froze it+ // in Chromium: no presented frame moved.+ await busyFixture(page);+ const box = page.locator('#box');+ await markElement(box, {mode: 'outline'});+ await box.evaluate(el => {+ el.style.willChange = 'transform';+ });+ const cast = await startScreencast(page);+ // A rAF loop registered before the helper's tick runs ahead of it in every+ // frame; on its 12th frame it creates the animation, and the helper's tick+ // looks in that same frame.+ await box.evaluate(el => {+ let n = 0;+ const loop = () => {+ n += 1;+ if (n === 12) {+ el.animate(+ [{transform: 'translateY(0px)'}, {transform: 'translateY(560px)'}],+ {duration: 1600, easing: 'linear'},+ );+ return;+ }+ requestAnimationFrame(loop);+ };+ requestAnimationFrame(loop);+ });+ const window = await blockMainThread(page, 700, {+ during: box,+ minProgressMs: 0,+ });+ await waitForRest(box);+ const tracked = await trackMarker(page, await cast.stop());+ const proof = busyWindowProof(tracked, window, {axis: 'y'});+ testInfo.annotations.push({+ type: 'busy-main-thread pending animation',+ description: proof.summary,+ });+ expect(proof.verdict.failures, proof.summary).toEqual([]);+ expect(proof.monotonic).toBe(true);+ });+});+test.describe('curve sampling', () => {test('a CSS transition samples onto its cubic-bezier to 0.05 px', async ({page,@@ -1429,6 +1611,33 @@ test.describe('storyboards', {tag: '@storyboard'}, () => {}});+ test("the harness's own waits run on the real timers: a frozen timer gate does not hold blockMainThread's block", async ({+ page,+ }) => {+ await fixture(page);+ await installTimerGate(page);+ const box = page.locator('#box');+ await box.evaluate(el => {+ el.animate(+ [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}],+ {duration: 1_500, easing: 'linear'},+ );+ });+ await freezeTimers(page);+ // Settled, or still held after 5 s.+ const within = (work: Promise<unknown>) =>+ Promise.race([+ work.then(() => 'settled'),+ new Promise<string>(resolve =>+ setTimeout(() => resolve('held'), 5_000),+ ),+ ]);+ expect(+ await within(blockMainThread(page, 100, {during: box, timeout: 2_000})),+ ).toBe('settled');+ await thawTimers(page);+ });+test('the timer gate holds a component timer until the filmstrip is done', async ({page,}, testInfo) => {diff --git a/internal/interaction-harness/src/index.ts b/internal/interaction-harness/src/index.tsindex 537bfc4d4b..2bcb127f36 100644--- a/internal/interaction-harness/src/index.ts+++ b/internal/interaction-harness/src/index.ts@@ -12,7 +12,9 @@* SYNC: When a module is added, export it here and describe it in README.md.*/+export * from './busy';export * from './curves';+export * from './frames';export * from './math';export * from './output';export * from './scripts';diff --git a/internal/interaction-harness/src/story.ts b/internal/interaction-harness/src/story.tsindex 598f75d852..5a1f2237c6 100644--- a/internal/interaction-harness/src/story.ts+++ b/internal/interaction-harness/src/story.ts@@ -119,7 +119,7 @@ const TIMER_GATE = `(() => {const realInterval = window.setInterval.bind(window);const realFrame = window.requestAnimationFrame.bind(window);const realCancelFrame = window.cancelAnimationFrame.bind(window);- const gate = {frozen: false, queue: [], frame: realFrame};+ const gate = {frozen: false, queue: [], frame: realFrame, timeout: realTimeout};window.__harnessTimerGate = gate;const drop = key => { gate.queue = gate.queue.filter(entry => entry.key !== key); };window.setTimeout = (fn, ms, ...args) => {