Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

test(interaction-harness): a starved waitForRest fails by name (#10505)

Christopher Chedeau9ba05488061b
diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.md
index 046916629c..41f384802a 100644
--- a/internal/interaction-harness/README.md
+++ b/internal/interaction-harness/README.md
@@ -284,7 +284,8 @@ decode only those. Pass either as `control`: the component passes when it moved
moved and its longest stall is no longer than the control's. Both are read from
the same screencast frames, so a compositor animation misses none. A window in
which the control moved on fewer than four frames fails, since the machine
-presented too little to judge. A run of frames with no new position counts as
+presented too little to judge; the failure names the cast's frames in the
+window and its longest gap. A run of frames with no new position counts as
moved when the change that ends it is a single cast step: the component moves
under one step a frame there, as on an easing tail, which the cast cannot show
on every frame. Where it moved under two steps a frame over the frames just
@@ -361,6 +362,10 @@ starve the page of frames. When no frame came for longer than
with `FrameStarvationError`, which names the frames drawn and the longest gap.
So a run that cannot prove anything still fails, and says why. With too few frames on pace, the motion did not last that
long, and the error says that instead.
+`waitForRest` fails the same way when it cannot tell. The element held still
+in its last frame, or no frame came after the deadline, and no frame came for
+longer than `FRAME_STARVATION_MS`: it throws `FrameStarvationError`, not
+"still moving".
**No Layout / Paint.** `traceRendering` records a Chromium trace around the
interaction; `countBetween` counts the steady middle of an animation (its
diff --git a/internal/interaction-harness/src/busy.control.test.ts b/internal/interaction-harness/src/busy.control.test.ts
index debb196d33..eabcea69fb 100644
--- a/internal/interaction-harness/src/busy.control.test.ts
+++ b/internal/interaction-harness/src/busy.control.test.ts
@@ -273,8 +273,9 @@ describe('busyWindowProof against a control', () => {
moving().map(f => ({...f, box: {...f.box!, y: 0}}) as TrackedFrame),
);
const proof = busyWindowProof(moving(), BLOCK, {control: still});
+ // It names how starved the cast was: here every frame came, 60 Hz.
expect(proof.verdict.failures.join('\n')).toMatch(
- /the control moved on 0 frames in the judged part/,
+ /the control moved on 0 frames in the judged part, under 4: the compositor presented too little in this window to judge \(the cast delivered \d+ frames in the \d+ ms block, none for 1[67] ms\)/,
);
});
});
diff --git a/internal/interaction-harness/src/busy.ts b/internal/interaction-harness/src/busy.ts
index 6b5aa0de5c..93f4117e81 100644
--- a/internal/interaction-harness/src/busy.ts
+++ b/internal/interaction-harness/src/busy.ts
@@ -1480,8 +1480,17 @@ export function busyWindowProof(
}
if (controlRead != null) {
if (controlRead.moved < minJudgedFrames) {
+ // How starved the cast was over the whole block, as the pacing and
+ // frame errors say it.
+ const blockFrames = tracked
+ .filter(f => f.box != null && f.t > window.from && f.t < window.to)
+ .map(f => f.t);
+ const castEdges = [window.from, ...blockFrames, window.to];
+ const castGapMs = Math.max(
+ ...castEdges.slice(1).map((t, k) => t - castEdges[k]),
+ );
failures.push(
- `the control moved on ${controlRead.moved} frames in the judged part, under ${minJudgedFrames}: the compositor presented too little in this window to judge`,
+ `the control moved on ${controlRead.moved} frames in the judged part, under ${minJudgedFrames}: the compositor presented too little in this window to judge (the cast delivered ${blockFrames.length} frame${blockFrames.length === 1 ? '' : 's'} in the ${Math.round(window.to - window.from)} ms block, none for ${Math.round(castGapMs)} ms)`,
);
}
const visible = controlRead.movedWith - controlRead.belowStep;
diff --git a/internal/interaction-harness/src/curves.ts b/internal/interaction-harness/src/curves.ts
index bf81f33443..58cda652a1 100644
--- a/internal/interaction-harness/src/curves.ts
+++ b/internal/interaction-harness/src/curves.ts
@@ -25,6 +25,7 @@
import type {Locator, Page} from '@playwright/test';
import {defineCurveEnd} from './curve-end';
+import {FRAME_STARVATION_MS, FrameStarvationError} from './frames';
import {parseTransform, type TimedValue, type Transform2D} from './math';
/** How far inside its end a seek past an animation's end lands, ms. */
@@ -2006,6 +2007,12 @@ export async function watchAnimations(
* easing or direction, waitForRest waits the hold out, and a hold longer than
* `timeout` rejects with "still moving".
*
+ * It throws `FrameStarvationError` instead when the page drew too few frames
+ * to tell. Then, since the element last moved, no frame came for longer than
+ * `FRAME_STARVATION_MS`. It held still in its last frame, or no frame came
+ * after the deadline. So the still frames must come by `timeout` plus
+ * `FRAME_STARVATION_MS`.
+ *
* While a rendering trace runs (`traceRendering`), it stands down as the
* animation recorder does: a per-frame callback would make every frame a
* main-thread frame, which the trace counts. It then waits on timers for the
@@ -2031,9 +2038,13 @@ export async function waitForRest(
locator,
Date.now() + timeout + 1_000,
async evaluateTimeout => {
- await locator.evaluate(
- (element, {ms, need}) =>
- new Promise<void>((resolve, reject) => {
+ const starved = await locator.evaluate(
+ (element, {ms, need, starvationMs}) =>
+ new Promise<{
+ frames: number;
+ spanMs: number;
+ longestGapMs: number;
+ } | null>((resolve, reject) => {
// The deadline reads the real clock, which a page freeze does not
// stop.
const realNow =
@@ -2065,29 +2076,66 @@ export async function waitForRest(
__harnessTimerGate?: {
frame: typeof requestAnimationFrame;
timeout?: typeof setTimeout;
+ clearTimeout?: typeof clearTimeout;
};
}
).__harnessTimerGate;
const frame = gate?.frame ?? requestAnimationFrame;
const timeout = gate?.timeout ?? setTimeout;
+ const clearTimer = gate?.clearTimeout ?? clearTimeout;
// Set by traceRendering while it traces (see the recorder above).
const tracing = () =>
(window as unknown as {__harnessRenderingTrace?: boolean})
.__harnessRenderingTrace === true;
+ // The deadline timer below holds this wait's scope, the element
+ // too, until it fires: every ending clears it.
+ const end = () => clearTimer(deadlineTimer);
const settle = () => {
+ end();
// Later waits count animations from here.
(
window as unknown as {__harnessAnimations?: {arm: () => number}}
).__harnessAnimations?.arm();
- resolve();
+ resolve(null);
};
let last = '';
let still = 0;
+ // When each frame of the per-frame check came since the element
+ // last moved, on the real clock: only those can show it at rest.
+ let watchedFrom = realNow();
+ const drawn: number[] = [];
+ const shortfall = () => {
+ const times = [watchedFrom, ...drawn, realNow()];
+ let longestGapMs = 0;
+ for (let i = 1; i < times.length; i += 1) {
+ longestGapMs = Math.max(longestGapMs, times[i] - times[i - 1]);
+ }
+ return {
+ frames: drawn.length,
+ spanMs: realNow() - watchedFrom,
+ longestGapMs,
+ };
+ };
const frames = () => {
last = read();
still = 0;
+ watchedFrom = realNow();
+ drawn.length = 0;
frame(tick);
};
+ // A frame that never comes cannot reach the deadline check in
+ // `tick`: a starved page is caught here.
+ let starvedOut = false;
+ const deadlineTimer = timeout(
+ () => {
+ const lastFrame = drawn[drawn.length - 1] ?? watchedFrom;
+ if (!tracing() && realNow() - lastFrame > starvationMs) {
+ starvedOut = true;
+ resolve(shortfall());
+ }
+ },
+ ms + starvationMs + 1,
+ );
// While traced: one look at the animations, then a timer to the latest
// scheduled curve end. One not started yet is looked at again a frame
// later; a trace that has ended hands over to the per-frame check.
@@ -2134,6 +2182,7 @@ export async function waitForRest(
timeout(
() => {
if (realNow() > deadline) {
+ end();
reject(
new Error('waitForRest: still moving at the deadline'),
);
@@ -2147,6 +2196,9 @@ export async function waitForRest(
);
};
const tick = () => {
+ if (starvedOut) {
+ return;
+ }
const now = read();
const moving = element.getAnimations({subtree: true}).some(
a =>
@@ -2159,12 +2211,25 @@ export async function waitForRest(
Number(a.effect?.getComputedTiming().delay ?? 0)
: Number(a.currentTime ?? 0) < curveEnd(a))),
);
- still = !moving && now === last ? still + 1 : 0;
+ const resting = !moving && now === last;
+ if (resting) {
+ drawn.push(realNow());
+ } else {
+ watchedFrom = realNow();
+ drawn.length = 0;
+ }
+ still = resting ? still + 1 : 0;
last = now;
if (still >= need) {
settle();
} else if (realNow() > deadline) {
- reject(new Error(`waitForRest: still moving (${now})`));
+ const short = shortfall();
+ end();
+ if (resting && short.longestGapMs > starvationMs) {
+ resolve(short);
+ } else {
+ reject(new Error(`waitForRest: still moving (${now})`));
+ }
} else if (tracing()) {
scheduled();
} else {
@@ -2177,9 +2242,18 @@ export async function waitForRest(
frames();
}
}),
- {ms: timeout, need: frames},
+ {ms: timeout, need: frames, starvationMs: FRAME_STARVATION_MS},
{timeout: evaluateTimeout},
);
+ if (starved != null) {
+ throw new FrameStarvationError({
+ by: 'waitForRest',
+ frames: starved.frames,
+ during: `it waited ${Math.round(starved.spanMs)} ms for the element to rest`,
+ min: frames,
+ longestGapMs: starved.longestGapMs,
+ });
+ }
return true;
},
);
diff --git a/internal/interaction-harness/src/frames.requireFrames.test.ts b/internal/interaction-harness/src/frames.requireFrames.test.ts
index 792304123e..adf881308d 100644
--- a/internal/interaction-harness/src/frames.requireFrames.test.ts
+++ b/internal/interaction-harness/src/frames.requireFrames.test.ts
@@ -6,7 +6,8 @@
* @output Unit cases for `requireFrames`. Enough selected frames come back.
* Too few, where no frame came for longer than FRAME_STARVATION_MS, fail
* with FrameStarvationError and its numbers. Too few on pace fail with a
- * plain Error. With none selected, every gap counts.
+ * plain Error. With none selected, every gap counts. A page that drew one
+ * frame at most in the whole sample was starved.
* @position Node unit tests beside the harness's browser self-tests.
*
* SYNC: When modified, update frames.ts (requireFrames).
@@ -86,4 +87,20 @@ describe('requireFrames', () => {
requireFrames(onPace, {min: 1, during: 'a push', where: () => false}),
).toThrow(/the motion did not last that long/);
});
+
+ it('names a page that drew one frame in the whole sample as starved', () => {
+ let error: unknown;
+ try {
+ requireFrames(trace([0]), {min: 2, during: 'a slide'});
+ } catch (thrown) {
+ error = thrown;
+ }
+ expect(error).toBeInstanceOf(FrameStarvationError);
+ expect((error as FrameStarvationError).numbers.longestGapMs).toBe(
+ Number.POSITIVE_INFINITY,
+ );
+ expect(String(error)).toMatch(
+ /the page drew 1 frame while a slide, where the proof needs 2\. No second frame came: the page was starved of frames/,
+ );
+ });
});
diff --git a/internal/interaction-harness/src/frames.ts b/internal/interaction-harness/src/frames.ts
index 89fc41475b..3578d044d2 100644
--- a/internal/interaction-harness/src/frames.ts
+++ b/internal/interaction-harness/src/frames.ts
@@ -806,13 +806,18 @@ const framesDrawn = (count: number) =>
/** A proof's frames when the page drew too few of them. */
export interface FrameShortfall {
+ /** The harness function that needed the frames (default `requireFrames`). */
+ readonly by?: string;
/** What the frames had to show. */
readonly during: string;
/** The frames the page drew during it. */
readonly frames: number;
/** The frames the proof needs. */
readonly min: number;
- /** The longest gap between frames around them, ms. */
+ /**
+ * The longest gap between frames around them, ms; Infinity when the page
+ * drew fewer than two frames.
+ */
readonly longestGapMs: number;
}
@@ -826,7 +831,7 @@ export class FrameStarvationError extends Error {
readonly numbers: FrameShortfall;
constructor(numbers: FrameShortfall) {
super(
- `requireFrames: the page drew ${framesDrawn(numbers.frames)} while ${numbers.during}, where the proof needs ${numbers.min}. No frame came for ${numbers.longestGapMs.toFixed(1)} ms: the page was starved of frames (a loaded machine, or a long task on the page), so the run proves nothing about the motion`,
+ `${numbers.by ?? 'requireFrames'}: the page drew ${framesDrawn(numbers.frames)} while ${numbers.during}, where the proof needs ${numbers.min}. ${Number.isFinite(numbers.longestGapMs) ? `No frame came for ${numbers.longestGapMs.toFixed(1)} ms` : 'No second frame came'}: the page was starved of frames (a loaded machine, or a long task on the page), so the run proves nothing about the motion`,
);
this.name = 'FrameStarvationError';
this.numbers = numbers;
@@ -865,10 +870,12 @@ export function requireFrames<T extends {readonly t: number}>(
? samples.indexOf(selected[selected.length - 1])
: samples.length - 1;
const around = samples.slice(Math.max(0, first - 1), last + 2);
- const longestGapMs = Math.max(
- 0,
- ...around.slice(1).map((sample, i) => sample.t - around[i].t),
- );
+ // With fewer than two frames there is no gap to measure. The page drew one
+ // frame at most in the whole sample, so it was starved.
+ const longestGapMs =
+ around.length < 2
+ ? Number.POSITIVE_INFINITY
+ : Math.max(...around.slice(1).map((sample, i) => sample.t - around[i].t));
const numbers = {during, frames: selected.length, min, longestGapMs};
if (longestGapMs > FRAME_STARVATION_MS) {
throw new FrameStarvationError(numbers);
diff --git a/internal/interaction-harness/src/harness.interaction.browser.spec.ts b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
index ad373e1125..53006e8136 100644
--- a/internal/interaction-harness/src/harness.interaction.browser.spec.ts
+++ b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
@@ -1925,6 +1925,83 @@ test.describe('frame evidence', () => {
expect(numbers.longestGapMs).toBeGreaterThan(FRAME_STARVATION_MS);
});
+ test('waitForRest on a page starved of frames fails by name, and an element still in motion still says so', async ({
+ page,
+ }) => {
+ await page.setContent(
+ '<!doctype html><div id="box" style="width:40px;height:40px;background:#000"></div>',
+ );
+ const box = page.locator('#box');
+ await waitForRest(box, {timeout: 1000});
+ // Motion at the deadline, on pace.
+ await box.evaluate(el =>
+ el.animate(
+ [{transform: 'translateX(0)'}, {transform: 'translateX(100px)'}],
+ {duration: 3000},
+ ),
+ );
+ await expect(waitForRest(box, {timeout: 500})).rejects.toThrow(
+ /waitForRest: still moving \(/,
+ );
+ await box.evaluate(el => {
+ for (const animation of el.getAnimations()) {
+ animation.cancel();
+ }
+ });
+ // Script motion every third frame, with one 120 ms long task early in
+ // the wait. Still frames between the moves are no starvation, and nor is
+ // a gap long before the deadline.
+ await box.evaluate(el => {
+ let n = 0;
+ const step = () => {
+ n += 1;
+ if (n % 3 === 0) {
+ (el as HTMLElement).style.transform = `translateX(${n % 90}px)`;
+ }
+ if (el.isConnected && el.id === 'box') {
+ requestAnimationFrame(step);
+ }
+ };
+ requestAnimationFrame(step);
+ setTimeout(() => {
+ const end = performance.now() + 120;
+ while (performance.now() < end) {
+ // the page's main thread is busy
+ }
+ }, 100);
+ });
+ await expect(waitForRest(box, {timeout: 900})).rejects.toThrow(
+ /waitForRest: still moving \(/,
+ );
+ // Stops the script motion: the element rests again.
+ await box.evaluate(el => {
+ el.id = 'rested';
+ (el as HTMLElement).style.transform = '';
+ });
+ await waitForRest(page.locator('#rested'), {timeout: 1000});
+ await page.locator('#rested').evaluate(el => {
+ el.id = 'box';
+ });
+ // Each frame now comes 1,000 ms after the one before, as on a starved
+ // machine: the element rests, but too few frames come to tell.
+ await page.evaluate(() => {
+ const frame = window.requestAnimationFrame.bind(window);
+ window.requestAnimationFrame = callback => {
+ setTimeout(() => frame(callback), 1000);
+ return 0;
+ };
+ });
+ const outcome = await waitForRest(box, {timeout: 1500}).then(
+ () => 'resolved',
+ (error: unknown) => error,
+ );
+ expect(outcome).toBeInstanceOf(FrameStarvationError);
+ const {numbers} = outcome as FrameStarvationError;
+ expect(numbers.frames).toBeLessThan(3);
+ expect(numbers.longestGapMs).toBeGreaterThan(FRAME_STARVATION_MS);
+ expect((outcome as Error).message).toMatch(/^waitForRest: the page drew /);
+ });
+
test('the trace counts no Layout for a transform animation and one per frame for `top`', async ({
page,
browserName,
diff --git a/internal/interaction-harness/src/story.ts b/internal/interaction-harness/src/story.ts
index fcf5a8358e..1e6fdb8c2c 100644
--- a/internal/interaction-harness/src/story.ts
+++ b/internal/interaction-harness/src/story.ts
@@ -177,6 +177,8 @@ export async function frames(page: Page, count = 2): Promise<void> {
// does not stop. The gate's freeze() and thaw() are its one way in:
// freezeTimers and thawTimers call them, and so does a page freeze
// (freeze.ts), which stops the page's clock too.
+// `clearTimeout` is the real clearTimeout, so a harness wait that ends early
+// cancels its own timer.
const TIMER_GATE = `(() => {
if (window.__harnessTimerGate) return;
const realTimeout = window.setTimeout.bind(window);
@@ -202,6 +204,7 @@ const TIMER_GATE = `(() => {
frame: realFrame,
now: realNow,
timeout: realTimeout,
+ clearTimeout: realClearTimeout,
// Holds every callback that comes due. With the page's clock stopped as
// well, the timeouts not yet due stop counting.
freeze(isClockStopped = false) {