An open source design system that's fully customizable and agent ready
TypeScript
test(interaction-harness): a busy proof can bound its settle in the cast's own steps (#9545)
Christopher Chedeau547d90a59b79
diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.mdindex c4371c5452..f9c17469db 100644--- a/internal/interaction-harness/README.md+++ b/internal/interaction-harness/README.md@@ -144,7 +144,13 @@ new position every frame of the block, and a main-thread animation shows none(nothing changed, so no frame is sent at all: `busyWindowProof` counts thewhole window as one stall). `settledWithin` (2 px) stops judging at the easingtail: the screencast rounds to whole pixels, so a tail that still moves shows-plateaus within a pixel or two of rest. A judged part shorter than four of the+plateaus within a pixel or two of rest. `settledWithinSteps` counts that bound+in the cast's own position steps, read from the cast (`castStep`, from the+`step` `trackMarker` gives each frame of the default marker; a control tracked+by another color carries none, and `castStep` refuses it): 1 px in Chromium's PNG cast and 2 px in+WebKit's JPEG cast, whose 4:2:0 chroma holds the marker's color in 2 by 2+pixel blocks and so makes the tail's plateaus longer. Two steps stop at the+same tail in either engine while a freeze further from rest is still judged. A judged part shorter than four of thescreencast's intervals fails: the animation under test was not moving duringthe block. With`during`, `blockMainThread` starts the loop from a task queued in the frame@@ -182,9 +188,9 @@ late on a loaded runner, too late to wait on before blocking. Only a hold thatstarts before the block, on two frames at least, is excused, and every frameafter its first change is judged, so a freeze later in the block still fails.Any other element already moving is judged from the block's start. WebKit's-casts are 2 px quantized: the frames come at the-viewport's size, but the page in them is captured at half resolution, so-positions move in 2 CSS px steps (Chromium's in 1). With under two steps of+casts are 2 px quantized: their JPEG frames hold the+marker's color in 2 by 2 pixel chroma blocks (4:2:0, which the frame header+reports), so positions move in 2 CSS px steps (Chromium's in 1). With under two steps ofmotion per frame, a frame delivered a few ms early shows no step and reads asnot moved. The proof reports `stepPx` and `pxPerFrame` and notes each stretchthat moves under two steps per frame, read over four frames at a time so a@@ -215,10 +221,16 @@ presented too little to judge. A run of frames with no new position counts asmoved when the change that ends it is a single cast step: the component movesunder one step a frame there, as on an easing tail, which the cast cannot showon every frame. Where it moved under two steps a frame over the frames just-before the run (a spring's slow tail inside a fast window), a frame delivered-early can show no step too, so the change may carry the motion of every frame+before the run (a spring's slow tail inside a fast window), or across the run+itself (its catch-up over the frames it spans) where the curve slows faster+than those frames show, a frame delivered early can show no step too, so the change may carry the motion of every framethe run held at that pace, to the nearest step. The run must also read as a-slowing curve, at most one frame longer than the component's run before it. A+slowing curve, at most one frame longer than the component's run before it.+Under `settledWithinSteps`, a run under one cast step a frame may instead last+ceil(step ÷ travel) frames, whatever the run before it. The travel is the+slower moving pace around the run, never less than its own catch-up per frame.+The run also lasts at most one frame more than the pace into it allows:+ceil(step ÷ max(pace before, catch-up per frame)) + 1. Afreeze ends in a catch-up of more than that, a stall in a frame-counted animation is longer than its runsbefore it, and a run that no change ends stays a miss. The component still hasto show a new position on at least four of the control's frames. The cast step (`stepPx`)diff --git a/internal/interaction-harness/src/busy.test.ts b/internal/interaction-harness/src/busy.test.tsindex 20c5b0f614..0ad22a1bcf 100644--- a/internal/interaction-harness/src/busy.test.ts+++ b/internal/interaction-harness/src/busy.test.ts@@ -2,20 +2,25 @@/*** @file busy.test.ts- * @input Synthetic tracked frames (no browser)+ * @input Synthetic tracked frames and recorded runs (no browser)* @output The busy-main-thread verdict's limits: dropped frames a loaded* machine produces pass, a frozen animation fails every criterion, a window* with nothing moving to judge fails, and out-of-order frames are judged in- * presentation order.+ * presentation order; a settle bound in cast steps reads the step from the+ * run (2 px in WebKit's cast, 1 px in Chromium's), stops a WebKit coast at+ * its tail's plateaus where a 2 px bound reads them as a stall, and still+ * fails a 40 ms freeze more than two steps from rest in either cast.* @position Unit test of ./busy and ./frames' ordering; the browser proof* with a real compositor animation and a rAF control is in* harness.interaction.browser.spec.ts.*/import {describe, expect, it} from 'vitest';-import {busyWindowProof} from './busy';+import {busyWindowProof, castStep} from './busy';import * as skip from './busy.skip.fixture';import * as slowTail from './busy.slowtail.fixture';+import * as webkitDecel from './busy.webkitdecel.fixture';+import * as webkitTail from './busy.webkittail.fixture';import {inPresentationOrder, type TrackedFrame} from './frames';const FRAME = 1000 / 60;@@ -653,3 +658,378 @@ describe('busyWindowProof in a slow tail', () => {expect(proof.verdict.failures.join('\n')).toMatch(/moved on \d+ of/);});});++describe('busyWindowProof settledWithinSteps', () => {+ /** Holds the position the frame at or before `fromT` shows for `ms`. */+ const freeze = (+ frames: readonly TrackedFrame[],+ fromT: number,+ ms: number,+ ): TrackedFrame[] => {+ const held = [...frames].reverse().find(f => f.t <= fromT)!.box!;+ return frames.map(f =>+ f.t > fromT && f.t <= fromT + ms+ ? ({...f, box: {...held}} as TrackedFrame)+ : f,+ );+ };+ /** Frames as `trackMarker` returns them, with the cast's step. */+ const cast = (+ frames: ReadonlyArray<readonly [number, number | null]>,+ step: {x: number; y: number},+ ): TrackedFrame[] => slowTailFrames(frames).map(f => ({...f, step}));+ // Chromium's cast is a PNG at the viewport's size: 1 px a step.+ const PNG = {x: 1, y: 1};+ const chromium = (frames: ReadonlyArray<readonly [number, number | null]>) =>+ cast(frames, PNG);+ const webkit = (frames: ReadonlyArray<readonly [number, number | null]>) =>+ cast(frames, webkitTail.step);++ it("reads the cast's position step from its frames, not from the motion", () => {+ expect(castStep(webkit(webkitTail.tracked), 'x')).toBe(2);+ expect(castStep(chromium(slowTail.tracked), 'x')).toBe(1);+ expect(() => castStep(slowTailFrames(slowTail.tracked), 'x')).toThrow(+ /no tracked frame carries the cast's step/,+ );+ });++ it("judges a WebKit coast to two of its cast's 2 px steps from rest, where a 2 px bound reads the tail's plateaus as a stall", () => {+ const control = webkit(webkitTail.control);+ const tracked = webkit(webkitTail.tracked);+ // Its tail during the block: 212, 212, 214, 214, 216 for four frames,+ // then 218 from +225 ms, the control moving on every frame.+ const inPx = busyWindowProof(tracked, webkitTail.block, {+ axis: 'x',+ settledWithin: 2,+ control,+ });+ expect(inPx.verdict.failures.join('\n')).toMatch(+ /longest stall 54\.\d ms, longer than the control's/,+ );+ const inSteps = busyWindowProof(tracked, webkitTail.block, {+ axis: 'x',+ settledWithinSteps: 2,+ control,+ });+ expect(inSteps.verdict.failures).toEqual([]);+ expect(inSteps.judged.to).toBeCloseTo(170, 0);+ });++ it('on a 1 px cast, judges a run whose changes are all even as settledWithin: 2 does', () => {+ // A coast toward 200 px slowing 12, 10, 8, 6, 4 and 2 px a frame, held+ // three frames at 196, 4 px from rest, the control moving every frame:+ // its changes share a factor of 2, but its cast's step is 1 px.+ const values = [+ ...Array<number>(4).fill(146),+ 158,+ 168,+ 176,+ 182,+ 186,+ 188,+ 190,+ 192,+ 194,+ 196,+ 196,+ 196,+ 196,+ 198,+ ...Array<number>(14).fill(200),+ ];+ const tracked = values.map(+ (x, i) =>+ ({+ t: -50 + i * 16.7,+ box: {x, y: 0, width: 1, height: 1},+ pixels: 1,+ step: PNG,+ }) as TrackedFrame,+ );+ const control = values.map(+ (_, i) =>+ ({+ t: -50 + i * 16.7,+ box: {x: 10 * i, y: 0, width: 1, height: 1},+ pixels: 1,+ step: PNG,+ }) as TrackedFrame,+ );+ const block = {from: 0, to: 400};+ const inPx = busyWindowProof(tracked, block, {+ axis: 'x',+ settledWithin: 2,+ control,+ });+ const inSteps = busyWindowProof(tracked, block, {+ axis: 'x',+ settledWithinSteps: 2,+ control,+ });+ expect(inPx.verdict.failures.join('\n')).toMatch(/longest stall/);+ expect(inSteps.judged).toEqual(inPx.judged);+ expect(inSteps.verdict).toEqual(inPx.verdict);+ });++ it('under one step a frame, reads a two-frame plateau right after two changes in a row as the cast’s step, not a stall', () => {+ // The tail shows 214 px one frame early, at +132 ms: 212, then 214 for+ // three frames, then 216 at +170. At about 0.7 px a frame the 2 px+ // position changes every three frames, so the plateau is the cast's.+ const tracked = webkit(webkitTail.tracked).map(f =>+ f.t > 130 && f.t < 135 && f.box != null+ ? ({...f, box: {...f.box, x: 214}} as TrackedFrame)+ : f,+ );+ const proof = busyWindowProof(tracked, webkitTail.block, {+ axis: 'x',+ settledWithinSteps: 2,+ control: webkit(webkitTail.control),+ });+ expect(proof.verdict.failures).toEqual([]);+ // A pixel bound reads it by the run before it, as before: a stall.+ const inPx = busyWindowProof(tracked, webkitTail.block, {+ axis: 'x',+ settledWithin: 4,+ control: webkit(webkitTail.control),+ });+ expect(inPx.verdict.failures.join('\n')).toMatch(+ /longest stall \d+\.\d ms, longer than the control's/,+ );+ });++ it('under one step a frame, still fails a long hold near rest, where the frames after the change no longer move', () => {+ // The tail holds 214 px, 4 px from rest, until the change to 218 px. It+ // holds for 82 ms (to +225 ms) and for 139 ms (to +282 ms). The frames+ // after the change are the settled plateau, so their pace is 0. The+ // motion before the hold still bounds how long one position may show.+ for (const holdTo of [225, 282]) {+ const tracked = webkit(webkitTail.tracked).map(f =>+ f.t > 160 && f.t < holdTo - 5 && f.box != null+ ? ({...f, box: {...f.box, x: 214}} as TrackedFrame)+ : f,+ );+ const proof = busyWindowProof(tracked, webkitTail.block, {+ axis: 'x',+ settledWithinSteps: 2,+ control: webkit(webkitTail.control),+ });+ expect(+ proof.verdict.failures.join('\n'),+ `held to +${holdTo} ms`,+ ).toMatch(/longest stall \d+\.\d ms, longer than the control's/);+ }+ });++ it('under one step a frame, fails a stall whose slow tail would excuse it: the allowance stays near the pace before the run', () => {+ // The tail holds 212 px for four more frames, 82 ms in all, then replays+ // the rest of the coast four frames late. The slow frames after the stall+ // set no allowance of their own. A pixel bound of the same 4 px fails it+ // too.+ const original = webkit(webkitTail.tracked);+ const held = original.findIndex(f => f.t > 130 && f.t < 135);+ const tracked = original.map((f, i) => {+ if (f.box == null || i <= held) {+ return f;+ }+ const x = i <= held + 4 ? 212 : original[i - 4].box?.x;+ return x == null ? f : ({...f, box: {...f.box, x}} as TrackedFrame);+ });+ const control = webkit(webkitTail.control);+ const inSteps = busyWindowProof(tracked, webkitTail.block, {+ axis: 'x',+ settledWithinSteps: 2,+ control,+ });+ expect(inSteps.verdict.failures.join('\n')).toMatch(+ /longest stall 82\.\d ms, longer than the control's/,+ );+ const inPx = busyWindowProof(tracked, webkitTail.block, {+ axis: 'x',+ settledWithin: 4,+ control,+ });+ expect(inPx.verdict.failures.join('\n')).toMatch(/longest stall 82\.\d ms/);+ });++ it('under one step a frame, still fails a three-frame freeze whose motion resumes from where it held', () => {+ // 1.2 px a frame on a 2 px cast, the control 3 px a frame. The coast+ // holds three frames, then moves on from where it held, so the change+ // that ends the still run is one step: the pace around the run, not+ // across it, bounds the run.+ const q = (x: number) => 2 * Math.floor(x / 2);+ const at = (t: number, x: number) =>+ ({+ t,+ box: {x, y: 0, width: 1, height: 1},+ step: {x: 2, y: 2},+ }) as TrackedFrame;+ const tracked: TrackedFrame[] = [];+ const control: TrackedFrame[] = [];+ let x = 100;+ for (let n = -12; n <= 54; n += 1) {+ if (n < 18 || n >= 21) {+ x += 1.2;+ }+ tracked.push(at(n * 16.7, q(x)));+ control.push(at(n * 16.7, q(3 * (n + 12))));+ }+ const proof = busyWindowProof(+ tracked,+ {from: 0, to: 700},+ {axis: 'x', settledWithinSteps: 2, control},+ );+ expect(proof.verdict.failures.join('\n')).toMatch(+ /longest stall \d+\.\d ms, longer than the control's/,+ );+ });++ it("still fails a 40 ms freeze in a WebKit coast's tail, more than two steps from rest", () => {+ // Held at 200 px, 20 px from rest, then a 10 px catch-up.+ const proof = busyWindowProof(+ freeze(webkit(webkitTail.tracked), 48, 40),+ webkitTail.block,+ {+ axis: 'x',+ settledWithinSteps: 2,+ control: webkit(webkitTail.control),+ },+ );+ expect(proof.verdict.failures.join('\n')).toMatch(+ /longest stall \d+\.\d ms, longer than the control's/,+ );+ });++ it('still fails a 40 ms freeze in a Chromium tail, more than two steps from rest', () => {+ const proof = busyWindowProof(+ freeze(chromium(slowTail.tracked), 600, 40),+ slowTail.block,+ {+ axis: 'x',+ settledWithinSteps: 2,+ control: chromium(slowTail.control),+ },+ );+ expect(proof.verdict.failures.join('\n')).toMatch(+ /longest stall \d+\.\d ms, longer than the control's/,+ );+ });++ it("reads a held frame in a fast-slowing WebKit coast at the run's own pace, under two steps a frame", () => {+ // 4.5 px a frame over the four frames before the hold, 2.3 across it:+ // the frame at +52 ms shows no step while the control moves, and the+ // next catches up 4 px, two frames' motion at the run's pace.+ const proof = busyWindowProof(+ webkit(webkitDecel.tracked),+ webkitDecel.block,+ {+ axis: 'x',+ settledWithinSteps: 2,+ control: webkit(webkitDecel.control),+ },+ );+ expect(proof.verdict.failures).toEqual([]);+ });++ it("reads the run's own pace only for a step-counted bound: with settledWithin, that held frame fails as before", () => {+ // The same coast with a pixel bound: the pace before the hold alone+ // judges the held frame, as for every proof that does not opt in.+ const proof = busyWindowProof(+ webkit(webkitDecel.tracked),+ webkitDecel.block,+ {+ axis: 'x',+ settledWithin: 4,+ control: webkit(webkitDecel.control),+ },+ );+ expect(proof.verdict.failures.join('\n')).toMatch(+ /moved on 7 of the 8 frames the control moved/,+ );+ });++ it('reads a three-frame plateau that ends in a two-step change at about 1.3 px a frame as slow motion', () => {+ // Held at 192 px from +63 ms through +91, then 196 at +109.+ const proof = busyWindowProof(+ freeze(webkit(webkitDecel.tracked), 80, 20),+ webkitDecel.block,+ {+ axis: 'x',+ settledWithinSteps: 2,+ control: webkit(webkitDecel.control),+ },+ );+ expect(proof.verdict.failures).toEqual([]);+ });++ it('still fails that plateau when its catch-up carries the time it held at the pace before it', () => {+ // The same hold, then a jump to 200 px, where 2.3 px a frame would have+ // carried it, and the rest of the coast 4 px further on.+ const tracked = webkit(webkitDecel.tracked).map(f =>+ f.box == null+ ? f+ : f.t > 80 && f.t <= 100+ ? ({...f, box: {...f.box, x: 192}} as TrackedFrame)+ : f.t > 100+ ? ({...f, box: {...f.box, x: f.box.x + 4}} as TrackedFrame)+ : f,+ );+ const proof = busyWindowProof(tracked, webkitDecel.block, {+ axis: 'x',+ settledWithinSteps: 2,+ control: webkit(webkitDecel.control),+ });+ expect(proof.verdict.failures.join('\n')).toMatch(+ /moved on \d+ of the \d+ frames the control moved/,+ );+ });++ it('still fails a one-frame hold in a WebKit coast moving two steps a frame or more', () => {+ // Held at 180 px at +25 ms, where the coast moves about 4.5 px a frame:+ // the next frame catches up 8 px, two frames' motion at that pace.+ const proof = busyWindowProof(+ freeze(webkit(webkitDecel.tracked), 7, 20),+ webkitDecel.block,+ {+ axis: 'x',+ settledWithinSteps: 2,+ control: webkit(webkitDecel.control),+ },+ );+ expect(proof.verdict.failures.join('\n')).toMatch(+ /moved on 7 of the 8 frames the control moved/,+ );+ });++ it('on a 1 px cast, two steps judge what settledWithin: 2 judges', () => {+ const tracked = chromium(slowTail.tracked);+ const control = chromium(slowTail.control);+ const inPx = busyWindowProof(tracked, slowTail.block, {+ axis: 'x',+ settledWithin: 2,+ control,+ });+ const inSteps = busyWindowProof(tracked, slowTail.block, {+ axis: 'x',+ settledWithinSteps: 2,+ control,+ });+ expect(inSteps.judged).toEqual(inPx.judged);+ expect(inSteps.verdict).toEqual(inPx.verdict);+ });++ it("takes settledWithin or settledWithinSteps, not both, and settledWithinSteps needs the cast's step", () => {+ expect(() =>+ busyWindowProof(chromium(slowTail.tracked), slowTail.block, {+ axis: 'x',+ settledWithin: 2,+ settledWithinSteps: 2,+ }),+ ).toThrow(/settledWithin or settledWithinSteps, not both/);+ expect(() =>+ busyWindowProof(slowTailFrames(slowTail.tracked), slowTail.block, {+ axis: 'x',+ settledWithinSteps: 2,+ }),+ ).toThrow(/no tracked frame carries the cast's step/);+ });+});diff --git a/internal/interaction-harness/src/busy.ts b/internal/interaction-harness/src/busy.tsindex b2daaa5a0a..4e1585a6c3 100644--- a/internal/interaction-harness/src/busy.ts+++ b/internal/interaction-harness/src/busy.ts@@ -7,8 +7,9 @@* that reports exactly when it ran —, `startBusyControl` — a plain* compositor-only control animation recorded in the same window —,* `busyWindowProof`, which reads frame evidence for that window, against- * the control when it has one, and `pooledBusyProof`, which judges several- * short blocks as one.+ * the control when it has one, `castStep`, the screencast's position step+ * read from its frames, and `pooledBusyProof`, which judges several short+ * blocks as one.* @position The busy-main-thread proof: an animation that runs on the* compositor keeps advancing at display rate while JavaScript holds the main* thread; one driven from the main thread (rAF, style writes, layout@@ -442,6 +443,48 @@ function slowStretches(return stretches;}+/**+ * The cast's position step along `axis`, CSS px, read from the cast: the+ * coarsest `step` the tracked frames carry (`trackMarker` sets it from each+ * frame's color resolution and scale, for the default marker only), so 2 px+ * in WebKit's 4:2:0 JPEG cast and 1 px in Chromium's PNG cast. Throws when no+ * frame carries one, as for a control tracked by another color: a step read+ * from the motion would take a run whose changes are all even on a 1 px cast+ * for a 2 px cast.+ */+export function castStep(+ tracked: readonly TrackedFrame[],+ axis: 'x' | 'y' | 'width' | 'height',+): number {+ const key = axis === 'x' || axis === 'width' ? 'x' : 'y';+ const steps = tracked.flatMap(f => (f.step == null ? [] : [f.step[key]]));+ if (steps.length === 0) {+ throw new Error(+ "castStep: no tracked frame carries the cast's step (trackMarker sets it)",+ );+ }+ return Math.max(...steps);+}++/**+ * The step a tracked value's changes share: the largest of 2, 1 and 0.5 CSS+ * px that every change between consecutive positions is a multiple of.+ * Changes rather than positions: a position the page commits once a block+ * ends can sit off that grid. With no change to read, the finest step: a+ * coarser one would let a larger jump pass for a single step.+ */+function motionStepPx(values: readonly number[]): number {+ const changes = values+ .slice(1)+ .map((p, i) => Math.abs(p - values[i]))+ .filter(d => d > 0.25);+ return changes.length === 0+ ? 0.5+ : ([2, 1, 0.5].find(step =>+ changes.every(d => Math.abs(d / step - Math.round(d / step)) < 0.01),+ ) ?? 0.5);+}+/*** Frame evidence for a busy window: how many frames the screencast presented* inside it, how many of them moved the tracked axis, and the longest stall.@@ -517,10 +560,17 @@ function slowStretches(* change that ends it is a single cast step (`stepPx`): the component moves* under one step per frame there, as on an easing tail, which the cast cannot* show on every frame. Where the component moved under two steps per frame- * over the frames just before the run, a frame delivered early can show no- * step too, so the change may carry the motion of every frame the run held- * at that pace, to the nearest step. The run also has to read as a slowing- * curve, at most one frame longer than the component's run before it. A+ * over the frames just before the run, or across the run itself (its+ * catch-up over the frames it spans) where the curve slows faster than those+ * frames show, a frame delivered early can show no step too, so the change+ * may carry the motion of every frame the run held at that pace, to the+ * nearest step. The run also has to read as a slowing+ * curve, at most one frame longer than the component's run before it.+ * Under `settledWithinSteps`, a run under one cast step a frame may instead+ * last ceil(step ÷ travel) frames, whatever the run before it. The travel is+ * the slower moving pace around the run, never less than its own catch-up+ * per frame. The run also lasts at most one frame more than the pace into it+ * allows: ceil(step ÷ max(pace before, catch-up per frame)) + 1. A* freeze ends in a larger catch-up, a stall in a frame-counted animation is* longer than its runs before it, and a run that no change ends stays a miss.* The component still has to show a new position on at least@@ -536,6 +586,7 @@ export function busyWindowProof(axis = 'y',marginMs = 20,settledWithin,+ settledWithinSteps,settledAt,maxMedianFrameMs = Infinity,maxStallFrames = 3,@@ -569,6 +620,17 @@ export function busyWindowProof(* as a stall: 2 is the usual value.*/settledWithin?: number;+ /**+ * `settledWithin` counted in the cast's own position steps: stop judging+ * once the value is within this many steps of where it ends, the step+ * read from the cast (`castStep`): 1 px in Chromium's PNG cast, 2 px in+ * WebKit's JPEG cast, whose 4:2:0 chroma holds the marker's color in+ * 2 by 2 pixel blocks. A coarser cast shows an easing tail's last steps as+ * longer plateaus, so a bound in px would read them as a stall in one+ * engine and not the other; a freeze further from the end than that is+ * still judged. Give this or `settledWithin`, not both.+ */+ settledWithinSteps?: number;/*** Where the value ends, for `settledWithin`: read it from the page once the* animation has ended (`waitForRest`). Defaults to the last frame's value,@@ -591,7 +653,19 @@ export function busyWindowProof(} = {},): BusyProof {let end = window.to - marginMs;- if (settledWithin != null) {+ if (settledWithin != null && settledWithinSteps != null) {+ throw new Error(+ 'busyWindowProof: give settledWithin or settledWithinSteps, not both',+ );+ }+ // The cast's own position step, for a proof that bounds its settle in it.+ const castStepPx =+ settledWithinSteps == null ? null : castStep(tracked, axis);+ const settledWithinPx =+ settledWithinSteps == null || castStepPx == null+ ? settledWithin+ : settledWithinSteps * castStepPx;+ if (settledWithinPx != null) {const valued = tracked.flatMap(f =>f.box == null ? [] : [{t: f.t, value: f.box[axis]}],);@@ -601,7 +675,7 @@ export function busyWindowProof(f =>f.t >= window.from &&final != null &&- Math.abs(f.value - final) <= settledWithin,+ Math.abs(f.value - final) <= settledWithinPx,);if (arrived != null) {end = Math.min(end, arrived.t);@@ -783,29 +857,15 @@ export function busyWindowProof(? expectedFrames: deliveredFrames;const movedFraction = (moved + recorderSkipped) / Math.max(1, judgedFrames);- // The cast's position step for the tracked value: the largest of 2, 1 and- // 0.5 CSS px that every change between its consecutive positions in the- // judged part is a multiple of. Changes rather than positions: a position- // the page commits once the block ends can sit off that grid.- const partValues = tracked.flatMap(f =>- f.box != null && f.t >= judged.from && f.t <= judged.to- ? [f.box[axis]]- : [],+ // The cast's position step for the tracked value, read from the judged+ // part.+ const stepPx = motionStepPx(+ tracked.flatMap(f =>+ f.box != null && f.t >= judged.from && f.t <= judged.to+ ? [f.box[axis]]+ : [],+ ),);- const stepChanges = partValues- .slice(1)- .map((p, i) => Math.abs(p - partValues[i]))- .filter(d => d > 0.25);- // With no change to read, the finest step: a coarser one would let a larger- // jump pass for a single step.- const stepPx =- stepChanges.length === 0- ? 0.5- : ([2, 1, 0.5].find(step =>- stepChanges.every(- d => Math.abs(d / step - Math.round(d / step)) < 0.01,- ),- ) ?? 0.5);const judgedValues = tracked.flatMap(f =>f.box != null && f.t >= judged.from && f.t <= judged.to? [{t: f.t, value: f.box[axis]}]@@ -888,6 +948,18 @@ export function busyWindowProof(? Math.abs(series[start].value - series[start - back].value) / frames: null;};+ // The component's motion per frame over up to four frames after frame+ // `end`, or null with fewer than two to read.+ const pxPerFrameAfter = (end: number): number | null => {+ const ahead = Math.min(4, series.length - 1 - end);+ if (ahead < 2) {+ return null;+ }+ const frames = (series[end + ahead].t - series[end].t) / cadence;+ return frames > 0+ ? Math.abs(series[end + ahead].value - series[end].value) / frames+ : null;+ };const underStep = (t: number): boolean => {const i = seriesIndex.get(t);if (i == null || changed.has(t)) {@@ -910,7 +982,19 @@ export function busyWindowProof(// frame long, and under one step a frame it adds at most one step to// the catch-up.const catchUp = Math.abs(series[k].value - series[i].value);- const pace = pxPerFrameBefore(start);+ // The pace at the run: the component's motion over the frames before+ // it. A proof that bounds its settle in the cast's own steps also reads+ // the pace across the run itself, its catch-up over the frames from the+ // run's start to the change, where the curve slows faster than the+ // frames before it show. A freeze still fails: its run is longer than+ // one frame at a step a frame or more, and at a steady pace of two steps+ // a frame its catch-up carries two frames of that pace.+ const paceBefore = pxPerFrameBefore(start);+ const runFrames = (series[k].t - series[start].t) / cadence;+ const pace =+ paceBefore == null || settledWithinSteps == null || runFrames <= 0+ ? paceBefore+ : Math.min(paceBefore, catchUp / runFrames);const shown =catchUp < 1.5 * stepPx ||(pace != null &&@@ -920,6 +1004,41 @@ export function busyWindowProof(if (!shown) {return false;}+ // Under one step a frame, a proof that bounds its settle in the cast's+ // own steps reads the run against the cast: the position changes about+ // every step ÷ pace frames, so a run shows one position for at most+ // ceil(step ÷ pace) frames, and a frame delivered early adds one. Only a+ // longer run is a stall, whatever the run before it was.+ // Here the pace is the travel at the run. It is the slower of the moving+ // paces around it, and never less than the run's catch-up per frame. A pace of 0 after the change is the settled plateau, not+ // motion. So a long hold near rest still fails: the motion before it+ // bounds how long one position may show.+ const moving = [paceBefore, pxPerFrameAfter(k)].filter(+ (value): value is number => value != null && value > 0,+ );+ const travel =+ moving.length === 0+ ? null+ : Math.max(+ Math.min(...moving),+ runFrames > 0 ? catchUp / runFrames : 0,+ );+ if (castStepPx != null && travel != null && travel < castStepPx) {+ // The ratio is read to a millionth: a pace of 0.5 that sums to+ // 0.4999… still allows two frames, not three.+ const allowed = Math.ceil(castStepPx / travel - 1e-6);+ // A slow tail after the run sets no allowance of its own. So the run+ // may last at most one frame more than the pace into it allows.+ const into = Math.max(+ paceBefore ?? 0,+ runFrames > 0 ? catchUp / runFrames : 0,+ );+ const cap =+ into > 0+ ? Math.ceil(castStepPx / into - 1e-6) + 1+ : Number.POSITIVE_INFINITY;+ return unchanged <= Math.min(allowed, cap);+ }// A slowing curve lengthens its runs a frame at a time: a run more than// one frame longer than the one before it is a stall.const before = start > 0 ? start - 1 - runStart(start - 1) : 0;diff --git a/internal/interaction-harness/src/busy.webkitdecel.fixture.ts b/internal/interaction-harness/src/busy.webkitdecel.fixture.tsnew file mode 100644index 0000000000..59e1ad3e9a--- /dev/null+++ b/internal/interaction-harness/src/busy.webkitdecel.fixture.ts@@ -0,0 +1,124 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file busy.webkitdecel.fixture.ts+ * @input One recorded busy-main-thread run of a short coast through a 400 ms+ * block in Playwright's WebKit, in the CI WebKit lane's container (4 CPUs,+ * one worker), with a compositor control box+ * @output `step`, the cast's position step; `tracked` and `control`: the screencast frames each was tracked in,+ * as [ms after the block started, x in CSS px, or null where it was not+ * found]; `block`: the block's span on the same clock. The coast slows from+ * about 4.5 px a frame to 1 within five frames on WebKit's 2 px cast: its+ * frame at +52 ms holds the position before while the control moves 7 px+ * (10 to 11 on the frames around it), and the next frame catches up 4 px.+ * @position Fixture for busy.test.ts+ */++/**+ * The cast's position step, CSS px: its frames are JPEGs at the viewport's+ * size whose 4:2:0 chroma holds color in 2 by 2 pixel blocks (their frame+ * headers: luma sampled 2x2, both chroma components 1x1).+ */+export const step = {x: 2, y: 2};++export const block = {from: 0, to: 400};++export const tracked: ReadonlyArray<readonly [number, number | null]> = [+ [-87.3, 146],+ [-76.7, 156],+ [-59.8, 156],+ [-49.4, 162],+ [-32.0, 162],+ [-21.5, 172],+ [-4.0, 176],+ [7.0, 180],+ [24.6, 184],+ [35.1, 188],+ [52.4, 188],+ [63.0, 192],+ [80.4, 192],+ [91.1, 194],+ [108.6, 196],+ [119.3, 198],+ [136.7, 198],+ [148.1, 198],+ [165.7, 200],+ [176.3, 200],+ [193.9, 200],+ [204.4, 202],+ [221.2, 202],+ [232.0, 202],+ [248.7, 202],+ [259.4, 202],+ [276.4, 202],+ [287.2, 202],+ [304.0, 202],+ [314.9, 202],+ [331.7, 202],+ [342.5, 202],+ [359.3, 202],+ [369.8, 202],+ [386.5, 202],+ [397.1, 202],+ [414.6, 202],+ [425.3, 202],+ [442.0, 202],+ [452.8, 202],+ [470.0, 202],+ [480.7, 202],+ [498.4, 202],+ [509.3, 202],+ [526.9, 202],+ [537.7, 202],+ [554.4, 202],+];++export const control: ReadonlyArray<readonly [number, number | null]> = [+ [-87.3, 117],+ [-76.7, 128],+ [-59.8, 134],+ [-49.4, 144],+ [-32.0, 150],+ [-21.5, 161],+ [-4.0, 167],+ [7.0, 178],+ [24.6, 184],+ [35.1, 194],+ [52.4, 201],+ [63.0, 212],+ [80.4, 218],+ [91.1, 228],+ [108.6, 235],+ [119.3, 237],+ [136.7, 230],+ [148.1, 220],+ [165.7, 213],+ [176.3, 202],+ [193.9, 196],+ [204.4, 186],+ [221.2, 178],+ [232.0, 169],+ [248.7, 162],+ [259.4, 153],+ [276.4, 146],+ [287.2, 136],+ [304.0, 130],+ [314.9, 120],+ [331.7, 113],+ [342.5, 103],+ [359.3, 96],+ [369.8, 86],+ [386.5, 80],+ [397.1, 70],+ [414.6, 64],+ [425.3, 53],+ [442.0, 46],+ [452.8, 37],+ [470.0, 30],+ [480.7, 20],+ [498.4, 14],+ [509.3, 2],+ [526.9, 6],+ [537.7, 16],+ [554.4, 23],+];diff --git a/internal/interaction-harness/src/busy.webkittail.fixture.ts b/internal/interaction-harness/src/busy.webkittail.fixture.tsnew file mode 100644index 0000000000..0f07368480--- /dev/null+++ b/internal/interaction-harness/src/busy.webkittail.fixture.ts@@ -0,0 +1,123 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file busy.webkittail.fixture.ts+ * @input One recorded busy-main-thread run of a coast settling through a+ * 400 ms block in Playwright's WebKit, in the CI WebKit lane's container+ * (4 CPUs, one worker), with a compositor control box+ * @output `step`, the cast's position step; `tracked` and `control`: the screencast frames each was tracked in,+ * as [ms after the block started, x in CSS px, or null where it was not+ * found]; `block`: the block's span on the same clock. WebKit's cast reports+ * positions in 2 px steps, so the coast's tail shows plateaus of a frame or+ * two within a few px of rest while the control keeps moving.+ * @position Fixture for busy.test.ts+ */++/**+ * The cast's position step, CSS px: its frames are JPEGs at the viewport's+ * size whose 4:2:0 chroma holds color in 2 by 2 pixel blocks (their frame+ * headers: luma sampled 2x2, both chroma components 1x1).+ */+export const step = {x: 2, y: 2};++export const block = {from: 0, to: 400};++export const tracked: ReadonlyArray<readonly [number, number | null]> = [+ [-91.9, 140],+ [-79.3, 146],+ [-64.3, 156],+ [-51.4, 156],+ [-35.8, 170],+ [-23.3, 176],+ [-7.8, 182],+ [4.1, 188],+ [20.4, 192],+ [31.1, 196],+ [48.3, 200],+ [59.3, 202],+ [76.2, 204],+ [87.0, 208],+ [103.9, 210],+ [115.4, 212],+ [131.9, 212],+ [142.8, 214],+ [159.5, 214],+ [170.3, 216],+ [187.2, 216],+ [197.6, 216],+ [214.9, 216],+ [225.1, 218],+ [242.5, 218],+ [253.2, 218],+ [270.1, 218],+ [281.6, 218],+ [297.4, 218],+ [309.3, 218],+ [325.8, 218],+ [337.3, 218],+ [353.8, 218],+ [364.7, 218],+ [382.0, 218],+ [392.3, 218],+ [409.4, 218],+ [419.7, 218],+ [436.6, 218],+ [446.9, 218],+ [464.2, 218],+ [474.6, 220],+ [491.5, 220],+ [502.4, 220],+ [519.8, 220],+ [530.3, 220],+ [547.9, 220],+];++export const control: ReadonlyArray<readonly [number, number | null]> = [+ [-91.9, 96],+ [-79.3, 105],+ [-64.3, 112],+ [-51.4, 121],+ [-35.8, 129],+ [-23.3, 138],+ [-7.8, 146],+ [4.1, 155],+ [20.4, 162],+ [31.1, 172],+ [48.3, 178],+ [59.3, 189],+ [76.2, 194],+ [87.0, 206],+ [103.9, 212],+ [115.4, 222],+ [131.9, 229],+ [142.8, 238],+ [159.5, 236],+ [170.3, 226],+ [187.2, 220],+ [197.6, 210],+ [214.9, 204],+ [225.1, 193],+ [242.5, 187],+ [253.2, 177],+ [270.1, 170],+ [281.6, 160],+ [297.4, 153],+ [309.3, 144],+ [325.8, 137],+ [337.3, 126],+ [353.8, 120],+ [364.7, 110],+ [382.0, 103],+ [392.3, 93],+ [409.4, 87],+ [419.7, 76],+ [436.6, 70],+ [446.9, 60],+ [464.2, 54],+ [474.6, 44],+ [491.5, 37],+ [502.4, 27],+ [519.8, 21],+ [530.3, 10],+ [547.9, 4],+];diff --git a/internal/interaction-harness/src/frames.colorblock.test.ts b/internal/interaction-harness/src/frames.colorblock.test.tsnew file mode 100644index 0000000000..cc4f927335--- /dev/null+++ b/internal/interaction-harness/src/frames.colorblock.test.ts@@ -0,0 +1,75 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file frames.colorblock.test.ts+ * @input frames.ts's color-resolution reader+ * @output Unit cases: a cast frame's color block is read from its own frame+ * header (a JPEG's chroma subsampling: 4:2:0 in 2 by 2 pixel blocks, 4:2:2+ * in 2 by 1, 4:4:4 and grayscale in 1 by 1) and is 1 by 1 for a PNG; a+ * JPEG with no frame header is an error, never a default.+ * @position Node unit tests beside the harness's browser self-tests.+ *+ * SYNC: When modified, update frames.ts (colorBlockPx).+ */++import {describe, expect, it} from 'vitest';+import {colorBlockPx} from './frames';++/**+ * A minimal JPEG as base64: the start-of-image marker, an application+ * segment, then a baseline frame header (402 by 714) whose components carry+ * `factors` (horizontal in the high nibble, vertical in the low one).+ */+const jpeg = (factors: readonly number[]): string => {+ const app0 = [+ 0xff, 0xe0, 0x00, 0x10, 0x4a, 0x46, 0x49, 0x46, 0x00, 0x01, 0x01, 0x00,+ 0x00, 0x01, 0x00, 0x01, 0x00, 0x00,+ ];+ const sof = [+ 0xff,+ 0xc0,+ 0x00,+ 8 + 3 * factors.length,+ 8,+ 0x02,+ 0xca,+ 0x01,+ 0x92,+ factors.length,+ ...factors.flatMap((f, k) => [k + 1, f, k === 0 ? 0 : 1]),+ ];+ return Buffer.from([0xff, 0xd8, ...app0, ...sof, 0xff, 0xd9]).toString(+ 'base64',+ );+};++describe('colorBlockPx', () => {+ it("reads a JPEG's chroma subsampling from its frame header", () => {+ expect(colorBlockPx(jpeg([0x22, 0x11, 0x11]), 'jpeg')).toEqual({+ x: 2,+ y: 2,+ });+ expect(colorBlockPx(jpeg([0x21, 0x11, 0x11]), 'jpeg')).toEqual({+ x: 2,+ y: 1,+ });+ expect(colorBlockPx(jpeg([0x11, 0x11, 0x11]), 'jpeg')).toEqual({+ x: 1,+ y: 1,+ });+ expect(colorBlockPx(jpeg([0x11]), 'jpeg')).toEqual({x: 1, y: 1});+ });++ it('takes a PNG at one block a pixel', () => {+ expect(colorBlockPx('', 'png')).toEqual({x: 1, y: 1});+ });++ it('fails on a JPEG with no frame header', () => {+ expect(() =>+ colorBlockPx(+ Buffer.from([0xff, 0xd8, 0xff, 0xd9]).toString('base64'),+ 'jpeg',+ ),+ ).toThrow(/no frame header/);+ });+});diff --git a/internal/interaction-harness/src/frames.ts b/internal/interaction-harness/src/frames.tsindex b5f6e72387..95ef1af5d6 100644--- a/internal/interaction-harness/src/frames.ts+++ b/internal/interaction-harness/src/frames.ts@@ -7,7 +7,8 @@* `Screencast` domain)* @output `startScreencast` (timestamped composited frames), `markElement`* (a marker outline the frames can find), `trackMarker` (per-frame marker- * box), `sampleRects` (per-rAF rect sampling), `frameStats`.+ * box and the cast's position step), `colorBlockPx` (a frame's color+ * resolution), `sampleRects` (per-rAF rect sampling), `frameStats`.* @position Frame evidence for motion proofs. The screencast is captured off* the page's main thread, from what the compositor presents, so it keeps* delivering frames while a script blocks the main thread: that is what makes@@ -292,6 +293,61 @@ export interface TrackedFrame {readonly box: Box | null;/** Marker pixels found (a fading element finds fewer at the same tolerance). */readonly pixels: number;+ /**+ * The cast's position step at this frame, CSS px per axis: one block of+ * the frame's color resolution (`colorBlockPx`) over the bitmap's scale to+ * the viewport. `trackMarker` sets it only for the default marker+ * (`MARKER_RGB`), whose edges were measured on these blocks: 2 px in+ * WebKit's JPEG cast, whose 4:2:0 chroma holds color in 2 by 2 pixel+ * blocks, and 1 px in Chromium's PNG cast, both at the viewport's size.+ * Another color carries none: a cyan control in the same WebKit frames+ * moves by odd px.+ */+ readonly step?: {readonly x: number; readonly y: number};+}++/**+ * The bitmap pixels per block of a cast frame's color resolution, per axis:+ * 1 for a PNG; for a JPEG, its chroma subsampling, read from the frame header+ * (4:2:0 holds color in 2 by 2 blocks, 4:2:2 in 2 by 1, 4:4:4 in 1 by 1).+ */+export function colorBlockPx(+ data: string,+ format: 'png' | 'jpeg',+): {x: number; y: number} {+ if (format === 'png') {+ return {x: 1, y: 1};+ }+ const bytes = Buffer.from(data, 'base64');+ // Segments follow the start-of-image marker, each FF, a marker byte and a+ // big-endian length that counts itself, up to a frame header (SOF0-SOF15,+ // which leave out C4, C8 and CC).+ let i = 2;+ while (i + 4 <= bytes.length && bytes[i] === 0xff) {+ const marker = bytes[i + 1];+ if (+ marker >= 0xc0 &&+ marker <= 0xcf &&+ marker !== 0xc4 &&+ marker !== 0xc8 &&+ marker !== 0xcc+ ) {+ const count = bytes[i + 9];+ const factors = Array.from(+ {length: count},+ (_, k) => bytes[i + 11 + 3 * k],+ );+ const horizontal = factors.map(f => f >> 4);+ const vertical = factors.map(f => f & 15);+ // The most subsampled component sets the color's resolution.+ return {+ x: Math.max(...horizontal) / Math.min(...horizontal),+ y: Math.max(...vertical) / Math.min(...vertical),+ };+ }+ i += 2 + bytes.readUInt16BE(i + 2);+ }+ throw new Error('colorBlockPx: the JPEG has no frame header (SOF)');}/**@@ -306,6 +362,7 @@ export async function trackMarker(tolerance = 40,}: {rgb?: readonly [number, number, number]; tolerance?: number} = {},): Promise<TrackedFrame[]> {+ const isDefaultMarker = rgb.every((value, i) => value === MARKER_RGB[i]);const scratch = await page.context().newPage();try {const out: TrackedFrame[] = [];@@ -317,6 +374,7 @@ export async function trackMarker(const results: {box: {x: number; y: number; width: number; height: number} | null;pixels: number;+ scale: number;}[] = [];for (const item of items) {const blob = await (@@ -376,6 +434,7 @@ export async function trackMarker(height: (maxY - minY + 1) / scale,},pixels,+ scale,});}return results;@@ -390,9 +449,19 @@ export async function trackMarker(tol: tolerance,},);- boxes.forEach((result, k) =>- out.push({t: batch[k].t, box: result.box, pixels: result.pixels}),- );+ boxes.forEach((result, k) => {+ if (!isDefaultMarker) {+ out.push({t: batch[k].t, box: result.box, pixels: result.pixels});+ return;+ }+ const block = colorBlockPx(batch[k].data, batch[k].format);+ out.push({+ t: batch[k].t,+ box: result.box,+ pixels: result.pixels,+ step: {x: block.x / result.scale, y: block.y / result.scale},+ });+ });}return out;} finally {