Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

test(interaction-harness): a WebKit touch play that does not watch each step waits for each answer, so no gap reaches the page more than 2 ms short (#9956)

Christopher Chedeauffa495bee5ed
diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.md
index 6dcaef089c..f6961ab19f 100644
--- a/internal/interaction-harness/README.md
+++ b/internal/interaction-harness/README.md
@@ -416,7 +416,7 @@ time: it fast-forwards whenever the page is idle.
| | Chromium 149 | WebKit 26.5 (Linux build) |
| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Trusted touch + pointer events, multi-touch | yes | yes (a `touchstart` lists only the fingers that landed in it; pointer events exact) |
-| Page sees the scripted timestamps | yes, exactly | no: real dispatch time; pacing keeps velocities within ~1 ms on an idle runner, but a CPU-starved one sends some steps late, then back to back; `pacing: 'spacing'` keeps the script's spacing after a stall instead, and fails a play whose page read a resting finger where the script moves, a throw where the script rests, or fewer than two moves, or, given `releaseTolerance`, a release that strays further than that from the script's (each read as Core's velocity tracker reads it): `TouchPageStallError` when the page's own touch or pointer listeners held an event back, `TouchPacingError` otherwise; the lift's `release` names what the page read |
+| Page sees the scripted timestamps | yes, exactly | no: real dispatch time; pacing keeps velocities within ~1 ms on an idle runner; after a stall each step waits its gap, less 2 ms, after WebKit answered the one before (unless a `'schedule'` play has `onEvent`); `pacing: 'spacing'` fails a play whose page read a resting finger where the script moves, a throw where the script rests, or fewer than two moves, or, given `releaseTolerance`, a release straying further from the script's (each read as Core's velocity tracker reads it): `TouchPageStallError` when the page's own touch or pointer listeners held an event back, `TouchPacingError` otherwise; the lift's `release` names what the page read |
| Sub-pixel touch positions | yes | no: integer CSS px |
| Native scroll, fling, pinch-zoom from injected touches | yes | no |
| Lifting one finger while others stay down | yes | pointer events right, but that `touchend` lists no remaining `touches`: skip with `PARTIAL_LIFT_SKIP` |
diff --git a/internal/interaction-harness/src/harness.interaction.browser.spec.ts b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
index a9ad422a61..117d2a8fa1 100644
--- a/internal/interaction-harness/src/harness.interaction.browser.spec.ts
+++ b/internal/interaction-harness/src/harness.interaction.browser.spec.ts
@@ -56,6 +56,7 @@ import {
import type {CastFrame, TrackedFrame} from './frames';
import {
advanceFrozenPage,
+ ANSWER_SLACK_MS,
armHold,
animationStoryboard,
castStoryboard,
@@ -305,6 +306,9 @@ test.describe('touch input', () => {
// the dispatch (it answers only once the page has the event), so a driver
// that holds the moves and sends them in a burst fails.
//
+ // This take watches every step (`onEvent`), so each step leaves at its
+ // scripted time and none waits for WebKit's answer to the one before.
+ //
// The pacing is bounded from above too. The harness waits only while it
// is ahead of the script and sends a late step at once, so a move it
// waited for (it left more than 2 ms after the step before was answered)
@@ -448,6 +452,52 @@ test.describe('touch input', () => {
);
});
+ test("a touch play reaches a page that takes touches late with no gap more than 2 ms short of the script's", async ({
+ page,
+ browserName,
+ }) => {
+ test.skip(
+ browserName !== 'webkit',
+ 'only WebKit stamps a touch when it arrives; Chromium carries the scripted times',
+ );
+ // A timer holds the main thread for 6 ms every 9 ms, so WebKit stamps
+ // some touches late and answers them soon after. When a touch did not
+ // wait for the answer to the one before, it could reach the page 1 to
+ // 3 ms after it, where the script spaces them 8.3 ms.
+ await installInputLog(page);
+ await page.route('http://late-taking-page.test/', route =>
+ route.fulfill({
+ contentType: 'text/html',
+ body: '<!doctype html><body style="margin:0;height:3000px;touch-action:none"><script>setInterval(()=>{const end=performance.now()+6;while(performance.now()<end){}},9);</script>',
+ }),
+ );
+ for (const pacing of ['schedule', 'spacing'] as const) {
+ await page.goto('http://late-taking-page.test/');
+ const driver = await touchDriver(page);
+ const played = await driver.play(
+ drag({x: 200, y: 400}, {x: 200, y: 300}, {durationMs: 100, hz: 120}),
+ {pacing},
+ );
+ const touches = (await readInputLog(page)).filter(record =>
+ record.type.startsWith('touch'),
+ );
+ expect(touches).toHaveLength(played.length);
+ const short = played
+ .slice(1)
+ .map(
+ (step, i) =>
+ step.t -
+ played[i].t -
+ (touches[i + 1].timeStamp - touches[i].timeStamp),
+ );
+ // WebKit stamps whole milliseconds, hence the 1 ms.
+ expect(
+ Math.max(...short),
+ `${pacing}: each gap's shortfall, ms: ${short.map(s => s.toFixed(1)).join(', ')}`,
+ ).toBeLessThanOrEqual(ANSWER_SLACK_MS + 1);
+ }
+ });
+
test('a flick flings a native scroller, farther for a faster finger', async ({
page,
browserName,
diff --git a/internal/interaction-harness/src/touch.pacing.test.ts b/internal/interaction-harness/src/touch.pacing.test.ts
index f55afdab30..d5da6280dd 100644
--- a/internal/interaction-harness/src/touch.pacing.test.ts
+++ b/internal/interaction-harness/src/touch.pacing.test.ts
@@ -6,13 +6,14 @@
* its pacing errors
* @output Unit cases: under `'schedule'` an event leaves at its scripted time
* from the start; under `'spacing'` its scripted interval after the event
- * before it left, or after that dispatch returned when it outlasted the
- * interval. The page's release (the last two move intervals, blended as
- * core blends them) decides a paced play: an early stall leaves it
- * intact; a short stall or a bunched pair at the end changes it, a fault
- * only under a tolerance. A lift more than the stillness interval after
- * the last move reads as rest, a fault unless the script rests too. A
- * throw where the script rests is a fault, and so are fewer than two
+ * before it left. Under `'spacing'`, and on a WebKit `'schedule'` play, it
+ * also waits its interval, less 2 ms, after the event before was answered,
+ * so no gap reaches the page more than 2 ms short. The page's release (the
+ * last two move intervals, blended as core blends them) decides a paced play:
+ * an early stall leaves it intact; a short stall or a bunched pair at the end
+ * changes it, a fault only under a tolerance. A lift more than the stillness
+ * interval after the last move reads as rest, a fault unless the script rests
+ * too. A throw where the script rests is a fault, and so are fewer than two
* moves. A gap the page's own listeners held is the page's; a late send is
* the runner's.
* @position Node unit tests beside the harness's browser self-tests.
@@ -25,6 +26,7 @@
import {describe, expect, it} from 'vitest';
import type {TouchEventStep} from './scripts';
import {
+ ANSWER_SLACK_MS,
pacedDeadline,
readRelease,
touchArrivalsFrom,
@@ -52,17 +54,22 @@ const FLICK: TouchEventStep[] = [0, 16, 32, 48, 64, 80, 96].map((t, i) => {
/**
* The harness's loop on a simulated clock: each event leaves once its
* deadline has come (or at once when it is late), and its dispatch returns
- * `latencyMs[i]` later, when the page has it and has handled it.
+ * `latencyMs[i]` later, when the page has it and has handled it. The page
+ * stamps each event at that answer, the latest WebKit can.
*/
function simulate(
pacing: TouchPacing,
latencyMs: readonly number[],
+ keepsGaps = false,
): TouchArrival[] {
const origin = 1000;
let now = origin;
let previous: {sentAt: number; returnedAt: number} | null = null;
return FLICK.map((_, index) => {
- now = Math.max(now, pacedDeadline(pacing, origin, FLICK, index, previous));
+ now = Math.max(
+ now,
+ pacedDeadline(pacing, origin, FLICK, index, previous, keepsGaps),
+ );
const sentAt = now;
now += latencyMs[index];
previous = {sentAt, returnedAt: now};
@@ -89,33 +96,106 @@ describe('pacedDeadline', () => {
).toBe(1048);
});
+ it("keeps a WebKit 'schedule' play's gap, less 2 ms, after a late answer", () => {
+ // Answered in time: at its scripted time.
+ expect(
+ pacedDeadline(
+ 'schedule',
+ 1000,
+ FLICK,
+ 3,
+ {sentAt: 1032, returnedAt: 1033},
+ true,
+ ),
+ ).toBe(1048);
+ // Answered at 1300: 14 ms after that answer.
+ expect(
+ pacedDeadline(
+ 'schedule',
+ 1000,
+ FLICK,
+ 3,
+ {sentAt: 1190, returnedAt: 1300},
+ true,
+ ),
+ ).toBe(1314);
+ });
+
it("sends the first event at the start and the rest after the one before, under 'spacing'", () => {
expect(pacedDeadline('spacing', 1000, FLICK, 0, null)).toBe(1000);
- // Answered within its 16 ms interval: 16 ms after it left.
+ // Answered within 2 ms: 16 ms after it left.
expect(
pacedDeadline('spacing', 1000, FLICK, 3, {
sentAt: 1200,
- returnedAt: 1205,
+ returnedAt: 1201,
}),
).toBe(1216);
- // Outlasted it: 16 ms after its answer.
+ // Answered after 5 ms: 14 ms after the answer, as the page may have
+ // stamped the event that late.
+ expect(
+ pacedDeadline('spacing', 1000, FLICK, 3, {
+ sentAt: 1200,
+ returnedAt: 1205,
+ }),
+ ).toBe(1219);
+ // Outlasted its interval: 14 ms after its answer.
expect(
pacedDeadline('spacing', 1000, FLICK, 3, {
sentAt: 1200,
returnedAt: 1300,
}),
- ).toBe(1316);
+ ).toBe(1314);
+ });
+
+ it('never lets a page that stamps on arrival read a gap more than 2 ms short of the script', () => {
+ // The closest two stamps can come is the earlier at its answer and the
+ // later at its send. Answers here take 1 to 30 ms.
+ const latency = [1, 5, 7, 30, 3, 8, 2];
+ const shortest = (pacing: TouchPacing, keepsGaps: boolean) => {
+ const sent: number[] = [];
+ const answered: number[] = [];
+ let now = 1000;
+ for (const index of FLICK.keys()) {
+ const previous =
+ index > 0
+ ? {sentAt: sent[index - 1], returnedAt: answered[index - 1]}
+ : null;
+ now = Math.max(
+ now,
+ pacedDeadline(pacing, 1000, FLICK, index, previous, keepsGaps),
+ );
+ sent.push(now);
+ now += latency[index];
+ answered.push(now);
+ }
+ return Math.min(
+ ...sent
+ .slice(1)
+ .map((at, i) => at - answered[i] - (FLICK[i + 1].t - FLICK[i].t)),
+ );
+ };
+ expect(shortest('spacing', false)).toBeGreaterThanOrEqual(-ANSWER_SLACK_MS);
+ expect(shortest('schedule', true)).toBeGreaterThanOrEqual(-ANSWER_SLACK_MS);
+ // A play that keeps its scripted times sends behind the 30 ms answer.
+ expect(shortest('schedule', false)).toBeLessThan(-ANSWER_SLACK_MS);
});
});
describe("touchPacingFault on a paced play's arrivals", () => {
- it("reads the script's release when every dispatch takes 5 ms", () => {
- expect(
- touchPacingFault(FLICK, simulate('spacing', [5, 5, 5, 5, 5, 5, 5])),
- ).toBeNull();
+ it("reads the script's release when every dispatch is answered within 2 ms", () => {
+ const arrivals = simulate('spacing', [2, 2, 2, 2, 2, 2, 2]);
+ expect(touchPacingFault(FLICK, arrivals, TOLERANCE)).toBeNull();
+ expect(readRelease(FLICK, arrivals)?.pagePxPerS).toBeCloseTo(1000, 6);
+ });
+
+ it('reads a slower release, never a faster one, when every dispatch takes 5 ms', () => {
+ // Each event waits 14 ms after the answer before it: 19 ms gaps.
+ const arrivals = simulate('spacing', [5, 5, 5, 5, 5, 5, 5]);
+ expect(touchPacingFault(FLICK, arrivals, TOLERANCE)).toBeNull();
+ expect(readRelease(FLICK, arrivals)?.pagePxPerS).toBeCloseTo(842, 0);
});
- it("reads a far faster release when 'schedule' bursts the events behind a stall", () => {
+ it("reads a far faster release when a 'schedule' play that keeps its scripted times bursts the events behind a stall", () => {
// The third dispatch stalls 100 ms; the four events behind it were due
// during the stall and reach the page 1 ms apart.
const fault = touchPacingFault(
@@ -128,6 +208,14 @@ describe("touchPacingFault on a paced play's arrivals", () => {
expect(fault?.scriptReleasePxPerS).toBeCloseTo(1000, 6);
});
+ it("stretches a WebKit 'schedule' play behind a stall instead of bursting it", () => {
+ // The same stall: each event behind it waits 14 ms after the answer
+ // before it, so the release reads 15 ms intervals.
+ const arrivals = simulate('schedule', [1, 1, 100, 1, 1, 1, 1], true);
+ expect(touchPacingFault(FLICK, arrivals, TOLERANCE)).toBeNull();
+ expect(readRelease(FLICK, arrivals)?.pagePxPerS).toBeCloseTo(1067, 0);
+ });
+
it("keeps the release intact behind an early stall under 'spacing'", () => {
// The stall stretches the second move's gap; the last two move intervals
// keep their 16 ms.
@@ -140,17 +228,17 @@ describe("touchPacingFault on a paced play's arrivals", () => {
).toBeNull();
});
- it('reads 554 px/s for a 20 ms stall on the move before the last', () => {
- // Its gap stretches to 35 ms and the next leaves 16 ms after its answer:
- // 0.2 × 16 px/17 ms + 0.8 × 16 px/35 ms.
+ it('reads 579 px/s for a 20 ms stall on the move before the last', () => {
+ // Its gap stretches to 35 ms and the next leaves 14 ms after its answer:
+ // 0.2 × 16 px/15 ms + 0.8 × 16 px/35 ms.
const fault = touchPacingFault(
FLICK,
simulate('spacing', [1, 1, 1, 1, 20, 1, 1]),
TOLERANCE,
);
expect(fault?.kind).toBe('release');
- expect(fault?.pageReleasePxPerS).toBeCloseTo(554, 0);
- expect(fault?.gapsMs).toEqual([35, 17, 16]);
+ expect(fault?.pageReleasePxPerS).toBeCloseTo(579, 0);
+ expect(fault?.gapsMs).toEqual([35, 15, 16]);
expect(fault?.scriptedGapsMs).toEqual([16, 16, 16]);
expect(fault?.heldByPage).toBeNull();
});
@@ -346,7 +434,7 @@ describe('the pacing errors', () => {
const error = new TouchPacingError(fault);
expect(error.name).toBe('TouchPacingError');
expect(error.message).toMatch(
- /the page read a release of 554 px\/s where the script throws 1000 px\/s \(gaps 35, 17, 16 ms where the script spaces 16, 16, 16 ms\): a dispatch stalled before the page/,
+ /the page read a release of 579 px\/s where the script throws 1000 px\/s \(gaps 35, 15, 16 ms where the script spaces 16, 16, 16 ms\): a dispatch stalled before the page/,
);
expect(error.fault).toBe(fault);
});
diff --git a/internal/interaction-harness/src/touch.ts b/internal/interaction-harness/src/touch.ts
index 8f40b90de0..8e9f47799a 100644
--- a/internal/interaction-harness/src/touch.ts
+++ b/internal/interaction-harness/src/touch.ts
@@ -24,9 +24,10 @@
* events derived from them are trusted and cancelable, multi-touch works.
* Limits, measured with WebKit 26.5: no timestamp parameter (the page sees
* the real dispatch time, so velocity comes from real-time pacing: within
- * ~1 ms on an idle runner, but a runner whose CPU quota stalls the process
- * sends some steps late, and the ones behind them back to back, unless
- * the play keeps the script's spacing: `pacing: 'spacing'`);
+ * ~1 ms on an idle runner; after a slow dispatch each step waits for its
+ * gap, less 2 ms, after WebKit answered the step before, so a stall
+ * stretches the gesture instead of bunching it, unless a `'schedule'` play
+ * watches each step, `onEvent`);
* integer coordinates only; no native scrolling, fling or pinch-zoom from
* these touches; a `touchstart` lists only the fingers that landed in it
* (pointer events are exact); no click synthesised from a raw touch sequence (use
@@ -56,14 +57,24 @@ export type Engine = 'chromium' | 'webkit';
/**
* How real-time pacing catches up after a slow dispatch (`PlayOptions.pacing`):
- * `'schedule'` sends each event at its scripted time from the start, so the
- * events behind a stalled command go out back to back; `'spacing'` sends each
- * event its scripted interval after the one before it left, or, when that
- * dispatch outlasted the interval (a stall), its interval after that dispatch
- * returned, so a stall shifts the rest of the gesture instead of bunching it.
+ * `'schedule'` sends each event at its scripted time from the start, and
+ * `'spacing'` its scripted interval after the event before it left. Under
+ * `'spacing'` an event also waits for its interval, less ANSWER_SLACK_MS,
+ * after the engine answered the event before. WebKit stamps a touch no later
+ * than its answer, so a stall stretches the gesture instead of bunching it.
+ * A WebKit `'schedule'` play waits for the answer too, unless it watches each
+ * step (`onEvent`).
*/
export type TouchPacing = 'schedule' | 'spacing';
+/**
+ * The answer time, ms, under which a touch play keeps the script's spacing
+ * exactly; a slower answer stretches the next gap. A page that stamps
+ * touches on arrival reads no gap shorter than the script's by more than
+ * this (see `pacedDeadline`).
+ */
+export const ANSWER_SLACK_MS = 2;
+
/**
* The release filter's stillness interval, 40 ms: a contact that has not
* moved for longer than this before it lifts releases at 0.
@@ -95,9 +106,10 @@ export interface PlayOptions {
* `'schedule'`; see `TouchPacing`). A proof whose result depends on a
* release velocity (a flick, a coast) passes `'spacing'`: on a driver
* without scripted timestamps the page reads the velocity from when events
- * arrive, and a burst after a stalled command reads as a much faster
- * finger. There, a paced play reads the release as the page did once its
- * last event is out (the lift's `PlayedEvent.release`). It throws when the
+ * arrive, and `'spacing'` keeps each gap from the event before even in a
+ * play that watches each step. There, a paced play reads the release as
+ * the page did once its last event is out (the lift's
+ * `PlayedEvent.release`). It throws when the
* page read a resting finger where the script moves, a throw where the
* script rests, or fewer than two moves. Given `releaseTolerance`, it also
* throws when the page's release strays further than that share. The error
@@ -181,13 +193,14 @@ async function until(deadline: number): Promise<void> {
}
/**
- * When event `index` of `steps` goes out (performance.now() ms): at its
- * scripted time from `origin`; or, under `'spacing'`, its scripted interval
- * after the event before it left, so the page reads the script's pace, unless
- * that dispatch outlasted the interval. WebKit answers a dispatch only once
- * the page has the event, so after such a stall the page may have received it
- * as late as the answer, and the next event leaves its interval after that:
- * never back to back.
+ * When event `index` of `steps` goes out, in performance.now() ms. Under
+ * `'schedule'` it goes at its scripted time from `origin`. Under `'spacing'`
+ * it goes its scripted interval after the event before it left. Under
+ * `'spacing'`, or with `keepsGaps`, it also waits for its interval, less
+ * ANSWER_SLACK_MS, after the engine answered the event before. WebKit stamps
+ * an event no later than it answers, and the next one no sooner than it
+ * leaves. So no gap reaches the page more than ANSWER_SLACK_MS short of the
+ * script's: a stall stretches the gesture.
*/
export function pacedDeadline(
pacing: TouchPacing,
@@ -195,14 +208,18 @@ export function pacedDeadline(
steps: readonly {readonly t: number}[],
index: number,
previous: {readonly sentAt: number; readonly returnedAt: number} | null,
+ keepsGaps = false,
): number {
- if (pacing === 'spacing' && index > 0 && previous != null) {
- const interval = steps[index].t - steps[index - 1].t;
- return previous.returnedAt - previous.sentAt <= interval
- ? previous.sentAt + interval
- : previous.returnedAt + interval;
+ const scripted = origin + steps[index].t;
+ if (index === 0 || previous == null) {
+ return scripted;
}
- return origin + steps[index].t;
+ const interval = steps[index].t - steps[index - 1].t;
+ const afterAnswer = previous.returnedAt + interval - ANSWER_SLACK_MS;
+ if (pacing === 'spacing') {
+ return Math.max(previous.sentAt + interval, afterAnswer);
+ }
+ return keepsGaps ? Math.max(scripted, afterAnswer) : scripted;
}
/** What the page saw of one touch event of a paced play. */
@@ -538,6 +555,9 @@ async function playSteps(
// Only where the page reads a velocity from arrival times: scripted
// timestamps carry it.
const judges = probe != null && realtime && pacing === 'spacing';
+ // A driver gives a probe only where the page stamps touches on arrival. A
+ // play that watches each step keeps its scripted times.
+ const keepsGaps = probe != null && onEvent == null;
if (judges) {
await probe.arm();
}
@@ -546,7 +566,9 @@ async function playSteps(
let previous: {sentAt: number; returnedAt: number} | null = null;
for (const [index, step] of steps.entries()) {
if (realtime) {
- await until(pacedDeadline(pacing, origin, steps, index, previous));
+ await until(
+ pacedDeadline(pacing, origin, steps, index, previous, keepsGaps),
+ );
}
const timestamp = stampFor(step.t);
const dispatchedAt = Date.now();