Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

ci: pull-request runs out of time start one at a time, at most one in ten minutes (#9640)

Cindy Zhang550ff216d586
diff --git a/.github/actions/ci-admission/action.yml b/.github/actions/ci-admission/action.yml
index d9284997ae..7a3d041665 100644
--- a/.github/actions/ci-admission/action.yml
+++ b/.github/actions/ci-admission/action.yml
@@ -15,7 +15,9 @@
# "Wait for room under the CI ceiling": a waiting run is found by that name.
# - A pull-request run that finds no room and ten other pull-request runs
# already waiting is turned away (`full`): CI is full, and the run stops with
-# that message. `main` runs are never turned away.
+# that message. So is one still without room at its longest wait, unless it
+# is the only pull-request run to start there in ten minutes and no run that
+# waited longer still waits. `main` runs are never turned away.
# - A run starts once the other active runs' estimated jobs plus `peak` fit
# under the ceiling. Small workflows that are not admitted (one or two slim
# jobs a run) always count at least eighteen jobs, which covers all but their
@@ -26,7 +28,7 @@
# - It waits at most `max-wait-minutes` (fifteen for `main`, twelve for a pull
# request, inside its fifteen-minute job budget), looking at even intervals
# plus a random 0-20 s so runs that look together spread out, and then
-# starts.
+# starts (a pull request: see above).
# - Each CI or Main run counts its live jobs, read once per look. A
# pull-request run still waiting here counts the jobs it runs, its waiting
# job and a Storybook build that found room, so waiting runs never hold each
@@ -151,6 +153,25 @@ runs:
return {spends: true, interval: LOOK_MS};
};
const STEP = 'Wait for room under the CI ceiling';
+ // A pull-request run that reaches its longest wait without room
+ // starts only if no other pull-request run started at its longest
+ // wait in the last ten minutes, about the time one run takes, and
+ // no run that has waited longer still waits; otherwise CI is full
+ // for it. So runs that waited out the same burst start one at a
+ // time, and CI passes ninety by about one run's jobs at most
+ // (spec:AST-030 DEC-16).
+ const AT_LIMIT_SPACING_MS = 10 * 60_000;
+ // A wait this close to the longest wait counts as one that ran out.
+ const AT_LIMIT_SLACK_MS = 30_000;
+ const isPullRun = run => run.event === 'pull_request' && run.path === '.github/workflows/ci.yml';
+ // When a run's wait began and, once admitted, ended: its first wait
+ // step's start and its last wait step's end.
+ const waitSpan = waits => {
+ const times = key => waits.map(step => Date.parse(step[key] ?? '')).filter(Number.isFinite);
+ const began = times('started_at');
+ const ended = times('completed_at');
+ return {began: began.length ? Math.min(...began) : NaN, ended: ended.length ? Math.max(...ended) : NaN};
+ };
// At most this many pull-request runs wait at once; one more finding
// no room is turned away (spec:AST-030 DEC-16).
const MAX_WAITING = 10;
@@ -219,11 +240,13 @@ runs:
// A look planned past the longest wait comes at its end instead.
const left = MAX_WAIT_MS - (Date.now() - started);
const next = Math.min((planned > 0 ? planned : started + Math.max(1, looked) * LOOK_MS) - Date.now(), left - JITTER_MS);
- if (looked >= MAX_LOOKS || left <= 0) {
+ // At its longest wait, a pull request decides on one more look
+ // whether it may start (AT_LIMIT_SPACING_MS); other runs start.
+ if ((looked >= MAX_LOOKS || left <= 0) && pr == null) {
core.warning(`Still no room after ${MAX_WAIT_MS / 60_000} minutes; starting.`);
return;
}
- if (next > 0) await new Promise(resolve => setTimeout(resolve, next + Math.floor(Math.random() * JITTER_MS)));
+ if (next > 0 && looked < MAX_LOOKS) await new Promise(resolve => setTimeout(resolve, next + Math.floor(Math.random() * JITTER_MS)));
}
// Looks that found no room; a look that only yielded to `main` is
// not one of them.
@@ -238,6 +261,12 @@ runs:
let small = 0;
let mainWaiting = false;
let waitingPulls = 0;
+ // Pull-request runs that started at their longest wait lately,
+ // and those still waiting that began waiting before this one.
+ const startedAtLimit = [];
+ // Pull-request runs still waiting, or out of time and not yet
+ // decided, by when their wait began.
+ const pullWaiters = [];
// What each run counted, logged on every look.
const counted = [];
const smallBy = {};
@@ -272,7 +301,9 @@ runs:
const waits = jobs
.flatMap(job => job.steps ?? [])
.filter(step => step.name === STEP);
+ const span = waitSpan(waits);
if (waits.some(step => step.status === 'in_progress')) {
+ if (isPullRun(run)) pullWaiters.push({id: run.id, began: span.began});
// Still waiting for room. A `main` run holds its whole
// peak, ahead of this one. A pull-request run holds what
// it runs: the job that waits and its Storybook build,
@@ -284,6 +315,16 @@ runs:
counted.push(`${short(run)} ${run.id} ${run.event}: ${holds} waiting`);
continue;
}
+ if (isPullRun(run) && span.ended - span.began >= MAX_WAIT_MS - AT_LIMIT_SLACK_MS) {
+ // Out of time: it started only once its admission job
+ // succeeded. Until admission decides, it ranks as a waiter;
+ // a run CI was full for never started.
+ const decided = jobs.find(job => job.name === 'admission' && job.status === 'completed');
+ if (decided == null) pullWaiters.push({id: run.id, began: span.began});
+ else if (decided.conclusion === 'success' && Date.now() - span.ended < AT_LIMIT_SPACING_MS) {
+ startedAtLimit.push({id: run.id, ago: Date.now() - span.ended});
+ }
+ }
const live = jobs.filter(job => LIVE.includes(job.status)).length;
const ended = waits.map(step => Date.parse(step.completed_at ?? '')).filter(Number.isFinite);
const admittedAt = ended.length > 0 ? Math.max(...ended) : NaN;
@@ -327,7 +368,39 @@ runs:
const left = MAX_WAIT_MS - (Date.now() - started);
const wait = Math.min(next.interval + Math.floor(Math.random() * JITTER_MS), left);
if (spent >= MAX_LOOKS || wait <= 0) {
- core.warning(`Still no room after ${MAX_WAIT_MS / 60_000} minutes (about ${others} other jobs${mainWaiting ? ', a main check waiting' : ''}); starting.`);
+ const why = `Still no room after ${MAX_WAIT_MS / 60_000} minutes (about ${others} other jobs${mainWaiting ? ', a main check waiting' : ''})`;
+ if (pr == null) {
+ core.warning(`${why}; starting.`);
+ return;
+ }
+ // Wait starts compared on one clock: this run's own first wait
+ // step as GitHub recorded it (whole seconds), else its clock
+ // floored to the second; a tie goes to the lower run id.
+ let mine = Math.floor(started / 1000) * 1000;
+ try {
+ calls += 1;
+ const {data: {jobs: own}} = await github.rest.actions.listJobsForWorkflowRun({
+ ...context.repo, run_id: context.runId, filter: 'latest', per_page: 100,
+ });
+ const began = waitSpan(own.flatMap(job => job.steps ?? []).filter(step => step.name === STEP)).began;
+ if (Number.isFinite(began)) mine = began;
+ } catch {
+ // The floored clock stands.
+ }
+ const waitedLonger = pullWaiters.filter(
+ other => other.began < mine || (other.began === mine && other.id < context.runId),
+ ).length;
+ if (startedAtLimit.length === 0 && waitedLonger === 0) {
+ core.warning(`${why}; starting, the one pull-request run to start at its longest wait for ${AT_LIMIT_SPACING_MS / 60_000} minutes.`);
+ // The whole run starts now, not only its Storybook build.
+ core.setOutput('fits', 'true');
+ return;
+ }
+ const other = startedAtLimit.length > 0
+ ? `run ${startedAtLimit[0].id} started at its longest wait ${Math.round(startedAtLimit[0].ago / 60_000)} minutes ago`
+ : `${waitedLonger} pull-request run${waitedLonger === 1 ? '' : 's'} that waited longer still wait${waitedLonger === 1 ? 's' : ''}`;
+ core.setOutput('full', 'true');
+ core.warning(`CI is full: ${why.charAt(0).toLowerCase()}${why.slice(1)}, and ${other}.`);
return;
}
core.info(`Waiting for room: about ${others} other jobs + ${peak}${mainWaiting ? ', a main check waiting' : ''}.`);
diff --git a/.github/scripts/ci-admission.test.mjs b/.github/scripts/ci-admission.test.mjs
index 729bf81b24..c47f7babd8 100644
--- a/.github/scripts/ci-admission.test.mjs
+++ b/.github/scripts/ci-admission.test.mjs
@@ -17,7 +17,7 @@ import path from 'node:path';
import {afterEach, describe, expect, it, vi} from 'vitest';
import yaml from 'yaml';
import {matrixFor} from './ci-matrix.mjs';
-import {PR_CI_GATE} from './ci-pr-gate.mjs';
+import {PR_CI_GATE, ungated} from './ci-pr-gate.mjs';
const root = path.resolve(import.meta.dirname, '../..');
const read = file => yaml.parse(fs.readFileSync(path.join(root, file), 'utf8'));
@@ -87,10 +87,13 @@ describe('where runs wait', () => {
it('stops a pull-request run that CI is full for, with the reason, and builds nothing', () => {
const job = ci.jobs['ci-full'];
- expect(needsOf(job)).toEqual(['room']);
- expect(job.if).toBe(
- "github.event_name == 'pull_request' && needs.room.outputs.full == 'true'",
+ // Either wait can find CI full: room, with ten runs waiting, or the rest
+ // of the run at its longest wait.
+ expect(needsOf(job)).toEqual(['room', 'admission']);
+ expect(ungated(job.if)).toBe(
+ "github.event_name == 'pull_request' && !cancelled() && (needs.room.outputs.full == 'true' || needs.admission.outputs.full == 'true')",
);
+ expect(admission.outputs.full).toBe('${{ steps.rest.outputs.full }}');
expect(job.permissions).toEqual({actions: 'write'});
const [stop] = job.steps;
expect(stop['continue-on-error']).toBeUndefined();
@@ -105,35 +108,86 @@ describe('where runs wait', () => {
});
it("fails a fork's run that CI is full for, which cannot cancel itself, in every required context", () => {
- const isStop = step =>
+ const ROOM_FULL = "needs.room.outputs.full == 'true'";
+ const EITHER_FULL = `${ROOM_FULL} || needs.admission.outputs.full == 'true'`;
+ const isStop = (step, condition = ROOM_FULL) =>
step.name === 'Stop when CI is full' &&
- step.if === "needs.room.outputs.full == 'true'" &&
+ step.if === condition &&
step.run.includes('CI is full: re-run all jobs or push later.') &&
step.run.trim().endsWith('exit 1') &&
step['continue-on-error'] === undefined;
// Admission fails first, so every job behind it without its own status
- // rule skips.
+ // rule skips; at its longest wait it fails after the wait.
expect(isStop(admission.steps[0])).toBe(true);
- // The required contexts run, and fail with the reason before any work.
+ const last = admission.steps.at(-1);
+ expect(last.name).toBe('Stop when CI is full at the longest wait');
+ expect(last.if).toBe("steps.rest.outputs.full == 'true'");
+ expect(last.run).toContain('CI is full: re-run all jobs or push later.');
+ expect(last.run.trim().endsWith('exit 1')).toBe(true);
+ // The required contexts run, and fail with the reason before any work,
+ // whichever wait found CI full.
for (const id of ['build', 'test', 'docsite-test']) {
const job = ci.jobs[id];
- expect(needsOf(job), id).toContain('room');
+ expect(needsOf(job), id).toEqual(
+ expect.arrayContaining(['room', 'admission']),
+ );
+ expect(isStop(job.steps[1], EITHER_FULL), id).toBe(true);
expect(job.steps[0].name, id).toBe(
'Require a branch in this repository or the ci:run label',
);
- expect(isStop(job.steps[1]), id).toBe(true);
}
// build runs although the Storybook build it reads skipped.
expect(String(ci.jobs.build.if)).toContain(
"needs.room.outputs.full == 'true'",
);
- // The jobs that would still run behind a failed admission build nothing.
- for (const id of ['typecheck', 'pr-interaction', 'pr-interaction-webkit']) {
- expect(needsOf(ci.jobs[id]), id).toContain('room');
- expect(String(ci.jobs[id].if), id).toContain(
- "needs.room.outputs.full != 'true'",
+ // Every job that would still run behind a failed admission builds
+ // nothing. It is every job with a status function that reaches admission
+ // or the Storybook build, unless it stops with the reason (the required
+ // contexts, ci-full), runs only after admission or the label lookup
+ // succeeded, never runs on a pull request, or does no work without a job
+ // behind admission.
+ const reaches = (id, target, seen = new Set()) =>
+ needsOf(ci.jobs[id]).some(
+ need =>
+ need === target ||
+ (!seen.has(need) && (seen.add(need), reaches(need, target, seen))),
);
+ const FULL_GUARD = "needs.admission.outputs.full != 'true'";
+ const ADMITTED =
+ /needs\.(admission|run-labels)\.result == 'success'|needs\.run-labels\.outputs\./;
+ const DISPATCH_ONLY =
+ /^(always\(\) && )?\(?github\.event_name == 'workflow_dispatch' && inputs\.operation == 'release-check'/;
+ // Its every step after the runner pick reads native-swift-runner's
+ // `applies`, empty when that job skipped behind admission.
+ const NO_WORK_ALONE = ['test-native-swift'];
+ expect(ADMITTED.test(String(ci.jobs['run-labels'].if))).toBe(true);
+ expect(String(ci.jobs['run-labels'].if)).toContain(
+ "needs.admission.result == 'success'",
+ );
+ const guarded = [];
+ for (const [id, job] of Object.entries(ci.jobs)) {
+ const condition = String(job.if ?? '');
+ if (!/\b(always|cancelled)\(\)/.test(condition)) continue;
+ if (!reaches(id, 'admission') && !reaches(id, 'build-storybook'))
+ continue;
+ if (['build', 'test', 'docsite-test', 'ci-full'].includes(id)) continue;
+ if (
+ ADMITTED.test(condition) ||
+ DISPATCH_ONLY.test(ungated(condition).trim())
+ )
+ continue;
+ if (NO_WORK_ALONE.includes(id)) continue;
+ expect(needsOf(job), id).toContain('admission');
+ expect(condition, id).toContain(FULL_GUARD);
+ guarded.push(id);
}
+ expect(guarded.sort()).toEqual([
+ 'pr-interaction',
+ 'pr-interaction-webkit',
+ 'pr-touched-components',
+ 'pr-visual',
+ 'typecheck',
+ ]);
});
it('skips only with every other job, and never fails, so every job and required check decides as before', () => {
@@ -183,6 +237,9 @@ describe('where runs wait', () => {
expect(admission.steps.filter(step => step !== wait)).toEqual([
expect.objectContaining({name: 'Stop when CI is full'}),
expect.objectContaining({uses: 'actions/checkout@v7'}),
+ expect.objectContaining({
+ name: 'Stop when CI is full at the longest wait',
+ }),
]);
expect(ci.jobs.room.steps.filter(step => step.name !== STEP)).toEqual([
expect.objectContaining({uses: 'actions/checkout@v7'}),
@@ -205,7 +262,11 @@ describe('where runs wait', () => {
]) {
const condition = String(ci.jobs[id].if);
expect(condition, id).toMatch(/\b(always|cancelled)\(\)/);
- expect(condition, id).not.toContain('needs.admission');
+ // Only whether CI was full for the run, never admission's result.
+ expect(
+ condition.replaceAll("needs.admission.outputs.full != 'true'", ''),
+ id,
+ ).not.toContain('needs.admission');
}
// lint and review-required are written by workflows admission does not touch.
for (const file of ['lint.yml', 'review-signal.yml', 'review-clear.yml']) {
@@ -461,7 +522,7 @@ describe('ci admission', () => {
expect(per_page).toBe(100);
calls += 1;
return {
- data: {jobs: runs().find(run => run.id === run_id).jobs ?? []},
+ data: {jobs: runs().find(run => run.id === run_id)?.jobs ?? []},
};
},
},
@@ -731,18 +792,33 @@ describe('ci admission', () => {
});
expect(near.lookTimes[0]).toBeGreaterThanOrEqual(4 * 60_000);
expect(near.lookTimes[0]).toBeLessThanOrEqual(4 * 60_000 + 20_000);
- // Room already spent both looks: the rest starts at once.
+ // Room already spent both looks: a pull request decides on one more
+ // look, at once, whether it may start at its longest wait; it may.
const spent = await admit({
active: prs(7),
peak: 16,
phase: 'rest',
fits: 'false',
- started: String(NOW),
+ started: String(NOW - 12 * 60_000),
looked: '2',
maxWait: 12,
maxLooks: 2,
});
- expect(spent).toMatchObject({looks: 0, calls: 0});
+ expect(spent.looks).toBe(1);
+ expect(spent.lookTimes[0]).toBe(0);
+ expect(spent.warning[0]).toContain('starting, the one pull-request run');
+ expect(spent.outputs.full).toBeUndefined();
+ // A main run there starts at once, without a look.
+ const mainSpent = await admit({
+ active: prs(7),
+ peak: 16,
+ isMain: true,
+ phase: 'rest',
+ fits: 'false',
+ started: String(NOW - 15 * 60_000),
+ looked: '8',
+ });
+ expect(mainSpent).toMatchObject({looks: 0, calls: 0});
});
it('leaves fits unset when a later look stops early, so the rest starts', async () => {
@@ -923,6 +999,238 @@ describe('ci admission', () => {
});
});
+ describe('one start at the longest wait at a time', () => {
+ const at = minutes => new Date(NOW + minutes * 60_000).toISOString();
+ /** A pull-request run admitted at `endedAt` minutes after its wait began `waited` minutes before. */
+ const startedRun = (
+ id,
+ endedAt,
+ waited,
+ {jobs = 12, admission = 'success'} = {},
+ ) => ({
+ id,
+ status: 'in_progress',
+ event: 'pull_request',
+ path: '.github/workflows/ci.yml',
+ jobs: [
+ {
+ name: 'room',
+ status: 'completed',
+ steps: [
+ {
+ name: STEP,
+ status: 'completed',
+ started_at: at(endedAt - waited),
+ completed_at: at(endedAt),
+ },
+ ],
+ },
+ admission == null
+ ? {name: 'admission', status: 'queued', steps: []}
+ : {
+ name: 'admission',
+ status: 'completed',
+ conclusion: admission,
+ steps: [],
+ },
+ ...busy(jobs),
+ ],
+ });
+ /** A pull-request run waiting since `beganAt` minutes. */
+ const waiter = (id, beganAt) => ({
+ id,
+ status: 'in_progress',
+ event: 'pull_request',
+ path: '.github/workflows/ci.yml',
+ jobs: [
+ {
+ name: 'room',
+ status: 'in_progress',
+ steps: [{name: STEP, status: 'in_progress', started_at: at(beganAt)}],
+ },
+ ],
+ });
+ // Far from room all along, even for one job: 7 x 12 + 18 + 1 > 90 + 10.
+ const full = prs(7);
+ const atLimit = (active, more = {}) =>
+ admit({
+ active,
+ peak: 16,
+ phase: 'room',
+ maxWait: 12,
+ maxLooks: 2,
+ ...more,
+ });
+
+ it('starts the whole run, once, when no other run started at its longest wait', async () => {
+ const result = await atLimit(full);
+ expect(result.looks).toBe(2);
+ expect(result.warning).toEqual([
+ expect.stringContaining(
+ 'starting, the one pull-request run to start at its longest wait for 10 minutes',
+ ),
+ ]);
+ // The rest of the run starts without another wait.
+ expect(result.outputs).toMatchObject({fits: 'true'});
+ expect(result.outputs.full).toBeUndefined();
+ });
+
+ it('turns the run away while another started at its longest wait in the last ten minutes', async () => {
+ // Run 80 waited twelve minutes and started three minutes before this
+ // run's longest wait ends (at 12).
+ const result = await atLimit([...full, startedRun(80, 9, 12)]);
+ expect(result.outputs.full).toBe('true');
+ expect(result.outputs.fits).toBe('false');
+ expect(result.warning[0]).toContain(
+ 'CI is full: still no room after 12 minutes',
+ );
+ expect(result.warning[0]).toContain(
+ 'run 80 started at its longest wait 3 minutes ago',
+ );
+ });
+
+ it('starts again once ten minutes have passed since the last start at the longest wait', async () => {
+ const result = await atLimit([...full, startedRun(80, 1, 12)]);
+ expect(result.outputs.full).toBeUndefined();
+ expect(result.outputs.fits).toBe('true');
+ });
+
+ it('lets a run that has waited longer start first', async () => {
+ const older = await atLimit([...full, waiter(81, -1)]);
+ expect(older.outputs.full).toBe('true');
+ expect(older.warning[0]).toContain(
+ '1 pull-request run that waited longer still waits',
+ );
+ // A run that began waiting after this one does not hold it back.
+ const younger = await atLimit([...full, waiter(82, 0.5)]);
+ expect(younger.outputs.full).toBeUndefined();
+ expect(younger.outputs.fits).toBe('true');
+ });
+
+ it('counts only pull-request runs that waited their longest, and not a fork CI was full for', async () => {
+ const result = await atLimit([
+ ...full,
+ // Found room after six minutes.
+ startedRun(83, 9, 6),
+ // A run CI was full for: its admission failed, so it never started.
+ startedRun(84, 9, 12, {admission: 'failure'}),
+ // A main check that waited its longest.
+ {...startedRun(85, 9, 15), event: 'workflow_dispatch'},
+ ]);
+ expect(result.outputs.full).toBeUndefined();
+ expect(result.outputs.fits).toBe('true');
+ });
+
+ it('ranks a run out of time whose admission has not decided as a waiter, never as a start', async () => {
+ // Run 86 began waiting first and ran out of time just before this one;
+ // its admission has not decided yet. It goes first; this run is stopped,
+ // and the reason names no run that did not start.
+ const older = await atLimit([
+ ...full,
+ startedRun(86, 11.9, 12.1, {admission: null}),
+ ]);
+ expect(older.outputs.full).toBe('true');
+ expect(older.warning[0]).toContain(
+ '1 pull-request run that waited longer still waits',
+ );
+ expect(older.warning[0]).not.toContain('run 86 started');
+ // One that began after this run does not hold it back.
+ const younger = await atLimit([
+ ...full,
+ startedRun(87, 12, 11.5, {admission: null}),
+ ]);
+ expect(younger.outputs.fits).toBe('true');
+ });
+
+ it("compares wait starts on GitHub's clock, ties going to the lower run id", async () => {
+ // This run's own wait step began 3 s before its script read the clock.
+ // A run that began 1 s before the script, after this run's step, is
+ // younger, not older.
+ const own = {
+ id: OWN,
+ status: 'completed',
+ event: 'pull_request',
+ path: '.github/workflows/ci.yml',
+ jobs: [
+ {
+ name: 'room',
+ status: 'in_progress',
+ steps: [{name: STEP, status: 'in_progress', started_at: at(-0.05)}],
+ },
+ ],
+ };
+ const younger = await atLimit([...full, own, waiter(88, -1 / 60)]);
+ expect(younger.outputs.fits).toBe('true');
+ // The same second: the lower run id waited longer.
+ const tieLower = await atLimit([...full, own, {...waiter(0, -0.05)}]);
+ expect(tieLower.outputs.full).toBe('true');
+ const tieHigher = await atLimit([...full, own, waiter(89, -0.05)]);
+ expect(tieHigher.outputs.fits).toBe('true');
+ });
+
+ it('never turns away a main run or a merge-queue entry at its longest wait', async () => {
+ const recent = [...full, startedRun(80, 12, 12), waiter(81, -5)];
+ const mainRun = await admit({
+ active: recent,
+ peak: 39,
+ isMain: true,
+ phase: 'room',
+ });
+ expect(mainRun.outputs.full).toBeUndefined();
+ expect(mainRun.warning[0]).toContain('starting.');
+ const entry = await admit({
+ active: recent,
+ peak: 16,
+ merge: true,
+ phase: 'room',
+ maxWait: 12,
+ maxLooks: 2,
+ });
+ expect(entry.outputs.full).toBeUndefined();
+ });
+
+ it('decides in the rest of the run too, when the Storybook build found room', async () => {
+ const result = await admit({
+ active: [...full, startedRun(80, 9, 12)],
+ peak: 16,
+ phase: 'rest',
+ fits: 'false',
+ started: String(NOW),
+ looked: '1',
+ next: String(NOW + 11.67 * 60_000),
+ maxWait: 12,
+ maxLooks: 2,
+ });
+ expect(result.outputs.full).toBe('true');
+ });
+
+ it('replays 11/02 19:06-19:20Z: ten runs out of time together start one at a time', async () => {
+ // Ten pull-request runs began waiting a few seconds apart while a main
+ // check waited; all ran out of time within two minutes. Each decides at
+ // its own limit, seeing the runs that decided before it.
+ const began = Array.from({length: 10}, (_, i) => i * 0.2);
+ const decided = [];
+ for (const [i, b] of began.entries()) {
+ const before = decided
+ .map(({id, start, at: endedAt}) =>
+ start ? startedRun(id, endedAt - b, 12) : null,
+ )
+ .filter(Boolean);
+ const later = began
+ .slice(i + 1)
+ .map((c, k) => waiter(200 + i + 1 + k, c - b));
+ const result = await atLimit([...full, ...before, ...later]);
+ decided.push({
+ id: 200 + i,
+ start: result.outputs.full !== 'true',
+ at: b + 12,
+ });
+ }
+ expect(decided.filter(d => d.start)).toHaveLength(1);
+ expect(decided[0].start).toBe(true);
+ });
+ });
+
it("keeps room for small workflows' measured most", async () => {
expect((await admit({active: [...prs(5), ...slims(18)]})).looks).toBe(1);
expect(
diff --git a/.github/scripts/ci-heavy-lanes.test.mjs b/.github/scripts/ci-heavy-lanes.test.mjs
index e0b4a7a100..d4bd126f02 100644
--- a/.github/scripts/ci-heavy-lanes.test.mjs
+++ b/.github/scripts/ci-heavy-lanes.test.mjs
@@ -67,7 +67,7 @@ describe('visual, accessibility and RTL owners', () => {
const visual = ci.jobs['pr-visual'];
const condition = ungated(visual.if);
expect(condition).toContain(
- "(github.event_name == 'pull_request' &&\n needs.build-storybook.result == 'success')",
+ "(github.event_name == 'pull_request' &&\n needs.build-storybook.result == 'success' &&\n needs.admission.outputs.full != 'true')",
);
expect(visual.needs).toContain('run-labels');
const step = name =>
diff --git a/.github/scripts/ci-interaction-webkit.test.mjs b/.github/scripts/ci-interaction-webkit.test.mjs
index 06e4e38c8d..168c129da4 100644
--- a/.github/scripts/ci-interaction-webkit.test.mjs
+++ b/.github/scripts/ci-interaction-webkit.test.mjs
@@ -56,7 +56,7 @@ describe('WebKit interaction proof routing', () => {
it('always reports, behind the same pull-request gate as the Chromium proofs', () => {
expect(webkit.if).toBe(chromium.if);
expect(ungated(webkit.if)).toBe(
- "github.event_name != 'workflow_dispatch' && always() && !cancelled() && needs.room.outputs.full != 'true'",
+ "github.event_name != 'workflow_dispatch' && always() && !cancelled() && needs.room.outputs.full != 'true' && needs.admission.outputs.full != 'true'",
);
// Admission too: the Storybook build starts beside it (spec:AST-030 FR16).
expect(webkit.needs).toEqual([
diff --git a/.github/scripts/ci-interaction.test.mjs b/.github/scripts/ci-interaction.test.mjs
index 296967dd64..bc8b5df6ca 100644
--- a/.github/scripts/ci-interaction.test.mjs
+++ b/.github/scripts/ci-interaction.test.mjs
@@ -47,7 +47,7 @@ describe('interaction proof routing', () => {
// Behind the pull-request gate (spec:AST-030 FR17).
expect(proofs.if).toMatch(/^\$\{\{ \(github\.event_name != 'pull_request'/);
expect(ungated(proofs.if)).toBe(
- "github.event_name != 'workflow_dispatch' && always() && !cancelled() && needs.room.outputs.full != 'true'",
+ "github.event_name != 'workflow_dispatch' && always() && !cancelled() && needs.room.outputs.full != 'true' && needs.admission.outputs.full != 'true'",
);
// Admission too: the Storybook build starts beside it (spec:AST-030 FR16).
expect(proofs.needs).toEqual([
diff --git a/.github/scripts/ci-tooling-routing.test.mjs b/.github/scripts/ci-tooling-routing.test.mjs
index ffaac1875a..2fc3450f12 100644
--- a/.github/scripts/ci-tooling-routing.test.mjs
+++ b/.github/scripts/ci-tooling-routing.test.mjs
@@ -443,9 +443,10 @@ describe('Node-tooling CI routing', () => {
});
it('preserves the historical joins and fails them on owned-lane failure', () => {
- // room only tells test when CI is full (ci-admission.test.mjs).
+ // room and admission only tell test when CI is full (ci-admission.test.mjs).
expect(ci.jobs.test.needs).toEqual([
'room',
+ 'admission',
'heavy-ci',
'release-bump',
'test-ui',
@@ -457,6 +458,7 @@ describe('Node-tooling CI routing', () => {
]);
expect(ci.jobs.build.needs).toEqual([
'room',
+ 'admission',
'heavy-ci',
'build-storybook',
'typecheck',
diff --git a/.github/scripts/touched-components.test.mjs b/.github/scripts/touched-components.test.mjs
index dde0f7c0dc..31a2cd1e96 100644
--- a/.github/scripts/touched-components.test.mjs
+++ b/.github/scripts/touched-components.test.mjs
@@ -488,9 +488,11 @@ describe('per-component results', () => {
describe('pr-touched-components routing', () => {
it('runs on every pull request after the Storybook build and pr-visual, on one standard arm runner', () => {
- expect(job.needs).toEqual(['build-storybook', 'pr-visual']);
+ // Admission too: a run CI is full for at its longest wait has its
+ // Storybook build, and builds nothing more (ci-admission.test.mjs).
+ expect(job.needs).toEqual(['admission', 'build-storybook', 'pr-visual']);
expect(ungated(job.if)).toBe(
- "!cancelled() && github.event_name == 'pull_request' && needs.build-storybook.result == 'success' && needs.build-storybook.outputs.lab_only != 'true'",
+ "!cancelled() && github.event_name == 'pull_request' && needs.build-storybook.result == 'success' && needs.build-storybook.outputs.lab_only != 'true' && needs.admission.outputs.full != 'true'",
);
expect(job['runs-on']).toBe('ubuntu-24.04-arm');
expect(job['timeout-minutes']).toBeLessThanOrEqual(15);
diff --git a/.github/scripts/visual-gate/workflow-concurrency.test.mjs b/.github/scripts/visual-gate/workflow-concurrency.test.mjs
index a83fb51cd4..a633b4371b 100644
--- a/.github/scripts/visual-gate/workflow-concurrency.test.mjs
+++ b/.github/scripts/visual-gate/workflow-concurrency.test.mjs
@@ -95,9 +95,13 @@ describe('PR report and deployment workflow contracts', () => {
value.indexOf(' pr-visual:'),
value.indexOf(' pr-rtl-shard:'),
);
- expect(visual).toContain(
- '[build-storybook, maintenance-request, visual-scout, run-labels]',
- );
+ expect(yaml.parse(value).jobs['pr-visual'].needs).toEqual([
+ 'admission',
+ 'build-storybook',
+ 'maintenance-request',
+ 'visual-scout',
+ 'run-labels',
+ ]);
expect(visual).toContain(
'name: storybook-${{ needs.build-storybook.outputs.short_hash }}',
);
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 3d6a8ec059..ea8d2eae0f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -152,11 +152,13 @@ jobs:
permissions:
actions: read
contents: read
+ outputs:
+ full: ${{ steps.rest.outputs.full }}
steps:
- name: Stop when CI is full
if: needs.room.outputs.full == 'true'
run: |
- echo "::error title=CI is full::CI is full: re-run all jobs or push later. Ten pull-request runs were already waiting for room."
+ echo "::error title=CI is full::CI is full: re-run all jobs or push later. The run found no room under the CI ceiling."
exit 1
- name: Check out the admission action
@@ -167,6 +169,7 @@ jobs:
persist-credentials: false
- name: Wait for room under the CI ceiling
+ id: rest
uses: ./.ci-admission/.github/actions/ci-admission
continue-on-error: true
timeout-minutes: ${{ github.event_name == 'workflow_dispatch' && 16 || 13 }}
@@ -182,14 +185,24 @@ jobs:
max-looks: ${{ github.event_name == 'workflow_dispatch' && 8 || 2 }}
label-legs: '{"ci:run-visual": 6, "ci:run-rtl": 9, "ci:run-a11y": 3}'
- # Stops a pull-request run that CI is full for (room found no room and ten
- # other pull-request runs waiting), with the reason, so it builds nothing.
- # It cancels the run when its token can. A fork's read-only token cannot;
- # then admission fails, the jobs behind it skip, and build, test and
+ # A pull-request run that waited its longest while another started at
+ # its longest wait is turned away (spec:AST-030 DEC-16): ci-full stops
+ # the run, and the jobs behind admission skip.
+ - name: Stop when CI is full at the longest wait
+ if: steps.rest.outputs.full == 'true'
+ run: |
+ echo "::error title=CI is full::CI is full: re-run all jobs or push later. The run found no room under the CI ceiling."
+ exit 1
+
+ # Stops a pull-request run that CI is full for, with the reason, so it builds
+ # nothing: room found no room and ten other pull-request runs waiting, or the
+ # run waited its longest while another started at its longest wait. It
+ # cancels the run when its token can. A fork's read-only token cannot; then
+ # admission fails, the jobs behind it skip or stop, and build, test and
# docsite-test fail with the same reason, never skip or pass.
ci-full:
- needs: [room]
- if: github.event_name == 'pull_request' && needs.room.outputs.full == 'true'
+ needs: [room, admission]
+ if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && (github.event_name == 'pull_request' && !cancelled() && (needs.room.outputs.full == 'true' || needs.admission.outputs.full == 'true')) }}
runs-on: ubuntu-slim
timeout-minutes: 5
permissions:
@@ -200,7 +213,7 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
run: |
- echo "::error title=CI is full::CI is full: re-run all jobs or push later. Ten pull-request runs were already waiting for room."
+ echo "::error title=CI is full::CI is full: re-run all jobs or push later. The run found no room under the CI ceiling."
gh run cancel "$GITHUB_RUN_ID" --repo "$GITHUB_REPOSITORY" || true
exit 1
@@ -562,9 +575,9 @@ jobs:
# reads the PR's file list and the base's trusted classifier, and
# everything is in scope when it cannot decide.
- name: Stop when CI is full
- if: needs.room.outputs.full == 'true'
+ if: needs.room.outputs.full == 'true' || needs.admission.outputs.full == 'true'
run: |
- echo "::error title=CI is full::CI is full: re-run all jobs or push later. Ten pull-request runs were already waiting for room."
+ echo "::error title=CI is full::CI is full: re-run all jobs or push later. The run found no room under the CI ceiling."
exit 1
# An agent lane's push defers the heavy suites (spec:AST-030 FR20): the
@@ -1118,6 +1131,7 @@ jobs:
needs:
[
room,
+ admission,
heavy-ci,
release-bump,
test-ui,
@@ -1142,9 +1156,9 @@ jobs:
# reads the PR's file list and the base's trusted classifier, and
# everything is in scope when it cannot decide.
- name: Stop when CI is full
- if: needs.room.outputs.full == 'true'
+ if: needs.room.outputs.full == 'true' || needs.admission.outputs.full == 'true'
run: |
- echo "::error title=CI is full::CI is full: re-run all jobs or push later. Ten pull-request runs were already waiting for room."
+ echo "::error title=CI is full::CI is full: re-run all jobs or push later. The run found no room under the CI ceiling."
exit 1
# The PR report says why the heavy checks did not run. Recorded here,
@@ -1682,7 +1696,7 @@ jobs:
# so a type error still fails both.
typecheck:
needs: [room, admission]
- if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && ((github.event_name != 'workflow_dispatch' || inputs.operation == 'release-check') && always() && !cancelled() && needs.room.outputs.full != 'true') }}
+ if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && ((github.event_name != 'workflow_dispatch' || inputs.operation == 'release-check') && always() && !cancelled() && needs.room.outputs.full != 'true' && needs.admission.outputs.full != 'true') }}
runs-on: ${{ (github.event_name == 'workflow_dispatch' && inputs.operation == 'release-check' && github.triggering_actor != 'github-actions[bot]') && '2-core-ubuntu-arm' || 'ubuntu-24.04-arm' }}
timeout-minutes: ${{ github.event_name == 'workflow_dispatch' && 360 || 15 }}
permissions:
@@ -1754,7 +1768,7 @@ jobs:
# gate passes. A skipped build-storybook job (a draft) skips this gate too;
# a fork pull request without `ci:run` still reports here.
build:
- needs: [room, heavy-ci, build-storybook, typecheck]
+ needs: [room, admission, heavy-ci, build-storybook, typecheck]
if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft) && (github.event_name != 'workflow_dispatch' && !cancelled() && (needs.build-storybook.result != 'skipped' || needs.room.outputs.full == 'true' || needs.heavy-ci.outputs.deferred == 'true' || github.event_name == 'pull_request' && !(github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run')))) }}
runs-on: ubuntu-slim
permissions: {}
@@ -1765,9 +1779,9 @@ jobs:
echo "::error::Waiting for a maintainer's ci:run label: an outside contributor's pull request runs CI once a maintainer adds it (spec:AST-030 FR17)."
exit 1
- name: Stop when CI is full
- if: needs.room.outputs.full == 'true'
+ if: needs.room.outputs.full == 'true' || needs.admission.outputs.full == 'true'
run: |
- echo "::error title=CI is full::CI is full: re-run all jobs or push later. Ten pull-request runs were already waiting for room."
+ echo "::error title=CI is full::CI is full: re-run all jobs or push later. The run found no room under the CI ceiling."
exit 1
# An agent lane's push defers the heavy suites (spec:AST-030 FR20): the
@@ -2274,7 +2288,7 @@ jobs:
pr-interaction:
needs: [room, admission, heavy-ci, build-storybook]
- if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && (!cancelled() && needs.heavy-ci.outputs.deferred != 'true' && (github.event_name != 'workflow_dispatch' && always() && !cancelled() && needs.room.outputs.full != 'true')) }}
+ if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && (!cancelled() && needs.heavy-ci.outputs.deferred != 'true' && (github.event_name != 'workflow_dispatch' && always() && !cancelled() && needs.room.outputs.full != 'true' && needs.admission.outputs.full != 'true')) }}
# Frame evidence measures timing: four cores, one Playwright worker.
runs-on: ubuntu-latest
timeout-minutes: 15
@@ -2392,7 +2406,7 @@ jobs:
# in WebKit.
pr-interaction-webkit:
needs: [room, admission, heavy-ci, build-storybook]
- if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && (!cancelled() && needs.heavy-ci.outputs.deferred != 'true' && (github.event_name != 'workflow_dispatch' && always() && !cancelled() && needs.room.outputs.full != 'true')) }}
+ if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && (!cancelled() && needs.heavy-ci.outputs.deferred != 'true' && (github.event_name != 'workflow_dispatch' && always() && !cancelled() && needs.room.outputs.full != 'true' && needs.admission.outputs.full != 'true')) }}
# Frame evidence measures timing: four cores, one Playwright worker.
runs-on: ubuntu-latest
timeout-minutes: 15
@@ -2716,11 +2730,19 @@ jobs:
# Audit and interaction fixtures stay with their existing browser checks.
pr-visual:
name: Stable visual regression
- needs: [build-storybook, maintenance-request, visual-scout, run-labels]
+ needs:
+ [
+ admission,
+ build-storybook,
+ maintenance-request,
+ visual-scout,
+ run-labels,
+ ]
if: >-
(github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && (!cancelled() &&
((github.event_name == 'pull_request' &&
- needs.build-storybook.result == 'success') ||
+ needs.build-storybook.result == 'success' &&
+ needs.admission.outputs.full != 'true') ||
(github.event_name == 'workflow_dispatch' &&
((inputs.operation == 'release-check' &&
needs.build-storybook.result == 'success') ||
@@ -3475,8 +3497,8 @@ jobs:
# pull-request comment. No required context depends on it.
pr-touched-components:
name: Touched components (visual, accessibility, RTL)
- needs: [build-storybook, pr-visual]
- if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && (!cancelled() && github.event_name == 'pull_request' && needs.build-storybook.result == 'success' && needs.build-storybook.outputs.lab_only != 'true') }}
+ needs: [admission, build-storybook, pr-visual]
+ if: ${{ (github.event_name != 'pull_request' || !github.event.pull_request.draft && (github.event.pull_request.head.repo.full_name == github.repository || contains(github.event.pull_request.labels.*.name, 'ci:run'))) && (!cancelled() && github.event_name == 'pull_request' && needs.build-storybook.result == 'success' && needs.build-storybook.outputs.lab_only != 'true' && needs.admission.outputs.full != 'true') }}
runs-on: ubuntu-24.04-arm
timeout-minutes: 15
permissions:
diff --git a/docs/specs/AST-030/spec.md b/docs/specs/AST-030/spec.md
index 7d68eff217..2f0903ead8 100644
--- a/docs/specs/AST-030/spec.md
+++ b/docs/specs/AST-030/spec.md
@@ -362,7 +362,10 @@ they are separately named.
jobs of room every four minutes, inside its longest wait. At most ten
pull-request runs wait at once: one more that finds no room stops at once with
"CI is full: re-run all jobs or push later", and its required checks report
- the stopped run. A pull request that may not run CI admits nothing, and
+ the stopped run. A pull-request run still without room at its longest wait
+ looks once more and starts only if no other pull-request run started at its
+ longest wait in the last ten minutes and no run that waited longer still
+ waits; otherwise it stops the same way. A pull request that may not run CI admits nothing, and
waiting never fails a job, so every required context reports as before.
- **FR17 — A pull request runs CI when it is ready and a maintainer can vouch
for it.** A draft pull request runs no CI, and marking it ready for review
@@ -559,6 +562,7 @@ The nightly perf lane is the one scheduled measurement outside them (FR18).
| FR14 | `workflow-paths.test.mjs` over every workflow | path allowlist; required-context workflow | a `paths-ignore`, a match-all or exclusion glob, or a path filter on a workflow that reports a required context fails |
| FR15 | workflow contract and reconciler tests | ENGOWNER spec PR; owner command; other comment; fork review; non-spec PR; run over the request budget; maintainer dispatch; fork PR, code-only: an owner's exact-head approval plus the owner command clears the code half | `review-required` succeeds while spec approval is missing, another workflow publishes a review context, a fork review starts the spec checks, a non-command comment starts a runner, an over-budget run decides, or a fork's approved code-only PR stays pending after the owner command |
| FR16 | `heavy-caps-action.test.mjs` cap cases and heavy-job contract; `ci-admission.test.mjs` | sixteenth-oldest PR; 101st heavy run of the day, beside cancelled, light and other-branch runs; agent lane and its five-PR and forty-run backstops; an engineering owner, including the shared login past those backstops, a commented-out or trailing-comment handle and an unreadable roster; bot; the deciding run; re-run; failed lookup; a run with room, without room, past its longest wait, beside a waiting `main` check, as a draft or unlabelled fork, with opt-in labels, beside small workflows past their measured most, and with the API unreadable; each workflow's most jobs; every required context's rule | a heavy suite that runs ahead of the author's fifteen oldest PRs, or past the daily cap, a capped job that passes, an engineering owner or a re-run that is held, or a roster read from the pull request's head, a run that starts without room before its longest wait, a pull-request run that starts while `main` waits, a job admission could skip, or a peak below the jobs its run holds fails |
+| FR16 | `ci-admission.test.mjs` cases for one start at the longest wait at a time and the workflow contract | a pull-request run at its longest wait alone, within ten minutes of another pull-request run's start there and after them, behind a run that waited longer and ahead of a younger one; another run that found room earlier, a fork's stopped run and a `main` run that waited its longest; a `main` run and a merge-queue entry at their longest wait; the rest of a run whose Storybook build found room; ten runs out of time together; the required contexts and ci-full for either wait | two pull-request runs start at their longest wait within ten minutes, a younger run starts ahead of a longer waiter, a `main` run or merge-queue entry stops, a run CI is full for builds, or a required context passes or skips |
| FR17 | `ci-pr-gate.test.mjs` over every `pull_request` workflow and the shared conditions | branch here, fork with and without `ci:run`, another label, draft with the label, ready, push, merge group, dispatch | a job that can start on a pull request without the condition, a label trigger, a required job that skips for a pull request that may not run CI, or a draft or unlabelled fork that runs fails |
| FR18 | `perf-nightly-workflow.test.mjs`, `perf-trend.test.mjs` and `run-labels.test.mjs` | nightly and dispatch; labelled, unlabelled, draft and unvouched-fork pull requests; write, read and no access; newest label event; failed lookup; 10% boundary; metric floors; seven-day window edges; fewer than three runs; a metric over once, twice in a row, or a different metric before; recorded and pull request runs | a run on an unlabelled pull request, a label honoured from read access, a regression called at exactly 10%, under its floor or on one run alone, a run outside the window in the baseline, a pull request run that records, or a second regression issue fails |
| FR19 | `ci-impact.test.mjs`, `pr-preview.test.mjs` and the maintenance template | CI change and non-CI change; standard, non-standard and expression runners; added, removed and changed jobs; trigger change | a CI change reports no impact, a move onto a non-standard runner or a pool or time-to-green rise past its threshold goes unflagged, or the section names a price |
@@ -889,14 +893,23 @@ budget, so `main` fits within fifteen minutes. Nothing is reserved for `main`
outside those minutes. At most ten pull-request runs wait at once, counting
pull-request runs only; one more that finds no room stops at once, "CI is full:
re-run all jobs or push later", rather than add a waiting job to a full CI. A
-`main` run is never stopped.
+pull-request run still without room at its longest wait looks once more, at a
+cost of one more look plus one call for its own jobs, and starts only if no
+other pull-request run started at its longest wait in the last ten minutes,
+about the time one run takes, and no run that waited longer still waits;
+otherwise it stops the same way. A run counts as started there once its
+admission job succeeded, and any run that waited out about its whole longest
+wait counts, room on its last look or not; until then it ranks as a waiter. Runs that wait out the same burst then start
+one at a time instead of together. A `main` run is never stopped.
CI can still pass ninety in five bounded ways, the first three rare:
- A run that has waited its longest (fifteen minutes for `main`, twelve for a
- pull request) starts whatever the load, even while a `main` check waits. That
- happens when a job outlives its budget, such as the thirty-minute macOS Swift
- job, and the run is over by at most its own jobs until that job ends.
+ pull request) starts whatever the load, even while a `main` check waits, at
+ most one pull-request run in ten minutes. That happens when a job outlives
+ its budget, such as the thirty-minute macOS Swift job, or when a `main` check
+ held the waiting runs until their longest waits; CI is over by at most one
+ pull-request run's jobs, and each `main` run's, until they end.
- Runs that look within the same few seconds can each see room. A random
0–20 s on every later look spreads them, and the excess is at most one run's
jobs until its next look.