Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

CI cancel recovery no longer re-runs a job that hit its own timeout

Claude707cba97d14b
diff --git a/docs/GITHUB_ACTIONS.md b/docs/GITHUB_ACTIONS.md
index c056431ea4..00a40ad579 100644
--- a/docs/GITHUB_ACTIONS.md
+++ b/docs/GITHUB_ACTIONS.md
@@ -789,6 +789,7 @@ retries only when every one of these holds:
| event is `pull_request` | Nightly, dispatch, and release-called runs are not ours to re-drive. |
| `run_attempt == 1` | The retry budget. `POST /rerun` makes attempt 2, whose cancel sees attempt 2 and stops — one retry per run, and a PR run is one run per head SHA. |
| no job concluded `failure`/`timed_out` | Fail-fast cancellation makes a genuinely red run *look* cancelled. Retrying it would re-run the suite on a broken tree. |
+| no job hit its own `timeout-minutes` | GitHub reports a timed-out job as `cancelled`; only its check-run annotation ("exceeded the maximum execution time") tells it from an external cancel (#11910). A re-run would time out again, so it skips as `job-timed-out`. |
| no newer run for the branch | That is a supersession; the newer run already covers this code. |
Any API lookup that cannot be completed skips the retry rather than assuming a
@@ -851,7 +852,7 @@ That summary lives on the recovery run, which nobody finds from the pull
request, so the workflow also **publishes the verdict back onto the cancelled
head SHA as a check run** named `CI cancel recovery` (issue #7438). It carries
the same fixed reason code the step summary renders — `superseded`,
-`job-failed`, `re-dispatched`, `retry-budget-exhausted`, `run-state-moved-on`,
+`job-failed`, `job-timed-out`, `re-dispatched`, `retry-budget-exhausted`, `run-state-moved-on`,
or one of the `*-unavailable` lookup failures — plus one sentence saying what
to do about it.
diff --git a/scripts/ci-retry-cancelled-run.js b/scripts/ci-retry-cancelled-run.js
index 2f7d4541df..6bc7433bd8 100644
--- a/scripts/ci-retry-cancelled-run.js
+++ b/scripts/ci-retry-cancelled-run.js
@@ -10,7 +10,7 @@
* reports a cancel as if the tests were red (issue 7437). The same SHA passes
* on the next attempt once the queue drains, so one re-dispatch clears it.
*
- * Five guards keep that from becoming a retry loop or from re-running work
+ * Six guards keep that from becoming a retry loop or from re-running work
* somebody deliberately stopped — the reasoning for each is the table under
* "External cancellation and one automatic retry" in docs/GITHUB_ACTIONS.md.
* The two least obvious, stated here because the code alone does not show them:
@@ -24,6 +24,9 @@
* - No job may have concluded `failure`/`timed_out`. This repository cancels
* its own run from a failing job (scripts/cancel-current-ci-run.js), which
* makes the RUN read `cancelled` while a job really did fail.
+ * - No job may have hit its own `timeout-minutes`. GitHub reports such a
+ * job as `cancelled`, so only its check-run annotation tells it apart
+ * from an external cancel (#11910).
*
* An API call that cannot be completed is NOT treated as "guard passed": every
* lookup failure returns without re-dispatching, because the cost of missing a
@@ -84,7 +87,18 @@ export const RETRY_DELAY_MS = 5 * 60_000;
* quietly eating the margin.
*/
export const MAX_RUNTIME_MS = RETRY_DELAY_MS
- + 2 * (MAX_JOB_PAGES + 2) * REQUEST_TIMEOUT_MS;
+ // +3: the live-run and sibling lookups, plus one round of annotation reads
+ // (issued in parallel, so one request timeout however many jobs it covers).
+ + 2 * (MAX_JOB_PAGES + 3) * REQUEST_TIMEOUT_MS;
+
+/**
+ * GitHub's annotation on a job killed by its own `timeout-minutes`. The jobs
+ * API reports that job as `cancelled`, not `timed_out` — a run cancelled only
+ * because a job timed out otherwise reads exactly like queue contention, and
+ * re-running it just times out again (#11910). The captured duration is a
+ * strict token (e.g. `20m0s`), so it is safe to render on the check.
+ */
+const JOB_TIMEOUT_PATTERN = /exceeded the maximum execution time(?: of ([0-9hms]{1,16}))?/i;
/** Real elapsed time. Injected in tests so no suite ever sleeps. */
const sleep = (ms) => new Promise((resolve) => { setTimeout(resolve, ms); });
@@ -116,6 +130,7 @@ const COMMIT_SHA_PATTERN = /^[0-9a-f]{40}$/i;
const CHECK_REASON_TEXT = {
're-dispatched': 'This run was cancelled with no successor run and no failing job, so CI was re-dispatched automatically — once. Watch the new run for the real result.',
superseded: 'This run was cancelled because a newer CI run started for the same pull request. That newer run carries the real result, so this one was deliberately not retried.',
+ 'job-timed-out': 'A job in this run exceeded its own `timeout-minutes`, which GitHub reports as cancelled rather than failed. That is a real red, not queue contention — a re-run would time out again — so it was not retried.',
'job-failed': 'This run was cancelled by CI itself after a job genuinely failed, which is why the run reads cancelled rather than failure. That is a real red, not queue contention, so it was not retried.',
'retry-budget-exhausted': 'This run is already a retry attempt, and recovery re-dispatches a run at most once. Re-run it manually if it was cancelled externally again.',
'run-state-moved-on': 'The run changed state after the cancellation event — it has already been re-run, or is no longer a completed cancellation — so the one-retry budget was left unspent.',
@@ -204,6 +219,31 @@ async function fetchAttemptJobs(fetchImpl, target, logger) {
return null;
}
+/**
+ * The cancelled jobs that were really killed by their own `timeout-minutes`,
+ * as `{name, limit}`, or null when any of their annotations could not be read
+ * — "we could not tell" must never read as "no job timed out".
+ *
+ * A job's id is its check-run id. The reads run in parallel so a pass costs
+ * one request timeout of wall clock however many jobs were cancelled.
+ */
+async function fetchTimedOutJobs(fetchImpl, target, jobs, logger) {
+ const verdicts = await Promise.all(jobs
+ .filter((job) => job?.conclusion === 'cancelled')
+ .map(async (job) => {
+ if (!/^\d+$/.test(String(job?.id ?? ''))) return null;
+ const url = `${target.repoPath}/check-runs/${job.id}/annotations?per_page=100`;
+ const annotations = await readJson(fetchImpl, url, target.token, logger);
+ if (!Array.isArray(annotations)) return null;
+ const match = annotations
+ .map((annotation) => JOB_TIMEOUT_PATTERN.exec(String(annotation?.message ?? '')))
+ .find(Boolean);
+ return match ? { name: safeJobName(job), limit: match[1] || 'its limit' } : {};
+ }));
+ if (verdicts.includes(null)) return null;
+ return verdicts.filter((verdict) => verdict.name);
+}
+
/**
* The run's state as GitHub holds it NOW, or null when it cannot be read.
*
@@ -277,6 +317,22 @@ async function evaluateApiGuards(fetchImpl, target, logger) {
result: { outcome: 'skipped', reason: 'job-failed' },
};
}
+ const timedOutJobs = await fetchTimedOutJobs(fetchImpl, target, jobs, logger);
+ if (!timedOutJobs) {
+ return {
+ level: 'error',
+ message: `⚠️ CI retry skipped: could not read the cancelled jobs' annotations for run ${target.runId}`,
+ result: { outcome: 'unavailable', reason: 'jobs-unavailable' },
+ };
+ }
+ if (timedOutJobs.length) {
+ const named = timedOutJobs.map(({ name, limit }) => `${name} after ${limit}`);
+ return {
+ level: 'log',
+ message: `ℹ️ CI retry skipped: run ${target.runId} had a job exceed its timeout (${named.join(', ')})`,
+ result: { outcome: 'skipped', reason: 'job-timed-out', timedOutJobs },
+ };
+ }
const live = await fetchLiveRunState(fetchImpl, target, logger);
if (!live) {
@@ -350,6 +406,11 @@ async function publishRecoveryCheck(fetchImpl, target, result, logger) {
+ `- outcome: \`${result.outcome}\`\n`
+ `- reason: \`${reason}\`\n\n`
+ `${CHECK_REASON_TEXT[reason] || CHECK_REASON_FALLBACK}\n\n`
+ // Job names come from the PR's own workflow file: a code span with
+ // its backticks removed renders them inert.
+ + (result.timedOutJobs || []).map(({ name, limit }) =>
+ `- \`${name.replace(/`/g, "'")}\` exceeded its timeout of \`${limit}\`\n`).join('')
+ + (result.timedOutJobs?.length ? '\n' : '')
+ 'This check is informational and always neutral — it never blocks a merge.',
},
}),
diff --git a/scripts/ci-retry-cancelled-run.test.js b/scripts/ci-retry-cancelled-run.test.js
index 035c46e771..5417b82f3b 100644
--- a/scripts/ci-retry-cancelled-run.test.js
+++ b/scripts/ci-retry-cancelled-run.test.js
@@ -46,7 +46,9 @@ const json = (body) => ({ ok: true, status: 200, json: async () => body });
* A fetch stub routed by URL shape. `jobs`/`runs` default to the clean
* external-cancel case so each test overrides only what it is about.
*/
-function stubFetch({ jobs = [], runs = [], rerunStatus = 201, checkStatus = 201, jobsStatus = 200, live } = {}) {
+function stubFetch({
+ jobs = [], runs = [], annotations = {}, rerunStatus = 201, checkStatus = 201, jobsStatus = 200, live,
+} = {}) {
const calls = [];
const impl = vi.fn(async (url, init = {}) => {
calls.push({ url, method: init.method || 'GET', body: init.body });
@@ -62,6 +64,8 @@ function stubFetch({ jobs = [], runs = [], rerunStatus = 201, checkStatus = 201,
workflow_runs: runs.map((r) => ({ head_repository: { id: 555 }, ...r })),
});
}
+ const annotated = url.match(/\/check-runs\/(\d+)\/annotations\?/);
+ if (annotated) return json(annotations[annotated[1]] || []);
if (url.endsWith('/rerun')) return { ok: rerunStatus < 300, status: rerunStatus };
if (url.endsWith('/check-runs')) return { ok: checkStatus < 300, status: checkStatus };
if (/\/actions\/runs\/\d+$/.test(url)) {
@@ -116,7 +120,7 @@ describe('retryTargetFromEnv', () => {
describe('retryCancelledCiRun guards', () => {
it('re-dispatches a cancelled pull-request run with no failing job and no successor', async () => {
const fetchStub = stubFetch({
- jobs: [{ name: 'DB tests', conclusion: 'success' }, { name: 'Client', conclusion: 'cancelled' }],
+ jobs: [{ id: 1, name: 'DB tests', conclusion: 'success' }, { id: 2, name: 'Client', conclusion: 'cancelled' }],
runs: [{ id: 4242, run_number: 900 }],
});
@@ -154,6 +158,29 @@ describe('retryCancelledCiRun guards', () => {
},
);
+ it('never retries a run whose job hit its own timeout-minutes, which GitHub reports as cancelled', async () => {
+ // The jobs API gives a timed-out job conclusion `cancelled` and leaves its
+ // running step unconcluded; only the check-run annotation names the
+ // timeout (#11910). A re-run would just time out again.
+ const fetchStub = stubFetch({
+ jobs: [
+ { id: 7, name: 'DB tests', conclusion: 'cancelled', steps: [{ conclusion: 'success' }, { conclusion: null }] },
+ { id: 8, name: 'Client', conclusion: 'cancelled' },
+ ],
+ annotations: {
+ 7: [{ annotation_level: 'failure', message: 'The job has exceeded the maximum execution time of 20m0s' }],
+ 8: [{ annotation_level: 'failure', message: 'The operation was canceled.' }],
+ },
+ });
+ await expect(run(BASE_ENV, fetchStub))
+ .resolves.toMatchObject({ outcome: 'skipped', reason: 'job-timed-out', phase: 'before-wait' });
+ expect(fetchStub.rerunRequested()).toBe(false);
+ const { summary } = fetchStub.publishedCheck().output;
+ expect(summary).toContain('reason: `job-timed-out`');
+ expect(summary).toContain('`DB tests` exceeded its timeout of `20m0s`');
+ expect(summary).not.toContain('`Client`');
+ });
+
it('retries a job whose STEP failed but whose conclusion reads cancelled', async () => {
// The fail-fast self-cancel can land while the failing job is still in its
// post-steps, and GitHub then records that job as `cancelled`. Trusting the