Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
CI cancel recovery no longer re-runs a job that hit its own timeout
Claude707cba97d14b
diff --git a/docs/GITHUB_ACTIONS.md b/docs/GITHUB_ACTIONS.mdindex c056431ea4..00a40ad579 100644--- a/docs/GITHUB_ACTIONS.md+++ b/docs/GITHUB_ACTIONS.md@@ -789,6 +789,7 @@ retries only when every one of these holds:| event is `pull_request` | Nightly, dispatch, and release-called runs are not ours to re-drive. || `run_attempt == 1` | The retry budget. `POST /rerun` makes attempt 2, whose cancel sees attempt 2 and stops — one retry per run, and a PR run is one run per head SHA. || no job concluded `failure`/`timed_out` | Fail-fast cancellation makes a genuinely red run *look* cancelled. Retrying it would re-run the suite on a broken tree. |+| no job hit its own `timeout-minutes` | GitHub reports a timed-out job as `cancelled`; only its check-run annotation ("exceeded the maximum execution time") tells it from an external cancel (#11910). A re-run would time out again, so it skips as `job-timed-out`. || no newer run for the branch | That is a supersession; the newer run already covers this code. |Any API lookup that cannot be completed skips the retry rather than assuming a@@ -851,7 +852,7 @@ That summary lives on the recovery run, which nobody finds from the pullrequest, so the workflow also **publishes the verdict back onto the cancelledhead SHA as a check run** named `CI cancel recovery` (issue #7438). It carriesthe same fixed reason code the step summary renders — `superseded`,-`job-failed`, `re-dispatched`, `retry-budget-exhausted`, `run-state-moved-on`,+`job-failed`, `job-timed-out`, `re-dispatched`, `retry-budget-exhausted`, `run-state-moved-on`,or one of the `*-unavailable` lookup failures — plus one sentence saying whatto do about it.diff --git a/scripts/ci-retry-cancelled-run.js b/scripts/ci-retry-cancelled-run.jsindex 2f7d4541df..6bc7433bd8 100644--- a/scripts/ci-retry-cancelled-run.js+++ b/scripts/ci-retry-cancelled-run.js@@ -10,7 +10,7 @@* reports a cancel as if the tests were red (issue 7437). The same SHA passes* on the next attempt once the queue drains, so one re-dispatch clears it.*- * Five guards keep that from becoming a retry loop or from re-running work+ * Six guards keep that from becoming a retry loop or from re-running work* somebody deliberately stopped — the reasoning for each is the table under* "External cancellation and one automatic retry" in docs/GITHUB_ACTIONS.md.* The two least obvious, stated here because the code alone does not show them:@@ -24,6 +24,9 @@* - No job may have concluded `failure`/`timed_out`. This repository cancels* its own run from a failing job (scripts/cancel-current-ci-run.js), which* makes the RUN read `cancelled` while a job really did fail.+ * - No job may have hit its own `timeout-minutes`. GitHub reports such a+ * job as `cancelled`, so only its check-run annotation tells it apart+ * from an external cancel (#11910).** An API call that cannot be completed is NOT treated as "guard passed": every* lookup failure returns without re-dispatching, because the cost of missing a@@ -84,7 +87,18 @@ export const RETRY_DELAY_MS = 5 * 60_000;* quietly eating the margin.*/export const MAX_RUNTIME_MS = RETRY_DELAY_MS- + 2 * (MAX_JOB_PAGES + 2) * REQUEST_TIMEOUT_MS;+ // +3: the live-run and sibling lookups, plus one round of annotation reads+ // (issued in parallel, so one request timeout however many jobs it covers).+ + 2 * (MAX_JOB_PAGES + 3) * REQUEST_TIMEOUT_MS;++/**+ * GitHub's annotation on a job killed by its own `timeout-minutes`. The jobs+ * API reports that job as `cancelled`, not `timed_out` — a run cancelled only+ * because a job timed out otherwise reads exactly like queue contention, and+ * re-running it just times out again (#11910). The captured duration is a+ * strict token (e.g. `20m0s`), so it is safe to render on the check.+ */+const JOB_TIMEOUT_PATTERN = /exceeded the maximum execution time(?: of ([0-9hms]{1,16}))?/i;/** Real elapsed time. Injected in tests so no suite ever sleeps. */const sleep = (ms) => new Promise((resolve) => { setTimeout(resolve, ms); });@@ -116,6 +130,7 @@ const COMMIT_SHA_PATTERN = /^[0-9a-f]{40}$/i;const CHECK_REASON_TEXT = {'re-dispatched': 'This run was cancelled with no successor run and no failing job, so CI was re-dispatched automatically — once. Watch the new run for the real result.',superseded: 'This run was cancelled because a newer CI run started for the same pull request. That newer run carries the real result, so this one was deliberately not retried.',+ 'job-timed-out': 'A job in this run exceeded its own `timeout-minutes`, which GitHub reports as cancelled rather than failed. That is a real red, not queue contention — a re-run would time out again — so it was not retried.','job-failed': 'This run was cancelled by CI itself after a job genuinely failed, which is why the run reads cancelled rather than failure. That is a real red, not queue contention, so it was not retried.','retry-budget-exhausted': 'This run is already a retry attempt, and recovery re-dispatches a run at most once. Re-run it manually if it was cancelled externally again.','run-state-moved-on': 'The run changed state after the cancellation event — it has already been re-run, or is no longer a completed cancellation — so the one-retry budget was left unspent.',@@ -204,6 +219,31 @@ async function fetchAttemptJobs(fetchImpl, target, logger) {return null;}+/**+ * The cancelled jobs that were really killed by their own `timeout-minutes`,+ * as `{name, limit}`, or null when any of their annotations could not be read+ * — "we could not tell" must never read as "no job timed out".+ *+ * A job's id is its check-run id. The reads run in parallel so a pass costs+ * one request timeout of wall clock however many jobs were cancelled.+ */+async function fetchTimedOutJobs(fetchImpl, target, jobs, logger) {+ const verdicts = await Promise.all(jobs+ .filter((job) => job?.conclusion === 'cancelled')+ .map(async (job) => {+ if (!/^\d+$/.test(String(job?.id ?? ''))) return null;+ const url = `${target.repoPath}/check-runs/${job.id}/annotations?per_page=100`;+ const annotations = await readJson(fetchImpl, url, target.token, logger);+ if (!Array.isArray(annotations)) return null;+ const match = annotations+ .map((annotation) => JOB_TIMEOUT_PATTERN.exec(String(annotation?.message ?? '')))+ .find(Boolean);+ return match ? { name: safeJobName(job), limit: match[1] || 'its limit' } : {};+ }));+ if (verdicts.includes(null)) return null;+ return verdicts.filter((verdict) => verdict.name);+}+/*** The run's state as GitHub holds it NOW, or null when it cannot be read.*@@ -277,6 +317,22 @@ async function evaluateApiGuards(fetchImpl, target, logger) {result: { outcome: 'skipped', reason: 'job-failed' },};}+ const timedOutJobs = await fetchTimedOutJobs(fetchImpl, target, jobs, logger);+ if (!timedOutJobs) {+ return {+ level: 'error',+ message: `⚠️ CI retry skipped: could not read the cancelled jobs' annotations for run ${target.runId}`,+ result: { outcome: 'unavailable', reason: 'jobs-unavailable' },+ };+ }+ if (timedOutJobs.length) {+ const named = timedOutJobs.map(({ name, limit }) => `${name} after ${limit}`);+ return {+ level: 'log',+ message: `ℹ️ CI retry skipped: run ${target.runId} had a job exceed its timeout (${named.join(', ')})`,+ result: { outcome: 'skipped', reason: 'job-timed-out', timedOutJobs },+ };+ }const live = await fetchLiveRunState(fetchImpl, target, logger);if (!live) {@@ -350,6 +406,11 @@ async function publishRecoveryCheck(fetchImpl, target, result, logger) {+ `- outcome: \`${result.outcome}\`\n`+ `- reason: \`${reason}\`\n\n`+ `${CHECK_REASON_TEXT[reason] || CHECK_REASON_FALLBACK}\n\n`+ // Job names come from the PR's own workflow file: a code span with+ // its backticks removed renders them inert.+ + (result.timedOutJobs || []).map(({ name, limit }) =>+ `- \`${name.replace(/`/g, "'")}\` exceeded its timeout of \`${limit}\`\n`).join('')+ + (result.timedOutJobs?.length ? '\n' : '')+ 'This check is informational and always neutral — it never blocks a merge.',},}),diff --git a/scripts/ci-retry-cancelled-run.test.js b/scripts/ci-retry-cancelled-run.test.jsindex 035c46e771..5417b82f3b 100644--- a/scripts/ci-retry-cancelled-run.test.js+++ b/scripts/ci-retry-cancelled-run.test.js@@ -46,7 +46,9 @@ const json = (body) => ({ ok: true, status: 200, json: async () => body });* A fetch stub routed by URL shape. `jobs`/`runs` default to the clean* external-cancel case so each test overrides only what it is about.*/-function stubFetch({ jobs = [], runs = [], rerunStatus = 201, checkStatus = 201, jobsStatus = 200, live } = {}) {+function stubFetch({+ jobs = [], runs = [], annotations = {}, rerunStatus = 201, checkStatus = 201, jobsStatus = 200, live,+} = {}) {const calls = [];const impl = vi.fn(async (url, init = {}) => {calls.push({ url, method: init.method || 'GET', body: init.body });@@ -62,6 +64,8 @@ function stubFetch({ jobs = [], runs = [], rerunStatus = 201, checkStatus = 201,workflow_runs: runs.map((r) => ({ head_repository: { id: 555 }, ...r })),});}+ const annotated = url.match(/\/check-runs\/(\d+)\/annotations\?/);+ if (annotated) return json(annotations[annotated[1]] || []);if (url.endsWith('/rerun')) return { ok: rerunStatus < 300, status: rerunStatus };if (url.endsWith('/check-runs')) return { ok: checkStatus < 300, status: checkStatus };if (/\/actions\/runs\/\d+$/.test(url)) {@@ -116,7 +120,7 @@ describe('retryTargetFromEnv', () => {describe('retryCancelledCiRun guards', () => {it('re-dispatches a cancelled pull-request run with no failing job and no successor', async () => {const fetchStub = stubFetch({- jobs: [{ name: 'DB tests', conclusion: 'success' }, { name: 'Client', conclusion: 'cancelled' }],+ jobs: [{ id: 1, name: 'DB tests', conclusion: 'success' }, { id: 2, name: 'Client', conclusion: 'cancelled' }],runs: [{ id: 4242, run_number: 900 }],});@@ -154,6 +158,29 @@ describe('retryCancelledCiRun guards', () => {},);+ it('never retries a run whose job hit its own timeout-minutes, which GitHub reports as cancelled', async () => {+ // The jobs API gives a timed-out job conclusion `cancelled` and leaves its+ // running step unconcluded; only the check-run annotation names the+ // timeout (#11910). A re-run would just time out again.+ const fetchStub = stubFetch({+ jobs: [+ { id: 7, name: 'DB tests', conclusion: 'cancelled', steps: [{ conclusion: 'success' }, { conclusion: null }] },+ { id: 8, name: 'Client', conclusion: 'cancelled' },+ ],+ annotations: {+ 7: [{ annotation_level: 'failure', message: 'The job has exceeded the maximum execution time of 20m0s' }],+ 8: [{ annotation_level: 'failure', message: 'The operation was canceled.' }],+ },+ });+ await expect(run(BASE_ENV, fetchStub))+ .resolves.toMatchObject({ outcome: 'skipped', reason: 'job-timed-out', phase: 'before-wait' });+ expect(fetchStub.rerunRequested()).toBe(false);+ const { summary } = fetchStub.publishedCheck().output;+ expect(summary).toContain('reason: `job-timed-out`');+ expect(summary).toContain('`DB tests` exceeded its timeout of `20m0s`');+ expect(summary).not.toContain('`Client`');+ });+it('retries a job whose STEP failed but whose conclusion reads cancelled', async () => {// The fail-fast self-cancel can land while the failing job is still in its// post-steps, and GitHub then records that job as `cancelled`. Trusting the