Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

fix(ok): stop cancelling in-flight smoke ticks, unlatch the alarm (PRD-7742) (#3129)

Andrew Mikofalvy83980c72a953
diff --git a/.github/scripts/evaluate-smoke-alarm.mjs b/.github/scripts/evaluate-smoke-alarm.mjs
index c0d025059..e168f803b 100644
--- a/.github/scripts/evaluate-smoke-alarm.mjs
+++ b/.github/scripts/evaluate-smoke-alarm.mjs
@@ -101,7 +101,12 @@ export function buildHistory({ runs, jobsForRun }) {
return runs.map((run) => {
const jobs = jobsForRun(run.databaseId ?? run.id) ?? [];
const smoke = jobs.find((j) => j.name === SMOKE_JOB_NAME);
- if (!smoke || smoke.conclusion === 'skipped') {
+ // A cancelled smoke is a superseded tick, not an unanswered gate: the run
+ // was killed by workflow concurrency before the gate could give a verdict,
+ // and no refusal fired. Counting it as qualified made condition 2 latch
+ // for the whole window on a cut whose fixes had already shipped through
+ // the standard path, paging every tick for hours about nothing.
+ if (!smoke || smoke.conclusion === 'skipped' || smoke.conclusion === 'cancelled') {
return { at: run.createdAt, qualified: false, verdict: null, promoted: false };
}
const dispatch = (smoke.steps ?? []).find((s) => s.name === DISPATCH_STEP_NAME);
diff --git a/.github/scripts/evaluate-smoke-alarm.test.mjs b/.github/scripts/evaluate-smoke-alarm.test.mjs
index 54e9496f2..32c49f048 100644
--- a/.github/scripts/evaluate-smoke-alarm.test.mjs
+++ b/.github/scripts/evaluate-smoke-alarm.test.mjs
@@ -160,6 +160,20 @@ describe('buildHistory', () => {
expect(buildHistory({ runs, jobsForRun: () => jobs })[0].qualified).toBe(false);
});
+ test('a cancelled smoke job is a superseded tick, not an unanswered gate', () => {
+ // Workflow concurrency killed the run before the gate answered; no refusal
+ // fired and no dispatch could have happened. Counting it as qualified
+ // latched the armed-and-reaching-nothing alarm for the whole window.
+ const jobs = [
+ { name: "Smoke the fast-tier candidate's DMG", conclusion: 'cancelled', steps: [] },
+ ];
+ expect(buildHistory({ runs, jobsForRun: () => jobs })[0]).toMatchObject({
+ qualified: false,
+ verdict: null,
+ promoted: false,
+ });
+ });
+
test('a smoke job whose dispatch step succeeded is a pass', () => {
const jobs = [
{
diff --git a/.github/workflows/select-beta-to-promote.yml b/.github/workflows/select-beta-to-promote.yml
index bae9a80ea..9b1d75cba 100644
--- a/.github/workflows/select-beta-to-promote.yml
+++ b/.github/workflows/select-beta-to-promote.yml
@@ -77,7 +77,9 @@ name: Select beta to promote
# Concurrency + de-dupe: this evaluator carries its own group (distinct from the
# ok-release-cadence lock that release.yml / promote-stable.yml share) so it
# never blocks a real cut. Double-promotion is prevented three ways:
-# (1) cancel-in-progress supersedes an overlapping eval, (2) a fail-loud in-flight
+# (1) the concurrency group serializes ticks (one running, newest pending —
+# see the concurrency block for why in-flight runs are never cancelled),
+# (2) a fail-loud in-flight
# check skips when a promote-stable run is already in_progress/queued, and
# (3) the script stops at the first already-shipped beta (commit contained in the
# latest stable) and never selects one whose changes are already released.
@@ -129,10 +131,19 @@ permissions:
concurrency:
# Deliberately NOT the shared ok-release-cadence group — a read-only evaluator
# must never queue behind (or delay) a real beta cut / stable promotion.
- # cancel-in-progress:true lets a newer eval (e.g. a push arriving mid-cron-tick)
- # supersede an older one; the eval is idempotent so a cancelled run loses nothing.
+ #
+ # cancel-in-progress is FALSE, and the reason is the smoke leg. "A cancelled
+ # run loses nothing" was true of the evaluation but not of an in-flight DMG
+ # smoke: cancelling one un-nominates a QUALIFIED candidate, and the next tick
+ # does not retry it — by then a newer (usually minor) beta has crossed the
+ # fast soak and shadows the patch candidate for the rest of the cycle. On a
+ # busy push day that cancels the fast tier exactly when it is wanted; the
+ # first live qualification died this way, mid-smoke, and the aggregate alarm
+ # then paged about it for hours. Queueing gives at most one pending tick
+ # behind a bounded run (all jobs carry timeout-minutes), which GitHub
+ # collapses to the newest — freshness without killing gates mid-answer.
group: ok-auto-promote-stable
- cancel-in-progress: true
+ cancel-in-progress: false
env:
# THE fast-tier arming switch, read by both the selector and the aggregate
@@ -298,12 +309,13 @@ jobs:
# verdict simply means no fast-tier dispatch happens and the beta waits out
# its normal 24h soak on a later tick.
#
- # Interrupt-safe by construction. The workflow's concurrency group is
- # `cancel-in-progress: true`, so a tick can be superseded mid-run at any
- # point. This job holds no state a killed run could corrupt: it downloads to
- # the runner's scratch, mounts through the helper (which detaches on SIGTERM),
- # and its only effect is the final dispatch. Re-running reaches the same
- # decision from the same inputs.
+ # Runs to completion once started: the concurrency group no longer cancels
+ # in-flight ticks (see the concurrency block), because a smoke killed mid-run
+ # is not harmlessly idempotent — the next tick evaluates NEWER inputs, a
+ # fresher beta shadows the nominated candidate, and the qualified cut never
+ # gets its answer. The job still holds no state a crash could corrupt: it
+ # downloads to the runner's scratch, mounts through the helper (which
+ # detaches on SIGTERM), and its only effect is the final dispatch.
smoke-fast-tier-candidate:
name: Smoke the fast-tier candidate's DMG
needs: evaluate