Beautiful, AI-native markdown IDE and LLM wiki
TypeScript 2nd-brainagent-skillsclaudecodexcompany-braindocsknowledge-knowledge-baseknowledge-managementllmllm-wikillm-wiki-karpathymarkdownmarkdown-editormdmdxnotespkmsecond-brainskills
fix(open-knowledge): per-attempt timeout in bun-install-ci.sh so a hung install self-heals (#1574)
Andrew Mikofalvy197c1e48e254
diff --git a/scripts/bun-install-ci.sh b/scripts/bun-install-ci.shindex 4bba756f8..54781fc58 100755--- a/scripts/bun-install-ci.sh+++ b/scripts/bun-install-ci.sh@@ -14,14 +14,26 @@# bug class is package-agnostic; specific run IDs live in the PR that# landed this wrapper (search git log for `bun-install-ci`).#+# A second shape is a HANG, not a fast failure: on a cold cache the+# network phase can stall indefinitely (bun 1.3.x has no install/network+# idle timeout — `bun install --help` exposes only --network-concurrency).+# A stalled process never exits, so a retry-on-exit wrapper alone never+# engages — the hang rides the GitHub job to its `timeout-minutes` cap,+# which CANCELS the whole job (and any required gate it feeds) and cannot+# retry. The per-attempt watchdog below converts that hang into a+# bounded, retryable non-zero exit.+## WHAT IT DOES# Runs `bun install --frozen-lockfile` (or whatever is in $BUN_INSTALL_CMD)# up to $BUN_INSTALL_MAX_ATTEMPTS times, sleeping# $BUN_INSTALL_RETRY_SLEEP_BASE * 2^(n-1) seconds between attempts-# (5s, 10s for the prod default of 3 attempts at 5s base). Emits a-# GitHub Actions ::warning:: annotation per retry and a single ::error::-# annotation on final exhaustion so the noise is visible in the Actions-# UI without masking persistent failures.+# (5s, 10s for the prod default of 3 attempts at 5s base). Each attempt is+# bounded by a $BUN_INSTALL_ATTEMPT_TIMEOUT-second watchdog: a hung attempt+# is SIGTERM'd (then SIGKILL'd after a grace window) and counts as a+# retryable failure. Emits a GitHub Actions ::warning:: annotation per+# retry (distinguishing a timeout from an ordinary failure) and a single+# ::error:: annotation on final exhaustion so the noise is visible in the+# Actions UI without masking persistent failures.## WHY NOT SOMETHING ELSE# - Wider Bun cache scope is orthogonal — it reduces frequency of cold@@ -29,6 +41,13 @@# not substitute.# - `.bun-version` bump: oven-sh/bun#26879 is still open. When it lands,# remove this wrapper and the workflow call sites.+# - GNU `timeout` / `gtimeout` for the per-attempt bound: not portable —+# neither ships with base macOS, and GitHub's macos runner images do not+# guarantee GNU coreutils. The watchdog here is pure bash so it behaves+# identically on the Linux and macOS runners.+# - A GitHub Actions step-level `timeout-minutes` on the install step would+# fail faster but still only CANCELS (red, no retry). The point is to+# self-heal, so the bound lives here where a retry can follow it.# - Inline retry at each call site: 11 call sites in total — 5 in# public-open-knowledge-validation.yml, 3 in root .github/workflows/# (beta-cut, main-reset, mirror-sync), and 3 in OK-mirrored workflows@@ -50,11 +69,27 @@# BUN_INSTALL_RETRY_SLEEP_BASE Base seconds between retries (default: 5).# Must be a non-negative integer. Doubled# each retry: 5s, 10s, 20s. Tests pass 0.+# BUN_INSTALL_ATTEMPT_TIMEOUT Per-attempt wall-clock budget in seconds+# (default: 240). Must be a non-negative+# integer; 0 disables the watchdog (run the+# install unbounded). The default is chosen+# to stay under a 15-min job cap even if+# every attempt times out AND ignores SIGTERM+# (so the watchdog burns the full grace+# window before SIGKILL):+# 3 * (240s + 10s grace) + backoff (5s + 10s)+# ≈ 12.75 min < 15 min. A healthy cold install+# finishes well inside 240s.+# BUN_INSTALL_KILL_GRACE Seconds to wait after SIGTERM before+# SIGKILL when a timed-out attempt ignores+# the term signal (default: 10). Non-negative+# integer. Tests pass a small value.## EXIT# 0 on success at any attempt.# 64 on invalid input (per sysexits.h EX_USAGE).-# Last attempt's exit code on retry exhaustion.+# 124 when the final attempt timed out (GNU `timeout` convention).+# Last attempt's exit code on retry exhaustion otherwise.## CALL FORM (workflow YAML)# - run: bash scripts/bun-install-ci.sh@@ -71,15 +106,17 @@ set -euo pipefailBUN_INSTALL_MAX_ATTEMPTS="${BUN_INSTALL_MAX_ATTEMPTS:-3}"BUN_INSTALL_RETRY_SLEEP_BASE="${BUN_INSTALL_RETRY_SLEEP_BASE:-5}"+BUN_INSTALL_ATTEMPT_TIMEOUT="${BUN_INSTALL_ATTEMPT_TIMEOUT:-240}"+BUN_INSTALL_KILL_GRACE="${BUN_INSTALL_KILL_GRACE:-10}"BUN_INSTALL_CMD="${BUN_INSTALL_CMD:-}"# Input validation. Reject anything other than a positive integer for-# attempts and a non-negative integer for sleep base. The previous-# attempt-counter loop used `[ -ge ]` to detect exhaustion, which silently-# returns false on non-integer rhs and produced an unbounded retry loop-# when MAX_ATTEMPTS was misconfigured (e.g. "3.0", "3 ", "abc"). Validate-# loudly here so misconfiguration fails the CI step in milliseconds rather-# than emitting hundreds of ::warning::s until the job times out.+# attempts and a non-negative integer for the sleep base / timeout / grace.+# The previous attempt-counter loop used `[ -ge ]` to detect exhaustion,+# which silently returns false on non-integer rhs and produced an unbounded+# retry loop when MAX_ATTEMPTS was misconfigured (e.g. "3.0", "3 ", "abc").+# Validate loudly here so misconfiguration fails the CI step in milliseconds+# rather than emitting hundreds of ::warning::s until the job times out.if ! [[ $BUN_INSTALL_MAX_ATTEMPTS =~ ^[1-9][0-9]*$ ]]; thenecho "::error::BUN_INSTALL_MAX_ATTEMPTS must be a positive integer, got '${BUN_INSTALL_MAX_ATTEMPTS}'" >&2exit 64@@ -88,29 +125,104 @@ if ! [[ $BUN_INSTALL_RETRY_SLEEP_BASE =~ ^[0-9]+$ ]]; thenecho "::error::BUN_INSTALL_RETRY_SLEEP_BASE must be a non-negative integer, got '${BUN_INSTALL_RETRY_SLEEP_BASE}'" >&2exit 64fi+if ! [[ $BUN_INSTALL_ATTEMPT_TIMEOUT =~ ^[0-9]+$ ]]; then+ echo "::error::BUN_INSTALL_ATTEMPT_TIMEOUT must be a non-negative integer, got '${BUN_INSTALL_ATTEMPT_TIMEOUT}'" >&2+ exit 64+fi+if ! [[ $BUN_INSTALL_KILL_GRACE =~ ^[0-9]+$ ]]; then+ echo "::error::BUN_INSTALL_KILL_GRACE must be a non-negative integer, got '${BUN_INSTALL_KILL_GRACE}'" >&2+ exit 64+fi-# Single source of truth for the install invocation. The `if -n` guard-# keeps the default path identical to what every prior workflow used —-# only test runs swap in a stub.+# Single source of truth for the install argv. The default path is identical+# to what every prior workflow used — only test runs swap in a stub via+# $BUN_INSTALL_CMD.+if [ -n "$BUN_INSTALL_CMD" ]; then+ INSTALL_ARGV=("$BUN_INSTALL_CMD")+else+ INSTALL_ARGV=(bun install --frozen-lockfile)+fi++# Run one attempt directly (no watchdog). Used when BUN_INSTALL_ATTEMPT_TIMEOUT+# is 0. Returns the install's own exit code; control returns to the retry loop.run_install() {- if [ -n "$BUN_INSTALL_CMD" ]; then- "$BUN_INSTALL_CMD" "$@"- else- bun install --frozen-lockfile "$@"+ "${INSTALL_ARGV[@]}" "$@"+}++# Run one attempt under a portable, bash-native watchdog. Returns the install's+# exit code, or 124 if the watchdog had to kill it (matching GNU `timeout`).+#+# The install runs in a backgrounded subshell that `exec`s the command, so the+# backgrounded PID *is* the install process — SIGTERM/SIGKILL hit bun directly+# rather than an intermediate shell that would orphan it. A sibling watchdog+# subshell polls in 1s ticks (rather than one long `sleep`) — both pre-SIGTERM+# and during the SIGTERM→SIGKILL grace window — so that, however it is torn+# down, no orphaned `sleep` can linger; it also self-exits within ~1s of the+# install finishing on its own, and skips the SIGKILL entirely when the+# install responds to SIGTERM before the grace window expires.+run_install_timed() {+ ( exec "${INSTALL_ARGV[@]}" "$@" ) &+ local install_pid=$!++ (+ waited=0 # subshell-local; `local` is illegal outside a function+ while [ "$waited" -lt "$BUN_INSTALL_ATTEMPT_TIMEOUT" ]; do+ kill -0 "$install_pid" 2>/dev/null || exit 0 # install finished on its own+ sleep 1+ waited=$((waited + 1))+ done+ kill -TERM "$install_pid" 2>/dev/null || exit 0+ grace_waited=0+ while [ "$grace_waited" -lt "$BUN_INSTALL_KILL_GRACE" ]; do+ kill -0 "$install_pid" 2>/dev/null || exit 0 # SIGTERM took; skip the SIGKILL+ sleep 1+ grace_waited=$((grace_waited + 1))+ done+ kill -KILL "$install_pid" 2>/dev/null || true+ ) &+ local watchdog_pid=$!++ local rc=0+ wait "$install_pid" 2>/dev/null || rc=$?++ # Install finished (success, own failure, or watchdog kill). Retire the+ # watchdog; SIGTERM unblocks its current 1s tick so the wait returns at once.+ kill -TERM "$watchdog_pid" 2>/dev/null || true+ wait "$watchdog_pid" 2>/dev/null || true++ # A watchdog kill surfaces as 143 (128+SIGTERM) or 137 (128+SIGKILL).+ # Normalize both to 124 so the retry annotation can name the hang and+ # downstream sees the stable "timed out" code.+ if [ "$rc" -eq 143 ] || [ "$rc" -eq 137 ]; then+ return 124fi+ return "$rc"}attempt=1while true; dorc=0- run_install "$@" || rc=$?+ if [ "$BUN_INSTALL_ATTEMPT_TIMEOUT" -eq 0 ]; then+ run_install "$@" || rc=$?+ else+ run_install_timed "$@" || rc=$?+ fiif [ "$rc" -eq 0 ]; thenexit 0fi++ # Name the failure mode so a hang (exit 124) is distinguishable from an+ # ordinary install failure in the Actions Annotations panel.+ if [ "$rc" -eq 124 ]; then+ reason="timed out after ${BUN_INSTALL_ATTEMPT_TIMEOUT}s"+ else+ reason="failed (exit ${rc})"+ fi+if [ "$attempt" -ge "$BUN_INSTALL_MAX_ATTEMPTS" ]; thennoun="attempts"[ "$BUN_INSTALL_MAX_ATTEMPTS" = "1" ] && noun="attempt"- echo "::error::bun install --frozen-lockfile failed after ${BUN_INSTALL_MAX_ATTEMPTS} ${noun} (last exit ${rc}). Tracker: https://github.com/oven-sh/bun/issues/26879"+ echo "::error::bun install --frozen-lockfile ${reason}; giving up after ${BUN_INSTALL_MAX_ATTEMPTS} ${noun}. Tracker: https://github.com/oven-sh/bun/issues/26879"exit "$rc"fisleep_for=$((BUN_INSTALL_RETRY_SLEEP_BASE * (1 << (attempt - 1))))@@ -118,7 +230,7 @@ while true; do# gh-api retry in public-open-knowledge-validation.yml (`(attempt N/M)`# parenthetical) so operators scanning the Annotations panel see a# uniform shape across CI jobs.- echo "::warning::bun install --frozen-lockfile failed (attempt ${attempt}/${BUN_INSTALL_MAX_ATTEMPTS}, exit ${rc}); retrying in ${sleep_for}s"+ echo "::warning::bun install --frozen-lockfile ${reason} (attempt ${attempt}/${BUN_INSTALL_MAX_ATTEMPTS}); retrying in ${sleep_for}s"sleep "$sleep_for"attempt=$((attempt + 1))donediff --git a/scripts/bun-install-ci.test.mjs b/scripts/bun-install-ci.test.mjsindex 8cd7eac3f..078c98631 100644--- a/scripts/bun-install-ci.test.mjs+++ b/scripts/bun-install-ci.test.mjs@@ -48,6 +48,33 @@ case "$TEST_STUB_BEHAVIOR" inecho "stub: synthetic non-1 failure (exit \${TEST_FAIL_EXIT_CODE:-1})" >&2exit "\${TEST_FAIL_EXIT_CODE:-1}";;+ hang-then-pass)+ # Attempt 1 blocks far past the wrapper's BUN_INSTALL_ATTEMPT_TIMEOUT so+ # the watchdog must kill it; attempt 2 succeeds. Trap TERM to exit promptly+ # (and reap the inner sleep) so the SIGTERM path resolves cleanly.+ if [ "$attempt" -le 1 ]; then+ sleep "\${TEST_HANG_SECONDS:-30}" & sleep_pid=$!+ trap 'kill "$sleep_pid" 2>/dev/null; exit 143' TERM+ wait "$sleep_pid"+ exit 0+ fi+ echo "stub: pass on attempt $attempt"+ exit 0+ ;;+ always-hang)+ # Every attempt hangs until the watchdog SIGTERMs it.+ sleep "\${TEST_HANG_SECONDS:-30}" & sleep_pid=$!+ trap 'kill "$sleep_pid" 2>/dev/null; exit 143' TERM+ wait "$sleep_pid"+ exit 0+ ;;+ always-hang-ignore-term)+ # Ignore SIGTERM so the watchdog must escalate to SIGKILL. Loop short+ # sleeps (rather than one long sleep) so the orphan the uncatchable SIGKILL+ # leaves behind is bounded to ~1s instead of the full hang duration.+ trap '' TERM+ while :; do sleep 1; done+ ;;*)echo "stub: unknown TEST_STUB_BEHAVIOR='$TEST_STUB_BEHAVIOR'" >&2exit 2@@ -215,4 +242,121 @@ describe('bun-install-ci.sh — retry wrapper for `bun install --frozen-lockfileexpect(out).toContain('::error::');expect(out).toContain('BUN_INSTALL_RETRY_SLEEP_BASE');});++ test('timeout recovery: attempt 1 hangs (killed by watchdog) then attempt 2 passes → exit 0 with one timeout ::warning::', () => {+ const ctx = setupStub();+ const result = runWrapper({+ ...ctx,+ behavior: 'hang-then-pass',+ maxAttempts: 3,+ extraEnv: { BUN_INSTALL_ATTEMPT_TIMEOUT: '3', BUN_INSTALL_KILL_GRACE: '1' },+ timeoutMs: 20000,+ });++ expect(result.signal).toBeNull();+ expect(result.status).toBe(0);+ const out = combinedOutput(result);+ expect(countMatches(out, '::warning::')).toBe(1);+ expect(out).toContain('timed out after 3s');+ expect(out).not.toContain('::error::');+ }, 30000); // Bun per-test timeout > the 3s budget + retries + the 20s spawnSync net++ test('timeout exhaustion: every attempt hangs → exit 124 with (N-1) ::warning::s + 1 ::error::', () => {+ const ctx = setupStub();+ const result = runWrapper({+ ...ctx,+ behavior: 'always-hang',+ maxAttempts: 2,+ extraEnv: { BUN_INSTALL_ATTEMPT_TIMEOUT: '3', BUN_INSTALL_KILL_GRACE: '1' },+ timeoutMs: 20000,+ });++ expect(result.signal).toBeNull();+ expect(result.status).toBe(124);+ const out = combinedOutput(result);+ expect(countMatches(out, '::warning::')).toBe(1);+ expect(countMatches(out, '::error::')).toBe(1);+ expect(out).toContain('timed out after 3s');+ }, 30000); // 2 attempts x 3s budget exceeds Bun's 5s default per-test timeout++ test('SIGKILL escalation: a SIGTERM-ignoring hang is force-killed and normalized to exit 124', () => {+ const ctx = setupStub();+ const result = runWrapper({+ ...ctx,+ behavior: 'always-hang-ignore-term',+ maxAttempts: 1,+ extraEnv: {+ BUN_INSTALL_ATTEMPT_TIMEOUT: '3',+ BUN_INSTALL_KILL_GRACE: '1',+ },+ timeoutMs: 20000,+ });++ expect(result.signal).toBeNull();+ expect(result.status).toBe(124);+ expect(countMatches(combinedOutput(result), '::error::')).toBe(1);+ }, 30000); // 3s budget + 1s grace + SIGKILL can edge past Bun's 5s default++ test('timeout disabled (BUN_INSTALL_ATTEMPT_TIMEOUT=0): direct path, happy install exits 0 with no annotations', () => {+ const ctx = setupStub();+ const result = runWrapper({+ ...ctx,+ behavior: 'always-pass',+ extraEnv: { BUN_INSTALL_ATTEMPT_TIMEOUT: '0' },+ });++ expect(result.status).toBe(0);+ const out = combinedOutput(result);+ expect(out).not.toContain('::warning::');+ expect(out).not.toContain('::error::');+ });++ test('timeout disabled + failure: direct path still retries and recovers', () => {+ const ctx = setupStub();+ const result = runWrapper({+ ...ctx,+ behavior: 'fail-then-pass',+ maxAttempts: 2,+ extraEnv: { BUN_INSTALL_ATTEMPT_TIMEOUT: '0' },+ });++ expect(result.status).toBe(0);+ const out = combinedOutput(result);+ expect(countMatches(out, '::warning::')).toBe(1);+ expect(out).toContain('failed (exit 1)');+ expect(out).not.toContain('timed out');+ expect(out).not.toContain('::error::');+ });++ test('input validation: non-integer BUN_INSTALL_ATTEMPT_TIMEOUT exits 64', () => {+ const ctx = setupStub();+ const result = runWrapper({+ ...ctx,+ behavior: 'always-pass',+ extraEnv: { BUN_INSTALL_ATTEMPT_TIMEOUT: 'soon' },+ timeoutMs: 2000,+ });++ expect(result.signal).toBeNull();+ expect(result.status).toBe(64);+ const out = combinedOutput(result);+ expect(out).toContain('::error::');+ expect(out).toContain('BUN_INSTALL_ATTEMPT_TIMEOUT');+ });++ test('input validation: non-integer BUN_INSTALL_KILL_GRACE exits 64', () => {+ const ctx = setupStub();+ const result = runWrapper({+ ...ctx,+ behavior: 'always-pass',+ extraEnv: { BUN_INSTALL_KILL_GRACE: '1.5' },+ timeoutMs: 2000,+ });++ expect(result.signal).toBeNull();+ expect(result.status).toBe(64);+ const out = combinedOutput(result);+ expect(out).toContain('::error::');+ expect(out).toContain('BUN_INSTALL_KILL_GRACE');+ });});