Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Big pull requests no longer break the stress check, and three persistence validation jobs run on smaller machines (GST-3002) (#9677)
Garry Tanaca965d45a50
diff --git a/.github/workflows/persistence-validation.yml b/.github/workflows/persistence-validation.ymlindex 682c0431f3..5d84f91f97 100644--- a/.github/workflows/persistence-validation.yml+++ b/.github/workflows/persistence-validation.yml@@ -74,7 +74,7 @@ jobs:deployment-matrix:name: PgBouncer, RLS and pool capacity / Bun ${{ matrix.bun }}if: ${{ github.event_name != 'pull_request' }}- runs-on: ubicloud-standard-8-ubuntu-2404+ runs-on: ubicloud-standard-4-ubuntu-2404timeout-minutes: 15strategy:fail-fast: false@@ -300,8 +300,9 @@ jobs:name: ${{ matrix.engine }} / Bun ${{ matrix.bun }}if: ${{ github.event_name != 'pull_request' }}# The soak uses 1.3-1.8 busy cores: the 2,500-write PGLite soak took 523s on- # 4 vCPUs, 539s on 8 and 500s on 16.- runs-on: ubicloud-standard-4-ubuntu-2404+ # 4 vCPUs, 539s on 8 and 500s on 16. On 2 vCPUs (GST-3002, three paired runs)+ # the pglite arm took a 7.0 minute median and postgres 10.0, under convergence.+ runs-on: ubicloud-standard-2-ubuntu-2404timeout-minutes: 110strategy:fail-fast: false@@ -461,7 +462,7 @@ jobs:reconciliation:name: Reconciliation crashes / ${{ matrix.engine }} / Bun ${{ matrix.bun }}if: ${{ github.event_name != 'pull_request' }}- runs-on: ubicloud-standard-4-ubuntu-2404+ runs-on: ubicloud-standard-2-ubuntu-2404timeout-minutes: 30strategy:fail-fast: falsediff --git a/.github/workflows/stress.yml b/.github/workflows/stress.ymlindex edbbabe127..c904b47a74 100644--- a/.github/workflows/stress.yml+++ b/.github/workflows/stress.yml@@ -93,7 +93,8 @@ jobs:needs: planif: needs.plan.outputs.run == 'true'runs-on: ubicloud-standard-4-ubuntu-2404- # Gate shards are planned under 12 minutes and race-hunt shards under 55;+ # Gate shards are planned under 12 minutes (25 when a 12-minute plan would pass+ # GitHub's 256-job matrix limit) and race-hunt shards under 55;# the margin covers checkout, install and the snapshot build.timeout-minutes: ${{ fromJSON(inputs.mode == 'gate' && '35' || '65') }}strategy:diff --git a/changelog.d/ci-trial-followup.md b/changelog.d/ci-trial-followup.mdnew file mode 100644index 0000000000..8105e71af7--- /dev/null+++ b/changelog.d/ci-trial-followup.md@@ -0,0 +1 @@+**Big pull requests no longer break the stress gate (it re-plans at 25 minutes past GitHub's 256-job matrix limit), an untimed `.slow.test.ts` file is planned at a fixed 120 s, and three persistence-validation lanes moved to smaller runners.**diff --git a/docs/TESTING.md b/docs/TESTING.mdindex 40fe377f1b..6ddbfe5746 100644--- a/docs/TESTING.md+++ b/docs/TESTING.md@@ -49,12 +49,12 @@ not grant access. No Ubicloud API token is passed to workflow jobs.| Workload | Runner | Capacity || --- | --- | --- |-| Unit shards, slow and eval jobs, BrainBench, admin browser, shared-skills compatibility, persistence soak, reconciliation crashes and read latency, native Linux cells, OpenClaw startup, JSONB parity, PR serial pool, E2E backend matrix, Tier 2, coverage reports and Semgrep | `ubicloud-standard-4-ubuntu-2404` | 4 vCPU, 16 GB RAM |-| Nightly coverage serial pool, `verify`, PgBouncer/RLS deployment matrix, E2E graduation shards | `ubicloud-standard-8-ubuntu-2404` | 8 vCPU, 32 GB RAM |+| PgBouncer/RLS deployment matrix, unit shards, slow and eval jobs, BrainBench, admin browser, shared-skills compatibility, read latency, native Linux cells, OpenClaw startup, JSONB parity, PR serial pool, E2E backend matrix, Tier 2, coverage reports and Semgrep | `ubicloud-standard-4-ubuntu-2404` | 4 vCPU, 16 GB RAM |+| Nightly coverage serial pool, `verify`, E2E graduation shards | `ubicloud-standard-8-ubuntu-2404` | 8 vCPU, 32 GB RAM || E2E Tier 1 (its CLI `init` spawns exceed their timeouts on 4 vCPUs), label-gated and nightly heavy-tests jobs | `ubicloud-standard-16-ubuntu-2404` | 16 vCPU, 64 GB RAM || Label-gated heavy test suite | `ubicloud-standard-30-ubuntu-2404` | 30 vCPU, 120 GB RAM || Native ARM64 glibc and musl tests | `ubicloud-standard-4-arm-ubuntu-2404` | 4 vCPU |-| Selected E2E (one Bun process each), planning, status aggregation, dependency audit, gitleaks, security regressions and actionlint | `ubicloud-standard-2-ubuntu-2404` | 2 vCPU, 8 GB RAM |+| Selected E2E (one Bun process each), persistence soak, reconciliation crashes, planning, status aggregation, dependency audit, gitleaks, security regressions and actionlint | `ubicloud-standard-2-ubuntu-2404` | 2 vCPU, 8 GB RAM |macOS and Windows matrices stay on GitHub-hosted runners. Release building andpublishing also stay unchanged. The pinned upstream OSV reusable workflow does@@ -69,7 +69,11 @@ rose from 542s to 608s and made Test the critical path. On matchedVMs a unit shard takes the same time on 4, 8 and 16 vCPUs; the serial pool and`verify` take the same time on 8 and 16 vCPUs (serial shard 2: 222s and 224s;277s on 4); the 2,500-write PGLite soak averages 1.3-1.8 busy cores and takes-523s on 4 vCPUs and 500s on 16. Selected E2E workers run one file at a time and+523s on 4 vCPUs and 500s on 16. Persistence-validation's soak, reconciliation+and PgBouncer/RLS lanes moved down one size after three paired runs (GST-3002):+the soak's peak memory stayed under 25% of 8 GB and its slowest arm (postgres)+went from a 5.2 to a 10.0 minute median, still shorter than the run's longest+job, Managed-sync convergence at about 17 minutes. Selected E2E workers run one file at a time andfit on 2 vCPUs. Re-measure (`ubi-runner.sh run -s standard-N`) before resizing arunner: more CPU does not shorten a single-process job; critical pathcomes first, vCPU-minutes second.diff --git a/scripts/stress/README.md b/scripts/stress/README.mdindex 34dad873d9..d974ed8519 100644--- a/scripts/stress/README.md+++ b/scripts/stress/README.md@@ -106,13 +106,16 @@ shards.file's estimate is its measured stress iteration(`scripts/stress/iteration-weights.json`, mined from stress shard job logswith `bun scripts/stress/gate.ts mine-iterations <log>…`, which merges into- the map), else its lane weight, else the lane's p95 (the race hunt uses p75):+ the map), else its lane weight, else the lane's p95 (the race hunt uses p75;+ an unmeasured `*.slow.test.ts` file is planned at a fixed 120 s):a fresh database per iteration can make a stress iteration slower than thefile's lane run, and a new file has no history. At p75, a new Postgres-armfile planned at 41 s an iteration ran 145 s and took its shard past the35-minute job timeout (Test run 44139096116).- There is no shard-count cap; `max-parallel: 12` keeps a large PR- from starving other jobs. Shard jobs record results and exit 0 on test+ GitHub refuses a job matrix over 256 entries, so a plan past 256 shards is+ re-planned at the old 25-minute budget, and a plan still past 256 fails the+ plan job with a named error; `max-parallel: 12` keeps a large PR from+ starving other jobs. Shard jobs record results and exit 0 on testfailures; the result job decides, so a planned iteration without a receiptfails the check as incomplete.- **Failure.** The result names the file, the failing iteration and thediff --git a/scripts/stress/gate.ts b/scripts/stress/gate.tsindex e4e2315ad2..d04db87336 100644--- a/scripts/stress/gate.ts+++ b/scripts/stress/gate.ts@@ -35,6 +35,9 @@ import {const ROOT = resolve(import.meta.dir, '..', '..');export const GATE_SHARD_BUDGET_MS = 12 * 60_000;export const GATE_UNMEASURED_QUANTILE = 0.95;+/** GitHub refuses a job matrix over 256 entries; past it the gate re-plans at the old 25-minute budget. */+export const GATE_MAX_SHARDS = 256;+export const GATE_FALLBACK_SHARD_BUDGET_MS = 25 * 60_000;export const RACE_SHARD_BUDGET_MS = 55 * 60_000;export const RACE_BASE_SHARDS = 6;export const RACE_MAX_SHARDS = 12;@@ -117,7 +120,9 @@ export function planGate(root: string, input: GatePlanInput, weights = loadWeigh}files = withWrappers(root, files);const { stress, notStressed } = classify(root, files, weights, GATE_UNMEASURED_QUANTILE);- const shards = planShards(stress, DEFAULT_ITERATIONS, GATE_SHARD_BUDGET_MS);+ let shards = planShards(stress, DEFAULT_ITERATIONS, GATE_SHARD_BUDGET_MS);+ if (shards.length > GATE_MAX_SHARDS) shards = planShards(stress, DEFAULT_ITERATIONS, GATE_FALLBACK_SHARD_BUDGET_MS);+ if (shards.length > GATE_MAX_SHARDS) return { error: `the plan needs ${shards.length} shards of up to ${GATE_FALLBACK_SHARD_BUDGET_MS / 60_000} minutes, past GitHub's ${GATE_MAX_SHARDS}-job matrix limit. Next: split the PR, or dispatch the stress workflow with stress_files naming a subset` };return {...emptyPlan('gate', stress.length ? `${stress.length} file(s) × ${DEFAULT_ITERATIONS} iterations` : 'no changed test files to stress'),run: stress.length > 0, diagnostic, base, head, helpers, notStressed, shards,@@ -436,7 +441,7 @@ async function main(argv: string[]): Promise<number> {if (mode === 'gate') {const result = planGate(ROOT, { event: env.EVENT ?? '', baseSha: env.BASE_SHA, headSha: env.HEAD_SHA, stressFiles: env.STRESS_FILES, stressBase: env.STRESS_BASE, stressHead: env.STRESS_HEAD });if ('error' in result) {- console.error(`::error::stress gate: the changed-file list could not be computed: ${result.error}. Why: the gate stresses every test file the diff touches, so a guessed list could skip one. Docs: ${STRESS_DOCS}`);+ console.error(`::error::stress gate: the plan could not be computed: ${result.error}. Why: the gate stresses every test file the diff touches, so a guessed list could skip one. Docs: ${STRESS_DOCS}`);return 1;}plan = result;diff --git a/scripts/structural-suites.tsv b/scripts/structural-suites.tsvindex 0b2f6ddb32..31192325b2 100644--- a/scripts/structural-suites.tsv+++ b/scripts/structural-suites.tsv@@ -381,7 +381,7 @@ test/scripts/run-verify-parallel.test.ts run-verify-parallel.sh — no-timeout-btest/scripts/serial-files.test.ts scripts/serial-files.tsv (serial lane manifest) 5 readFileSynctest/scripts/stable-daily.test.ts stable-daily.yml 2 readFileSynctest/scripts/stress-gate.test.ts docs anchors 1 readFileSync-test/scripts/stress-gate.test.ts gate plan 6 readFileSync+test/scripts/stress-gate.test.ts gate plan 8 readFileSynctest/scripts/stress-plan.test.ts execution profiles 3 readFileSynctest/scripts/stress-plan.test.ts shards 5 readFileSynctest/scripts/test-env-opt-ins-guard.test.ts check-test-env-opt-ins 5 readFileSyncdiff --git a/test/scripts/ci-runner-routing.test.ts b/test/scripts/ci-runner-routing.test.tsindex b837480ba4..7b3ca1c34e 100644--- a/test/scripts/ci-runner-routing.test.ts+++ b/test/scripts/ci-runner-routing.test.ts@@ -34,7 +34,7 @@ describe('CI runner routing', () => {expect(load('osv-scanner.yml').jobs['osv-scan']['runs-on']).toBeUndefined();});- test('single-process lanes and soaks use 4 vCPUs, pooled lanes 8, and the label-gated heavy suite keeps 30', () => {+ test('single-process lanes use 4 vCPUs, persistence soak and reconciliation 2, pooled lanes 8, and the label-gated heavy suite keeps 30', () => {// Measured: a unit shard is one bun process (1.1-1.5 busy cores on average)// and took the same time on 4, 8 and 16 vCPUs; verify matches 16 vCPUs at 8.for (const name of ['slow-brainbench-e2e', 'brainbench', 'slow-entity-resolve-perf', 'admin-browser', 'shared-skills-compatibility']) {@@ -51,11 +51,13 @@ describe('CI runner routing', () => {// Graduation files run one at a time: 712-958 s whole on 8 vCPUs vs 751-770 s on 16 (GBRA-135 pilot).expect(load('e2e.yml').jobs['tier1-graduation']['runs-on']).toBe(pooled);expect(load('persistence-validation.yml').jobs['read-performance']['runs-on']).toBe(single);- expect(load('persistence-validation.yml').jobs.invariants['runs-on']).toBe(single);- expect(load('persistence-validation.yml').jobs.reconciliation['runs-on']).toBe(single);+ // Three paired runs (GST-3002): on 2 vCPUs the soak and reconciliation lanes peaked under 25% of+ // 8 GB and stayed shorter than convergence; PgBouncer/RLS on 4 vCPUs peaked at 18% of 16 GB.+ expect(load('persistence-validation.yml').jobs.invariants['runs-on']).toBe(small);+ expect(load('persistence-validation.yml').jobs.reconciliation['runs-on']).toBe(small);expect(load('persistence-validation.yml').jobs.convergence['runs-on']).toBe(single);expect(load('convergence-nightly.yml').jobs.convergence['runs-on']).toBe(single);- expect(load('persistence-validation.yml').jobs['deployment-matrix']['runs-on']).toBe(pooled);+ expect(load('persistence-validation.yml').jobs['deployment-matrix']['runs-on']).toBe(single);expect(load('heavy-tests.yml').jobs.heavy['runs-on']).toBe(heavy);});diff --git a/test/scripts/stress-gate.test.ts b/test/scripts/stress-gate.test.tsindex 35022221c3..7ae60bf838 100644--- a/test/scripts/stress-gate.test.ts+++ b/test/scripts/stress-gate.test.ts@@ -68,6 +68,27 @@ describe('gate plan', () => {expect(plan.shards.flatMap(s => s.items).reduce((n, i) => n + i.count, 0)).toBe(10);});+ test('a plan past 256 shards at 12 minutes is re-planned at 25 minutes so GitHub accepts the matrix', () => {+ const { root, base } = repo({ 'test/a.test.ts': "test('a', () => {});\n" });+ const files = Array.from({ length: 130 }, (_, i) => `test/n${i}.test.ts`);+ const head = commit(root, Object.fromEntries(files.map(f => [f, "test('n', () => {});\n"])));+ const unit = new Map(files.map(f => [f, 100_000] as [string, number]));+ const plan = isPlan(planGate(root, { event: 'pull_request', baseSha: base, headSha: head }, { ...weights(), unit }));+ expect(plan.shards.length).toBeLessThanOrEqual(256);+ expect(plan.shards.some(s => s.estimateMs > 12 * 60_000)).toBe(true);+ expect(plan.shards.every(s => s.estimateMs <= 25 * 60_000)).toBe(true);+ expect(plan.shards.flatMap(s => s.items).reduce((n, i) => n + i.count, 0)).toBe(1300);+ });++ test('a plan still past 256 shards at 25 minutes fails with a named error instead of a matrix GitHub refuses', () => {+ const { root, base } = repo({ 'test/a.test.ts': "test('a', () => {});\n" });+ const files = Array.from({ length: 257 }, (_, i) => `test/n${i}.test.ts`);+ const head = commit(root, Object.fromEntries(files.map(f => [f, "test('n', () => {});\n"])));+ const unit = new Map(files.map(f => [f, 100_000] as [string, number]));+ const result = planGate(root, { event: 'pull_request', baseSha: base, headSha: head }, { ...weights(), unit });+ expect(result).toMatchObject({ error: expect.stringContaining("GitHub's 256-job matrix limit") });+ });+test('a missing merge base fails the plan with the fetch next step', () => {const { root } = repo({ 'test/a.test.ts': '' });const result = planGate(root, { event: 'merge_group', baseSha: 'f'.repeat(40), headSha: 'HEAD' }, weights());