Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Big pull requests no longer break the stress check, and three persistence validation jobs run on smaller machines (GST-3002) (#9677)

Garry Tanaca965d45a50
diff --git a/.github/workflows/persistence-validation.yml b/.github/workflows/persistence-validation.yml
index 682c0431f3..5d84f91f97 100644
--- a/.github/workflows/persistence-validation.yml
+++ b/.github/workflows/persistence-validation.yml
@@ -74,7 +74,7 @@ jobs:
deployment-matrix:
name: PgBouncer, RLS and pool capacity / Bun ${{ matrix.bun }}
if: ${{ github.event_name != 'pull_request' }}
- runs-on: ubicloud-standard-8-ubuntu-2404
+ runs-on: ubicloud-standard-4-ubuntu-2404
timeout-minutes: 15
strategy:
fail-fast: false
@@ -300,8 +300,9 @@ jobs:
name: ${{ matrix.engine }} / Bun ${{ matrix.bun }}
if: ${{ github.event_name != 'pull_request' }}
# The soak uses 1.3-1.8 busy cores: the 2,500-write PGLite soak took 523s on
- # 4 vCPUs, 539s on 8 and 500s on 16.
- runs-on: ubicloud-standard-4-ubuntu-2404
+ # 4 vCPUs, 539s on 8 and 500s on 16. On 2 vCPUs (GST-3002, three paired runs)
+ # the pglite arm took a 7.0 minute median and postgres 10.0, under convergence.
+ runs-on: ubicloud-standard-2-ubuntu-2404
timeout-minutes: 110
strategy:
fail-fast: false
@@ -461,7 +462,7 @@ jobs:
reconciliation:
name: Reconciliation crashes / ${{ matrix.engine }} / Bun ${{ matrix.bun }}
if: ${{ github.event_name != 'pull_request' }}
- runs-on: ubicloud-standard-4-ubuntu-2404
+ runs-on: ubicloud-standard-2-ubuntu-2404
timeout-minutes: 30
strategy:
fail-fast: false
diff --git a/.github/workflows/stress.yml b/.github/workflows/stress.yml
index edbbabe127..c904b47a74 100644
--- a/.github/workflows/stress.yml
+++ b/.github/workflows/stress.yml
@@ -93,7 +93,8 @@ jobs:
needs: plan
if: needs.plan.outputs.run == 'true'
runs-on: ubicloud-standard-4-ubuntu-2404
- # Gate shards are planned under 12 minutes and race-hunt shards under 55;
+ # Gate shards are planned under 12 minutes (25 when a 12-minute plan would pass
+ # GitHub's 256-job matrix limit) and race-hunt shards under 55;
# the margin covers checkout, install and the snapshot build.
timeout-minutes: ${{ fromJSON(inputs.mode == 'gate' && '35' || '65') }}
strategy:
diff --git a/changelog.d/ci-trial-followup.md b/changelog.d/ci-trial-followup.md
new file mode 100644
index 0000000000..8105e71af7
--- /dev/null
+++ b/changelog.d/ci-trial-followup.md
@@ -0,0 +1 @@
+**Big pull requests no longer break the stress gate (it re-plans at 25 minutes past GitHub's 256-job matrix limit), an untimed `.slow.test.ts` file is planned at a fixed 120 s, and three persistence-validation lanes moved to smaller runners.**
diff --git a/docs/TESTING.md b/docs/TESTING.md
index 40fe377f1b..6ddbfe5746 100644
--- a/docs/TESTING.md
+++ b/docs/TESTING.md
@@ -49,12 +49,12 @@ not grant access. No Ubicloud API token is passed to workflow jobs.
| Workload | Runner | Capacity |
| --- | --- | --- |
-| Unit shards, slow and eval jobs, BrainBench, admin browser, shared-skills compatibility, persistence soak, reconciliation crashes and read latency, native Linux cells, OpenClaw startup, JSONB parity, PR serial pool, E2E backend matrix, Tier 2, coverage reports and Semgrep | `ubicloud-standard-4-ubuntu-2404` | 4 vCPU, 16 GB RAM |
-| Nightly coverage serial pool, `verify`, PgBouncer/RLS deployment matrix, E2E graduation shards | `ubicloud-standard-8-ubuntu-2404` | 8 vCPU, 32 GB RAM |
+| PgBouncer/RLS deployment matrix, unit shards, slow and eval jobs, BrainBench, admin browser, shared-skills compatibility, read latency, native Linux cells, OpenClaw startup, JSONB parity, PR serial pool, E2E backend matrix, Tier 2, coverage reports and Semgrep | `ubicloud-standard-4-ubuntu-2404` | 4 vCPU, 16 GB RAM |
+| Nightly coverage serial pool, `verify`, E2E graduation shards | `ubicloud-standard-8-ubuntu-2404` | 8 vCPU, 32 GB RAM |
| E2E Tier 1 (its CLI `init` spawns exceed their timeouts on 4 vCPUs), label-gated and nightly heavy-tests jobs | `ubicloud-standard-16-ubuntu-2404` | 16 vCPU, 64 GB RAM |
| Label-gated heavy test suite | `ubicloud-standard-30-ubuntu-2404` | 30 vCPU, 120 GB RAM |
| Native ARM64 glibc and musl tests | `ubicloud-standard-4-arm-ubuntu-2404` | 4 vCPU |
-| Selected E2E (one Bun process each), planning, status aggregation, dependency audit, gitleaks, security regressions and actionlint | `ubicloud-standard-2-ubuntu-2404` | 2 vCPU, 8 GB RAM |
+| Selected E2E (one Bun process each), persistence soak, reconciliation crashes, planning, status aggregation, dependency audit, gitleaks, security regressions and actionlint | `ubicloud-standard-2-ubuntu-2404` | 2 vCPU, 8 GB RAM |
macOS and Windows matrices stay on GitHub-hosted runners. Release building and
publishing also stay unchanged. The pinned upstream OSV reusable workflow does
@@ -69,7 +69,11 @@ rose from 542s to 608s and made Test the critical path. On matched
VMs a unit shard takes the same time on 4, 8 and 16 vCPUs; the serial pool and
`verify` take the same time on 8 and 16 vCPUs (serial shard 2: 222s and 224s;
277s on 4); the 2,500-write PGLite soak averages 1.3-1.8 busy cores and takes
-523s on 4 vCPUs and 500s on 16. Selected E2E workers run one file at a time and
+523s on 4 vCPUs and 500s on 16. Persistence-validation's soak, reconciliation
+and PgBouncer/RLS lanes moved down one size after three paired runs (GST-3002):
+the soak's peak memory stayed under 25% of 8 GB and its slowest arm (postgres)
+went from a 5.2 to a 10.0 minute median, still shorter than the run's longest
+job, Managed-sync convergence at about 17 minutes. Selected E2E workers run one file at a time and
fit on 2 vCPUs. Re-measure (`ubi-runner.sh run -s standard-N`) before resizing a
runner: more CPU does not shorten a single-process job; critical path
comes first, vCPU-minutes second.
diff --git a/scripts/stress/README.md b/scripts/stress/README.md
index 34dad873d9..d974ed8519 100644
--- a/scripts/stress/README.md
+++ b/scripts/stress/README.md
@@ -106,13 +106,16 @@ shards.
file's estimate is its measured stress iteration
(`scripts/stress/iteration-weights.json`, mined from stress shard job logs
with `bun scripts/stress/gate.ts mine-iterations <log>…`, which merges into
- the map), else its lane weight, else the lane's p95 (the race hunt uses p75):
+ the map), else its lane weight, else the lane's p95 (the race hunt uses p75;
+ an unmeasured `*.slow.test.ts` file is planned at a fixed 120 s):
a fresh database per iteration can make a stress iteration slower than the
file's lane run, and a new file has no history. At p75, a new Postgres-arm
file planned at 41 s an iteration ran 145 s and took its shard past the
35-minute job timeout (Test run 44139096116).
- There is no shard-count cap; `max-parallel: 12` keeps a large PR
- from starving other jobs. Shard jobs record results and exit 0 on test
+ GitHub refuses a job matrix over 256 entries, so a plan past 256 shards is
+ re-planned at the old 25-minute budget, and a plan still past 256 fails the
+ plan job with a named error; `max-parallel: 12` keeps a large PR from
+ starving other jobs. Shard jobs record results and exit 0 on test
failures; the result job decides, so a planned iteration without a receipt
fails the check as incomplete.
- **Failure.** The result names the file, the failing iteration and the
diff --git a/scripts/stress/gate.ts b/scripts/stress/gate.ts
index e4e2315ad2..d04db87336 100644
--- a/scripts/stress/gate.ts
+++ b/scripts/stress/gate.ts
@@ -35,6 +35,9 @@ import {
const ROOT = resolve(import.meta.dir, '..', '..');
export const GATE_SHARD_BUDGET_MS = 12 * 60_000;
export const GATE_UNMEASURED_QUANTILE = 0.95;
+/** GitHub refuses a job matrix over 256 entries; past it the gate re-plans at the old 25-minute budget. */
+export const GATE_MAX_SHARDS = 256;
+export const GATE_FALLBACK_SHARD_BUDGET_MS = 25 * 60_000;
export const RACE_SHARD_BUDGET_MS = 55 * 60_000;
export const RACE_BASE_SHARDS = 6;
export const RACE_MAX_SHARDS = 12;
@@ -117,7 +120,9 @@ export function planGate(root: string, input: GatePlanInput, weights = loadWeigh
}
files = withWrappers(root, files);
const { stress, notStressed } = classify(root, files, weights, GATE_UNMEASURED_QUANTILE);
- const shards = planShards(stress, DEFAULT_ITERATIONS, GATE_SHARD_BUDGET_MS);
+ let shards = planShards(stress, DEFAULT_ITERATIONS, GATE_SHARD_BUDGET_MS);
+ if (shards.length > GATE_MAX_SHARDS) shards = planShards(stress, DEFAULT_ITERATIONS, GATE_FALLBACK_SHARD_BUDGET_MS);
+ if (shards.length > GATE_MAX_SHARDS) return { error: `the plan needs ${shards.length} shards of up to ${GATE_FALLBACK_SHARD_BUDGET_MS / 60_000} minutes, past GitHub's ${GATE_MAX_SHARDS}-job matrix limit. Next: split the PR, or dispatch the stress workflow with stress_files naming a subset` };
return {
...emptyPlan('gate', stress.length ? `${stress.length} file(s) × ${DEFAULT_ITERATIONS} iterations` : 'no changed test files to stress'),
run: stress.length > 0, diagnostic, base, head, helpers, notStressed, shards,
@@ -436,7 +441,7 @@ async function main(argv: string[]): Promise<number> {
if (mode === 'gate') {
const result = planGate(ROOT, { event: env.EVENT ?? '', baseSha: env.BASE_SHA, headSha: env.HEAD_SHA, stressFiles: env.STRESS_FILES, stressBase: env.STRESS_BASE, stressHead: env.STRESS_HEAD });
if ('error' in result) {
- console.error(`::error::stress gate: the changed-file list could not be computed: ${result.error}. Why: the gate stresses every test file the diff touches, so a guessed list could skip one. Docs: ${STRESS_DOCS}`);
+ console.error(`::error::stress gate: the plan could not be computed: ${result.error}. Why: the gate stresses every test file the diff touches, so a guessed list could skip one. Docs: ${STRESS_DOCS}`);
return 1;
}
plan = result;
diff --git a/scripts/structural-suites.tsv b/scripts/structural-suites.tsv
index 0b2f6ddb32..31192325b2 100644
--- a/scripts/structural-suites.tsv
+++ b/scripts/structural-suites.tsv
@@ -381,7 +381,7 @@ test/scripts/run-verify-parallel.test.ts run-verify-parallel.sh — no-timeout-b
test/scripts/serial-files.test.ts scripts/serial-files.tsv (serial lane manifest) 5 readFileSync
test/scripts/stable-daily.test.ts stable-daily.yml 2 readFileSync
test/scripts/stress-gate.test.ts docs anchors 1 readFileSync
-test/scripts/stress-gate.test.ts gate plan 6 readFileSync
+test/scripts/stress-gate.test.ts gate plan 8 readFileSync
test/scripts/stress-plan.test.ts execution profiles 3 readFileSync
test/scripts/stress-plan.test.ts shards 5 readFileSync
test/scripts/test-env-opt-ins-guard.test.ts check-test-env-opt-ins 5 readFileSync
diff --git a/test/scripts/ci-runner-routing.test.ts b/test/scripts/ci-runner-routing.test.ts
index b837480ba4..7b3ca1c34e 100644
--- a/test/scripts/ci-runner-routing.test.ts
+++ b/test/scripts/ci-runner-routing.test.ts
@@ -34,7 +34,7 @@ describe('CI runner routing', () => {
expect(load('osv-scanner.yml').jobs['osv-scan']['runs-on']).toBeUndefined();
});
- test('single-process lanes and soaks use 4 vCPUs, pooled lanes 8, and the label-gated heavy suite keeps 30', () => {
+ test('single-process lanes use 4 vCPUs, persistence soak and reconciliation 2, pooled lanes 8, and the label-gated heavy suite keeps 30', () => {
// Measured: a unit shard is one bun process (1.1-1.5 busy cores on average)
// and took the same time on 4, 8 and 16 vCPUs; verify matches 16 vCPUs at 8.
for (const name of ['slow-brainbench-e2e', 'brainbench', 'slow-entity-resolve-perf', 'admin-browser', 'shared-skills-compatibility']) {
@@ -51,11 +51,13 @@ describe('CI runner routing', () => {
// Graduation files run one at a time: 712-958 s whole on 8 vCPUs vs 751-770 s on 16 (GBRA-135 pilot).
expect(load('e2e.yml').jobs['tier1-graduation']['runs-on']).toBe(pooled);
expect(load('persistence-validation.yml').jobs['read-performance']['runs-on']).toBe(single);
- expect(load('persistence-validation.yml').jobs.invariants['runs-on']).toBe(single);
- expect(load('persistence-validation.yml').jobs.reconciliation['runs-on']).toBe(single);
+ // Three paired runs (GST-3002): on 2 vCPUs the soak and reconciliation lanes peaked under 25% of
+ // 8 GB and stayed shorter than convergence; PgBouncer/RLS on 4 vCPUs peaked at 18% of 16 GB.
+ expect(load('persistence-validation.yml').jobs.invariants['runs-on']).toBe(small);
+ expect(load('persistence-validation.yml').jobs.reconciliation['runs-on']).toBe(small);
expect(load('persistence-validation.yml').jobs.convergence['runs-on']).toBe(single);
expect(load('convergence-nightly.yml').jobs.convergence['runs-on']).toBe(single);
- expect(load('persistence-validation.yml').jobs['deployment-matrix']['runs-on']).toBe(pooled);
+ expect(load('persistence-validation.yml').jobs['deployment-matrix']['runs-on']).toBe(single);
expect(load('heavy-tests.yml').jobs.heavy['runs-on']).toBe(heavy);
});
diff --git a/test/scripts/stress-gate.test.ts b/test/scripts/stress-gate.test.ts
index 35022221c3..7ae60bf838 100644
--- a/test/scripts/stress-gate.test.ts
+++ b/test/scripts/stress-gate.test.ts
@@ -68,6 +68,27 @@ describe('gate plan', () => {
expect(plan.shards.flatMap(s => s.items).reduce((n, i) => n + i.count, 0)).toBe(10);
});
+ test('a plan past 256 shards at 12 minutes is re-planned at 25 minutes so GitHub accepts the matrix', () => {
+ const { root, base } = repo({ 'test/a.test.ts': "test('a', () => {});\n" });
+ const files = Array.from({ length: 130 }, (_, i) => `test/n${i}.test.ts`);
+ const head = commit(root, Object.fromEntries(files.map(f => [f, "test('n', () => {});\n"])));
+ const unit = new Map(files.map(f => [f, 100_000] as [string, number]));
+ const plan = isPlan(planGate(root, { event: 'pull_request', baseSha: base, headSha: head }, { ...weights(), unit }));
+ expect(plan.shards.length).toBeLessThanOrEqual(256);
+ expect(plan.shards.some(s => s.estimateMs > 12 * 60_000)).toBe(true);
+ expect(plan.shards.every(s => s.estimateMs <= 25 * 60_000)).toBe(true);
+ expect(plan.shards.flatMap(s => s.items).reduce((n, i) => n + i.count, 0)).toBe(1300);
+ });
+
+ test('a plan still past 256 shards at 25 minutes fails with a named error instead of a matrix GitHub refuses', () => {
+ const { root, base } = repo({ 'test/a.test.ts': "test('a', () => {});\n" });
+ const files = Array.from({ length: 257 }, (_, i) => `test/n${i}.test.ts`);
+ const head = commit(root, Object.fromEntries(files.map(f => [f, "test('n', () => {});\n"])));
+ const unit = new Map(files.map(f => [f, 100_000] as [string, number]));
+ const result = planGate(root, { event: 'pull_request', baseSha: base, headSha: head }, { ...weights(), unit });
+ expect(result).toMatchObject({ error: expect.stringContaining("GitHub's 256-job matrix limit") });
+ });
+
test('a missing merge base fails the plan with the fetch next step', () => {
const { root } = repo({ 'test/a.test.ts': '' });
const result = planGate(root, { event: 'merge_group', baseSha: 'f'.repeat(40), headSha: 'HEAD' }, weights());