Never stop coding. Free MIT AI gateway: one endpoint, 375 providers (150+ free), 1,400+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A, Desktop/PWA. Built by 900+ contributors
title: Self-Hosted Runner Box Operations
Self-Hosted Runner Box Operations (.113 pool)
The self-hosted pool runs on the .113 box (tomni-proxmox-113). Since 2026-10-18 each
OmniRoute runner has one role, one label, and a memory/CPU budget sized to what its workload
measured. The Turbopack slot gets what the build needs; the other slots are sized to their
measured peaks. The box only overcommits when a Turbopack build coincides with a Release-Green
webpack build and a lint job; host swap (16 GB) absorbs that, and because pull-request Builds run
hosted, the Turbopack slot is idle most of the time:
| runner | label | role | MemoryHigh / MemoryMax |
CPUQuota |
|---|---|---|---|---|
omniroute-113-5 |
omni-build |
Turbopack builds: ci.yml Build (push/dispatch only), Docker amd64, npm-publish, nightly-v4-build, preview-artifact, the nightly a11y build |
56 GB / 60 GB (swap 0) | 1600 % |
omniroute-113-6 |
omni-release |
Release-Green (release-green + main-green), webpack build |
20 GB / 24 GB (swap 4 GB) | 800 % |
omniroute-113-2 |
omni-light |
nightlies (Schemathesis, promptfoo, garak, k6) and quality.yml lint-guard |
12 GB / 16 GB (swap 4 GB) | 600 % |
omniroute-113 |
omni-light |
same pool as above | 8 GB / 12 GB | 400 % |
| OmniHeuris, OmniMind | (their repos) | Dependabot only (peak 155 MB each, 2026-10-18) | 4 GB / 6 GB | 200 % each |
Measured 2026-10-18:
| resource | value | what it means for scheduling |
|---|---|---|
| RAM / CPU | 78 GB / 32 cores | a Turbopack production build peaked at 49 GB uncapped (tests/unit/ci-build-bundler-webpack.test.ts); on Next 16.3.8 the compile phase alone holds ~40 GB (36 GB RSS in next-build plus 17 loader workers at 3 GB, measured 2026-10-18); a webpack build at ~14–16 GB. One Turbopack + one webpack build fit together; two Turbopack builds do not |
| swap | 16 GB | omniroute-113-5 runs with swap off (MemorySwapMax=0) so a runaway build is cut at its cap instead of swapping the box |
/tmp |
12 GB tmpfs = RAM | anything parked there is memory; leftovers are swept after 3 h |
| CPU | load 60 on 32 cores observed | before the quotas, one Turbopack build (~1166 % CPU) plus a coverage suite on the same box starved the other listeners' heartbeats (The self-hosted runner lost communication, #16853) |
| runners | 6 listeners | 4 OmniRoute + OmniHeuris + OmniMind; omniroute-113-3/-4/-7/-8 are disabled (systemctl enable --now brings one back) |
The limits live in each unit's /etc/systemd/system/actions.runner.<…>.service.d/override.conf
(previous values kept as override.conf.bak-20261018) and were applied live with
systemctl set-property --runtime … MemoryHigh=… MemoryMax=… CPUQuota=… — no restart, so no
running job was touched. Labels are changed through the runners API
(gh api -X DELETE repos/<repo>/actions/runners/<id>/labels/<label> /
gh api -X POST repos/<repo>/actions/runners/<id>/labels -f 'labels[]=<label>'), also live.
Install the janitor (one-time, on the box)
scp scripts/ops/runner-janitor.sh [email protected]:/opt/omniroute-ops/runner-janitor.sh
ssh [email protected] 'chmod +x /opt/omniroute-ops/runner-janitor.sh; apt-get install -y lsof'
# cron (root): every 30 min, log to /var/log/runner-janitor.log
*/30 * * * * MAX_ACTIVE_RUNNERS=6 /opt/omniroute-ops/runner-janitor.sh >> /var/log/runner-janitor.log 2>&1
lsof is required: the janitor proves a path is idle with one snapshot of open
files before removing it, and without the tool it removes nothing and says so
(exit 1). Try any change with --dry-run first — it prints exactly what it would
do and touches nothing.
What it does every run: sweeps our own leftovers (runner-*, omniroute-*,
next-build*, e2e-build.tar.gz) after 3 h on tmpfs and 24 h on disk
_work/_temp; kills a next-build older than 75 min (no job runs that long — on
2026-08-27 one ran 70 min after GitHub had declared its job lost); prunes 48 h-old
checkouts of runners whose unit is stopped; alerts on disk ≥ 85 %, memory PSI
full/avg60 ≥ 10 %, and more listeners than MAX_ACTIVE_RUNNERS (with an
omniroute/other breakdown). Exit 1 = attention needed; read the log.
The 75-minute rule does not catch a build whose job GitHub already gave up on. When the
Runner.Listener cannot renew its job, _diag/Runner_*.log shows
POST request to …/renewjob timed out after 100 seconds, the API reports the job completed
(failure) about ten minutes after the last successful renewal, and next-build keeps running with
no job attached (2026-10-18: the Build of #17066 was failed at 19:43 and its build was still
holding 40 GB five minutes later). Confirm the ref in the newest _diag/Worker_*.log, then
SIGTERM only the build tree by PID — next-build, npm run build, build-next-isolated.mjs and
the _work workers. Never touch Runner.Listener, RunnerService.js or Runner.Worker: the
worker reports the job as cancelled and the listener goes back online by itself.
Runner units: KillMode
The runner's default KillMode=process leaves Runner.Worker → npm → next-build
alive when a unit is stopped or restarted — an orphan build keeps eating RAM and
CPU with no job attached. Every OmniRoute unit carries a drop-in
(/etc/systemd/system/actions.runner.diegosouzapw-OmniRoute.<name>.service.d/10-killmode.conf)
with KillMode=mixed: SIGTERM to the listener first, SIGKILL to the whole cgroup at
TimeoutStop. It takes effect on the unit's next restart — restart one runner at
a time, only when idle, with the idle check and the restart in the same command.
Operating rules
- One Turbopack build at a time — enforced by label. Every job that runs a full Turbopack
next buildtargets[self-hosted, omni-build], and onlyomniroute-113-5carries that label:ci.ymlBuild,npm-publish.ymlpublish,nightly-v4-build,preview-artifact, the nightly a11y build (#16853) anddocker-publish.ymlamd64 (webpack inside BuildKit — Turbopack panicked there, run 33253576569). The arm64 Docker leg stays onubuntu-24.04-arm.docker-publishamd64 shares theheavy-build-mainconcurrency group withci.ymlBuild(cancel-in-progress: false). Docker Engine must be onomniroute-113-5. A second Turbopack slot does not fit 78 GB (2 × 33–37 GB plus the rest of the box); a second slot is only safe for webpack builds. - Pull-request Builds run hosted (2026-10-18).
ci.ymlBuildreachesomni-buildonly forpushandworkflow_dispatch; everypull_request(own-origin or fork) andmerge_groupBuild runs onubuntu-latestwith webpack. With every own-origin PR routed to the slot, a Turbopack Build held it 29–58 min, 18 Builds queued for up to 18 h, and the release line's own push Build got the slot once in 40 jobs. Hosted, the same PR Build takes 8–12 min next to the PR's other jobs. Turbopack, the bundlernpm-publishships, stays exercised on the release line by the nightly a11y build of the default branch (04:41 UTC), by every push or dispatch Build that reaches the slot, and bynpm-publishitself.tests/unit/build/heavy-build-queue.test.tspins the four-clause gate. - Never size the Turbopack slot below its measured need. On 2026-10-18 at 18:06 the slot was
lowered from 56G/60G to 40G/44G to make room for
omni-release. The compile phase sat at theMemoryHighfor its whole run (memory PSIfull30–43 %), and theRunner.Listener, which shares the cgroup, could not renew its job. The owner's 56G/60G of 2026-10-17 was restored the same day. Below that value the build does not fail; it thrashes, and the job is lost. - Release-Green has its own slot (2026-10-18). Until then both Release-Green validations ran
on
omni-build: on every push (~10/hour, 97 of 100 push runs cancelled mid-work in one day) and three ~2 h full sweeps a day (the 05:29 sweep waited 6 h for the slot) — the single build slot was oversubscribed several times over and own-origin PR Builds waited for hours.release-greenandmain-greennow targetomni-release(omniroute-113-6only) withOMNIROUTE_USE_TURBOPACK=0, and a started run is no longer cancelled by a newer push (the pending slot still collapses the storm).tests/unit/build/runner-capacity-routing.test.tspins the routing, the webpack switch and that nothing onomni-releasebuilds with Turbopack. - Light pool:
omni-light.omniroute-113andomniroute-113-2run the nightlies that need a backend-onlynext build(~5–6 GB; webpack) andquality.ymllint-guard.omniroute-113hadMemoryHigh=4Guntil 2026-10-18 — the promptfoo build reclaimed its way through 54 minutes there; 8 GB removes the throttling. - Fewer listeners on purpose. Four OmniRoute units were disabled on 2026-08-29 — each extra
listener is another tenant for the memory budget above. The janitor ceiling is 6 (
MAX_ACTIVE_RUNNERS=6in cron): it counts everyRunner.Listeneron the box, and OmniHeuris + OmniMind add two to our four. - Never clean
/tmpor_workby hand while any runner is busy. A check-then-delete with a gap between the two is how a live Build job lost its_workon 2026-08-27. The janitor does the check and the removal in one step; let it. - Stopping a runner mid-job cancels the job (observed live):
systemctl stoponly when its listener has noRunner.Workerchild — and do it in one command. - Workflows must not park artefacts in
/tmp(it is RAM). Download to$RUNNER_TEMP(on disk, per runner) — the 1.3 GBnext-buildartefact took 27–32 minutes to land on the tmpfs and 2 minutes to upload from disk. - The
.15VPS is homologation-only — never runs CI runners.
Hosted lane (GitHub-hosted runners)
The account is on the Pro plan: 40 concurrent hosted jobs, shared by every PR check,
every push-triggered workflow and CodeQL's default setup. Measured on 2026-10-18 during a
merge campaign (~10 merges/hour on release/v3.8.52): 2,500 workflow runs created in
24 h, 44 pushes to the branch in 6 h, 106 hosted jobs queued while PR checks waited for
hours and PRs were merged without them. Nine workflows fired on every push to the branch:
per push to release/v* |
hosted jobs | outcome under the storm | since 2026-10-18 |
|---|---|---|---|
ci.yml (full matrix) |
up to 30 | cancelled by the next push, 24 of 24 in a day | not started on release/** pushes (PR run + Release-Green) |
quality.yml |
~6 | cancelled, 23 of 23 | not started on release/** pushes |
Release-Green --quick |
0 | ran on omni-build, cancelled 97 of 100 |
omni-release slot, never cancelled, pending run collapses |
Lock released branch, OpenSSF Scorecard |
2 | one queued run per push | one run per ref (tests/unit/build/hosted-lane-concurrency.test.ts) |
Release acceptance (shadow report) |
1 | one hosted npm ci per push, never cancelled |
unchanged — candidate for a paths: filter |
Publish to Docker Hub → Deploy to VPS |
2 | cancelled per push; amd64 leg competes on omni-build |
unchanged — owner's deploy channel |
CodeQL default setup (dynamic) |
4 | javascript-typescript, actions, c-cpp, python per push |
unchanged — Settings → Code security; c-cpp/python scan nothing here |
tests/unit/build/push-storm-diet.test.ts pins the main-only push trigger of ci.yml and
quality.yml. Side effect: Codecov no longer receives a base upload per release push (the
step is informational and continue-on-error); the in-repo coverage gate and PR comment are
unaffected.
More self-hosted capacity does not fix the hosted queue. The devbox
(devbox-vm-06-dev002, 24 vCPU / 54 GB) could host a runner, but the agent sessions already
keep its load at 20–26, the agent container has no Docker daemon, and the saturated lane is
the hosted one. The .113 box had 71 GB free and load 5–8 at the same moment; its constraint
is the per-listener memory budget above (ceiling 6 listeners), not CPU. The remaining levers
are process ones: land campaign PRs through the Mergify queue (label=queue, batches share
one merge_group CI run instead of one run per PR plus one per push), trim the CodeQL
default-setup languages, and filter release-acceptance.yml by the files it reads.