Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

ci: keep only short macOS legs in the merge queue (#541)

Ruben Lütke3c8afd190ea5
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 40af70f1b5..a96c650c27 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -39,10 +39,12 @@
# very same commit.
# pr pull request: the whole Linux + Windows suite and every affected
# lane on Linux and Windows. No macOS, no 18-minute real-runtime leg.
-# queue merge-queue group: everything `pr` runs, one macOS leg per affected
-# lane and the slow legs. The only blocking macOS proof per change.
+# queue merge-queue group: everything `pr` runs, the slow legs and one
+# SHORT macOS leg per affected lane. The blocking macOS proof per
+# change; up to four groups share the five macOS slots.
# full nightly, release tags, manual full dispatch: Intel macOS, the ARM
-# runners, every interpreter, the macOS test shards.
+# runners, every interpreter, the macOS test shards, and the real
+# runtimes and the real browser on macOS (5-10 minutes per leg).
# A push to main whose commit the merge queue already passed runs nothing
# but `detect` and the gate (`proven`); a push that bypassed the queue runs
# the `queue` tier. A branch gets macOS feedback before the queue with
@@ -1045,8 +1047,9 @@ jobs:
# user PATH and shell profiles stayed put (scripts/ci/agent_runtime_e2e.py).
# No key, no network to a model. The weekly canary does the same at
# upstream latest and raises the pin. A pull request runs both runtimes on
- # Linux and OpenClaw on Windows; the 18-minute Hermes install on Windows
- # and both macOS legs run in the merge queue (scripts/ci/plan_run.py).
+ # Linux and OpenClaw on Windows; the merge queue adds the 18-minute Hermes
+ # install on Windows; nightly and release runs add both macOS legs
+ # (scripts/ci/plan_run.py).
agent-runtimes-e2e:
needs: detect
if: needs.detect.outputs.agent_runtimes == 'true'
diff --git a/docs/agent-runtimes.md b/docs/agent-runtimes.md
index 24408a8120..f27109c9f7 100644
--- a/docs/agent-runtimes.md
+++ b/docs/agent-runtimes.md
@@ -404,13 +404,15 @@ each of them:
|---|---|---|
| `agent-runtimes-unit` | Linux and Windows on a pull request, macOS added in the merge queue; the portable base install | The runtime drivers, gateway, ACP runner, chat service, send queue, controls, society binding, roster, routines and society routes (`python scripts/ci/agent_runtime_suite.py`). Fakes only. |
| `agent-runtimes-headless` | `python:3.11-slim` | The runtime package imports and its tests pass on a server with no GPU, audio or desktop stack, and the app's Hermes setup without curl stops with "Setup needs curl". |
-| `agent-runtimes-e2e` | Pull request: Linux × Hermes, OpenClaw and Windows × OpenClaw. Merge queue: Linux, Windows, macOS × Hermes, OpenClaw | `scripts/ci/agent_runtime_e2e.py`: the app's own setup job (`manager.wait_ready`) installs the release pinned in `jarvis/agent_runtimes/runtime-versions.json` with a desktop app's PATH, never npm or a manual installer. Then two turns per model-gateway protocol against scripted providers (`--replay`), the fake-model driver check with turn-1 budgets (Hermes 30 s, OpenClaw 90 s), `pytest tests/unit/agent_runtimes`, and proof that the Windows user PATH gained no `agent_runtimes` entry and OpenClaw edited no shell profile. No key and no network to a model. |
-
-Every change has passed all three jobs on all three OSes before it lands:
-the merge queue is the blocking proof (`scripts/ci/plan_run.py` sizes the
-matrices). The Hermes install on Windows takes about 18 minutes and both
-macOS legs share five runner slots with every other pull request, so they
-run once per change, in the queue, not on every push to a branch. The
+| `agent-runtimes-e2e` | Pull request: Linux × Hermes, OpenClaw and Windows × OpenClaw. Merge queue: Linux, Windows × Hermes, OpenClaw. Nightly and release: macOS added | `scripts/ci/agent_runtime_e2e.py`: the app's own setup job (`manager.wait_ready`) installs the release pinned in `jarvis/agent_runtimes/runtime-versions.json` with a desktop app's PATH, never npm or a manual installer. Then two turns per model-gateway protocol against scripted providers (`--replay`), the fake-model driver check with turn-1 budgets (Hermes 30 s, OpenClaw 90 s), `pytest tests/unit/agent_runtimes`, and proof that the Windows user PATH gained no `agent_runtimes` entry and OpenClaw edited no shell profile. No key and no network to a model. |
+
+Before a change lands, the merge queue has run the unit suite on all three
+OSes and both real runtimes on Linux and Windows (`scripts/ci/plan_run.py`
+sizes the matrices). The Hermes install on Windows takes about 18 minutes,
+so it runs once per change, in the queue, not on every push to a branch. The
+real runtimes on macOS (10 and 7 minutes, on five runner slots shared with
+every merge group) run nightly, on release tags and on request
+(`gh workflow run ci.yml --ref <branch> -f full=false -f include_macos=true`). The
impact selection of `scripts/agent_land.py` adds the same suite whenever the
lane is on, so a runtime change also runs the chat and society tests, not
only the tests that import the changed module.
diff --git a/docs/ci-pipeline.md b/docs/ci-pipeline.md
index be19df3277..a28ee20d5b 100644
--- a/docs/ci-pipeline.md
+++ b/docs/ci-pipeline.md
@@ -25,8 +25,8 @@ to prove (`scripts/ci/plan_run.py` emits the tier and every job matrix):
| Tier | When | What runs | Target |
| --- | --- | --- | --- |
| `pr` | pull request, branch run dispatched by the merge train | Static gates, the whole Linux + Windows suite, every affected lane on Linux and Windows. No macOS, and not the 18-minute Hermes install on Windows. | ~10 min |
-| `queue` | merge-queue group; a push that reached main without the queue | Everything `pr` runs, plus one macOS leg per affected lane and the slow legs. The blocking proof of the exact tree main gets. | ~10 min, ~20 min when the agent-runtime lane is affected |
-| `full` | nightly, release tag, manual full dispatch | The wide matrix: Intel macOS, the ARM runners, every interpreter, the macOS test shards, all three macOS desktop rows. | ~30 min |
+| `queue` | merge-queue group; a push that reached main without the queue | Everything `pr` runs, the slow legs, and one short macOS leg per affected lane (realtime contract, updater, agent-runtime unit suite, native crypto on Intel, the Apple Silicon desktop bundle). The blocking proof of the exact tree main gets. | ~10 min, ~20 min when the agent-runtime lane is affected |
+| `full` | nightly, release tag, manual full dispatch | The wide matrix: Intel macOS, the ARM runners, every interpreter, the macOS test shards, all three macOS desktop rows, and the real runtimes and the real browser on macOS. | ~30 min |
| proven | push to main of a commit the merge queue already passed | `detect` and `CI gate` only. | under 1 min |
A queue group and the push that lands it are one commit, so the push run
@@ -34,7 +34,14 @@ asks the API for a green `merge_group` run of its own SHA and, when there is
one, turns every lane off. Any doubt (a direct push, an API error) fails open
to the `queue` tier. A nightly run never accepts an earlier proof.
-A branch gets macOS feedback before the queue with
+Up to four merge groups build at once on five macOS slots. The first queue
+runs of this design carried six to eight macOS legs each; three groups and
+one manual full run then waited more than half an hour on macOS alone. The
+two real-runtime legs (10 and 7 minutes) and the real-browser leg therefore
+run on macOS nightly and on release tags, and a group keeps only legs of one
+to four minutes.
+
+A branch gets the whole macOS matrix before the queue with
`gh workflow run ci.yml --ref <branch> -f full=false -f include_macos=true`.
Measured before the tiers (338 runs, 2026-10-07 to 2026-10-12): a run fanned
@@ -156,8 +163,8 @@ and manual runs have their own slots.
squash, `CI gate` required). A pull request's own CI is its admission ticket;
the queue then builds each entry on top of the newest main plus every entry
ahead of it, runs `ci.yml` on that exact tree (`merge_group` event: the
-`queue` tier, whole Linux + Windows suite plus one macOS leg per affected
-lane), and squash-merges only when `CI gate` is green there.
+`queue` tier, whole Linux + Windows suite plus one short macOS leg per
+affected lane), and squash-merges only when `CI gate` is green there.
Changes that pass alone and break together stop in the queue instead of on
main. A failed entry leaves the queue; the entries behind it are rebuilt
without it.
diff --git a/scripts/ci/plan_run.py b/scripts/ci/plan_run.py
index a784984a26..82f6682322 100644
--- a/scripts/ci/plan_run.py
+++ b/scripts/ci/plan_run.py
@@ -11,11 +11,14 @@ run has to prove, not what could be run:
Linux and Windows, no macOS, and without the slowest real-runtime
leg. It is the admission ticket to the merge queue.
``queue`` A merge-queue group, or a push that reached main without one. The
- blocking proof of the exact tree main gets: everything ``pr`` runs
- plus one macOS leg per affected lane and the slow legs.
+ blocking proof of the exact tree main gets: everything ``pr`` runs,
+ the slow legs, and one SHORT macOS leg per affected lane. Up to
+ four groups build at once on five macOS slots, so a group may
+ spend minutes there, not half an hour: the real-runtime and the
+ real-browser macOS legs (5 to 10 minutes each) stay in ``full``.
``full`` Nightly, a release tag, a manual full dispatch. The wide matrix:
Intel macOS, the ARM runners, every interpreter, the macOS test
- shards. Unchanged in scope; it only stopped running per change.
+ shards, the real runtimes and the real browser on macOS.
``macos`` (nightly, tags, a dispatch with ``include_macos``) selects the wide
matrix on any tier: ``gh workflow run ci.yml --ref <branch> -f full=false
@@ -137,14 +140,14 @@ def matrices(tier: str, *, macos: bool) -> dict[str, Any]:
]
if slow_legs:
e2e.append({"os": WINDOWS, "runtime": "hermes"})
- if mac_lanes:
+ # 10 and 7 minutes on macOS: with three groups in the queue these two
+ # legs alone held every macOS slot for half an hour. Nightly and release.
+ if macos:
e2e += [{"os": MACOS, "runtime": "hermes"}, {"os": MACOS, "runtime": "openclaw"}]
browser = list(base_os)
- if mac_lanes:
- browser.append(MACOS)
if macos:
- browser += [LINUX_ARM, WINDOWS_ARM, MACOS_INTEL]
+ browser += [MACOS, LINUX_ARM, WINDOWS_ARM, MACOS_INTEL]
return {
# jarvisctl, dragdrop and the installer smoke stay off macOS unless
diff --git a/tests/unit/ci/test_agent_runtime_lane.py b/tests/unit/ci/test_agent_runtime_lane.py
index 7ce3260841..eda03c3ab4 100644
--- a/tests/unit/ci/test_agent_runtime_lane.py
+++ b/tests/unit/ci/test_agent_runtime_lane.py
@@ -104,12 +104,16 @@ def test_ci_runs_the_lane_on_every_os_and_the_gate_waits_for_it():
jobs = yaml.safe_load((ROOT / ".github/workflows/ci.yml").read_text(encoding="utf-8"))["jobs"]
assert "agent_runtimes" in jobs["detect"]["outputs"]
assert LANE_JOBS <= set(jobs["gate"]["needs"])
- # Before anything lands (the merge queue) the lane has run on all three
- # OSes with both runtimes; ci.yml reads those matrices from the plan.
+ # Before anything lands (the merge queue) the unit suite has run on all
+ # three OSes and both real runtimes on Linux and Windows; the nightly and
+ # release runs add the real runtimes on macOS. ci.yml reads the plan.
oses = {"ubuntu-latest", "windows-latest", "macos-latest"}
queue = plan_run.plan("merge_group")
assert set(queue["lane_os"]) == oses
assert {(leg["os"], leg["runtime"]) for leg in queue["e2e_matrix"]} == {
+ (os, runtime) for os in oses - {"macos-latest"} for runtime in ("hermes", "openclaw")
+ }
+ assert {(leg["os"], leg["runtime"]) for leg in plan_run.plan("schedule")["e2e_matrix"]} == {
(os, runtime) for os in oses for runtime in ("hermes", "openclaw")
}
unit = jobs["agent-runtimes-unit"]["strategy"]["matrix"]
diff --git a/tests/unit/ci/test_pipeline_audit.py b/tests/unit/ci/test_pipeline_audit.py
index 13f35d28de..09f520be0a 100644
--- a/tests/unit/ci/test_pipeline_audit.py
+++ b/tests/unit/ci/test_pipeline_audit.py
@@ -255,13 +255,18 @@ def test_a_pull_request_never_takes_a_macos_runner():
assert {"os": "windows-latest", "python": "3.12"} in sized["updater_matrix"]
-def test_the_merge_queue_proves_one_macos_leg_per_lane_and_the_slow_legs():
+def test_the_merge_queue_proves_one_short_macos_leg_per_lane_and_the_slow_legs():
sized = plan_run.plan("merge_group")
assert sized["mac_lanes"] and not sized["macos"]
assert sized["lane_os"] == ["ubuntu-latest", "windows-latest", "macos-latest"]
- assert sized["browser_os"] == ["ubuntu-latest", "windows-latest", "macos-latest"]
assert sized["desktop_legs"] == "primary"
- assert {(leg["os"], leg["runtime"]) for leg in sized["e2e_matrix"]} == {
+ assert {"os": "windows-latest", "runtime": "hermes"} in sized["e2e_matrix"]
+ # Five macOS slots serve up to four merge groups at once: the long macOS
+ # legs (real runtimes, real browser) wait for the nightly and release runs.
+ assert sized["browser_os"] == ["ubuntu-latest", "windows-latest"]
+ assert "macos" not in json.dumps(sized["e2e_matrix"])
+ wide = plan_run.plan("schedule")
+ assert {(leg["os"], leg["runtime"]) for leg in wide["e2e_matrix"]} == {
(os, runtime)
for os in ("ubuntu-latest", "windows-latest", "macos-latest")
for runtime in ("hermes", "openclaw")