Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

ci: run the macOS tests a change can reach in the merge queue (#605)

Ruben Lütkeb5b172a05577
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a928336f3a..2e3aab0060 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -144,6 +144,8 @@ jobs:
browser_os: ${{ steps.mode.outputs.browser_os }}
desktop_legs: ${{ steps.mode.outputs.desktop_legs }}
matrix_parallel: ${{ steps.mode.outputs.matrix_parallel }}
+ mac_tests: ${{ steps.mactests.outputs.mode != 'none' }}
+ mac_test_shards: ${{ steps.mactests.outputs.shards == '[]' && '[1]' || steps.mactests.outputs.shards }}
duration_snapshot: ${{ steps.snapshot.outputs.name }}
release: ${{ github.ref_type == 'tag' }}
steps:
@@ -221,6 +223,24 @@ jobs:
git diff --name-only --no-renames "$BASE" HEAD | tee changed.txt | python3 scripts/ci/classify_changes.py
fi
+ # A merge group runs the macOS tests its change can reach: the whole
+ # suite on macOS is 27 runner-minutes on five shared slots. No diff
+ # (changed.txt missing or empty) selects everything.
+ - name: Select the macOS tests this change can reach
+ id: mactests
+ shell: bash
+ env:
+ PYTHON_LANE: ${{ steps.lanes.outputs.python }}
+ MAC_IMPACT: ${{ steps.mode.outputs.mac_impact }}
+ run: |
+ : > mac_tests.txt
+ if [ "$PYTHON_LANE" = "true" ] && [ "$MAC_IMPACT" = "true" ]; then
+ test -f changed.txt || : > changed.txt
+ python3 scripts/ci/select_tests.py --changed-from changed.txt --out mac_tests.txt
+ else
+ { echo "mode=none"; echo "shards=[]"; } | tee -a "$GITHUB_OUTPUT"
+ fi
+
# Resolve the mutable cache ONCE. All shards download the same immutable
# snapshot; staggered runners must never compute different partitions.
- name: Restore duration history once
@@ -257,6 +277,7 @@ jobs:
path: |
test_durations.json
test_durations_windows.json
+ mac_tests.txt
if-no-files-found: error
retention-days: 7
@@ -601,6 +622,57 @@ jobs:
path: out/
retention-days: 14
+ # The macOS tests the change can reach, before it lands (merge queue). One
+ # runner for an ordinary change; a change that selects everything (the
+ # pipeline, conftest, the lockfile) gets the full three-way split. The
+ # nightly and release runs keep the whole suite in `tests-macos`.
+ tests-macos-impact:
+ needs: detect
+ if: needs.detect.outputs.python == 'true' && needs.detect.outputs.mac_tests == 'true'
+ strategy:
+ fail-fast: false
+ matrix:
+ shard: ${{ fromJSON(needs.detect.outputs.mac_test_shards) }}
+ runs-on: macos-latest
+ timeout-minutes: 40
+ steps:
+ - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+ with:
+ persist-credentials: false
+ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+ with:
+ python-version: "3.11"
+ - name: Install
+ uses: ./.github/actions/install-locked
+ with:
+ extras: dev telephony
+ - name: Download the shared duration snapshot
+ uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
+ with:
+ name: ${{ needs.detect.outputs.duration_snapshot }}
+ - name: Run the selected tests
+ continue-on-error: true
+ env:
+ SHARD: ${{ matrix.shard }}
+ SHARDS: ${{ needs.detect.outputs.mac_test_shards }}
+ run: |
+ total=$(python -c "import json,os; print(len(json.loads(os.environ['SHARDS'])))")
+ python scripts/ci/run_tests_parallel.py tests --files-from mac_tests.txt \
+ --shard "$SHARD/$total" \
+ --report "out/test-report-macos-impact-$SHARD.json" \
+ --junit-out "out/junit-macos-impact-$SHARD.xml"
+ - name: Ratchet (a failure not in the baseline blocks)
+ env:
+ SHARD: ${{ matrix.shard }}
+ run: python scripts/ci/ratchet_tests.py check --baseline scripts/ci/test-baseline-macos.json "out/test-report-macos-impact-$SHARD.json"
+ - name: Upload shard report
+ if: always()
+ uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
+ with:
+ name: tests-macos-impact-${{ matrix.shard }}
+ path: out/
+ retention-days: 14
+
test-report:
name: test report + floor
needs: [detect, tests-linux]
@@ -1183,6 +1255,7 @@ jobs:
- tests-linux
- tests-windows
- tests-macos
+ - tests-macos-impact
- test-report
- duration-cache
- frontend
diff --git a/docs/ci-pipeline.md b/docs/ci-pipeline.md
index 7c44a890ab..bb38c6179d 100644
--- a/docs/ci-pipeline.md
+++ b/docs/ci-pipeline.md
@@ -54,7 +54,10 @@ well (one morning, 2026-10-13), pull requests waited 6 to 25 minutes for a
slot. The queue builds two groups at a time, so at most sixteen macOS legs
compete for the five slots; in the first such group all eight were done 14
minutes in. The whole Python suite on macOS (three shards of nine minutes)
-still runs nightly and on release tags only.
+runs nightly and on release tags; a merge group runs the macOS tests its
+change can reach instead (`tests macos impact`), so a test that the change
+breaks on macOS stops it in the queue. A macOS failure in a test the
+selection does not reach is found by the nightly run.
The nightly run has no reader waiting for it, so its matrices run one leg at
a time (`matrix_parallel`) instead of queueing 22 macOS jobs ahead of every
@@ -82,7 +85,8 @@ request, merge queue, and again on the push of the same commit).
| `python contracts (fast)` | Import cleanliness on the bare install, the named contract guards, skill-routing precision and recall, plugin auth. Blocking, no baseline. | — |
| `tests linux 1..6` | The whole suite in six shards. Batches of files run in fresh processes with a wall-clock budget; a failed batch is re-run file by file, a failed file once more (a pass there is reported as flaky). A file that runs out of its budget alone gets one more attempt with twice that budget, capped at the file ceiling: a file without history starts on the generic floor, and a pass there is reported as flaky and enters the history with its real duration. | `scripts/ci/run_tests_parallel.py` |
| `tests windows 1..6` | The whole suite in six shards on every tier, pull requests included, so a Windows-only failure blocks the pull request instead of ejecting it from the merge queue. | `scripts/ci/run_tests_parallel.py` |
-| `tests macos 1..3` | Nightly, release and manual runs with macOS only. | — |
+| `tests macos 1..3` | The whole suite on macOS: nightly, release and manual runs with macOS only. | — |
+| `tests macos impact` | Merge queue: the macOS tests the change can reach, before it lands. One runner for an ordinary change, the full three-way split when the change selects everything (the pipeline, conftest, the lockfile). Selected once, in `detect`. | `scripts/ci/select_tests.py` |
| `test report + floor` | Proves the six Linux shards cover every discovered file exactly once, enforces the min-passed floor. Detection freezes one shared duration snapshot for all shards, including partial reruns. | `scripts/ci/ratchet_tests.py` |
| `duration cache` | Runs on main only (nightly, a manual run, a push that bypassed the queue) and only after every Linux and Windows shard succeeded: merges the shard timings into one history per OS, on top of the history the run started from, so a file that was flaky in one run keeps its timing. Windows shards by its own history; until one exists it uses the Linux one. | `scripts/ci/ratchet_tests.py durations` |
| Lanes | `frontend` (three test shards and the build, side by side), `jarvisctl`, `deps`, `realtime` (+ slim container), `updater` (in-app update, native handover, restart helper), `agent runtimes` (+ slim container + the real Hermes and OpenClaw at their pinned version; see [agent-runtimes.md](agent-runtimes.md#ci-and-release-gates)), `dragdrop`, `browser`, `macOS desktop`, `installer smoke` — each only when its paths change, each as wide as the tier: Linux + Windows on a pull request, + macOS in the merge queue, + Intel and ARM in the wide matrix. | `scripts/ci/plan_run.py` |
diff --git a/scripts/ci/plan_run.py b/scripts/ci/plan_run.py
index 215329cec1..b029617e8a 100644
--- a/scripts/ci/plan_run.py
+++ b/scripts/ci/plan_run.py
@@ -18,6 +18,9 @@ run has to prove, not what could be run:
real runtimes and the real browser included). Nothing
macOS-specific that a lane tests lands unproven, and the five
slots are spent once per change, not once per push.
+ The whole Python suite on macOS is 27 runner-minutes: a group
+ runs the macOS tests its change can reach (select_tests.py), in
+ one job for an ordinary change.
``full`` Nightly, a release tag, a manual full dispatch. The wide matrix:
Intel macOS, the ARM runners, every interpreter, the macOS test
shards, and the Hermes install on Windows (14 to 23 minutes). A
@@ -101,6 +104,9 @@ def decide(
# slots with every merge group: its matrices run one leg at a time
# instead of queueing 22 macOS jobs ahead of everybody else's.
"matrix_parallel": 1 if event == "schedule" else UNTHROTTLED,
+ # The macOS tests the change can reach (select_tests.py) run before it
+ # lands; the wide matrix runs the whole macOS suite instead.
+ "mac_impact": tier == "queue" and not macos,
}
diff --git a/scripts/ci/required_results.py b/scripts/ci/required_results.py
index 9c413ee27f..e6c51f6da9 100644
--- a/scripts/ci/required_results.py
+++ b/scripts/ci/required_results.py
@@ -52,6 +52,8 @@ def expected_jobs(outputs: dict[str, str]) -> set[str]:
required.update(jobs)
if outputs.get("macos") == "true" and outputs.get("python") == "true":
required.add("tests-macos")
+ if outputs.get("mac_tests") == "true" and outputs.get("python") == "true":
+ required.add("tests-macos-impact")
if outputs.get("macos_desktop") == "true" and outputs.get("desktop_legs") != "none":
required.add("macos-desktop")
if outputs.get("release") == "true":
diff --git a/scripts/ci/select_tests.py b/scripts/ci/select_tests.py
index 24b297e874..47408c9614 100644
--- a/scripts/ci/select_tests.py
+++ b/scripts/ci/select_tests.py
@@ -2,8 +2,9 @@
"""Pick the test files a change can plausibly affect (impact selection).
Used where running everything is expensive and a full run follows anyway:
-the Windows leg of a pull request (the full Windows suite runs on every push
-to main and nightly) and ``scripts/agent_land.py``'s fast local check.
+the macOS tests of a merge-queue group (the whole suite on macOS is 27
+runner-minutes on five shared slots and runs nightly) and
+``scripts/agent_land.py``'s fast local check.
Selection is textual and deliberately generous — FAIL OPEN:
@@ -17,13 +18,15 @@ Selection is textual and deliberately generous — FAIL OPEN:
* conftest, packaging, lockfiles or the pipeline itself select EVERYTHING,
and so does a selection larger than :data:`ALL_RATIO` of the suite.
-Writes the file list to ``--out`` and ``mode=all|selected|none`` plus
-``count=N`` to ``$GITHUB_OUTPUT``.
+Writes the file list to ``--out`` and ``mode=all|selected|none``,
+``count=N`` and ``shards=[...]`` (how many runners the selection needs) to
+``$GITHUB_OUTPUT``.
"""
from __future__ import annotations
import argparse
+import json
import os
import re
import sys
@@ -37,6 +40,10 @@ from scripts.ci import agent_runtime_suite # noqa: E402
from scripts.ci.run_tests_parallel import discover # noqa: E402
ALL_RATIO = 0.35
+#: A selection up to this many files still fits one runner in about the time
+#: the other legs of a run take; a larger one gets two, everything gets three.
+SHARD_FILES = 450
+FULL_SHARDS = 3
_EVERYTHING_PREFIXES = (".github/workflows/", "scripts/ci/", "tests/fakes/", "tests/fixtures/")
_EVERYTHING_FILES = {
"conftest.py",
@@ -112,6 +119,15 @@ def select(changed: list[str], tests: list[str], texts: dict[str, str]) -> tuple
return "selected", sorted(chosen)
+def shards_for(mode: str, count: int) -> list[int]:
+ """The runners a selection needs: none, one, two, or the full split."""
+ if mode == "none":
+ return []
+ if mode == "all":
+ return list(range(1, FULL_SHARDS + 1))
+ return [1] if count <= SHARD_FILES else [1, 2]
+
+
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--changed-from", type=Path, help="File with changed paths (else stdin).")
@@ -133,7 +149,8 @@ def main(argv: list[str] | None = None) -> int:
output = os.environ.get("GITHUB_OUTPUT")
if output:
with open(output, "a", encoding="utf-8") as handle:
- handle.write(f"mode={mode}\ncount={len(chosen)}\n")
+ shards = json.dumps(shards_for(mode, len(chosen)), separators=(",", ":"))
+ handle.write(f"mode={mode}\ncount={len(chosen)}\nshards={shards}\n")
return 0
diff --git a/tests/unit/ci/test_pipeline_audit.py b/tests/unit/ci/test_pipeline_audit.py
index 87848ff44f..e5980af24f 100644
--- a/tests/unit/ci/test_pipeline_audit.py
+++ b/tests/unit/ci/test_pipeline_audit.py
@@ -20,6 +20,7 @@ from scripts.ci import (
release_admit,
release_assets,
required_results,
+ select_tests,
)
ROOT = Path(__file__).resolve().parents[3]
@@ -493,6 +494,57 @@ def test_the_nightly_run_never_queues_its_macos_legs_ahead_of_a_merge_group():
assert widest < plan_run.UNTHROTTLED
+def test_a_merge_group_runs_the_macos_tests_its_change_can_reach():
+ ci = workflows()["ci.yml"]["jobs"]
+ # Only the queue tier: a pull request takes no macOS runner, and the wide
+ # matrix runs the whole macOS suite instead.
+ assert plan_run.plan("merge_group")["mac_impact"] is True
+ assert plan_run.plan("push")["mac_impact"] is True
+ assert plan_run.plan("pull_request")["mac_impact"] is False
+ assert plan_run.plan("schedule")["mac_impact"] is False
+ # One runner for an ordinary change, two for a large one, the full split
+ # when the change selects everything, none when it reaches no test.
+ assert select_tests.shards_for("selected", 12) == [1]
+ assert select_tests.shards_for("selected", select_tests.SHARD_FILES + 1) == [1, 2]
+ assert select_tests.shards_for("all", 2500) == [1, 2, 3]
+ assert select_tests.shards_for("none", 0) == []
+ job = ci["tests-macos-impact"]
+ assert job["runs-on"] == "macos-latest"
+ assert job["if"] == (
+ "needs.detect.outputs.python == 'true' && needs.detect.outputs.mac_tests == 'true'"
+ )
+ assert job["strategy"]["matrix"]["shard"] == (
+ "${{ fromJSON(needs.detect.outputs.mac_test_shards) }}"
+ )
+ run = next(s for s in job["steps"] if s.get("name") == "Run the selected tests")["run"]
+ assert "--files-from mac_tests.txt" in run and '--shard "$SHARD/$total"' in run
+ ratchet = next(s for s in job["steps"] if s.get("name", "").startswith("Ratchet"))["run"]
+ assert "test-baseline-macos.json" in ratchet
+ # The selection is made once, in detect, and shipped in the snapshot; an
+ # empty matrix can never reach the job.
+ detect = ci["detect"]
+ select = next(s for s in detect["steps"] if s.get("id") == "mactests")["run"]
+ assert "select_tests.py --changed-from changed.txt --out mac_tests.txt" in select
+ assert "'[1]'" in detect["outputs"]["mac_test_shards"]
+ shared = next(
+ s for s in detect["steps"] if s.get("name") == "Share duration snapshot with every shard"
+ )
+ assert "mac_tests.txt" in shared["with"]["path"].split()
+ # The gate waits for it and requires it when it was selected.
+ assert "tests-macos-impact" in ci["gate"]["needs"]
+ flags = {name: "true" for name in classify_changes.LANES}
+ flags.update(full="false", macos="false", release="false", proven="false")
+ flags.update(desktop_legs="primary", mac_tests="true")
+ assert "tests-macos-impact" in required_results.expected_jobs(flags)
+ needs = {name: {"result": "success"} for name in required_results.expected_jobs(flags)}
+ needs["detect"]["outputs"] = flags
+ assert required_results.evaluate(needs, pipeline=True)["ok"]
+ needs["tests-macos-impact"]["result"] = "skipped"
+ assert not required_results.evaluate(needs, pipeline=True)["ok"]
+ flags["mac_tests"] = "false"
+ assert "tests-macos-impact" not in required_results.expected_jobs(flags)
+
+
def test_only_a_complete_green_run_on_main_writes_the_duration_caches():
ci = workflows()["ci.yml"]["jobs"]
job = ci["duration-cache"]
@@ -541,7 +593,10 @@ def test_windows_shards_by_its_own_history_and_falls_back_to_the_linux_one():
ensure = next(s for s in detect if s.get("id") == "snapshot")["run"]
assert "test -s test_durations_windows.json || cp test_durations.json" in ensure
shared = next(s for s in detect if s.get("name") == "Share duration snapshot with every shard")
- assert shared["with"]["path"].split() == ["test_durations.json", "test_durations_windows.json"]
+ assert shared["with"]["path"].split()[:2] == [
+ "test_durations.json",
+ "test_durations_windows.json",
+ ]
windows = next(s for s in ci["tests-windows"]["steps"] if s.get("name") == "Run shard")["run"]
assert "--durations test_durations_windows.json" in windows
assert '--durations-out "out/durations-windows-$SHARD.json"' in windows