Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.
Python agent-orchestrationai-agentai-coding-assistantclaude-codecoding-agentcomputer-usedesktop-applocal-firstmcppersonal-jarvisvoice-assistantvoice-coding
ci: run the macOS tests a change can reach in the merge queue (#605)
Ruben Lütkeb5b172a05577
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex a928336f3a..2e3aab0060 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -144,6 +144,8 @@ jobs:browser_os: ${{ steps.mode.outputs.browser_os }}desktop_legs: ${{ steps.mode.outputs.desktop_legs }}matrix_parallel: ${{ steps.mode.outputs.matrix_parallel }}+ mac_tests: ${{ steps.mactests.outputs.mode != 'none' }}+ mac_test_shards: ${{ steps.mactests.outputs.shards == '[]' && '[1]' || steps.mactests.outputs.shards }}duration_snapshot: ${{ steps.snapshot.outputs.name }}release: ${{ github.ref_type == 'tag' }}steps:@@ -221,6 +223,24 @@ jobs:git diff --name-only --no-renames "$BASE" HEAD | tee changed.txt | python3 scripts/ci/classify_changes.pyfi+ # A merge group runs the macOS tests its change can reach: the whole+ # suite on macOS is 27 runner-minutes on five shared slots. No diff+ # (changed.txt missing or empty) selects everything.+ - name: Select the macOS tests this change can reach+ id: mactests+ shell: bash+ env:+ PYTHON_LANE: ${{ steps.lanes.outputs.python }}+ MAC_IMPACT: ${{ steps.mode.outputs.mac_impact }}+ run: |+ : > mac_tests.txt+ if [ "$PYTHON_LANE" = "true" ] && [ "$MAC_IMPACT" = "true" ]; then+ test -f changed.txt || : > changed.txt+ python3 scripts/ci/select_tests.py --changed-from changed.txt --out mac_tests.txt+ else+ { echo "mode=none"; echo "shards=[]"; } | tee -a "$GITHUB_OUTPUT"+ fi+# Resolve the mutable cache ONCE. All shards download the same immutable# snapshot; staggered runners must never compute different partitions.- name: Restore duration history once@@ -257,6 +277,7 @@ jobs:path: |test_durations.jsontest_durations_windows.json+ mac_tests.txtif-no-files-found: errorretention-days: 7@@ -601,6 +622,57 @@ jobs:path: out/retention-days: 14+ # The macOS tests the change can reach, before it lands (merge queue). One+ # runner for an ordinary change; a change that selects everything (the+ # pipeline, conftest, the lockfile) gets the full three-way split. The+ # nightly and release runs keep the whole suite in `tests-macos`.+ tests-macos-impact:+ needs: detect+ if: needs.detect.outputs.python == 'true' && needs.detect.outputs.mac_tests == 'true'+ strategy:+ fail-fast: false+ matrix:+ shard: ${{ fromJSON(needs.detect.outputs.mac_test_shards) }}+ runs-on: macos-latest+ timeout-minutes: 40+ steps:+ - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2+ with:+ persist-credentials: false+ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0+ with:+ python-version: "3.11"+ - name: Install+ uses: ./.github/actions/install-locked+ with:+ extras: dev telephony+ - name: Download the shared duration snapshot+ uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8+ with:+ name: ${{ needs.detect.outputs.duration_snapshot }}+ - name: Run the selected tests+ continue-on-error: true+ env:+ SHARD: ${{ matrix.shard }}+ SHARDS: ${{ needs.detect.outputs.mac_test_shards }}+ run: |+ total=$(python -c "import json,os; print(len(json.loads(os.environ['SHARDS'])))")+ python scripts/ci/run_tests_parallel.py tests --files-from mac_tests.txt \+ --shard "$SHARD/$total" \+ --report "out/test-report-macos-impact-$SHARD.json" \+ --junit-out "out/junit-macos-impact-$SHARD.xml"+ - name: Ratchet (a failure not in the baseline blocks)+ env:+ SHARD: ${{ matrix.shard }}+ run: python scripts/ci/ratchet_tests.py check --baseline scripts/ci/test-baseline-macos.json "out/test-report-macos-impact-$SHARD.json"+ - name: Upload shard report+ if: always()+ uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3+ with:+ name: tests-macos-impact-${{ matrix.shard }}+ path: out/+ retention-days: 14+test-report:name: test report + floorneeds: [detect, tests-linux]@@ -1183,6 +1255,7 @@ jobs:- tests-linux- tests-windows- tests-macos+ - tests-macos-impact- test-report- duration-cache- frontenddiff --git a/docs/ci-pipeline.md b/docs/ci-pipeline.mdindex 7c44a890ab..bb38c6179d 100644--- a/docs/ci-pipeline.md+++ b/docs/ci-pipeline.md@@ -54,7 +54,10 @@ well (one morning, 2026-10-13), pull requests waited 6 to 25 minutes for aslot. The queue builds two groups at a time, so at most sixteen macOS legscompete for the five slots; in the first such group all eight were done 14minutes in. The whole Python suite on macOS (three shards of nine minutes)-still runs nightly and on release tags only.+runs nightly and on release tags; a merge group runs the macOS tests its+change can reach instead (`tests macos impact`), so a test that the change+breaks on macOS stops it in the queue. A macOS failure in a test the+selection does not reach is found by the nightly run.The nightly run has no reader waiting for it, so its matrices run one leg ata time (`matrix_parallel`) instead of queueing 22 macOS jobs ahead of every@@ -82,7 +85,8 @@ request, merge queue, and again on the push of the same commit).| `python contracts (fast)` | Import cleanliness on the bare install, the named contract guards, skill-routing precision and recall, plugin auth. Blocking, no baseline. | — || `tests linux 1..6` | The whole suite in six shards. Batches of files run in fresh processes with a wall-clock budget; a failed batch is re-run file by file, a failed file once more (a pass there is reported as flaky). A file that runs out of its budget alone gets one more attempt with twice that budget, capped at the file ceiling: a file without history starts on the generic floor, and a pass there is reported as flaky and enters the history with its real duration. | `scripts/ci/run_tests_parallel.py` || `tests windows 1..6` | The whole suite in six shards on every tier, pull requests included, so a Windows-only failure blocks the pull request instead of ejecting it from the merge queue. | `scripts/ci/run_tests_parallel.py` |-| `tests macos 1..3` | Nightly, release and manual runs with macOS only. | — |+| `tests macos 1..3` | The whole suite on macOS: nightly, release and manual runs with macOS only. | — |+| `tests macos impact` | Merge queue: the macOS tests the change can reach, before it lands. One runner for an ordinary change, the full three-way split when the change selects everything (the pipeline, conftest, the lockfile). Selected once, in `detect`. | `scripts/ci/select_tests.py` || `test report + floor` | Proves the six Linux shards cover every discovered file exactly once, enforces the min-passed floor. Detection freezes one shared duration snapshot for all shards, including partial reruns. | `scripts/ci/ratchet_tests.py` || `duration cache` | Runs on main only (nightly, a manual run, a push that bypassed the queue) and only after every Linux and Windows shard succeeded: merges the shard timings into one history per OS, on top of the history the run started from, so a file that was flaky in one run keeps its timing. Windows shards by its own history; until one exists it uses the Linux one. | `scripts/ci/ratchet_tests.py durations` || Lanes | `frontend` (three test shards and the build, side by side), `jarvisctl`, `deps`, `realtime` (+ slim container), `updater` (in-app update, native handover, restart helper), `agent runtimes` (+ slim container + the real Hermes and OpenClaw at their pinned version; see [agent-runtimes.md](agent-runtimes.md#ci-and-release-gates)), `dragdrop`, `browser`, `macOS desktop`, `installer smoke` — each only when its paths change, each as wide as the tier: Linux + Windows on a pull request, + macOS in the merge queue, + Intel and ARM in the wide matrix. | `scripts/ci/plan_run.py` |diff --git a/scripts/ci/plan_run.py b/scripts/ci/plan_run.pyindex 215329cec1..b029617e8a 100644--- a/scripts/ci/plan_run.py+++ b/scripts/ci/plan_run.py@@ -18,6 +18,9 @@ run has to prove, not what could be run:real runtimes and the real browser included). NothingmacOS-specific that a lane tests lands unproven, and the fiveslots are spent once per change, not once per push.+ The whole Python suite on macOS is 27 runner-minutes: a group+ runs the macOS tests its change can reach (select_tests.py), in+ one job for an ordinary change.``full`` Nightly, a release tag, a manual full dispatch. The wide matrix:Intel macOS, the ARM runners, every interpreter, the macOS testshards, and the Hermes install on Windows (14 to 23 minutes). A@@ -101,6 +104,9 @@ def decide(# slots with every merge group: its matrices run one leg at a time# instead of queueing 22 macOS jobs ahead of everybody else's."matrix_parallel": 1 if event == "schedule" else UNTHROTTLED,+ # The macOS tests the change can reach (select_tests.py) run before it+ # lands; the wide matrix runs the whole macOS suite instead.+ "mac_impact": tier == "queue" and not macos,}diff --git a/scripts/ci/required_results.py b/scripts/ci/required_results.pyindex 9c413ee27f..e6c51f6da9 100644--- a/scripts/ci/required_results.py+++ b/scripts/ci/required_results.py@@ -52,6 +52,8 @@ def expected_jobs(outputs: dict[str, str]) -> set[str]:required.update(jobs)if outputs.get("macos") == "true" and outputs.get("python") == "true":required.add("tests-macos")+ if outputs.get("mac_tests") == "true" and outputs.get("python") == "true":+ required.add("tests-macos-impact")if outputs.get("macos_desktop") == "true" and outputs.get("desktop_legs") != "none":required.add("macos-desktop")if outputs.get("release") == "true":diff --git a/scripts/ci/select_tests.py b/scripts/ci/select_tests.pyindex 24b297e874..47408c9614 100644--- a/scripts/ci/select_tests.py+++ b/scripts/ci/select_tests.py@@ -2,8 +2,9 @@"""Pick the test files a change can plausibly affect (impact selection).Used where running everything is expensive and a full run follows anyway:-the Windows leg of a pull request (the full Windows suite runs on every push-to main and nightly) and ``scripts/agent_land.py``'s fast local check.+the macOS tests of a merge-queue group (the whole suite on macOS is 27+runner-minutes on five shared slots and runs nightly) and+``scripts/agent_land.py``'s fast local check.Selection is textual and deliberately generous — FAIL OPEN:@@ -17,13 +18,15 @@ Selection is textual and deliberately generous — FAIL OPEN:* conftest, packaging, lockfiles or the pipeline itself select EVERYTHING,and so does a selection larger than :data:`ALL_RATIO` of the suite.-Writes the file list to ``--out`` and ``mode=all|selected|none`` plus-``count=N`` to ``$GITHUB_OUTPUT``.+Writes the file list to ``--out`` and ``mode=all|selected|none``,+``count=N`` and ``shards=[...]`` (how many runners the selection needs) to+``$GITHUB_OUTPUT``."""from __future__ import annotationsimport argparse+import jsonimport osimport reimport sys@@ -37,6 +40,10 @@ from scripts.ci import agent_runtime_suite # noqa: E402from scripts.ci.run_tests_parallel import discover # noqa: E402ALL_RATIO = 0.35+#: A selection up to this many files still fits one runner in about the time+#: the other legs of a run take; a larger one gets two, everything gets three.+SHARD_FILES = 450+FULL_SHARDS = 3_EVERYTHING_PREFIXES = (".github/workflows/", "scripts/ci/", "tests/fakes/", "tests/fixtures/")_EVERYTHING_FILES = {"conftest.py",@@ -112,6 +119,15 @@ def select(changed: list[str], tests: list[str], texts: dict[str, str]) -> tuplereturn "selected", sorted(chosen)+def shards_for(mode: str, count: int) -> list[int]:+ """The runners a selection needs: none, one, two, or the full split."""+ if mode == "none":+ return []+ if mode == "all":+ return list(range(1, FULL_SHARDS + 1))+ return [1] if count <= SHARD_FILES else [1, 2]++def main(argv: list[str] | None = None) -> int:parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])parser.add_argument("--changed-from", type=Path, help="File with changed paths (else stdin).")@@ -133,7 +149,8 @@ def main(argv: list[str] | None = None) -> int:output = os.environ.get("GITHUB_OUTPUT")if output:with open(output, "a", encoding="utf-8") as handle:- handle.write(f"mode={mode}\ncount={len(chosen)}\n")+ shards = json.dumps(shards_for(mode, len(chosen)), separators=(",", ":"))+ handle.write(f"mode={mode}\ncount={len(chosen)}\nshards={shards}\n")return 0diff --git a/tests/unit/ci/test_pipeline_audit.py b/tests/unit/ci/test_pipeline_audit.pyindex 87848ff44f..e5980af24f 100644--- a/tests/unit/ci/test_pipeline_audit.py+++ b/tests/unit/ci/test_pipeline_audit.py@@ -20,6 +20,7 @@ from scripts.ci import (release_admit,release_assets,required_results,+ select_tests,)ROOT = Path(__file__).resolve().parents[3]@@ -493,6 +494,57 @@ def test_the_nightly_run_never_queues_its_macos_legs_ahead_of_a_merge_group():assert widest < plan_run.UNTHROTTLED+def test_a_merge_group_runs_the_macos_tests_its_change_can_reach():+ ci = workflows()["ci.yml"]["jobs"]+ # Only the queue tier: a pull request takes no macOS runner, and the wide+ # matrix runs the whole macOS suite instead.+ assert plan_run.plan("merge_group")["mac_impact"] is True+ assert plan_run.plan("push")["mac_impact"] is True+ assert plan_run.plan("pull_request")["mac_impact"] is False+ assert plan_run.plan("schedule")["mac_impact"] is False+ # One runner for an ordinary change, two for a large one, the full split+ # when the change selects everything, none when it reaches no test.+ assert select_tests.shards_for("selected", 12) == [1]+ assert select_tests.shards_for("selected", select_tests.SHARD_FILES + 1) == [1, 2]+ assert select_tests.shards_for("all", 2500) == [1, 2, 3]+ assert select_tests.shards_for("none", 0) == []+ job = ci["tests-macos-impact"]+ assert job["runs-on"] == "macos-latest"+ assert job["if"] == (+ "needs.detect.outputs.python == 'true' && needs.detect.outputs.mac_tests == 'true'"+ )+ assert job["strategy"]["matrix"]["shard"] == (+ "${{ fromJSON(needs.detect.outputs.mac_test_shards) }}"+ )+ run = next(s for s in job["steps"] if s.get("name") == "Run the selected tests")["run"]+ assert "--files-from mac_tests.txt" in run and '--shard "$SHARD/$total"' in run+ ratchet = next(s for s in job["steps"] if s.get("name", "").startswith("Ratchet"))["run"]+ assert "test-baseline-macos.json" in ratchet+ # The selection is made once, in detect, and shipped in the snapshot; an+ # empty matrix can never reach the job.+ detect = ci["detect"]+ select = next(s for s in detect["steps"] if s.get("id") == "mactests")["run"]+ assert "select_tests.py --changed-from changed.txt --out mac_tests.txt" in select+ assert "'[1]'" in detect["outputs"]["mac_test_shards"]+ shared = next(+ s for s in detect["steps"] if s.get("name") == "Share duration snapshot with every shard"+ )+ assert "mac_tests.txt" in shared["with"]["path"].split()+ # The gate waits for it and requires it when it was selected.+ assert "tests-macos-impact" in ci["gate"]["needs"]+ flags = {name: "true" for name in classify_changes.LANES}+ flags.update(full="false", macos="false", release="false", proven="false")+ flags.update(desktop_legs="primary", mac_tests="true")+ assert "tests-macos-impact" in required_results.expected_jobs(flags)+ needs = {name: {"result": "success"} for name in required_results.expected_jobs(flags)}+ needs["detect"]["outputs"] = flags+ assert required_results.evaluate(needs, pipeline=True)["ok"]+ needs["tests-macos-impact"]["result"] = "skipped"+ assert not required_results.evaluate(needs, pipeline=True)["ok"]+ flags["mac_tests"] = "false"+ assert "tests-macos-impact" not in required_results.expected_jobs(flags)++def test_only_a_complete_green_run_on_main_writes_the_duration_caches():ci = workflows()["ci.yml"]["jobs"]job = ci["duration-cache"]@@ -541,7 +593,10 @@ def test_windows_shards_by_its_own_history_and_falls_back_to_the_linux_one():ensure = next(s for s in detect if s.get("id") == "snapshot")["run"]assert "test -s test_durations_windows.json || cp test_durations.json" in ensureshared = next(s for s in detect if s.get("name") == "Share duration snapshot with every shard")- assert shared["with"]["path"].split() == ["test_durations.json", "test_durations_windows.json"]+ assert shared["with"]["path"].split()[:2] == [+ "test_durations.json",+ "test_durations_windows.json",+ ]windows = next(s for s in ci["tests-windows"]["steps"] if s.get("name") == "Run shard")["run"]assert "--durations test_durations_windows.json" in windowsassert '--durations-out "out/durations-windows-$SHARD.json"' in windows