Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

docs(ci): document the new pipeline, merge train and release flow

rubenluetke10-beepfe85a94dca85
diff --git a/.github/AGENT_WORKFLOW_AUDIT.md b/.github/AGENT_WORKFLOW_AUDIT.md
index 97facabf2a..242e96a486 100644
--- a/.github/AGENT_WORKFLOW_AUDIT.md
+++ b/.github/AGENT_WORKFLOW_AUDIT.md
@@ -1,5 +1,10 @@
# Coding-agent workflow audit
+> **Superseded 2026-09-28.** The pipeline was rebuilt around a single required
+> check, `CI gate`, with change-classified lanes, sharded tests, a known-failure
+> ratchet and a merge train. The current contract is `docs/ci-pipeline.md`; this
+> file stays as the record of the state before.
+
Snapshot: 2026-09-27, `PersonalJarvis/PersonalJarvis`, `main`. This records the
live GitHub API state, not a remembered setting. Re-read protection before any
change. GitHub documents the [status-check protection API](https://docs.github.com/en/rest/branches/branch-protection#update-status-check-protection)
diff --git a/AGENTS.md b/AGENTS.md
index ebd5ec3c00..614de55860 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -198,11 +198,15 @@ required in CI: `test_routing`, `test_output_filter`,
`test_hangup_reason_parity`, `test_turn_language`.
The commit and push hooks block confirmed secret, private-key, withheld-path,
-language, and broken-bundle additions. CI keeps privacy, repository integrity,
-changed-file code policy, the four contract guards, and the frontend build as
-required checks. Broader tests remain visible evidence. A red non-required job
-is not proof of a new regression or permission to ignore it: triage it against
-the exact base before merging, and fix any new failure.
+language, and broken-bundle additions. CI (`docs/ci-pipeline.md`) has ONE
+required check, `CI gate`, over change-classified lanes; a test failure blocks
+unless it is listed in `scripts/ci/test-baseline-<os>.json`. Finished work in
+your own worktree lands with `python scripts/agent_land.py` (rebase onto main,
+auto-resolve generated files, gates, relevant tests, push); a `codex/`,
+`claude/`, `agent/` branch or an `auto-merge` label puts a PR on the merge
+train, which keeps it current with main and squash-merges it once green.
+Triage any red job against the exact base; never add to a baseline to hide a
+new failure.
Run `check_boot_budget.py` after touching startup; CI cannot measure the live
voice-ready path.
diff --git a/CLAUDE.md b/CLAUDE.md
index ebd5ec3c00..614de55860 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -198,11 +198,15 @@ required in CI: `test_routing`, `test_output_filter`,
`test_hangup_reason_parity`, `test_turn_language`.
The commit and push hooks block confirmed secret, private-key, withheld-path,
-language, and broken-bundle additions. CI keeps privacy, repository integrity,
-changed-file code policy, the four contract guards, and the frontend build as
-required checks. Broader tests remain visible evidence. A red non-required job
-is not proof of a new regression or permission to ignore it: triage it against
-the exact base before merging, and fix any new failure.
+language, and broken-bundle additions. CI (`docs/ci-pipeline.md`) has ONE
+required check, `CI gate`, over change-classified lanes; a test failure blocks
+unless it is listed in `scripts/ci/test-baseline-<os>.json`. Finished work in
+your own worktree lands with `python scripts/agent_land.py` (rebase onto main,
+auto-resolve generated files, gates, relevant tests, push); a `codex/`,
+`claude/`, `agent/` branch or an `auto-merge` label puts a PR on the merge
+train, which keeps it current with main and squash-merges it once green.
+Triage any red job against the exact base; never add to a baseline to hide a
+new failure.
Run `check_boot_budget.py` after touching startup; CI cannot measure the live
voice-ready path.
diff --git a/docs/ci-pipeline.md b/docs/ci-pipeline.md
new file mode 100644
index 0000000000..6e2f8301b4
--- /dev/null
+++ b/docs/ci-pipeline.md
@@ -0,0 +1,130 @@
+# CI/CD pipeline
+
+How a change travels from a coding agent's branch to a user's machine. The
+design borrows the strongest ideas of the
+[Hermes Agent pipeline](https://github.com/NousResearch/hermes-agent/tree/main/.github/workflows)
+and adapts them to this repository: several coding agents working in
+parallel, a Python + TypeScript desktop app, three operating systems, and
+signed installers.
+
+```
+agent branch ──► pull request ──► CI (lanes) ──► CI gate ──► merge train ──► main
+ │
+ release-cut (manual) ──► tag ──► release gate ──► PyPI / installers / signatures
+ └─► GitHub Release ──► in-app updater
+```
+
+## 1. CI — `.github/workflows/ci.yml`
+
+| Stage | What it does | Script |
+| --- | --- | --- |
+| `detect` | Classifies the diff into lanes. Fails open: an empty diff, a pipeline change, a push to main, the nightly run and a manual run turn every lane on. | `scripts/ci/classify_changes.py` |
+| `static gates` | ~20 repository gates (keys, bundle, mirrors, privacy, docs, CLI coverage, ratchets, bash 3.2, no new German) in one job. Every gate reports. | `scripts/ci/run_gates.py` |
+| `python contracts (fast)` | Import cleanliness on the bare install, the named contract guards, skill-routing precision and recall, plugin auth. Blocking, no baseline. | — |
+| `tests linux 1..6` | The whole suite in six shards. Batches of files run in fresh processes with a wall-clock budget; a failed batch is re-run file by file, a failed file once more (a pass there is reported as flaky). | `scripts/ci/run_tests_parallel.py` |
+| `tests windows` | Four shards on full runs; on a pull request one runner takes only the tests the diff can reach. | `scripts/ci/select_tests.py` |
+| `tests macos 1..3` | Nightly and manual runs only (~10x runner cost). | — |
+| `test report + floor` | Sums all Linux shards, enforces the min-passed floor, lists baselined failures that now pass, and on main refreshes the per-file duration cache that balances the shards. | `scripts/ci/ratchet_tests.py` |
+| Lanes | `frontend`, `jarvisctl`, `deps`, `realtime` (3 OS + slim container), `dragdrop`, `browser`, `macOS desktop`, `installer smoke` — each only when its paths change. | — |
+| `CI gate` | Aggregates every job. **The only required check.** Skipped lanes pass; the nightly run is strict and fails on any skip. | `scripts/ci/required_results.py` |
+
+### Known failures: the ratchet
+
+The suite carries a backlog of failures, mostly platform-specific. They are
+listed per OS in `scripts/ci/test-baseline-{linux,windows,macos}.json`. A
+failure in the list is reported and never blocks; any other failure blocks
+the change that introduced it. The list only shrinks: `test report` names the
+entries that pass again. Regenerate a list from a full run with:
+
+```bash
+gh run download <run-id> -p 'tests-linux-*' -D reports
+python scripts/ci/ratchet_tests.py update --out scripts/ci/test-baseline-linux.json reports
+```
+
+A missing list puts that OS in report-only mode until the first full run
+produces one.
+
+### Concurrency
+
+A pull request (or a branch run the merge train dispatched) keeps only its
+newest run. Pushes to main, the nightly run and manual full runs are never
+cancelled — every commit on main gets its own verdict.
+
+## 2. Integrating several agents — `.github/workflows/merge-train.yml`
+
+Agents push branches named `codex/…`, `claude/…`, `agent/…`, `gemini/…`,
+`cursor/…` or `bot/…`, or add the `auto-merge` label to any pull request. The
+merge train runs on every push to main, after every CI run, and every 15
+minutes:
+
+1. A pull request behind main gets main merged into it
+ (`scripts/ci/agent_integrate.py`). No force push, so an agent that keeps
+ pushing to its branch is never overwritten.
+2. Conflicts resolve by file class: **generated** files (frontend `dist/`,
+ agent mirrors, CLI reference docs, lockfiles, timing caches) take main's
+ copy and are regenerated; **append-only** files (CHANGELOG, allowlists,
+ `.gitignore`, `docs/BUGS.md`) are union-merged; **everything else** goes
+ to the optional AI resolver (`ANTHROPIC_API_KEY` → Claude Code,
+ `OPENAI_API_KEY` → Codex CLI). Its result must leave no conflict markers
+ and must still parse, and CI re-runs on it before anything lands. What
+ stays unresolved aborts cleanly, gets the `needs-rebase` label and a
+ comment listing the files. `git rerere` replays recorded resolutions.
+3. The first up-to-date pull request with a green `CI gate` is squash-merged.
+ One merge per tick means every landing was tested on top of the exact main
+ it lands on.
+
+Opt out with `no-auto-merge`, `do-not-merge`, `wip` or `needs-human`.
+`priority` moves a pull request to the front. Forks and Dependabot never ride
+the train.
+
+**Token.** With the optional `INTEGRATION_TOKEN` secret (a fine-grained token
+with contents and pull-request write access), the train's pushes and merges
+fire the normal events. Without it the train uses `GITHUB_TOKEN`, whose pushes
+fire no events, and dispatches `ci.yml` itself for each updated branch and for
+main after each merge.
+
+### Locally: `scripts/agent_land.py`
+
+An agent in its own worktree runs one command when its work is done:
+
+```bash
+python scripts/agent_land.py # feature branch -> PR labelled auto-merge
+python scripts/agent_land.py --direct # fast-forward main itself
+```
+
+It rebases onto the latest main with the same conflict engine, runs the
+static gates, runs the tests the diff can reach, and pushes. It refuses a
+dirty worktree and never stashes or force-pushes.
+
+## 3. Releases, installers and updates
+
+A release happens **only** when the maintainer asks for one.
+
+* **`release-cut.yml`** (manual): refuses unless main is green, bumps
+ `pyproject.toml` + `jarvis/__init__.py`, moves the `[Unreleased]` notes (or
+ the Conventional Commits since the last tag) into a dated CHANGELOG section
+ (`scripts/ci/cut_release.py`), commits, dispatches CI, tags, and dispatches
+ the three publishing workflows on the tag.
+* **`release-gate.yml`** is the first job of `release.yml` (PyPI),
+ `desktop-installers.yml` and `sign-installer.yml`. It admits a tag only
+ when tag, versions and CHANGELOG agree, the commit is on main, and
+ `CI gate` passed on that exact commit (`scripts/ci/release_admit.py`,
+ waiting while CI still runs).
+* The GitHub Release (notes + the resumable `personal-jarvis-src.tar.gz`) is
+ published after the same admission. The in-app updater follows
+ `releases/latest`, so a release reaches managed installs and installer
+ users only after main's CI proved it.
+
+## 4. Adapted from Hermes, and what was left out
+
+| Hermes idea | Here |
+| --- | --- |
+| Orchestrator + change classifier, fail-open lanes | `detect` + `classify_changes.py` |
+| One aggregate required check (`all-checks-pass`) | `CI gate` |
+| Per-file process isolation, duration cache written only by main | `run_tests_parallel.py`, batched to amortise this suite's start-up cost |
+| Only PR runs are cancelled; main and release never | `concurrency` block |
+| Strict mode where a skipped lane fails | nightly run |
+| Unrelated-history check | `history` gate |
+| Stable release admits a claim before building | `release-gate.yml` |
+| Autofix PRs with a privileged/unprivileged split | not adopted: generated files are regenerated during integration instead |
+| 96-core runners, daily canary tags, Docker/Nix lanes | not adopted: standard runners, releases stay manual, no such artefacts |