The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
docs(ci): the CI rules agents follow, and what PR CI no longer runs
Autonomous Devops531e27fe6128
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 8fed92f8b..7d2da8cb0 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -1,8 +1,9 @@name: CI# Fast PR checks: each changed component runs its own Linux unit tests. Native,-# macOS, browser and end-to-end acceptance run in the release workflows. Merge-# candidates rerun only the plan and process checks; the PR run covered the code.+# macOS, coverage-gate, browser and end-to-end checks are outside PR CI; see+# docs/validation-and-release.md#what-pr-ci-does-not-run. Merge candidates rerun+# only the plan and process checks; the PR run covered the code.# Contributor code runs with a read-only token and no release credentials.on:pull_request:diff --git a/AGENTS.md b/AGENTS.mdindex ece9cf45e..df11fafa5 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -25,23 +25,34 @@ For product names, terminology, and visible copy, follow thevalidation cycle. Verify published versions and checksums, then report completion.Desktop's `--wait` follows the exact tag/SHA through the workflow's six-artifactverification; reuse that receipt instead of repeating the downloads manually.-- Prepare the PR and complete code/native review while automatic CI runs. For an- agent implementation in progress, push to a draft PR: CI runs cheap workflow- and process checks. Mark ready after targeted local checks to start complete- affected suites, and keep the revision stable while CI and review finish.- The integration gate remains blocked on drafts. For an authorized merge, use- `make merge-pr` with the reviewed head/base SHAs and- `--queue --merge`; it enqueues the exact reviewed head, follows the queue and- verifies the merged tree against successful merge-group CI. Main advancing- does not invalidate the review base; the queue checks the combined candidate.- GitHub approval requirements remain as configured; this does not introduce- an additional human approval count. Keep review independent of implementation.- Existing `--run/--scope` commands automatically select queue mode when the- target's merge-queue rule is enabled; direct mode remains for rollout only.- For evidence outside the queue, use `scripts/record-ci-validation.py` and the- existing verified Process/Desktop source-input contracts. Other affected- native, browser, engine and hardware checks remain explicit review requirements.- Never replay an uncertain enqueue/merge mutation; inspect the same PR first.+- CI rules (details in [docs/validation-and-release.md](docs/validation-and-release.md#pr-ci)):+ - PR CI is fast on purpose. It runs only the Linux unit checks of the components+ the PR changes; docs, workflow and script changes run process checks only. The+ one required check is `ci/required`.+ - Push work in progress to a draft PR (process checks only). Mark it ready to run+ the component checks, then keep the head stable while CI and review finish.+ Do the code review while CI runs, and keep review independent of implementation.+ - Base the branch on current `main`. A branch older than #991 runs the old+ planner and fails `ci/required`; use **Update branch** or rebase first.+ - Merge with `make merge-pr ARGS="N --queue --reviewed-head SHA --reviewed-base SHA --merge"`.+ The queue reruns only the plan and process checks. Do not use `--run/--scope`+ (legacy direct mode). Never replay an uncertain enqueue or merge; inspect the PR.+ - Do not start extra runs for a PR: no manual **CI → Run workflow** while its+ automatic run exists, no repeated pushes to retrigger a red check. Read the+ failure, fix it, push once. Rerun a job only for a recorded flaky test.+ - Checks PR CI no longer runs are yours to run locally when you touch their area:+ CLI coverage gates (`npm run test:core`, `test:harnessd`, `test:resume`,+ `test:orchestrator`, `test:sharing`, `test:remote-viewers`, `test:portability`,+ `test:local-models`), serial and PTY specs, and Desktop tests on macOS for+ macOS-specific code. Record what you ran in the PR.+ - Heavy workflows run only when started by hand: OS images and VM tests+ (**Harness OS** and the other OS workflows) and CLI end to end (nightly too).+ Start one only when the change needs it, once, and report its run link.+ - The account runs at most 20 jobs at once, 5 on macOS. Do not run several+ heavy manual workflows at the same time.+ - Release workflows are not a full test suite (CLI release runs typecheck, Desktop+ release builds and signs). A release failure is still a code failure: fix it+ and cut the next version; never move a published tag.- For an authorized Desktop release, start `make release-desktop ARGS="--prepare"`from the final pushed PR branch alongside validation and review. It preparesverified packages without publishing; merge and release only after checks pass.diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 80175a75f..7013c72b3 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -200,8 +200,9 @@ source satisfies that full-suite check; do not run it again locally before releaManual CI offers `cli`, `tui`, `backend`, `desktop`, and `full` scopes; `full` remains the default.CLI scope runs the typecheck and the default Vitest suite in four Linux shards.Desktop scope runs the VM suite in four Linux shards. PR CI runs only the suites of-the components a PR changes; native, macOS, coverage-gate and end-to-end checks run in-the release workflows.+the components a PR changes. Coverage gates, serial/PTY specs, macOS Desktop tests and+end to end are not part of PR CI: run the ones your change touches yourself (see+[What PR CI does not run](docs/validation-and-release.md#what-pr-ci-does-not-run)).Two further suites exist and are**opt-in**, because they need software the machine may not have — they skip themselves rather thandiff --git a/cli/AGENTS.md b/cli/AGENTS.mdindex d422da186..9ab431c14 100644--- a/cli/AGENTS.md+++ b/cli/AGENTS.md@@ -85,7 +85,8 @@ sessions to the owner's other machines, the Share relay for a harness shared wit3. **The core does not wait on a service and does not crash with one.** Services start through`serviceHost.start()`; every port declares fallbacks beside it in `core/api.ts`.4. **100% coverage, per file,** for `src/core/`, `src/services/` (`npm run test:core`) and `src/harnessd/`- (`npm run test:harnessd`). CI enforces both. Write the test that fails without your change. The one+ (`npm run test:harnessd`). PR CI does not run these gates: run them locally when you change those+ files and record the result in the PR. Write the test that fails without your change. The onefile outside it is `src/core/main.ts`, the wiring, which the end-to-end suite runs.5. **End to end for every user-facing flow** (`npm run test:e2e`, `e2e/`): the real daemon, a privatetmux server and fake Claude Code and Codex engines (`e2e/harness/fakeEngine.mjs`). Prefer extendingdiff --git a/docs/validation-and-release.md b/docs/validation-and-release.mdindex 13049db5f..16c84e042 100644--- a/docs/validation-and-release.md+++ b/docs/validation-and-release.md@@ -25,26 +25,57 @@ compiling, or starting a broad suite. Do not clean other sessions' files to makeRun independent checks concurrently, with enough capacity for their workers;two commands each spawning every CPU is not useful parallelism.+## PR CI+PR CI is deliberately fast. Each changed component runs only its own Linux suite:-| Changed directory | PR check |-| --- | --- |-| `cli/`, `tests/` | `cli-typecheck` and `cli-tests` (default Vitest suite, four shards) |-| `desktop/` | `desktop-tests` (VM suite, four Linux shards) |-| `tui/` | `tui-test` (`cargo test`) |-| `backend/`, `companions/`, `website/`, `os/`, `provider/`, `mobile/`, `daemons/` | that component's check |-| `devices/` | `firmware-checks` |-| anything else (docs, workflows, scripts, store) | `process-checks` only |+| Changed directory | PR check | Typical time |+| --- | --- | --- |+| `cli/`, `tests/` | `cli-typecheck` and `cli-tests` (default Vitest suite, four shards) | 3–5 min |+| `desktop/` | `desktop-tests` (VM suite, four Linux shards) | 5–7 min |+| `tui/` | `tui-test` (`cargo test`) | a few minutes |+| `backend/`, `companions/`, `website/`, `os/`, `provider/`, `mobile/`, `daemons/` | that component's one check | 1–4 min |+| `devices/` | `firmware-checks` | 1–2 min |+| anything else (docs, workflows, scripts, store) | `process-checks` only | about 1.5 min |`plan` (actionlint and suite selection), `process-checks` and `ci/required` always-run. Merge-queue runs rerun only those three jobs: the PR run already tested the-change. macOS, native process and serial, per-file coverage gates, browser-acceptance and CLI end-to-end checks run in the release workflows, on the nightly-CLI end-to-end run, or on demand from their own workflows. A release failure is a-code failure.+run; `ci/required` is the only required check. On a draft PR only those three run+and `ci/required` stays blocked; marking the PR ready runs the component checks.+Merge-queue runs rerun only the three always-on jobs (about 2 minutes): the PR run+already tested the change.++The `plan` and `ci/required` jobs run `scripts/ci-plan.py` from the PR head. A branch+created before #991 still carries the old planner, which expects removed jobs and+fails `ci/required` for CLI, Desktop, workflow and script changes. Update the branch+from `main` before relying on its CI.++### What PR CI does not run++| Check | Where it runs now | When you must run it yourself |+| --- | --- | --- |+| CLI per-file coverage gates (`test:core`, `test:harnessd`, `test:resume`, `test:orchestrator`, `test:sharing`, `test:remote-viewers`, `test:portability`, `test:local-models`) | nowhere automatically | when you change files those gates cover (`src/core/`, `src/services/`, `src/harnessd/` and each gate's area) |+| CLI serial, PTY and login-shell specs; macOS process images | CLI release builds and verifies process images only | when you change `src/cable/` serial code, shell startup or process images; run the specs on macOS |+| CLI end to end (`cli/e2e`) | nightly and **Actions → CLI end to end → Run workflow** | for user-facing CLI flows; start one run on the branch |+| Desktop tests on macOS | nowhere automatically | when you change macOS-specific Desktop code; `make desktop-test` on a Mac |+| TUI native fixtures and connected shells | the TUI release (`release-tui.yml`) | optional locally: `scripts/validate-tui-native.py` |+| Experience, authoring, Home Assistant and Desktop logging browser checks | **Run workflow** on their own workflows | when you change their area |+| OS image builds, VM and hardware acceptance | **Actions → Harness OS** and the other OS workflows, by hand only | for installer, image or session changes; start once with the needed inputs |++Record what you ran, with run links, in the PR. Release workflows are not a full+test suite: the CLI release runs typecheck and bundle checks, and the Desktop release+builds, signs and verifies downloads. A release failure is still a code failure: fix+it and cut the next version.++Do not add CI runs to a PR. Don't start **CI → Run workflow** for a PR that already has+an automatic run, and don't push again just to retrigger a failed check. Read the+failure, fix it and push once. The account runs at most 20 jobs at once (5 on macOS),+and every extra run delays every other PR.++### Suite detailsManual **CI → Run workflow** accepts `scope`: `cli`, `tui`, `backend`, `desktop`,-`full` (the default, all five core components), `all`, or one extra component.+`full` (the default, all five core components), `all`, or one extra component. It+runs the same fast jobs as PR CI.`desktop-tests` uses the release's pinned Flutter SDK and shared dependency caches.Each shard has four test workers and the bounded runner's ten-minute budget. Logs@@ -170,9 +201,8 @@ candidates are removed by their owning workflow. For process maintenance, dispatit requires reuse, promotes to disposable paths, verifies all six downloads, thenremoves those objects and the consumed candidate. It cannot publish a product.-Native TUI CI tests and builds the shipped musl target in the same Cargo output-directory. Dependency caches are keyed by target, Rust toolchain, and Cargo inputs;-cache hits still run every test. The native TUI fixtures run two at a time,+PR CI runs the TUI unit tests only. The TUI release builds the shipped musl targets+and runs the native fixtures against them. The native TUI fixtures run two at a time,using their own homes, socket names, and mock ports. CI retains each fixture's logand validation receipt as an artifact. Native comparisons use tmux 3.7c at the pinnedcommit in `.github/actions/reference-tmux`; Ubuntu's 3.4 loses the exit status in the@@ -279,8 +309,11 @@ before merge and then again on the merged commit. Check the merged source identiand rerun only validation invalidated by the merge. Native checks absent from CIstill need their own evidence.-Prepare the PR description and review the diff while CI is running. Once it passes,-collect its record with one read-only command:+Prepare the PR description and review the diff while CI is running. For a normal+merge you need nothing more: `make merge-pr --queue` collects and verifies the+`ci/required` receipt itself. The collector below is for manual runs only, and its+`cli` and `desktop` scopes still expect the job set from before #991, so they fail+on current runs; `process` still works. To collect a manual run's record:```shpython3 scripts/record-ci-validation.py RUN_ID --scope cli --pr PR_NUMBER@@ -430,53 +463,24 @@ Queue rejection, a changed head/target, missing coverage, failed/skipped requirejobs or a lost response cannot report success. An uncertain enqueue is inspectedwithout replaying the mutation. If the wait budget expires, follow that same queuedPR; do not start another validation or merge request. GitHub enforces the required-check and queue after rollout. Existing `--run/--scope` commands automatically use-queue mode when the branch has that rule; legacy direct mode is retained for the-bootstrap merge only and requires unchanged reviewed main.+check and the queue. Do not use `--run/--scope`: that legacy direct mode depends on+the collector's pre-#991 job lists.## Automatic CI and rollout`ci.yml` runs on every PR revision and on `merge_group: checks_requested`.`scripts/ci-plan.py` reads the complete immutable Git diff, including both sidesof renames, rather than GitHub's limited file list. PR runs check out the exact-head; queue runs check out the combined candidate. Unknown paths fail planning-until their checks are registered. CI uses read-only repository permissions and-executes no contributor code with release credentials.--Agents should open a draft PR while implementing and run targeted local checks-before marking it ready. Draft pushes run workflow lint and repository process-checks; expensive component suites wait for `ready_for_review`. The plan still-records all affected suites, and `ci/required` stays blocked while the PR is a-draft. Marking ready runs complete affected validation for that exact head;-returning to draft defers component checks on later pushes. Keep a ready PR stable while CI and-independent review finish. Push again when a correction is needed, rather than-repeatedly dispatching the same full suite. Merge candidates and explicit runs-always execute their complete selected scopes.--Coarse component selection retains the existing complete CLI coverage/native-matrix and Desktop inventory on both platforms. Shared CLI and process/workflow-inputs conservatively select all core suites; source boundaries are not narrowed-as part of rollout. CLI changes also select Mobile's protocol contracts; changes to-the shared planner or CI job graph select every component. Website, Store-experiences/browser, logging, OS source contracts,-provider, firmware host tests, Mobile VM tests and daemon contracts are selected-for their declared inputs. Physical devices, full OS images/VM journeys, real engines,-visual inspection and other acceptance selected by review remain separate required-validation; automatic host tests do not substitute for those.--The dependency map determines test coverage, not workflow cancellation groups:--```mermaid-flowchart LR- CLI[CLI protocols and runtime APIs] --> Desktop[Desktop contracts]- CLI --> Phone[Mobile protocol contracts]- CLI --> Companions[Companion cross-imports]- CLI --> TUI[TUI protocol and daemon integration]- Fixtures[Shared desk and daemon fixtures] --> Core[Core and Mobile checks]- Store[Store runtimes and viewers] --> Desktop- Store --> Browser[Experience and authoring browser checks]- Planner[Shared CI planner and job graph] --> All[Every component]-```+head; queue runs check out the combined candidate. Each changed path selects only+its own component's suite ([PR CI](#pr-ci)); paths outside a component select no+suite. CI uses read-only repository permissions and executes no contributor code+with release credentials.++Cross-component effects are not tested on PRs. A CLI change does not run the+Desktop, TUI, companion or mobile suites even where they read CLI protocol files.+When a change alters a protocol, wire format or shared fixture another component+reads, say so in the PR and run that component's tests locally or with+**CI → Run workflow** (`scope=desktop`, `tui`, `companions` or `mobile`).Separate PR groups prevent cross-cancellation, but still share available runnercapacity. Related branches validate independently; the merge queue validates@@ -485,11 +489,10 @@ shared interfaces and frequently changed files. Main background validation maycoalesce pending snapshots; distinct merge candidates must not share that group.The stable `ci/required` job runs even after an upstream failure. It accepts only-successful selected jobs and complete shard summaries. Deliberately inapplicable-jobs may be skipped; a required skip, cancellation, failure or missing result fails-integration. Manual `full` retains its prior core-suite meaning; `all` selects every-automatic suite. Manual dispatch is extra evidence and cannot substitute for the-automatic PR/merge-group gate.+successful selected jobs. Deliberately inapplicable jobs may be skipped; a required+skip, cancellation, failure or missing result fails integration. Manual `full`+selects the five core suites; `all` selects every suite. Manual dispatch is extra+evidence and cannot substitute for the automatic PR gate.Enable main's queue only after passing automatic CI and a disposable-branch queuetrial. Start with three candidate builds, squash, ALLGREEN, a minimum of one PR and@@ -505,11 +508,15 @@ locks belong to work jobs, leaving summaries and the always-running final gate igate waiting for a runner cannot block the next revision. Checks no longer selectedby the new plan may finish, including when a ready PR returns to draft; they cannotvalidate a newer head. Explicit runs and distinct candidates have independent-concurrency identities. Broad main E2E-finishes its active snapshot and retains the newest pending snapshot; nightly and-explicit E2E requests are separate and preserved. A passing older snapshot does-not validate newer source. Broad E2E remains advisory until reliable, and missing-required acceptance still prevents an authorized release.+concurrency identities. CLI end to end runs nightly and on demand; it no longer+runs after each merge. A passing older run does not validate newer source. It+remains advisory, and missing required acceptance still prevents an authorized+release.++OS workflows run only when started by hand. Pushing to an `os/**`, `os-polish/*` or+`os-candidate/*` branch builds nothing. Start **Harness OS** (or the specific OS+workflow) on the branch once, with the inputs the change needs, and reuse its+`image_run_id` for follow-up assessments instead of building again.Hourly **CI health** retains a bounded latest-100-run workload sample, executionobserved in cancelled runs and the age/source of completed main E2E. These are jobdiff --git a/scripts/ci-plan.py b/scripts/ci-plan.pyindex 69f33f123..06175a6a3 100644--- a/scripts/ci-plan.py+++ b/scripts/ci-plan.py@@ -2,8 +2,8 @@"""Select fast CI suites from a complete Git diff and verify their final results.A PR runs the unit suite of each component it changes, and nothing else. Merge-groups run only the plan and process checks: the PR run already covered the code,-and the release workflows carry native, macOS, browser and end-to-end acceptance.+groups run only the plan and process checks: the PR run already covered the code.+Checks outside PR CI are listed in docs/validation-and-release.md#what-pr-ci-does-not-run."""import argparseimport json