Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.
Select the smallest sufficient validation
Run python3 scripts/test-plan.py --base <comparison-ref> at the repository root.
The comparison includes committed changes since the merge base, staged/unstaged edits,
and untracked files. Renames cover both old and new paths. Inspect the JSON plan before
submitting its Cargo commands through the configured validation entry. Use --full for
an explicit complete short check set. A missing or invalid base is an error, not a
reason to select nothing.
Use --since CHECKPOINT_SHA for a separate diagnostic iteration plan while retaining
the top-level final plan. See validation feedback for checkpoint
semantics, explicit focused/final run associations and the required agent feedback loop.
Selection rules
| Change | Checks |
|---|---|
| Root README/AGENTS, docs, durable-spec Markdown only | Review links and requirement consistency; no Rust build. A behavior change still requires the task's explicit acceptance tests. |
Repository .gitignore |
Publication-policy regressions and contract checks. Review ignored paths and any affected producer or packaging behavior; ignore rules alone do not change tracked product sources. |
| Known repository agent instructions and shared validation skill | Link, instruction-consistency and host-discovery checks; no product build from instruction paths alone. Known skill scripts retain their own tests; unknown or product-installed skill paths retain consumer investigation. |
apps/website, website/release workflows and their .github/scripts |
Dedicated website Node tests/build, static browser checks and publication-script checks; no Desktop frontend or Rust build. Decision API/source changes keep their own broader contract scope. |
apps/website/data/releases.json |
Website checks plus hiroute-host-runtime library tests and package Clippy, because the native upgrade reader consumes this catalog; no Desktop build. |
Release contract snapshots under contracts/releases/, their generator/PR publisher and release workflow |
Snapshot/source/index checks and isolated Git/PR-publisher regressions; no Rust or Desktop build. Release workflow edits also retain the existing website workflow checks. Full plans include these contract-record checks. |
| Release candidate workflow and bounded packaging/build/installer scripts | CPA target mapping, candidate-to-publication, Desktop inventory, standalone installer and license regressions; no Rust product suite. Website consumers keep their dedicated workflow. Changes to the pinned CPA source or runtime code retain their broader checks. |
| Desktop frontend | Frontend build and tests; this is not native Desktop evidence. |
| CLI/client core | CLI, client core and daemon tests, plus the default production smoke once. |
| Integrations | Integrations, application, CPA bridge, storage, release-facts, daemon and CLI tests, plus smoke. |
| Storage | Storage, observation, application, daemon, client core and CLI tests, plus smoke. |
| Observation | Observation, daemon, client core and CLI tests, plus smoke. |
| Daemon | Daemon and CLI tests, plus smoke. Plans running the daemon package suite also build the candidate console required by installed management tests. |
| CPA bridge | CPA bridge, daemon, integrations and CLI tests, plus smoke. |
| Gateway/core | Both Gateway packages, protocol E2E, integrations, CPA bridge, storage, release-facts, daemon, CLI and smoke; retain the distinct no-default-features loopback. |
| Native Desktop | Frontend, CLI/client core/daemon and core smoke, plus the configured-Mac desktop.compile gate and affected native behavior checks; do not automatically expand to all backend packages. Hosted success does not prove native behavior. |
| Shared Worker API and domain harness/launch contracts consumed by Desktop | Preserve backend scope and add desktop.compile, even when no Desktop file changes. This bounded mapping does not make every shared-library change a native obligation. |
| Diagnostics implementation/settings/filtering/writer and its tests | Diagnostics package tests and Clippy, plus Pilot launcher tests; no unrelated Worker/Gateway E2E or frontend build. Add the release-profile regression when changing the build default. |
| Diagnostics event/record/identity/error contracts | Full short set by default; review cross-boundary consumers. |
Hosted backend workflow, shared Linux setup and execution adapters (gateway-core.yml, ci-setup/action.yml, ci-prerequisites.py, ci-run.py, ci-shards.py) |
Full hosted matrix, because these files decide what backend validation actually runs; their unit tests alone do not prove the real Actions entry path. |
scripts/test-plan.py, scripts/test-test-plan.py, hosted adapter tests, and the operator skill .agents/skills/hiroute-integrate/SKILL.md |
Selection and shard-coverage regression tests; no Rust or frontend build when the execution adapters themselves are unchanged. |
| HOME validation routing and local/SSH Rust runner tooling | Routing, runner and Pilot launcher Python regressions; no Rust/frontend build unless the change also affects product code. |
| Pilot CPA manifest producer/consumer, reusable-build tests or selection rules | Default local Pilot checks use fake build/launch tools. In Actions (GITHUB_ACTIONS=true), affected paths and full plans additionally select the explicit test-pilot-builds.py --real-consumer command; its single real Rust case is required and cannot pass by skipping or selecting nothing. No full Desktop acceptance is added. |
| Bounded manual local GC wrapper | Wrapper and shared local-runner Python regressions; deletion policy remains in the runner. No Rust/frontend build. |
| Bounded Gateway runtime/protocol tests, routing fixtures, Responses request helper | Package Clippy, explicit target/consumer tests and coverage gates; runtime test changes also select the protocol matrix, and the Responses helper retains the coverage gate and indirect smoke consumer. Gateway runtime includes the known serde_json/preserve_order combination from #94. |
| Real Worker test entries and native-context fixtures | Bounded package compilation plus separate required product_checks; ignored native cases are never counted as ordinary Cargo success. Shared native-context fixtures select both Codex/Claude and Qoder consumers. |
| Qoder native integration, its Worker profile, and shared collaboration settings/Skill boundaries | Keep ordinary owner checks and require the Qoder main-Agent/Worker journey; model-specific Codex/Claude settings do not automatically select this gate or all older Worker journeys. |
| Shared application/domain/API, Cargo manifests/lockfile, other unclassified scripts, shared E2E harness, assets/contracts, unknown paths | Conservative full proposal until impact is understood; do not equate an unclassified path with proven cross-product impact. |
Mixed changes take the union. The tool proposes checks from paths; it cannot infer the semantics of every edit. Review the diff and consumers before starting costly checks:
- Docs only: links and consistency. Instructions do not count as implementation evidence.
- Local implementation: affected behavior, package tests and actual consumers. A shared library is not automatically a shared protocol. Logging defaults/filters do not require unrelated routing or Worker E2E. Tests that assert an explicit logging threshold remain.
- Cross-boundary contract/dependency: public API shape, serialized event/config shape, authentication, routing semantics, process lifetime or dependency changes require their consumers and explicit gates; use the full set when several boundaries are affected.
- Unclear impact: investigate callers and tests first. If impact remains unknown, retain the full set. Do not narrow merely because a test is slow or failing.
For an evidence-based adjustment, record changed behavior, affected consumers, selected
commands and why omitted checks cannot observe this change. No extra user permission is
needed for this routine decision. CI still uses the conservative generated plan; add a
bounded classification with regressions for recurring known cases instead of repeatedly
ignoring it. An implementation change inside a normally focused file can still require
--full or additional consumer/native checks when its actual semantics cross a boundary.
Embedded Markdown and product Skills are not automatically docs-only.
The mapping is implemented in scripts/test-plan.py and regression-tested by
python3 scripts/test-test-plan.py.
For focused observation-library checks on many-core hosts, explicitly use
-- --test-threads=8, matching the existing workspace DAG's per-library bound.
Each isolated store has its own query workers and a real 500 ms query deadline;
starting one test per host CPU can create dozens of concurrent SQLite stores and
interrupt otherwise bounded reads. Keep deadline/cancellation tests and their
production limits intact. Diagnose suspected scheduling failures with the same
candidate, feature graph and full selected library suite at both thread settings;
one passing isolated test does not establish contention. Run process targets
separately with their documented one-thread setting.
The final workspace DAG runs the complete Observation library at eight threads after
the isolated CPA library and before preparation/other target processes. This isolates
external workbench load without changing query limits, fixtures, assertions or coverage.
Its target records retain start/end times; historical runs without these timestamps
cannot establish precise query-phase resource contention from a later passing rerun.
Every non-empty change plan also runs the contract-convergence unit tests and
python3 scripts/check-contract-convergence.py. These are repository contract checks, not Rust
evidence, so docs-only and frontend-only changes retain their existing rust: false
classification.
Explicit native Agent product gates
product_checks records required opt-in journeys separately from ordinary commands.
The selector uses one entry per test target and bounded fixture-consumer mappings. Review
these obligations even when ordinary package tests pass: an ignored target with zero
executed cases is not product evidence. Missing installations or required environment
leave the gate failed or unexecuted, never green. The selector retains the gate when
those environment variables are absent; the product runner enforces their validity.
The Qoder entry selects all five tests in qoder_delegation: the installed user Skill
calling the public HiRoute CLI to delegate real work, native Worker skills with exact
Continue of the frozen task, independent concurrent routes with cancellation of
only owned work, native tool summarization/automatic compaction on the frozen route, and ordinary
startup using persisted additional model routes through the production Gateway.
The fourth journey requires both worker.context.tool-summary-route and
worker.context.compaction-route, including Continue from the persisted native summary.
Its harnesses value is ["qoder"]; Codex/Claude evidence does not
replace this entry. Run the printed command through the configured validation runner:
cargo test --locked -p hiroute-product-e2e --test qoder_delegation -- --ignored --nocapture --test-threads=1
The Qoder journey guide defines setup.
It requires the exact full HIROUTE_PRODUCT_CANDIDATE_SHA, an absolute installed
HIROUTE_WORKER_QODER_BINARY, and explicit existing HIROUTE_QODER_CONTEXT_HOME and
HIROUTE_QODER_CONFIG_DIR directories with normal native login. Do not copy authentication
material or fall back to an unrelated daily context. No adapter or Node installation is
required for Qoder. The fifth journey additionally requires HIROUTE_QODER_MODEL_CONTEXT_HOME and
HIROUTE_QODER_MODEL_CONFIG_DIR: a dedicated, normally logged-in acceptance context
whose settings may be changed and restored. Borrowed daily context for a read-only
Worker journey does not authorize writing persistent model providers. No transient
provider or credential override can substitute for this fifth journey.
Record all five journey outcomes and their revision, not only the
Cargo exit code. Python fixture tests check the oracle, not the installed Agent journey.
The OpenCode entry selects all seven tests in opencode_delegation: saved ordinary
Responses/Messages routes and independent restore, the installed user Skill calling
the public HiRoute CLI, native Worker tools with exact Continue for both protocols,
concurrent routing/cancellation and missing-history refusal, stale-safe static API
import, and automatic native compaction plus auxiliary calls on the frozen route.
The OpenCode journey guide
defines the selected official installation, isolated context and per-case evidence.
Run every required case; a successful build or an ignored test does not replace it.
Shared native_context_* fixtures retain their registered Codex/Claude, Qoder,
Pi, DeepSeek Harness and OpenCode native consumers;
worker_product_support and shared delegation/runtime setup retain all known Worker
consumers. Qoder-only fixtures and collaboration settings select the Qoder gate without
adding the old Codex/Claude lifecycle/read journeys. Registered journey guides and files
under docs/ remain documentation-only; changing executable fixtures is different.
native_required marks changes under apps/desktop/src-tauri/ and the known shared
Worker contracts consumed by that entry. native_checks supplies the separate
desktop.compile command template through the configured Mac, enabling desktop-pilot
(which includes desktop-runtime) to match the managed frontend/Pilot ACL configuration,
and compiling all targets, including test consumers. Run it early after shared
contracts integrate; native compilation
explains setup and evidence reuse. These commands stay outside backend commands;
hosted backend success does not execute them. A docs-only --since iteration does
not remove the top-level final obligation.
The mapping is intentionally bounded, not automatic dependency discovery. Review
new or moved contract consumers and extend it where justified. Neither field is an
umbrella for native Agent execution: native_required: false cannot waive
product_checks, and a successful headless Agent journey cannot replace required
Desktop acceptance.
Integration preflight
Use python3 scripts/test-plan.py --base BASE_SHA --integration after combining task branches.
The added integration view prints early checks without modifying the top-level final commands
or running anything. Selected Python gates precede the coverage-manifest target; the protocol
and runtime targets follow in one bounded Cargo command. Native/frontend final requirements
remain visible at the top level. See the integration skill
for branch convergence and evidence bookkeeping.
Known early consumers include Gateway adapters/model IR, runtime/protocol test changes, the Responses helper and the coverage producer/manifest/schema. This is a small supported mapping, not comprehensive dependency discovery. Unknown changes still retain conservative final expansion; no early targets is not evidence that a branch is safe. Runtime changes select the protocol matrix as well as their coverage gate because protocol semantics can invalidate cross-ingress assertions while runtime tests pass.
Command descriptor changes select both the CLI contracts and Product E2E product_oracle
targets early. Regenerate both contracts/cli/ and e2e/product/schema/: they share the
command descriptor digest, so updating only the CLI outputs leaves the product contract stale.
For a full/group plan, early tests preserve the final Cargo package scope, adding the E2E
owner if needed, and restrict execution with explicit --test targets. Focused runtime
checks preserve the known serde_json/preserve_order combination. This reduces late failures;
it does not promise a cheap cold build or complete feature equivalence in every context.
After an observed failure, --integration --collect-failures explicitly adds --no-fail-fast
to the bounded protocol/runtime command only. Review the proposed scope before submitting
through validation.py. Coverage runs first; repair a failed coverage gate before continuing.
Collection does not continue later commands, include expensive product smoke, or establish a
complete workspace failure set. Final commands retain their default fail-fast policy.
Combine --since CHECKPOINT_SHA --integration to base early checks on the iteration diff;
the top-level obligations and Cargo package context remain against BASE_SHA, so narrowing
the changed paths does not silently narrow dependency features. Both integration options are planning-only
and unavailable with hosted CI execution/output. Existing CI continues to execute the final
plan and its selection regressions, with no second smoke invocation.
Preflight success does not waive final obligations or automatically reuse evidence. Keep command, SHA, run ID, package/features/platform, outcome and remaining checks together in the existing task handoff. Small early checks may repeat inside final workspace validation; reserve costly product runs for affected acceptance paths and unresolved diagnosis. Preserve original evidence and explain intervening changes before reusing unchanged-code results.
When expansion selects the complete locked backend workspace test, the SSH workbench
can run that exact command with --schedule workspace-dag. The executor checks the
compiled target list and each target's full case-name set, runs doctests separately,
and requires the three real smoke cases through the existing smoke_cli target.
Preparation is build evidence only. Its published binaries are verified by role;
ordinary tests cannot silently build replacements. A failing target stops admission
of later targets by default, so its unexecuted set remains an obligation. Use an
explicit bounded diagnostic scope before a new full final attempt. The older
workspace-smoke schedule is an explicit rollback and does not inherit a failed
DAG run's evidence.
Failure policy and completeness
Validation phase, selected scope and failure policy are separate choices. A group/full
plan or --phase final does not automatically enable failure collection.
| Purpose | Failure policy | Evidence requirement |
|---|---|---|
| Routine iteration | Default Cargo fail-fast | Classify observed failures; an early stop leaves the failure set incomplete and remaining checks pending. |
| Explicit batch diagnosis | Caller adds --no-fail-fast for a declared scope |
Collect failures across that Cargo command's selected test executables; structured feedback still arrives only at command completion. |
| Final candidate validation | Default Cargo fail-fast | Any failed or unexecuted required check prevents final success; all selected checks and explicit gates need passing evidence. |
--no-fail-fast continues other selected test executables after a test failure. It does
not continue later validation commands: test-plan.py --run-ci returns after the first
failed command. Build failures, timeouts, interruption or missing output can leave even
an explicit collection incomplete. Claim a complete failure set only for a declared
scope whose required tests finished with recorded outcomes; unknown coverage is not
completion or passing evidence.
After failure, record observed failures and remaining or unknown coverage. Decide whether batch diagnosis needs an explicit collection run, preserving the candidate SHA and required build context. Select its scope from the diagnostic need; do not automatically restart full just to fill a report. Batch related fixes and keep unexecuted checks as final obligations.
Iteration, merge and specialist checks
Use three explicit validation phases:
- Diagnostic: inspect the current run's observed failures and execution limits. Classify branch-caused failures, exact-base failures and external blockers before editing. Use explicit batch collection when needed under the policy above; a default fail-fast run need not exhaust its selected scope before diagnosis or fixes begin.
- Focused: batch related fixes, then run only affected package/downstream checks, deterministic generators and explicit task gates. Intermediate pushed revisions are diagnostic candidates, not final evidence.
- Final: freeze the candidate and complete the selected checks once, pushing first only when managed Rust validation requires it. A focused scope stays focused at final validation; “final” does not mean full workspace. If a check fails, classify observed failures, decide whether explicit collection is needed, batch fixes and validate the affected next candidate. Remaining checks still need passing evidence before final success. Docs/tooling-only follow-ups run their own checks; preserve unchanged Rust evidence under its original SHA.
This sequencing does not waive required assertions or explicit task acceptance gates. Do not skip a selected command or unique assertion, relabel evidence from an earlier SHA, or rerun an unaffected generator merely because an unrelated test-only edit changed the candidate SHA.
- PR CI computes the affected plan against the PR base. Affected plans remain linear; complete plans run every backend target exactly once across coverage-checked hosted shards, including the E2E packages. The separate Gateway diagnostics workflow is manual/reusable; it no longer duplicates these tests on every PR. No-default-features transport testing remains distinct.
- Minute-scale stability, high connection counts and fixed-machine performance are specialist checks: run them when changing resource lifetime, buffering, transport, concurrency or performance-sensitive behavior, or when explicitly required for release.
- Real Agent/account, native Desktop and installation tests retain their own environment requirements. A synthetic peer, valid JSON contract or missing dependency is not a substitute for a successful production journey.
Examples of sufficient scope
A Debug default change in diagnostics needs package tests that cover actual event filtering, startup, persisted values, damaged settings and explicit overrides, plus Pilot launcher checks and the release-profile default regression. Expand to Desktop/native validation only when its entry, UI or lifecycle behavior changes. Do not add unrelated model requests, Worker lifecycles, Gateway protocols or frontend builds solely because diagnostics is shared.
Selection rule edits need python3 scripts/test-test-plan.py, a generated plan for the
actual changed paths, and regressions for focused, mixed, unknown and explicit-full cases.
Review the CI selection job too: it already runs these regressions before deciding whether
to start Rust/frontend jobs. No Rust build is needed to validate this Python mapping.
If a run is stopped by the user, preserve its evidence and mark it incomplete regardless of partial passing counts. Do not restart or replace it with another broad run.
Keep these quality boundaries
Do not remove the only assertion for authentication/credentials, preview zero side effects, Apply idempotency/crash recovery, precommit fallback/postcommit no replay, protocol/tool IDs, context isolation, Worker cancel/continue, content retention or bounded resource cleanup. Move an assertion to its owning component or a surviving core journey before removing its old test. A duplicate launcher can be removed; a unique failure condition cannot.
The quick smoke currently contains three real cases: Gateway controlled response,
CLI/control discovery, and invalid release rejection. It runs once in smoke_cli; all
scope/step/report assertions remain there. Explicit case selection is still available for
diagnosis, while selection correctness uses cheap validate tests. The quick set does not
claim full model-save, Agent restore, Worker lifecycle, record-cleanup or Desktop coverage.
Those behaviors continue to rely on their owning package tests and required product checks.
Full and Gateway selections also retain the complete core-routing definition check:
cargo test --locked -p hiroute-e2e --test case_shards \
core_routing_validation_covers_the_complete_unsharded_scenario -- --exact --nocapture
Submit this through the configured managed backend runner. The existing
case-selection CLI tests invoke the candidate's
validate command with the full scenario and local-process profile, without --case.
The 17 steps and 10 declared shards are contract counts, not executed business steps;
this check does not start CPA/SUT or replace production smoke. The Cargo test entry
is accepted by both managed runners without admitting arbitrary cargo run commands.
Historical sealed production smoke is explicitly ignored unless requested with its SUT. When explicitly selected, missing SUT fails. Current-candidate smoke remains the normal quick entry; it cannot be replaced by a historical result or a fixture success.
Diagnostic logs
Development/test runtimes default to Debug when no saved level exists. Product smoke uses the isolated Pilot Debug override. Before retrying a product-path failure, follow diagnostic troubleshooting to verify applied levels, inspect both roles and distinguish business failure from missing diagnostic output. Tests without a diagnostics runtime still use their assertion and runner output.
Build and evidence rules
For Rust execution use HOME-configured routing; it selects
the existing SSH workbench or local-rust runner on the execution host. test-plan.py --run-ci is restricted to Actions and
uses the independent CI adapter. Selection changes no SSH
configuration, checkout retention, capacity locks, or target directory rules.
The first reduction removes repeated executions and unused binaries. Trusted build
attestation, per-run random challenge, output identity, private directories and cleanup
remain unchanged. Do not bypass these checks using an arbitrary cached binary. Further
Build reuse is available only when explicitly retaining and leasing the same managed
backend checkout, as described in validation routing.
The smoke runner verifies a per-candidate recipe and staged binary digest for each
package. Gateway and Control each retain their own hirouted binary; a changed or
corrupt receipt fails instead of silently rebuilding. Separate checkouts still share
compiler objects only through sccache.
Report command exit and case states separately, with exact revision, expected/actual cases, and failure location. Zero selected required tests, unavailable required environments, or missing scenario results cannot produce a green verdict. Preserve old evidence under its original SHA; never relabel it after retry.