Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.
Tests as product documentation
Code map · Desktop scenarios · Agent/Worker scenarios
Execution locations owns caller HOME routing,
per-origin SSH workbenches, shared compiler capacity and separate candidate/driver
revision identities. A runner setup failure is not a product test result. Ordinary local and SSH runs
save results and private evidence before
reclaiming their managed checkout/TMPDIR, including failed or blocked runs.
Explicit retention covers diagnosis and consumers that still need build artifacts;
local lifecycle regressions and
SSH worker regressions protect both paths.
Scheduler preparation regressions
verify complete smoke receipt export without preserving executable snapshots;
the shared exporter keeps Pilot's existing target-artifact consumer paths.
The console artifact owner checks selected SSH
installed-package prerequisites before Rust execution and verifies the same-candidate
frontend from an execution-host staging directory under the runner's checkout lease.
Standard file tools supply that directory; the runner does not transport frontend bytes.
The SSH checkout owner shares retained-state admission
with recovery-command selection; continuation regressions
cover repeated source advances, preparation blocks and execution of the generated retry.
Windows development execution is owned by windows-dev.py
and its native executor, reached through
validation.py windows. HOME configuration owns machine/session/client locations;
the project owns artifact admission, exact-case execution and revision-bound evidence.
Its four temporary-directory native artifact ACL/recovery cases can execute without
replacing an existing installation; installed-product cases still require a fresh profile.
Reusable Desktop Pilot build admission belongs to
pilot-builds.py; its
lifecycle regressions cover manifest-content
identity, frozen snapshots and checks before reuse/start using fake build/launch tools.
The separate --real-consumer case is required by the affected CI selection and
checks the production Rust build-script and embedding declaration; local default
checks never compile or execute those temporary Rust programs.
The optional native App entry belongs to
pilot-native-app.py, with
fixture process regressions for ordinary
reopen, profile retention, artifact refusal and lease retirement. Actual macOS
LaunchServices/native CU and daemon/CLI acceptance remain separate from those
local entry tests; follow the App lifecycle guide.
Orphaned data preparation preserves complete
backups after explicit user authorization; it does not establish a product scenario.
Native Desktop session actions admit
the installed NSIS member and collect actual Windows UI Automation/screenshot
evidence. Public CLI readback is distinguished from a UI action, and a cross-build
replacement is distinguished from a newer-version update.
The Windows uninstall restore plan
keeps Codex root collaboration separate from its model-profile projection; its
portable unit cases cover owner selection and refusal, while the actual NSIS
uninstall remains the native evidence for restoring Agent files.
The native registration publication regression
uses a unique disposable registry key to reproduce inherited system-file permissions,
verify private publication and refuse changed retained bytes or a failed export.
It verifies a generic recovery boundary; it does not establish a supported
historical Windows release or a complete ordinary-user restoration journey.
The standalone journal cases
also prove that cutover facts stop rollback before manager effects, forward
reapplication registers only the current definition and preserves the original
backups, and changed native entries are refused before any manager call.
Current-installation failure pages and finite installer waits have bounded
component and production-style checks. Development cross-build simulations are
not supported historical Windows upgrade evidence. The general complete-source
recovery design remains; extending the development historical matrix is separate.
Choose a user promise, then the layer that can observe its failure. Keep tests beside that owner and preserve independent assertions. A focused filesystem conflict test can document a product guarantee more precisely than another broad happy-path UI journey.
What each layer proves
| Layer | Appropriate assertion | Evidence limit |
|---|---|---|
| Pure state / policy | Intent, thresholds, branch isolation and request construction | No service dispatch or production wiring |
| Component with real filesystem/database/controlled service | Ownership, rollback, authorization, cancellation and recovery | Not every production entry or OS |
| Desktop React + mock IPC | Controls, edits, pending/error feedback and emitted commands | No Tauri effects, native clipboard or actual daemon mutation |
| Desktop command authority / native compilation | ACL check, configured Mac desktop-pilot --all-targets consumer compilation |
An allowed/compiled command is not a WebView invocation or native scenario |
| Real CLI/daemon or Gateway listener | Published plans, actual upstream attempts and durable effects | Controlled upstreams do not establish live-model answer quality |
| Native Desktop / installed Agent / live provider | The selected real integration path and independently observed outcome | Bound to its exact environment, revision and executed scenarios |
| Frozen compatibility / expected-red / oracle fixture | A historical reader, intended rejection or test oracle | Not current product completion |
Start with the representative cases in the capability map. A file
named product, a golden containing green, or a zero process exit does not prove
execution. In tools/product-e2e, transaction recovery
contains expected-red composition, adapter contracts
are component evidence, and oracle tests
validate the oracle. The real installed management loop is in
publication_process.rs.
Its https-feed lifecycle
checks registered-source admission, published sessions, new Workers and exact Continue
through withdrawal, cache removal, restart and later embedded absorption. The daemon's
integration-test-hooks feature supplies only the external feed/inventory fixtures;
Gateway calls reuse the existing TLS E2E dial map. The Gateway's registered-feed TLS case connects its production transport directly to that same external OpenSSL fixture, exposing handshake failures before the installed lifecycle. Normal builds have no feed fixture
trust root or transport override. The test does not establish live production signing.
Reuse the fixture at the owning boundary
| Need | Reuse / preserve |
|---|---|
| Prepare a daemon's current sources, grants and plans | Publication Product: its Product.cli is internal Local Control notation, not the standalone CLI; protected input belongs to that fixture's daemon |
| Save an explicit HTTP model source or correct its target | Custom HTTP journey: real control/save/restart/recheck with synthetic literal/hostname targets and no directory; connection form browser checks separately cover plaintext notices, localized errors and retained input |
| Configure a real native Agent | Agent product support: shared Preview → Apply → status and bounded process capture; native argv/ownership/result interpretation remain in the leaf |
| Prove saved model configuration | Additional-model journey, Windows Pi/DSH journey, Windows Qoder account journey: ordinary native startup reads saved settings without provider/endpoint/credential overrides; agent-user-models and protected account-qoder-user-models remain separate from Worker profiles |
| Observe native routing and rejected attempts | NativeContextUpstream: count every model POST, including rejected endpoints/models/credentials; keep the ledger for positive route/history assertions |
| Prove exact Continue, concurrent routes and cancellation | Native context, boundary journey, boundary oracle guide |
| Prove restoration of old durable state | Frozen producer fixtures, crash-boundary tests: old bytes through the current consumer, not a new serializer pretending to be history |
| Upgrade connected Agents from a released package | Released grant upgrade: pass the repository and checksum-pinned v0.3.1 Linux archive, with current binaries in HIROUTE_VALIDATION_PRODUCT_BIN_DIR; released transactions create both connections, current startup preserves grants/native files, and real requests plus public session queries prove attribution across restart. Native probes and the upstream are bounded fixtures. |
| Recover interrupted Agent settings before the service receipt | Publication process runs Agent recovery at after_target and after_active; startup completes the journal before Apply replay, authenticated requests acquire the Agent, user native files remain byte-identical, and the recovered managed profile is stable across restart. |
| Locate recovery material while the service is down | Shared presentation, production-style geometry, offline CLI formatter: verified batch and source location, platform folder route and guide names; unsafe/absent locations remain unverified. Native opening/restoration needs its own installed evidence. |
| Preserve native probe failure classification | Pi probe tests, immutable launcher: mutable test bodies are interpreter inputs; executable availability and version/output failures have separate assertions |
| Exercise actual Desktop controls | Browser scenario map and native acceptance: select stable scenario IDs and require every declared outcome |
The publication fixture leases its listener address across daemon restarts; do
not substitute a released bind(0) probe. Its protected-input channel cannot be
attached to an existing Desktop. Prepare that instance first, then use real UI
mutations. Main-Agent delegation must separately invoke the actual public CLI.
The isolated subscription startup test uses the listener port lease
outside the host's outbound ephemeral range; the parent keeps that lease through
the isolated child's Gateway shutdown. On Unix it shares the publication fixture's
per-user address-lock namespace across worktrees and runner temporary directories.
The Gateway runtime fixture
holds its product and optional control listeners together during preparation,
then releases both immediately before starting the child. Releasing a bind(0)
probe before choosing the other address can assign both roles the same port.
The shared product session owns confirmation, exact-digest authority and command orchestration tests. Its permit-drop and broker-abort cases verify that cancelled callers can retry, while stale permits cannot cancel a newer prompt or another context. The console composition cases exercise the real role-all dispatcher while preserving the external Local Control Released gate. Authentication cases cover launch replay, revocation, hash-only storage and rate limits. The listener lifecycle cases verify off/failed minting without binds, actual network origins, the pre-established shell loopback path and desired configuration pending restart. Unix installer regressions cover explicit exposure validation, omitted-option preservation and configuration rollback. Network launch authentication tests cover exact origin/port, remote local-code refusal, expiry, replay and credential redaction. The console transport cases exercise independent peer admission, stalled-write permit release, progressing responses, bounded idle headers and confirmations on aged keep-alive connections through the production Hyper connection owner and shared confirmation broker. Browser scenarios run through both host transports; their production CSS screenshots include the console and Windows startup/update pages, without establishing native effects.
The Rust installation journal covers current-schema interruption recovery. Package tests verify x64 and ARM64 inventories and reject a target/executable mismatch. There is no released Windows Python producer to preserve. The frozen release reader proves that the V2 projection remains consumable by supported macOS Desktop. Windows shell origin/handoff tests live in the shell library; installed NSIS, WebView, Task and upgrade effects require the separate native acceptance inventory, bound to the actual standard-user OS, candidate and installer hash. The shell transaction regression uses real Windows file sharing to verify backup handles close before hash admission; the native installer candidate build runs it before exporting the measured bundle. The bounded NSIS journey uses the verified development bundle and configured existing standard session to check silent defaults, repeat install, console/Gateway readiness and uninstall with data retained. It verifies the installed shell against the NSIS member image hash after install, repeat and passive update. Its result does not establish WebView, Agent restore or upgrade recovery acceptance; see the execution entry. The Windows package inspector extracts the actual NSIS shell member with 7-Zip for its SHA-256 and CRT audit. The native case compares that measurement with installed bytes; the workspace executable is not the artifact identity source.
Keep borrowed user resources, writable native acceptance configuration, fresh product storage and per-run materials separate. Each product instance needs a private runtime/storage namespace. Worker task submissions recover with the original key and request, with title/detail lookup only as an aid. Agent collaboration configuration separately freezes credential-free Preview/Apply bytes in the CLI receipt owner; Windows publishes them without replacement and checks native ownership, link count and ancestors before replay. Its native regressions do not substitute for an installed Agent invoking the public CLI. CLI submission regressions cover denied local writes and uncertain responses; durable admission and native Main journeys remain separate. Expose the selected CLI and Pi's selected Node consistently to main-Agent checks and Worker launch. Never repair permissions on pre-existing user directories to satisfy a fixture; cleanup verifies the owned object's identity and preserves foreign files.
For a native run, read the ecosystem's acceptance entry for explicit inputs, writable roots, required outcomes and cleanup. Report a preparation/cleanup failure separately while retaining the first product failure. Process exit, successful setup or the model's success text cannot replace a Gateway receipt and an independent tool artifact.
Keep assertions independent of implementation layout
Build legal current state through current producers; compute expected outcomes independently. A historical journal fixture must retain its original producer bytes, including relevant feature combinations such as JSON ordering. Register recovery readers and removal conditions in the compatibility registry.
Require the capability under test rather than a fixed total number of Agents. Keep inventory completeness at the registry boundary: registry, bundle source and bundle producer must agree. Changed deterministic generators run twice; the second run must leave zero diff.
Use shared enable/adjust/disable and blocked-action recovery scenarios from the Agent feature guide. Native protocol differences alone do not justify an extra verification control. Browser fixtures, including standalone editors, select stable IDs rather than positional slices or translated labels; require nonempty selection and all expected outcomes.
Before consolidating tests, list their unique failure conditions. Preserve crash, stale-preview, cancellation, foreign-file, authorization, timeout and compatibility cases even when happy paths overlap. For a broken gate, demonstrate that a realistic bad case fails. For managed native overrides, retain a positive control showing that the conflicting native setting would otherwise be consumed.
Protocol changes reuse saved-route and native-context journeys across Responses and Messages. Assert actual request paths, trusted receipt identity, rotation and independent restore. A removed-route witness uses its previously working protocol and proves a typed authorization rejection plus zero upstream requests; an unknown HTTP path or old restore fixture does not prove route revocation or native execution.
Select and report validation
Use HiRoute validation for the
current selection, execution, failure or handoff operation. Its routes lead to
the owning guides; branch convergence remains a separate integration responsibility.
Hosted prerequisite installation belongs to
ci-prerequisites.py, invoked by the shared
ci-setup action.
test-ci-setup.py covers Ubuntu source selection
and bounded failure handling; the real hosted matrix verifies package installation.
Hosted shard inventory and harness policy belong to ci-shards.py; ci-run.py captures launch concurrency settings for the shared report. See the hosted guide for the explicit process-target exception and unknown effective concurrency. The host adapter derives admitted local/Desktop run help from the local runner parser, while retaining configured repository/jobs and Desktop origin-fetch admission.
- Run
python3 scripts/test-plan.py --base <exact-base>; use--integrationfor branch integration. Review affected consumers and explain semantic adjustments. - Follow the selected commands and CI definitions.
Use
scripts/validation.pyfor configured hosts, exact committed Rust candidates, feature plan, phase and related runs. Compile affected native consumers early; featureless Desktop checks and zero-selected tests do not satisfy that gate. - Diagnose focused failures, freeze after affected checks pass, then complete the selected final checks once. Missing required cases remain unexecuted, not green.
- Record revision, command, platform, cases, result and evidence location. Preserve reused evidence at its original SHA; assess the three slowest measured modules once without automatically refactoring the frozen candidate.
Final backend DAG admission and preparation are owned by validation-schedule.py, with regressions. CPA bridge process fixtures run in an isolated lane before other listeners; Gateway runtime consumers wait for prepared product roles. Smoke preparation and Gateway preparation bind the exact receipts consumed by admission; same-byte caches from another recipe do not replace them. Failed admission retains independent results and missing coverage.
Local/SSH cache admission in remote-rust.py connects/starts sccache under the existing capacity lease and probes effective scratch with a unique cache-miss compilation; local lifecycle coverage includes isolated real-cache cleanup and obsolete-server refusal.
Ordinary local/SSH totals and module reports share Cargo-target attribution in validation-report.py. The runner regressions exercise worker results and compact waits with nested child summaries, failed and unfinished parents, and multiple targets.
The control-shell startup fixture retains bounded private failure evidence before temporary-root cleanup; its regressions cover timeout, early exit, normal readiness and incomplete diagnostics. Follow the diagnostic guide for retention and interpretation.
Shared private filesystem behavior has one platform contract suite: exclusive creation, protected handle reads, permission preservation and identity-bound cleanup. Storage and Worker tests keep domain errors, recovery and lifecycle assertions; the native Windows cases additionally exercise ACLs and held handles.
The Windows iteration route uses managed MSVC cross-compilation and affected execution on the configured Windows development machine. The release workflow requires the reusable native Windows matrix before package builds; PR/main pushes do not start that expensive matrix. The Windows CI entry requires nonempty native inventories for primitives, shared libraries, installed foreground and Task business paths, retained low-level SCM tests, and fixed upstream clients. The Windows 11 ARM install job builds a native ARM64 package and invokes the Rust transaction; the other Windows 11 jobs retain their explicit x64 replay boundary. The canonical native link configuration also owns the Debug checks' static CRT; captured-source inputs bind its CMake toolchain, and the PE import gate rejects redistributable VC runtime dependencies before replay. The fixture producer binds exact npm integrities and official Qoder archive hashes; the feedback adapter retains the original SHA and native verdict. A build or client initialize is not evidence of tool execution, provider login, Windows 11, or another processor architecture.
Native scope dispositions own the
complete background recipe names and the hosted runtime/deferred partition.
Hosted producer and replay retain every compiler artifact and exact nonempty
inventory while deferring the three logged-owner Task bodies. The feedback adapter
checks zero execution, null exit and actual capture/list facts against the original
verified manifest; missing compilation or a failed executable gate stays red.
Its successful hosted assessment remains not_assessed for background acceptance,
with no full-scope or complete-candidate claim. Use default full for actual
logged-owner acceptance; that mode keeps every original runtime gate. See
hosted evidence limits and the adapter's
regressions before changing this boundary.
The installed management journey keeps Host collaboration receipts, native defaults and model authority separate. Its foreground doctor/logs negatives do not replace the real Task doctor/logs helper. The standard-owner Task journey executes as its own actual logged-on standard user, with no administrator group or alternate token. It requires an absent installation and SID-derived Task, then verifies native identity, Control/Gateway recovery, public service restart, packaged login lifecycle, recovery and guarded uninstall. The Task action supervisor keeps the registered arguments and persisted definition unchanged while owning a same-user business child and its complete Job. The real crash case must observe a new authenticated business process, and a requested stop must cancel recovery; Scheduler action completion or restart settings alone do not prove that behavior. The forced-stop case stalls the real protected-input reader and checks the held daemon's native exit time. The lifecycle client must retain its kernel peer proof when the stop acknowledgement times out, then mark manager stop intent before terminating that held process; malformed replies still fail closed. The shared authorization checks cover public missing/wrong-kind/stale Result/Release rejection; they do not prove valid provider authorization. WTS account lookup retains actual SID validity and equality independently of the API's input buffer capacity; follow this fixture for identity prerequisites. The self-reexecuted owner fixture starts its closed result frame on a new line, so the parent can require an exact prefix independently of libtest's pretty caption; a successful child exit alone does not establish a received result. Owner command envelopes share the public Control response bound, so real diagnostic log tails are admissible on repeated runs; other fixture metadata and stderr retain their smaller capture bounds. Its policy toggles do not establish console closure, RDP reconnection or a login trigger; those need external session evidence. The Task upgrade journey shares its original recovery, rollback and business-state assertions between the alternate-token gate and a separate actual-standard-owner gate. Its process and cleanup boundary uses ordinary process creation for the latter and refuses cleanup without the exact installation record. Replays retain prior saved Sources and use fresh intent identities; native fixture reuse requires the exact restored settings and pinned executable, with no authenticated Codex profile. The untouched settings leave the native default model to Codex so Restore needs no account catalog. Same-OS package qualification does not establish the original full Windows 11 baseline or execution of either upgrade case. The console owner keeps the GUI daemon detached for background launch and preserves inherited pipes when explicit foreground use attaches the parent console. Its native primitive cases use isolated consoles; the public foreground cases require the actual CLI to wait for natural daemon drainage after Ctrl-C and Ctrl-Break. Their test-owned installation does not replace complete-package or logged-owner Task acceptance. The complete-package foreground CPA journey executes before the required logged-owner Task, so missing WTS prerequisites cannot hide stock CPA startup, public PKCE cancellation/expiry or launch-snapshot failures. Its marker is test-owned; installer, Task and authorized outbound model traffic remain separate required evidence.
The native Source upstream owns the independent model endpoint and bounded wire receipts; public command and settings assertions stay in the business journey. Its accepted TCP sockets and the rejected package proxy observer explicitly clear nonblocking mode before bounded reads. Real delayed header/body and post-rejection peer EOF regressions cover the inherited Winsock mode without extending deadlines or weakening request validation.
The Worker dependency selection fixture binds the locked npm fixture directory before daemon startup, then submits actual discovery candidates only when their held FileIDs match the pinned CLI and Node. The transaction keeps its canonical tuple and digest contract; same-byte foreign files and duplicate candidates cannot replace an admitted dependency. The public journey rejects a noncanonical direct LocalControl request, then proves ordinary CLI input and canonical replay share one Operation and revision. CLI native input tests also preserve completed canonical replay after file deletion. The Pi locator regressions cover original npm/package paths and reparse rejection before canonicalization. Worker terminal facts publish typed result states and bounded, independently observed oracle counters; they retain neither arbitrary result text nor a guessed per-run failure cause. The cancel publication fixture waits for the complete nonce receipt within the original deadline; file existence alone does not prove that the native writer has published its contents. The shared model oracle follows the declared shell timeout units. OpenCode's Windows shell receipt uses the runner's pinned Node with encoded paths, so the same cancellation check works with PowerShell, cmd and Git Bash. Its residual binding reader distinguishes incomplete publication from fatal native identity errors. The residual oracle keeps correlated tool errors observable, with fixed stage/error facts and no native body. These diagnostics preserve the original cancellation and recovery assertions. The system PowerShell parser probe retains its original timeout while recording actual entry, parse and completion stages through bounded private captures; a forced Job cleanup never turns that timeout into successful parsing. The readonly control keeps pipeline entry and language-mode observations separate from actual write attempt and access-denial receipts; diagnostic output cannot satisfy the permission gate. Its Windows controls prove actual writable and ACL-denied behavior in both system PowerShell and pwsh; FullLanguage retains File.WriteAllText, while ConstrainedLanguage uses the native filesystem cmdlet and an independently nonce-bound typed denial. The background state fixture preserves persistent state after uninstall; the Lua journey requires an absent state at its start, while the upgrade journey retains its existing-state recovery contract. For Qoder Skill receipt mismatches, follow the retained Windows Skill binding and its HTTP challenge owner. Windows admits a directory spelling only through its held native directory/file identity, captured permissions and content checks before and after a same-candidate user-array receipt with exact body. Linux retains the literal path rule. Test-only closed observations distinguish these dimensions without publishing paths or bodies; diagnostic facts alone cannot authorize a match. The output-read controls distinguish the live file-size guard from a failed final read after the native scope stops. Their closed observations retain actual file length, I/O kind and sampling failure without exposing output. A size-related error caption alone does not prove that the file exceeded its limit or that account execution passed. The owned capture reader reads the retained file only after its native process scope stops, preserving exclusive Windows sharing and the exact byte limit. Its native causal control uses the production private-file creator; borrowed files keep their path reader. The actual Git Bash control executes the selected candidate CLI, including literal special-character drive paths, and retains exact output equality. Its required native recipe selects an explicit Git for Windows image through a validation-only input. The held installation and MSYS runtime imports determine its distribution; a compiler triplet alone cannot admit another shell. WSL or an absent shell cannot supply a passing result. The Qoder tool fixture and saved-model Skill oracle use the same Bash execution contract with separate causal controls. Their native path binding retains ordinary and proven canonical spellings, permits the stable CLI's existing hardlink, and releases Skill leases before production restoration. These fixtures do not substitute for actual account-backed tool execution. The native tool caller also rechecks the same file effects after each New/Continue execution has stopped its owned process scope. A correlated receipt observed before exit cannot replace that final-state check; the shell fixture includes a same-identity late-write control.
The same native entry's -InventoryOnly exports static scope recipes through its
actual required-test producer without Cargo, package preparation or client launch.
Protected Qoder acceptance uses that exact source inventory, including library and
installed integration-test targets; a missing saved-model recipe cannot be accepted
as complete account coverage. Its borrowed official profile retains native identity
without reading authentication bytes, and controlled upstreams prove saved routing
and tool execution separately from provider-account model inference.
Use the exact old checkout's export to compare upgrade baseline cases, filters and
harness arguments; the export itself records no runtime execution. Workstation
replay provenance binds its actual OS and original manifest/package hashes to the
producer run, so another build of the same source is not interchangeable evidence.
Code-map and local ownership-guide edits require link/owner review and the selected contract checks, not new model calls. Runtime-embedded Markdown and shipped Skills can change behavior and need their actual consumers. Instructions and compilation remain distinct from executed business evidence.
Repository agent policy under .agents/ and issue-spec/project/ is separate from
the product-installed collaboration Skill. The selector maps known project skill
scripts to their own tests; unknown skill code and runtime-embedded documents
still require consumer investigation. Repository ignore rules select publication-policy
regressions; optional ignored skills are not public-checkout test dependencies.
See instruction ownership
for pinned generation and actual host discovery, including the shared validation skill.
Complete-model-data regressions cover canonical wire identity, manual versus scoped defaults, source/default commit and restart, defaults, metadata pagination and removal dialogs, price activation races, and slow shared-session reads. The producer regressions reject uncommitted and committed same-revision corrections and rollback during bootstrap; the service reopen scenario preserves settings, acceptance and a captured old snapshot across an embedded v2 data upgrade. Directory scenarios use real generated inferred and observed records, including snapshot-bound provenance reads and retry; IPC is mocked, not native acceptance. The installed HTTPS lifecycle uses complete v2 publications, exact documented cache components, frozen main/Worker/Continue execution, removal, restart and embedded absorption.