Skip to content

Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.

25.1 KB
Raw Blame

Desktop behavior tests

Console and Windows host coverage

The V3 browser runner accepts HIROUTE_BROWSER_TRANSPORT=console to execute the same required product scenarios through the browser host seam. --console-only checks the production login/session pages with synthetic HTTP, Bearer/context headers, plaintext warnings and host capabilities, and captures zh/en light/dark at 1200×813 and 755×600. These component scenarios do not establish real daemon, native authorization or installation acceptance.

--shell-host-only adds the actual Windows startup HTML/styles and production shell settings/update states, host-only command routing and credential-free reconnect presentation. It uses synthetic host replies and captures the same language/theme/size matrix. Native origin ACLs, NSIS, Task/login effects and real upgrades keep their installed gates. It also distinguishes an unavailable Windows update catalog from installation recovery, and checks the bundled local-listener recovery instructions and usable Recheck action at both supported window sizes. Incomplete installation has a neutral heading and exact version/build installer repair, without storage retry or an unrelated recovery location.

Start with the user capability, then choose the smallest layer that can observe its contract. Node state tests explain decision rules; browser scenarios exercise the production React components and their wiring through synthetic IPC. Neither layer proves native authorization, persistence, process cleanup or real model calls.

Capability map

User capability Representative tests Unique guarantee
Restore a lost native connection while the daemon responds desktop.connection.loss-retry-recovery, desktop.connection.pending-and-failed-recovery, desktop.connection.unowned-service in shell scenarios, production captures Accurate Home/Sessions read errors, explicit recovery, duplicate prevention, pending/failure states, external ownership refusal, and zh/en light/dark geometry at both window sizes. Native Resident/daemon faults, exact reply synchronization with unchanged daemon identity, running work, old confirmation expiry and retained data require separate macOS execution.
Recover after a storage upgrade refuses startup Recovery presentation, production-style scenarios, storage owner proof Original-program reinstall requires the authoritative admitted unchanged group; unknown causes never grant downgrade. Verified source guidance names the exact batch, source directory and guides; long names wrap in both languages/themes. Windows supplies a route from the displayed log folder; macOS offers the preserved-material action. Missing/unsafe locations grant neither located recovery nor resume. Native installed upgrade, complete restoration and real requests remain separate acceptance.
Search session content and continue reading matches Session scenarios, session layout captures Results represent distinct matching sessions; keyword edits discard stale pages, failed continuation withdraws its cursor, and clearing search resumes automatic selection after an in-flight detail read. Pending indexing and complete empty results remain distinct. Chinese/English searching, results, empty, pending and retry states stay readable at 1200×813 and 755×600 using production styles. Real lookup, fresh authorization and Worker activity require native acceptance.
Connect an API through the service directory Connection flow scenarios, directory and draft rules Full branded directory, separate product drafts and keys, checked source name, cancellation, explicit model selection and inherited append behavior. Uses production styles; native save and live calls need separate acceptance.
Enable or edit Codex routing codex.enable.draft-cancel, codex.routing.fixed-binding, codex.routing.mixed-native-default in Agent scenarios Cancelling never applies; configured fixed overrides, native reasoning and the chosen default survive save; service responsibility is explained.
Configure Claude routing and collaboration claude.enable.shared-presets, agent.configuration.independent-facets, claude.collaboration.check-before-save, claude.collaboration.retry-preserves-edit Ordinary Claude entry and account Default limitations remain clear; prerequisite checks precede mutation; retry preserves edits.
Enable, recover or disable Qoder task collaboration qoder.collaboration.enable-without-model, qoder.collaboration.retry-and-restore No model connection or published Plan is required; required capability checks happen during save. Login and changed-Skill failures retain edits and enabled collaboration with the shared retry action; restore keeps model configuration unchanged.
Add and adjust Qoder model routes qoder.routing.additional-plans, qoder.routing.adjust-and-restore Cancelled drafts do not write; only selected Plans are submitted; unavailable choices stay visible; token changes and independent restores keep the other facet. Native default/provider preservation and real model calls require production acceptance.
Remove Pi or DSH routes used by the native default pi.routing.default-in-use, dsh.routing.default-in-use A blocked removal retains model routes and task collaboration, directs the user to their selected client, and never changes the native default or calls a model. Both reuse the additional-provider fixture.
Configure Agents through common interactions agent.configuration.shared-interactions; Agent status projection Codex, Claude Code and Qoder show configured routing without verification badges or extra check buttons. Ordinary navigation never probes a model; backend verification facts cannot hide actual configuration/recovery state.
Discover Agents and retain a selected local profile hermes.discovery.parallel-entry, hermes.discovery.brand-and-known-profile, agent.discovery.remember-and-revalidate in Agent scenarios Hermes stays alongside other Agents before discovery; only its details offer directory selection. Cancel/error retains context without writes; a successful selected-target read replaces the unscoped entry and enables the existing settings flow.
Prepare independent management and Hermes task routing agent.management.all-supported-families, agent.management.owned-restore, hermes.collaboration.native-model-independent Six management controls, independent owned restore, retained shared/external files and existing Hermes trigger modes without a managed model or Plan. Real installed Skill loading and Worker completion retain separate native acceptance.
Diagnose or recover an Agent codex.restore.conflict-active, agent.recovery.non-runnable A file conflict preserves current access and unrelated settings; executable diagnostics do not hide recovery.
Choose or replace a Worker installation worker.discovery.visible-only, worker.installation.harness-isolation, worker.installation.replace, worker.installation.latest-edit in routing/Worker scenarios Discovery follows visible configuration; installations stay independent; one explicit save commits the latest edit exactly once.
Resume incomplete Qoder model configuration qoder.routing.resume-pending Recovery requires trusted authority and the exact original context/Operation; it refreshes backend state, creates no new setup, preserves collaboration and does not fabricate verification.
Select Qoder as a Worker qoder.installation.single-cli A single CLI is saved through the existing confirmation flow without adapter/Node placeholders; switching execution Agents preserves each installation.
Continue editing after route save routing.save.unobserved-editable An unobserved result does not lock the editor or other routes.
Publish a smaller budget used by installed Agents routing.publish.qoder-budget-conflict, desktop.routing.qoder-budget-checkpoint, desktop.routing.additional-budget-consumers All actual Pi/DSH consumers and output reductions are visible; both immediate rejection and an accepted operation's checkpoint failure keep edits and explain disabling only Qoder model routing before publication and reconfiguration; neither disables collaboration nor calls models.
Follow a submitted operation desktop.operation.dismiss-observes, desktop.operation.identity-retry-converges, desktop.operation.repeated-submission in shell scenarios; operation rules Dismissal does not cancel observation; mismatched identity stays neutral through grace/retry; repeated accepted or unknown saves restart observation; success refreshes and dismisses. Pure tests retain identity, sequence and terminal-state edges.
Adjust presentation desktop.settings.fixed-scale; preference tests Settings retain language and appearance controls; application text stays at 100% with no size selector. Previous 150/200% browser preferences normalize to 100% without changing language or theme, including after remount. Storage failure rules remain separately testable.
Configure the shared subscription proxy from a device scan desktop.models.subscription-proxy.inline-cancel, desktop.models.subscription-proxy.read-retry, desktop.models.subscription-proxy.save-retry-return; restart hint tests The inline editor preserves scan results and selection, removes the Settings entry, prevents an unread-policy overwrite and retains failed-save input. A simulated restart reloads the scan and saved policy; native restart, persistence and provider transport require production acceptance.
Choose independent subscription sign-in from a device scan desktop.models.subscription-scan.recommended-login, desktop.models.subscription-scan.append-to-login Scan explains native credential refresh and links to the existing independent sign-in without starting authorization or saving. Changing from an append scan to independent sign-in clears the previous connection target, so a new account saves as a new connection. Chinese/English at the fixed 100% size and 755×600 protect wrapping, visible actions and the initial login explanation while session loading is delayed, including previously enlarged browser preferences. Real OAuth refresh retains its separate native gate.
Return to routing or configure a classifier desktop.routing.reactivation-models, desktop.routing.classifier-protocol An existing editor refreshes saved model names; quality uses active models; selection, curl copy and native-save outcomes are visible.
Connect decision models and custom extensions desktop.decisions.save-test-revision, desktop.decisions.discard-navigation; authentication and reference rules Save-and-test uses the newly saved credential revision; failures remain actionable, origin changes clear credentials, and discard clears retained editors.
Read retained C counts and coverage desktop.quality.c-observations in C scenarios Counts retain explicit denominators; a pending refresh clears stale counts and preserves the expanded section when results arrive. Incomplete coverage hides rates and explains unconfirmed origins; long identities and empty scopes remain readable in Chinese and English. Mock IPC covers rendering, while daemon product tests own event ingestion and retained reads.
Edit native default/custom assessments desktop.routing.native-default, desktop.routing.custom-assessment in routing scenarios; released intent adapter One native default without capability/background controls or route test; exact custom text, thresholds and reset. Side-by-side threshold controls and helper rows stay aligned when descriptions wrap. Browser IPC is synthetic; native publication/reopen and real provider diagnostics have separate evidence.
Correct an invalid collapsed connection field desktop.decisions.hidden-invalid-field Submission opens and focuses the field, scrolls the complete field and error above sticky actions, and does not save the invalid value.
Inspect models used by branch routes desktop.models.branch-route-references General-model details include regular and upgrade candidates of enabled branch routes, omit disabled routes, and open the referenced route.
Add a decision connection from an unfinished route desktop.decisions.route-detour The shell suspends the route draft, saves and returns the exact typed connection, preserves input on cancel, and never publishes automatically.
Inspect a model or open its documentation desktop.models.documentation-retains-input; model-to-route scenario Local readiness does not claim inference success; browser failure retains form input; web anchors retain their default action; route creation uses the selected model.
Change model grouping or inspect page controls desktop.models.view-switch, desktop.presentation.control-layout; presentation scenarios Production styles preserve model/connection counts, selected model and visible selection. Chinese/English and light/dark checks reject browser-default buttons and horizontal overflow from long recent-session titles at the supported small window.
Try a new model, judge its evidence and follow live routing timeline.initial-failure-retry-recovers.*, timeline.polls-only-while-displayed-and-visible, trust.scoring-states.both-groupings.*, quickstart.*.inline-connection-guard.model-change-and-back in trial scenarios A failed timeline read keeps its tab, error and Retry until recovery, while only a confirmed empty read hides the tab; polling continues only while the timeline is shown and the window is visible, after one availability probe. Per-model and per-executor groupings both show scoring coverage and complete/partial/awaiting/unscored counts. Changing models, going back or leaving the quick start asks before discarding the inline connection form; Cancel keeps the model and input. HIROUTE_BROWSER_TRANSPORT=console runs the same scenarios over the console host. Real feed retrieval, saved trials and live sessions require native acceptance.
Keep selected sessions and delayed excerpts current Live session refresh scenarios Narrow summary refresh preserves list pages, selection and scroll. Visible unresolved excerpts retry independently of timeline digests; hidden/inactive views pause. Production-shaped empty catalogs after cleanup, late responses, externally deleted/expired summaries, reopening and new-request revival preserve current body visibility.
Read exact turn ranges and recover live timeline reads Timeline presentation scenarios Assessment Agent-turn IDs resolve to display coordinates across producer epochs; unavailable or ambiguous identities remain explicit. Transient reads preserve the last view and recover with bounded backoff; real capacity failures require Retry. An extension's selected group and reason survive timeout recovery and a full refresh when a late low score arrives, without a built-in floor or protection claim. Fixed labels occlude moving cards without hiding keyboard focus.

Run the focused production-component cases with node tests/v3/visual/new-model-trial.mjs <evidence-directory> --session-live-refresh or --presentation from the Desktop directory. They use production styles and controlled IPC; they do not establish native persistence or real model outcomes.

The focused timeline pricing checks run with node tests/v3/visual/new-model-trial.mjs <evidence-directory> --timeline-pricing. They also run in the default trial suite. Production components must remove a provisional row absorbed by exact completed-request membership, retain selection, actual keyboard focus and correct counts, and show known API-equivalent subtotals alongside unknown contributions in Chinese and English. Mock IPC proves presentation, while the native model-save publication journey covers installed price edits, real Gateway valuation, frozen history and restart.

Completion focus scenarios exercise overview and lane controls with real keyboard and mouse events, including visible keyboard focus. They retain focus on an unrelated search field, a surviving turn or the page body instead of moving it because selection changed. English/light at 755×600 and Chinese/dark at 1200×813 use production components and styles; native WebView acceptance remains separate.

The historical protection and overview scenarios run with node tests/v3/visual/new-model-trial.mjs <evidence-directory> --protection. They verify scope-matched release counts and explicit unavailable reasons, a selectable 24-hour interval inside independently clipped retention boundaries, the opening selection's historical-protection outcome/previous score, and forty-turn overview navigation, viewport tracking, keyboard access and live append. Use the same HIROUTE_BROWSER_TRANSPORT=console switch for the console host. These remain production-component scenarios with synthetic data; real protection/release records and native acceptance need their own evidence. Run browser suites serially.

The trial interaction refinements run with node tests/v3/visual/new-model-trial.mjs <evidence-directory> --polish. They cover cached trust-level replay, precise-value debounce, shared reason labels, bounded two/forty-turn lanes and sticky labels, visible arrivals and unread counts, automatic following until an explicit selection, reduced motion, suggestion hierarchy, model-card facts and footer clearance across both supported sizes, languages and themes. Small-window timeline captures use the product's Compact view. Browser suites in one checkout share the Vite dependency cache and should run serially. These fixtures do not establish native or provider effects.

Task-route save scenarios also cover Pi and Qoder executor reselection: an explicit Messages choice survives clicking the selected executor and reaches the saved draft. They reuse the real Plan editor and existing IPC trace, without launching a native Agent.

The focused Agent fixture explicitly spans the application grid and keeps its evidence note outside the page viewport. Layout assertions must observe usable component width; a DOM element that exists outside a valid viewport does not establish readability.

The shell fixture mounts the real DesktopApp, including navigation, forms and observation effects. It controls only IPC responses and records boundary calls; it does not call private component callbacks. In particular, browser checks of save_classifier_openapi and open_external_url verify UI dispatch/outcomes, not the native OS action.

For an interactive presentation regression check, open product-shell.html?checks=presentation on the fixture server at 755×600 and click Run presentation checks. It runs the same two required presentation scenarios with production styles and reports each result; IPC remains mocked.

The full model-information views run with node tests/v3/visual/new-model-trial.mjs /path/to/evidence --metadata. They mount production Models, default-intent controls, metadata directory and removal dialogs with synthetic IPC. The scenarios cover Chinese/English, light/dark, 755×600 and 1200×813, retained edits after read failures, stale-cursor restart, four price components and confirmation only after all impact pages are reviewed. Actual generated inferred and observed records cover visible field markers, linked source/rule reads bound to the selected snapshot, and retry after source-read failure. Fact presentation tests preserve unknown, conditional, conflicting and not-applicable meanings. Installed CLI/daemon tests separately own persistence and actual feed/default/price effects.

Session quality presentation

Session quality names come from the exact retained execution request (same session and request ID), then the current configuration display name when available. Opaque configuration/branch IDs belong in diagnostic tooltips, not primary labels. Mixed or unknown attribution must not be presented as one model. History gaps and partial assessment evidence remain separate visible flags; an unrated stage is not assigned a score.

plan-quality-models.test.mjs protects evidence identity, missing evidence and request deduplication. The session-window browser scenarios exercise the real Sessions/PlanQuality wiring, both model stages, coverage flags, unrated copy and evidence navigation; these are mock IPC checks, not native or model-quality evidence.

Run levels

Custom API transport and error presentation use the production form in node tests/v3/visual/connection-flow.mjs /path/to/evidence. It exercises manual HTTP save without a directory, closed backend error reasons with retained input, and plaintext notices in zh/en, light/dark, at 1200×813 and 755×600. Browser IPC is synthetic; real_process_custom_http_save_recheck_and_validation_errors in the daemon publication tests separately proves actual control/save/restart behavior.

From apps/desktop, with the repository's Node 24 and installed dependencies:

npm test
npm run build
node_modules/.bin/tsc -p tests/v3/browser/tsconfig.json --noEmit
CHROME_BIN=/path/to/chrome node tests/v3/visual/agent-trust.mjs /tmp/hiroute-agent-tests

The full browser run discovers all Agent, routing/Worker and shell scenarios, plus the model-to-route scenario. Focused browser flags remain available: --claude-collaboration-only, --worker-replacement-only, --route-save-only, --subscription-proxy-only, and --shell-only (after the output directory). Unknown flags fail. For selection logic alone, run node --test tests/scenario-selection.test.mjs.

Choose the configured native Desktop/Pilot entry separately for real authorization, recovery, file changes, task cancellation and external-browser/download acceptance. See validation runner and test planner. Do not cite a mock-IPC report as native or daemon acceptance.

Keep selection and assertions stable

Scenarios carry a stable id, capability tags and a display name. IDs describe the user guarantee and remain stable when names, files or implementation details change. The reviewed required sets are independent of the discovered catalog: deleting a required scenario, selecting nothing, returning duplicate results or omitting a selected case fails. Full runs include newly added scenarios automatically. Batch size is a transport detail, never a scenario selector. Keep required-ID changes in the same review as an explicit replacement of the removed guarantee. green, red and missing execution remain distinct.

UI consistency retains static stylesheet conventions, shared Disclosure usage, localized copy, native capabilities and the public OpenAPI byte contract. The native command authority check compares all bundled Tauri handlers with the AppManifest, generated permission files and explicit main-window permissions. Adding a command requires reviewing all four together; mock IPC and native compilation cannot establish that the real WebView may invoke it. Its former source-shape assertions for Agent forms are covered by the Agent IDs above; settings, classifier actions, model links, route reactivation and operation effect wiring are covered by the shell IDs. Moving a component no longer requires preserving a callback expression, a hook dependency spelling or a source file location. Keep pure state tests when they protect independent identity, cancellation, ordering or recovery edges that one browser path cannot cover.

When reviewing churn, use git log -p -- <test> <producer> to distinguish a product contract change from fixture repair. Do not use test-count reduction as an acceptance target, or rewrite an expectation merely to match the implementation. Current fixtures follow current product types; historical recovery fixtures keep their separate registration and meaning.

Remaining work outside this batch: Observation contract constants that compare to themselves need a producer/consumer oracle; duplicated Rust process harnesses need an ownership review before extraction. Those changes must preserve their unique cancellation, identity and cleanup assertions and do not belong in these UI fixtures.