Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.
HiRoute black-box E2E
This directory holds environment-independent product-path contracts. It sits above unit,
component, and gateway-core lifecycle tests and verifies user-visible behavior across a
real TCP boundary.
E2E HTTP client
-> external HiRoute SUT
-> stock CPA Claude (one source)
-> Anthropic-native mock
-> stock CPA Codex (one source)
-> Responses-native mock
hiroute-e2e does not depend on hiroute-poc or the gateway-core Rust crate. A future
replacement of the POC binary by the production daemon/Desktop backend does not require a
scenario rewrite as long as the external CLI and HTTP product contracts remain stable.
Current coverage
scenarios/core-routing.json contains seventeen ordered
steps:
| Downstream protocol | Task | Real native mock | Expected source |
|---|---|---|---|
| Responses | simple | Anthropic /v1/messages |
Claude-compatible |
| Responses | complex | Codex /responses |
Codex subscription |
| Messages | simple | Anthropic /v1/messages |
Claude-compatible |
| Messages | complex | Codex /responses |
Codex subscription |
| Responses / Messages | large Agent envelope + simple user | Anthropic /v1/messages |
Claude-compatible |
| Responses / Messages | tool continuation | Codex /responses |
original Turn affinity hit |
| Responses | simple + Claude 500 | Claude /v1/messages, then Codex /responses |
precommit fallback to Codex |
| Responses | function output after fallback | Codex /responses, no Claude Attempt |
written-back Turn affinity hit |
| Responses | simple full-history append after complex | Codex /responses |
same Session prefix remains on Codex |
| Responses | keep first two user items, replace later history with summary | Anthropic /v1/messages |
rebuilt history reselects Claude |
| Messages | complex full-history append after simple | Anthropic /v1/messages |
same Session prefix remains on Claude |
| Messages | edit system while history text is unchanged | Codex /responses |
changed instructions reselect Codex |
| Responses | non-empty previous_response_id |
no native Attempt | explicit 400 before Planner |
Every step also asserts:
- downstream status, SSE content type, and protocol terminal event;
x-hiroute-source;- ordered Attempts actually observed by the native mock, including native path/model, response-head status, and use of that source's unique upstream authentication;
- whether a Messages/Responses protocol conversion actually occurred;
- each Attempt's routing receipt, including protocol, source, class, affinity, disposition, and status;
- accepted-source writeback after fallback, followed by a Codex affinity hit for function output without probing Claude again;
- source stability for a full visible-history append in one stable Session, and reselection after history reconstruction or instruction changes;
- early rejection of unsupported Responses server-side continuation before Planner or native upstream, with no new ledger or receipt entry.
The current implementation proves only these basic paths, sources, and limited terminal
events. It does not prove full protocol-semantic fidelity. At present,
protocol_conversion=true means only that client and native-mock protocol names differ.
The native ledger checks only the top-level messages/input shape; it does not yet compare
Vision, tool schema/choice/call/result/ID, native reasoning fields, usage/finish reason,
provider state, or the complete SSE event sequence. Do not describe the current seventeen
steps as passed protocol-conversion conformance.
The formal P0 target is defined by the Agent integration and model protocol-conversion contract and split into:
E2E-P028: resolve AgentIntegrationProfile from exact Codex/Claude Code installations and valid configuration, select the unique Responses/Messages entry, apply and roll back configuration diffs, and execute a real cross-protocol Turn;E2E-D011: semantic fidelity for the six ingress/upstream combinations across Responses, Chat Completions, and Messages;E2E-D012: isolated ingress effort and exact AgentPlan reasoning configuration rendered natively for each target and again after fallback;E2E-D013: tool IDs/continuation, provider-state affinity, normalized streaming events, in-stream errors, and zero postcommit handoff.
These items must begin as failing tests with bounded schemas, a canonical Model IR corpus, request/SSE goldens, and a native semantic ledger, then be implemented in the production Adapter. HTTP 200, final text, a target path, or merely different protocol names are not sufficient evidence.
The native mock ledger is the primary source-selection evidence. A response model or a Gateway self-reported header is only independent supporting evidence. Native mocks assert exact authentication without writing credentials to the ledger or result artifact.
Environment contract
A Scenario describes product behavior only. It cannot contain executable paths, shell
commands, ports, delays, arbitrary response bodies, or credentials. A v2 success step allows
one or two source-native Attempts and the bounded response-head statuses
200/429/500/502/503/504/529; the final Attempt must be 200 Accept. Entry-validation steps
allow only a fixed 400 + error code + 0 Attempt, proving that the request did not reach the
Planner/native upstream. Headers are limited to bounded session-id/thread-id values and
cannot inject authentication or environment orchestration. #[serde(deny_unknown_fields)],
semantic validation, and JSON Schema prevent orchestration from leaking into cases.
profiles/local-process.json permits only two fixed
references:
HIROUTE_E2E_CPA_BIN -> stock CLIProxyAPI executable
HIROUTE_E2E_SUT_BIN -> external HiRoute executable
Each run uses dynamic ports and a 0700 TempDir, generates random capabilities, and starts two
single-source CPAs, two native mocks, and one SUT. Secret-bearing files and process logs are
0600. Every subprocess starts from a separate empty 0700 work directory used as current_dir,
HOME, XDG, and TMPDIR, so stock CPA cannot search upward into a repository .env.
Subprocesses start after env_clear() with a minimal environment and do not inherit real
provider tokens. External HTTP proxies point to a loopback black hole and native sources are
loopback-only. Readiness uses connection probes, not fixed sleeps; shutdown is bounded and a
drop guard performs a second kill if necessary.
The result artifact contains product evidence only, never:
- request or model-response bodies;
- capabilities or CPA/native API keys;
- ports, temporary paths, or executable paths;
- command argv or raw process logs.
Usage
Validate the contract without prebuilt binaries:
cargo run -p hiroute-e2e -- validate \
--scenario e2e/scenarios/core-routing.json \
--profile e2e/profiles/local-process.json
Run the complete black-box path:
HIROUTE_E2E_CPA_BIN=/absolute/path/to/CLIProxyAPI \
HIROUTE_E2E_SUT_BIN=/absolute/path/to/hiroute-poc \
cargo run -p hiroute-e2e -- run \
--scenario e2e/scenarios/core-routing.json \
--profile e2e/profiles/local-process.json \
--result /tmp/hiroute-e2e-result.json
The runner allocates dynamic ports and a private temporary directory, then starts two native
mocks, two isolated stock CPAs, and an external HiRoute SUT. Across real TCP it verifies the
four protocol/source quadrants, two Agent-envelope counterexamples, Responses/Messages tool
continuation affinity, ContextHold/rebuild invalidation, early rejection of Responses
server-side continuation, and the full
Claude 500 -> Codex Accept -> function output hits Codex directly path.
Recommended gates:
cargo fmt --package hiroute-e2e -- --check
cargo check -p hiroute-e2e
cargo clippy -p hiroute-e2e --all-targets -- -D warnings
cargo test -p hiroute-e2e
Driving later features with E2E
Follow product-contract-first-red, implementation-green:
- If current fields express the behavior, add a
ScenarioStepand run it to establish a real not-yet-implemented failure. - Use the v2 bounded
attemptsfor response-head fallback. Upgrade the Rust contract andschema_versionwithe2e/schema/only when a feature requires first-chunk disconnect, semantic SSE failure, another step type, or new evidence. - Never put shell, paths, sleep, fixed ports, or secrets in a scenario; orchestration belongs to runner/profile.
- Prefer native-ledger assertions for routing behavior and add native-event sequence plus normalized-semantic assertions for protocol behavior.
- Implement product code until the same local and CI scenario is green.
- Scan the result artifact for sensitive data before retaining it as review evidence.
Good next scenarios include interleaved Turns without affinity leakage; the response-head and
first-semantic-SSE-error precommit fallback matrix; zero replay after commit and total Attempt
limits; arbitrary SSE byte slicing, long streams, cancellation, and memory bounds;
multimodal, count_tokens, compact, and usage/cost evidence; and real-local Codex CLI /
Claude Code smoke profiles.
The v2 native mock currently returns fixed text SSE for success and supports bounded
response-head failure. ScenarioStep handles only an HTTP Turn. Do not evade this boundary
with arbitrary shell fields; model a stream failure or AgentTurn as a new bounded enum and
upgrade the schema when needed.
CI layers
Public CI compiles the harness on Linux, macOS, and Windows, runs mock/unit tests, and statically validates checked-in scenarios and profiles through the CLI. This gate needs no external binary, account, or credential.
The complete process path still requires an explicit stock CPA and HiRoute SUT via the run
command above. Static validate never represents that path as executed. After building a
production SUT from the repository, expand in layers:
e2e-core-routing: build stock CPA + SUT and run the current hermetic suite;e2e-product: add fallback, tool, cancellation, memory, and related suites;e2e-real-local: manual/nightly only, using real local Agents and explicit authorization; it is not public CI and does not replace the hermetic suite.
P0 Gateway exact Oracle
schema/p0-production-manifest.json seals the final gateway-isolated contract. The manifest
locks one real Responses → Responses production smoke, its profile/scenario, and the
launcher/readiness/collector/result schemas. SHA-256 values cover canonical JSON, so an
unsealed semantic change fails before startup. Old manifests, corpora, goldens, and fixture
adversaries remain historical negative inputs and cannot prove release completion.
The runner uses only the normal
hirouted --role gateway --listen ... --publication ... --credentials ... path. The final
contract has no --fixture, --expect-status, expected_red, or skip. Each run creates
private publication/credential input, random client/provider capabilities, an unpredictable
freshness challenge, a dynamic loopback listener, and a native provider mock. The profile
pins the Gateway product source revision. Within the binary's checkout, the runner verifies
that source and build inputs have no tracked or untracked drift from that revision, then
performs a trusted cargo rustc --locked --all-features rebuild with a random build nonce and
the checkout's current target/toolchain. Launcher evidence binds revision, source-tree and
build-input digests, target, toolchain, command, artifact path, and executable SHA-256.
Legitimate builds need not share a local debug hash, while wrappers and forged build inputs
cannot qualify as the SUT. Before and after readiness probing, OS socket-owner proof requires
the dynamic listener to belong to the same live child PID. HIROUTE_E2E_SUT_REVISION adds an
optional external revision assertion.
Provider evidence retains the complete native JSON payload and exact connection count. The
collector independently reads Lifecycle, ExecutionFact, ConversationContent, and
content-free OTel JSONL. Each must have its own producer/epoch/stream identity, contiguous
sequence beginning at 1, exact channel schema digest, strongly typed product envelope that
rejects unknown fields, unique event IDs, common request correlation, no gaps/loss, and its
own terminal. Lifecycle request_finished is last; after ExecutionFact request_finished,
only independent usage_and_cache is allowed. ConversationContent has only request and
response directions, each with begin … finish; request_finished is never treated as a
content barrier. Ten exact result facts project uniquely to four ordered checkpoints. Missing,
orphaned, damaged, hard-coded, wrong-ready, wrong-payload, or incomplete-terminal evidence
fails closed.
The result schema allows only scenario green|red and separates
process_exit.test_process_code from scenario_state. The CLI returns zero only when real
production evidence establishes green. A mismatch writes a red result and exits non-zero;
startup, provenance, readiness, or collector-contract failure is also non-zero and cannot
become release proof. The full result contains a synthetic challenge and canonical content
evidence, so it is written mode 0600; stdout contains only a body-free summary.
Validate, reseal intentionally, and run the real production smoke:
cargo build -p hiroute-gateway --bin hirouted --all-features
cargo run -p hiroute-e2e -- validate \
--schema e2e/schema --scenario e2e/scenarios/p0-gateway.json
# Run only after intentionally changing a locked artifact. Run twice; the second run must
# produce no diff.
cargo run -p hiroute-e2e -- seal-production --e2e-root e2e
HIROUTE_E2E_SUT_BIN="$PWD/target/debug/hirouted" \
cargo run -p hiroute-e2e -- run \
--profile e2e/profiles/gateway-isolated.json \
--scenario e2e/scenarios/p0-gateway.json \
--result target/e2e/p0-gateway-result.json
This process freezes only one production smoke and stable extension points. The broader protocol, fault, resource, and privacy matrix remains owned by its dedicated convergence work; this one green scenario must not be extrapolated to matrix completion.
matrix/p0-gateway-coverage.json is an exact independent coverage index, not a self-reported
completion flag. A frozen typed registry defines every required row. Each row references a
registry receipt that fixes real hirouted + native-provider execution mode, test target,
exact symbol, complete source SHA-256, and assertion identity. seal-coverage only projects
that registry:
cargo run -p hiroute-e2e -- seal-coverage --e2e-root e2e
p0_gateway_matrix_coverage verifies schema, complete registry projection, source digest,
symbol/test binding, and assertion identity. Missing rows or forged execution fail. The index
contains no skip, expected_red, or manual passed state. The personal P0 listener's
transport row is loopback H1 only; release H2 benchmarks and Enterprise TLS listeners are
outside this matrix.