Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.
Rust ai-agentsai-gatewayclaude-codecodexcoding-agentcost-optimizationdeepseekllm-gatewayllm-routerlocal-firstmodel-routingrusttask-routingtauri
fix(e2e): preserve control-shell startup diagnostics on timeout (#203)
aias0080ab44102dcb
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.mdindex a924a91c..43d234c1 100644--- a/docs/code-map/testing.md+++ b/docs/code-map/testing.md@@ -247,6 +247,12 @@ Ordinary local/SSH totals and module reports share Cargo-target attribution in[runner regressions](../../scripts/test-remote-rust.py) exercise worker results andcompact waits with nested child summaries, failed and unfinished parents, and multiple targets.+The [control-shell startup fixture](../../tools/product-e2e/tests/control_shell_support/startup.rs)+retains bounded private failure evidence before temporary-root cleanup; its+[regressions](../../tools/product-e2e/tests/control_shell_support/startup_tests.rs)+cover timeout, early exit, normal readiness and incomplete diagnostics. Follow the+[diagnostic guide](../diagnostic-troubleshooting.md) for retention and interpretation.+Shared private filesystem behavior has one [platform contract suite](../../crates/platform/src/private_fs/tests.rs): exclusive creation, protected handle reads, permission preservation and identity-bound cleanup. Storage and Worker tests keep domain errors, recovery and lifecycle assertions; the native Windows cases additionally exercise ACLs and held handles.The [Windows iteration route](../validation-routing.md#windows-iteration-and-release-validation)diff --git a/docs/diagnostic-troubleshooting.md b/docs/diagnostic-troubleshooting.mdindex b1cb9902..5b9b690f 100644--- a/docs/diagnostic-troubleshooting.md+++ b/docs/diagnostic-troubleshooting.md@@ -33,6 +33,19 @@ Unit tests that do not start the diagnostics runtime do not generate product logDebug does not relax log-content boundaries. Do not add bodies, credentials, complete configuration or raw third-party output for temporary diagnosis. Before sharing, distinguish safe product JSONL from private merged output, real configuration and request evidence. Do not upload an entire test root.+The `control_shell` integration test retains startup failures under+`target/product-e2e-evidence/control-shell-startup-*/` before its temporary fixture+and daemon guard are cleaned up. The failure message names the private directory;+`report.json` records the child PID/status, original ten-second readiness budget,+observed boot/stage and missing or truncated files. Each retained stdout, stderr+and current daemon JSONL tail is limited to 64 KiB. The daemon uses a fresh Debug+diagnostics root, separate from the rejected role-composition check. No configuration+or data files are copied. Raw output tails are private and must not be posted directly.+An available diagnostic snapshot does not prove an asynchronous writer flush or+absence of lost events; missing, partial or ambiguous boot evidence remains explicit.+The managed runner preserves this non-debug target directory with its run artifacts.+The snapshot explains the last observed startup boundary, not the cause of a timeout.+For upstream authentication or gateway rejection, correlate Debug `upstream_wire` by `request_token` and `attempt_index`. Request events record whether Bearer / x-api-key is present, Authorization byte length, Content-Length, the actual `native_model` with bounded length/characters, and reasoning effort, Messages thinking type/budget and boolean switches read from the final serialized template. Unknown tiers are `other`; absence and an unavailable projection remain distinct. These facts describe prepared parameters, not delivery or the provider's internal computation. Response and attempt terminal events record HTTP status, actual H1/H2, Content-Type, safe `provider_error` and HMAC correlation of the upstream request ID, without retaining header values, addresses, bodies or raw provider error text. `request_kind=decision_service` uses the same safe channel for actual decision-service calls; `completed`/`failure` denotes the parsed terminal result. Combine upstream request and downstream headers/body facts in `attempt_end.commits` with the next attempt/binding in `fallback` to determine why retries or budget exhaustion occurred before body delivery, or why delivery ended in failure after it began. Missing optional fields in old records mean unknown. If request events are absent, first check levels, diagnostic loss and candidate-preparation failures; absence does not prove another upstream rejection.## Stream decoding failures and private replaydiff --git a/tools/product-e2e/tests/control_shell.rs b/tools/product-e2e/tests/control_shell.rsindex 96be5a66..655acf94 100644--- a/tools/product-e2e/tests/control_shell.rs+++ b/tools/product-e2e/tests/control_shell.rs@@ -2,15 +2,15 @@#[path = "../../../crates/daemon/tests/support/isolated_agent_environment.rs"]mod child_environment;+#[path = "control_shell_support/startup.rs"]+mod startup;use std::collections::{BTreeMap, BTreeSet};use std::fs::File;-use std::io::{BufRead, BufReader, Read, Write};-use std::os::unix::fs::FileTypeExt;+use std::io::{BufRead, BufReader, Write};use std::os::unix::net::UnixStream;use std::path::{Path, PathBuf};use std::process::{Child, Command, Output, Stdio};-use std::time::{Duration, Instant};use hiroute_application_api::{ClientHelloV1, LOCAL_CONTROL_SCHEMA_V2, LocalControlWireRequestV2, MACHINE_ENVELOPE_SCHEMA_V2,@@ -92,6 +92,8 @@ fn control_shell_uses_real_hiroute_and_hirouted_processes() {let mut command = Command::new(&binaries.hirouted);child_environment::configure(&mut command, &agent_home);+ let capture = startup::StartupCapture::new(temporary.path()).unwrap();+ capture.configure(&mut command).unwrap();let daemon = command.args(["--role",@@ -102,12 +104,16 @@ fn control_shell_uses_real_hiroute_and_hirouted_processes() {runtime_root.to_str().unwrap(),]).env("PATH", agent_path)- .stdout(Stdio::piped())- .stderr(Stdio::piped()).spawn().expect("launch production hirouted control role");let mut daemon = DaemonGuard(daemon);- wait_for_endpoint(&endpoint, &mut daemon.0);+ capture+ .wait_for_endpoint(+ &endpoint,+ &mut daemon.0,+ &workspace.join("target/product-e2e-evidence"),+ )+ .unwrap_or_else(|failure| panic!("{failure}"));let mut stalled_peer = UnixStream::connect(&endpoint).expect("stalled Local Control peer");stalled_peer.write_all(b"{").unwrap();@@ -500,27 +506,6 @@ fn durable_store_snapshot(storage_root: &Path) -> BTreeMap<String, String> {snapshot}-fn wait_for_endpoint(endpoint: &Path, child: &mut Child) {- let deadline = Instant::now() + Duration::from_secs(10);- loop {- if endpoint- .metadata()- .is_ok_and(|metadata| metadata.file_type().is_socket())- {- return;- }- if let Some(status) = child.try_wait().unwrap() {- let mut stderr = String::new();- if let Some(mut stream) = child.stderr.take() {- let _ = stream.read_to_string(&mut stderr);- }- panic!("hirouted exited before readiness: {status}; stderr: {stderr}");- }- assert!(Instant::now() < deadline, "hirouted readiness timed out");- std::thread::sleep(Duration::from_millis(25));- }-}-fn build_product_binaries(workspace: &Path) -> ProductBinaries {if std::env::var_os("HIROUTE_VALIDATION_EXECUTION").is_some() {let directory = PathBuf::from(diff --git a/tools/product-e2e/tests/control_shell_support/startup.rs b/tools/product-e2e/tests/control_shell_support/startup.rsnew file mode 100644index 00000000..054d6460--- /dev/null+++ b/tools/product-e2e/tests/control_shell_support/startup.rs@@ -0,0 +1,238 @@+//! Private, bounded failure snapshots for the control-shell startup assertion.++use std::fs::{File, OpenOptions};+use std::io::{self, Read, Seek, SeekFrom, Write};+use std::os::unix::fs::{DirBuilderExt, FileTypeExt, OpenOptionsExt, PermissionsExt};+use std::path::{Path, PathBuf};+use std::process::{Child, Command};+use std::time::{Duration, Instant};++use serde_json::{Value, json};++const READINESS_TIMEOUT: Duration = Duration::from_secs(10);+const SNAPSHOT_BYTES: u64 = 64 * 1024;++pub(super) struct StartupCapture {+ root: PathBuf,+}++impl StartupCapture {+ pub(super) fn new(fixture: &Path) -> io::Result<Self> {+ let root = fixture.join("control-startup");+ std::fs::DirBuilder::new().mode(0o700).create(&root)?;+ Ok(Self { root })+ }++ pub(super) fn configure(&self, command: &mut Command) -> io::Result<()> {+ // Regular files cannot fill an undrained pipe or wait for a descendant's EOF.+ // They live only with the fixture; failure retention copies bounded tails.+ command+ .stdout(private_file(&self.root.join("stdout"))?)+ .stderr(private_file(&self.root.join("stderr"))?)+ // This fresh root excludes the earlier rejected role=all boot.+ .arg("--diagnostics-root")+ .arg(self.root.join("diagnostics"))+ .args(["--diagnostic-level-override", "debug"]);+ Ok(())+ }++ pub(super) fn wait_for_endpoint(+ &self,+ endpoint: &Path,+ child: &mut Child,+ evidence_root: &Path,+ ) -> Result<(), String> {+ let started = Instant::now();+ let deadline = started + READINESS_TIMEOUT;+ loop {+ if endpoint+ .metadata()+ .is_ok_and(|metadata| metadata.file_type().is_socket())+ {+ return Ok(());+ }+ let (reason, status) = match child.try_wait() {+ Ok(Some(status)) => ("hirouted exited before readiness", status.to_string()),+ Err(error) => (+ "hirouted readiness status unavailable",+ format!("{:?}", error.kind()),+ ),+ Ok(None) if Instant::now() >= deadline => {+ ("hirouted readiness timed out", "running".to_owned())+ }+ Ok(None) => {+ std::thread::sleep(Duration::from_millis(25));+ continue;+ }+ };+ return Err(self.failure(+ reason,+ &status,+ child.id(),+ started.elapsed(),+ evidence_root,+ ));+ }+ }++ fn failure(+ &self,+ reason: &str,+ status: &str,+ pid: u32,+ elapsed: Duration,+ evidence_root: &Path,+ ) -> String {+ match self.preserve(reason, status, pid, elapsed, evidence_root) {+ Ok(path) => format!(+ "{reason}; child={status}; private evidence: {}",+ path.display()+ ),+ Err(error) => format!(+ "{reason}; child={status}; evidence unavailable: {:?}",+ error.kind()+ ),+ }+ }++ fn preserve(+ &self,+ reason: &str,+ status: &str,+ pid: u32,+ elapsed: Duration,+ evidence_root: &Path,+ ) -> io::Result<PathBuf> {+ std::fs::DirBuilder::new()+ .recursive(true)+ .mode(0o700)+ .create(evidence_root)?;+ let directory = tempfile::Builder::new()+ .prefix("control-shell-startup-")+ .permissions(std::fs::Permissions::from_mode(0o700))+ .tempdir_in(evidence_root)?;+ // Keep even an incomplete export if a later write fails.+ let destination = directory.keep();+ let mut files = serde_json::Map::new();+ let mut diagnostics = json!({"state": "unavailable", "current_boot": null});+ for (source, name) in [+ ("stdout", "stdout.tail"),+ ("stderr", "stderr.tail"),+ (+ "diagnostics/daemon/current.jsonl",+ "daemon-current.tail.jsonl",+ ),+ ] {+ let metadata = match read_tail(&self.root.join(source)) {+ Ok((bytes, length, offset)) => {+ private_file(&destination.join(name))?.write_all(&bytes)?;+ if name == "daemon-current.tail.jsonl" {+ diagnostics = diagnostic_summary(&bytes, offset > 0);+ }+ json!({+ "state": "captured", "source_bytes": length,+ "offset": offset, "captured_bytes": bytes.len(),+ "truncated": offset > 0 || bytes.len() as u64 != length,+ })+ }+ Err(error) => {+ json!({"state": "unavailable", "error": format!("{:?}", error.kind())})+ }+ };+ files.insert(name.into(), metadata);+ }+ let report = json!({+ "reason": reason, "pid": pid, "child_status": status,+ "elapsed_ms": elapsed.as_millis(), "deadline_ms": READINESS_TIMEOUT.as_millis(),+ "snapshot_bytes_per_file": SNAPSHOT_BYTES,+ "files": files, "diagnostics": diagnostics,+ // The writer is asynchronous. A live snapshot never establishes a flush+ // or absence of dropped events, even if every retained line parses.+ "writer_flush": "unknown", "snapshot_before_cleanup": true,+ });+ private_file(&destination.join("report.json"))?+ .write_all(&serde_json::to_vec_pretty(&report)?)?;+ Ok(destination)+ }+}++fn private_file(path: &Path) -> io::Result<File> {+ OpenOptions::new()+ .write(true)+ .create_new(true)+ .mode(0o600)+ .open(path)+}++fn read_tail(path: &Path) -> io::Result<(Vec<u8>, u64, u64)> {+ // Refuse special files: diagnostics collection must never wait on a FIFO.+ if !std::fs::symlink_metadata(path)?.file_type().is_file() {+ return Err(io::Error::from(io::ErrorKind::InvalidInput));+ }+ let mut file = File::open(path)?;+ let length = file.metadata()?.len();+ let offset = length.saturating_sub(SNAPSHOT_BYTES);+ file.seek(SeekFrom::Start(offset))?;+ let mut bytes = Vec::new();+ file.take(length - offset).read_to_end(&mut bytes)?;+ Ok((bytes, length, offset))+}++fn diagnostic_summary(bytes: &[u8], truncated: bool) -> Value {+ let mut boots = std::collections::BTreeSet::new();+ let mut records = 0;+ let mut invalid = 0;+ let mut last_stage = None;+ let mut level_applied = false;+ for line in bytes+ .split(|byte| *byte == b'\n')+ .filter(|line| !line.is_empty())+ {+ let Ok(record) = serde_json::from_slice::<Value>(line) else {+ invalid += 1;+ continue;+ };+ let Some(boot) = record["boot_id"]+ .as_str()+ .filter(|boot| boot.len() == 32 && boot.bytes().all(|byte| byte.is_ascii_hexdigit()))+ else {+ invalid += 1;+ continue;+ };+ boots.insert(boot.to_owned());+ records += 1;+ level_applied |= record["event"]["level_applied"].is_object();+ for kind in ["stage_begin", "stage_end", "stage_progress"] {+ if let Some(stage) = record["event"][kind]["stage"].as_str()+ && [+ "root_validate",+ "resident_lock",+ "artifact_validate",+ "spawn",+ "ready_wait",+ "catalog_load",+ "gateway_start",+ "worker_revalidate",+ "control_bind",+ "publication_reconcile",+ ]+ .contains(&stage)+ {+ last_stage = Some(stage.to_owned());+ }+ }+ }+ let single_boot = boots.len() == 1;+ json!({+ "state": if records == 0 { "unavailable" } else if truncated || invalid > 0 || !single_boot || !level_applied { "partial" } else { "available" },+ "current_boot": if single_boot { boots.first() } else { None },+ "last_observed_stage": if single_boot { last_stage } else { None },+ "level_applied_observed": level_applied,+ "record_count": records, "invalid_record_count": invalid,+ "truncated": truncated,+ })+}++#[cfg(test)]+#[path = "startup_tests.rs"]+mod tests;diff --git a/tools/product-e2e/tests/control_shell_support/startup_tests.rs b/tools/product-e2e/tests/control_shell_support/startup_tests.rsnew file mode 100644index 00000000..095f3ffe--- /dev/null+++ b/tools/product-e2e/tests/control_shell_support/startup_tests.rs@@ -0,0 +1,203 @@+use super::*;+use crate::DaemonGuard;+use std::os::unix::fs::PermissionsExt;+use std::os::unix::net::UnixListener;++const BOOT: &str = "0123456789abcdef0123456789abcdef";+const SECRET: &str = "private-startup-sentinel";++fn spawn(capture: &StartupCapture, script: &str) -> DaemonGuard {+ let mut command = Command::new("/bin/sh");+ command.args(["-c", script]);+ capture.configure(&mut command).unwrap();+ DaemonGuard(command.spawn().unwrap())+}++fn evidence(root: &Path) -> (PathBuf, Value) {+ let paths: Vec<_> = std::fs::read_dir(root).unwrap().collect();+ assert_eq!(paths.len(), 1);+ let path = paths[0].as_ref().unwrap().path();+ let report = serde_json::from_slice(&std::fs::read(path.join("report.json")).unwrap()).unwrap();+ (path, report)+}++#[test]+fn stalled_startup_retains_bounded_private_evidence_after_fixture_cleanup() {+ let fixture = tempfile::tempdir().unwrap();+ let retained = tempfile::tempdir().unwrap();+ let capture = StartupCapture::new(fixture.path()).unwrap();+ let logs = capture.root.join("diagnostics/daemon");+ std::fs::create_dir_all(&logs).unwrap();+ let records = [+ json!({"boot_id": BOOT, "event": {"level_applied": {"level": "debug"}}}),+ json!({"boot_id": BOOT, "event": {"stage_begin": {"stage": "root_validate"}}}),+ ];+ std::fs::write(+ logs.join("current.jsonl"),+ format!("{}\n{}\n", records[0], records[1]),+ )+ .unwrap();+ std::fs::write(fixture.path().join("configuration-secret"), SECRET).unwrap();+ let mut child = spawn(+ &capture,+ &format!(+ "dd if=/dev/zero bs=1024 count=128 2>/dev/null; printf '{SECRET}' >&2; exec sleep 60"+ ),+ );+ let started = Instant::now();+ let failure = capture+ .wait_for_endpoint(+ &fixture.path().join("absent.sock"),+ &mut child.0,+ retained.path(),+ )+ .unwrap_err();+ assert!(started.elapsed() >= READINESS_TIMEOUT);+ assert!(started.elapsed() < READINESS_TIMEOUT + Duration::from_secs(5));+ assert!(failure.starts_with("hirouted readiness timed out"));+ assert!(!failure.contains(SECRET));+ assert!(+ child.0.try_wait().unwrap().is_none(),+ "snapshot must precede guard cleanup"+ );+ drop(child);+ drop(fixture);+ let (path, report) = evidence(retained.path());+ assert_eq!(report["child_status"], "running");+ assert_eq!(report["deadline_ms"], 10_000);+ assert_eq!(report["diagnostics"]["current_boot"], BOOT);+ assert_eq!(+ report["diagnostics"]["last_observed_stage"],+ "root_validate"+ );+ assert_eq!(report["diagnostics"]["state"], "available");+ assert_eq!(report["writer_flush"], "unknown");+ assert_eq!(+ report["files"]["stdout.tail"]["captured_bytes"],+ SNAPSHOT_BYTES+ );+ assert_eq!(report["files"]["stdout.tail"]["truncated"], true);+ assert_eq!(+ std::fs::read_to_string(path.join("stderr.tail")).unwrap(),+ SECRET+ );+ assert_eq!(+ std::fs::metadata(&path).unwrap().permissions().mode() & 0o777,+ 0o700+ );+ let mut names = Vec::new();+ for entry in std::fs::read_dir(path).unwrap() {+ let entry = entry.unwrap();+ assert_eq!(+ entry.metadata().unwrap().permissions().mode() & 0o777,+ 0o600+ );+ names.push(entry.file_name().to_string_lossy().into_owned());+ }+ names.sort();+ assert_eq!(+ names,+ [+ "daemon-current.tail.jsonl",+ "report.json",+ "stderr.tail",+ "stdout.tail"+ ]+ );+}++#[test]+fn early_exit_retains_status_without_printing_child_output() {+ let fixture = tempfile::tempdir().unwrap();+ let retained = tempfile::tempdir().unwrap();+ let capture = StartupCapture::new(fixture.path()).unwrap();+ let mut child = spawn(&capture, &format!("printf '{SECRET}' >&2; exit 7"));+ // An inherited/duplicated writer remains open. Collection cannot wait for EOF.+ let _open_writer = OpenOptions::new()+ .append(true)+ .open(capture.root.join("stderr"))+ .unwrap();+ let started = Instant::now();+ let failure = capture+ .wait_for_endpoint(+ &fixture.path().join("absent.sock"),+ &mut child.0,+ retained.path(),+ )+ .unwrap_err();+ assert!(started.elapsed() < Duration::from_secs(5));+ assert!(failure.starts_with("hirouted exited before readiness"));+ assert!(!failure.contains(SECRET));+ assert_eq!(child.0.try_wait().unwrap().unwrap().code(), Some(7));+ drop(child);+ drop(fixture);+ let (path, report) = evidence(retained.path());+ assert!(report["child_status"].as_str().unwrap().contains('7'));+ assert_eq!(report["diagnostics"]["state"], "unavailable");+ assert_eq!(+ report["files"]["daemon-current.tail.jsonl"]["error"],+ "NotFound"+ );+ assert_eq!(+ std::fs::read_to_string(path.join("stderr.tail")).unwrap(),+ SECRET+ );+}++#[test]+fn ready_socket_keeps_the_child_alive_and_creates_no_evidence() {+ let fixture = tempfile::tempdir().unwrap();+ let capture = StartupCapture::new(fixture.path()).unwrap();+ let mut child = spawn(&capture, "exec sleep 60");+ let endpoint = fixture.path().join("ready.sock");+ let _listener = UnixListener::bind(&endpoint).unwrap();+ let retained = fixture.path().join("evidence");+ capture+ .wait_for_endpoint(&endpoint, &mut child.0, &retained)+ .unwrap();+ assert!(child.0.try_wait().unwrap().is_none());+ assert!(!retained.exists());+ drop(child);+ drop(_listener);+ fixture.close().unwrap();+}++#[test]+fn incomplete_or_ambiguous_diagnostics_are_not_claimed_complete() {+ let valid = json!({"boot_id": BOOT, "event": {"stage_begin": {"stage": "control_bind"}}});+ let summary = diagnostic_summary(format!("{valid}\n{{broken").as_bytes(), true);+ assert_eq!(summary["state"], "partial");+ assert_eq!(summary["invalid_record_count"], 1);+ assert_eq!(summary["last_observed_stage"], "control_bind");+ assert_eq!(summary["level_applied_observed"], false);+ let second = json!({"boot_id": "ffffffffffffffffffffffffffffffff", "event": {}});+ let summary = diagnostic_summary(format!("{valid}\n{second}\n").as_bytes(), false);+ assert_eq!(summary["state"], "partial");+ assert!(summary["current_boot"].is_null());+ assert!(summary["last_observed_stage"].is_null());+ assert_eq!(diagnostic_summary(b"", false)["state"], "unavailable");+}++#[test]+fn evidence_failure_preserves_the_startup_failure_and_refuses_special_files() {+ let fixture = tempfile::tempdir().unwrap();+ let capture = StartupCapture::new(fixture.path()).unwrap();+ let mut child = spawn(&capture, "exit 9");+ let invalid_destination = fixture.path().join("not-a-directory");+ std::fs::write(&invalid_destination, "").unwrap();+ let failure = capture+ .wait_for_endpoint(+ &fixture.path().join("absent.sock"),+ &mut child.0,+ &invalid_destination,+ )+ .unwrap_err();+ assert!(failure.starts_with("hirouted exited before readiness"));+ assert!(failure.contains("evidence unavailable"));+ let link = fixture.path().join("link");+ std::os::unix::fs::symlink(&invalid_destination, &link).unwrap();+ assert_eq!(+ read_tail(&link).unwrap_err().kind(),+ io::ErrorKind::InvalidInput+ );+}