Skip to content

Business operating system for Claude Code — 57 skills, 21 agents, smart daemon. Unified inbox (WhatsApp/Email/Slack/Telegram), autonomous PR merge, full-AWS monitoring, revenue (Stripe+RevenueCat), e-commerce (Shopify), marketing (Klaviyo/Meta/GA4), voice (Bland/ElevenLabs), APM (Datadog/NewRelic/OTEL), YOLO mode.

Add a session-safe local failover gateway (#834)

Aurora Capitalf8a7f8d6fffe
diff --git a/claude-ops/scripts/local-failover/README.md b/claude-ops/scripts/local-failover/README.md
new file mode 100644
index 00000000..0b1bb1a4
--- /dev/null
+++ b/claude-ops/scripts/local-failover/README.md
@@ -0,0 +1,317 @@
+# Local Failover Gateway
+
+This package provides stable loopback HTTP endpoints for LLM-compatible CLIs and MCP HTTP
+clients. It selects only pre-provisioned, semantically healthy routes and never changes cloud,
+tailnet, client, credential, tmux, or existing service state by itself.
+
+Nothing in this directory is installed or activated by the repository. The example uses unused
+high loopback ports. A live install and client cutover require separate authorization and
+coordination with the Mac runtime owner.
+
+## Architecture
+
+```text
+CLI -> 127.0.0.1:<stable> -> cloud-edge primary
+ Tailscale direct secondary
+ owned SSM tunnel tertiary
+
+MCP -> 127.0.0.1:<stable> -> existing local aggregate primary
+ provisioned remote aggregate alternatives
+```
+
+The configuration array is the route order. `config.example.json` deliberately orders the LLM
+paths as `cloudflare-primary`, `tailscale-secondary`, then `ssm-tertiary`. The two optional routes
+ship with `provisioned: false`; an operator must approve and provision them before they can enter
+selection.
+
+A route is selectable only when all applicable gates pass:
+
+1. It is explicitly marked provisioned.
+2. A network-interface gate confirms the expected local route where configured.
+3. A tunnel route was started by this daemon, its port was observed free before startup, its owned
+ process is still alive, and every listener on that port belongs to the owned process group.
+4. Authenticated semantic health closes the circuit after the configured recovery hysteresis.
+
+The health engine treats `/v1/models` only as a read-only semantic response-shape check and
+immediately discards the payload. It does not discover, normalize, persist, rank, refresh, or
+rewrite models, aliases, provider inventory, auth, or cache state. An empty inventory does not by
+itself admit a route: the separately configured minimal streaming probe must also succeed. Missing
+health credentials and HTTP 401/403 are configuration-unknown, not provider failures; stale
+unknown health eventually makes the route unavailable.
+
+MCP route health uses the explicitly configured read-only HTTP `GET` `mcp_health_path`, such as the
+upstream gateway's `/_failover/status`. Success requires a bounded JSON body that explicitly reports
+a ready state or an MCP protocol with an active route. Health probes never create MCP sessions or
+send JSON-RPC requests.
+
+## Request and stream safety
+
+- Only `GET`, `HEAD`, and `OPTIONS` are eligible for bounded retry, and never after client-visible
+ bytes. The configured maximum includes the first attempt.
+- `POST`, `PUT`, `PATCH`, and `DELETE` are sent at most once. This includes model generation and
+ every MCP JSON-RPC request, even when its method name appears read-only.
+- A stream is never spliced between origins. If an origin interrupts after response headers or
+ bytes, the gateway closes the client response and records the route failure.
+- Hop-by-hop and proxy-auth headers are stripped. End-to-end authorization and MCP session headers
+ are preserved. Request bodies and response payloads are never logged.
+- An upstream `Mcp-Session-Id` is pinned in memory to its route. An unknown, expired, or unhealthy
+ pinned session receives `503 session_reconnect_required`; it is never migrated to another route.
+- Request bodies, health responses, status responses, connect time, and idle time are bounded.
+ Client header/body reads also have a deadline, and each listener has a hard concurrent-request
+ cap; excess loopback clients receive `503 gateway_overloaded` without allocating another worker.
+
+Circuit breakers start unknown, require stable successes, open after a failure threshold, use
+capped exponential cooldown, and require a failback hold. Status and Prometheus-format metrics are
+available only on each loopback listener at `/_failover/status` and `/_failover/metrics`. They omit
+upstream URLs, model names, credentials, headers, request bodies, and session IDs. Optional state
+snapshots are atomic mode-0600 informational files; health starts unknown after every restart.
+
+## Isolated validation
+
+These commands parse configuration and run deterministic mocks only. The tests bind ephemeral or
+unused loopback ports and never contact configured providers:
+
+```bash
+cd claude-ops
+python3 -m json.tool scripts/local-failover/config.example.json >/dev/null
+PYTHONPATH=scripts/local-failover python3 -m local_failover check \
+ --config scripts/local-failover/config.example.json
+python3 -m unittest discover \
+ -s scripts/local-failover/tests -p 'test_*.py' -v
+python3 -m py_compile scripts/local-failover/local_failover/*.py \
+ scripts/local-failover/vpc_paths.py
+plutil -lint scripts/local-failover/launchd/com.example.local-failover.plist.template
+```
+
+Run the module from `scripts/local-failover` or put that directory on `PYTHONPATH`:
+
+```bash
+cd scripts/local-failover
+python3 -m local_failover check --config config.example.json
+python3 -m local_failover status --config config.example.json
+```
+
+`check` never resolves or prints URL environment values. `status` performs only loopback GETs.
+`run` starts listeners, health probes, and explicitly enabled owned tunnels, so it is an activation
+step and is intentionally not run by repository tests.
+
+## Future macOS installation and rollback (not executed)
+
+The LaunchAgent template contains no credentials. LaunchAgents do not inherit an interactive shell
+environment reliably; the runtime owner must approve an existing secret-injection mechanism for the
+environment variable names referenced by config. Never write credential values into JSON, plist,
+shell history, logs, or this repository.
+
+After isolated validation and explicit installation approval, render a private plist rather than
+editing the template:
+
+```bash
+set -eu
+PACKAGE_ROOT="/absolute/path/to/scripts/local-failover"
+PYTHON_BIN="$(command -v python3)"
+CONFIG_PATH="$HOME/Library/Application Support/local-failover/config.json"
+LOG_DIR="$HOME/Library/Logs/local-failover"
+PLIST="$HOME/Library/LaunchAgents/com.example.local-failover.plist"
+install -d -m 700 "$(dirname "$CONFIG_PATH")" "$LOG_DIR"
+install -m 600 "$PACKAGE_ROOT/config.example.json" "$CONFIG_PATH"
+python3 - "$PACKAGE_ROOT" "$PYTHON_BIN" "$CONFIG_PATH" "$LOG_DIR" \
+ "$PACKAGE_ROOT/launchd/com.example.local-failover.plist.template" "$PLIST" <<'PY'
+import pathlib, sys
+from xml.sax.saxutils import escape
+root, python, config, logs, source, destination = sys.argv[1:]
+text = pathlib.Path(source).read_text()
+values = {
+ "__PYTHON_BIN__": python,
+ "__PACKAGE_ROOT__": root,
+ "__CONFIG_PATH__": config,
+ "__STDOUT_LOG__": f"{logs}/stdout.log",
+ "__STDERR_LOG__": f"{logs}/stderr.log",
+}
+for token, value in values.items():
+ text = text.replace(token, escape(value))
+path = pathlib.Path(destination)
+path.write_text(text)
+path.chmod(0o600)
+PY
+plutil -lint "$PLIST"
+PYTHONPATH="$PACKAGE_ROOT" "$PYTHON_BIN" -m local_failover check --config "$CONFIG_PATH"
+```
+
+The following are **activation commands, not validation commands**. They require approval from the
+Mac runtime owner after configuration backups and isolated probes:
+
+```bash
+# REQUIRES EXPLICIT AUTHORIZATION
+launchctl bootstrap "gui/$(id -u)" "$PLIST"
+launchctl kickstart -k "gui/$(id -u)/com.example.local-failover"
+```
+
+Before client cutover, make owner-only backups for every client independently:
+
+```bash
+umask 077
+stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+cp -p "$CLIENT_CONFIG" "$CLIENT_CONFIG.before-local-failover.$stamp"
+```
+
+Only after a loopback status check and a real read-only client smoke test should an authorized
+operator change a client base URL. Rollback restores each exact backup and unloads only this agent:
+
+```bash
+# REQUIRES EXPLICIT AUTHORIZATION
+launchctl bootout "gui/$(id -u)" "$PLIST"
+cp -p "$CLIENT_CONFIG.before-local-failover.$stamp" "$CLIENT_CONFIG"
+```
+
+The gateway never modifies existing MCP child services. Rollback therefore leaves their original
+ports and processes untouched.
+
+## Optional VPC attachment paths
+
+Run identifier-redacted detection only when approved to use the already-authenticated CLIs:
+
+```bash
+python3 scripts/local-failover/vpc_paths.py detect \
+ --region "$AWS_REGION" \
+ --route-destination "$VPC_DESTINATION"
+```
+
+Detection invokes only CLI version/status, `describe-client-vpn-endpoints`,
+`describe-client-vpn-target-networks`, `describe-instance-information`, `tailscale status --json`,
+and local route inspection. It reports counts and coarse state; it never prints resource IDs,
+tailnet names, addresses, command stderr, or credential locations. It does not start sessions.
+
+### A. AWS Client VPN: managed full-VPC laptop route
+
+AWS Client VPN is an OpenVPN-based managed remote-access service. AWS documents certificate
+(mutual), AWS Managed Microsoft AD, and SAML federated authentication; every endpoint also needs an
+ACM server certificate. A production design requires:
+
+- a non-overlapping client CIDR and split/full-tunnel decision;
+- endpoint creation and at least one target subnet association;
+- authorization rules distinct from routes;
+- VPC and additional destination routes as needed;
+- existing security groups and reachable DNS servers;
+- certificate lifecycle or AD/SAML configuration and a macOS client rollout;
+- connection logging and retention decisions.
+
+The managed convenience has the broadest route scope and largest blast radius of these options.
+Least-privilege authorization rules, narrow security groups, split tunneling, MFA-capable user auth,
+and DNS testing are important. SAML uses the AWS-provided client; OpenVPN-compatible client choices
+depend on the selected auth mode.
+
+Client VPN is paid. AWS bills endpoint association-hours and active client connection-hours; public
+IPv4, data transfer, logging, and related services can add cost. The official pricing page currently
+illustrates US East (Ohio) with one association at $0.10/hour and ten connections totaling
+$0.50/hour, but rates vary by Region and can change—review the current pricing page and a change set
+before authorization.
+
+The repository template is review-only:
+
+```bash
+python3 scripts/local-failover/vpc_paths.py plan client-vpn \
+ --region "$AWS_REGION" --stack-name local-access-review
+```
+
+It prints exact `validate-template`, `create-change-set`, and `describe-change-set` commands around
+`plans/aws-client-vpn.yaml`. A change set does not deploy. Executing it would create an endpoint,
+association, authorization rule, and optional route and therefore needs separate paid/network/
+security approval. The template consumes existing certificates and security groups; it does not
+create or alter either.
+
+Rollback after a real deployment is destructive and must be separately approved. CloudFormation
+should first preview deletion impact. The exact final operation is:
+
+```bash
+# ⚠ REQUIRES EXPLICIT CONFIRMATION; DO NOT RUN AS PART OF DETECTION/PLANNING
+aws cloudformation delete-stack --region "$AWS_REGION" --stack-name "$STACK_NAME"
+```
+
+Exported client profiles, DNS, security groups, certificates, and identity-provider objects have
+independent lifecycles and must not be removed merely because the endpoint stack is rolled back.
+
+Official AWS guidance:
+
+- [What is AWS Client VPN?](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/what-is.html)
+- [Client authentication](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html)
+- [Mutual authentication](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-auth-mutual-enable.html)
+- [SAML federation](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/federated-authentication.html)
+- [Endpoint workflow](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-endpoints.html)
+- [Target networks](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-target.html)
+- [Routes](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-routes.html)
+- [Authorization rules](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-rules.html)
+- [Security groups](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-vpn-security-groups.html)
+- [Best practices](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/what-is-best-practices.html)
+- [macOS and other clients](https://docs.aws.amazon.com/vpn/latest/clientvpn-user/connect.html)
+- [Current AWS VPN pricing](https://aws.amazon.com/vpn/pricing/)
+
+### B. Systems Manager: narrow per-service tunnel
+
+`AWS-StartPortForwardingSessionToRemoteHost` forwards one local port through an SSM managed node to
+a remote host reachable from that node. It does not provide a broad laptop route. Current AWS
+guidance requires SSM Agent 3.1.1374.0 or later for remote-host forwarding, the Session Manager
+plugin, IAM permission, managed-node Systems Manager connectivity, and remote DNS/network access.
+The remote host itself need not be managed by SSM. Port-forward payloads are not available in
+Session Manager session logs, reducing content auditability.
+
+Render, but do not execute, the exact command:
+
+```bash
+python3 scripts/local-failover/vpc_paths.py plan ssm \
+ --region "$AWS_REGION" --remote-port 8317 --local-port 18317
+```
+
+An enabled SSM route is still unavailable until the gateway owns the tunnel process group, observed
+its port transition, verifies every listener on the port belongs to that process group, and passes
+authenticated semantic health through it. A bind race or listener replacement revokes readiness
+and signals only the recorded process group. Process exit removes the gate immediately; if an owned
+descendant still holds the listener, the supervisor terminates the continuously existing group with
+TERM and bounded KILL escalation. Restart uses capped backoff. A pre-existing listener is reported
+as a conflict and never adopted or killed.
+
+Official AWS guidance:
+
+- [Start Session Manager sessions and remote-host forwarding](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html)
+- [Session Manager prerequisites](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-prerequisites.html)
+- [Session Manager overview](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html)
+- [`aws ssm start-session`](https://docs.aws.amazon.com/cli/latest/reference/ssm/start-session.html)
+
+### C. Tailscale subnet router: simplest existing overlay route
+
+When Tailscale already connects the laptop and an approved VPC host, a subnet router is usually the
+simplest non-AWS-managed broad route. The router host must enable IP forwarding, advertise the VPC
+CIDR, receive tailnet route approval (or an approved `autoApprovers` rule), and have access controls
+that permit the intended identities. These are host and shared-tailnet mutations, so this repository
+only prints a dry-run plan:
+
+```bash
+python3 scripts/local-failover/vpc_paths.py plan tailscale --vpc-cidr 10.0.0.0/16
+```
+
+The failover route remains `provisioned: false` until those approvals exist and local route plus
+semantic health checks pass. Rollback withdraws and unapproves the route and restores forwarding;
+each step requires the same owners' authorization.
+
+Official guidance: [Tailscale subnet routers](https://tailscale.com/kb/1019/subnets).
+
+## Paths that are not laptop VPC routing
+
+AWS Site-to-Site VPN connects networks through customer-gateway infrastructure, and Direct Connect
+is a physical/partner private connection. Neither is an individual macOS laptop solution. AWS
+Verified Access can protect application-level access without a VPN, but it is not a general laptop
+route into a VPC and is not used as a gateway failover path here.
+
+## Permissions still required for live end-to-end proof
+
+Repository tests prove local forwarding and simulated path failures. A live claim that existing CLIs
+continue across the real cloud-edge, Tailscale, and SSM paths still requires all of the following:
+
+1. Mac runtime-owner approval to render and bootstrap the LaunchAgent on unused production ports.
+2. Approval for an existing credential injector to expose only named health variables to the agent.
+3. Tailscale host, route-advertisement, route-approval, and access-control approval if that path does
+ not already exist.
+4. SSM managed-node/IAM confirmation before enabling the owned tunnel.
+5. Separate cost/network/security authorization before any Client VPN change set is executed.
+6. Per-client backup, base-URL cutover approval, and a rollback window.
+
+No permission in this list is implied by repository review or passing tests.
diff --git a/claude-ops/scripts/local-failover/config.example.json b/claude-ops/scripts/local-failover/config.example.json
new file mode 100644
index 00000000..98daf515
--- /dev/null
+++ b/claude-ops/scripts/local-failover/config.example.json
@@ -0,0 +1,153 @@
+{
+ "version": 1,
+ "state_file": "~/.local/state/local-failover/status.json",
+ "listeners": [
+ {
+ "name": "llm",
+ "host": "127.0.0.1",
+ "port": 18080,
+ "protocol": "llm",
+ "max_body_bytes": 8388608,
+ "max_concurrent_requests": 64,
+ "max_idempotent_attempts": 2,
+ "client_timeout_seconds": 10,
+ "connect_timeout_seconds": 5,
+ "idle_timeout_seconds": 120,
+ "session_ttl_seconds": 3600,
+ "routes": [
+ {
+ "name": "cloudflare-primary",
+ "url_env": "FAILOVER_CLOUDFLARE_URL",
+ "provisioned": true,
+ "trusted_overlay_http": false,
+ "health": {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "inventory_path": "/v1/models",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ "interval_seconds": 60,
+ "timeout_seconds": 8,
+ "first_event_timeout_seconds": 5,
+ "max_event_gap_seconds": 3,
+ "max_response_bytes": 65536
+ },
+ "breaker": {
+ "failure_threshold": 2,
+ "recovery_successes": 3,
+ "base_backoff_seconds": 5,
+ "max_backoff_seconds": 120,
+ "failback_hold_seconds": 15,
+ "max_stale_seconds": 180
+ }
+ },
+ {
+ "name": "tailscale-secondary",
+ "url_env": "FAILOVER_TAILSCALE_URL",
+ "provisioned": false,
+ "trusted_overlay_http": true,
+ "network_gate": {
+ "kind": "route_interface",
+ "destination": "100.64.0.10",
+ "interface_prefixes": ["utun", "tailscale"]
+ },
+ "health": {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "inventory_path": "/v1/models",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ "interval_seconds": 60
+ },
+ "breaker": {
+ "failure_threshold": 2,
+ "recovery_successes": 3,
+ "base_backoff_seconds": 5,
+ "max_backoff_seconds": 120,
+ "failback_hold_seconds": 15,
+ "max_stale_seconds": 180
+ }
+ },
+ {
+ "name": "ssm-tertiary",
+ "url": "http://127.0.0.1:18317",
+ "provisioned": false,
+ "trusted_overlay_http": false,
+ "tunnel": {
+ "enabled": false,
+ "argv": [
+ "aws",
+ "ssm",
+ "start-session",
+ "--region",
+ "${AWS_REGION}",
+ "--target",
+ "${SSM_MANAGED_NODE_ID}",
+ "--document-name",
+ "AWS-StartPortForwardingSessionToRemoteHost",
+ "--parameters",
+ "host=[${SSM_REMOTE_HOST}],portNumber=[8317],localPortNumber=[18317]"
+ ],
+ "listener_host": "127.0.0.1",
+ "listener_port": 18317,
+ "startup_timeout_seconds": 20,
+ "restart_base_seconds": 5,
+ "restart_max_seconds": 120
+ },
+ "health": {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "inventory_path": "/v1/models",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ "interval_seconds": 60
+ },
+ "breaker": {
+ "failure_threshold": 2,
+ "recovery_successes": 3,
+ "base_backoff_seconds": 5,
+ "max_backoff_seconds": 120,
+ "failback_hold_seconds": 15,
+ "max_stale_seconds": 180
+ }
+ }
+ ]
+ },
+ {
+ "name": "mcp",
+ "host": "127.0.0.1",
+ "port": 18090,
+ "protocol": "mcp",
+ "max_body_bytes": 4194304,
+ "max_concurrent_requests": 64,
+ "max_idempotent_attempts": 1,
+ "client_timeout_seconds": 10,
+ "connect_timeout_seconds": 5,
+ "idle_timeout_seconds": 120,
+ "session_ttl_seconds": 3600,
+ "routes": [
+ {
+ "name": "local-primary",
+ "url": "http://127.0.0.1:18091",
+ "provisioned": true,
+ "health": {
+ "kind": "mcp",
+ "mcp_health_path": "/_failover/status",
+ "interval_seconds": 30
+ }
+ },
+ {
+ "name": "remote-secondary",
+ "url_env": "FAILOVER_REMOTE_MCP_URL",
+ "provisioned": false,
+ "health": {
+ "kind": "mcp",
+ "auth_env": "MCP_HEALTH_TOKEN",
+ "mcp_health_path": "/_failover/status",
+ "interval_seconds": 30
+ }
+ }
+ ]
+ }
+ ]
+}
diff --git a/claude-ops/scripts/local-failover/launchd/com.example.local-failover.plist.template b/claude-ops/scripts/local-failover/launchd/com.example.local-failover.plist.template
new file mode 100644
index 00000000..ebce87fe
--- /dev/null
+++ b/claude-ops/scripts/local-failover/launchd/com.example.local-failover.plist.template
@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>Label</key>
+ <string>com.example.local-failover</string>
+ <key>ProgramArguments</key>
+ <array>
+ <string>__PYTHON_BIN__</string>
+ <string>-m</string>
+ <string>local_failover</string>
+ <string>run</string>
+ <string>--config</string>
+ <string>__CONFIG_PATH__</string>
+ </array>
+ <key>WorkingDirectory</key>
+ <string>__PACKAGE_ROOT__</string>
+ <key>RunAtLoad</key>
+ <true/>
+ <key>KeepAlive</key>
+ <dict>
+ <key>SuccessfulExit</key>
+ <false/>
+ </dict>
+ <key>ThrottleInterval</key>
+ <integer>10</integer>
+ <key>ProcessType</key>
+ <string>Background</string>
+ <key>Umask</key>
+ <integer>63</integer>
+ <key>EnvironmentVariables</key>
+ <dict>
+ <key>PYTHONUNBUFFERED</key>
+ <string>1</string>
+ </dict>
+ <key>StandardOutPath</key>
+ <string>__STDOUT_LOG__</string>
+ <key>StandardErrorPath</key>
+ <string>__STDERR_LOG__</string>
+</dict>
+</plist>
diff --git a/claude-ops/scripts/local-failover/local_failover/__init__.py b/claude-ops/scripts/local-failover/local_failover/__init__.py
new file mode 100644
index 00000000..63ff946c
--- /dev/null
+++ b/claude-ops/scripts/local-failover/local_failover/__init__.py
@@ -0,0 +1,5 @@
+"""Loopback-only, health-gated local failover gateway."""
+
+from .config import ConfigError, GatewayConfig, load_config, parse_config
+
+__all__ = ["ConfigError", "GatewayConfig", "load_config", "parse_config"]
diff --git a/claude-ops/scripts/local-failover/local_failover/__main__.py b/claude-ops/scripts/local-failover/local_failover/__main__.py
new file mode 100644
index 00000000..5a555af0
--- /dev/null
+++ b/claude-ops/scripts/local-failover/local_failover/__main__.py
@@ -0,0 +1,4 @@
+from .main import main
+
+
+raise SystemExit(main())
diff --git a/claude-ops/scripts/local-failover/local_failover/circuit.py b/claude-ops/scripts/local-failover/local_failover/circuit.py
new file mode 100644
index 00000000..d1499345
--- /dev/null
+++ b/claude-ops/scripts/local-failover/local_failover/circuit.py
@@ -0,0 +1,152 @@
+"""Circuit breaker and fixed-order route selection."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Iterable
+
+from .config import BreakerPolicy, RouteConfig
+
+
+class CircuitBreaker:
+ """Health-driven breaker with recovery hysteresis and capped backoff."""
+
+ def __init__(self, policy: BreakerPolicy, now: float) -> None:
+ self.policy = policy
+ self.state = "unknown"
+ self.reason = "not_checked"
+ self.consecutive_failures = 0
+ self.consecutive_successes = 0
+ self.open_count = 0
+ self.open_until = 0.0
+ self.healthy_since: float | None = None
+ self.last_success: float | None = None
+ self.last_failure: float | None = None
+ self.last_transition = now
+
+ def _transition(self, state: str, now: float) -> None:
+ if self.state != state:
+ self.state = state
+ self.last_transition = now
+
+ def _open(self, now: float, reason: str) -> None:
+ self.open_count += 1
+ exponent = min(self.open_count - 1, 30)
+ backoff = min(
+ self.policy.base_backoff_seconds * (2**exponent),
+ self.policy.max_backoff_seconds,
+ )
+ self.open_until = now + backoff
+ self.healthy_since = None
+ self.consecutive_successes = 0
+ self.reason = reason
+ self._transition("open", now)
+
+ def observe_success(self, now: float, reason: str) -> None:
+ if self.state == "open" and now < self.open_until:
+ return
+ if self.state in {"unknown", "open"}:
+ self._transition("half_open", now)
+ self.consecutive_successes = 0
+ self.healthy_since = None
+ self.consecutive_failures = 0
+ self.consecutive_successes += 1
+ self.last_success = now
+ self.reason = reason
+ if self.consecutive_successes >= self.policy.recovery_successes:
+ if self.healthy_since is None:
+ self.healthy_since = now
+ self._transition("closed", now)
+
+ def observe_failure(self, now: float, reason: str) -> None:
+ self.last_failure = now
+ self.reason = reason
+ self.consecutive_successes = 0
+ self.healthy_since = None
+ if self.state in {"half_open", "open", "unknown"}:
+ self.consecutive_failures += 1
+ if self.state == "half_open" or self.consecutive_failures >= self.policy.failure_threshold:
+ self._open(now, reason)
+ return
+ self.consecutive_failures += 1
+ if self.consecutive_failures >= self.policy.failure_threshold:
+ self._open(now, reason)
+
+ def observe_unknown(self, now: float, reason: str) -> None:
+ self.reason = reason
+ if self.state == "unknown":
+ self.last_transition = now
+
+ def available(self, now: float) -> bool:
+ if self.state != "closed" or self.healthy_since is None or self.last_success is None:
+ return False
+ if now - self.last_success > self.policy.max_stale_seconds:
+ return False
+ return now - self.healthy_since >= self.policy.failback_hold_seconds
+
+ def snapshot(self, now: float) -> dict[str, object]:
+ return {
+ "state": self.state,
+ "reason": self.reason,
+ "available": self.available(now),
+ "consecutive_failures": self.consecutive_failures,
+ "consecutive_successes": self.consecutive_successes,
+ "open_until_monotonic": self.open_until if self.state == "open" else None,
+ "last_transition_monotonic": self.last_transition,
+ }
+
+
+@dataclass
+class RouteRuntime:
+ config: RouteConfig
+ breaker: CircuitBreaker
+ gate_ready: bool = True
+ gate_reason: str = "ready"
+ requests: int = 0
+ failures: int = 0
+ bytes_to_client: int = 0
+
+ def available(self, now: float) -> bool:
+ return self.config.provisioned and self.gate_ready and self.breaker.available(now)
+
+ def snapshot(self, now: float) -> dict[str, object]:
+ return {
+ "name": self.config.name,
+ "provisioned": self.config.provisioned,
+ "gate_ready": self.gate_ready,
+ "gate_reason": self.gate_reason,
+ "circuit": self.breaker.snapshot(now),
+ "requests": self.requests,
+ "failures": self.failures,
+ "bytes_to_client": self.bytes_to_client,
+ }
+
+
+class RouteSelector:
+ """Selects the first available route and preserves MCP session affinity."""
+
+ def __init__(self, routes: Iterable[RouteRuntime]) -> None:
+ self.routes = tuple(routes)
+ self.by_name = {route.config.name: route for route in self.routes}
+ self.active_route: str | None = None
+
+ def choose(
+ self, session_route: str | None, now: float, excluded: set[str] | None = None
+ ) -> RouteRuntime | None:
+ excluded = excluded or set()
+ if session_route:
+ route = self.by_name.get(session_route)
+ if route and route.config.name not in excluded and route.available(now):
+ return route
+ return None
+ for route in self.routes:
+ if route.config.name in excluded:
+ continue
+ if route.available(now):
+ self.active_route = route.config.name
+ return route
+ self.active_route = None
+ return None
+
+ def snapshot(self) -> dict[str, object]:
+ return {"active_route": self.active_route}
diff --git a/claude-ops/scripts/local-failover/local_failover/config.py b/claude-ops/scripts/local-failover/local_failover/config.py
new file mode 100644
index 00000000..46a57c3c
--- /dev/null
+++ b/claude-ops/scripts/local-failover/local_failover/config.py
@@ -0,0 +1,569 @@
+"""Strict configuration schema for the local failover gateway."""
+
+from __future__ import annotations
+
+import ipaddress
+import json
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Mapping
+from urllib.parse import urlsplit
+
+
+ENV_NAME = re.compile(r"^[A-Z_][A-Z0-9_]*$")
+SAFE_NAME = re.compile(r"^[a-z][a-z0-9-]{0,62}$")
+
+
+class ConfigError(ValueError):
+ """The failover configuration is unsafe or internally inconsistent."""
+
+
+@dataclass(frozen=True)
+class BreakerPolicy:
+ failure_threshold: int = 2
+ recovery_successes: int = 3
+ base_backoff_seconds: float = 5.0
+ max_backoff_seconds: float = 60.0
+ failback_hold_seconds: float = 5.0
+ max_stale_seconds: float = 60.0
+
+
+@dataclass(frozen=True)
+class HealthConfig:
+ kind: str
+ auth_env: str | None = None
+ auth_header: str = "Authorization"
+ auth_prefix: str = "Bearer "
+ inventory_path: str = "/v1/models"
+ stream_path: str | None = None
+ stream_model: str | None = None
+ stream_model_env: str | None = None
+ mcp_health_path: str | None = None
+ interval_seconds: float = 15.0
+ timeout_seconds: float = 8.0
+ first_event_timeout_seconds: float = 5.0
+ max_event_gap_seconds: float = 3.0
+ max_response_bytes: int = 65536
+
+
+@dataclass(frozen=True)
+class TunnelConfig:
+ enabled: bool
+ argv: tuple[str, ...]
+ listener_host: str
+ listener_port: int
+ startup_timeout_seconds: float = 20.0
+ restart_base_seconds: float = 5.0
+ restart_max_seconds: float = 120.0
+
+
+@dataclass(frozen=True)
+class NetworkGateConfig:
+ kind: str = "none"
+ destination: str | None = None
+ interface_prefixes: tuple[str, ...] = ()
+
+
+@dataclass(frozen=True)
+class RouteConfig:
+ name: str
+ url: str | None
+ url_env: str | None
+ provisioned: bool
+ trusted_overlay_http: bool
+ health: HealthConfig
+ breaker: BreakerPolicy
+ tunnel: TunnelConfig | None = None
+ network_gate: NetworkGateConfig = NetworkGateConfig()
+
+ def resolve_url(self, env: Mapping[str, str]) -> str:
+ value = self.url
+ if self.url_env:
+ value = env.get(self.url_env)
+ if not value:
+ raise ConfigError(
+ f"route {self.name!r} requires environment variable {self.url_env}"
+ )
+ assert value is not None
+ _validate_upstream_url(value, self.trusted_overlay_http, self.name)
+ return value.rstrip("/")
+
+
+@dataclass(frozen=True)
+class ListenerConfig:
+ name: str
+ host: str
+ port: int
+ protocol: str
+ max_body_bytes: int
+ max_concurrent_requests: int
+ max_idempotent_attempts: int
+ client_timeout_seconds: float
+ connect_timeout_seconds: float
+ idle_timeout_seconds: float
+ session_ttl_seconds: float
+ routes: tuple[RouteConfig, ...]
+
+
+@dataclass(frozen=True)
+class GatewayConfig:
+ version: int
+ state_file: Path | None
+ listeners: tuple[ListenerConfig, ...]
+
+
+def _object(value: Any, context: str) -> dict[str, Any]:
+ if not isinstance(value, dict):
+ raise ConfigError(f"{context} must be an object")
+ return value
+
+
+def _list(value: Any, context: str) -> list[Any]:
+ if not isinstance(value, list):
+ raise ConfigError(f"{context} must be an array")
+ return value
+
+
+def _only_keys(value: dict[str, Any], allowed: set[str], context: str) -> None:
+ extra = sorted(set(value) - allowed)
+ if extra:
+ raise ConfigError(f"{context} contains unknown keys: {', '.join(extra)}")
+
+
+def _name(value: Any, context: str) -> str:
+ if not isinstance(value, str) or not SAFE_NAME.fullmatch(value):
+ raise ConfigError(f"{context} must match {SAFE_NAME.pattern}")
+ return value
+
+
+def _env_name(value: Any, context: str, optional: bool = False) -> str | None:
+ if value is None and optional:
+ return None
+ if not isinstance(value, str) or not ENV_NAME.fullmatch(value):
+ raise ConfigError(f"{context} must be an uppercase environment variable name")
+ return value
+
+
+def _bounded_int(value: Any, minimum: int, maximum: int, context: str) -> int:
+ if isinstance(value, bool) or not isinstance(value, int) or not minimum <= value <= maximum:
+ raise ConfigError(f"{context} must be an integer from {minimum} to {maximum}")
+ return value
+
+
+def _bounded_float(value: Any, minimum: float, maximum: float, context: str) -> float:
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ raise ConfigError(f"{context} must be a number")
+ result = float(value)
+ if not minimum <= result <= maximum:
+ raise ConfigError(f"{context} must be from {minimum} to {maximum}")
+ return result
+
+
+def _loopback(host: str, context: str) -> None:
+ try:
+ address = ipaddress.ip_address(host)
+ except ValueError as exc:
+ raise ConfigError(f"{context} must be a literal loopback address") from exc
+ if not address.is_loopback:
+ raise ConfigError(f"{context} must be loopback-only")
+
+
+def _safe_path(value: Any, context: str, optional: bool = False) -> str | None:
+ if value is None and optional:
+ return None
+ if not isinstance(value, str) or not value.startswith("/") or "\x00" in value:
+ raise ConfigError(f"{context} must be an absolute HTTP path")
+ if "?" in value or "#" in value:
+ raise ConfigError(f"{context} cannot contain a query or fragment")
+ return value
+
+
+def _validate_upstream_url(value: str, trusted_overlay_http: bool, context: str) -> None:
+ try:
+ parsed = urlsplit(value)
+ port = parsed.port
+ except ValueError as exc:
+ raise ConfigError(f"route {context!r} has an invalid upstream URL") from exc
+ if parsed.scheme not in {"http", "https"} or not parsed.hostname:
+ raise ConfigError(f"route {context!r} upstream must use http or https")
+ if parsed.username or parsed.password or parsed.query or parsed.fragment:
+ raise ConfigError(
+ f"route {context!r} upstream cannot contain credentials, query, or fragment"
+ )
+ if port is not None and not 1 <= port <= 65535:
+ raise ConfigError(f"route {context!r} upstream port is invalid")
+ if parsed.scheme == "http":
+ try:
+ loopback = ipaddress.ip_address(parsed.hostname).is_loopback
+ except ValueError:
+ loopback = parsed.hostname == "localhost"
+ if not loopback and not trusted_overlay_http:
+ raise ConfigError(
+ f"route {context!r} cleartext upstream requires trusted_overlay_http"
+ )
+
+
+def _parse_breaker(raw: Any, context: str) -> BreakerPolicy:
+ data = _object(raw or {}, context)
+ _only_keys(
+ data,
+ {
+ "failure_threshold",
+ "recovery_successes",
+ "base_backoff_seconds",
+ "max_backoff_seconds",
+ "failback_hold_seconds",
+ "max_stale_seconds",
+ },
+ context,
+ )
+ policy = BreakerPolicy(
+ failure_threshold=_bounded_int(
+ data.get("failure_threshold", 2), 1, 20, f"{context}.failure_threshold"
+ ),
+ recovery_successes=_bounded_int(
+ data.get("recovery_successes", 3), 1, 20, f"{context}.recovery_successes"
+ ),
+ base_backoff_seconds=_bounded_float(
+ data.get("base_backoff_seconds", 5), 0.1, 3600, f"{context}.base_backoff_seconds"
+ ),
+ max_backoff_seconds=_bounded_float(
+ data.get("max_backoff_seconds", 60), 0.1, 86400, f"{context}.max_backoff_seconds"
+ ),
+ failback_hold_seconds=_bounded_float(
+ data.get("failback_hold_seconds", 5), 0, 3600, f"{context}.failback_hold_seconds"
+ ),
+ max_stale_seconds=_bounded_float(
+ data.get("max_stale_seconds", 60), 1, 86400, f"{context}.max_stale_seconds"
+ ),
+ )
+ if policy.max_backoff_seconds < policy.base_backoff_seconds:
+ raise ConfigError(f"{context}.max_backoff_seconds must be at least base_backoff_seconds")
+ return policy
+
+
+def _parse_health(raw: Any, context: str) -> HealthConfig:
+ data = _object(raw, context)
+ _only_keys(
+ data,
+ {
+ "kind",
+ "auth_env",
+ "auth_header",
+ "auth_prefix",
+ "inventory_path",
+ "stream_path",
+ "stream_model",
+ "stream_model_env",
+ "mcp_health_path",
+ "interval_seconds",
+ "timeout_seconds",
+ "first_event_timeout_seconds",
+ "max_event_gap_seconds",
+ "max_response_bytes",
+ },
+ context,
+ )
+ kind = data.get("kind")
+ if kind not in {"llm", "mcp"}:
+ raise ConfigError(f"{context}.kind must be llm or mcp")
+ auth_env = _env_name(data.get("auth_env"), f"{context}.auth_env", optional=True)
+ auth_header = data.get("auth_header", "Authorization")
+ auth_prefix = data.get("auth_prefix", "Bearer ")
+ if not isinstance(auth_header, str) or not re.fullmatch(r"[A-Za-z0-9-]{1,64}", auth_header):
+ raise ConfigError(f"{context}.auth_header is invalid")
+ if not isinstance(auth_prefix, str) or len(auth_prefix) > 32 or "\r" in auth_prefix or "\n" in auth_prefix:
+ raise ConfigError(f"{context}.auth_prefix is invalid")
+ stream_model = data.get("stream_model")
+ if stream_model is not None and (not isinstance(stream_model, str) or not stream_model):
+ raise ConfigError(f"{context}.stream_model must be a non-empty string")
+ stream_model_env = _env_name(
+ data.get("stream_model_env"), f"{context}.stream_model_env", optional=True
+ )
+ if stream_model and stream_model_env:
+ raise ConfigError(f"{context} must use stream_model or stream_model_env, not both")
+ inventory_path = _safe_path(
+ data.get("inventory_path", "/v1/models"), f"{context}.inventory_path"
+ )
+ stream_path = _safe_path(data.get("stream_path"), f"{context}.stream_path", optional=True)
+ mcp_health_path = _safe_path(
+ data.get("mcp_health_path"), f"{context}.mcp_health_path", optional=True
+ )
+ if kind == "llm" and not stream_path:
+ raise ConfigError(f"{context}.stream_path is required for llm health")
+ if kind == "mcp" and not mcp_health_path:
+ raise ConfigError(f"{context}.mcp_health_path is required for mcp health")
+ return HealthConfig(
+ kind=kind,
+ auth_env=auth_env,
+ auth_header=auth_header,
+ auth_prefix=auth_prefix,
+ inventory_path=inventory_path or "/v1/models",
+ stream_path=stream_path,
+ stream_model=stream_model,
+ stream_model_env=stream_model_env,
+ mcp_health_path=mcp_health_path,
+ interval_seconds=_bounded_float(
+ data.get("interval_seconds", 15), 1, 3600, f"{context}.interval_seconds"
+ ),
+ timeout_seconds=_bounded_float(
+ data.get("timeout_seconds", 8), 0.1, 120, f"{context}.timeout_seconds"
+ ),
+ first_event_timeout_seconds=_bounded_float(
+ data.get("first_event_timeout_seconds", 5),
+ 0.1,
+ 120,
+ f"{context}.first_event_timeout_seconds",
+ ),
+ max_event_gap_seconds=_bounded_float(
+ data.get("max_event_gap_seconds", 3), 0.1, 120, f"{context}.max_event_gap_seconds"
+ ),
+ max_response_bytes=_bounded_int(
+ data.get("max_response_bytes", 65536), 1024, 1048576, f"{context}.max_response_bytes"
+ ),
+ )
+
+
+def _parse_tunnel(raw: Any, context: str, provisioned: bool) -> TunnelConfig | None:
+ if raw is None:
+ return None
+ data = _object(raw, context)
+ _only_keys(
+ data,
+ {
+ "enabled",
+ "argv",
+ "listener_host",
+ "listener_port",
+ "startup_timeout_seconds",
+ "restart_base_seconds",
+ "restart_max_seconds",
+ },
+ context,
+ )
+ enabled = data.get("enabled")
+ if not isinstance(enabled, bool):
+ raise ConfigError(f"{context}.enabled must be boolean")
+ if enabled and not provisioned:
+ raise ConfigError(f"{context} cannot be enabled until the route is provisioned")
+ argv_raw = _list(data.get("argv"), f"{context}.argv")
+ if not argv_raw or any(not isinstance(item, str) or not item for item in argv_raw):
+ raise ConfigError(f"{context}.argv must contain non-empty strings")
+ host = data.get("listener_host")
+ if not isinstance(host, str):
+ raise ConfigError(f"{context}.listener_host must be a string")
+ _loopback(host, f"{context} tunnel listener")
+ config = TunnelConfig(
+ enabled=enabled,
+ argv=tuple(argv_raw),
+ listener_host=host,
+ listener_port=_bounded_int(
+ data.get("listener_port"), 1, 65535, f"{context}.listener_port"
+ ),
+ startup_timeout_seconds=_bounded_float(
+ data.get("startup_timeout_seconds", 20),
+ 1,
+ 300,
+ f"{context}.startup_timeout_seconds",
+ ),
+ restart_base_seconds=_bounded_float(
+ data.get("restart_base_seconds", 5), 0.1, 3600, f"{context}.restart_base_seconds"
+ ),
+ restart_max_seconds=_bounded_float(
+ data.get("restart_max_seconds", 120), 0.1, 86400, f"{context}.restart_max_seconds"
+ ),
+ )
+ if config.restart_max_seconds < config.restart_base_seconds:
+ raise ConfigError(f"{context}.restart_max_seconds must be at least restart_base_seconds")
+ return config
+
+
+def _parse_network_gate(raw: Any, context: str) -> NetworkGateConfig:
+ if raw is None:
+ return NetworkGateConfig()
+ data = _object(raw, context)
+ _only_keys(data, {"kind", "destination", "interface_prefixes"}, context)
+ kind = data.get("kind", "none")
+ if kind not in {"none", "route_interface"}:
+ raise ConfigError(f"{context}.kind must be none or route_interface")
+ if kind == "none":
+ return NetworkGateConfig()
+ destination = data.get("destination")
+ if not isinstance(destination, str) or not destination:
+ raise ConfigError(f"{context}.destination is required")
+ prefixes = _list(data.get("interface_prefixes"), f"{context}.interface_prefixes")
+ if not prefixes or any(
+ not isinstance(prefix, str) or not re.fullmatch(r"[A-Za-z0-9._-]{1,32}", prefix)
+ for prefix in prefixes
+ ):
+ raise ConfigError(f"{context}.interface_prefixes must contain safe prefixes")
+ return NetworkGateConfig(kind=kind, destination=destination, interface_prefixes=tuple(prefixes))
+
+
+def _parse_route(raw: Any, context: str) -> RouteConfig:
+ data = _object(raw, context)
+ _only_keys(
+ data,
+ {
+ "name",
+ "url",
+ "url_env",
+ "provisioned",
+ "trusted_overlay_http",
+ "health",
+ "breaker",
+ "tunnel",
+ "network_gate",
+ },
+ context,
+ )
+ name = _name(data.get("name"), f"{context}.name")
+ url = data.get("url")
+ url_env = _env_name(data.get("url_env"), f"{context}.url_env", optional=True)
+ if (url is None) == (url_env is None):
+ raise ConfigError(f"{context} must set exactly one of url or url_env")
+ if url is not None and not isinstance(url, str):
+ raise ConfigError(f"{context}.url must be a string")
+ trusted_overlay_http = data.get("trusted_overlay_http", False)
+ provisioned = data.get("provisioned", False)
+ if not isinstance(trusted_overlay_http, bool) or not isinstance(provisioned, bool):
+ raise ConfigError(f"{context} provisioned/trusted_overlay_http must be boolean")
+ if url is not None:
+ _validate_upstream_url(url, trusted_overlay_http, name)
+ health = _parse_health(data.get("health"), f"{context}.health")
+ breaker = _parse_breaker(data.get("breaker"), f"{context}.breaker")
+ tunnel = _parse_tunnel(data.get("tunnel"), f"{context}.tunnel", provisioned)
+ gate = _parse_network_gate(data.get("network_gate"), f"{context}.network_gate")
+ return RouteConfig(
+ name=name,
+ url=url,
+ url_env=url_env,
+ provisioned=provisioned,
+ trusted_overlay_http=trusted_overlay_http,
+ health=health,
+ breaker=breaker,
+ tunnel=tunnel,
+ network_gate=gate,
+ )
+
+
+def _parse_listener(raw: Any, context: str) -> ListenerConfig:
+ data = _object(raw, context)
+ _only_keys(
+ data,
+ {
+ "name",
+ "host",
+ "port",
+ "protocol",
+ "max_body_bytes",
+ "max_concurrent_requests",
+ "max_idempotent_attempts",
+ "client_timeout_seconds",
+ "connect_timeout_seconds",
+ "idle_timeout_seconds",
+ "session_ttl_seconds",
+ "routes",
+ },
+ context,
+ )
+ name = _name(data.get("name"), f"{context}.name")
+ host = data.get("host")
+ if not isinstance(host, str):
+ raise ConfigError(f"{context}.host must be a string")
+ _loopback(host, f"{context} listener")
+ protocol = data.get("protocol")
+ if protocol not in {"llm", "mcp"}:
+ raise ConfigError(f"{context}.protocol must be llm or mcp")
+ routes = tuple(
+ _parse_route(item, f"{context}.routes[{index}]")
+ for index, item in enumerate(_list(data.get("routes"), f"{context}.routes"))
+ )
+ if not routes:
+ raise ConfigError(f"{context}.routes cannot be empty")
+ names = [route.name for route in routes]
+ if len(names) != len(set(names)):
+ raise ConfigError(f"{context} route names must be unique")
+ return ListenerConfig(
+ name=name,
+ host=host,
+ port=_bounded_int(data.get("port"), 1, 65535, f"{context}.port"),
+ protocol=protocol,
+ max_body_bytes=_bounded_int(
+ data.get("max_body_bytes", 1048576), 1, 16777216, f"{context}.max_body_bytes"
+ ),
+ max_concurrent_requests=_bounded_int(
+ data.get("max_concurrent_requests", 64),
+ 1,
+ 1024,
+ f"{context}.max_concurrent_requests",
+ ),
+ max_idempotent_attempts=_bounded_int(
+ data.get("max_idempotent_attempts", 2),
+ 1,
+ 8,
+ f"{context}.max_idempotent_attempts",
+ ),
+ client_timeout_seconds=_bounded_float(
+ data.get("client_timeout_seconds", 10),
+ 0.1,
+ 300,
+ f"{context}.client_timeout_seconds",
+ ),
+ connect_timeout_seconds=_bounded_float(
+ data.get("connect_timeout_seconds", 5),
+ 0.1,
+ 120,
+ f"{context}.connect_timeout_seconds",
+ ),
+ idle_timeout_seconds=_bounded_float(
+ data.get("idle_timeout_seconds", 30),
+ 0.1,
+ 3600,
+ f"{context}.idle_timeout_seconds",
+ ),
+ session_ttl_seconds=_bounded_float(
+ data.get("session_ttl_seconds", 3600), 1, 86400, f"{context}.session_ttl_seconds"
+ ),
+ routes=routes,
+ )
+
+
+def parse_config(raw: Any) -> GatewayConfig:
+ data = _object(raw, "config")
+ _only_keys(data, {"version", "state_file", "listeners"}, "config")
+ if data.get("version") != 1:
+ raise ConfigError("config.version must be 1")
+ state_file_raw = data.get("state_file")
+ if state_file_raw is not None and (
+ not isinstance(state_file_raw, str) or not state_file_raw.strip()
+ ):
+ raise ConfigError("config.state_file must be a non-empty path")
+ listeners = tuple(
+ _parse_listener(item, f"config.listeners[{index}]")
+ for index, item in enumerate(_list(data.get("listeners"), "config.listeners"))
+ )
+ if not listeners:
+ raise ConfigError("config.listeners cannot be empty")
+ names = [listener.name for listener in listeners]
+ binds = [(listener.host, listener.port) for listener in listeners]
+ if len(names) != len(set(names)):
+ raise ConfigError("listener names must be unique")
+ if len(binds) != len(set(binds)):
+ raise ConfigError("listener bind addresses must be unique")
+ return GatewayConfig(
+ version=1,
+ state_file=Path(state_file_raw).expanduser() if state_file_raw else None,
+ listeners=listeners,
+ )
+
+
+def load_config(path: Path) -> GatewayConfig:
+ try:
+ raw = json.loads(path.read_text(encoding="utf-8"))
+ except OSError as exc:
+ raise ConfigError(f"cannot read config: {exc}") from exc
+ except json.JSONDecodeError as exc:
+ raise ConfigError(f"invalid config JSON at line {exc.lineno}, column {exc.colno}") from exc
+ return parse_config(raw)
diff --git a/claude-ops/scripts/local-failover/local_failover/health.py b/claude-ops/scripts/local-failover/local_failover/health.py
new file mode 100644
index 00000000..e54a5ce8
--- /dev/null
+++ b/claude-ops/scripts/local-failover/local_failover/health.py
@@ -0,0 +1,330 @@
+"""Authenticated semantic health probes with bounded stream validation."""
+
+from __future__ import annotations
+
+import http.client
+import json
+import socket
+import ssl
+import time
+from dataclasses import dataclass
+from typing import Mapping, Protocol
+from urllib.parse import urlsplit
+
+from .config import ConfigError, RouteConfig
+
+
+@dataclass(frozen=True)
+class ProbeResult:
+ kind: str
+ reason: str
+ latency_ms: int
+
+
+@dataclass(frozen=True)
+class ProbeHTTPResponse:
+ status: int
+ headers: Mapping[str, str]
+ body: bytes
+ events: tuple[tuple[float, bytes], ...]
+ latency_ms: int
+ error: str | None = None
+
+
+class ProbeTransport(Protocol):
+ def request(self, **kwargs) -> ProbeHTTPResponse: ...
+
+
+def _http_classification(status: int, latency_ms: int, prefix: str) -> ProbeResult | None:
+ if status in {400, 401, 403, 404, 405, 409, 422}:
+ return ProbeResult("unknown", "health_configuration_rejected", latency_ms)
+ if status == 429:
+ return ProbeResult("failure", f"{prefix}_rate_limited", latency_ms)
+ if status >= 500:
+ return ProbeResult("failure", f"{prefix}_server_error", latency_ms)
+ if not 200 <= status < 300:
+ return ProbeResult("failure", f"{prefix}_http_status", latency_ms)
+ return None
+
+
+def classify_inventory_response(status: int, body: bytes, latency_ms: int) -> ProbeResult:
+ classified = _http_classification(status, latency_ms, "inventory")
+ if classified:
+ return classified
+ try:
+ payload = json.loads(body)
+ except (UnicodeDecodeError, json.JSONDecodeError):
+ return ProbeResult("failure", "inventory_json", latency_ms)
+ if not isinstance(payload, dict) or not isinstance(payload.get("data"), list):
+ return ProbeResult("failure", "inventory_shape", latency_ms)
+ return ProbeResult("success", "inventory_ok", latency_ms)
+
+
+def classify_stream_response(
+ response: ProbeHTTPResponse,
+ first_event_timeout_seconds: float,
+ max_event_gap_seconds: float,
+ max_response_bytes: int,
+) -> ProbeResult:
+ if response.error:
+ return ProbeResult("failure", response.error, response.latency_ms)
+ classified = _http_classification(response.status, response.latency_ms, "stream")
+ if classified:
+ return classified
+ content_type = response.headers.get("content-type", "").lower()
+ if "text/event-stream" not in content_type and "ndjson" not in content_type:
+ return ProbeResult("failure", "stream_content_type", response.latency_ms)
+ total = sum(len(line) for _, line in response.events)
+ if total > max_response_bytes:
+ return ProbeResult("failure", "stream_response_too_large", response.latency_ms)
+ data_events = [
+ (elapsed, line)
+ for elapsed, line in response.events
+ if line.lstrip().startswith((b"data:", b"{"))
+ ]
+ if not data_events:
+ return ProbeResult("failure", "stream_no_events", response.latency_ms)
+ if data_events[0][0] > first_event_timeout_seconds:
+ return ProbeResult("failure", "stream_first_event_timeout", response.latency_ms)
+ if len(data_events) < 2:
+ return ProbeResult("failure", "stream_interrupted", response.latency_ms)
+ for previous, current in zip(data_events, data_events[1:]):
+ if current[0] - previous[0] > max_event_gap_seconds:
+ return ProbeResult("failure", "stream_event_gap", response.latency_ms)
+ return ProbeResult("success", "stream_cadence_ok", response.latency_ms)
+
+
+def classify_mcp_response(response: ProbeHTTPResponse) -> ProbeResult:
+ if response.error:
+ return ProbeResult("failure", response.error, response.latency_ms)
+ classified = _http_classification(response.status, response.latency_ms, "mcp")
+ if classified:
+ return classified
+ try:
+ payload = json.loads(response.body)
+ except (UnicodeDecodeError, json.JSONDecodeError):
+ return ProbeResult("failure", "mcp_health_shape", response.latency_ms)
+ if not isinstance(payload, dict):
+ return ProbeResult("failure", "mcp_health_shape", response.latency_ms)
+ status = payload.get("status", payload.get("state"))
+ semantic_ready = (
+ isinstance(status, str)
+ and status.lower() in {"healthy", "ok", "online", "ready"}
+ ) or payload.get("ready") is True or (
+ payload.get("protocol") == "mcp"
+ and isinstance(payload.get("active_route"), str)
+ and bool(payload["active_route"])
+ )
+ if not semantic_ready:
+ return ProbeResult("failure", "mcp_health_shape", response.latency_ms)
+ return ProbeResult("success", "mcp_health_ok", response.latency_ms)
+
+
+class HTTPProbeTransport:
+ """Small HTTP/1.1 client that never retains or logs probe payloads."""
+
+ def __init__(self, clock=time.monotonic) -> None:
+ self.clock = clock
+
+ @staticmethod
+ def _target(base_url: str, path: str) -> tuple[str, str, int, str]:
+ parsed = urlsplit(base_url)
+ assert parsed.hostname is not None
+ port = parsed.port or (443 if parsed.scheme == "https" else 80)
+ base_path = parsed.path.rstrip("/")
+ target = f"{base_path}/{path.lstrip('/')}" or "/"
+ return parsed.scheme, parsed.hostname, port, target
+
+ def request(
+ self,
+ *,
+ method: str,
+ url: str,
+ path: str,
+ headers: Mapping[str, str],
+ body: bytes | None,
+ timeout_seconds: float,
+ max_response_bytes: int,
+ stream: bool,
+ event_timeout_seconds: float,
+ ) -> ProbeHTTPResponse:
+ started = self.clock()
+ connection: http.client.HTTPConnection | None = None
+ try:
+ scheme, host, port, target = self._target(url, path)
+ if scheme == "https":
+ connection = http.client.HTTPSConnection(
+ host,
+ port,
+ timeout=timeout_seconds,
+ context=ssl.create_default_context(),
+ )
+ else:
+ connection = http.client.HTTPConnection(host, port, timeout=timeout_seconds)
+ request_headers = dict(headers)
+ request_headers["Connection"] = "close"
+ connection.request(method, target, body=body, headers=request_headers)
+ response = connection.getresponse()
+ response_headers = {key.lower(): value for key, value in response.getheaders()}
+ latency_ms = int((self.clock() - started) * 1000)
+ if stream and 200 <= response.status < 300:
+ if connection.sock:
+ connection.sock.settimeout(event_timeout_seconds)
+ events: list[tuple[float, bytes]] = []
+ total = 0
+ error: str | None = None
+ try:
+ while len(events) < 32:
+ line = response.readline(max_response_bytes - total + 1)
+ if not line:
+ break
+ total += len(line)
+ elapsed = self.clock() - started
+ events.append((elapsed, line))
+ if total > max_response_bytes:
+ error = "stream_response_too_large"
+ break
+ stripped = line.strip()
+ if stripped in {b"data: [DONE]", b"event: message_stop"}:
+ break
+ data_count = sum(
+ 1
+ for _, event in events
+ if event.lstrip().startswith((b"data:", b"{"))
+ )
+ if data_count >= 2:
+ break
+ except (OSError, http.client.HTTPException, socket.timeout):
+ error = "stream_interrupted"
+ return ProbeHTTPResponse(
+ response.status,
+ response_headers,
+ b"",
+ tuple(events),
+ latency_ms,
+ error,
+ )
+ body_bytes = response.read(max_response_bytes + 1)
+ error = "probe_response_too_large" if len(body_bytes) > max_response_bytes else None
+ if error:
+ body_bytes = b""
+ return ProbeHTTPResponse(
+ response.status,
+ response_headers,
+ body_bytes,
+ (),
+ latency_ms,
+ error,
+ )
+ except ConfigError:
+ return ProbeHTTPResponse(0, {}, b"", (), 0, "route_configuration")
+ except (OSError, http.client.HTTPException, ssl.SSLError, ValueError):
+ return ProbeHTTPResponse(
+ 0,
+ {},
+ b"",
+ (),
+ int((self.clock() - started) * 1000),
+ "connect_error",
+ )
+ finally:
+ if connection:
+ connection.close()
+
+
+class SemanticHealth:
+ def __init__(self, transport: ProbeTransport | None = None) -> None:
+ self.transport = transport or HTTPProbeTransport()
+
+ @staticmethod
+ def _headers(route: RouteConfig, env: Mapping[str, str]) -> dict[str, str] | ProbeResult:
+ health = route.health
+ headers = {"Accept": "application/json, text/event-stream"}
+ if health.auth_env:
+ credential = env.get(health.auth_env)
+ if not credential:
+ return ProbeResult("unknown", "health_auth_missing", 0)
+ headers[health.auth_header] = f"{health.auth_prefix}{credential}"
+ return headers
+
+ def probe(self, route: RouteConfig, env: Mapping[str, str]) -> ProbeResult:
+ headers = self._headers(route, env)
+ if isinstance(headers, ProbeResult):
+ return headers
+ try:
+ url = route.resolve_url(env)
+ except ConfigError:
+ return ProbeResult("unknown", "route_url_missing", 0)
+ health = route.health
+ if health.kind == "mcp":
+ response = self.transport.request(
+ method="GET",
+ url=url,
+ path=health.mcp_health_path or "/_failover/status",
+ headers=headers,
+ body=None,
+ timeout_seconds=health.timeout_seconds,
+ max_response_bytes=health.max_response_bytes,
+ stream=False,
+ event_timeout_seconds=health.max_event_gap_seconds,
+ )
+ return classify_mcp_response(response)
+
+ model = health.stream_model
+ if health.stream_model_env:
+ model = env.get(health.stream_model_env)
+ if not model:
+ return ProbeResult("unknown", "health_model_missing", 0)
+ inventory = self.transport.request(
+ method="GET",
+ url=url,
+ path=health.inventory_path,
+ headers=headers,
+ body=None,
+ timeout_seconds=health.timeout_seconds,
+ max_response_bytes=health.max_response_bytes,
+ stream=False,
+ event_timeout_seconds=health.max_event_gap_seconds,
+ )
+ if inventory.error:
+ return ProbeResult("failure", inventory.error, inventory.latency_ms)
+ inventory_result = classify_inventory_response(
+ inventory.status, inventory.body, inventory.latency_ms
+ )
+ if inventory_result.kind != "success":
+ return inventory_result
+ stream_body = json.dumps(
+ {
+ "model": model,
+ "messages": [{"role": "user", "content": "Reply OK"}],
+ "max_tokens": 2,
+ "stream": True,
+ },
+ separators=(",", ":"),
+ ).encode()
+ stream = self.transport.request(
+ method="POST",
+ url=url,
+ path=health.stream_path or "/v1/chat/completions",
+ headers={**headers, "Content-Type": "application/json"},
+ body=stream_body,
+ timeout_seconds=health.timeout_seconds,
+ max_response_bytes=health.max_response_bytes,
+ stream=True,
+ event_timeout_seconds=max(
+ health.first_event_timeout_seconds, health.max_event_gap_seconds
+ ),
+ )
+ stream_result = classify_stream_response(
+ stream,
+ health.first_event_timeout_seconds,
+ health.max_event_gap_seconds,
+ health.max_response_bytes,
+ )
+ if stream_result.kind != "success":
+ return stream_result
+ return ProbeResult(
+ "success",
+ "semantic_ok",
+ inventory_result.latency_ms + stream_result.latency_ms,
+ )
diff --git a/claude-ops/scripts/local-failover/local_failover/main.py b/claude-ops/scripts/local-failover/local_failover/main.py
new file mode 100644
index 00000000..cf80a9f3
--- /dev/null
+++ b/claude-ops/scripts/local-failover/local_failover/main.py
@@ -0,0 +1,105 @@
+"""Command-line entry point for the local failover gateway."""
+
+from __future__ import annotations
+
+import argparse
+import http.client
+import json
+import signal
+import sys
+import threading
+from collections.abc import Sequence
+from pathlib import Path
+
+from .config import ConfigError, GatewayConfig, load_config
+from .proxy import GatewayService
+
+
+def check_summary(config: GatewayConfig) -> dict[str, object]:
+ return {
+ "valid": True,
+ "listeners": [
+ {
+ "name": listener.name,
+ "protocol": listener.protocol,
+ "bind": f"{listener.host}:{listener.port}",
+ "route_order": [route.name for route in listener.routes],
+ }
+ for listener in config.listeners
+ ],
+ }
+
+
+def local_status(config: GatewayConfig) -> dict[str, object]:
+ listeners: list[dict[str, object]] = []
+ for listener in config.listeners:
+ connection = http.client.HTTPConnection(
+ listener.host,
+ listener.port,
+ timeout=min(listener.connect_timeout_seconds, 2),
+ )
+ try:
+ connection.request("GET", "/_failover/status", headers={"Connection": "close"})
+ response = connection.getresponse()
+ body = response.read(1048577)
+ if response.status != 200 or len(body) > 1048576:
+ raise ValueError("invalid status response")
+ payload = json.loads(body)
+ if not isinstance(payload, dict):
+ raise ValueError("invalid status payload")
+ listeners.append({"name": listener.name, "state": "online", "status": payload})
+ except (OSError, http.client.HTTPException, UnicodeError, json.JSONDecodeError, ValueError):
+ listeners.append({"name": listener.name, "state": "unavailable"})
+ finally:
+ connection.close()
+ return {"listeners": listeners}
+
+
+def run(config: GatewayConfig) -> int:
+ service = GatewayService(config)
+ stopped = threading.Event()
+
+ def request_stop(_signum: int, _frame: object) -> None:
+ stopped.set()
+
+ previous = {
+ signum: signal.signal(signum, request_stop)
+ for signum in (signal.SIGINT, signal.SIGTERM)
+ }
+ try:
+ service.start(run_health=True)
+ stopped.wait()
+ finally:
+ service.shutdown()
+ for signum, handler in previous.items():
+ signal.signal(signum, handler)
+ return 0
+
+
+def _parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(description=__doc__)
+ subcommands = parser.add_subparsers(dest="command", required=True)
+ for name in ("run", "check", "status"):
+ command = subcommands.add_parser(name)
+ command.add_argument("--config", type=Path, required=True)
+ return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ args = _parser().parse_args(argv)
+ try:
+ config = load_config(args.config)
+ except ConfigError as exc:
+ print(f"configuration error: {exc}", file=sys.stderr)
+ return 2
+ if args.command == "check":
+ print(json.dumps(check_summary(config), sort_keys=True, separators=(",", ":")))
+ return 0
+ if args.command == "status":
+ print(json.dumps(local_status(config), sort_keys=True, separators=(",", ":")))
+ return 0
+ return run(config)
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/claude-ops/scripts/local-failover/local_failover/proxy.py b/claude-ops/scripts/local-failover/local_failover/proxy.py
new file mode 100644
index 00000000..1756af83
--- /dev/null
+++ b/claude-ops/scripts/local-failover/local_failover/proxy.py
@@ -0,0 +1,718 @@
+"""Loopback-only streaming reverse proxy with ordered, session-safe failover."""
+
+from __future__ import annotations
+
+import http.client
+import json
+import os
+import socket
+import ssl
+import subprocess
+import tempfile
+import threading
+import time
+from dataclasses import dataclass
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from typing import Mapping
+from urllib.parse import urlsplit
+
+from .circuit import CircuitBreaker, RouteRuntime, RouteSelector
+from .config import GatewayConfig, ListenerConfig, NetworkGateConfig, RouteConfig
+from .health import ProbeResult, SemanticHealth
+from .tunnel import TunnelSupervisor
+
+
+IDEMPOTENT_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
+RETRYABLE_STATUSES = frozenset({502, 503, 504})
+HOP_BY_HOP_HEADERS = frozenset(
+ {
+ "connection",
+ "keep-alive",
+ "proxy-authenticate",
+ "proxy-authorization",
+ "te",
+ "trailer",
+ "transfer-encoding",
+ "upgrade",
+ }
+)
+
+
+@dataclass
+class SessionBinding:
+ route_name: str
+ last_seen: float
+
+
+def _connection_tokens(value: str | None) -> set[str]:
+ if not value:
+ return set()
+ return {token.strip().lower() for token in value.split(",") if token.strip()}
+
+
+def _network_gate_status(gate: NetworkGateConfig) -> tuple[bool, str]:
+ """Inspect the local route table without changing network state."""
+
+ if gate.kind == "none":
+ return True, "ready"
+ assert gate.destination is not None
+ try:
+ result = subprocess.run(
+ ["/sbin/route", "-n", "get", gate.destination],
+ stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL,
+ text=True,
+ timeout=2,
+ check=False,
+ )
+ except (OSError, subprocess.TimeoutExpired):
+ return False, "network_gate_unknown"
+ if result.returncode != 0:
+ return False, "network_route_missing"
+ interface = ""
+ for line in result.stdout.splitlines():
+ key, separator, value = line.strip().partition(":")
+ if separator and key == "interface":
+ interface = value.strip()
+ break
+ if not interface:
+ return False, "network_interface_unknown"
+ if not any(interface.startswith(prefix) for prefix in gate.interface_prefixes):
+ return False, "network_interface_unapproved"
+ return True, "network_route_ready"
+
+
+class ListenerRuntime:
+ """Mutable state for one configured listener."""
+
+ def __init__(self, service: GatewayService, config: ListenerConfig) -> None:
+ self.service = service
+ self.config = config
+ now = service.clock()
+ self.routes = [
+ RouteRuntime(
+ route,
+ CircuitBreaker(route.breaker, now),
+ gate_ready=route.provisioned
+ and route.tunnel is None
+ and route.network_gate.kind == "none",
+ gate_reason="ready"
+ if route.provisioned
+ and route.tunnel is None
+ and route.network_gate.kind == "none"
+ else "gate_not_checked",
+ )
+ for route in config.routes
+ ]
+ self.selector = RouteSelector(self.routes)
+ self.sessions: dict[str, SessionBinding] = {}
+ self.lock = threading.RLock()
+ self.requests = 0
+ self.responses = 0
+ self.errors = 0
+
+ def _expire_sessions(self, now: float) -> None:
+ expired = [
+ session_id
+ for session_id, binding in self.sessions.items()
+ if now - binding.last_seen > self.config.session_ttl_seconds
+ ]
+ for session_id in expired:
+ del self.sessions[session_id]
+
+ def session_route(self, session_id: str, now: float) -> str | None:
+ with self.lock:
+ self._expire_sessions(now)
+ binding = self.sessions.get(session_id)
+ if binding is None:
+ return None
+ binding.last_seen = now
+ return binding.route_name
+
+ def bind_session(self, session_id: str, route_name: str, now: float) -> None:
+ if not session_id or len(session_id) > 512 or "\r" in session_id or "\n" in session_id:
+ return
+ with self.lock:
+ self._expire_sessions(now)
+ self.sessions[session_id] = SessionBinding(route_name, now)
+
+ def choose(
+ self, session_route: str | None, now: float, excluded: set[str] | None = None
+ ) -> RouteRuntime | None:
+ with self.lock:
+ return self.selector.choose(session_route, now, excluded)
+
+ def begin_attempt(self, route: RouteRuntime) -> None:
+ with self.lock:
+ self.requests += 1
+ route.requests += 1
+
+ def observe_failure(self, route: RouteRuntime, reason: str) -> None:
+ with self.lock:
+ self.errors += 1
+ route.failures += 1
+ route.breaker.observe_failure(self.service.clock(), reason)
+
+ def add_response_bytes(self, route: RouteRuntime, count: int) -> None:
+ with self.lock:
+ route.bytes_to_client += count
+
+ def complete_response(self) -> None:
+ with self.lock:
+ self.responses += 1
+
+ def snapshot(self, now: float) -> dict[str, object]:
+ with self.lock:
+ self._expire_sessions(now)
+ return {
+ "name": self.config.name,
+ "protocol": self.config.protocol,
+ "active_route": self.selector.active_route,
+ "requests": self.requests,
+ "responses": self.responses,
+ "errors": self.errors,
+ "sessions": len(self.sessions),
+ "routes": [route.snapshot(now) for route in self.routes],
+ }
+
+
+class _GatewayHTTPServer(ThreadingHTTPServer):
+ daemon_threads = True
+ allow_reuse_address = False
+
+ def __init__(
+ self,
+ server_address: tuple[str, int],
+ handler_class: type[BaseHTTPRequestHandler],
+ *,
+ max_concurrent_requests: int,
+ ) -> None:
+ self._request_slots = threading.BoundedSemaphore(max_concurrent_requests)
+ super().__init__(server_address, handler_class)
+
+ @staticmethod
+ def _reject_overload(request: socket.socket) -> None:
+ body = b'{"error":"gateway_overloaded"}'
+ response = (
+ b"HTTP/1.1 503 Service Unavailable\r\n"
+ b"Content-Type: application/json\r\n"
+ + f"Content-Length: {len(body)}\r\n".encode()
+ + b"Retry-After: 1\r\nConnection: close\r\n\r\n"
+ + body
+ )
+ try:
+ request.settimeout(0.5)
+ request.sendall(response)
+ except OSError:
+ pass
+
+ def process_request(self, request: socket.socket, client_address: object) -> None:
+ if not self._request_slots.acquire(blocking=False):
+ self._reject_overload(request)
+ self.shutdown_request(request)
+ return
+ try:
+ super().process_request(request, client_address)
+ except BaseException:
+ self._request_slots.release()
+ raise
+
+ def process_request_thread(self, request: socket.socket, client_address: object) -> None:
+ try:
+ super().process_request_thread(request, client_address)
+ finally:
+ self._request_slots.release()
+
+
+class _GatewayHandler(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+ runtime: ListenerRuntime
+
+ def setup(self) -> None:
+ super().setup()
+ self.connection.settimeout(self.runtime.config.client_timeout_seconds)
+
+ def log_message(self, _format: str, *args: object) -> None:
+ return
+
+ def do_GET(self) -> None:
+ self._dispatch()
+
+ def do_HEAD(self) -> None:
+ self._dispatch()
+
+ def do_OPTIONS(self) -> None:
+ self._dispatch()
+
+ def do_POST(self) -> None:
+ self._dispatch()
+
+ def do_PUT(self) -> None:
+ self._dispatch()
+
+ def do_PATCH(self) -> None:
+ self._dispatch()
+
+ def do_DELETE(self) -> None:
+ self._dispatch()
+
+ def _json_response(self, status: int, code: str) -> None:
+ body = json.dumps({"error": code}, separators=(",", ":")).encode()
+ self.send_response(status)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ self.send_header("Connection", "close")
+ self.end_headers()
+ self.close_connection = True
+ if self.command != "HEAD":
+ try:
+ self.wfile.write(body)
+ except (BrokenPipeError, ConnectionResetError):
+ pass
+
+ def _status_response(self, metrics: bool) -> None:
+ if self.command not in {"GET", "HEAD"}:
+ self._json_response(405, "method_not_allowed")
+ return
+ if metrics:
+ body = self.runtime.service.metrics(self.runtime).encode()
+ content_type = "text/plain; version=0.0.4"
+ else:
+ body = json.dumps(
+ self.runtime.snapshot(self.runtime.service.clock()),
+ sort_keys=True,
+ separators=(",", ":"),
+ ).encode()
+ content_type = "application/json"
+ self.send_response(200)
+ self.send_header("Content-Type", content_type)
+ self.send_header("Content-Length", str(len(body)))
+ self.send_header("Cache-Control", "no-store")
+ self.send_header("Connection", "close")
+ self.end_headers()
+ self.close_connection = True
+ if self.command != "HEAD":
+ try:
+ self.wfile.write(body)
+ except (BrokenPipeError, ConnectionResetError):
+ pass
+
+ def _read_body(self) -> bytes | None:
+ if self.headers.get("Transfer-Encoding"):
+ self._json_response(400, "request_transfer_encoding_unsupported")
+ return None
+ values = self.headers.get_all("Content-Length", failobj=[])
+ if len(set(values)) > 1:
+ self._json_response(400, "conflicting_content_length")
+ return None
+ try:
+ length = int(values[0]) if values else 0
+ except ValueError:
+ self._json_response(400, "invalid_content_length")
+ return None
+ if length < 0:
+ self._json_response(400, "invalid_content_length")
+ return None
+ if length > self.runtime.config.max_body_bytes:
+ self._json_response(413, "request_body_too_large")
+ return None
+ try:
+ body = self.rfile.read(length) if length else b""
+ except socket.timeout:
+ self._json_response(408, "request_body_timeout")
+ return None
+ if len(body) != length:
+ self._json_response(400, "incomplete_request_body")
+ return None
+ return body
+
+ def _request_headers(self) -> dict[str, str]:
+ excluded = set(HOP_BY_HOP_HEADERS)
+ excluded.update(_connection_tokens(self.headers.get("Connection")))
+ excluded.update({"expect", "host", "x-local-failover-route"})
+ headers = {
+ key: value
+ for key, value in self.headers.items()
+ if key.lower() not in excluded
+ }
+ headers["Connection"] = "close"
+ return headers
+
+ def _upstream(
+ self, route: RouteConfig
+ ) -> tuple[http.client.HTTPConnection, str]:
+ base_url = route.resolve_url(self.runtime.service.env)
+ parsed = urlsplit(base_url)
+ assert parsed.hostname is not None
+ port = parsed.port or (443 if parsed.scheme == "https" else 80)
+ base_path = parsed.path.rstrip("/")
+ target = f"{base_path}/{self.path.lstrip('/')}"
+ if not target.startswith("/"):
+ target = f"/{target}"
+ if parsed.scheme == "https":
+ connection: http.client.HTTPConnection = http.client.HTTPSConnection(
+ parsed.hostname,
+ port,
+ timeout=self.runtime.config.connect_timeout_seconds,
+ context=ssl.create_default_context(),
+ )
+ else:
+ connection = http.client.HTTPConnection(
+ parsed.hostname,
+ port,
+ timeout=self.runtime.config.connect_timeout_seconds,
+ )
+ return connection, target
+
+ def _response_headers(
+ self, response: http.client.HTTPResponse, route_name: str
+ ) -> tuple[list[tuple[str, str]], int | None]:
+ connection_value = response.headers.get("Connection")
+ excluded = set(HOP_BY_HOP_HEADERS)
+ excluded.update(_connection_tokens(connection_value))
+ excluded.add("x-local-failover-route")
+ headers: list[tuple[str, str]] = []
+ content_length: int | None = None
+ for key, value in response.getheaders():
+ if key.lower() in excluded:
+ continue
+ if key.lower() == "content-length":
+ try:
+ content_length = int(value)
+ except ValueError:
+ continue
+ headers.append((key, value))
+ headers.append(("X-Local-Failover-Route", route_name))
+ headers.append(("Connection", "close"))
+ return headers, content_length
+
+ def _forward_response(
+ self,
+ response: http.client.HTTPResponse,
+ route: RouteRuntime,
+ ) -> None:
+ headers, content_length = self._response_headers(response, route.config.name)
+ session_id = response.headers.get("Mcp-Session-Id")
+ if (
+ self.runtime.config.protocol == "mcp"
+ and 200 <= response.status < 300
+ and session_id
+ ):
+ self.runtime.bind_session(
+ session_id, route.config.name, self.runtime.service.clock()
+ )
+ self.send_response(response.status, response.reason)
+ for key, value in headers:
+ self.send_header(key, value)
+ self.end_headers()
+ self.close_connection = True
+ if self.command == "HEAD":
+ self.runtime.complete_response()
+ return
+
+ written = 0
+ upstream_failed = False
+ while True:
+ try:
+ chunk = response.read1(65536)
+ except http.client.IncompleteRead as exc:
+ chunk = exc.partial
+ upstream_failed = True
+ except (OSError, http.client.HTTPException, socket.timeout):
+ chunk = b""
+ upstream_failed = True
+ if chunk:
+ try:
+ self.wfile.write(chunk)
+ self.wfile.flush()
+ except (BrokenPipeError, ConnectionResetError):
+ return
+ written += len(chunk)
+ self.runtime.add_response_bytes(route, len(chunk))
+ if upstream_failed or not chunk:
+ break
+ if content_length is not None and written != content_length:
+ upstream_failed = True
+ if upstream_failed:
+ self.runtime.observe_failure(route, "response_interrupted")
+ else:
+ self.runtime.complete_response()
+
+ def _dispatch(self) -> None:
+ if self.path == "/_failover/status":
+ self._status_response(metrics=False)
+ return
+ if self.path == "/_failover/metrics":
+ self._status_response(metrics=True)
+ return
+ body = self._read_body()
+ if body is None:
+ return
+
+ now = self.runtime.service.clock()
+ session_route: str | None = None
+ if self.runtime.config.protocol == "mcp":
+ incoming_session = self.headers.get("Mcp-Session-Id")
+ if incoming_session:
+ session_route = self.runtime.session_route(incoming_session, now)
+ if session_route is None:
+ self._json_response(503, "session_reconnect_required")
+ return
+
+ safe_retry = self.command in IDEMPOTENT_METHODS and session_route is None
+ attempts = self.runtime.config.max_idempotent_attempts if safe_retry else 1
+ excluded: set[str] = set()
+ headers = self._request_headers()
+
+ for attempt in range(attempts):
+ route = self.runtime.choose(session_route, self.runtime.service.clock(), excluded)
+ if route is None:
+ code = (
+ "session_reconnect_required"
+ if session_route is not None
+ else "no_healthy_route"
+ )
+ self._json_response(503, code)
+ return
+ self.runtime.begin_attempt(route)
+ connection: http.client.HTTPConnection | None = None
+ try:
+ connection, target = self._upstream(route.config)
+ connection.request(self.command, target, body=body, headers=headers)
+ response = connection.getresponse()
+ if connection.sock:
+ connection.sock.settimeout(self.runtime.config.idle_timeout_seconds)
+ except (OSError, http.client.HTTPException, ssl.SSLError, ValueError):
+ if connection:
+ connection.close()
+ self.runtime.observe_failure(route, "upstream_connect_error")
+ excluded.add(route.config.name)
+ if attempt + 1 < attempts:
+ continue
+ self._json_response(502, "upstream_unavailable")
+ return
+
+ if response.status in RETRYABLE_STATUSES:
+ self.runtime.observe_failure(route, f"upstream_http_{response.status}")
+ if safe_retry and attempt + 1 < attempts:
+ excluded.add(route.config.name)
+ alternative = self.runtime.choose(
+ None, self.runtime.service.clock(), excluded
+ )
+ if alternative is not None:
+ response.close()
+ connection.close()
+ continue
+ try:
+ self._forward_response(response, route)
+ finally:
+ response.close()
+ connection.close()
+ self.runtime.service.persist_state()
+ return
+
+
+class GatewayService:
+ """Owns loopback listeners, health workers, and supervised route tunnels."""
+
+ def __init__(
+ self,
+ config: GatewayConfig,
+ *,
+ env: Mapping[str, str] | None = None,
+ clock=time.monotonic,
+ semantic_health: SemanticHealth | None = None,
+ ) -> None:
+ self.config = config
+ self.env = dict(os.environ if env is None else env)
+ self.clock = clock
+ self.semantic_health = semantic_health or SemanticHealth()
+ self.listeners = {
+ listener.name: ListenerRuntime(self, listener)
+ for listener in config.listeners
+ }
+ self._servers: dict[str, _GatewayHTTPServer] = {}
+ self._server_threads: list[threading.Thread] = []
+ self._health_thread: threading.Thread | None = None
+ self._stop = threading.Event()
+ self._started = False
+ self._persist_lock = threading.Lock()
+ self._state_persistence = "disabled" if config.state_file is None else "pending"
+ self._next_probe: dict[tuple[str, str], float] = {}
+ self._tunnels: dict[tuple[str, str], TunnelSupervisor] = {}
+ for listener in self.listeners.values():
+ for route in listener.routes:
+ if route.config.tunnel is not None:
+ self._tunnels[(listener.config.name, route.config.name)] = (
+ TunnelSupervisor(route.config.tunnel, env=self.env)
+ )
+
+ def start(self, *, run_health: bool = True) -> None:
+ if self._started:
+ raise RuntimeError("gateway is already started")
+ self._stop.clear()
+ try:
+ for name, runtime in self.listeners.items():
+ handler = type(
+ f"GatewayHandler_{name}",
+ (_GatewayHandler,),
+ {"runtime": runtime},
+ )
+ server = _GatewayHTTPServer(
+ (runtime.config.host, runtime.config.port),
+ handler,
+ max_concurrent_requests=runtime.config.max_concurrent_requests,
+ )
+ thread = threading.Thread(
+ target=server.serve_forever,
+ name=f"local-failover-{name}",
+ daemon=True,
+ )
+ self._servers[name] = server
+ self._server_threads.append(thread)
+ thread.start()
+ self._started = True
+ if run_health:
+ self._health_thread = threading.Thread(
+ target=self._health_loop,
+ name="local-failover-health",
+ daemon=True,
+ )
+ self._health_thread.start()
+ self.persist_state()
+ except BaseException:
+ self.shutdown()
+ raise
+
+ def _set_gate(self, route: RouteRuntime, ready: bool, reason: str) -> None:
+ route.gate_ready = ready and route.config.provisioned
+ route.gate_reason = reason if route.config.provisioned else "not_provisioned"
+
+ def _route_gate(self, listener: ListenerRuntime, route: RouteRuntime, now: float) -> bool:
+ tunnel = self._tunnels.get((listener.config.name, route.config.name))
+ if tunnel is not None:
+ status = tunnel.tick(now)
+ self._set_gate(route, status.ready, status.reason)
+ return route.gate_ready
+ ready, reason = _network_gate_status(route.config.network_gate)
+ self._set_gate(route, ready, reason)
+ return route.gate_ready
+
+ def _observe_probe(
+ self, route: RouteRuntime, result: ProbeResult, now: float
+ ) -> None:
+ if result.kind == "success":
+ route.breaker.observe_success(now, result.reason)
+ elif result.kind == "failure":
+ route.breaker.observe_failure(now, result.reason)
+ else:
+ route.breaker.observe_unknown(now, result.reason)
+
+ def _health_once(self) -> None:
+ now = self.clock()
+ for listener in self.listeners.values():
+ for route in listener.routes:
+ if not self._route_gate(listener, route, now):
+ continue
+ key = (listener.config.name, route.config.name)
+ if now < self._next_probe.get(key, 0.0):
+ continue
+ if route.breaker.state == "open" and now < route.breaker.open_until:
+ self._next_probe[key] = route.breaker.open_until
+ continue
+ result = self.semantic_health.probe(route.config, self.env)
+ with listener.lock:
+ self._observe_probe(route, result, now)
+ if result.kind == "failure" and route.breaker.state == "open":
+ self._next_probe[key] = max(now + 0.1, route.breaker.open_until)
+ else:
+ self._next_probe[key] = now + route.config.health.interval_seconds
+ self.persist_state()
+
+ def _health_loop(self) -> None:
+ while not self._stop.is_set():
+ self._health_once()
+ self._stop.wait(0.5)
+
+ def metrics(self, runtime: ListenerRuntime) -> str:
+ now = self.clock()
+ snapshot = runtime.snapshot(now)
+ lines = [
+ "# HELP local_failover_requests_total Upstream request attempts.",
+ "# TYPE local_failover_requests_total counter",
+ ]
+ for route in snapshot["routes"]:
+ name = route["name"]
+ labels = f'listener="{runtime.config.name}",route="{name}"'
+ lines.append(
+ f"local_failover_route_requests_total{{{labels}}} {route['requests']}"
+ )
+ lines.append(
+ f"local_failover_route_failures_total{{{labels}}} {route['failures']}"
+ )
+ lines.append(
+ f"local_failover_route_available{{{labels}}} "
+ f"{1 if route['circuit']['available'] and route['gate_ready'] else 0}"
+ )
+ return "\n".join(lines) + "\n"
+
+ def _snapshot(self) -> dict[str, object]:
+ now = self.clock()
+ return {
+ "version": 1,
+ "state_persistence": self._state_persistence,
+ "listeners": [runtime.snapshot(now) for runtime in self.listeners.values()],
+ }
+
+ def persist_state(self) -> None:
+ path = self.config.state_file
+ if path is None:
+ return
+ with self._persist_lock:
+ path = Path(path)
+ descriptor = -1
+ temporary: str | None = None
+ try:
+ path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
+ descriptor, temporary = tempfile.mkstemp(
+ dir=path.parent, prefix=f".{path.name}.", text=True
+ )
+ os.fchmod(descriptor, 0o600)
+ with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
+ descriptor = -1
+ json.dump(self._snapshot(), handle, sort_keys=True, separators=(",", ":"))
+ handle.write("\n")
+ handle.flush()
+ os.fsync(handle.fileno())
+ os.replace(temporary, path)
+ temporary = None
+ os.chmod(path, 0o600)
+ self._state_persistence = "ok"
+ except OSError:
+ self._state_persistence = "error"
+ finally:
+ if descriptor >= 0:
+ os.close(descriptor)
+ if temporary is not None:
+ try:
+ os.unlink(temporary)
+ except FileNotFoundError:
+ pass
+
+ def shutdown(self) -> None:
+ self._stop.set()
+ for server in self._servers.values():
+ server.shutdown()
+ server.server_close()
+ for thread in self._server_threads:
+ thread.join(timeout=3)
+ if self._health_thread is not None:
+ self._health_thread.join(timeout=3)
+ for tunnel in self._tunnels.values():
+ tunnel.shutdown()
+ self._servers.clear()
+ self._server_threads.clear()
+ self._health_thread = None
+ self._started = False
diff --git a/claude-ops/scripts/local-failover/local_failover/tunnel.py b/claude-ops/scripts/local-failover/local_failover/tunnel.py
new file mode 100644
index 00000000..adef562d
--- /dev/null
+++ b/claude-ops/scripts/local-failover/local_failover/tunnel.py
@@ -0,0 +1,272 @@
+"""Ownership-safe supervision for an optional local forwarding tunnel."""
+
+from __future__ import annotations
+
+import os
+import re
+import shutil
+import signal
+import socket
+import subprocess
+import time
+from dataclasses import dataclass
+from typing import Callable, Mapping
+
+from .config import TunnelConfig
+
+
+ENV_REFERENCE = re.compile(r"\$\{([A-Z_][A-Z0-9_]*)\}")
+
+
+@dataclass(frozen=True)
+class TunnelStatus:
+ state: str
+ ready: bool
+ restart_count: int
+ reason: str
+
+
+def listener_bound(host: str, port: int) -> bool:
+ try:
+ with socket.create_connection((host, port), timeout=0.2):
+ return True
+ except OSError:
+ return False
+
+
+def listener_owned_by_process_group(host: str, port: int, process_group: int) -> bool:
+ """Fail closed unless every listener on the port belongs to the owned process group."""
+
+ del host # lsof filters the port; listener_bound already checks the configured address.
+ lsof = shutil.which("lsof")
+ if lsof is None or process_group <= 0:
+ return False
+ try:
+ result = subprocess.run(
+ [lsof, "-nP", "-a", f"-iTCP:{port}", "-sTCP:LISTEN", "-Fp"],
+ stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL,
+ text=True,
+ timeout=2,
+ check=False,
+ )
+ except (OSError, ProcessLookupError, subprocess.TimeoutExpired):
+ return False
+ if result.returncode != 0:
+ return False
+ listener_pids = {
+ int(line[1:])
+ for line in result.stdout.splitlines()
+ if line.startswith("p") and line[1:].isdigit()
+ }
+ if not listener_pids:
+ return False
+ try:
+ return all(
+ os.getpgid(listener_pid) == process_group for listener_pid in listener_pids
+ )
+ except (OSError, ProcessLookupError):
+ return False
+
+
+class TunnelSupervisor:
+ """Starts only an unbound listener and trusts only its live child process."""
+
+ def __init__(
+ self,
+ config: TunnelConfig,
+ *,
+ env: Mapping[str, str] | None = None,
+ process_factory: Callable = subprocess.Popen,
+ listener_checker: Callable[[str, int], bool] = listener_bound,
+ ownership_checker: Callable[[str, int, int], bool] = (
+ listener_owned_by_process_group
+ ),
+ process_group_getter: Callable[[int], int] = os.getpgid,
+ process_group_signaler: Callable[[int, int], None] = os.killpg,
+ ) -> None:
+ self.config = config
+ self.env = dict(env if env is not None else os.environ)
+ self.process_factory = process_factory
+ self.listener_checker = listener_checker
+ self.ownership_checker = ownership_checker
+ self.process_group_getter = process_group_getter
+ self.process_group_signaler = process_group_signaler
+ self.process = None
+ self.process_group: int | None = None
+ self.started_at: float | None = None
+ self.next_restart = 0.0
+ self.restart_count = 0
+ self._state = "disabled" if not config.enabled else "stopped"
+ self._reason = "disabled" if not config.enabled else "not_started"
+ self._observed_free_before_start = False
+
+ @property
+ def ready(self) -> bool:
+ return self._state == "bound" and self.process is not None and self.process.poll() is None
+
+ def _status(self) -> TunnelStatus:
+ return TunnelStatus(self._state, self.ready, self.restart_count, self._reason)
+
+ def _resolve_argv(self) -> tuple[str, ...] | None:
+ resolved: list[str] = []
+ for item in self.config.argv:
+ missing = False
+
+ def replace(match: re.Match[str]) -> str:
+ nonlocal missing
+ value = self.env.get(match.group(1))
+ if value is None or value == "":
+ missing = True
+ return ""
+ return value
+
+ value = ENV_REFERENCE.sub(replace, item)
+ if missing:
+ return None
+ resolved.append(value)
+ return tuple(resolved)
+
+ def _backoff(self, now: float, reason: str) -> None:
+ self.restart_count += 1
+ exponent = min(self.restart_count - 1, 30)
+ delay = min(
+ self.config.restart_base_seconds * (2**exponent),
+ self.config.restart_max_seconds,
+ )
+ self.next_restart = now + delay
+ self._state = "backoff"
+ self._reason = reason
+ self.process = None
+ self.process_group = None
+ self.started_at = None
+ self._observed_free_before_start = False
+
+ def _stop_owned(self) -> None:
+ process = self.process
+ process_group = self.process_group
+ if process is None:
+ return
+ if process_group is None:
+ if process.poll() is None:
+ try:
+ process.terminate()
+ process.wait(timeout=3)
+ except (OSError, subprocess.TimeoutExpired):
+ try:
+ process.kill()
+ process.wait(timeout=1)
+ except (OSError, subprocess.TimeoutExpired):
+ pass
+ return
+ if process.poll() is None:
+ try:
+ if self.process_group_getter(process.pid) != process_group:
+ return
+ except (OSError, ProcessLookupError):
+ return
+ elif not (
+ self.listener_checker(self.config.listener_host, self.config.listener_port)
+ and self.ownership_checker(
+ self.config.listener_host,
+ self.config.listener_port,
+ process_group,
+ )
+ ):
+ return
+ try:
+ self.process_group_signaler(process_group, signal.SIGTERM)
+ except (OSError, ProcessLookupError):
+ return
+ try:
+ process.wait(timeout=0.2)
+ except (OSError, subprocess.TimeoutExpired):
+ pass
+ deadline = time.monotonic() + 3
+ while time.monotonic() < deadline:
+ try:
+ self.process_group_signaler(process_group, 0)
+ except (OSError, ProcessLookupError):
+ return
+ time.sleep(0.05)
+ try:
+ self.process_group_signaler(process_group, signal.SIGKILL)
+ except (OSError, ProcessLookupError):
+ return
+ try:
+ process.wait(timeout=1)
+ except (OSError, subprocess.TimeoutExpired):
+ pass
+
+ def tick(self, now: float) -> TunnelStatus:
+ if not self.config.enabled:
+ return self._status()
+ if self.process is not None:
+ if self.process.poll() is not None:
+ self._stop_owned()
+ self._backoff(now, "process_exited")
+ return self._status()
+ bound = self.listener_checker(
+ self.config.listener_host, self.config.listener_port
+ )
+ if bound and self._observed_free_before_start:
+ if not self.ownership_checker(
+ self.config.listener_host,
+ self.config.listener_port,
+ self.process_group or 0,
+ ):
+ self._stop_owned()
+ self._backoff(now, "listener_ownership_mismatch")
+ return self._status()
+ self._state = "bound"
+ self._reason = "owned_process_bound"
+ return self._status()
+ if (
+ self.started_at is not None
+ and now - self.started_at > self.config.startup_timeout_seconds
+ ):
+ self._stop_owned()
+ self._backoff(now, "startup_timeout")
+ return self._status()
+ self._state = "starting"
+ self._reason = "waiting_for_listener"
+ return self._status()
+ if now < self.next_restart:
+ self._state = "backoff"
+ return self._status()
+ if self.listener_checker(self.config.listener_host, self.config.listener_port):
+ self._state = "conflict"
+ self._reason = "preexisting_listener"
+ return self._status()
+ argv = self._resolve_argv()
+ if argv is None:
+ self._state = "configuration_unknown"
+ self._reason = "tunnel_environment_missing"
+ return self._status()
+ self._observed_free_before_start = True
+ try:
+ self.process = self.process_factory(
+ argv,
+ stdin=subprocess.DEVNULL,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ shell=False,
+ start_new_session=True,
+ close_fds=True,
+ )
+ except OSError:
+ self._backoff(now, "process_start_failed")
+ return self._status()
+ self.started_at = now
+ self.process_group = self.process.pid
+ self._state = "starting"
+ self._reason = "process_started"
+ return self._status()
+
+ def shutdown(self) -> None:
+ self._stop_owned()
+ self.process = None
+ self.process_group = None
+ self._state = "stopped"
+ self._reason = "shutdown"
diff --git a/claude-ops/scripts/local-failover/plans/aws-client-vpn.yaml b/claude-ops/scripts/local-failover/plans/aws-client-vpn.yaml
new file mode 100644
index 00000000..274ccdc7
--- /dev/null
+++ b/claude-ops/scripts/local-failover/plans/aws-client-vpn.yaml
@@ -0,0 +1,129 @@
+AWSTemplateFormatVersion: "2010-09-09"
+Description: >-
+ REVIEW ONLY. Optional managed laptop access with AWS Client VPN. Deploying this
+ template creates hourly-billed resources and requires separate authorization.
+
+Parameters:
+ VpcId:
+ Type: AWS::EC2::VPC::Id
+ TargetSubnetId:
+ Type: AWS::EC2::Subnet::Id
+ ExistingSecurityGroupId:
+ Type: AWS::EC2::SecurityGroup::Id
+ ServerCertificateArn:
+ Type: String
+ Description: Existing ACM server-certificate ARN; required for every auth mode.
+ ClientCidrBlock:
+ Type: String
+ Default: 172.31.240.0/22
+ Description: Non-overlapping client CIDR, /12 through /22.
+ VpcDestinationCidr:
+ Type: String
+ Description: CIDR clients are authorized to reach.
+ DnsServers:
+ Type: CommaDelimitedList
+ Description: Existing reachable DNS server addresses.
+ AuthType:
+ Type: String
+ AllowedValues: [certificate, directory, saml]
+ ClientRootCertificateChainArn:
+ Type: String
+ Default: ""
+ Description: Required when AuthType is certificate.
+ DirectoryId:
+ Type: String
+ Default: ""
+ Description: Required when AuthType is directory.
+ SAMLProviderArn:
+ Type: String
+ Default: ""
+ Description: Required when AuthType is saml.
+ SelfServiceSAMLProviderArn:
+ Type: String
+ Default: ""
+ AccessGroupId:
+ Type: String
+ Default: ""
+ Description: Optional AD/SAML group; empty authorizes all authenticated users.
+ AdditionalDestinationCidr:
+ Type: String
+ Default: ""
+ Description: Optional additional route; empty creates no explicit route.
+
+Conditions:
+ UseCertificate: !Equals [!Ref AuthType, certificate]
+ UseDirectory: !Equals [!Ref AuthType, directory]
+ UseSAML: !Equals [!Ref AuthType, saml]
+ HasSelfServiceSAML: !Not [!Equals [!Ref SelfServiceSAMLProviderArn, ""]]
+ HasAccessGroup: !Not [!Equals [!Ref AccessGroupId, ""]]
+ HasAdditionalRoute: !Not [!Equals [!Ref AdditionalDestinationCidr, ""]]
+
+Resources:
+ Endpoint:
+ Type: AWS::EC2::ClientVpnEndpoint
+ Properties:
+ VpcId: !Ref VpcId
+ ClientCidrBlock: !Ref ClientCidrBlock
+ ServerCertificateArn: !Ref ServerCertificateArn
+ SplitTunnel: true
+ TransportProtocol: udp
+ VpnPort: 443
+ SecurityGroupIds: [!Ref ExistingSecurityGroupId]
+ DnsServers: !Ref DnsServers
+ ConnectionLogOptions:
+ Enabled: false
+ AuthenticationOptions:
+ - !If
+ - UseCertificate
+ - Type: certificate-authentication
+ MutualAuthentication:
+ ClientRootCertificateChainArn: !Ref ClientRootCertificateChainArn
+ - !Ref AWS::NoValue
+ - !If
+ - UseDirectory
+ - Type: directory-service-authentication
+ ActiveDirectory:
+ DirectoryId: !Ref DirectoryId
+ - !Ref AWS::NoValue
+ - !If
+ - UseSAML
+ - Type: federated-authentication
+ FederatedAuthentication:
+ SAMLProviderArn: !Ref SAMLProviderArn
+ SelfServiceSAMLProviderArn: !If
+ - HasSelfServiceSAML
+ - !Ref SelfServiceSAMLProviderArn
+ - !Ref AWS::NoValue
+ - !Ref AWS::NoValue
+ TagSpecifications:
+ - ResourceType: client-vpn-endpoint
+ Tags:
+ - Key: Purpose
+ Value: local-access-review
+
+ TargetNetwork:
+ Type: AWS::EC2::ClientVpnTargetNetworkAssociation
+ Properties:
+ ClientVpnEndpointId: !Ref Endpoint
+ SubnetId: !Ref TargetSubnetId
+
+ Authorization:
+ Type: AWS::EC2::ClientVpnAuthorizationRule
+ Properties:
+ ClientVpnEndpointId: !Ref Endpoint
+ TargetNetworkCidr: !Ref VpcDestinationCidr
+ AuthorizeAllGroups: !If [HasAccessGroup, false, true]
+ AccessGroupId: !If [HasAccessGroup, !Ref AccessGroupId, !Ref AWS::NoValue]
+
+ AdditionalRoute:
+ Type: AWS::EC2::ClientVpnRoute
+ Condition: HasAdditionalRoute
+ DependsOn: TargetNetwork
+ Properties:
+ ClientVpnEndpointId: !Ref Endpoint
+ DestinationCidrBlock: !Ref AdditionalDestinationCidr
+ TargetVpcSubnetId: !Ref TargetSubnetId
+
+Outputs:
+ ReviewWarning:
+ Value: Do not deploy without explicit cost, network, authentication, and security approval.
diff --git a/claude-ops/scripts/local-failover/tests/run.sh b/claude-ops/scripts/local-failover/tests/run.sh
new file mode 100755
index 00000000..802ac7ad
--- /dev/null
+++ b/claude-ops/scripts/local-failover/tests/run.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+PLUGIN_ROOT="$(cd "$ROOT/../.." && pwd)"
+
+cd "$PLUGIN_ROOT"
+python3 -m unittest discover -s "$ROOT/tests" -p 'test_*.py' -v
+python3 -m py_compile "$ROOT"/local_failover/*.py "$ROOT/vpc_paths.py"
+python3 -m json.tool "$ROOT/config.example.json" >/dev/null
+PYTHONPATH="$ROOT" python3 -m local_failover check --config "$ROOT/config.example.json" >/dev/null
+plutil -lint "$ROOT/launchd/com.example.local-failover.plist.template"
+git diff --check
diff --git a/claude-ops/scripts/local-failover/tests/test_config_circuit.py b/claude-ops/scripts/local-failover/tests/test_config_circuit.py
new file mode 100644
index 00000000..93146b2b
--- /dev/null
+++ b/claude-ops/scripts/local-failover/tests/test_config_circuit.py
@@ -0,0 +1,268 @@
+from __future__ import annotations
+
+import sys
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from local_failover.circuit import CircuitBreaker, RouteRuntime, RouteSelector
+from local_failover.config import ConfigError, parse_config
+
+
+def valid_config() -> dict:
+ return {
+ "version": 1,
+ "state_file": "~/.local/state/local-failover/status.json",
+ "listeners": [
+ {
+ "name": "cli",
+ "host": "127.0.0.1",
+ "port": 18316,
+ "protocol": "llm",
+ "max_body_bytes": 1048576,
+ "max_concurrent_requests": 64,
+ "max_idempotent_attempts": 2,
+ "client_timeout_seconds": 10,
+ "session_ttl_seconds": 3600,
+ "routes": [
+ {
+ "name": "cloud-primary",
+ "url": "https://gateway.example.com",
+ "provisioned": True,
+ "health": {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "inventory_path": "/v1/models",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ },
+ },
+ {
+ "name": "overlay-secondary",
+ "url": "http://100.64.0.10:8317",
+ "trusted_overlay_http": True,
+ "provisioned": True,
+ "health": {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "inventory_path": "/v1/models",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ },
+ },
+ {
+ "name": "tunnel-tertiary",
+ "url": "http://127.0.0.1:18317",
+ "provisioned": False,
+ "health": {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "inventory_path": "/v1/models",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ },
+ "tunnel": {
+ "enabled": False,
+ "argv": ["aws", "ssm", "start-session"],
+ "listener_host": "127.0.0.1",
+ "listener_port": 18317,
+ },
+ },
+ ],
+ }
+ ],
+ }
+
+
+class ConfigTests(unittest.TestCase):
+ def test_parses_and_preserves_route_order(self) -> None:
+ parsed = parse_config(valid_config())
+
+ self.assertEqual(parsed.version, 1)
+ self.assertEqual(parsed.listeners[0].host, "127.0.0.1")
+ self.assertEqual(
+ [route.name for route in parsed.listeners[0].routes],
+ ["cloud-primary", "overlay-secondary", "tunnel-tertiary"],
+ )
+
+ def test_listener_rejects_non_loopback_bind(self) -> None:
+ raw = valid_config()
+ raw["listeners"][0]["host"] = "0.0.0.0"
+
+ with self.assertRaisesRegex(ConfigError, "loopback"):
+ parse_config(raw)
+
+ def test_cleartext_non_loopback_requires_explicit_overlay_trust(self) -> None:
+ raw = valid_config()
+ raw["listeners"][0]["routes"][1].pop("trusted_overlay_http")
+
+ with self.assertRaisesRegex(ConfigError, "trusted_overlay_http"):
+ parse_config(raw)
+
+ def test_upstream_url_rejects_credentials_query_and_fragment(self) -> None:
+ for url in (
+ "https://user:[email protected]",
+ "https://gateway.example.com?token=hidden",
+ "https://gateway.example.com/#fragment",
+ ):
+ with self.subTest(url=url):
+ raw = valid_config()
+ raw["listeners"][0]["routes"][0]["url"] = url
+ with self.assertRaises(ConfigError):
+ parse_config(raw)
+
+ def test_route_can_resolve_url_from_environment_without_storing_value(self) -> None:
+ raw = valid_config()
+ route = raw["listeners"][0]["routes"][0]
+ route.pop("url")
+ route["url_env"] = "FAILOVER_PRIMARY_URL"
+ parsed = parse_config(raw)
+
+ self.assertEqual(
+ parsed.listeners[0].routes[0].resolve_url(
+ {"FAILOVER_PRIMARY_URL": "https://gateway.example.com"}
+ ),
+ "https://gateway.example.com",
+ )
+ with self.assertRaisesRegex(ConfigError, "FAILOVER_PRIMARY_URL"):
+ parsed.listeners[0].routes[0].resolve_url({})
+
+ def test_rejects_duplicate_route_names_and_invalid_environment_names(self) -> None:
+ duplicate = valid_config()
+ duplicate["listeners"][0]["routes"][1]["name"] = "cloud-primary"
+ with self.assertRaisesRegex(ConfigError, "unique"):
+ parse_config(duplicate)
+
+ invalid_env = valid_config()
+ invalid_env["listeners"][0]["routes"][0]["health"]["auth_env"] = "BAD-NAME"
+ with self.assertRaisesRegex(ConfigError, "environment"):
+ parse_config(invalid_env)
+
+ def test_enabled_tunnel_must_be_loopback_and_provisioned(self) -> None:
+ raw = valid_config()
+ tunnel_route = raw["listeners"][0]["routes"][2]
+ tunnel_route["provisioned"] = True
+ tunnel_route["tunnel"]["enabled"] = True
+ tunnel_route["tunnel"]["listener_host"] = "100.64.0.10"
+
+ with self.assertRaisesRegex(ConfigError, "tunnel listener"):
+ parse_config(raw)
+
+ def test_listener_request_limits_are_bounded(self) -> None:
+ raw = valid_config()
+ raw["listeners"][0]["max_concurrent_requests"] = 0
+ with self.assertRaisesRegex(ConfigError, "max_concurrent_requests"):
+ parse_config(raw)
+
+ raw = valid_config()
+ raw["listeners"][0]["client_timeout_seconds"] = 0
+ with self.assertRaisesRegex(ConfigError, "client_timeout_seconds"):
+ parse_config(raw)
+
+
+class CircuitTests(unittest.TestCase):
+ def setUp(self) -> None:
+ config = parse_config(valid_config())
+ self.policy = config.listeners[0].routes[0].breaker
+
+ def test_starts_unknown_and_requires_stable_successes(self) -> None:
+ breaker = CircuitBreaker(self.policy, now=0)
+
+ self.assertFalse(breaker.available(0))
+ breaker.observe_success(1, "semantic_ok")
+ breaker.observe_success(2, "semantic_ok")
+ self.assertFalse(breaker.available(2))
+ breaker.observe_success(3, "semantic_ok")
+ self.assertEqual(breaker.state, "closed")
+ self.assertFalse(breaker.available(7))
+ self.assertTrue(breaker.available(8))
+
+ def test_failure_threshold_opens_with_capped_exponential_backoff(self) -> None:
+ breaker = CircuitBreaker(self.policy, now=0)
+ for timestamp in (1, 2, 3):
+ breaker.observe_success(timestamp, "semantic_ok")
+ breaker.observe_failure(8, "connect")
+ self.assertEqual(breaker.state, "closed")
+ breaker.observe_failure(9, "connect")
+ self.assertEqual(breaker.state, "open")
+ first_until = breaker.open_until
+
+ breaker.observe_success(first_until, "semantic_ok")
+ breaker.observe_failure(first_until + 1, "connect")
+ self.assertEqual(breaker.state, "open")
+ self.assertGreater(breaker.open_until, first_until + self.policy.base_backoff_seconds)
+ self.assertLessEqual(
+ breaker.open_until - (first_until + 1), self.policy.max_backoff_seconds
+ )
+
+ def test_unknown_probe_does_not_count_as_provider_failure_but_stales(self) -> None:
+ breaker = CircuitBreaker(self.policy, now=0)
+ for timestamp in (1, 2, 3):
+ breaker.observe_success(timestamp, "semantic_ok")
+ self.assertTrue(breaker.available(8))
+
+ breaker.observe_unknown(9, "health_auth_rejected")
+ self.assertEqual(breaker.state, "closed")
+ self.assertEqual(breaker.consecutive_failures, 0)
+ self.assertFalse(breaker.available(3 + self.policy.max_stale_seconds + 1))
+
+ def test_recovery_failure_reopens_immediately(self) -> None:
+ breaker = CircuitBreaker(self.policy, now=0)
+ breaker.observe_failure(1, "connect")
+ breaker.observe_failure(2, "connect")
+ breaker.observe_success(breaker.open_until, "semantic_ok")
+ self.assertEqual(breaker.state, "half_open")
+
+ breaker.observe_failure(breaker.open_until + 1, "connect")
+
+ self.assertEqual(breaker.state, "open")
+
+
+class SelectorTests(unittest.TestCase):
+ def _runtime(self, route, successes=(1, 2, 3), gate_ready=True) -> RouteRuntime:
+ runtime = RouteRuntime(route, CircuitBreaker(route.breaker, now=0))
+ runtime.gate_ready = gate_ready
+ for timestamp in successes:
+ runtime.breaker.observe_success(timestamp, "semantic_ok")
+ return runtime
+
+ def test_uses_fixed_order_and_skips_unprovisioned_or_unready_routes(self) -> None:
+ listener = parse_config(valid_config()).listeners[0]
+ primary = self._runtime(listener.routes[0], gate_ready=False)
+ secondary = self._runtime(listener.routes[1])
+ tertiary = self._runtime(listener.routes[2])
+ selector = RouteSelector([primary, secondary, tertiary])
+
+ selected = selector.choose(None, now=8)
+
+ self.assertEqual(selected.config.name, "overlay-secondary")
+
+ def test_failback_waits_for_primary_stability_and_resists_flapping(self) -> None:
+ listener = parse_config(valid_config()).listeners[0]
+ primary = RouteRuntime(listener.routes[0], CircuitBreaker(listener.routes[0].breaker, 0))
+ secondary = self._runtime(listener.routes[1])
+ selector = RouteSelector([primary, secondary])
+ self.assertEqual(selector.choose(None, 8).config.name, "overlay-secondary")
+
+ for timestamp in (9, 10, 11):
+ primary.breaker.observe_success(timestamp, "semantic_ok")
+ self.assertEqual(selector.choose(None, 15).config.name, "overlay-secondary")
+ primary.breaker.observe_failure(15, "stream")
+ primary.breaker.observe_failure(16, "stream")
+ self.assertEqual(selector.choose(None, 16).config.name, "overlay-secondary")
+
+ def test_session_route_never_migrates(self) -> None:
+ listener = parse_config(valid_config()).listeners[0]
+ primary = self._runtime(listener.routes[0], gate_ready=False)
+ secondary = self._runtime(listener.routes[1])
+ selector = RouteSelector([primary, secondary])
+
+ self.assertIsNone(selector.choose("cloud-primary", now=8))
+ self.assertEqual(selector.choose(None, now=8).config.name, "overlay-secondary")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/claude-ops/scripts/local-failover/tests/test_gateway_integration.py b/claude-ops/scripts/local-failover/tests/test_gateway_integration.py
new file mode 100644
index 00000000..aef53f61
--- /dev/null
+++ b/claude-ops/scripts/local-failover/tests/test_gateway_integration.py
@@ -0,0 +1,511 @@
+from __future__ import annotations
+
+import http.client
+import json
+import socket
+import stat
+import sys
+import tempfile
+import threading
+import time
+import unittest
+from dataclasses import replace
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from local_failover.config import parse_config
+from local_failover.health import HTTPProbeTransport, SemanticHealth
+from local_failover.proxy import GatewayService
+
+
+def free_port() -> int:
+ with socket.socket() as sock:
+ sock.bind(("127.0.0.1", 0))
+ return sock.getsockname()[1]
+
+
+class OriginState:
+ def __init__(self, name: str) -> None:
+ self.name = name
+ self.requests: list[dict] = []
+ self.resource_status = 200
+ self.expected_auth = "Bearer test-health-credential"
+ self.block_started = threading.Event()
+ self.block_release = threading.Event()
+
+
+class MockOriginHandler(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ @property
+ def state(self) -> OriginState:
+ return self.server.state
+
+ def log_message(self, _format, *args) -> None:
+ return
+
+ def _body(self) -> bytes:
+ length = int(self.headers.get("Content-Length", "0"))
+ return self.rfile.read(length) if length else b""
+
+ def _record(self, body: bytes) -> None:
+ self.state.requests.append(
+ {
+ "method": self.command,
+ "path": self.path,
+ "headers": {key.lower(): value for key, value in self.headers.items()},
+ "body": body,
+ }
+ )
+
+ def _send(self, status: int, body: bytes, **headers: str) -> None:
+ self.send_response(status)
+ self.send_header("Content-Length", str(len(body)))
+ self.send_header("Content-Type", headers.pop("content_type", "application/json"))
+ for key, value in headers.items():
+ self.send_header(key, value)
+ self.end_headers()
+ if self.command != "HEAD":
+ self.wfile.write(body)
+
+ def do_HEAD(self) -> None:
+ self.do_GET()
+
+ def do_GET(self) -> None:
+ self._record(b"")
+ if self.path == "/v1/models":
+ if self.headers.get("Authorization") != self.state.expected_auth:
+ self._send(401, b'{"error":"unauthorized"}')
+ return
+ self._send(200, b'{"object":"list","data":[{"id":"configured"}]}')
+ return
+ if self.path == "/resource":
+ self._send(
+ self.state.resource_status,
+ json.dumps({"origin": self.state.name}).encode(),
+ )
+ return
+ if self.path == "/block":
+ self.state.block_started.set()
+ self.state.block_release.wait(timeout=2)
+ self._send(200, json.dumps({"origin": self.state.name}).encode())
+ return
+ self._send(404, b'{"error":"missing"}')
+
+ def do_POST(self) -> None:
+ body = self._body()
+ self._record(body)
+ if self.path == "/v1/chat/completions":
+ if self.headers.get("Authorization") != self.state.expected_auth:
+ self._send(401, b'{"error":"unauthorized"}')
+ return
+ payload = b"data: first\n\ndata: [DONE]\n\n"
+ self._send(200, payload, content_type="text/event-stream")
+ return
+ if self.path == "/interrupt-before-headers":
+ self.connection.shutdown(socket.SHUT_RDWR)
+ self.connection.close()
+ return
+ if self.path == "/stream-interrupt":
+ self.send_response(200)
+ self.send_header("Content-Type", "text/event-stream")
+ self.send_header("Content-Length", "100")
+ self.end_headers()
+ self.wfile.write(b"data: partial\n\n")
+ self.wfile.flush()
+ self.connection.shutdown(socket.SHUT_RDWR)
+ self.connection.close()
+ return
+ if self.path == "/mcp":
+ response = b'{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-06-18"}}'
+ self._send(200, response, **{"Mcp-Session-Id": "session-primary"})
+ return
+ self._send(200, json.dumps({"origin": self.state.name}).encode())
+
+
+class MockOrigin:
+ def __init__(self, name: str) -> None:
+ self.state = OriginState(name)
+ self.server = ThreadingHTTPServer(("127.0.0.1", 0), MockOriginHandler)
+ self.server.state = self.state
+ self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
+ self.thread.start()
+
+ @property
+ def url(self) -> str:
+ return f"http://127.0.0.1:{self.server.server_port}"
+
+ def close(self) -> None:
+ self.server.shutdown()
+ self.server.server_close()
+ self.thread.join(timeout=2)
+
+
+def gateway_config(
+ port: int,
+ primary: str,
+ secondary: str,
+ protocol: str = "llm",
+ *,
+ client_timeout_seconds: float = 1,
+ max_concurrent_requests: int = 64,
+):
+ if protocol == "llm":
+ health = {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "inventory_path": "/v1/models",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ "interval_seconds": 60,
+ }
+ else:
+ health = {
+ "kind": "mcp",
+ "mcp_health_path": "/_failover/status",
+ "interval_seconds": 60,
+ }
+ breaker = {
+ "failure_threshold": 1,
+ "recovery_successes": 1,
+ "base_backoff_seconds": 1,
+ "max_backoff_seconds": 5,
+ "failback_hold_seconds": 0,
+ "max_stale_seconds": 300,
+ }
+ return parse_config(
+ {
+ "version": 1,
+ "listeners": [
+ {
+ "name": protocol,
+ "host": "127.0.0.1",
+ "port": port,
+ "protocol": protocol,
+ "max_body_bytes": 1024,
+ "max_concurrent_requests": max_concurrent_requests,
+ "max_idempotent_attempts": 2,
+ "client_timeout_seconds": client_timeout_seconds,
+ "connect_timeout_seconds": 1,
+ "idle_timeout_seconds": 2,
+ "session_ttl_seconds": 60,
+ "routes": [
+ {
+ "name": "primary",
+ "url": primary,
+ "provisioned": True,
+ "health": health,
+ "breaker": breaker,
+ },
+ {
+ "name": "secondary",
+ "url": secondary,
+ "provisioned": True,
+ "health": health,
+ "breaker": breaker,
+ },
+ ],
+ }
+ ],
+ }
+ )
+
+
+class GatewayIntegrationTests(unittest.TestCase):
+ def setUp(self) -> None:
+ self.primary = MockOrigin("primary")
+ self.secondary = MockOrigin("secondary")
+ self.port = free_port()
+ self.env = {
+ "FAILOVER_HEALTH_TOKEN": "test-health-credential",
+ "FAILOVER_HEALTH_MODEL": "configured-model",
+ }
+ self.service = GatewayService(
+ gateway_config(self.port, self.primary.url, self.secondary.url),
+ env=self.env,
+ )
+ self.service.start(run_health=False)
+ self.runtime = self.service.listeners["llm"]
+ now = self.service.clock()
+ for route in self.runtime.routes:
+ route.breaker.observe_success(now, "test_ready")
+
+ def tearDown(self) -> None:
+ self.service.shutdown()
+ self.primary.close()
+ self.secondary.close()
+
+ def request(
+ self,
+ method: str,
+ path: str,
+ body: bytes | None = None,
+ headers: dict[str, str] | None = None,
+ ) -> tuple[int, dict[str, str], bytes]:
+ connection = http.client.HTTPConnection("127.0.0.1", self.port, timeout=3)
+ connection.request(method, path, body=body, headers=headers or {})
+ response = connection.getresponse()
+ response_headers = {key.lower(): value for key, value in response.getheaders()}
+ data = response.read()
+ connection.close()
+ return response.status, response_headers, data
+
+ def test_new_post_uses_healthy_secondary_when_primary_is_open(self) -> None:
+ now = self.service.clock()
+ self.runtime.routes[0].breaker.observe_failure(now, "simulated_down")
+
+ status, headers, body = self.request("POST", "/v1/messages", b"{}")
+
+ self.assertEqual(status, 200)
+ self.assertEqual(json.loads(body)["origin"], "secondary")
+ self.assertEqual(headers["x-local-failover-route"], "secondary")
+ self.assertEqual(len(self.primary.state.requests), 0)
+ self.assertEqual(len(self.secondary.state.requests), 1)
+
+ def test_idempotent_get_retries_503_on_next_healthy_route(self) -> None:
+ self.primary.state.resource_status = 503
+
+ status, _, body = self.request("GET", "/resource")
+
+ self.assertEqual(status, 200)
+ self.assertEqual(json.loads(body)["origin"], "secondary")
+ self.assertEqual(len(self.primary.state.requests), 1)
+ self.assertEqual(len(self.secondary.state.requests), 1)
+
+ def test_post_is_never_replayed_after_unknown_delivery(self) -> None:
+ status, _, _ = self.request("POST", "/interrupt-before-headers", b"{}")
+
+ self.assertEqual(status, 502)
+ self.assertEqual(len(self.primary.state.requests), 1)
+ self.assertEqual(len(self.secondary.state.requests), 0)
+
+ def test_interrupted_stream_is_not_spliced_to_secondary(self) -> None:
+ connection = http.client.HTTPConnection("127.0.0.1", self.port, timeout=3)
+ connection.request("POST", "/stream-interrupt", body=b"{}")
+ response = connection.getresponse()
+ with self.assertRaises(http.client.IncompleteRead):
+ response.read()
+ connection.close()
+
+ self.assertEqual(len(self.primary.state.requests), 1)
+ self.assertEqual(len(self.secondary.state.requests), 0)
+
+ def test_hop_headers_are_stripped_and_end_to_end_auth_is_preserved(self) -> None:
+ status, _, _ = self.request(
+ "POST",
+ "/v1/messages",
+ b"{}",
+ {
+ "Authorization": "Bearer client-credential",
+ "Connection": "keep-alive, X-Remove-Me",
+ "X-Remove-Me": "not-forwarded",
+ },
+ )
+
+ received = self.primary.state.requests[0]["headers"]
+ self.assertEqual(status, 200)
+ self.assertEqual(received["authorization"], "Bearer client-credential")
+ self.assertNotIn("x-remove-me", received)
+ self.assertEqual(received["connection"], "close")
+
+ def test_body_limit_is_enforced_before_any_upstream_request(self) -> None:
+ status, _, _ = self.request("POST", "/v1/messages", b"x" * 1025)
+
+ self.assertEqual(status, 413)
+ self.assertEqual(self.primary.state.requests, [])
+ self.assertEqual(self.secondary.state.requests, [])
+
+ def test_stalled_request_body_times_out_without_reaching_upstream(self) -> None:
+ self.service.shutdown()
+ self.port = free_port()
+ self.service = GatewayService(
+ gateway_config(
+ self.port,
+ self.primary.url,
+ self.secondary.url,
+ client_timeout_seconds=0.2,
+ ),
+ env=self.env,
+ )
+ self.service.start(run_health=False)
+ self.runtime = self.service.listeners["llm"]
+ now = self.service.clock()
+ for route in self.runtime.routes:
+ route.breaker.observe_success(now, "test_ready")
+ client = socket.create_connection(("127.0.0.1", self.port), timeout=2)
+ client.sendall(
+ b"POST /v1/messages HTTP/1.1\r\n"
+ b"Host: localhost\r\n"
+ b"Content-Length: 10\r\n"
+ b"Connection: close\r\n\r\n"
+ b"x"
+ )
+
+ response = b""
+ while True:
+ chunk = client.recv(4096)
+ if not chunk:
+ break
+ response += chunk
+ client.close()
+
+ self.assertIn(b" 408 ", response)
+ self.assertIn(b"request_body_timeout", response)
+ self.assertEqual(self.primary.state.requests, [])
+ self.assertEqual(self.secondary.state.requests, [])
+
+ def test_concurrency_limit_rejects_excess_client_without_new_thread(self) -> None:
+ self.service.shutdown()
+ self.port = free_port()
+ self.service = GatewayService(
+ gateway_config(
+ self.port,
+ self.primary.url,
+ self.secondary.url,
+ max_concurrent_requests=1,
+ ),
+ env=self.env,
+ )
+ self.service.start(run_health=False)
+ self.runtime = self.service.listeners["llm"]
+ now = self.service.clock()
+ for route in self.runtime.routes:
+ route.breaker.observe_success(now, "test_ready")
+ first_result: list[tuple[int, dict[str, str], bytes]] = []
+ first = threading.Thread(
+ target=lambda: first_result.append(self.request("GET", "/block")), daemon=True
+ )
+ first.start()
+ self.assertTrue(self.primary.state.block_started.wait(timeout=1))
+
+ status, _, body = self.request("GET", "/resource")
+ self.primary.state.block_release.set()
+ first.join(timeout=2)
+
+ self.assertEqual(status, 503)
+ self.assertIn(b"gateway_overloaded", body)
+ self.assertEqual(first_result[0][0], 200)
+ self.assertEqual(
+ [request["path"] for request in self.primary.state.requests], ["/block"]
+ )
+
+ def test_status_and_metrics_do_not_expose_urls_credentials_or_models(self) -> None:
+ status, _, body = self.request("GET", "/_failover/status")
+ metrics_status, _, metrics = self.request("GET", "/_failover/metrics")
+ combined = body + metrics
+
+ self.assertEqual(status, 200)
+ self.assertEqual(metrics_status, 200)
+ self.assertIn(b'"primary"', body)
+ self.assertNotIn(self.primary.url.encode(), combined)
+ self.assertNotIn(b"test-health-credential", combined)
+ self.assertNotIn(b"configured-model", combined)
+
+ def test_real_semantic_probe_checks_inventory_and_stream_cadence(self) -> None:
+ route = self.runtime.routes[0].config
+
+ result = SemanticHealth(HTTPProbeTransport()).probe(route, self.env)
+
+ self.assertEqual(result.kind, "success")
+ self.assertEqual(result.reason, "semantic_ok")
+
+ def test_state_snapshot_is_private_and_redacted(self) -> None:
+ self.service.shutdown()
+ with tempfile.TemporaryDirectory() as directory:
+ state_file = Path(directory) / "status.json"
+ config = replace(
+ gateway_config(free_port(), self.primary.url, self.secondary.url),
+ state_file=state_file,
+ )
+ self.service = GatewayService(config, env=self.env)
+ self.service.start(run_health=False)
+ now = self.service.clock()
+ for route in self.service.listeners["llm"].routes:
+ route.breaker.observe_success(now, "test_ready")
+ self.service.persist_state()
+
+ body = state_file.read_bytes()
+ mode = stat.S_IMODE(state_file.stat().st_mode)
+
+ self.assertEqual(mode, 0o600)
+ self.assertIn(b'"state_persistence":"ok"', body)
+ self.assertNotIn(self.primary.url.encode(), body)
+ self.assertNotIn(b"test-health-credential", body)
+ self.assertNotIn(b"configured-model", body)
+
+
+class MCPSessionIntegrationTests(unittest.TestCase):
+ def setUp(self) -> None:
+ self.primary = MockOrigin("primary")
+ self.secondary = MockOrigin("secondary")
+ self.port = free_port()
+ self.service = GatewayService(
+ gateway_config(self.port, self.primary.url, self.secondary.url, "mcp"), env={}
+ )
+ self.service.start(run_health=False)
+ self.runtime = self.service.listeners["mcp"]
+ now = self.service.clock()
+ for route in self.runtime.routes:
+ route.breaker.observe_success(now, "test_ready")
+
+ def tearDown(self) -> None:
+ self.service.shutdown()
+ self.primary.close()
+ self.secondary.close()
+
+ def request(self, session_id: str | None = None):
+ headers = {"Content-Type": "application/json"}
+ if session_id:
+ headers["Mcp-Session-Id"] = session_id
+ connection = http.client.HTTPConnection("127.0.0.1", self.port, timeout=3)
+ connection.request(
+ "POST",
+ "/mcp",
+ body=b'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}',
+ headers=headers,
+ )
+ response = connection.getresponse()
+ response_headers = {key.lower(): value for key, value in response.getheaders()}
+ body = response.read()
+ connection.close()
+ return response.status, response_headers, body
+
+ def test_session_is_pinned_and_requires_reconnect_when_route_fails(self) -> None:
+ status, headers, _ = self.request()
+ session_id = headers["mcp-session-id"]
+ self.assertEqual(status, 200)
+ self.runtime.routes[0].gate_ready = False
+
+ failed_status, _, failed_body = self.request(session_id)
+
+ self.assertEqual(failed_status, 503)
+ self.assertIn(b"session_reconnect_required", failed_body)
+ self.assertEqual(len(self.primary.state.requests), 1)
+ self.assertEqual(len(self.secondary.state.requests), 0)
+
+ def test_session_remains_pinned_across_repeated_requests(self) -> None:
+ status, headers, _ = self.request()
+ session_id = headers["mcp-session-id"]
+
+ second_status, _, _ = self.request(session_id)
+
+ self.assertEqual(status, 200)
+ self.assertEqual(second_status, 200)
+ self.assertEqual(len(self.primary.state.requests), 2)
+ self.assertEqual(len(self.secondary.state.requests), 0)
+
+ def test_unknown_session_is_not_sent_to_arbitrary_route(self) -> None:
+ status, _, body = self.request("unknown-session")
+
+ self.assertEqual(status, 503)
+ self.assertIn(b"session_reconnect_required", body)
+ self.assertEqual(self.primary.state.requests, [])
+ self.assertEqual(self.secondary.state.requests, [])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/claude-ops/scripts/local-failover/tests/test_health_tunnel.py b/claude-ops/scripts/local-failover/tests/test_health_tunnel.py
new file mode 100644
index 00000000..e1d38664
--- /dev/null
+++ b/claude-ops/scripts/local-failover/tests/test_health_tunnel.py
@@ -0,0 +1,518 @@
+from __future__ import annotations
+
+import os
+import shutil
+import signal
+import socket
+import sys
+import time
+import unittest
+from dataclasses import replace
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from local_failover.config import parse_config
+from local_failover.health import (
+ ProbeHTTPResponse,
+ SemanticHealth,
+ classify_inventory_response,
+ classify_stream_response,
+)
+from local_failover.tunnel import (
+ TunnelSupervisor,
+ listener_bound,
+ listener_owned_by_process_group,
+)
+from test_config_circuit import valid_config
+
+
+class FakeTransport:
+ def __init__(self, responses: list[ProbeHTTPResponse]) -> None:
+ self.responses = list(responses)
+ self.calls: list[dict] = []
+
+ def request(self, **kwargs) -> ProbeHTTPResponse:
+ self.calls.append(kwargs)
+ return self.responses.pop(0)
+
+
+class HealthClassificationTests(unittest.TestCase):
+ def test_authenticated_inventory_requires_list_and_discards_payload(self) -> None:
+ success = classify_inventory_response(200, b'{"object":"list","data":[{"id":"a"}]}', 12)
+ malformed = classify_inventory_response(200, b'{"data":{}}', 12)
+
+ self.assertEqual(success.kind, "success")
+ self.assertEqual(success.reason, "inventory_ok")
+ self.assertFalse(hasattr(success, "payload"))
+ self.assertEqual(malformed.kind, "failure")
+ self.assertEqual(malformed.reason, "inventory_shape")
+
+ def test_rejected_health_authorization_is_unknown_not_provider_failure(self) -> None:
+ for status in (400, 401, 403):
+ with self.subTest(status=status):
+ result = classify_inventory_response(status, b"", 4)
+ self.assertEqual(result.kind, "unknown")
+ self.assertEqual(result.reason, "health_configuration_rejected")
+
+ def test_inventory_rate_limit_and_server_error_are_failures(self) -> None:
+ self.assertEqual(classify_inventory_response(429, b"", 4).kind, "failure")
+ self.assertEqual(classify_inventory_response(503, b"", 4).kind, "failure")
+
+ def test_stream_requires_fast_first_event_and_bounded_cadence(self) -> None:
+ response = ProbeHTTPResponse(
+ status=200,
+ headers={"content-type": "text/event-stream"},
+ body=b"",
+ events=((0.5, b"data: one\n"), (1.5, b"data: two\n")),
+ latency_ms=500,
+ )
+
+ result = classify_stream_response(
+ response,
+ first_event_timeout_seconds=2,
+ max_event_gap_seconds=2,
+ max_response_bytes=1024,
+ )
+
+ self.assertEqual(result.kind, "success")
+ self.assertEqual(result.reason, "stream_cadence_ok")
+
+ def test_stream_rejects_buffering_large_gaps_and_interruption(self) -> None:
+ cases = (
+ (
+ ProbeHTTPResponse(
+ 200,
+ {"content-type": "text/event-stream"},
+ b"",
+ ((3.0, b"data: late\n"), (3.1, b"data: two\n")),
+ 1,
+ ),
+ "stream_first_event_timeout",
+ ),
+ (
+ ProbeHTTPResponse(
+ 200,
+ {"content-type": "text/event-stream"},
+ b"",
+ ((0.1, b"data: one\n"), (4.0, b"data: two\n")),
+ 1,
+ ),
+ "stream_event_gap",
+ ),
+ (
+ ProbeHTTPResponse(
+ 200,
+ {"content-type": "text/event-stream"},
+ b"",
+ ((0.1, b"data: one\n"),),
+ 1,
+ error="stream_interrupted",
+ ),
+ "stream_interrupted",
+ ),
+ )
+ for response, reason in cases:
+ with self.subTest(reason=reason):
+ result = classify_stream_response(response, 2, 2, 1024)
+ self.assertEqual(result.kind, "failure")
+ self.assertEqual(result.reason, reason)
+
+
+class SemanticHealthTests(unittest.TestCase):
+ def _route(self):
+ return parse_config(valid_config()).listeners[0].routes[0]
+
+ def test_missing_health_auth_or_model_is_configuration_unknown_without_io(self) -> None:
+ transport = FakeTransport([])
+ health = SemanticHealth(transport=transport)
+
+ missing_auth = health.probe(self._route(), env={})
+ missing_model = health.probe(
+ self._route(), env={"FAILOVER_HEALTH_TOKEN": "not-a-real-token"}
+ )
+
+ self.assertEqual(missing_auth.kind, "unknown")
+ self.assertEqual(missing_auth.reason, "health_auth_missing")
+ self.assertEqual(missing_model.kind, "unknown")
+ self.assertEqual(missing_model.reason, "health_model_missing")
+ self.assertEqual(transport.calls, [])
+
+ def test_llm_probe_requires_inventory_then_stream_and_never_returns_payload(self) -> None:
+ transport = FakeTransport(
+ [
+ ProbeHTTPResponse(
+ 200,
+ {"content-type": "application/json"},
+ b'{"data":[{"id":"private-model-name"}]}',
+ (),
+ 3,
+ ),
+ ProbeHTTPResponse(
+ 200,
+ {"content-type": "text/event-stream"},
+ b"",
+ ((0.1, b"data: first\n"), (0.2, b"data: [DONE]\n")),
+ 5,
+ ),
+ ]
+ )
+ health = SemanticHealth(transport=transport)
+
+ result = health.probe(
+ self._route(),
+ env={
+ "FAILOVER_HEALTH_TOKEN": "not-a-real-token",
+ "FAILOVER_HEALTH_MODEL": "configured-model",
+ },
+ )
+
+ self.assertEqual(result.kind, "success")
+ self.assertEqual(result.reason, "semantic_ok")
+ self.assertNotIn("private-model-name", repr(result))
+ self.assertEqual([call["method"] for call in transport.calls], ["GET", "POST"])
+ self.assertEqual(
+ transport.calls[0]["headers"]["Authorization"], "Bearer not-a-real-token"
+ )
+
+ def test_mcp_probe_uses_only_read_only_health_gets(self) -> None:
+ raw = valid_config()
+ route_raw = raw["listeners"][0]["routes"][0]
+ route_raw["health"] = {
+ "kind": "mcp",
+ "mcp_health_path": "/servers/example/healthz",
+ }
+ route = parse_config(raw).listeners[0].routes[0]
+ transport = FakeTransport(
+ [
+ ProbeHTTPResponse(
+ 200,
+ {"content-type": "application/json"},
+ b'{"status":"ready"}',
+ (),
+ 2,
+ ),
+ ProbeHTTPResponse(
+ 200,
+ {"content-type": "application/json"},
+ b'{"status":"ready"}',
+ (),
+ 2,
+ ),
+ ]
+ )
+
+ health = SemanticHealth(transport=transport)
+ first = health.probe(route, env={})
+ second = health.probe(route, env={})
+
+ self.assertEqual(first.kind, "success")
+ self.assertEqual(first.reason, "mcp_health_ok")
+ self.assertEqual(second.kind, "success")
+ self.assertEqual([call["method"] for call in transport.calls], ["GET", "GET"])
+ self.assertTrue(all(call["body"] is None for call in transport.calls))
+
+ def test_mcp_probe_rejects_nonsemantic_success_body(self) -> None:
+ raw = valid_config()
+ route_raw = raw["listeners"][0]["routes"][0]
+ route_raw["health"] = {"kind": "mcp", "mcp_health_path": "/healthz"}
+ route = parse_config(raw).listeners[0].routes[0]
+ transport = FakeTransport(
+ [ProbeHTTPResponse(200, {"content-type": "text/plain"}, b"welcome", (), 2)]
+ )
+
+ result = SemanticHealth(transport=transport).probe(route, env={})
+
+ self.assertEqual(result.kind, "failure")
+ self.assertEqual(result.reason, "mcp_health_shape")
+
+
+class FakeProcess:
+ def __init__(self) -> None:
+ self.pid = 12345
+ self.returncode = None
+ self.terminated = False
+ self.killed = False
+
+ def poll(self):
+ return self.returncode
+
+ def terminate(self) -> None:
+ self.terminated = True
+ self.returncode = -15
+
+ def wait(self, timeout=None):
+ return self.returncode
+
+ def kill(self) -> None:
+ self.killed = True
+ self.returncode = -9
+
+
+class FakeProcessFactory:
+ def __init__(self) -> None:
+ self.calls: list[tuple[tuple[str, ...], dict]] = []
+ self.processes: list[FakeProcess] = []
+ self.signals: list[tuple[int, int]] = []
+
+ def __call__(self, argv, **kwargs):
+ process = FakeProcess()
+ self.calls.append((tuple(argv), kwargs))
+ self.processes.append(process)
+ return process
+
+ def get_process_group(self, pid: int) -> int:
+ if any(process.pid == pid for process in self.processes):
+ return pid
+ raise ProcessLookupError
+
+ def signal_process_group(self, process_group: int, signum: int) -> None:
+ self.signals.append((process_group, signum))
+ process = next(
+ (process for process in self.processes if process.pid == process_group), None
+ )
+ if process is None or process.poll() is not None:
+ raise ProcessLookupError
+ if signum == 0:
+ return
+ if signum == signal.SIGTERM:
+ process.terminate()
+ elif signum == signal.SIGKILL:
+ process.kill()
+
+
+def enabled_tunnel_config():
+ raw = valid_config()
+ route = raw["listeners"][0]["routes"][2]
+ route["provisioned"] = True
+ route["tunnel"].update(
+ {
+ "enabled": True,
+ "argv": ["tunnel-command", "--target", "${TUNNEL_TARGET}"],
+ "startup_timeout_seconds": 10,
+ "restart_base_seconds": 5,
+ "restart_max_seconds": 20,
+ }
+ )
+ return parse_config(raw).listeners[0].routes[2].tunnel
+
+
+class TunnelSupervisorTests(unittest.TestCase):
+ def test_will_not_adopt_preexisting_listener(self) -> None:
+ factory = FakeProcessFactory()
+ supervisor = TunnelSupervisor(
+ enabled_tunnel_config(),
+ env={"TUNNEL_TARGET": "target-placeholder"},
+ process_factory=factory,
+ listener_checker=lambda *_: True,
+ )
+
+ status = supervisor.tick(0)
+
+ self.assertEqual(status.state, "conflict")
+ self.assertEqual(factory.calls, [])
+
+ def test_requires_owned_live_process_and_newly_bound_port(self) -> None:
+ factory = FakeProcessFactory()
+ bound = {"value": False}
+ supervisor = TunnelSupervisor(
+ enabled_tunnel_config(),
+ env={"TUNNEL_TARGET": "target-placeholder"},
+ process_factory=factory,
+ listener_checker=lambda *_: bound["value"],
+ ownership_checker=lambda *_: bound["value"],
+ )
+
+ self.assertEqual(supervisor.tick(0).state, "starting")
+ self.assertFalse(supervisor.ready)
+ bound["value"] = True
+ self.assertEqual(supervisor.tick(1).state, "bound")
+ self.assertTrue(supervisor.ready)
+ self.assertFalse(factory.calls[0][1]["shell"])
+ self.assertTrue(factory.calls[0][1]["start_new_session"])
+
+ def test_process_exit_removes_readiness_and_restarts_with_backoff(self) -> None:
+ factory = FakeProcessFactory()
+ bound = {"value": False}
+ supervisor = TunnelSupervisor(
+ enabled_tunnel_config(),
+ env={"TUNNEL_TARGET": "target-placeholder"},
+ process_factory=factory,
+ listener_checker=lambda *_: bound["value"],
+ ownership_checker=lambda *_: bound["value"],
+ )
+ supervisor.tick(0)
+ bound["value"] = True
+ self.assertTrue(supervisor.tick(1).ready)
+ bound["value"] = False
+ factory.processes[0].returncode = 1
+
+ self.assertEqual(supervisor.tick(2).state, "backoff")
+ self.assertFalse(supervisor.tick(6).ready)
+ self.assertEqual(len(factory.calls), 1)
+ self.assertEqual(supervisor.tick(7).state, "starting")
+ self.assertEqual(len(factory.calls), 2)
+
+ def test_bind_race_never_adopts_listener_owned_by_another_process(self) -> None:
+ factory = FakeProcessFactory()
+ bound = {"value": False}
+ supervisor = TunnelSupervisor(
+ enabled_tunnel_config(),
+ env={"TUNNEL_TARGET": "target-placeholder"},
+ process_factory=factory,
+ listener_checker=lambda *_: bound["value"],
+ ownership_checker=lambda *_: False,
+ process_group_getter=factory.get_process_group,
+ process_group_signaler=factory.signal_process_group,
+ )
+ self.assertEqual(supervisor.tick(0).state, "starting")
+ bound["value"] = True
+
+ status = supervisor.tick(1)
+
+ self.assertEqual(status.state, "backoff")
+ self.assertEqual(status.reason, "listener_ownership_mismatch")
+ self.assertFalse(status.ready)
+ self.assertTrue(factory.processes[0].terminated)
+
+ def test_listener_replacement_revokes_tunnel_readiness(self) -> None:
+ factory = FakeProcessFactory()
+ bound = {"value": False}
+ owned = {"value": True}
+ supervisor = TunnelSupervisor(
+ enabled_tunnel_config(),
+ env={"TUNNEL_TARGET": "target-placeholder"},
+ process_factory=factory,
+ listener_checker=lambda *_: bound["value"],
+ ownership_checker=lambda *_: owned["value"],
+ process_group_getter=factory.get_process_group,
+ process_group_signaler=factory.signal_process_group,
+ )
+ supervisor.tick(0)
+ bound["value"] = True
+ self.assertTrue(supervisor.tick(1).ready)
+ owned["value"] = False
+
+ status = supervisor.tick(2)
+
+ self.assertEqual(status.state, "backoff")
+ self.assertEqual(status.reason, "listener_ownership_mismatch")
+ self.assertFalse(status.ready)
+ self.assertTrue(factory.processes[0].terminated)
+
+ def test_pid_reuse_guard_never_signals_a_different_process_group(self) -> None:
+ factory = FakeProcessFactory()
+ bound = {"value": False}
+ supervisor = TunnelSupervisor(
+ enabled_tunnel_config(),
+ env={"TUNNEL_TARGET": "target-placeholder"},
+ process_factory=factory,
+ listener_checker=lambda *_: bound["value"],
+ ownership_checker=lambda *_: True,
+ process_group_getter=lambda _pid: 54321,
+ process_group_signaler=factory.signal_process_group,
+ )
+ supervisor.tick(0)
+ bound["value"] = True
+ self.assertTrue(supervisor.tick(1).ready)
+
+ supervisor.shutdown()
+
+ self.assertEqual(factory.signals, [])
+ self.assertFalse(factory.processes[0].terminated)
+
+ def test_missing_argv_environment_is_unknown_without_starting(self) -> None:
+ factory = FakeProcessFactory()
+ supervisor = TunnelSupervisor(
+ enabled_tunnel_config(),
+ env={},
+ process_factory=factory,
+ listener_checker=lambda *_: False,
+ )
+
+ self.assertEqual(supervisor.tick(0).state, "configuration_unknown")
+ self.assertEqual(factory.calls, [])
+
+ def test_startup_timeout_terminates_only_owned_process(self) -> None:
+ factory = FakeProcessFactory()
+ supervisor = TunnelSupervisor(
+ enabled_tunnel_config(),
+ env={"TUNNEL_TARGET": "target-placeholder"},
+ process_factory=factory,
+ listener_checker=lambda *_: False,
+ process_group_getter=factory.get_process_group,
+ process_group_signaler=factory.signal_process_group,
+ )
+ supervisor.tick(0)
+
+ status = supervisor.tick(11)
+
+ self.assertEqual(status.state, "backoff")
+ self.assertTrue(factory.processes[0].terminated)
+ self.assertFalse(status.ready)
+
+ @unittest.skipUnless(shutil.which("lsof"), "lsof is required for listener ownership")
+ def test_shutdown_terminates_process_group_when_descendant_owns_listener(self) -> None:
+ with socket.socket() as reservation:
+ reservation.bind(("127.0.0.1", 0))
+ port = reservation.getsockname()[1]
+ unrelated = socket.socket()
+ unrelated.bind(("127.0.0.1", 0))
+ unrelated.listen()
+ unrelated_port = unrelated.getsockname()[1]
+ child_code = (
+ "import socket,time;"
+ "listener=socket.socket();"
+ "listener.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1);"
+ f"listener.bind(('127.0.0.1',{port}));"
+ "listener.listen();"
+ "time.sleep(30)"
+ )
+ parent_code = (
+ "import subprocess,sys;"
+ f"subprocess.Popen([sys.executable,'-c',{child_code!r}])"
+ )
+ config = replace(
+ enabled_tunnel_config(),
+ argv=(sys.executable, "-c", parent_code),
+ listener_port=port,
+ )
+ supervisor = TunnelSupervisor(config, env={})
+ process_group: int | None = None
+ try:
+ self.assertEqual(supervisor.tick(time.monotonic()).state, "starting")
+ process_group = supervisor.process.pid
+ deadline = time.monotonic() + 3
+ while time.monotonic() < deadline and not (
+ listener_bound("127.0.0.1", port)
+ and supervisor.process.poll() is not None
+ and listener_owned_by_process_group("127.0.0.1", port, process_group)
+ ):
+ time.sleep(0.05)
+ self.assertTrue(listener_bound("127.0.0.1", port))
+ self.assertIsNotNone(supervisor.process.poll())
+ self.assertTrue(
+ listener_owned_by_process_group("127.0.0.1", port, process_group)
+ )
+
+ supervisor.shutdown()
+
+ deadline = time.monotonic() + 3
+ while time.monotonic() < deadline and listener_bound("127.0.0.1", port):
+ time.sleep(0.05)
+ self.assertFalse(listener_bound("127.0.0.1", port))
+ self.assertTrue(listener_bound("127.0.0.1", unrelated_port))
+ finally:
+ supervisor.shutdown()
+ unrelated.close()
+ if process_group is not None:
+ try:
+ os.killpg(process_group, signal.SIGKILL)
+ except ProcessLookupError:
+ pass
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/claude-ops/scripts/local-failover/tests/test_main.py b/claude-ops/scripts/local-failover/tests/test_main.py
new file mode 100644
index 00000000..8cb3a7db
--- /dev/null
+++ b/claude-ops/scripts/local-failover/tests/test_main.py
@@ -0,0 +1,109 @@
+from __future__ import annotations
+
+import contextlib
+import io
+import json
+import socket
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from local_failover.main import main
+
+
+def free_port() -> int:
+ with socket.socket() as sock:
+ sock.bind(("127.0.0.1", 0))
+ return sock.getsockname()[1]
+
+
+def config_document(port: int) -> dict:
+ return {
+ "version": 1,
+ "listeners": [
+ {
+ "name": "llm",
+ "host": "127.0.0.1",
+ "port": port,
+ "protocol": "llm",
+ "routes": [
+ {
+ "name": "primary",
+ "url": "https://primary.invalid",
+ "provisioned": True,
+ "health": {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ },
+ },
+ {
+ "name": "secondary",
+ "url": "http://127.0.0.1:18001",
+ "provisioned": True,
+ "health": {
+ "kind": "llm",
+ "auth_env": "FAILOVER_HEALTH_TOKEN",
+ "stream_path": "/v1/chat/completions",
+ "stream_model_env": "FAILOVER_HEALTH_MODEL",
+ },
+ },
+ ],
+ }
+ ],
+ }
+
+
+class MainTests(unittest.TestCase):
+ def write_config(self, payload: dict) -> Path:
+ directory = tempfile.TemporaryDirectory()
+ self.addCleanup(directory.cleanup)
+ path = Path(directory.name) / "config.json"
+ path.write_text(json.dumps(payload), encoding="utf-8")
+ return path
+
+ def test_check_prints_order_and_bind_without_upstream_urls(self) -> None:
+ path = self.write_config(config_document(18000))
+ output = io.StringIO()
+
+ with contextlib.redirect_stdout(output):
+ result = main(["check", "--config", str(path)])
+
+ rendered = output.getvalue()
+ self.assertEqual(result, 0)
+ self.assertIn('"route_order":["primary","secondary"]', rendered)
+ self.assertIn('"bind":"127.0.0.1:18000"', rendered)
+ self.assertNotIn("primary.invalid", rendered)
+
+ def test_status_reports_unavailable_without_starting_or_mutating(self) -> None:
+ path = self.write_config(config_document(free_port()))
+ output = io.StringIO()
+
+ with contextlib.redirect_stdout(output):
+ result = main(["status", "--config", str(path)])
+
+ self.assertEqual(result, 0)
+ self.assertEqual(
+ json.loads(output.getvalue()),
+ {"listeners": [{"name": "llm", "state": "unavailable"}]},
+ )
+
+ def test_invalid_config_returns_bounded_error(self) -> None:
+ path = self.write_config({"version": 1, "listeners": []})
+ error = io.StringIO()
+
+ with contextlib.redirect_stderr(error):
+ result = main(["check", "--config", str(path)])
+
+ self.assertEqual(result, 2)
+ self.assertIn("configuration error", error.getvalue())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/claude-ops/scripts/local-failover/tests/test_vpc_paths.py b/claude-ops/scripts/local-failover/tests/test_vpc_paths.py
new file mode 100644
index 00000000..9b5a9a80
--- /dev/null
+++ b/claude-ops/scripts/local-failover/tests/test_vpc_paths.py
@@ -0,0 +1,180 @@
+from __future__ import annotations
+
+import json
+import sys
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+import vpc_paths
+
+
+class FakeRunner:
+ def __init__(self) -> None:
+ self.calls: list[tuple[str, ...]] = []
+
+ def __call__(self, argv, **_kwargs):
+ command = tuple(argv)
+ self.calls.append(command)
+ joined = " ".join(command)
+ if "describe-client-vpn-endpoints" in command:
+ payload = {
+ "ClientVpnEndpoints": [
+ {
+ "ClientVpnEndpointId": "cvpn-endpoint-private-identifier",
+ "Status": {"Code": "available"},
+ }
+ ]
+ }
+ elif "describe-client-vpn-target-networks" in command:
+ payload = {
+ "ClientVpnTargetNetworks": [
+ {"AssociationId": "cvpn-assoc-private", "Status": {"Code": "associated"}}
+ ]
+ }
+ elif "describe-instance-information" in command:
+ payload = {
+ "InstanceInformationList": [
+ {
+ "InstanceId": "i-privateidentifier",
+ "PingStatus": "Online",
+ "AgentVersion": "3.2.0.0",
+ }
+ ]
+ }
+ elif command[:2] == ("tailscale", "status"):
+ payload = {
+ "BackendState": "Running",
+ "TailscaleIPs": ["100.64.0.10"],
+ "Self": {"Online": True, "DNSName": "private-name.example.ts.net."},
+ }
+ elif command[:3] == ("/sbin/route", "-n", "get"):
+ return SimpleNamespace(returncode=0, stdout="interface: utun9\n", stderr="")
+ elif command[:2] in {
+ ("aws", "--version"),
+ ("session-manager-plugin", "--version"),
+ }:
+ return SimpleNamespace(returncode=0, stdout="installed\n", stderr="")
+ else:
+ self.fail_unexpected(joined)
+ return SimpleNamespace(returncode=0, stdout=json.dumps(payload), stderr="")
+
+ def fail_unexpected(self, command: str) -> None:
+ raise AssertionError(f"unexpected command: {command}")
+
+
+class DetectionTests(unittest.TestCase):
+ def test_detection_is_read_only_and_redacts_resource_identifiers(self) -> None:
+ runner = FakeRunner()
+
+ result = vpc_paths.detect(
+ region="us-east-1",
+ profile=None,
+ ssm_target=None,
+ route_destination="10.0.0.10",
+ runner=runner,
+ )
+
+ rendered = json.dumps(result, sort_keys=True)
+ self.assertEqual(result["client_vpn"]["available_endpoints"], 1)
+ self.assertEqual(result["client_vpn"]["associated_target_networks"], 1)
+ self.assertEqual(result["ssm"]["online_managed_nodes"], 1)
+ self.assertTrue(result["tailscale"]["online"])
+ self.assertEqual(result["local_route"]["interface_class"], "utun")
+ for private_value in (
+ "cvpn-endpoint-private-identifier",
+ "cvpn-assoc-private",
+ "i-privateidentifier",
+ "private-name.example.ts.net",
+ "100.64.0.10",
+ "utun9",
+ ):
+ self.assertNotIn(private_value, rendered)
+ commands = "\n".join(" ".join(command).lower() for command in runner.calls)
+ for mutation in (
+ " create-",
+ " delete-",
+ " modify-",
+ " authorize-",
+ " start-session",
+ "advertise-routes",
+ ):
+ self.assertNotIn(mutation, commands)
+
+ def test_missing_tools_are_reported_without_error_text(self) -> None:
+ def missing(_argv, **_kwargs):
+ raise FileNotFoundError("private installation path")
+
+ result = vpc_paths.detect(
+ region="us-east-1",
+ profile=None,
+ ssm_target=None,
+ route_destination=None,
+ runner=missing,
+ )
+
+ rendered = json.dumps(result)
+ self.assertIn('"state": "unavailable"', rendered)
+ self.assertNotIn("private installation path", rendered)
+
+
+class PlanTests(unittest.TestCase):
+ def test_client_vpn_template_covers_auth_network_and_access_resources(self) -> None:
+ text = vpc_paths.TEMPLATE.read_text(encoding="utf-8")
+
+ for expected in (
+ "AWS::EC2::ClientVpnEndpoint",
+ "AWS::EC2::ClientVpnTargetNetworkAssociation",
+ "AWS::EC2::ClientVpnAuthorizationRule",
+ "AWS::EC2::ClientVpnRoute",
+ "certificate-authentication",
+ "directory-service-authentication",
+ "federated-authentication",
+ "ExistingSecurityGroupId",
+ "DnsServers",
+ ):
+ self.assertIn(expected, text)
+
+ def test_client_vpn_plan_is_change_set_only_and_marks_paid_gate(self) -> None:
+ text = vpc_paths.render_client_vpn_plan(
+ region="us-east-1", profile="example", stack_name="local-access-review"
+ )
+
+ self.assertIn("validate-template", text)
+ self.assertIn("create-change-set", text)
+ self.assertIn("describe-change-set", text)
+ self.assertIn("PAID RESOURCE", text)
+ self.assertIn("DO NOT execute", text)
+ self.assertNotIn("execute-change-set --", text)
+ self.assertNotIn("/Users/", text)
+
+ def test_ssm_plan_is_narrow_remote_host_forwarding(self) -> None:
+ text = vpc_paths.render_ssm_plan(
+ region="us-east-1",
+ profile=None,
+ target_env="SSM_MANAGED_NODE_ID",
+ remote_host_env="SSM_REMOTE_HOST",
+ remote_port=8317,
+ local_port=18317,
+ )
+
+ self.assertIn("AWS-StartPortForwardingSessionToRemoteHost", text)
+ self.assertIn('"localPortNumber":["18317"]', text)
+ self.assertIn("supervisor", text.lower())
+ self.assertIn("semantic health", text.lower())
+
+ def test_tailscale_plan_requires_forwarding_and_tailnet_approval(self) -> None:
+ text = vpc_paths.render_tailscale_plan(vpc_cidr="10.0.0.0/16")
+
+ self.assertIn("net.ipv4.ip_forward", text)
+ self.assertIn("--advertise-routes", text)
+ self.assertIn("tailnet", text.lower())
+ self.assertIn("REQUIRES EXPLICIT AUTHORIZATION", text)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/claude-ops/scripts/local-failover/vpc_paths.py b/claude-ops/scripts/local-failover/vpc_paths.py
new file mode 100644
index 00000000..c0f668e4
--- /dev/null
+++ b/claude-ops/scripts/local-failover/vpc_paths.py
@@ -0,0 +1,365 @@
+#!/usr/bin/env python3
+"""Read-only VPC-path detection and non-executing provisioning plans."""
+
+from __future__ import annotations
+
+import argparse
+import ipaddress
+import json
+import re
+import shlex
+import subprocess
+from collections.abc import Callable, Sequence
+from pathlib import Path
+from typing import Any
+
+
+Runner = Callable[..., Any]
+TEMPLATE = Path(__file__).resolve().parent / "plans" / "aws-client-vpn.yaml"
+TEMPLATE_ARGUMENT = "file://scripts/local-failover/plans/aws-client-vpn.yaml"
+
+
+def _run(runner: Runner, argv: Sequence[str]) -> Any | None:
+ try:
+ result = runner(
+ list(argv),
+ stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True,
+ timeout=15,
+ check=False,
+ )
+ except (FileNotFoundError, OSError, subprocess.TimeoutExpired):
+ return None
+ return result if result.returncode == 0 else None
+
+
+def _json_output(result: Any | None) -> dict[str, Any] | None:
+ if result is None:
+ return None
+ try:
+ value = json.loads(result.stdout)
+ except (TypeError, json.JSONDecodeError):
+ return None
+ return value if isinstance(value, dict) else None
+
+
+def _aws_prefix(region: str, profile: str | None) -> list[str]:
+ argv = ["aws"]
+ if profile:
+ argv.extend(["--profile", profile])
+ argv.extend(["--region", region])
+ return argv
+
+
+def _client_vpn_detection(
+ region: str, profile: str | None, runner: Runner
+) -> dict[str, object]:
+ prefix = _aws_prefix(region, profile)
+ payload = _json_output(
+ _run(
+ runner,
+ [*prefix, "ec2", "describe-client-vpn-endpoints", "--output", "json"],
+ )
+ )
+ if payload is None:
+ return {"state": "unavailable"}
+ endpoints = payload.get("ClientVpnEndpoints")
+ if not isinstance(endpoints, list):
+ return {"state": "unknown"}
+ available = 0
+ associations = 0
+ association_state = "checked"
+ for endpoint in endpoints:
+ if not isinstance(endpoint, dict):
+ continue
+ status = endpoint.get("Status")
+ if isinstance(status, dict) and status.get("Code") == "available":
+ available += 1
+ endpoint_id = endpoint.get("ClientVpnEndpointId")
+ if not isinstance(endpoint_id, str):
+ association_state = "partial"
+ continue
+ target_payload = _json_output(
+ _run(
+ runner,
+ [
+ *prefix,
+ "ec2",
+ "describe-client-vpn-target-networks",
+ "--client-vpn-endpoint-id",
+ endpoint_id,
+ "--output",
+ "json",
+ ],
+ )
+ )
+ if target_payload is None:
+ association_state = "partial"
+ continue
+ targets = target_payload.get("ClientVpnTargetNetworks")
+ if isinstance(targets, list):
+ associations += sum(
+ 1
+ for target in targets
+ if isinstance(target, dict)
+ and isinstance(target.get("Status"), dict)
+ and target["Status"].get("Code") == "associated"
+ )
+ return {
+ "state": "detected",
+ "endpoints": len(endpoints),
+ "available_endpoints": available,
+ "associated_target_networks": associations,
+ "association_check": association_state,
+ }
+
+
+def _ssm_detection(
+ region: str, profile: str | None, target: str | None, runner: Runner
+) -> dict[str, object]:
+ prefix = _aws_prefix(region, profile)
+ argv = [*prefix, "ssm", "describe-instance-information"]
+ if target:
+ argv.extend(["--filters", f"Key=InstanceIds,Values={target}"])
+ argv.extend(["--output", "json"])
+ payload = _json_output(_run(runner, argv))
+ if payload is None:
+ return {"state": "unavailable"}
+ instances = payload.get("InstanceInformationList")
+ if not isinstance(instances, list):
+ return {"state": "unknown"}
+ online = sum(
+ 1
+ for instance in instances
+ if isinstance(instance, dict) and instance.get("PingStatus") == "Online"
+ )
+ return {
+ "state": "detected",
+ "matching_managed_nodes": len(instances),
+ "online_managed_nodes": online,
+ "target_filter_applied": bool(target),
+ }
+
+
+def _tailscale_detection(runner: Runner) -> dict[str, object]:
+ payload = _json_output(_run(runner, ["tailscale", "status", "--json"]))
+ if payload is None:
+ return {"state": "unavailable"}
+ self_status = payload.get("Self")
+ online = (
+ payload.get("BackendState") == "Running"
+ and isinstance(self_status, dict)
+ and self_status.get("Online") is True
+ )
+ addresses = payload.get("TailscaleIPs")
+ return {
+ "state": "detected",
+ "online": online,
+ "address_count": len(addresses) if isinstance(addresses, list) else 0,
+ }
+
+
+def _route_detection(destination: str | None, runner: Runner) -> dict[str, object]:
+ if not destination:
+ return {"state": "not_requested"}
+ result = _run(runner, ["/sbin/route", "-n", "get", destination])
+ if result is None:
+ return {"state": "unavailable"}
+ interface = ""
+ for line in result.stdout.splitlines():
+ key, separator, value = line.strip().partition(":")
+ if separator and key == "interface":
+ interface = value.strip()
+ break
+ match = re.match(r"[A-Za-z]+", interface)
+ return {
+ "state": "detected" if match else "unknown",
+ "interface_class": match.group(0).lower() if match else None,
+ }
+
+
+def detect(
+ *,
+ region: str,
+ profile: str | None,
+ ssm_target: str | None,
+ route_destination: str | None,
+ runner: Runner = subprocess.run,
+) -> dict[str, object]:
+ """Return identifier-free readiness facts using only read operations."""
+
+ aws_installed = _run(runner, ["aws", "--version"]) is not None
+ plugin_installed = _run(runner, ["session-manager-plugin", "--version"]) is not None
+ return {
+ "schema_version": 1,
+ "read_only": True,
+ "aws_cli": {"state": "installed" if aws_installed else "unavailable"},
+ "session_manager_plugin": {
+ "state": "installed" if plugin_installed else "unavailable"
+ },
+ "client_vpn": _client_vpn_detection(region, profile, runner),
+ "ssm": _ssm_detection(region, profile, ssm_target, runner),
+ "tailscale": _tailscale_detection(runner),
+ "local_route": _route_detection(route_destination, runner),
+ }
+
+
+def _aws_flags(region: str, profile: str | None) -> str:
+ profile_flag = f" --profile {shlex.quote(profile)}" if profile else ""
+ return f"--region {shlex.quote(region)}{profile_flag}"
+
+
+def render_client_vpn_plan(
+ *, region: str, profile: str | None, stack_name: str
+) -> str:
+ flags = _aws_flags(region, profile)
+ template = shlex.quote(TEMPLATE_ARGUMENT)
+ safe_stack_name = shlex.quote(stack_name)
+ return f"""AWS Client VPN review plan (renders commands; executes nothing)
+
+PAID RESOURCE / EXPLICIT AUTHORIZATION GATE
+- A deployed endpoint incurs an hourly charge for each target-network association.
+- Each active client connection incurs a connection-hour charge; public IPv4 and data charges may apply.
+- The template requires existing VPC, subnet, security group, ACM server certificate, DNS, and auth identifiers.
+- Authentication can be certificate, Directory Service, or SAML. Do not put private keys in parameters.
+
+Safe validation:
+aws cloudformation validate-template {flags} --template-body {template}
+
+Safe review-only change set (replace placeholders; this does not deploy):
+aws cloudformation create-change-set {flags} \\
+ --stack-name {safe_stack_name} \\
+ --change-set-name client-vpn-review \\
+ --change-set-type CREATE \\
+ --template-body {template} \\
+ --parameters file://client-vpn-parameters.review.json
+
+aws cloudformation describe-change-set {flags} \\
+ --stack-name {safe_stack_name} --change-set-name client-vpn-review
+
+DO NOT execute the change set without separate authorization for hourly cost, certificates,
+subnet association, authorization rules, routes, security groups, DNS, and client rollout.
+"""
+
+
+def render_ssm_plan(
+ *,
+ region: str,
+ profile: str | None,
+ target_env: str,
+ remote_host_env: str,
+ remote_port: int,
+ local_port: int,
+) -> str:
+ for name in (target_env, remote_host_env):
+ if not re.fullmatch(r"[A-Z_][A-Z0-9_]*", name):
+ raise ValueError("environment variable names must be uppercase shell identifiers")
+ if not 1 <= remote_port <= 65535 or not 1 <= local_port <= 65535:
+ raise ValueError("ports must be from 1 to 65535")
+ flags = _aws_flags(region, profile)
+ return f"""AWS Systems Manager remote-host port-forward plan (not executed)
+
+This is a narrow per-service path, not a broad laptop VPC route. It requires an online
+managed node, IAM permission, Session Manager plugin, SSM Agent 3.1.1374.0+, and managed-node
+DNS/network reachability to the remote host. Port-forward payloads are not Session Manager logged.
+
+Supervised tunnel argv/command:
+aws ssm start-session {flags} \\
+ --target "${{{target_env}}}" \\
+ --document-name AWS-StartPortForwardingSessionToRemoteHost \\
+ --parameters '{{"host":["'"${{{remote_host_env}}}"'"],"portNumber":["{remote_port}"],"localPortNumber":["{local_port}"]}}'
+
+The failover supervisor may admit this path only after it observed 127.0.0.1:{local_port}
+transition from free to bound by its owned child and semantic health succeeds through the tunnel.
+"""
+
+
+def render_tailscale_plan(*, vpc_cidr: str) -> str:
+ network = ipaddress.ip_network(vpc_cidr, strict=True)
+ safe_cidr = shlex.quote(str(network))
+ return f"""Tailscale subnet-router plan (not executed)
+
+REQUIRES EXPLICIT AUTHORIZATION from the host owner and tailnet administrator.
+On the approved Linux subnet-router host:
+ sudo sysctl -w net.ipv4.ip_forward=1
+ printf 'net.ipv4.ip_forward = 1\\n' | sudo tee /etc/sysctl.d/99-tailscale.conf
+ sudo tailscale set --advertise-routes={safe_cidr}
+
+Then approve {network} in the tailnet admin console (or an approved autoApprovers policy)
+and authorize the intended identities in tailnet access controls. On macOS, verify read-only:
+ tailscale status
+ /sbin/route -n get <VPC_DESTINATION>
+
+Rollback (also requires explicit authorization): withdraw the advertised route, remove tailnet
+approval, restore the prior forwarding setting, and disable this route in failover configuration.
+"""
+
+
+def _parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(description=__doc__)
+ subcommands = parser.add_subparsers(dest="command", required=True)
+ detection = subcommands.add_parser("detect", help="run identifier-redacted read checks")
+ detection.add_argument("--region", required=True)
+ detection.add_argument("--profile")
+ detection.add_argument("--ssm-target")
+ detection.add_argument("--route-destination")
+
+ plans = subcommands.add_parser("plan", help="print commands without running them")
+ plan_kind = plans.add_subparsers(dest="plan_kind", required=True)
+ client_vpn = plan_kind.add_parser("client-vpn")
+ client_vpn.add_argument("--region", required=True)
+ client_vpn.add_argument("--profile")
+ client_vpn.add_argument("--stack-name", default="local-access-review")
+ ssm = plan_kind.add_parser("ssm")
+ ssm.add_argument("--region", required=True)
+ ssm.add_argument("--profile")
+ ssm.add_argument("--target-env", default="SSM_MANAGED_NODE_ID")
+ ssm.add_argument("--remote-host-env", default="SSM_REMOTE_HOST")
+ ssm.add_argument("--remote-port", type=int, required=True)
+ ssm.add_argument("--local-port", type=int, required=True)
+ tailscale = plan_kind.add_parser("tailscale")
+ tailscale.add_argument("--vpc-cidr", required=True)
+ return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ args = _parser().parse_args(argv)
+ if args.command == "detect":
+ result = detect(
+ region=args.region,
+ profile=args.profile,
+ ssm_target=args.ssm_target,
+ route_destination=args.route_destination,
+ )
+ print(json.dumps(result, sort_keys=True, separators=(",", ":")))
+ return 0
+ if args.plan_kind == "client-vpn":
+ print(
+ render_client_vpn_plan(
+ region=args.region,
+ profile=args.profile,
+ stack_name=args.stack_name,
+ ),
+ end="",
+ )
+ elif args.plan_kind == "ssm":
+ print(
+ render_ssm_plan(
+ region=args.region,
+ profile=args.profile,
+ target_env=args.target_env,
+ remote_host_env=args.remote_host_env,
+ remote_port=args.remote_port,
+ local_port=args.local_port,
+ ),
+ end="",
+ )
+ else:
+ print(render_tailscale_plan(vpc_cidr=args.vpc_cidr), end="")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/docs/superpowers/plans/2026-08-14-local-failover-gateway.md b/docs/superpowers/plans/2026-08-14-local-failover-gateway.md
new file mode 100644
index 00000000..988f1b63
--- /dev/null
+++ b/docs/superpowers/plans/2026-08-14-local-failover-gateway.md
@@ -0,0 +1,288 @@
+# Local Failover Gateway Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build and deterministically verify loopback-only CLI and MCP failover listeners with health-gated network paths and no automatic side-effect request replay.
+
+**Architecture:** A dependency-free Python package separates validated configuration, circuit/selection state, semantic health probes, tunnel ownership, and byte-stream proxying. A read-only VPC helper reports existing Client VPN, SSM, and Tailscale readiness and prints provisioning plans without executing them.
+
+**Tech Stack:** Python 3.11+ standard library, `unittest`, local HTTP mock servers, macOS launchd template, AWS CLI and Tailscale CLI as optional read-only detector dependencies.
+
+## Global Constraints
+
+- Work only in `claude-ops/scripts/local-failover/` and failover-specific docs/tests.
+- Bind listeners to loopback; never modify live services, tmux panes, client configs, credentials, AWS resources, or tailnet state.
+- Use `/v1/models` only for authenticated semantic health and never persist its payload.
+- Treat health-check `401`/`403` or missing health authorization as configuration-unknown, not provider failure.
+- Never automatically replay `POST`, `PUT`, `PATCH`, or `DELETE`, including MCP JSON-RPC and model generation calls.
+- Admit the SSM route only when the daemon owns the tunnel process group, observed the configured local port transition from free to exclusively group-owned, and semantic health passes.
+
+---
+
+### Task 1: Validated configuration and circuit state
+
+**Files:**
+- Create: `claude-ops/scripts/local-failover/local_failover/config.py`
+- Create: `claude-ops/scripts/local-failover/local_failover/circuit.py`
+- Create: `claude-ops/scripts/local-failover/tests/test_config_circuit.py`
+
+**Interfaces:**
+- Produces: `load_config(path: Path) -> GatewayConfig`, `CircuitBreaker`, `RouteRuntime`, `RouteSelector.choose(session_route: str | None, now: float) -> RouteRuntime | None`.
+- Consumes: JSON configuration and an injected monotonic clock value.
+
+- [ ] **Step 1: Write failing validation and state-machine tests**
+
+```python
+def test_listener_rejects_non_loopback_bind(self):
+ raw = valid_config()
+ raw["listeners"][0]["host"] = "0.0.0.0"
+ with self.assertRaisesRegex(ConfigError, "loopback"):
+ parse_config(raw)
+
+def test_circuit_requires_stable_recovery_before_failback(self):
+ breaker = CircuitBreaker(policy(), now=0)
+ breaker.observe_failure(0, "connect")
+ breaker.observe_failure(1, "connect")
+ self.assertFalse(breaker.available(1))
+ breaker.observe_success(11)
+ breaker.observe_success(12)
+ self.assertFalse(breaker.available(12))
+ breaker.observe_success(13)
+ self.assertTrue(breaker.available(18))
+```
+
+- [ ] **Step 2: Run tests and verify RED**
+
+Run: `python3 -m unittest discover -s scripts/local-failover/tests -p 'test_*.py' -v`
+
+Expected: import failure because `local_failover.config` and `local_failover.circuit` do not exist.
+
+- [ ] **Step 3: Implement immutable dataclasses, URL/bind validation, and breaker transitions**
+
+Validate environment variable names, URL schemes/credentials/query fragments, overlay HTTP opt-in, body/time bounds, unique route names, and loopback listeners. Implement `unknown`, `closed`, `open`, and `half_open` states with threshold opening, capped exponential cooldown, recovery success count, and stable hold time.
+
+- [ ] **Step 4: Run focused tests and verify GREEN**
+
+Run: `python3 -m unittest scripts/local-failover/tests/test_config_circuit.py -v`
+
+Expected: all configuration and state-machine tests pass without sleeps.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add claude-ops/scripts/local-failover/local_failover/{config,circuit}.py \
+ claude-ops/scripts/local-failover/tests/test_config_circuit.py
+git commit -m "feat: add failover policy state machine"
+```
+
+### Task 2: Semantic health and supervised tunnel ownership
+
+**Files:**
+- Create: `claude-ops/scripts/local-failover/local_failover/health.py`
+- Create: `claude-ops/scripts/local-failover/local_failover/tunnel.py`
+- Create: `claude-ops/scripts/local-failover/tests/test_health_tunnel.py`
+
+**Interfaces:**
+- Consumes: `RouteConfig`, `RouteRuntime`, environment mapping, injected HTTP connector/process factory/listener checker.
+- Produces: `ProbeResult(kind, reason, latency_ms)`, `SemanticHealth.probe(route)`, and `TunnelSupervisor.tick(now) -> TunnelStatus`.
+
+- [ ] **Step 1: Write failing semantic and ownership tests**
+
+```python
+def test_missing_or_rejected_health_auth_is_unknown(self):
+ result = health.probe_models(route, env={})
+ self.assertEqual(result.kind, "unknown")
+ result = health.classify_models_response(401, b"")
+ self.assertEqual(result.kind, "unknown")
+
+def test_tunnel_will_not_adopt_preexisting_listener(self):
+ supervisor = TunnelSupervisor(config, listener_checker=lambda *_: True)
+ self.assertEqual(supervisor.tick(0).state, "conflict")
+ self.assertEqual(process_factory.calls, [])
+```
+
+- [ ] **Step 2: Run tests and verify RED**
+
+Run: `python3 -m unittest scripts/local-failover/tests/test_health_tunnel.py -v`
+
+Expected: import failure because the health and tunnel modules do not exist.
+
+- [ ] **Step 3: Implement bounded authenticated probes and no-shell tunnel supervision**
+
+Inventory success requires a JSON object with a list-valued `data` field but discards the list immediately. Streaming success requires SSE/NDJSON content, a first event before the configured deadline, multiple cadence events or a terminal marker, a byte cap, and no response-body logging. MCP health uses a bounded read-only status GET and never creates a protocol session. The tunnel starts only from an unbound port, uses `shell=False` and its own recorded process group, verifies the child remains alive while every listener on the port belongs to that group, terminates the continuously existing group with bounded TERM/KILL escalation, and applies capped restart backoff.
+
+- [ ] **Step 4: Run focused tests and verify GREEN**
+
+Run: `python3 -m unittest scripts/local-failover/tests/test_health_tunnel.py -v`
+
+Expected: all health classification, cadence, listener-conflict, process-exit, and restart-backoff tests pass.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add claude-ops/scripts/local-failover/local_failover/{health,tunnel}.py \
+ claude-ops/scripts/local-failover/tests/test_health_tunnel.py
+git commit -m "feat: supervise semantic route health"
+```
+
+### Task 3: Streaming gateway and MCP session affinity
+
+**Files:**
+- Create: `claude-ops/scripts/local-failover/local_failover/proxy.py`
+- Create: `claude-ops/scripts/local-failover/local_failover/main.py`
+- Create: `claude-ops/scripts/local-failover/local_failover/__init__.py`
+- Create: `claude-ops/scripts/local-failover/tests/test_gateway_integration.py`
+
+**Interfaces:**
+- Consumes: validated `GatewayConfig`, selector/runtime state, semantic health, tunnel supervisors.
+- Produces: `GatewayService.start()`, `GatewayService.shutdown()`, loopback HTTP listeners, `/_failover/status`, and `/_failover/metrics`.
+
+- [ ] **Step 1: Write failing local integration tests**
+
+```python
+def test_post_is_sent_once_when_selected_origin_interrupts(self):
+ response = self.request("POST", "/v1/messages", body=b"{}")
+ self.assertIn(response.status, (502, 503))
+ self.assertEqual(primary.request_count, 1)
+ self.assertEqual(secondary.request_count, 0)
+
+def test_mcp_session_does_not_migrate_between_routes(self):
+ session_id = self.initialize_mcp_on_primary()
+ primary.mark_unhealthy()
+ response = self.mcp_request(session_id)
+ self.assertEqual(response.status, 503)
+ self.assertEqual(secondary.request_count, 0)
+```
+
+- [ ] **Step 2: Run tests and verify RED**
+
+Run: `python3 -m unittest scripts/local-failover/tests/test_gateway_integration.py -v`
+
+Expected: import failure because the proxy and main modules do not exist.
+
+- [ ] **Step 3: Implement bounded forwarding and status**
+
+Strip hop-by-hop and proxy credentials, replace `Host`, preserve end-to-end authorization and MCP session headers, enforce body limits, client read deadlines, and per-listener concurrency caps, and stream with `Connection: close` when no upstream length exists. Retry only idempotent methods and only before client-visible bytes. Bind returned MCP session IDs to a route with idle expiry. Persist redacted informational status atomically with mode `0600`.
+
+- [ ] **Step 4: Run focused tests and verify GREEN**
+
+Run: `python3 -m unittest scripts/local-failover/tests/test_gateway_integration.py -v`
+
+Expected: tests pass for ordered failure, bounded GET retry, POST no-replay, interrupted stream no-splice, recovery, anti-flapping, MCP reconnect-required, and redacted status.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add claude-ops/scripts/local-failover/local_failover \
+ claude-ops/scripts/local-failover/tests/test_gateway_integration.py
+git commit -m "feat: add session-safe local failover proxy"
+```
+
+### Task 4: Read-only VPC detection, dry-run plans, launchd template, and operations guide
+
+**Files:**
+- Create: `claude-ops/scripts/local-failover/vpc_paths.py`
+- Create: `claude-ops/scripts/local-failover/config.example.json`
+- Create: `claude-ops/scripts/local-failover/launchd/com.example.local-failover.plist.template`
+- Create: `claude-ops/scripts/local-failover/plans/aws-client-vpn.yaml`
+- Create: `claude-ops/scripts/local-failover/README.md`
+- Create: `claude-ops/scripts/local-failover/tests/test_vpc_paths.py`
+
+**Interfaces:**
+- Produces: `vpc_paths.py detect` (read-only, identifier-redacted JSON) and `vpc_paths.py plan {client-vpn,ssm,tailscale}` (stdout only).
+- Consumes: explicit region/profile/target/subnet/CIDR placeholders; it never reads credential files or executes plan output.
+
+- [ ] **Step 1: Write failing detection and dry-run tests**
+
+```python
+def test_detect_invokes_only_allowlisted_read_operations(self):
+ detect(runner=fake_runner)
+ self.assertNotIn("create", " ".join(fake_runner.argv).lower())
+ self.assertNotIn("modify", " ".join(fake_runner.argv).lower())
+
+def test_plans_print_without_executing(self):
+ text = render_ssm_plan(plan_args())
+ self.assertIn("AWS-StartPortForwardingSessionToRemoteHost", text)
+ self.assertEqual(fake_runner.calls, [])
+```
+
+- [ ] **Step 2: Run tests and verify RED**
+
+Run: `python3 -m unittest scripts/local-failover/tests/test_vpc_paths.py -v`
+
+Expected: import failure because `vpc_paths.py` does not exist.
+
+- [ ] **Step 3: Implement redacted detection and exact non-executing plans**
+
+Use only `describe-client-vpn-endpoints`, `describe-client-vpn-target-networks`, `describe-instance-information`, local CLI version/status, and route inspection. Render Client VPN CloudFormation validation/change-set commands, SSM remote-host forwarding argv, Tailscale IP-forwarding/advertise/approval steps, cost implications, rollback, and explicit authorization gates.
+
+- [ ] **Step 4: Run focused tests and documentation checks**
+
+Run: `python3 -m unittest scripts/local-failover/tests/test_vpc_paths.py -v`
+
+Run: `python3 -m json.tool scripts/local-failover/config.example.json >/dev/null`
+
+Run: `plutil -lint scripts/local-failover/launchd/com.example.local-failover.plist.template`
+
+Expected: tests pass, JSON parses, and the plist template is structurally valid.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add claude-ops/scripts/local-failover
+git commit -m "docs: add failover deployment and VPC plans"
+```
+
+### Task 5: Full isolated verification and review artifact
+
+**Files:**
+- Modify: `claude-ops/tests/run-all.sh`
+- Create: `claude-ops/tests/test-local-failover.sh`
+- Create outside Git: review patch generated from the exact base commit.
+
+**Interfaces:**
+- Consumes: all failover modules and fixtures.
+- Produces: one repository test-suite entry and a patch with branch/base/head provenance.
+
+- [ ] **Step 1: Add a shell suite that invokes the deterministic Python tests and static validators**
+
+```bash
+python3 -m unittest discover -s "$ROOT/scripts/local-failover/tests" -p 'test_*.py' -v
+python3 -m json.tool "$ROOT/scripts/local-failover/config.example.json" >/dev/null
+```
+
+- [ ] **Step 2: Run focused and repository verification**
+
+Run: `bash tests/test-local-failover.sh`
+
+Run: `npm run lint`
+
+Run: `npm test`
+
+Run: `bash tests/run-all.sh`
+
+Expected: all failover tests, static checks, lint, secret scan, and repository suites pass without touching external or existing local ports.
+
+- [ ] **Step 3: Verify source safety and worktree scope**
+
+Run: `git diff --check origin/main...HEAD`
+
+Run: `bash tests/test-no-secrets.sh`
+
+Run: `git status --short`
+
+Expected: no whitespace errors, no secret/PII findings, and only intended failover files before the final commit.
+
+- [ ] **Step 4: Commit and create the review artifact**
+
+```bash
+git add claude-ops/tests/run-all.sh claude-ops/tests/test-local-failover.sh
+git commit -m "test: verify local failover scenarios"
+git format-patch --stdout origin/main..HEAD > ../local-failover-gateway.patch
+```
+
+- [ ] **Step 5: Record exact provenance**
+
+Run: `git rev-parse origin/main HEAD && git branch --show-current && shasum -a 256 ../local-failover-gateway.patch`
+
+Expected: exact base/head SHAs, branch name, and patch checksum are available for the completion report.
diff --git a/docs/superpowers/specs/2026-08-14-local-failover-gateway-design.md b/docs/superpowers/specs/2026-08-14-local-failover-gateway-design.md
new file mode 100644
index 00000000..2c37a907
--- /dev/null
+++ b/docs/superpowers/specs/2026-08-14-local-failover-gateway-design.md
@@ -0,0 +1,70 @@
+# Local Failover Gateway Design
+
+## Goal
+
+Provide stable loopback endpoints for OpenAI/Anthropic-compatible CLIs and HTTP MCP clients while selecting among pre-provisioned network paths in a fixed order. The gateway must preserve local MCP child services, avoid request duplication, supervise optional per-service tunnels, and expose enough state to diagnose failover without logging credentials or request bodies.
+
+## Scope and ownership
+
+The implementation lives entirely under `claude-ops/scripts/local-failover/` plus failover-specific documentation and tests. It does not edit or signal existing tmux panes, local MCP processes, client configurations, credentials, cloud resources, VPN routes, or tailnet policy. Runtime installation and client cutover are explicit later operator actions.
+
+The model inventory endpoint is used only as an authenticated semantic health check. The gateway never stores, rewrites, aliases, ranks, or refreshes model inventory.
+
+## Architecture
+
+One dependency-free Python daemon owns one or more loopback-only listeners. Each listener has an ordered route set and protocol policy:
+
+```text
+CLI -> stable loopback listener -> Cloud edge -> overlay address -> supervised tunnel
+MCP -> stable loopback listener -> local aggregate -> optional remote aggregate paths
+```
+
+Routes are eligible only when provisioned, their network gate is satisfied, and semantic health is closed. A supervised tunnel route additionally requires a process group started by this daemon, a newly bound local listener whose owning processes all belong to that group, and successful semantic checks through that listener. A pre-existing, raced, or replaced listener is never adopted as proof of tunnel health.
+
+The daemon contains five bounded units:
+
+1. **Configuration** validates loopback binds, safe upstream URLs, environment-only health credentials, timeouts, body limits, and route order.
+2. **Health engine** performs authenticated inventory and minimal stream checks for LLM routes and read-only bounded status GETs for MCP routes, classifying missing/invalid health authorization as configuration-unknown rather than provider failure. It never creates a health-check MCP session.
+3. **Circuit and selector** apply failure thresholds, exponential cooldown, recovery successes, failback hold time, and active-route stickiness.
+4. **Streaming reverse proxy** forwards bytes once, strips hop-by-hop headers, preserves authorization and MCP session headers, and never changes upstream after client-visible bytes.
+5. **Tunnel supervisor** starts an argv array without a shell, records and owns only its new process group, verifies a previously free local port became bound exclusively by that group, terminates the whole continuously existing group with bounded TERM/KILL escalation, and restarts with capped backoff.
+
+## Retry and stream safety
+
+Automatic retries are allowed only for `GET`, `HEAD`, and `OPTIONS`, are bounded by configuration, and stop as soon as response bytes become client-visible. `POST`, `PUT`, `PATCH`, and `DELETE` are never replayed. This includes OpenAI/Anthropic generation requests and MCP JSON-RPC calls, regardless of whether a particular method appears read-only.
+
+An interrupted stream is terminated and recorded against the selected route. The gateway does not splice a second upstream into an existing stream. The client can decide whether to issue a new request.
+
+## MCP session behavior
+
+When an upstream returns `Mcp-Session-Id`, the gateway binds that opaque value to the selected route in memory. Requests carrying the session ID stay on that route. If the route becomes unavailable, the gateway returns a reconnect-required `503` rather than sending the session to another upstream. Session bindings expire after a configured idle TTL and are never persisted.
+
+The existing local aggregate and its child services remain unchanged. A future MCP cutover points clients at the new listener while its primary route points at the existing aggregate.
+
+## Observability
+
+Loopback-only `/_failover/status` and `/_failover/metrics` endpoints expose route names, circuit states, health reason codes, transition timestamps, counters, active route, and supervised-tunnel state. They do not expose upstream URLs, headers, credentials, model lists, request bodies, or query strings. Optional state snapshots are atomically written with owner-only permissions and are informational only; stale health is not trusted after restart.
+
+## Optional VPC paths
+
+The implementation provides read-only detection and dry-run planning only:
+
+- **AWS Client VPN** is the managed full-VPC option. AWS documents it as an OpenVPN-based managed client VPN with certificate, Active Directory, or SAML authentication. It requires a server certificate, endpoint, target subnet association, authorization rules, routes, security groups, and DNS design. AWS bills each endpoint association and each active client connection hourly; additional public IPv4, transfer, logging, and optional integration charges can apply. SAML endpoints require the AWS-provided client, including on Apple Silicon. Sources: [AWS Client VPN overview](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/what-is.html), [authentication](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html), [target networks](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-target.html), [routes](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-routes.html), [authorization](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-rules.html), [macOS/OpenVPN clients](https://docs.aws.amazon.com/vpn/latest/clientvpn-user/connect.html), and [pricing](https://aws.amazon.com/vpn/pricing/#AWS_Client_VPN_pricing).
+- **Systems Manager port forwarding to a remote host** is the narrow per-service option. It requires the Session Manager plugin, a managed node, IAM authorization, outbound Systems Manager connectivity, and SSM Agent 3.1.1374.0 or later for `AWS-StartPortForwardingSessionToRemoteHost`. Port-forwarded payloads are not available in Session Manager session logs. Sources: [start a remote-host port-forwarding session](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html#sessions-remote-port-forwarding) and [Session Manager prerequisites](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-prerequisites.html).
+- **Tailscale subnet routing** is the simplest existing non-AWS-managed route when an approved subnet router already exists. It requires IP forwarding, route advertisement, route approval (or an approved `autoApprovers` policy), and access rules. Source: [Tailscale subnet routers](https://tailscale.com/kb/1019/subnets).
+
+Site-to-Site VPN connects a VPC to an on-premises network through customer-gateway equipment, and Direct Connect is a physical/partner Ethernet connection; neither is a laptop remote-access design. AWS Verified Access is appropriate for application-level access, not general laptop VPC routing.
+
+## Security model
+
+Trust boundaries are the local client request, route configuration, upstream responses, and optional tunnel child process. Controls are loopback-only binds, strict config validation, TLS verification, no shell execution, bounded client concurrency and request-body deadlines, bounded upstream timeouts, credential references by environment variable name, status redaction, and zero automatic replay of side-effect-capable requests.
+
+The gateway does not create broad network access. A route enters selection only after separate provisioning approval and local semantic validation.
+
+## Test strategy
+
+Deterministic unit tests use a fake clock for circuit transitions, hysteresis, backoff, anti-flapping, selection, MCP affinity, and tunnel ownership. A real local subprocess regression verifies that shutdown removes an exited leader's descendant listener while preserving an unrelated listener. Local mock origins verify authenticated inventory checks, stream cadence, read-only MCP health, fixed route order, bounded client concurrency, stalled-body deadlines, bounded idempotent retries, non-idempotent no-replay, recovery, stream interruption, status redaction, and reconnect-required MCP behavior. No test uses operator credentials, external provider requests, or existing local service ports.
+
+## Rollback
+
+Before a future cutover, copy each client config to a timestamped owner-only backup. Rollback restores those backups and unloads only the failover LaunchAgent. Existing origins, MCP children, and network paths remain untouched, so rollback does not require cloud or tailnet changes.