Secure, Fast, and Extensible Sandbox runtime for AI agents.
fix(egress): notify H2 clients before retired streams finish
leo pein5e130ee0148a
diff --git a/.github/workflows/egress-test.yml b/.github/workflows/egress-test.ymlindex bb71eca1d..a3e3f3251 100644--- a/.github/workflows/egress-test.yml+++ b/.github/workflows/egress-test.yml@@ -143,6 +143,30 @@ jobs:run: |python -m unittest discover -v -s tests -p "test_*.py"+ - name: Set up exact Go for held-old HTTP/2 fixture+ # Keep the repository's Go 1.25.9 tests above on their pinned toolchain.+ uses: actions/setup-go@v6+ with:+ go-version: '1.27.2'+ cache: false++ - name: Test and build held-old Go fixture+ env:+ GOTOOLCHAIN: local+ GOWORK: off+ GOPROXY: off+ GO111MODULE: off+ GOCACHE: ${{ runner.temp }}/go1272-cache+ GOMODCACHE: ${{ runner.temp }}/go1272-modcache+ run: |+ test "$(go version)" = "go version go1.27.2 linux/amd64"+ mkdir -p "$GOCACHE" "$GOMODCACHE"+ go test components/egress/tests/fixtures/h2_retry_client.go \+ components/egress/tests/fixtures/h2_retry_client_test.go+ go build -trimpath \+ -o "$RUNNER_TEMP/h2_retry_client-go1.27.2" \+ components/egress/tests/fixtures/h2_retry_client.go+- name: Set up Python for H2 runtime coverage# Unit discovery above covers 3.10; exercise real runtime on 3.11 too.uses: actions/setup-python@v6@@ -163,13 +187,34 @@ jobs:-s components/egress/tests -p 'test_credential_bound_h2*.py'- name: Run H2 loopback runtime regressions+ id: h2_loopback_runtime# A real loopback destination on port 443 is part of this fixture.# This does not cover transparent redirects, the production UID or quarantine.+ # With no --go-client, this selects only the manual-peer denominator,+ # including the real pre_admission_barrier_h2 runtime case; it does not+ # select optional Go diagnostics or imply client interoperability.run: |sudo "$RUNNER_TEMP/h2-runtime-venv/bin/python" \components/egress/tests/test_credential_bound_h2_runtime.py \--run-real-runtime --artifacts "$RUNNER_TEMP/h2-runtime-evidence"+ - name: Run Go held-old early-notice H1 and H2 cases+ # Only the two explicit early-notice cases provide standard-client+ # parser/interoperability evidence. Historical go_retry cases remain+ # non-gating diagnostics and are not selected here. Raw stderr is+ # discarded by the fixture; only filtered GOAWAY JSON enters artifacts.+ if: steps.h2_loopback_runtime.outcome == 'success'+ run: |+ sudo env -u GODEBUG \+ "$RUNNER_TEMP/h2-runtime-venv/bin/python" \+ components/egress/tests/test_credential_bound_h2_runtime.py \+ --run-real-runtime \+ --artifacts "$RUNNER_TEMP/h2-runtime-evidence/go-early-notice" \+ --case go_early_notice_h1_origin \+ --case go_early_notice_h2_origin \+ --go-client "$RUNNER_TEMP/h2_retry_client-go1.27.2" \+ --require-go-version go1.27.2+- name: Run static TLS passthrough runtime regressions# Actual destination 443 and authenticated Unix IPC are required here.# Explicit CONNECT proves neither transparent redirects nor nft enforcement.diff --git a/components/egress/mitmscripts/credential_bound_h2.py b/components/egress/mitmscripts/credential_bound_h2.pyindex f37e91975..62a50fa1c 100644--- a/components/egress/mitmscripts/credential_bound_h2.py+++ b/components/egress/mitmscripts/credential_bound_h2.py@@ -22,10 +22,12 @@ import h2.errorsimport h2.eventsimport h2.exceptionsimport h2.stream+from hyperframe.frame import GoAwayFramefrom mitmproxy import httpfrom mitmproxy.net.http import status_codesfrom mitmproxy.proxy import commands, eventsfrom mitmproxy.proxy.layers import http as layers+from mitmproxy.proxy.layers.http._http_h2 import BufferedH2Connectionfrom mitmproxy.proxy.layers.http._base import HttpEvent, ReceiveHttpfrom mitmproxy.proxy.layers.http._http1 import Http1Clientfrom mitmproxy.proxy.layers.http._http2 import Http2Client, Http2Server@@ -105,7 +107,10 @@ class BoundH2Server(Http2Server):self.failed = Falseself.close_started = Falseself.goaway_sent = False+ self.retirement_notice_sent = Falseself._close_commands_sent = False+ self._frozen_goaway_bound = None+ self._minimum_serialized_goaway_bound = Noneself._pending: dict[int, RequestIdentity] = {}self._claimed: dict[int, RequestIdentity] = {}self._refused: dict[int, http.HTTPFlow] = {}@@ -113,6 +118,79 @@ class BoundH2Server(Http2Server):self._wakeup = Noneself._reset_stream = Noneself._event_depth = 0+ self._install_retirement_serializer()++ def _install_retirement_serializer(self) -> None:+ """Clamp GOAWAY frames on this fresh server connection only.++ The pinned BufferedH2Connection continues to own parsing, HPACK,+ CONTINUATION assembly, flow control, and output buffering. This+ retirement-only wrapper observes frames immediately before the+ dependency serializes them and never changes its state machine.+ """+ connection = self.h2_conn+ if (+ type(connection) is not BufferedH2Connection+ or connection._data_to_send+ ):+ raise H2IdentityError(+ "credential proxy: H2 serializer installed after connection start"+ )+ original_prepare = connection._prepare_for_sending++ def prepare_for_sending(frames):+ state = self.runtime.registry.drain_state(self.token)+ if (+ state is not None+ and state.fenced+ and state.frozen_last_stream_id is not None+ ):+ bound = state.frozen_last_stream_id+ if self._frozen_goaway_bound is None:+ self._frozen_goaway_bound = bound+ else:+ self._frozen_goaway_bound = min(+ self._frozen_goaway_bound, bound+ )++ serialized_bound = self._minimum_serialized_goaway_bound+ for frame in frames:+ if not isinstance(frame, GoAwayFrame):+ continue+ limits = [frame.last_stream_id]+ if self._frozen_goaway_bound is not None:+ limits.append(self._frozen_goaway_bound)+ if serialized_bound is not None:+ limits.append(serialized_bound)+ frame.last_stream_id = min(limits)+ serialized_bound = frame.last_stream_id++ original_prepare(frames)+ if serialized_bound is not None:+ self._minimum_serialized_goaway_bound = serialized_bound++ # An instance attribute shadows the inherited bound method without+ # changing BufferedH2Connection or H2Connection for other connections.+ connection._prepare_for_sending = prepare_for_sending++ def _send_retirement_notice(self, last_stream_id: int):+ """Queue one advisory NO_ERROR GOAWAY without closing h2's state machine."""+ if self.retirement_notice_sent:+ return+ self.h2_conn._prepare_for_sending(+ [+ GoAwayFrame(+ stream_id=0,+ last_stream_id=last_stream_id,+ error_code=h2.errors.ErrorCodes.NO_ERROR,+ additional_data=b"",+ )+ ]+ )+ self.retirement_notice_sent = True+ data = self.h2_conn.data_to_send()+ if data:+ yield commands.SendData(self.conn, data)def fail_closed(self) -> None:"""Hook-safe flag; normal layer commands run on the next layer event."""@@ -524,6 +602,12 @@ class BoundH2Server(Http2Server):or (state.deadline is not None and time.monotonic() >= state.deadline)or any(sid not in self._active for sid in self.h2_conn.stream_trailers)):return True+ if (+ state.fenced+ and state.frozen_last_stream_id is not None+ and not self.retirement_notice_sent+ ):+ return Truetry:for sid, record in self._active.items():upstream_done, upstream_trailers = self._upstream_done(record, observe=False)@@ -596,18 +680,31 @@ class BoundH2Server(Http2Server):returnif self.close_started:return- if (- state.fenced- and state.frozen_last_stream_id is not None- and not self._active+ drained = (+ not self._activeand not self._pendingand not self._claimedand not self._refusedand not self.http_layer.streamsand not self.h2_conn.stream_buffersand not self.h2_conn.stream_trailers+ )+ if (+ state.fenced+ and state.frozen_last_stream_id is not None+ and not drained+ and not self.retirement_notice_sent+ ):+ yield from self._send_retirement_notice(+ state.frozen_last_stream_id+ )+ if (+ state.fenced+ and state.frozen_last_stream_id is not None+ and drained):self.goaway_sent = True+ self.retirement_notice_sent = Trueself.h2_conn.close_connection(error_code=h2.errors.ErrorCodes.NO_ERROR,last_stream_id=state.frozen_last_stream_id,diff --git a/components/egress/tests/fixtures/h2_production_observer.py b/components/egress/tests/fixtures/h2_production_observer.pyindex d1e6023ec..49a1df5aa 100644--- a/components/egress/tests/fixtures/h2_production_observer.py+++ b/components/egress/tests/fixtures/h2_production_observer.py@@ -1,11 +1,13 @@# Copyright 2026 The OpenSandbox Authors# SPDX-License-Identifier: Apache-2.0-"""Read-only P2 metadata observer; never an adapter or publication owner.+"""P2 test observer; never an adapter or publication owner.-Load AFTER the actual system.py. The sole fault is an explicitly named async-ServerConnectedHook gate in an isolated child. It never closes a writer,-changes library queues, changes destinations, or replaces production methods.+Load AFTER the actual system.py. Its only scenario faults are an explicitly+named async ServerConnectedHook gate and a one-stream barrier immediately+before the real LiveRuntime admission method. The barrier delegates to that+same method after parent release. It never closes a writer, changes library+queues, changes destinations, or alters shared production code."""from __future__ import annotations@@ -13,6 +15,7 @@ from __future__ import annotationsimport asyncioimport jsonimport os+import threadingimport timefrom pathlib import Path@@ -23,6 +26,10 @@ _SCENARIO = os.environ.get("H2_PRODUCTION_SCENARIO", "normal")_sequence = 0_task = None_last = None+_barrier_release_value = os.environ.get("H2_PRODUCTION_BARRIER_RELEASE")+_barrier_release = None if _barrier_release_value is None else Path(_barrier_release_value)+_barrier_used = False+_barrier_stop = threading.Event()def emit(kind, **metadata):@@ -109,10 +116,41 @@ def running():if not _DIRECTORY.is_dir():raise RuntimeError("P2 observation directory is missing")emit("observer_ready", scenario=_SCENARIO)+ if _SCENARIO == "pre_admission_barrier_h2":+ live = credential_bound.runtime()+ if live is None or _barrier_release is None:+ raise RuntimeError("pre-admission test barrier has no live runtime or release path")+ original = live.acquire_request++ def gated_acquire(token, *, client_stream_id=None):+ global _barrier_used+ if client_stream_id == 3 and not _barrier_used:+ _barrier_used = True+ emit("pre_admission_barrier_entered", stream_id=client_stream_id,+ before_registry_acquire=True)+ deadline = time.monotonic() + 20+ while not _barrier_release.exists() and time.monotonic() < deadline:+ if _barrier_stop.wait(0.01):+ break+ if not _barrier_release.exists():+ emit("pre_admission_barrier_timeout", stream_id=client_stream_id)+ raise RuntimeError("pre-admission test barrier timed out")+ admission = original(token, client_stream_id=client_stream_id)+ state = live.registry.drain_state(token)+ emit("pre_admission_result", stream_id=client_stream_id,+ action=admission.action, reason=admission.reason,+ handle_created=admission.handle is not None,+ fenced=None if state is None else state.fenced,+ registry_request_count=live.registry.request_count)+ return admission+ return original(token, client_stream_id=client_stream_id)++ live.acquire_request = gated_acquire_task = asyncio.create_task(observe())def done():+ _barrier_stop.set()if _task is not None:_task.cancel()@@ -130,7 +168,8 @@ def requestheaders(flow):fenced=None if state is None else state.fenced,stream_id=None if handle is None else handle.client_stream_id,epoch=None if handle is None else handle.revision.decision_epoch,- admitted=handle is not None)+ admitted=handle is not None,+ private_token_present=bool(flow.request.headers.get_all("private-token")))def response(flow):diff --git a/components/egress/tests/fixtures/h2_retry_client.go b/components/egress/tests/fixtures/h2_retry_client.goindex 7f199005f..5ceac773c 100644--- a/components/egress/tests/fixtures/h2_retry_client.go+++ b/components/egress/tests/fixtures/h2_retry_client.go@@ -29,6 +29,7 @@ import ("errors""flag""io"+ "log""net""net/http""net/http/httptrace"@@ -37,6 +38,7 @@ import ("regexp""runtime""strconv"+ "strings""sync""sync/atomic""syscall"@@ -63,6 +65,80 @@ func (o *output) emit(event map[string]any) {}}+const maxH2DebugLine = 4096++var goAwayReadLog = regexp.MustCompile(`^http2: Framer 0x[0-9a-fA-F]+: read GOAWAY len=([0-9]+) LastStreamID=([0-9]+) ErrCode=([A-Z0-9_]+) Debug=`)++// filteredH2DebugWriter consumes Go 1.27.2 http2debug=2 diagnostics only in+// the explicit observer mode. It never forwards raw logger lines: DATA,+// headers, debug payloads and pointer identities are discarded in memory.+type filteredH2DebugWriter struct {+ mu sync.Mutex+ line []byte+ dropping bool+ out *output+ started time.Time+ ordinal uint64+}++func newFilteredH2DebugWriter(out *output) *filteredH2DebugWriter {+ return &filteredH2DebugWriter{out: out, started: time.Now()}+}++func (w *filteredH2DebugWriter) Write(data []byte) (int, error) {+ w.mu.Lock()+ defer w.mu.Unlock()+ for _, b := range data {+ if b == '\n' {+ if w.dropping {+ w.out.emit(map[string]any{"event": "observer_log_line_overflow"})+ } else if len(w.line) != 0 {+ w.process(string(w.line))+ }+ clear(w.line)+ w.line = w.line[:0]+ w.dropping = false+ continue+ }+ if w.dropping {+ continue+ }+ if len(w.line) == maxH2DebugLine {+ clear(w.line)+ w.line = w.line[:0]+ w.dropping = true+ continue+ }+ w.line = append(w.line, b)+ }+ return len(data), nil+}++func (w *filteredH2DebugWriter) process(line string) {+ if !strings.HasPrefix(line, "http2: Framer ") || !strings.Contains(line, ": read GOAWAY ") {+ return+ }+ match := goAwayReadLog.FindStringSubmatch(line)+ if match == nil {+ w.out.emit(map[string]any{"event": "goaway_observer_invalid", "reason": "format"})+ return+ }+ frameLength, lengthErr := strconv.ParseUint(match[1], 10, 24)+ lastStreamID, lastErr := strconv.ParseUint(match[2], 10, 32)+ if lengthErr != nil || lastErr != nil || frameLength < 8 ||+ lastStreamID > 0x7fffffff || match[3] != "NO_ERROR" {+ w.out.emit(map[string]any{"event": "goaway_observer_invalid", "reason": "fields"})+ return+ }+ w.ordinal+++ w.out.emit(map[string]any{+ "event": "goaway_frame_parsed", "source": "go_http2debug_filtered_framer_read",+ "frame_type": "GOAWAY", "stream_id": 0,+ "last_stream_id": lastStreamID, "error_code": 0,+ "framer_ordinal": w.ordinal, "monotonic_elapsed_ns": time.Since(w.started).Nanoseconds(),+ })+}+// The TLS connection identities seen by GotConn are opaque sequential IDs.// They are not stream IDs, transmission counts, or origin-processing counts.type observations struct {@@ -331,6 +407,13 @@ func controls(ctx context.Context, commands chan<- string) {}}+func goAwayObserverModeAllowed(enabled bool, debugSetting string) bool {+ if enabled {+ return debugSetting == "http2debug=2"+ }+ return debugSetting == ""+}+func run() int {out := &output{writer: json.NewEncoder(os.Stdout), failed: make(chan struct{})}requiredVersion := ""@@ -344,6 +427,7 @@ func run() int {flags := flag.NewFlagSet("h2_retry_client", flag.ContinueOnError)flags.SetOutput(io.Discard)flags.StringVar(&requiredVersion, "require-go-version", "", "required exact approved stable Go runtime version")+ observeGoAway := flags.Bool("observe-goaway-parse", false, "filter exact Go 1.27.2 HTTP/2 parser logs into sanitized events")socks := flags.String("socks-address", "", "IPv4 loopback SOCKS5 endpoint")caFile := flags.String("ca-file", "", "combined synthetic CA PEM file")timeout := flags.Duration("timeout", 25*time.Second, "overall deadline")@@ -357,7 +441,17 @@ func run() int {if runtime.Version() != requiredVersion {return fail("unsupported_go_version")}- if os.Getenv("GODEBUG") != "" {+ debugSetting := os.Getenv("GODEBUG")+ if !goAwayObserverModeAllowed(*observeGoAway, debugSetting) {+ if *observeGoAway {+ return fail("goaway_observer_requires_exact_godebug")+ }+ return fail("ambient_godebug_not_supported")+ }+ if *observeGoAway {+ log.SetFlags(0)+ log.SetOutput(newFilteredH2DebugWriter(out))+ } else if debugSetting != "" {return fail("ambient_godebug_not_supported")}host, port, err := net.SplitHostPort(*socks)@@ -400,7 +494,8 @@ func run() int {"event": "ready", "schema": 1, "go_version": runtime.Version(), "required_go_version": requiredVersion,"transport": map[string]any{"force_attempt_http2": true, "proxy_environment": false, "dial": "loopback_socks5_ipv4",- "origin": "127.0.0.1:443", "authority": targetHost, "tls_server_name": targetHost,+ "goaway_observer_mode": map[bool]string{true: "filtered_go_http2debug_parser_event", false: "disabled"}[*observeGoAway],+ "origin": "127.0.0.1:443", "authority": targetHost, "tls_server_name": targetHost,"insecure_skip_verify": false, "trust": "supplied_ca_only", "tls_min_version": "1.2","disable_compression": true, "redirects": false, "manual_retries": 0,"max_idle_conns": 2, "max_idle_conns_per_host": 2, "max_conns_per_host": 0,diff --git a/components/egress/tests/fixtures/h2_retry_client_README.md b/components/egress/tests/fixtures/h2_retry_client_README.mdindex 6f9c23198..cd544b45f 100644--- a/components/egress/tests/fixtures/h2_retry_client_README.md+++ b/components/egress/tests/fixtures/h2_retry_client_README.md@@ -1,18 +1,45 @@# Fixed Go HTTP/2 GET observer-This fixture covers only the explicitly selected Go `net/http` synthetic GET observations. It does not replay requests in application code or establish any non-idempotent guarantee. The user-approved D2 revision replaces the original Go 1.25.9 pin with the latest stable Go installed for the acceptance run. Build the single `h2_retry_client.go` file with that independently verified toolchain and pass its exact `goX.Y.Z` version using the required `-require-go-version` option. There is no default or fallback. `ready` and `fatal` report both `required_go_version` and `go_version`. Version mismatch and nonempty `GODEBUG` are rejected before dialing.+This fixture uses the standard Go 1.27.2 `net/http.Transport` and one `http.Client.Do` per logical GET. It never replays a request in application code and makes no non-idempotent safety claim. Build the single `h2_retry_client.go` file using the independently verified Go 1.27.2 compiler; pass its exact `go1.27.2` identity with `-require-go-version`. The program rejects any runtime mismatch. It has no non-standard dependencies or implicit toolchain fallback.-CLI: `h2_retry_client -require-go-version goX.Y.Z -socks-address 127.0.0.1:<port> -ca-file <combined-ca.pem> -timeout 25s -new-timeout 8s`. The CA bundle must contain the synthetic MITM and origin roots. TLS verifies `bound.example.com`; SOCKS5 CONNECT always targets the real `127.0.0.1:443`. No environment proxy, redirect, alternate target, insecure TLS, or external Go module is used. Both requests share one HTTP/2-enabled Transport.+The fixed topology is `127.0.0.1:<SOCKS port>` to the actual `127.0.0.1:443` origin, with TLS SNI/authority `bound.example.com`. The synthetic CA bundle contains the MITM and origin test roots. TLS verification, standard `Transport` TLS setup, and automatic HTTP/2 negotiation remain in place. No environment proxy, redirect, alternate target, insecure TLS, or Go module dependency is used.-Control protocol (UTF-8 JSON lines; keep stdin open):+## Case roles-1. Wait for `ready` and `old_headers` (`proto=HTTP/2.0`, verified TLS). The old GET is `/v1/old`, with `X-H2-Logical-ID: old`; its body drains concurrently.-2. After the parent has committed removal through authenticated revision IPC, write `{"command":"continue_new"}` plus newline exactly once while the old response is still open (early completion is rejected). The single new `client.Do` issues `/v1/new`, with `X-H2-Logical-ID: new` on the same Transport.-3. The parent harness chooses one of two separate old-response release policies:- - `go_retry` holds old until `new_result`, then releases it. This tests whether the library can migrate while old remains unfinished.- - `go_retry_bounded_old` releases old after observing this new GET at the original connection's post-ACK fence, without waiting for `new_result`. This tests recovery when old independently finishes early in the retry window; it cannot establish recovery for arbitrary old-response durations. Await `old_result`, `new_result` and `done` under either policy. The new GET has its own deadline and may legitimately negotiate HTTP/1.1 on a fresh, unbound pass-through connection. Its failure does not cancel the old GET. The overall deadline still bounds both requests. The helper's Transport and single-Do behavior are identical in both scenarios.-4. `{"command":"cancel"}`, stdin EOF, invalid control, SIGINT, or SIGTERM cancels the run and closes its owned connections. Control lines are limited to 4096 bytes.+- When `--go-client` is supplied without explicit `--case` arguments, the runner selects only the two `go_early_notice_*` candidate cases; historical diagnostics run only when named explicitly.+- `go_retry` retains the earlier delayed/refusal diagnostic: new GET must hit the old connection's post-ACK fence before the fixture releases the held response. It is historical diagnostic evidence, not a candidate acceptance gate; Go Transport scheduling may legitimately change after early GOAWAY.+- `go_retry_bounded_old` retains the old bounded-release diagnostic, which releases only after the same old-connection fence. It does not prove arbitrary held-old migration and is also not a candidate gate.+- `go_early_notice_h1_origin` and `go_early_notice_h2_origin` are separate held-old interoperability cases. The source holds old's body before END_STREAM. After authenticated removal ACK, the fixture waits for Go's standard HTTP/2 Framer to parse the production GOAWAY, then starts exactly one new `Do` while old remains unfinished. The new request may use a directly fresh connection or be refused on the old connection in a natural race and then retried by the same standard Transport. It must ultimately complete on a fresh connection before old is released. H1-origin and true H2-to-H2 results remain separate.+- `pre_admission_barrier_h2` is a production-addon/LiveRuntime/Registry case driven by the manual peer. It pauses stream 3 before the real `LiveRuntime.acquire_request`; its only admitted old stream is 1, so this IPC case proves the post-ACK hook path above frozen bound 1. It commits and reads back the fence over authenticated Unix IPC, then resumes the real admission call. It is not a normal-client interoperability claim or a below-bound proof.-`got_conn` uses process-local IDs keyed by actual `net.Conn` identity. Each result contains `do_calls`, `got_conn_count`, observed `connection_ids` (including repeats), `write_count`, HTTP status/protocol, body bytes/SHA-256/completeness, and a bounded `error_class`. Trace/write counts do not prove wire attempts or origin processing; the parent independently measures at-most-once and refusal/connection ordering. No HTTP/2 stream IDs, body contents, arbitrary errors, or response headers are logged.+The socket-free lifecycle suite separately tests a lower-numbered late admission through the actual `LiveRuntime` and Registry owners: stream 3 is held before acquire, stream 5 receives a real handle, the new revision fences the connection at bound 5, then acquire for stream 3 is denied. That owner-ordering test does not simulate wire order, a `system.requestheaders` hook, or IPC.-`done.outcome=observed` and exit 0 mean both observations finished with a complete old response, even if the new GET timed out or failed. They are **not acceptance**: the parent must assert expected status/body, old-stream preservation, library retry outcome, origin at-most-once, and actual new-connection revision behavior. Setup, control, old-response, and overall-deadline failures exit 2. Runtime verification must be reported separately; source preparation is not a Go build or runtime pass. See the [versioned integration record](../../../../docs/architecture/network/osep-0023-h2-integration.md#observed-go-retry-limit-and-staged-delivery) for the separate held-old failure, bounded-old result and remaining acceptance gates.+The runtime suite's `ManualH2Peer` records GOAWAY but deliberately does not feed it to hyper-h2's protocol state machine. Its frame-order and late-frame cases are protocol stress tests, not compliant-client migration evidence. Only the standard Go cases above provide client parser/interoperability evidence.++## GOAWAY parser-event filtering++The default fixture rejects every non-empty `GODEBUG` and does not enable Go HTTP/2 diagnostics. The two `go_early_notice_*` runner cases explicitly pass `-observe-goaway-parse` and set exactly `GODEBUG=http2debug=2`. For the pinned Go 1.27.2 standard library, the fixture filters the `Framer ...: read GOAWAY` parser log into a fixed JSON allowlist: frame type, stream ID zero, last stream ID, NO_ERROR code, run-local ordinal, and a monotonic elapsed time. Unsupported or oversized lines become fixed redacted observer errors. DATA, headers, GOAWAY debug bytes, pointer IDs, raw log lines, and other diagnostics are discarded in memory; child stderr is discarded and raw logs never enter artifacts.++This is a **standard-library parser event**, not wire capture. The Framer logs after parsing and before the H2 transport processes GOAWAY; the one old H2 connection is the only successful Go connection before the new `Do` begins, so the pre-request event is attributed by that causal window. Final request success, old completion ordering, `httptrace`, and origin attempt counts are checked separately. A `GotConn` event alone is never treated as GOAWAY proof.++## Control and evidence++CLI:++```text+h2_retry_client -require-go-version go1.27.2 \+ -socks-address 127.0.0.1:<port> -ca-file <combined-ca.pem> \+ -timeout 25s -new-timeout 8s [-observe-goaway-parse]+```++Keep stdin open. Wait for `ready` and verified H2 `old_headers`; after the parent confirms authenticated removal, send one JSON line:++```json+{"command":"continue_new"}+```++The early-notice cases wait for a filtered `goaway_frame_parsed` event before sending that command. The fixture emits sanitized request/result events only: `do_calls`, `got_conn_count`, opaque connection IDs, write count, HTTP status and protocol, body hash/completeness, and bounded error classes. Neither headers, body bytes, arbitrary errors nor Go's raw HTTP/2 diagnostic lines are saved. Go 1.27.2 omits zero-valued frame stream IDs from this debug summary; the filtered event records `stream_id=0` because the standard Framer successfully parsed the frame as GOAWAY, whose parser rejects a nonzero stream ID. `done.outcome=observed` and exit 0 mean the fixture finished collecting results; only the parent runner's individual assertions can decide whether a case passed.++The H2 origin keeps the old stream open while continuing to read the same H2 connection, accept other streams, and process flow-control updates. Once the new result or its fixed deadline is observed, the parent releases old; the origin sends the remaining bytes within the stream window and then END_STREAM. The source records one attempt per logical ID and reports H1/H2 separately.++The runtime runner still requires an explicitly authorized environment with canonical loopback port 443 available. It has no high-port fallback. See the [versioned integration record](../../../../docs/architecture/network/osep-0023-h2-integration.md#observed-go-retry-limit-and-staged-delivery) for prior held-old results and outstanding acceptance limits.diff --git a/components/egress/tests/fixtures/h2_retry_client_test.go b/components/egress/tests/fixtures/h2_retry_client_test.gonew file mode 100644index 000000000..5ace6405e--- /dev/null+++ b/components/egress/tests/fixtures/h2_retry_client_test.go@@ -0,0 +1,104 @@+// Copyright 2026 The OpenSandbox Authors+// SPDX-License-Identifier: Apache-2.0++package main++import (+ "bytes"+ "encoding/json"+ "strings"+ "testing"+)++func TestGoAwayObserverRequiresExactOptIn(t *testing.T) {+ for _, test := range []struct {+ name string+ enabled bool+ debug string+ want bool+ }{+ {name: "disabled without debug", want: true},+ {name: "enabled exact", enabled: true, debug: "http2debug=2", want: true},+ {name: "enabled combined", enabled: true, debug: "http2debug=2,netdns=go"},+ {name: "enabled wrong level", enabled: true, debug: "http2debug=1"},+ {name: "disabled ambient", debug: "http2debug=2"},+ } {+ t.Run(test.name, func(t *testing.T) {+ if got := goAwayObserverModeAllowed(test.enabled, test.debug); got != test.want {+ t.Fatalf("mode allowed=%t, want %t", got, test.want)+ }+ })+ }+}++func filteredEvents(t *testing.T, writes ...string) ([]map[string]any, string) {+ t.Helper()+ var outputBytes bytes.Buffer+ out := &output{writer: json.NewEncoder(&outputBytes), failed: make(chan struct{})}+ filter := newFilteredH2DebugWriter(out)+ for _, value := range writes {+ if _, err := filter.Write([]byte(value)); err != nil {+ t.Fatal("filtered logger write failed")+ }+ }+ var events []map[string]any+ decoder := json.NewDecoder(&outputBytes)+ for decoder.More() {+ var event map[string]any+ if err := decoder.Decode(&event); err != nil {+ t.Fatal("decode sanitized observer event")+ }+ events = append(events, event)+ }+ return events, outputBytes.String()+}++func TestFilteredH2DebugWriterEmitsOnlySanitizedParsedGoAwayFields(t *testing.T) {+ events, encoded := filteredEvents(t,+ `http2: Framer 0xcafe: read DATA stream=1 len=24 data="synthetic-private-value"`+"\n",+ `http2: Framer 0xbeef: read HEADERS flags=END_HEADERS stream=3 len=18`++ ` fields=["private-token: synthetic-secret"]`+"\n",+ `http2: Framer 0x1234: read GOAWAY len=12 LastStreamID=3 ErrCode=NO_ERROR Debug="synthetic-debug-secret"`+"\n",+ )+ if len(events) != 1 {+ t.Fatalf("expected one filtered GOAWAY parser event, got %d", len(events))+ }+ event := events[0]+ if event["event"] != "goaway_frame_parsed" ||+ event["source"] != "go_http2debug_filtered_framer_read" ||+ event["frame_type"] != "GOAWAY" || event["stream_id"] != float64(0) ||+ event["last_stream_id"] != float64(3) || event["error_code"] != float64(0) ||+ event["framer_ordinal"] != float64(1) {+ t.Fatalf("unexpected sanitized GOAWAY fields: %#v", event)+ }+ if event["monotonic_elapsed_ns"].(float64) < 0 {+ t.Fatalf("invalid monotonic observation time: %#v", event)+ }+ for _, forbidden := range []string{"synthetic-private-value", "synthetic-secret", "synthetic-debug-secret", "0x1234", "Debug", "data=", "fields="} {+ if strings.Contains(encoded, forbidden) {+ t.Fatalf("raw Go diagnostic content escaped filtering: %s", encoded)+ }+ }+}++func TestFilteredH2DebugWriterRejectsUnsupportedAndOversizedRecords(t *testing.T) {+ long := "http2: Framer 0x1234: " + strings.Repeat("private-header-value", maxH2DebugLine)+ events, encoded := filteredEvents(t,+ `http2: Framer 0x1234: read GOAWAY unusual-format synthetic-secret`+"\n",+ `http2: Framer 0x5678: read GOAWAY len=8 LastStreamID=1 ErrCode=PROTOCOL_ERROR Debug="secret"`+"\n",+ long+"\n",+ )+ if len(events) != 3 {+ t.Fatalf("expected only bounded observer failure summaries, got %d", len(events))+ }+ if events[0]["event"] != "goaway_observer_invalid" || events[0]["reason"] != "format" ||+ events[1]["event"] != "goaway_observer_invalid" || events[1]["reason"] != "fields" ||+ events[2]["event"] != "observer_log_line_overflow" {+ t.Fatalf("unexpected sanitized failure summaries: %#v", events)+ }+ for _, forbidden := range []string{"synthetic-secret", "PROTOCOL_ERROR", "secret", "private-header-value", "0x1234"} {+ if strings.Contains(encoded, forbidden) {+ t.Fatalf("diagnostic filter leaked rejected raw line: %s", encoded)+ }+ }+}diff --git a/components/egress/tests/test_credential_bound_h2.py b/components/egress/tests/test_credential_bound_h2.pyindex ea76d33a1..a5a6f3db0 100644--- a/components/egress/tests/test_credential_bound_h2.py+++ b/components/egress/tests/test_credential_bound_h2.py@@ -1,7 +1,12 @@# Copyright 2026 The OpenSandbox Authors# SPDX-License-Identifier: Apache-2.0-"""Pure pinned-library layer tests. No sockets, TLS handshake, IPC, or Go claims."""+"""Socket-free H2/manual-peer stress tests; no TLS, real IPC or client interoperability claim.++The companion opt-in runner's ManualH2Peer records and withholds GOAWAY from+hyper-h2 state processing. Its frame timing is useful for late-frame stress,+not proof that a normal client received GOAWAY or migrated correctly.+"""import sysimport unittest@@ -14,8 +19,13 @@ from unittest.mock import patchtry:import h2.configimport h2.connection+ import h2.errorsimport h2.eventsimport h2.settings+ from hyperframe.frame import (+ ContinuationFrame, DataFrame, Frame, GoAwayFrame, HeadersFrame,+ RstStreamFrame, WindowUpdateFrame,+ )from mitmproxy import connection, httpfrom mitmproxy.proxy import commands, eventsfrom mitmproxy.proxy.context import Context@@ -164,6 +174,8 @@ class Harness:self.identities = []self.output = []self.peer_events = []+ self.client_wire = bytearray()+ self.parse_client_wire = Trueself.origin_events = []self.origin_bytes = bytearray()self.opens = []@@ -181,6 +193,7 @@ class Harness:self.identities.append(identity)if (self.deny+ or self.runtime.state.fencedor len(self.adapter._active) >= self.runtime.registry.request_capacity):self.adapter.refuse(identity, flow)@@ -215,7 +228,9 @@ class Harness:self.opens.append(command)elif isinstance(command, commands.SendData):if command.connection is self.client:- self.peer_events.extend(self.peer.receive_data(command.data))+ self.client_wire.extend(command.data)+ if self.parse_client_wire:+ self.peer_events.extend(self.peer.receive_data(command.data))else:self.upstream_conn = command.connectionself.origin_bytes.extend(command.data)@@ -249,6 +264,25 @@ class Harness:if data:self.run(events.DataReceived(self.client, data))+ @staticmethod+ def decode_frames(data):+ """Decode synthetic wire bytes with pinned hyperframe."""+ frames = []+ offset = 0+ while offset < len(data):+ header = memoryview(data[offset : offset + 9])+ frame, body_length = Frame.parse_frame_header(header)+ end = offset + 9 + body_length+ if end > len(data):+ raise AssertionError("truncated synthetic server frame")+ frame.parse_body(memoryview(data[offset + 9 : end]))+ frames.append(frame)+ offset = end+ return frames++ def client_frames(self):+ return self.decode_frames(self.client_wire)+def tick(self):wake = self.adapter._wakeupif wake is not None:@@ -334,7 +368,7 @@ class Harness:dependency_versions_supported(),"requires mitmproxy 11.0.2 / h2 4.1.0 / hyperframe 6.0.1",)-class BoundH2Tests(unittest.TestCase):+class SocketFreeManualPeerStressTests(unittest.TestCase):def test_instance_identity_is_raw_exact_and_one_shot(self):h = Harness(deny=True)h.headers(@@ -415,6 +449,41 @@ class BoundH2Tests(unittest.TestCase):h.assert_identity_empty(self)h.assert_no_origin(self)+ def test_socket_free_stub_paused_stream_is_refused_after_fence_not_ac07_acceptance(self):+ h = Harness(hold={("requestheaders", 1)})+ h.headers(1)+ h.headers(3, end=True)+ self.assertEqual(list(h.adapter._active), [3])+ self.assertEqual(list(h.adapter._pending.values())[0].stream_id, 1)++ h.fence(last=3)+ h.tick()+ self.assertEqual(+ [event.last_stream_id for event in h.peer_events+ if isinstance(event, h2.events.ConnectionTerminated)],+ [3],+ )+ h.parse_client_wire = False+ h.release(("requestheaders", 1))++ self.assertEqual(list(h.adapter._active), [3])+ self.assertNotIn(1, h.adapter._claimed)+ self.assertNotIn(1, h.adapter._pending)+ self.assertNotIn(1, h.adapter._refused)+ self.assertFalse(h.adapter.failed)+ origin_requests = [+ event for event in h.origin_events+ if isinstance(event, h2.events.RequestReceived)+ ]+ self.assertEqual(len(origin_requests), 1)+ self.assertEqual(dict(origin_requests[0].headers)[b":path"], b"/3")+ resets = [+ frame for frame in h.client_frames()+ if isinstance(frame, RstStreamFrame) and frame.stream_id == 1+ ]+ self.assertEqual(len(resets), 1)+ self.assertEqual(resets[0].error_code, h2.errors.ErrorCodes.REFUSED_STREAM)+def test_active_capacity_exhaustion_refuses_new_stream_preserving_old(self):h = Harness(capacity=1)h.headers(1)@@ -544,34 +613,121 @@ class BoundH2Tests(unittest.TestCase):h.adapter.owned_upstreams(), ((association[0], association[1]),))- def test_h2_backpressure_retains_handle_and_delays_goaway(self):+ def test_manual_peer_late_goaway_stress_retains_handle_and_delays_close(self):h = Harness(downstream_window=0)h.headers(end=True)h.from_origin()h.response(body=b"payload")+ state = h.adapter.h2_conn.state_machine.stateh.fence()h.tick()self.assertFalse(h.runtime.finished)self.assertTrue(h.adapter.h2_conn.stream_buffers[1])self.assertFalse(h.adapter.goaway_sent)+ early = [+ event for event in h.peer_events+ if isinstance(event, h2.events.ConnectionTerminated)+ ]+ self.assertEqual([(event.last_stream_id, event.error_code) for event in early], [(1, 0)])+ self.assertNotIn(+ 1,+ [event.stream_id for event in h.peer_events+ if isinstance(event, h2.events.StreamEnded)],+ )+ self.assertIs(h.adapter.h2_conn.state_machine.state, state)handle = h.flows[1].metadata["opensandbox_credential_request_handle"]- h.peer.increment_flow_control_window(100, stream_id=1)- h.flush()+ h.parse_client_wire = False+ window_update = WindowUpdateFrame(stream_id=1)+ window_update.window_increment = 100+ h.run(events.DataReceived(h.client, window_update.serialize()))h.tick()self.assertEqual(h.runtime.finished, [handle])self.assertNotIn("opensandbox_credential_request_handle", h.flows[1].metadata)self.assertTrue(h.adapter.goaway_sent)- terms = [- x for x in h.peer_events if isinstance(x, h2.events.ConnectionTerminated)+ frames = h.client_frames()+ goaways = [+ (index, frame) for index, frame in enumerate(frames)+ if isinstance(frame, GoAwayFrame)+ ]+ self.assertEqual(+ [(frame.last_stream_id, frame.error_code) for _, frame in goaways],+ [(1, 0), (1, 0)],+ )+ early_index, final_index = goaways[0][0], goaways[1][0]+ body_frames = [+ (index, frame) for index, frame in enumerate(frames)+ if isinstance(frame, DataFrame) and frame.stream_id == 1]- self.assertEqual([(x.last_stream_id, x.error_code) for x in terms], [(1, 0)])- types = [type(x) for x in h.peer_events]- self.assertLess(- types.index(h2.events.StreamEnded),- types.index(h2.events.ConnectionTerminated),+ self.assertEqual(+ b"".join(frame.data for _, frame in body_frames), b"payload")+ self.assertIn("END_STREAM", body_frames[-1][1].flags)+ self.assertLess(early_index, body_frames[0][0])+ self.assertLess(body_frames[-1][0], final_index)self.assertFalse(h.adapter.failed)+ def test_goaway_serialization_clamps_all_paths_and_preserves_instance_scope(self):+ for path in ("close", "terminate", "protocol_error"):+ with self.subTest(path=path):+ h = Harness()+ h.headers(end=True)+ h.runtime.state.fenced = True+ h.runtime.state.frozen_last_stream_id = 0+ connection_class_method = type(h.adapter.h2_conn)._prepare_for_sending++ if path == "close":+ h.adapter.h2_conn.close_connection(+ error_code=h2.errors.ErrorCodes.NO_ERROR,+ last_stream_id=9,+ )+ data = h.adapter.h2_conn.data_to_send()+ expected_error = h2.errors.ErrorCodes.NO_ERROR+ elif path == "terminate":+ h.adapter.h2_conn._terminate_connection(+ h2.errors.ErrorCodes.PROTOCOL_ERROR+ )+ data = h.adapter.h2_conn.data_to_send()+ expected_error = h2.errors.ErrorCodes.PROTOCOL_ERROR+ else:+ commands_out = list(h.adapter.protocol_error(+ "synthetic protocol error",+ h2.errors.ErrorCodes.PROTOCOL_ERROR,+ ))+ data = next(+ command.data for command in commands_out+ if isinstance(command, commands.SendData)+ )+ expected_error = h2.errors.ErrorCodes.PROTOCOL_ERROR++ events = h.peer.receive_data(data)+ goaway = next(+ event for event in events+ if isinstance(event, h2.events.ConnectionTerminated)+ )+ self.assertEqual(goaway.last_stream_id, 0)+ self.assertEqual(goaway.error_code, expected_error)+ self.assertEqual(h.adapter.h2_conn.highest_inbound_stream_id, 1)+ self.assertIs(type(h.adapter.h2_conn)._prepare_for_sending, connection_class_method)+ self.assertNotIn("_prepare_for_sending", type(h.adapter.h2_conn).__dict__)++ def test_serialized_goaway_bound_never_increases(self):+ h = Harness()+ h.runtime.state.fenced = True+ h.runtime.state.frozen_last_stream_id = 5+ h.adapter.h2_conn._prepare_for_sending([+ GoAwayFrame(0, last_stream_id=2, error_code=h2.errors.ErrorCodes.NO_ERROR)+ ])+ h.adapter.h2_conn._prepare_for_sending([+ GoAwayFrame(0, last_stream_id=9, error_code=h2.errors.ErrorCodes.PROTOCOL_ERROR)+ ])+ events = h.peer.receive_data(h.adapter.h2_conn.data_to_send())+ goaways = [event for event in events if isinstance(event, h2.events.ConnectionTerminated)]+ self.assertEqual([event.last_stream_id for event in goaways], [2, 2])+ self.assertEqual(+ [event.error_code for event in goaways],+ [h2.errors.ErrorCodes.NO_ERROR, h2.errors.ErrorCodes.PROTOCOL_ERROR],+ )+def test_h1_normal_response_releases_only_after_upload_finishes(self):h = Harness(upstream="h1")h.headers(extra=[("content-length", "3")])@@ -770,10 +926,47 @@ class BoundH2Tests(unittest.TestCase):h = Harness(max_streams=1)h.headers(1, end=True)h.from_origin()- h.headers(3, end=True)+ h.headers(3, end=True, extra=[("x-dynamic", "cached-value")])+ self.assertIn(+ (b"x-dynamic", b"cached-value"),+ h.peer.encoder.header_table.dynamic_entries,+ )+ h.peer.max_outbound_frame_size = 16384+ long_value = "".join(chr(33 + (index * 17) % 90) for index in range(45000))+ h.peer.send_headers(+ 5,+ [+ (":method", "POST"),+ (":scheme", "https"),+ (":authority", "example.test"),+ (":path", "/5"),+ ("x-dynamic", "cached-value"),+ ("x-long", long_value),+ ],+ end_stream=False,+ )+ h.peer.send_data(5, b"post-bound", end_stream=True)+ post_bound_bytes = h.peer.data_to_send()+ post_bound_frames = h.decode_frames(post_bound_bytes)+ self.assertTrue(any(isinstance(frame, ContinuationFrame)+ for frame in post_bound_frames))h.fence()+ h.tick()h.deny = True- h.headers(5, end=True)+ h.parse_client_wire = False+ data_window = h.adapter.h2_conn._inbound_flow_control_window_manager+ bytes_processed = data_window._bytes_processed+ h.run(events.DataReceived(h.client, post_bound_bytes))+ self.assertEqual(data_window._bytes_processed - bytes_processed, len(b"post-bound"))+ self.assertNotIn(5, h.adapter._active)+ self.assertNotIn(5, h.root.streams)+ h.assert_identity_empty(self)+ self.assertTrue(any(+ (b"x-dynamic", b"cached-value") in identity.headers+ for identity in h.identities if identity.stream_id == 5+ ))+ self.assertNotIn(5, h.adapter.h2_conn.stream_buffers)+ self.assertNotIn(5, h.adapter.h2_conn.stream_trailers)self.assertEqual(h.adapter._association(h.adapter._active[3])[3], "queued")h.response(1, body=b"first")h.tick()@@ -783,10 +976,26 @@ class BoundH2Tests(unittest.TestCase):h.tick()self.assertEqual(len(h.runtime.finished), 2)self.assertFalse(h.adapter.failed)+ frames = h.client_frames()terms = [- x for x in h.peer_events if isinstance(x, h2.events.ConnectionTerminated)+ frame for frame in frames if isinstance(frame, GoAwayFrame)]- self.assertEqual([(x.last_stream_id, x.error_code) for x in terms], [(3, 0)])+ self.assertEqual(+ [(frame.last_stream_id, frame.error_code) for frame in terms],+ [(3, 0), (3, 0)],+ )+ refused = [+ frame for frame in frames+ if isinstance(frame, RstStreamFrame) and frame.stream_id == 5+ ]+ self.assertEqual(len(refused), 1)+ self.assertEqual(refused[0].error_code, h2.errors.ErrorCodes.REFUSED_STREAM)+ self.assertFalse(+ any(+ isinstance(frame, (HeadersFrame, DataFrame)) and frame.stream_id == 5+ for frame in frames+ )+ )self.assertEqual([x.stream_iddiff --git a/components/egress/tests/test_credential_bound_h2_lifecycle.py b/components/egress/tests/test_credential_bound_h2_lifecycle.pyindex 9c9f733a9..357e11e74 100644--- a/components/egress/tests/test_credential_bound_h2_lifecycle.py+++ b/components/egress/tests/test_credential_bound_h2_lifecycle.py@@ -838,6 +838,57 @@ class H2RuntimeLifecycleTest(unittest.TestCase):self.assertEqual((denied.action, denied.reason), ("deny", "connection_fenced"))self.assertEqual(f.process.registry.request_count, 1)+ def test_live_registry_denies_late_lower_stream_below_frozen_high_water(self):+ """Real LiveRuntime/Registry owner ordering; no wire or hook claim."""+ f = self.fixture()+ token = f.item.token+ # Stream 3 is held at the pre-acquire point; no handle is fabricated.+ paused_stream_id = 3++ higher = f.process.acquire_request(token, client_stream_id=5)+ self.assertEqual((higher.action, higher.reason), ("allow", "admitted"))+ self.assertIsNotNone(higher.handle)+ self.assertEqual(higher.handle.client_stream_id, 5)+ self.assertEqual(+ f.process.registry.drain_state(token).max_admitted_stream_id, 5+ )++ old_snapshot = f.process._receiver.acquire()+ self.assertIsNotNone(old_snapshot)+ payload = json.loads(old_snapshot.payload)+ payload.update(+ state="active-empty",+ vaultRevision=2,+ tlsBindingHostSelectors=[],+ fullRenderedBindings=[],+ redactions=[],+ )+ encoded = json.dumps(payload, separators=(",", ":")).encode()+ revision = Revision(+ "control", "subject", 2, 2, 0, hashlib.sha256(encoded).hexdigest()+ )+ receiver = f.process._receiver+ self.assertEqual(receiver.prepare(revision, encoded), revision)+ self.assertEqual(receiver.commit(revision), revision)+ self.assertEqual(receiver.readback(), revision)++ state = f.process.registry.drain_state(token)+ self.assertTrue(state.fenced)+ self.assertEqual(state.frozen_last_stream_id, 5)+ self.assertLessEqual(paused_stream_id, state.frozen_last_stream_id)+ self.assertEqual(f.request_handle.snapshot.revision.decision_epoch, 1)+ self.assertEqual(higher.handle.snapshot.revision.decision_epoch, 1)++ late = f.process.acquire_request(token, client_stream_id=paused_stream_id)+ self.assertEqual((late.action, late.reason), ("deny", "connection_fenced"))+ self.assertIsNone(late.handle)+ self.assertEqual(f.process.registry.request_count, 2)+ self.assertEqual(f.process.registry.count, 1)++ f.process.finish_request(higher.handle)+ f.process.finish_request(f.request_handle)+ self.assertEqual(f.process.registry.request_count, 0)+def test_close_exception_is_not_consumption_or_completion(self):f = self.fixture(full_lifecycle=True, delay_cancel=True, handler_type=FailingCloseHandlerdiff --git a/components/egress/tests/test_credential_bound_h2_runtime.py b/components/egress/tests/test_credential_bound_h2_runtime.pyindex 3ce217724..9e03b854f 100644--- a/components/egress/tests/test_credential_bound_h2_runtime.py+++ b/components/egress/tests/test_credential_bound_h2_runtime.py@@ -12,7 +12,10 @@ Without --run-real-runtime this exits 2 with NOT_RUN. --list is static. Theorigin MUST really bind 127.0.0.1:443; occupied/denied 443 is BLOCKED, never ahigh-port fallback. The client uses actual SOCKS5 CONNECT to that address,then verified TLS SNI bound.example.com and H2. The real system.py, receiver,-registry and adapter are untouched. Test observation reads metadata only.+registry and adapter execute unchanged. The scenario-specific observer may+pause immediately before the real LiveRuntime admission call. ManualH2Peer+records but withholds GOAWAY from hyper-h2; only Go early-notice cases establish+normal client receipt and must be used for interoperability claims.Exit 0 means every selected case passed; 1 is assertion/protocol failure;2 is blocked or not run. Inspect report.json's full denominator: unselected@@ -52,20 +55,28 @@ SECRET_TWO = "runtime-rotated-secret-two"BODY = (b"synthetic-production-h2-body-" * 6000)[:131089]BODY_SHA256 = hashlib.sha256(BODY).hexdigest()SMALL_BODY = b"synthetic-production-h2-ok"-CASES = [f"{name}_{protocol}" for name in (+H1_HELD_PREFIX = b"data: synthetic-old-start\n\n"+H2_HELD_PREFIX = BODY[:24]+MANUAL_PEER_CASES = [f"{name}_{protocol}" for name in ("attachment", "removal_drain", "rotation", "remove_readd", "reset_isolation", "disconnect") for protocol in ("h1", "h2")] + ["expiry_short_h2", "expiry_default_30s_h2", "d3_pending_writer"]-GO_CASES = ("go_retry", "go_retry_bounded_old")+REAL_RUNTIME_CASES = ("pre_admission_barrier_h2",)+CASES = [*MANUAL_PEER_CASES, *REAL_RUNTIME_CASES]+GO_DELAYED_CASES = ("go_retry", "go_retry_bounded_old")+GO_EARLY_CASES = ("go_early_notice_h1_origin", "go_early_notice_h2_origin")+GO_CASES = (*GO_DELAYED_CASES, *GO_EARLY_CASES)GO_SCOPE = {- "go_retry": "Old response held until new_result; success requires migration while the old response remains unfinished.",- "go_retry_bounded_old": "Old response released after the new GET reaches the post-ACK old-connection fence, before its 8-second retry budget ends. Success would not prove migration while old never finishes or when a finite old response outlasts that budget.",+ "go_retry": "Historical non-gating diagnostic, available only by explicit --case selection: old response is held until new_result and new must first reach the post-ACK old-connection fence. This scheduling requirement is not a candidate gate after early GOAWAY.",+ "go_retry_bounded_old": "Historical non-gating diagnostic, available only by explicit --case selection: old response is released after new reaches the post-ACK old-connection fence. It does not prove arbitrary held-old migration.",+ "go_early_notice_h1_origin": "H1 origin holds the old response. After the standard Go Framer parses production GOAWAY, one new Do must complete on a fresh connection while old is unfinished; direct fresh selection or a natural old-fence refusal followed by the same Transport retry are both allowed.",+ "go_early_notice_h2_origin": "A real H2 origin holds the old response. After the standard Go Framer parses production GOAWAY, one new Do must complete on a fresh connection while old is unfinished; direct fresh selection or a natural old-fence refusal followed by the same Transport retry are both allowed. This is a separate H2-to-H2 cell.",}MISSING = ["Linux transparent redirects and actual original-destination lookup","Restricted mitmproxy user, IPv6 and production CA provisioning","Go generation owner, readiness, recovery and kernel quarantine readback","Fragmented HEADERS, same-batch DATA/RST, HPACK and flow-control variants",- "Pre-ACK fence timing / commit-before-admission race and path-only/policy-only rotations",+ "Pre-ACK fence timing and path-only/policy-only rotations","Upstream concurrency queue, reordered IDs, early-response upload/reset variants","Capacity exhaustion, invalid authority, stranded trailers and fatal parser variants","Non-idempotent requests and other client libraries or versions",@@ -76,6 +87,17 @@ class Blocked(RuntimeError):"""An unavailable prerequisite; never a successful skip."""+def select_cases(requested, has_go_client):+ """Default to candidate coverage; historical Go diagnostics are explicit."""+ if requested is not None:+ return list(requested)+ return [*CASES, *(GO_EARLY_CASES if has_go_client else ())]+++def exit_code_for_status(status):+ return {"PASS": 0, "FAIL": 1, "BLOCKED": 2, "NOT_RUN": 2}[status]++def require(condition, message):if not condition:raise AssertionError(message)@@ -114,8 +136,8 @@ def exact_read(stream, count):return output-class H2Client:- """One verified SOCKS5/TLS connection, no retries and explicit flow credit."""+class ManualH2Peer:+ """Protocol stress peer; records GOAWAY but does not feed it to hyper-h2."""def __init__(self, proxy_port, ca_file):import h2.config@@ -126,6 +148,7 @@ class H2Client:self.closed = Falseself.eof = Falseself.terminated = False+ self.goaway_seen = Falseself.frames = []self.pending_frames = bytearray()self.bodies = {}@@ -169,12 +192,13 @@ class H2Client:if data:self.tls.sendall(data)- def request(self, stream_id, path="/v1/old"):- self.sent_requests.append({"stream_id": stream_id, "path": path, "logical_id": f"manual-{stream_id}"})+ def request(self, stream_id, path="/v1/old", logical_id=None):+ logical_id = logical_id or f"manual-{stream_id}"+ self.sent_requests.append({"stream_id": stream_id, "path": path, "logical_id": logical_id})self.h2.send_headers(stream_id, [(":method", "GET"), (":scheme", "https"),(":authority", BOUND), (":path", path),- ("x-h2-logical-id", f"manual-{stream_id}"),+ ("x-h2-logical-id", logical_id),], end_stream=True)self.flush()@@ -198,10 +222,12 @@ class H2Client:def _frames(self, data):from hyperframe.frame import Frame, GoAwayFrame, RstStreamFrameself.pending_frames.extend(data)+ forwarded = bytearray()while len(self.pending_frames) >= 9:frame, length = Frame.parse_frame_header(memoryview(bytes(self.pending_frames[:9])))if len(self.pending_frames) < length + 9:- return+ break+ raw = bytes(self.pending_frames[:9 + length])frame.parse_body(memoryview(bytes(self.pending_frames[9:9 + length])))del self.pending_frames[:9 + length]item = {"sequence": len(self.frames) + 1, "type": type(frame).__name__,@@ -212,7 +238,11 @@ class H2Client:item["error_code"] = int(frame.error_code)if isinstance(frame, GoAwayFrame):item["last_stream_id"] = frame.last_stream_id+ self.goaway_seen = Trueself.frames.append(item)+ if not isinstance(frame, GoAwayFrame):+ forwarded.extend(raw)+ return bytes(forwarded)def poll(self):import h2.events@@ -228,9 +258,8 @@ class H2Client:self.eof = Truerequire(not self.pending_frames, "EOF split a received H2 frame")return- self._frames(data)- events = self.h2.receive_data(data)- self.terminated = self.terminated or any(isinstance(event, h2.events.ConnectionTerminated) for event in events)+ protocol_data = self._frames(data)+ events = self.h2.receive_data(protocol_data) if protocol_data else []for event in events:if isinstance(event, h2.events.DataReceived):self.bodies.setdefault(event.stream_id, bytearray()).extend(event.data)@@ -244,6 +273,8 @@ class H2Client:self.ended.add(event.stream_id)elif isinstance(event, h2.events.StreamReset):self.resets[event.stream_id] = int(event.error_code)+ elif isinstance(event, h2.events.ConnectionTerminated):+ self.terminated = Trueif not self.terminated:self.flush()@@ -262,7 +293,9 @@ class H2Client:self.tls.close()def evidence(self):- return {"eof": self.eof, "frames": self.frames, "requests": self.sent_requests,+ return {"eof": self.eof, "goaway_observed_but_withheld_from_hyper_h2": self.goaway_seen,+ "connection_terminated_in_hyper_h2": self.terminated,+ "frames": self.frames, "requests": self.sent_requests,"ended": sorted(self.ended), "received_resets": self.resets,"sent_resets": sorted(self.locally_reset),"bodies": {str(sid): {"bytes": len(body), "sha256": hashlib.sha256(body).hexdigest()}@@ -348,7 +381,7 @@ class H1Handler(http.server.BaseHTTPRequestHandler):path, token = self.server.record(headers, "http/1.1")body, fields = self.server.response(path, token)self.send_response(200)- if self.server.scenario == "go_retry" and path == "/v1/old":+ if self.server.scenario in ("go_retry", "go_retry_early_notice") and path == "/v1/old":self.send_header("content-type", "text/event-stream")self.send_header("transfer-encoding", "chunked")self.end_headers()@@ -427,7 +460,14 @@ class OriginHandler(socketserver.BaseRequestHandler):path, token = self.server.record(dict(event.headers), "h2", event.stream_id)body, fields = self.server.response(path, token)conn.send_headers(event.stream_id, [(":status", "200"), *fields])- pending[event.stream_id] = [body, 0]+ hold_old = (self.server.scenario in ("go_retry", "go_retry_early_notice")+ and path == "/v1/old")+ offset = 0+ if hold_old:+ prefix = H2_HELD_PREFIX+ conn.send_data(event.stream_id, prefix, end_stream=False)+ offset = len(prefix)+ pending[event.stream_id] = [body, offset, hold_old]elif isinstance(event, h2.events.DataReceived):conn.acknowledge_received_data(event.flow_controlled_length, event.stream_id)elif isinstance(event, h2.events.StreamReset):@@ -435,7 +475,10 @@ class OriginHandler(socketserver.BaseRequestHandler):elif isinstance(event, h2.events.ConnectionTerminated):returnfor sid, pair in list(pending.items()):- body, offset = pair+ body, offset, held = pair+ if held and not self.server.release.is_set():+ continue+ pair[2] = Falsetry:while offset < len(body):size = min(conn.local_flow_control_window(sid), conn.max_outbound_frame_size, len(body) - offset)@@ -480,6 +523,7 @@ class RuntimeHarness:self.ipc.socket_path = str(Path(self.socket_directory.name) / "receiver.sock")self.socket_path = self.ipc.socket_pathself.events_path = root / "proxy-events.jsonl"+ self.barrier_release_path = root / "release-pre-admission"generate_origin_certificate(root)self.protocol = protocolself.scenario = scenario@@ -501,7 +545,8 @@ class RuntimeHarness:or key.startswith("OPENSANDBOX_EGRESS_REVISION_")or key == "OPENSANDBOX_EGRESS_MITMPROXY_PASSTHROUGH_HOSTS")}environment.update(H2_PRODUCTION_OBSERVATIONS=str(self.root),- H2_PRODUCTION_SCENARIO=self.scenario)+ H2_PRODUCTION_SCENARIO=self.scenario,+ H2_PRODUCTION_BARRIER_RELEASE=str(self.barrier_release_path))executable = str(Path(sys.executable).parent / "mitmdump")args = [executable, "--mode", "socks5", "--listen-host", "127.0.0.1","--listen-port", str(self.proxy_port),@@ -581,7 +626,7 @@ class RuntimeHarness:self.ipc_events.append({"operation": "invalid_token_rejected", "status": 401})def connect(self):- client = H2Client(self.proxy_port, self.root / "ca/mitmproxy-ca-cert.pem")+ client = ManualH2Peer(self.proxy_port, self.root / "ca/mitmproxy-ca-cert.pem")self.clients.append(client)return client@@ -694,17 +739,33 @@ def assert_no_goaway(client):require(not any(frame["type"] == "GoAwayFrame" for frame in client.frames), "GOAWAY arrived before old response completion")+def assert_early_goaway(client, frozen):+ client.until(+ lambda: any(frame["type"] == "GoAwayFrame" for frame in client.frames),+ message="fenced H2 connection did not deliver its early GOAWAY notice",+ )+ goaways = [frame for frame in client.frames if frame["type"] == "GoAwayFrame"]+ require(len(goaways) == 1, "fence queued more than one early GOAWAY notice")+ require(goaways[0]["error_code"] == 0+ and goaways[0]["last_stream_id"] == frozen,+ "early GOAWAY did not preserve frozen client stream bound")+ require(client.goaway_seen and not client.terminated,+ "manual peer did not keep GOAWAY outside hyper-h2 protocol state")++def assert_graceful(client, stream_ids, frozen):client.credit()client.until(lambda: client.eof, timeout=10, message="drained connection did not reach EOF")goaways = [frame for frame in client.frames if frame["type"] == "GoAwayFrame"]- require(len(goaways) == 1, "normal drain did not deliver exactly one GOAWAY")- goaway = goaways[0]- require(goaway["error_code"] == 0 and goaway["last_stream_id"] == frozen, "GOAWAY did not retain frozen client stream bound")+ require(len(goaways) == 2, "held-stream retirement did not deliver one notice and one final GOAWAY")+ notice, final = goaways+ require(all(frame["error_code"] == 0 and frame["last_stream_id"] == frozen+ for frame in goaways), "GOAWAY did not retain the frozen client stream bound")for sid in stream_ids:require(bytes(client.bodies[sid]) == BODY and sid in client.ended, "old response body/END_STREAM was incomplete")endings = [frame for frame in client.frames if frame["stream_id"] == sid and frame["end_stream"]]- require(len(endings) == 1 and endings[0]["sequence"] < goaway["sequence"], "old END_STREAM did not precede unique GOAWAY")+ require(len(endings) == 1 and notice["sequence"] < endings[0]["sequence"] < final["sequence"],+ "old END_STREAM was not retained between the early notice and final GOAWAY")def assert_refused(client, sid):@@ -737,10 +798,12 @@ def case_removal_drain(harness):row = harness.wait_connection(lambda state: state["fenced"])require(row["frozen"] == 3 and len(row["active"]) == 2, "ACK fence/high-water or retained old handles missing")require(row["deadline"] is not None, "removal failed to retain a drain deadline")+ assert_early_goaway(client, 3)for sid in (5, 7):client.request(sid, "/v1/new")assert_refused(client, sid)- assert_no_goaway(client)+ require(len([frame for frame in client.frames if frame["type"] == "GoAwayFrame"]) == 1,+ "repeated maintenance queued another early GOAWAY")require(not any(record["path"] == "/v1/new" for record in harness.origin.records), "refused request reached origin")assert_graceful(client, [1, 3], 3)for path in ("/v1/old", "/v1/old-two"):@@ -783,6 +846,7 @@ def case_remove_readd(harness):harness.backpressured(client)harness.install(2, None)first = harness.wait_connection(lambda state: state["fenced"])+ assert_early_goaway(client, 1)harness.install(3, SECRET_TWO)client.request(3, "/v1/new")assert_refused(client, 3)@@ -790,7 +854,8 @@ def case_remove_readd(harness):require(current["fenced"] and current["frozen"] == first["frozen"] == 1and current["deadline"] == first["deadline"], "remove/readd reopened or extended the old connection")require(not any(record["path"] == "/v1/new" for record in harness.origin.records), "readded old connection reached origin")- assert_no_goaway(client)+ require(len([frame for frame in client.frames if frame["type"] == "GoAwayFrame"]) == 1,+ "remove/readd queued a duplicate retirement notice")assert_graceful(client, [1], 1)harness.expect_clean_runtime()@@ -805,7 +870,7 @@ def case_reset_isolation(harness):require(harness.state()["registry_requests"] == 1, "reset released the other active stream")harness.install(2, None)harness.wait_connection(lambda state: state["fenced"] and state["frozen"] == 3)- assert_no_goaway(client)+ assert_early_goaway(client, 3)assert_graceful(client, [3], 3)harness.expect_clean_runtime()@@ -819,6 +884,70 @@ def case_disconnect(harness):require(harness.process.poll() is None, "normal disconnect incorrectly exited the child")+def case_pre_admission_barrier(harness):+ """Exercise real IPC/hook barrier for stream 3 above frozen bound 1."""+ harness.install(1)+ client = harness.connect()+ row = harness.backpressured(client)+ assert_attached(row)+ require([item["stream_id"] for item in row["active"]] == [1],+ "production barrier precondition must have only admitted stream 1")+ client.request(3, "/v1/new", logical_id="new")+ entered = wait_for(+ lambda: next((item for item in harness.events()+ if item.get("kind") == "pre_admission_barrier_entered"), None),+ timeout=10, message="real requestheaders path did not pause before registry admission")+ require(entered.get("stream_id") == 3 and entered.get("before_registry_acquire") is True,+ "barrier did not pause the real stream before LiveRuntime.acquire_request")++ harness.install(2, None, retry=True)+ fenced = harness.wait_connection(lambda state: state["fenced"])+ require(fenced["frozen"] == 1,+ "real IPC barrier scope must freeze the admitted stream-1 high-water mark")+ require(harness.final_ack_ns > 0 and harness.ipc.active() is not None,+ "authenticated Unix IPC fence was not acknowledged before release")+ harness.barrier_release_path.write_text("release\n")++ admission = wait_for(+ lambda: next((item for item in harness.events()+ if item.get("kind") == "pre_admission_result"), None),+ timeout=10, message="paused real admission did not resume after the fence ACK")+ require(admission.get("action") == "deny"+ and admission.get("reason") == "connection_fenced"+ and admission.get("handle_created") is False+ and admission.get("fenced") is True,+ "Registry did not deny the request that resumed after fence publication")+ client.until(lambda: 3 in client.resets,+ message="post-ACK pre-admission request did not receive REFUSED_STREAM")+ require(client.resets[3] == 7 and 3 not in client.headers,+ "pre-admission denial emitted an HTTP response instead of REFUSED_STREAM")+ hook = wait_for(+ lambda: next((item for item in harness.events()+ if item.get("kind") == "requestheaders" and item.get("logical_id") == "new"), None),+ timeout=5, message="post-hook observer did not record the denied request")+ require(hook.get("attached") is True and hook.get("admitted") is False+ and hook.get("fenced") is True and hook.get("private_token_present") is False,+ "denied request acquired a handle or received credential injection")+ harness.wait_connection(lambda value: value["fenced"]+ and [item["stream_id"] for item in value["active"]] == [1])+ registry = wait_for(lambda: (state := harness.state())+ if state.get("registry_requests") == 1 else None,+ timeout=5, message="rejected stream changed the Registry request count")+ require(registry["registry_requests"] == 1,+ "the rejected stream changed the Registry request count or released the old request")+ require(harness.evidence()["origin_attempt_counts"].get("new", 0) == 0,+ "post-ACK pre-admission request reached the origin")+ assert_credential(harness, "/v1/old", "one")++ client.credit()+ client.until(lambda: 1 in client.ended, message="already-admitted old response did not finish")+ require(bytes(client.bodies[1]) == BODY,+ "already-admitted old response did not retain its original body")+ harness.wait_connection(lambda value: not value["active"])+ client.close()+ harness.expect_clean_runtime()++def case_expiry(harness):harness.install(1)client = harness.connect()@@ -882,12 +1011,11 @@ def case_go_retry(harness, binary, required_version, *, bounded_old=False):cleanup_observation = {"attempted": False, "completed": False,"scope": "after Go client process exit, before harness teardown","client_driven": True, "proxy_first_goaway_proven": False}- stderr = (harness.root / "go-stderr.log").open("w")process = subprocess.Popen([str(binary), "-socks-address", f"127.0.0.1:{harness.proxy_port}","-ca-file", str(harness.combined_ca), "-timeout", "25s", "-new-timeout", "8s","-require-go-version", required_version],- stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=stderr, text=True,+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True,env={key: value for key, value in os.environ.items() if key != "SSLKEYLOGFILE"})def read_output():@@ -994,7 +1122,6 @@ def case_go_retry(harness, binary, required_version, *, bounded_old=False):process.stdin.close()process.stdout.close()reader.join(timeout=5)- stderr.close()(harness.root / "go-events.jsonl").write_text("".join(json.dumps(item, sort_keys=True) + "\n" for item in events))harness.go_evidence = {"required_go_version": required_version,"actual_go_version": None if event("ready") is None else event("ready").get("go_version"),@@ -1014,6 +1141,176 @@ def case_go_retry(harness, binary, required_version, *, bounded_old=False):"proxy_replay": False, "non_idempotent_guarantee": False}+def case_go_early_notice(harness, binary, required_version):+ """Require Go to read GOAWAY before issuing its one fresh-connection GET."""+ harness.install(1)+ events = []+ cleanup_observation = {"attempted": False, "completed": False,+ "scope": "after Go client process exit, before harness teardown",+ "client_driven": True, "proxy_first_goaway_proven": False}+ child_env = {key: value for key, value in os.environ.items()+ if key not in ("SSLKEYLOGFILE", "GODEBUG")}+ child_env["GODEBUG"] = "http2debug=2"+ process = subprocess.Popen(+ [str(binary), "-socks-address", f"127.0.0.1:{harness.proxy_port}",+ "-ca-file", str(harness.combined_ca), "-timeout", "25s", "-new-timeout", "8s",+ "-require-go-version", required_version, "-observe-goaway-parse"],+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True,+ env=child_env)++ def read_output():+ for line in process.stdout:+ try:+ item = json.loads(line)+ if item.get("event") == "goaway_frame_parsed":+ item["harness_received_ns"] = time.monotonic_ns()+ events.append(item)+ except json.JSONDecodeError:+ events.append({"event": "invalid_json"})++ reader = threading.Thread(target=read_output, name="h2-p2-go-early-output")+ reader.start()++ def event(name):+ return next((item for item in events if item.get("event") == name), None)++ try:+ ready = wait_for(lambda: event("ready") or event("fatal"), timeout=5,+ message="Go fixture did not announce exact version/config")+ require(ready.get("event") == "ready" and ready.get("go_version") == required_version+ and ready.get("required_go_version") == required_version,+ "Go fixture runtime differs from independently expected version")+ old_headers = wait_for(lambda: event("old_headers"), timeout=10,+ message="Go old GET did not receive streaming H2 headers")+ require(old_headers["proto"] == "HTTP/2.0" and old_headers["tls_verified"],+ "Go old request was not verified H2")+ row = harness.wait_connection(lambda state: bool(state["active"]))+ assert_attached(row)+ old_stream = row["active"][0]["stream_id"]+ old_connection_serial = row["connection_serial"]+ harness.install(2, None, retry=True)+ harness.wait_connection(lambda state: state["fenced"] and state["frozen"] == old_stream)++ notice = wait_for(+ lambda: next((item for item in events+ if item.get("event") == "goaway_frame_parsed"+ and item.get("last_stream_id") == old_stream+ and item.get("error_code") == 0), None),+ timeout=8, message="Go did not parse the production early GOAWAY through its standard H2 Framer")+ require(notice.get("harness_received_ns", 0) >= harness.final_ack_ns,+ "Go parsed the notice before the authenticated fence ACK")+ before_new = [item for item in events+ if item.get("event") == "got_conn"]+ require(len(before_new) == 1 and before_new[0].get("logical_id") == "old"+ and not any(item.get("logical_id") == "new" for item in before_new),+ "GOAWAY parser event did not arrive while old was the only successful connection")+ require(sum(item.get("event") == "goaway_frame_parsed" for item in events) == 1,+ "unexpected repeated GOAWAY parse events arrived before the new Do")+ require(not event("old_result"), "old Go response completed before the new GET was released")+ command_ns = time.monotonic_ns()+ process.stdin.write('{"command":"continue_new"}\n')+ process.stdin.flush()++ new = wait_for(lambda: event("new_result"), timeout=12,+ message="Go new GET did not reach its bounded result after GOAWAY")+ old_was_held = event("old_result") is None+ require(old_was_held, "old Go response completed before the new result")+ harness.origin.release.set()+ release_ns = time.monotonic_ns()+ old = wait_for(lambda: event("old_result"), timeout=8,+ message="Go old response did not complete after origin release")+ wait_for(lambda: event("done"), timeout=5, message="Go fixture did not finish")+ code = process.wait(timeout=5)+ reader.join(timeout=5)+ require(not reader.is_alive(), "Go stdout reader was not reaped")++ cleanup_observation.update(attempted=True, go_exit_code=code, normal_exit=code == 0,+ started_ns=time.monotonic_ns())+ harness.cleanup["go_cleanup_origin"] = "client_driven_after_Go_process_exit_before_harness_teardown"+ harness.cleanup["proxy_first_goaway_proven"] = False+ harness.expect_clean_runtime()+ cleanup_observation.update(completed=True, finished_ns=time.monotonic_ns())+ require(harness.process.poll() is None, "Go normal cleanup unexpectedly exited mitmdump")+ require(old["do_calls"] == new["do_calls"] == 1,+ "Go fixture made application-level replay calls")+ require(old["body_complete"] and old["status"] == 200 and not old["error_class"],+ "Go Transport did not preserve the old response through migration")+ if harness.protocol == "h1":+ expected_old = H1_HELD_PREFIX+ expected_new_proto = "HTTP/1.1"+ else:+ expected_old = BODY+ expected_new_proto = "HTTP/2.0"+ require(old["body_sha256"] == hashlib.sha256(expected_old).hexdigest(),+ "held old response body hash differs from the selected origin protocol")+ require(new["status"] == 200 and new["body_complete"] and not new["error_class"],+ "Go fresh-connection GET did not complete successfully")+ require(new["proto"] == expected_new_proto,+ "final fresh connection negotiated a different origin protocol")+ require(new["body_sha256"] == hashlib.sha256(SMALL_BODY).hexdigest(),+ "Go fresh-connection response body hash differs")+ old_connections = set(old["connection_ids"])+ require(new["connection_ids"] and new["connection_ids"][-1] not in old_connections,+ "the single logical Do did not ultimately complete on a fresh connection")+ fence_refusals = [item for item in harness.events()+ if item.get("kind") == "requestheaders" and item.get("logical_id") == "new"+ and item.get("admitted") is False and item.get("fenced") is True]+ counts = harness.evidence()["origin_attempt_counts"]+ require(counts.get("old") == counts.get("new") == 1,+ "origin did not process each logical GET exactly once")+ assert_credential(harness, "/v1/old", "one")+ assert_credential(harness, "/v1/new", "missing")+ harness.go_evidence = {+ "required_go_version": required_version,+ "actual_go_version": ready.get("go_version"),+ "exit_code": code, "exact_child_reaped": True, "reader_joined": True,+ "events": events, "observed_goaway_parse_before_command": notice,+ "command_ns": command_ns, "release_ns": release_ns,+ "new_old_connection_fence_events": len(fence_refusals),+ "goaway_evidence": "standard_library_parser_event_not_wire_capture",+ "old_unfinished_when_new_completed": old_was_held,+ "origin_attempt_counts": counts,+ "origin_protocol": harness.protocol,+ "client_protocols": {"old": old["proto"], "new": new["proto"]},+ "logical_do_calls_required": 1, "manual_retries": 0,+ "cleanup_observation": cleanup_observation,+ "scope": GO_SCOPE["go_early_notice_h1_origin" if harness.protocol == "h1"+ else "go_early_notice_h2_origin"],+ "proxy_replay": False, "non_idempotent_guarantee": False,+ }+ finally:+ harness.origin.release.set()+ forced_go_cleanup = process.poll() is None+ if forced_go_cleanup:+ process.terminate()+ try:+ process.wait(timeout=5)+ except subprocess.TimeoutExpired:+ process.kill()+ process.wait(timeout=5)+ if process.stdin is not None:+ process.stdin.close()+ if process.stdout is not None:+ process.stdout.close()+ reader.join(timeout=5)+ (harness.root / "go-events.jsonl").write_text(+ "".join(json.dumps(item, sort_keys=True) + "\n" for item in events))+ if not hasattr(harness, "go_evidence"):+ harness.go_evidence = {+ "required_go_version": required_version,+ "actual_go_version": None if event("ready") is None else event("ready").get("go_version"),+ "exit_code": process.returncode,+ "exact_child_reaped": process.returncode is not None,+ "reader_joined": not reader.is_alive(), "events": events,+ "origin_protocol": harness.protocol,+ "cleanup_observation": cleanup_observation,+ "forced_go_cleanup": forced_go_cleanup,+ "scope": GO_SCOPE["go_early_notice_h1_origin" if harness.protocol == "h1"+ else "go_early_notice_h2_origin"],+ "proxy_replay": False, "non_idempotent_guarantee": False,+ }++def fingerprint():def git(*args):return subprocess.check_output(["git", *args], cwd=REPO, text=True).strip()@@ -1051,8 +1348,12 @@ def preflight(metadata):def run_case(name, artifacts, args):- protocol = "h1" if name.endswith("_h1") or name in GO_CASES else "h2"- scenario = "d3_pending_server_connected" if name == "d3_pending_writer" else "go_retry" if name in GO_CASES else "normal"+ protocol = ("h1" if name.endswith("_h1") or name in GO_DELAYED_CASES+ or name == "go_early_notice_h1_origin" else "h2")+ scenario = ("d3_pending_server_connected" if name == "d3_pending_writer"+ else "pre_admission_barrier_h2" if name == "pre_admission_barrier_h2"+ else "go_retry_early_notice" if name in GO_EARLY_CASES+ else "go_retry" if name in GO_DELAYED_CASES else "normal")seconds = 30 if name == "expiry_default_30s_h2" else 2 if name == "expiry_short_h2" else 20 if name in GO_CASES else 8harness = Noneresult = {"case": name, "status": "FAIL", "started_ns": time.monotonic_ns()}@@ -1063,9 +1364,11 @@ def run_case(name, artifacts, args):case_expiry(harness)elif name == "d3_pending_writer":case_d3(harness)- elif name in GO_CASES:+ elif name in GO_DELAYED_CASES:case_go_retry(harness, args.go_client, args.require_go_version,bounded_old=name == "go_retry_bounded_old")+ elif name in GO_EARLY_CASES:+ case_go_early_notice(harness, args.go_client, args.require_go_version)else:globals()["case_" + name.rsplit("_", 1)[0]](harness)require(name == "d3_pending_writer" or harness.process.poll() is None,@@ -1094,9 +1397,16 @@ def run_case(name, artifacts, args):def summarize_results(cases, selected):- """Keep the manual, held-old and bounded-old evidence denominators distinct."""- groups = {"manual_peer": CASES, "go_held_old": ("go_retry",),- "go_bounded_old": ("go_retry_bounded_old",)}+ """Keep manual stress, real admission, delayed and GOAWAY cases distinct."""+ groups = {+ "manual_peer_late_goaway_stress": MANUAL_PEER_CASES,+ "real_live_runtime_pre_admission_barrier": REAL_RUNTIME_CASES,+ "candidate_acceptance": (*REAL_RUNTIME_CASES, *GO_EARLY_CASES),+ "go_delayed_held_old_diagnostic_not_candidate_gate": ("go_retry",),+ "go_bounded_old_diagnostic_not_candidate_gate": ("go_retry_bounded_old",),+ "go_early_notice_h1_origin": ("go_early_notice_h1_origin",),+ "go_early_notice_h2_origin": ("go_early_notice_h2_origin",),+ }vocabulary = ("PASS", "FAIL", "BLOCKED", "NOT_RUN")def group(names):@@ -1123,7 +1433,12 @@ def main(argv=None):args = parser.parse_args(argv)if args.list:print(json.dumps({"status": "NOT_RUN", "manual_denominator": CASES,- "optional_client_cases": GO_CASES, "go_scope": GO_SCOPE,+ "manual_peer_stress_denominator": MANUAL_PEER_CASES,+ "real_runtime_cases": REAL_RUNTIME_CASES,+ "optional_client_cases": GO_CASES,+ "default_go_candidate_cases": GO_EARLY_CASES,+ "explicit_go_diagnostic_cases": GO_DELAYED_CASES,+ "go_scope": GO_SCOPE,"missing_matrix": MISSING}, indent=2))return 0if not args.run_real_runtime:@@ -1135,7 +1450,7 @@ def main(argv=None):parser.error("--go-client and --require-go-version must be supplied together")if args.go_client and not args.go_client.is_file():parser.error("--go-client must name an existing previously built fixture")- selected = args.cases or [*CASES, *(GO_CASES if args.go_client else [])]+ selected = select_cases(args.cases, bool(args.go_client))if any(name in GO_CASES for name in selected) and not args.go_client:parser.error("Go cases require --go-client and independently verified --require-go-version")if len(set(selected)) != len(selected):@@ -1143,7 +1458,9 @@ def main(argv=None):args.artifacts = args.artifacts.resolve()args.artifacts.mkdir(mode=0o700, parents=False, exist_ok=False)report = {"scope": "P2 production addon / loopback SOCKS5 TLS / authenticated Unix IPC only",- "manual_denominator": CASES, "optional_client_cases": GO_CASES, "go_scope": GO_SCOPE,+ "manual_denominator": CASES, "manual_peer_stress_denominator": MANUAL_PEER_CASES,+ "real_runtime_cases": REAL_RUNTIME_CASES,+ "optional_client_cases": GO_CASES, "go_scope": GO_SCOPE,"missing_matrix": MISSING,"selected": selected, "cases": [{"case": name, "status": "NOT_RUN"} for name in [*CASES, *GO_CASES]],"status": "NOT_RUN", "fingerprint": fingerprint()}@@ -1179,7 +1496,7 @@ def main(argv=None):save()print(json.dumps({"status": report["status"], "summary": report["summary"],"denominator_summaries": report["denominator_summaries"], "report": str(destination), "missing_matrix_count": len(MISSING)}), flush=True)- return {"PASS": 0, "FAIL": 1, "BLOCKED": 2, "NOT_RUN": 2}[report["status"]]+ return exit_code_for_status(report["status"])if __name__ == "__main__":diff --git a/components/egress/tests/test_credential_bound_h2_runtime_selection.py b/components/egress/tests/test_credential_bound_h2_runtime_selection.pynew file mode 100644index 000000000..f7e3b5cb3--- /dev/null+++ b/components/egress/tests/test_credential_bound_h2_runtime_selection.py@@ -0,0 +1,86 @@+# Copyright 2026 The OpenSandbox Authors+# SPDX-License-Identifier: Apache-2.0++"""Pure case-selection and denominator checks for the opt-in runtime runner."""++import unittest++import test_credential_bound_h2_runtime as runner+++def initial_cases(overrides=None):+ overrides = overrides or {}+ return [+ {"case": name, "status": overrides.get(name, "NOT_RUN")}+ for name in [*runner.CASES, *runner.GO_CASES]+ ]+++class RuntimeSelectionTests(unittest.TestCase):+ def test_no_go_client_keeps_existing_runtime_denominator(self):+ self.assertEqual(runner.select_cases(None, False), runner.CASES)++ def test_go_client_default_selects_candidate_cases_only(self):+ selected = runner.select_cases(None, True)+ self.assertEqual(selected, [*runner.CASES, *runner.GO_EARLY_CASES])+ self.assertTrue(set(runner.GO_EARLY_CASES).issubset(selected))+ self.assertFalse(set(runner.GO_DELAYED_CASES).intersection(selected))++ def test_historical_diagnostic_failure_does_not_contaminate_default_gate(self):+ selected = runner.select_cases(None, True)+ statuses = {name: "PASS" for name in selected}+ statuses.update({name: "FAIL" for name in runner.GO_DELAYED_CASES})+ summary = runner.summarize_results(initial_cases(statuses), selected)++ self.assertEqual(summary["status"], "PASS")+ self.assertEqual(+ summary["denominator_summaries"]["candidate_acceptance"]["status"],+ "PASS",+ )+ self.assertEqual(+ summary["denominator_summaries"][+ "go_delayed_held_old_diagnostic_not_candidate_gate"+ ]["status"],+ "FAIL",+ )+ self.assertEqual(+ summary["denominator_summaries"][+ "go_bounded_old_diagnostic_not_candidate_gate"+ ]["status"],+ "FAIL",+ )++ def test_explicit_historical_diagnostic_retains_its_own_failure(self):+ selected = runner.select_cases(["go_retry"], True)+ summary = runner.summarize_results(+ initial_cases({"go_retry": "FAIL"}), selected+ )++ self.assertEqual(summary["status"], "FAIL")+ self.assertEqual(+ summary["denominator_summaries"]["candidate_acceptance"]["status"],+ "NOT_RUN",+ )+ self.assertEqual(+ summary["denominator_summaries"][+ "go_delayed_held_old_diagnostic_not_candidate_gate"+ ]["status"],+ "FAIL",+ )++ def test_early_notice_failure_keeps_candidate_gate_nonzero(self):+ selected = runner.select_cases(None, True)+ statuses = {name: "PASS" for name in selected}+ statuses["go_early_notice_h2_origin"] = "FAIL"+ summary = runner.summarize_results(initial_cases(statuses), selected)++ self.assertEqual(summary["status"], "FAIL")+ self.assertEqual(+ summary["denominator_summaries"]["candidate_acceptance"]["status"],+ "FAIL",+ )+ self.assertEqual(runner.exit_code_for_status(summary["status"]), 1)+++if __name__ == "__main__":+ unittest.main()diff --git a/components/egress/tests/test_credential_bound_h2_wakeup.py b/components/egress/tests/test_credential_bound_h2_wakeup.pyindex 7e5a50c9f..a2bd403e6 100644--- a/components/egress/tests/test_credential_bound_h2_wakeup.py+++ b/components/egress/tests/test_credential_bound_h2_wakeup.py@@ -166,7 +166,14 @@ class WakeupTests(unittest.TestCase):state = self.runtime.registry.drain_state(self.adapter.token)self.assertTrue(state.fenced)self.assertIsNotNone(state.deadline)- self.assertFalse(self.adapter.maintenance_needed())+ self.assertTrue(self.adapter.maintenance_needed())+ first_deadline = state.deadline+ self.advance()+ self.assertTrue(self.adapter.retirement_notice_sent)+ self.assertFalse(self.adapter.goaway_sent)+ self.assertEqual(self.runtime.registry.drain_state(self.adapter.token).deadline,+ first_deadline)+ self.assertEqual(self.runtime.registry.request_count, 1)self.assert_timeout_and_release()def test_retirement_deadline_closes_held_stream_without_network(self):diff --git a/docs/architecture/network/osep-0023-h2-integration.md b/docs/architecture/network/osep-0023-h2-integration.mdindex 63c13eeae..c69558d94 100644--- a/docs/architecture/network/osep-0023-h2-integration.md+++ b/docs/architecture/network/osep-0023-h2-integration.md@@ -25,10 +25,11 @@ Before credential injection, the adapter validates the original pseudo-headers:1. Before mutation acknowledgement, publication atomically installs the snapshot, new-request fence, frozen stream bound and first retirement deadlines. Acknowledgement does not wait for drain.2. Valid requests refused at `requestheaders` receive REFUSED_STREAM and do not reach the origin. Previously admitted requests retain their own snapshots.-3. Once old requests and their local downstream/upstream output obligations are terminal, ordinary h2 APIs send one NO_ERROR GOAWAY with the frozen bound, followed by close.-4. Deadline expiry force-closes the whole connection and can interrupt other streams after rotation of a still-covered host. This is abnormal closure, not graceful completion.+3. If old requests or their local downstream/upstream output obligations remain, H2-A5 queues one advisory NO_ERROR GOAWAY as a `GoAwayFrame` through that `BoundH2Server` instance's `BufferedH2Connection._prepare_for_sending` and its existing output buffer. It does not call `close_connection` or send h2's `SEND_GOAWAY` input for this notice, so the pinned state machine and old streams remain live. Once old obligations are terminal, ordinary h2 close emits the final GOAWAY and closes the connection. An already-drained connection uses the final GOAWAY as its only notice.+4. At the same instance serialization boundary, every GOAWAY is limited to the minimum of its proposed last-stream ID, the frozen Registry bound when available, and the smallest bound previously serialized on that connection. Public close, internal `_terminate_connection` and protocol-error GOAWAYs retain their error codes; the adapter does not alter `highest_inbound_stream_id`. The local send buffer and ACK do not prove the peer received either notice.+5. Deadline expiry force-closes the whole connection and can interrupt other streams after rotation of a still-covered host. This is abnormal closure, not graceful completion.-Maintenance polling uses the exact connection handler's registered wakeup task. No-op housekeeping is filtered before dependency event dispatch, so it does not continually refresh the network inactivity watchdog. Actionable drain and cleanup work still runs without requiring new client traffic. The integration does not replace dependency classes or globally patch task scheduling.+Maintenance polling uses the exact connection handler's registered wakeup task. A fenced connection whose notice has not yet been queued is actionable work, even while an old stream is held and no new client traffic arrives. After one notice, idle polling is filtered before dependency event dispatch, so it does not continually refresh the network inactivity watchdog or reset the first deadline. Actionable drain and cleanup work still runs without new client traffic. The integration does not replace dependency classes or globally patch task scheduling.Response/error hooks alone do not release a handle: DATA, trailers, queued upstream requests and unfinished uploads remain obligations. The adapter observes actual upstream children and client/upstream stream mappings without rewriting their queues or forcing H1 upstreams. Reset-stranded trailers wait only for an existing first deadline, or close abnormally if none exists. An early response followed by reset during an unfinished upload can orphan upstream work; the client and verified owned upstreams then close immediately.@@ -36,12 +37,26 @@ Only the marked kill/NO_RESPONSE path represents normal refusal. Pre-hook 400/41### Client retry limitation-Delayed GOAWAY does not guarantee migration while an old response is unfinished. In the recorded Go 1.27.2 `net/http` synthetic GET configuration, retries after REFUSED_STREAM repeatedly selected the same connection and exhausted an eight-second deadline without reaching the origin. The held-old acceptance case remains **FAIL**.+The original `go_retry` diagnostic keeps the historical delayed/refusal shape: it requires the new GET to reach the old connection's post-ACK fence before the harness releases old. A previously recorded Go 1.27.2 run repeatedly selected that same connection after REFUSED_STREAM and exhausted an eight-second deadline without reaching the origin. This is useful historical evidence about delayed notice, but it is **not a candidate gate** once early GOAWAY is sent; standard `net/http.Transport` may have already retired the old connection before the new `Do` starts. The separate `go_retry_bounded_old` diagnostic also retains its old fence-dependent release rule and is not a candidate gate.-A separate bounded-old case passed only when the harness released the old response early. The new GET completed once on a fresh connection with **HTTP/1.1 200**, while the old HTTP/2 response completed intact. This does not prove H2-to-H2 migration, migration while old remains unfinished, or success for arbitrary finite response durations. Go-peer frames were not captured, so GOAWAY causality and wire ordering remain unproven. Neither outcome should be generalized to other client versions or configurations.+When a Go client fixture is supplied without explicit `--case` selection, the runtime runner defaults to the two early-notice H1/H2 candidate cases only. The historical `go_retry` and `go_retry_bounded_old` diagnostics require explicit selection and retain separate result groups, so their reuse-specific failure cannot contaminate the candidate gate.++A separate historical bounded-old case passed only when the harness released the old response early. The new GET completed once on a fresh connection with **HTTP/1.1 200**, while the old HTTP/2 response completed intact. This does not prove H2-to-H2 migration, migration while old remains unfinished, or success for arbitrary finite response durations.++The current runner adds `go_early_notice_h1_origin` and `go_early_notice_h2_origin`. Each keeps the old response unfinished, commits removal over authenticated Unix IPC, waits for Go 1.27.2's standard HTTP/2 Framer to parse the NO_ERROR GOAWAY, and only then starts the single new `Do`. The new `Do` may directly use a fresh connection or hit the old fence in the natural parser/transport scheduling race and be retried by the same standard Transport. Acceptance requires a final fresh-connection success, the old response still unfinished at that result, both origin logical IDs exactly once, and complete old body after release. The H2-origin case now holds the first body prefix while its H2 loop continues processing other streams and WINDOW_UPDATE. These new cases remain **NOT_RUN** on the current host; no interoperability result is claimed.++Their `goaway_frame_parsed` record comes from the exact Go 1.27.2 `http2debug=2` Framer read diagnostic after successful standard-library parsing. In this explicit opt-in mode the fixture filters raw diagnostics in memory to an allowlist, drops child stderr and never stores DATA, headers, debug bytes, pointers or raw log lines. This is a parser event, not packet capture, and it precedes `processGoAway`; the old request is the only successful Go connection before the new `Do`, which supplies a causal attribution window. `GotConn` alone is not treated as GOAWAY evidence.The proxy does not replay requests or change error codes to influence client pooling. The narrower passing case does not replace the failed acceptance gate.+### H2-A5 candidate evidence++The current isolated branch adds the early notice and per-instance monotonic GOAWAY clamp. On the exact pinned libraries, socket-free regressions observed the notice before an old response completed, retained buffered DATA and its request handle, then observed DATA/END_STREAM before the final GOAWAY. Additional cases cover ordinary close, `_terminate_connection`, protocol-error GOAWAY, a previously serialized lower bound, valid post-fence refusal, fragmented CONTINUATION headers, dynamic-table decoding and same-batch post-bound DATA accounting.++The socket-free layer suite uses a manual H2 peer that records but deliberately does not submit GOAWAY to hyper-h2's receive state machine. Its late-frame ordering tests are stress coverage only, not normal-client interoperability. The existing paused-hook stub is likewise not AC-07 acceptance. The opt-in `pre_admission_barrier_h2` runtime case pauses an actual `system.requestheaders` flow before the real `LiveRuntime.acquire_request`; stream 3 is above its frozen bound 1. It commits and ACKs the Registry fence over authenticated Unix IPC, resumes the call, and checks no handle, injection or origin request while an already-admitted old response still completes. Separately, a socket-free test uses the actual `LiveRuntime` and Registry: stream 3 is held before acquire, stream 5 obtains a real handle, revision commit freezes bound 5, then the actual late acquire for stream 3 is denied even though `3 <= 5`; this owner-level test does not simulate wire order, a `system.requestheaders` hook, or IPC. Together they cover the above-bound real-hook/ACK path and below-bound owner race without claiming one test covers both. The runtime barrier and Go cases are currently **NOT_RUN** because canonical loopback port 443 is unavailable without a host privilege change that is not authorized here.++Since that five-file snapshot, the focused Python H2 suite passed 83 tests, the Go 1.27.2 filtered-observer tests passed, and the candidate Go fixture built successfully; these are socket-free/unit results. The real Go H1/H2 held-old cases and Registry admission barrier remain **NOT_RUN** because the runner binds its synthetic origin only to `127.0.0.1:443`; this UID's observed `ip_unprivileged_port_start` is 1024 and an unprivileged bind returned EACCES. No service was stopped and no port fallback was used. The independent environment report records the separate temporary socket capability probe. The GitHub-hosted Go/H2 steps are wired but not yet executed. Current held-old and early-notice runtime results are therefore **NOT_RUN**, not PASS or a newly measured FAIL; candidate green, P2/P3 and H2-to-H2 acceptance remain unclaimed.+## Failure containmentClient disconnect fences admission immediately, but cleanup retains exact ownership until related upstreams are verified terminal. A disconnected hook, a CLOSED flag or an exit code alone does not establish transport closure. Cleanup observes actual task/IO identities and writer-close confirmation; callbacks must remain effective through canceled wakeups while handler registration remains valid.