Skip to content

The open-source app everyone uses to manage agents at work

refactor(adapter-utils): replace the process-wide byte ledger with route-local byte bounds (#12465)

Nicky Leach64b7dce0adfb
diff --git a/doc/observability.md b/doc/observability.md
index 8f0d193e29..5da3157eff 100644
--- a/doc/observability.md
+++ b/doc/observability.md
@@ -630,24 +630,6 @@ field.
| `sandbox_duplex_loss_total` | One terminal duplex channel loss. The `loss_class` dimension records the phase. |
| `sandbox_duplex_session_leak_total` | One leaked provider session at teardown. |
-### Aggregate byte ledger metrics
-
-The host aggregate byte ledger owns one process-scoped gauge and two
-process-scoped counters. The ledger bounds the retained bytes across every live
-duplex route in one process. It sets the gauge on each reserve and each release.
-It increments a counter on a rejected reservation and on an accounting defect.
-These records carry no dimension label. The guarded counter store keys each
-counter on `(companyId, metric)`, and the gauge reports one process value, so no
-dynamic dimension rides them. The code owner is
-`packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts`, and the metric
-names are literal constants in `duplex-observability.ts`.
-
-| Metric | Type | Scope |
-| --- | --- | --- |
-| `sandbox_duplex_aggregate_bytes_in_use` | gauge | The aggregate retained bytes across every live duplex route. The ledger sets it on each reserve and each release. |
-| `sandbox_duplex_aggregate_byte_reservation_rejections_total` | counter | One rejected aggregate byte reservation. The ledger increments it when a reservation would pass the aggregate ceiling. |
-| `sandbox_duplex_aggregate_byte_accounting_underflow_total` | counter | One aggregate byte accounting defect. The ledger increments it on a double release or on a transfer of a token it does not hold. |
-
### Dimension keys
Counters carry no dimension labels. The guarded counter store keys each counter
@@ -662,10 +644,25 @@ key never reaches a sink by accident.
| `provider` | string | no | `daytona`, or `other` for any other plugin key. |
| `transport` | string | no | `duplex`, `http2`, or `file`. `duplex` names the retired bespoke frame protocol; `http2` names the Node HTTP/2 session over the sandbox channel; a fallback record uses `file`. |
| `outcome` | string | yes | `ok` or `error`. |
-| `fallback_reason` | string | yes | `gate_off`, `capability_absent`, `route_busy`, `entrypoint_sync_failed`, `broker_construction_failed`, `channel_open_failed`, `ready_invalid`, `ready_nonce_mismatch`, `ready_timeout`, `contaminated`, `aggregate_bytes_exceeded`, or `preface_missing`. It rides only a fallback record. `route_busy` marks the process-scoped route ceiling full. `entrypoint_sync_failed` and `broker_construction_failed` mark the named build step. `channel_open_failed` marks a failed channel open. `aggregate_bytes_exceeded` marks a readiness handshake, or an `http2` post-preface pre-bind buffer, where the host fell back because the process aggregate byte ceiling had no room. `preface_missing` marks a missing or an invalid HTTP/2 client connection preface inside the bounded readiness buffer: the host found no valid preface after the accepted READY line, aborted the `http2` open, and moved the run to the file bridge (`queue_v1`) one time. |
+| `fallback_reason` | string | yes | `gate_off`, `capability_absent`, `route_busy`, `entrypoint_sync_failed`, `broker_construction_failed`, `channel_open_failed`, `ready_invalid`, `ready_nonce_mismatch`, `ready_timeout`, `contaminated`, or `preface_missing`. It rides only a fallback record. `route_busy` marks the process-scoped route ceiling full. `entrypoint_sync_failed` and `broker_construction_failed` mark the named build step. `channel_open_failed` marks a failed channel open. `preface_missing` marks a missing or an invalid HTTP/2 client connection preface inside the bounded readiness buffer: the host found no valid preface after the accepted READY line, aborted the `http2` open, and moved the run to the file bridge (`queue_v1`) one time. |
| `loss_class` | string | yes | `pre_dispatch` or `post_dispatch`, relative to the first request dispatch. It rides only a loss record. |
| `loss_reason` | string | yes | `stdin_eof`, `provider_exit`, `heartbeat_timeout`, `rpc_failure`, `write_error`, `transport_closed`, or `other`. The host maps every loss cause to one of these values, so no raw provider text reaches a sink. `write_error` marks a rejected host-to-sandbox write. `transport_closed` marks a reason-less provider transport close with no exit data. It rides only a loss record. |
To add a name or an enum value, extend the literal constant in
`duplex-observability.ts` first, then update the test that asserts the closed set.
+
+### Known behavior: aggregate retained body bytes
+
+The HTTP/2 bridge bounds retained body bytes for one route only. Each route
+holds up to 8,388,608 bytes (8 MiB) at its own peak (see
+`HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS` in `http2-bridge-server.ts`). The host
+process admits up to 128 concurrent routes (see
+`DEFAULT_MAX_CONCURRENT_DUPLEX_ROUTES` in `plugin-worker-manager.ts`). The
+process can therefore retain up to 1,073,741,824 bytes (1 GiB) of body data
+across every route at the same time.
+
+This is accepted, known behavior. The process tracks no aggregate byte
+ledger across routes: a per-route bound stops one busy route from starving
+another route's own budget, but the host enforces no smaller ceiling on the
+sum across every route.
Keep every dimension low-cardinality and free of user content.
diff --git a/packages/adapter-utils/src/duplex-aggregate-byte-ledger.test.ts b/packages/adapter-utils/src/duplex-aggregate-byte-ledger.test.ts
deleted file mode 100644
index 6711bf625b..0000000000
--- a/packages/adapter-utils/src/duplex-aggregate-byte-ledger.test.ts
+++ /dev/null
@@ -1,223 +0,0 @@
-import { describe, expect, it } from "vitest";
-import {
- DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES,
- DUPLEX_AGGREGATE_TOKEN_OWNERS,
- DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED,
- DuplexAggregateByteLedger,
- MIN_HOST_PROCESS_MEMORY_BYTES_FOR_DUPLEX,
- assertDuplexAggregateCeilingBytes,
- resolveDuplexAggregateCeilingBytes,
- type DuplexAggregateByteLedgerTelemetry,
-} from "./duplex-aggregate-byte-ledger.js";
-
-// A telemetry surface that counts each fixed record, so a test asserts the exact
-// gauge value and the exact counter counts.
-function createCountingTelemetry(): DuplexAggregateByteLedgerTelemetry & {
- gauge: number;
- rejections: number;
- underflows: number;
-} {
- const state = { gauge: 0, rejections: 0, underflows: 0 };
- return {
- ...state,
- setBytesInUse(bytes: number) {
- this.gauge = bytes;
- },
- recordReservationRejection() {
- this.rejections += 1;
- },
- recordAccountingUnderflow() {
- this.underflows += 1;
- },
- };
-}
-
-describe("DuplexAggregateByteLedger", () => {
- it("reserves under the ceiling and tracks the gauge and the live-token count", () => {
- const telemetry = createCountingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1000, telemetry });
-
- const a = ledger.reserve("buffered_chunk", 400);
- const b = ledger.reserve("buffered_chunk", 500);
-
- expect(a).not.toBeNull();
- expect(b).not.toBeNull();
- expect(ledger.bytesInUse).toBe(900);
- expect(ledger.liveTokenCount).toBe(2);
- expect(telemetry.gauge).toBe(900);
- expect(a?.bytes).toBe(400);
- expect(a?.owner).toBe("buffered_chunk");
- expect(a?.state).toBe("held");
- });
-
- it("rejects a one-byte-over reservation, retains nothing, and increments the rejection counter", () => {
- const telemetry = createCountingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1000, telemetry });
-
- ledger.reserve("buffered_chunk", 1000);
- const over = ledger.reserve("buffered_chunk", 1);
-
- expect(over).toBeNull();
- expect(ledger.bytesInUse).toBe(1000);
- expect(ledger.liveTokenCount).toBe(1);
- expect(telemetry.rejections).toBe(1);
- // The public rejection marker is the fixed string the caller reports.
- expect(DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED).toBe("DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED");
- });
-
- it("transfers ownership without a decrement or a re-reserve, so no admission gap opens", () => {
- const telemetry = createCountingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 100, telemetry });
-
- // Reserve the whole ceiling as a pre-bind event, then transfer to buffered.
- const token = ledger.reserve("pre_bind_event", 100);
- expect(token).not.toBeNull();
- // A second reservation cannot fit while the token stays held.
- expect(ledger.reserve("buffered_chunk", 1)).toBeNull();
-
- ledger.transfer(token!, "buffered_chunk");
-
- // The transfer changed only the owner. The gauge never moved, so no window
- // opened where the bytes were released and re-admitted.
- expect(ledger.bytesInUse).toBe(100);
- expect(ledger.liveTokenCount).toBe(1);
- expect(token?.owner).toBe("buffered_chunk");
- expect(telemetry.gauge).toBe(100);
- // No reservation fit during the transfer, so no rejection or underflow fired
- // beyond the one rejection above.
- expect(telemetry.rejections).toBe(1);
- expect(telemetry.underflows).toBe(0);
- });
-
- it("releases a token one time and decrements the gauge", () => {
- const telemetry = createCountingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1000, telemetry });
-
- const token = ledger.reserve("response_body", 250);
- expect(token).not.toBeNull();
- ledger.release(token!);
-
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(token?.state).toBe("released");
- expect(telemetry.gauge).toBe(0);
- expect(telemetry.underflows).toBe(0);
- });
-
- it("records the underflow counter on a second release and does not clamp the gauge again", () => {
- const telemetry = createCountingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1000, telemetry });
-
- const kept = ledger.reserve("request_frame", 300);
- const doubled = ledger.reserve("request_frame", 200);
- expect(kept).not.toBeNull();
- expect(doubled).not.toBeNull();
- ledger.release(doubled!);
- expect(ledger.bytesInUse).toBe(300);
-
- // A forced second release of the same token is an accounting defect.
- ledger.release(doubled!);
-
- expect(telemetry.underflows).toBe(1);
- // The gauge did not drop a second time, so the defect stays visible.
- expect(ledger.bytesInUse).toBe(300);
- expect(ledger.liveTokenCount).toBe(1);
- });
-
- it("records the underflow counter on a transfer of a released token", () => {
- const telemetry = createCountingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1000, telemetry });
- const token = ledger.reserve("pre_bind_event", 100);
- ledger.release(token!);
-
- ledger.transfer(token!, "buffered_chunk");
-
- expect(telemetry.underflows).toBe(1);
- expect(ledger.bytesInUse).toBe(0);
- });
-
- it("throws on a non-integer or negative reservation, so a programming error fails loud", () => {
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1000 });
- expect(() => ledger.reserve("buffered_chunk", -1)).toThrow();
- expect(() => ledger.reserve("buffered_chunk", 1.5)).toThrow();
- expect(() => ledger.reserve("buffered_chunk", Number.NaN)).toThrow();
- });
-
- it("admits a zero-byte reservation as a valid held token", () => {
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1000 });
- const token = ledger.reserve("seen_request_id", 0);
- expect(token).not.toBeNull();
- expect(token?.bytes).toBe(0);
- expect(ledger.liveTokenCount).toBe(1);
- });
-
- it("exposes the closed owner-label set", () => {
- expect([...DUPLEX_AGGREGATE_TOKEN_OWNERS]).toEqual([
- "pre_bind_event",
- "buffered_chunk",
- "terminal_buffered",
- "request_frame",
- "request_payload",
- "response_body",
- "seen_request_id",
- "decoder_buffer",
- "readiness_buffer",
- "readiness_replay",
- "http2_preface_scan",
- "http2_preface_replay",
- "pending_write",
- "stdin_write",
- ]);
- });
-});
-
-describe("aggregate ceiling configuration", () => {
- it("uses the default when the override is absent", () => {
- expect(resolveDuplexAggregateCeilingBytes()).toBe(DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES);
- expect(resolveDuplexAggregateCeilingBytes(null)).toBe(DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES);
- expect(DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES).toBe(256 * 1024 * 1024);
- });
-
- it("accepts a valid positive override under the process allocation minus the reserve", () => {
- expect(resolveDuplexAggregateCeilingBytes(64 * 1024 * 1024)).toBe(64 * 1024 * 1024);
- const maxCeiling =
- MIN_HOST_PROCESS_MEMORY_BYTES_FOR_DUPLEX - DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES;
- expect(resolveDuplexAggregateCeilingBytes(maxCeiling)).toBe(maxCeiling);
- });
-
- it("rejects an invalid override to the safe default and reports each rejection", () => {
- const overMaximum = MIN_HOST_PROCESS_MEMORY_BYTES_FOR_DUPLEX;
- const invalidOverrides = [0, -1, 1.5, Number.POSITIVE_INFINITY, Number.NaN, overMaximum];
- for (const invalid of invalidOverrides) {
- const rejected: number[] = [];
- // An invalid present override returns the safe default. It never throws, so a
- // single bad environment value never fails host startup.
- expect(resolveDuplexAggregateCeilingBytes(invalid, (value) => rejected.push(value))).toBe(
- DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES,
- );
- // The resolver reports the rejection one time with the rejected numeric value.
- expect(rejected).toEqual([invalid]);
- }
- });
-
- it("does not report a rejection for a valid or a missing override", () => {
- const rejected: number[] = [];
- const reporter = (value: number): void => {
- rejected.push(value);
- };
- expect(resolveDuplexAggregateCeilingBytes(64 * 1024 * 1024, reporter)).toBe(64 * 1024 * 1024);
- expect(resolveDuplexAggregateCeilingBytes(undefined, reporter)).toBe(
- DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES,
- );
- expect(resolveDuplexAggregateCeilingBytes(null, reporter)).toBe(
- DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES,
- );
- expect(rejected).toEqual([]);
- });
-
- it("still throws for a direct ceiling assertion on an unsafe integer", () => {
- // The ledger constructor asserts its ceiling and must fail loud on a
- // programming error, so `assertDuplexAggregateCeilingBytes` keeps throwing.
- expect(() => assertDuplexAggregateCeilingBytes(Number.MAX_SAFE_INTEGER + 2)).toThrow();
- });
-});
diff --git a/packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts b/packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts
deleted file mode 100644
index 9924659ea8..0000000000
--- a/packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts
+++ /dev/null
@@ -1,413 +0,0 @@
-/**
- * The process-owned aggregate byte ledger for the sandbox duplex channel.
- *
- * The route-count controller bounds how many concurrent duplex routes stay live.
- * It does not bound the aggregate bytes those routes retain. At the maximum route
- * count the per-route byte bounds multiply to many gigabytes of retained bytes.
- * This ledger closes that gap. One ledger per host server process owns a single
- * byte ceiling. Every host-side retention site reserves its exact retained bytes
- * against this ledger before it allocates, so the aggregate retained bytes across
- * all live routes never pass the ceiling.
- *
- * The ledger is fail-closed. A reservation that would pass the ceiling returns no
- * token. The caller retains nothing and reports the fixed rejection marker
- * {@link DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED}.
- *
- * Ownership is transfer-only. A token holds an immutable byte amount, an owner
- * label from a closed enum, and a one-way state: `held` then `released`. The
- * ledger moves a token from one representation to the next with
- * {@link DuplexAggregateByteLedger.transfer}, which changes only the owner label.
- * It never decrements and re-reserves, so no admission gap opens between two
- * representations of the same retained bytes.
- *
- * The ledger holds every live token in one owner registry. Registry removal and
- * the release of a token run in one synchronous critical section, before any
- * `await` or external callback. A second release of the same token is an
- * accounting defect. The ledger records the fixed underflow counter and does not
- * clamp the gauge down a second time, so a real defect stays visible.
- *
- * The ledger carries no payload data and no arbitrary labels. It exposes only the
- * byte gauge, the live-token count, and the closed owner enum.
- */
-
-import {
- DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL,
- DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL,
- DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE,
-} from "./duplex-observability.js";
-
-/**
- * The public rejection marker. A retention site that cannot reserve its bytes
- * reports this fixed string. It carries no payload, no route, and no raw value.
- */
-export const DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED = "DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED";
-
-/**
- * The default aggregate ceiling, in bytes: 256 MiB. This is one eighth of the
- * documented minimum host process memory allocation for the sandbox duplex
- * transport ({@link MIN_HOST_PROCESS_MEMORY_BYTES_FOR_DUPLEX}, 2 GiB). The
- * remaining seven eighths cover the Node runtime, ordinary server work, parser
- * and transient headroom, and allocator or garbage-collector variance.
- */
-export const DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES = 256 * 1024 * 1024;
-
-/**
- * The documented minimum host process memory allocation to enable the sandbox
- * duplex transport, in bytes: 2 GiB. An operator override of the aggregate
- * ceiling must not pass this value minus the documented reserve.
- */
-export const MIN_HOST_PROCESS_MEMORY_BYTES_FOR_DUPLEX = 2 * 1024 * 1024 * 1024;
-
-/**
- * The closed set of owner labels a reservation token can carry. Each label names
- * one host-side representation that retains bytes. The ledger accepts no label
- * outside this set, so no arbitrary label ever reaches a token.
- *
- * - `pre_bind_event`: a held pre-bind duplex event, before route bind.
- * - `buffered_chunk`: a buffered data chunk retained for a late listener.
- * - `terminal_buffered`: a buffered chunk moved to the terminal registry at
- * route terminalization, held for a late listener drain.
- * - `request_frame`: a bounded raw request frame the broker retains at dispatch.
- * - `request_payload`: a normalized request payload the broker retains at dispatch.
- * - `response_body`: a response-body chunk or replacement buffer the reader retains.
- * - `seen_request_id`: one entry of the broker no-replay request-id set.
- * - `decoder_buffer`: the raw partial-frame bytes the host frame decoder retains
- * between chunks, plus the peak replacement buffer it allocates on concat.
- * - `readiness_buffer`: the raw untrusted bytes the readiness gate retains before
- * the READY frame completes, before the broker binds and the decoder takes over.
- * - `readiness_replay`: the post-READY suffix bytes the readiness gate holds in the
- * pending replay buffer, from the READY accept until the broker binds and the
- * frame decoder charges its own retention. This is a separate retention from
- * `readiness_buffer`: the gate drops the whole pre-READY buffer on READY, then
- * charges only the retained suffix and each later pre-bind chunk under this owner.
- * - `http2_preface_scan`: the raw untrusted bytes the `http2_v1` preface scan
- * retains while it searches for the client connection preface, from the
- * readiness-replay handoff until the preface match or the scan cap. This
- * is a separate retention from `readiness_replay`: it starts only after
- * the readiness gate hands its own retained suffix to the preface scan.
- * - `http2_preface_replay`: the bytes the `http2_v1` preface scan holds after the
- * client connection preface, from the preface match until the HTTP/2 server
- * binds its downstream listener. This is a separate retention from
- * `http2_preface_scan`: the scan drops its own buffer on the preface match,
- * then charges only the retained suffix and each later pre-bind chunk under
- * this owner.
- * - `pending_write`: the raw host-to-worker write payload a pending duplex write
- * RPC retains, from the enqueue seam until the RPC settles.
- * - `stdin_write`: the serialized host-to-worker frame the child-stdin transport
- * buffer retains, from the write until the stream flushes the chunk, the stream
- * errors, the stream closes, or the worker exits. This is a separate retention
- * from `pending_write`: the RPC holds the raw payload while the transport buffer
- * holds the larger serialized frame, so the two tokens cover the peak of both.
- */
-export const DUPLEX_AGGREGATE_TOKEN_OWNERS = [
- "pre_bind_event",
- "buffered_chunk",
- "terminal_buffered",
- "request_frame",
- "request_payload",
- "response_body",
- "seen_request_id",
- "decoder_buffer",
- "readiness_buffer",
- "readiness_replay",
- "http2_preface_scan",
- "http2_preface_replay",
- "pending_write",
- "stdin_write",
-] as const;
-
-/** One owner label from the closed {@link DUPLEX_AGGREGATE_TOKEN_OWNERS} set. */
-export type DuplexAggregateTokenOwner = (typeof DUPLEX_AGGREGATE_TOKEN_OWNERS)[number];
-
-/** The one-way lifecycle state of a reservation token. */
-export type DuplexReservationTokenState = "held" | "released";
-
-/**
- * An opaque reservation token. The byte amount is immutable. The owner label
- * moves through the closed enum by transfer only. The state moves one way, from
- * `held` to `released`. A caller reads these fields but never mutates them; only
- * the owning ledger changes the owner and the state.
- */
-export interface ReservationToken {
- /** The immutable exact byte amount this token reserves. */
- readonly bytes: number;
- /** The current owner label. The ledger changes it only through `transfer`. */
- readonly owner: DuplexAggregateTokenOwner;
- /** The one-way lifecycle state. The ledger sets it to `released` one time. */
- readonly state: DuplexReservationTokenState;
-}
-
-/**
- * The internal mutable token. The ledger owns it and returns it typed as the
- * read-only {@link ReservationToken}. Only code in this module changes the owner
- * or the state.
- */
-class ReservationTokenImpl implements ReservationToken {
- readonly bytes: number;
- owner: DuplexAggregateTokenOwner;
- state: DuplexReservationTokenState = "held";
-
- constructor(bytes: number, owner: DuplexAggregateTokenOwner) {
- this.bytes = bytes;
- this.owner = owner;
- }
-}
-
-/**
- * The fixed telemetry surface for the aggregate byte ledger. The host binds it to
- * the closed duplex telemetry contract. Each record uses only fixed constant
- * dimensions. No company, agent, run, route, URL, header, request id, token, byte
- * payload, or raw provider value ever reaches a record. The default is a no-op.
- */
-export interface DuplexAggregateByteLedgerTelemetry {
- /** Set the aggregate-bytes-in-use gauge to the current value. */
- setBytesInUse(bytes: number): void;
- /** Increment the reservation-rejection counter one time. */
- recordReservationRejection(): void;
- /** Increment the accounting-underflow counter one time. */
- recordAccountingUnderflow(): void;
-}
-
-/** A no-op telemetry surface. Every method does nothing, so the ledger stays inert. */
-export const NOOP_DUPLEX_AGGREGATE_BYTE_LEDGER_TELEMETRY: DuplexAggregateByteLedgerTelemetry = {
- setBytesInUse() {},
- recordReservationRejection() {},
- recordAccountingUnderflow() {},
-};
-
-/**
- * The low-level process metric sink the host binds. It carries the fixed metric
- * name and the value only, so no dynamic dimension reaches a sink. The host maps
- * the gauge and the two counters to the process metric pipeline and to a warn log
- * for a defect.
- */
-export interface DuplexAggregateByteLedgerMetricSink {
- /** Set a named process gauge to a value. */
- setGauge(name: string, value: number): void;
- /** Increment a named process counter one time. */
- incrementCounter(name: string): void;
-}
-
-/**
- * Build a {@link DuplexAggregateByteLedgerTelemetry} that maps each ledger event
- * to the fixed metric name on the process metric sink. The gauge uses
- * {@link DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE}. The rejection counter uses
- * {@link DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL}. The defect
- * counter uses {@link DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL}.
- * Each call sits in an error swallow, so a sink failure never breaks the reserve
- * or the release path.
- */
-export function createDuplexAggregateByteLedgerTelemetry(
- sink: DuplexAggregateByteLedgerMetricSink,
-): DuplexAggregateByteLedgerTelemetry {
- return {
- setBytesInUse(bytes: number): void {
- try {
- sink.setGauge(DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE, bytes);
- } catch {
- // A telemetry failure never breaks the reserve or the release path.
- }
- },
- recordReservationRejection(): void {
- try {
- sink.incrementCounter(DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL);
- } catch {
- // A telemetry failure never breaks the reserve path.
- }
- },
- recordAccountingUnderflow(): void {
- try {
- sink.incrementCounter(DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL);
- } catch {
- // A telemetry failure never breaks the release path.
- }
- },
- };
-}
-
-/** The options for {@link DuplexAggregateByteLedger}. */
-export interface DuplexAggregateByteLedgerOptions {
- /** The aggregate ceiling, in bytes. It must be a positive safe integer. */
- ceilingBytes: number;
- /** The fixed telemetry surface. The default is the no-op surface. */
- telemetry?: DuplexAggregateByteLedgerTelemetry;
-}
-
-/**
- * Assert an aggregate ceiling byte value. The value must be a positive safe
- * integer that is no larger than the documented process allocation minus the
- * documented reserve. The function throws on any other value, so an invalid
- * ceiling fails startup instead of silently disabling the ledger.
- *
- * The reserve equals the documented process allocation minus the default
- * ceiling ({@link DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES}), so the maximum
- * accepted ceiling is exactly the documented process allocation minus that
- * reserve.
- */
-export function assertDuplexAggregateCeilingBytes(value: number): void {
- const maxCeiling = maxDuplexAggregateCeilingBytes();
- if (!Number.isSafeInteger(value) || value < 1) {
- throw new Error(
- `The duplex aggregate byte ceiling must be a positive safe integer; got ${String(value)}.`,
- );
- }
- if (value > maxCeiling) {
- throw new Error(
- `The duplex aggregate byte ceiling ${value} must not pass the documented process allocation minus the reserve (${maxCeiling}).`,
- );
- }
-}
-
-/**
- * The maximum accepted aggregate ceiling, in bytes: the documented process
- * allocation minus the documented reserve. Both {@link assertDuplexAggregateCeilingBytes}
- * and {@link isValidDuplexAggregateCeilingBytes} read this one bound.
- */
-function maxDuplexAggregateCeilingBytes(): number {
- return MIN_HOST_PROCESS_MEMORY_BYTES_FOR_DUPLEX - DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES;
-}
-
-/**
- * Report whether `value` is a valid aggregate ceiling. A valid ceiling is a
- * positive safe integer that is no larger than {@link maxDuplexAggregateCeilingBytes}.
- * This is the non-throwing form of {@link assertDuplexAggregateCeilingBytes}. The
- * override resolver uses it to reject an invalid override without a throw.
- */
-export function isValidDuplexAggregateCeilingBytes(value: number): boolean {
- return Number.isSafeInteger(value) && value >= 1 && value <= maxDuplexAggregateCeilingBytes();
-}
-
-/**
- * A reporter the host binds to receive one invalid-override rejection signal.
- * {@link resolveDuplexAggregateCeilingBytes} calls it one time when it rejects a
- * present invalid override and falls back to the safe default. The signal carries
- * only the rejected numeric value. It carries no route, company, run, or payload
- * value.
- */
-export type DuplexAggregateCeilingOverrideRejectionReporter = (rejectedValue: number) => void;
-
-/**
- * Resolve an aggregate ceiling from an optional operator override. A missing or
- * `null` override uses {@link DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES}. A present
- * valid value passes through unchanged.
- *
- * A present invalid override does not fail host startup. The host process is
- * multi-tenant, so one invalid environment value must not brick the whole host.
- * The resolver rejects the invalid override, reports it through the optional
- * `onRejectedOverride` reporter, and returns the conservative safe default. This
- * fails loud without failing closed on availability.
- */
-export function resolveDuplexAggregateCeilingBytes(
- override?: number | null,
- onRejectedOverride?: DuplexAggregateCeilingOverrideRejectionReporter,
-): number {
- if (override === undefined || override === null) {
- return DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES;
- }
- if (!isValidDuplexAggregateCeilingBytes(override)) {
- onRejectedOverride?.(override);
- return DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES;
- }
- return override;
-}
-
-/**
- * The process-owned aggregate byte ledger. Create one per host server process.
- * Inject the same object into every host-side retention site, so one shared gauge
- * bounds the aggregate retained bytes across all live duplex routes.
- */
-export class DuplexAggregateByteLedger {
- private readonly ceiling: number;
- private readonly telemetry: DuplexAggregateByteLedgerTelemetry;
- private used = 0;
- /** The owner registry. It holds every live token. */
- private readonly liveTokens = new Set<ReservationTokenImpl>();
-
- constructor(options: DuplexAggregateByteLedgerOptions) {
- assertDuplexAggregateCeilingBytes(options.ceilingBytes);
- this.ceiling = options.ceilingBytes;
- this.telemetry = options.telemetry ?? NOOP_DUPLEX_AGGREGATE_BYTE_LEDGER_TELEMETRY;
- }
-
- /** The configured aggregate ceiling, in bytes. */
- get ceilingBytes(): number {
- return this.ceiling;
- }
-
- /** The current aggregate bytes in use across every live token. */
- get bytesInUse(): number {
- return this.used;
- }
-
- /** The current number of live tokens in the owner registry. */
- get liveTokenCount(): number {
- return this.liveTokens.size;
- }
-
- /**
- * Reserve `bytes` for `owner`. Check and increment the gauge synchronously
- * before the caller allocates. Return a held token when the reservation fits
- * under the ceiling. Return `null` when `used + bytes` would pass the ceiling;
- * the caller retains nothing and reports
- * {@link DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED}.
- *
- * The byte amount must be a non-negative safe integer. A caller that passes any
- * other value made a programming error, so the ledger throws and fails loud.
- */
- reserve(owner: DuplexAggregateTokenOwner, bytes: number): ReservationToken | null {
- if (!Number.isSafeInteger(bytes) || bytes < 0) {
- throw new Error(
- `A duplex byte reservation must be a non-negative safe integer; got ${String(bytes)}.`,
- );
- }
- if (this.used + bytes > this.ceiling) {
- this.telemetry.recordReservationRejection();
- return null;
- }
- this.used += bytes;
- const token = new ReservationTokenImpl(bytes, owner);
- this.liveTokens.add(token);
- this.telemetry.setBytesInUse(this.used);
- return token;
- }
-
- /**
- * Transfer a held token to a new owner label. Change only the owner. Never
- * decrement and never re-reserve. This is the atomic pre-bind handoff: it moves
- * the same reserved bytes from one representation to the next with no admission
- * gap.
- *
- * A transfer of a token the ledger does not hold, or of an already-released
- * token, is an accounting defect. The ledger records the fixed underflow
- * counter and leaves the gauge unchanged.
- */
- transfer(token: ReservationToken, owner: DuplexAggregateTokenOwner): void {
- const impl = token as ReservationTokenImpl;
- if (impl.state !== "held" || !this.liveTokens.has(impl)) {
- this.telemetry.recordAccountingUnderflow();
- return;
- }
- impl.owner = owner;
- }
-
- /**
- * Release a held token. Remove it from the owner registry, mark it `released`,
- * and decrement the gauge, all in one synchronous critical section before any
- * `await` or external callback. A token releases one time.
- *
- * A second release of the same token, or a release of a token the ledger does
- * not hold, is an accounting defect. The ledger records the fixed underflow
- * counter and does not clamp the gauge a second time, so a real defect stays
- * visible on the counter.
- */
- release(token: ReservationToken): void {
- const impl = token as ReservationTokenImpl;
- if (impl.state === "released" || !this.liveTokens.delete(impl)) {
- this.telemetry.recordAccountingUnderflow();
- return;
- }
- impl.state = "released";
- this.used -= impl.bytes;
- this.telemetry.setBytesInUse(this.used);
- }
-}
diff --git a/packages/adapter-utils/src/duplex-frame-codec.ts b/packages/adapter-utils/src/duplex-frame-codec.ts
index b2e848c737..d5f44459cc 100644
--- a/packages/adapter-utils/src/duplex-frame-codec.ts
+++ b/packages/adapter-utils/src/duplex-frame-codec.ts
@@ -22,9 +22,11 @@ export const DUPLEX_FRAME_VERSION = 2;
/**
* The default maximum size of one frame, in bytes. The decoder rejects a longer
* frame with a `frame_too_large` protocol error. The value matches the per-chunk
- * character bound of the host duplex route.
+ * byte bound of the host duplex route, and the host body limit for one HTTP/2
+ * bridge stream ({@link DEFAULT_SANDBOX_CALLBACK_BRIDGE_MAX_BODY_BYTES} in
+ * `sandbox-callback-bridge.ts`).
*/
-export const DEFAULT_MAX_DUPLEX_FRAME_BYTES = 1_000_000;
+export const DEFAULT_MAX_DUPLEX_FRAME_BYTES = 262_144;
/**
* The READY control frame. The gateway sends it one time after it binds the
@@ -74,8 +76,7 @@ export type DuplexProtocolErrorCode =
| "unknown_type"
| "version_mismatch"
| "frame_too_large"
- | "id_too_large"
- | "aggregate_bytes_exceeded";
+ | "id_too_large";
/** A decode-time protocol error. The read path returns it; it never throws. */
export interface DuplexProtocolError {
diff --git a/packages/adapter-utils/src/duplex-frame-vectors.json b/packages/adapter-utils/src/duplex-frame-vectors.json
index 99a6762b7a..fb15ba5091 100644
--- a/packages/adapter-utils/src/duplex-frame-vectors.json
+++ b/packages/adapter-utils/src/duplex-frame-vectors.json
@@ -1,6 +1,6 @@
{
"frameVersion": 2,
- "defaultMaxFrameBytes": 1000000,
+ "defaultMaxFrameBytes": 262144,
"description": "Shared wire-compatibility vectors for the duplex frame codec. Every codec copy decodes the same bytes. bytes is a UTF-8 byte stream; expected lists the decode result (frame or protocol-error code).",
"vectors": [
{
diff --git a/packages/adapter-utils/src/duplex-observability.ts b/packages/adapter-utils/src/duplex-observability.ts
index 1d144fa60f..db798bbe14 100644
--- a/packages/adapter-utils/src/duplex-observability.ts
+++ b/packages/adapter-utils/src/duplex-observability.ts
@@ -35,40 +35,6 @@ export const DUPLEX_COUNTER_LOSS_TOTAL = "sandbox_duplex_loss_total";
/** The guarded counter for one leaked provider session on teardown. */
export const DUPLEX_COUNTER_SESSION_LEAK_TOTAL = "sandbox_duplex_session_leak_total";
-/**
- * The process-scoped gauge for the aggregate retained bytes across every live
- * duplex route. The host aggregate byte ledger sets it on each reserve and each
- * release. The record carries no dynamic dimension.
- */
-export const DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE = "sandbox_duplex_aggregate_bytes_in_use";
-/**
- * The counter for one rejected aggregate byte reservation. The host ledger
- * increments it when a reservation would pass the aggregate ceiling. The record
- * carries no dynamic dimension.
- */
-export const DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL =
- "sandbox_duplex_aggregate_byte_reservation_rejections_total";
-/**
- * The counter for one aggregate byte accounting defect. The host ledger
- * increments it on a double release or a transfer of a token it does not hold.
- * The record carries no dynamic dimension.
- */
-export const DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL =
- "sandbox_duplex_aggregate_byte_accounting_underflow_total";
-
-/**
- * The closed set of aggregate byte ledger metric names. A test pins this exact
- * set, so a new ledger metric name needs an explicit review. Each record uses
- * only closed constant dimensions and no dynamic label. The Observability
- * contract documents these metrics under "Aggregate byte ledger metrics" in
- * `doc/observability.md`.
- */
-export const DUPLEX_AGGREGATE_BYTE_LEDGER_METRIC_NAMES = [
- DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE,
- DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL,
- DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL,
-] as const;
-
/**
* The closed dimension-key set. Every span attribute, counter label, and event
* field uses only these keys. A test asserts the exact set, so a new key never
@@ -102,13 +68,10 @@ export type DuplexOutcomeValue = "ok" | "error";
* stage: the process-scoped route ceiling was full (`route_busy`), the entrypoint
* sync failed (`entrypoint_sync_failed`), the broker construction failed
* (`broker_construction_failed`), or the channel open failed (`channel_open_failed`).
- * The `aggregate_bytes_exceeded` reason names a readiness handshake, or an
- * `http2` post-preface pre-bind buffer, the host fell back because the
- * process aggregate byte ceiling had no room. The `preface_missing` reason
- * names a missing or an invalid HTTP/2 client connection preface inside the
- * bounded readiness buffer: the host found no valid preface after the
- * accepted READY line, aborted the HTTP/2 open, and moved the run to
- * `queue_v1` one time.
+ * The `preface_missing` reason names a missing or an invalid HTTP/2 client
+ * connection preface inside the bounded readiness buffer: the host found no
+ * valid preface after the accepted READY line, aborted the HTTP/2 open, and
+ * moved the run to `queue_v1` one time.
*/
export type DuplexFallbackReason =
| "gate_off"
@@ -121,7 +84,6 @@ export type DuplexFallbackReason =
| "ready_nonce_mismatch"
| "ready_timeout"
| "contaminated"
- | "aggregate_bytes_exceeded"
| "preface_missing";
/** The class of a terminal loss, relative to the first request dispatch. */
diff --git a/packages/adapter-utils/src/execution-target-sandbox.test.ts b/packages/adapter-utils/src/execution-target-sandbox.test.ts
index 3a26fbc439..54dea1c5aa 100644
--- a/packages/adapter-utils/src/execution-target-sandbox.test.ts
+++ b/packages/adapter-utils/src/execution-target-sandbox.test.ts
@@ -42,10 +42,6 @@ import {
type StartupTraceContext,
type StartupTracer,
} from "./acpx-engine/startup-timing.js";
-import {
- DuplexAggregateByteLedger,
- DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED,
-} from "./duplex-aggregate-byte-ledger.js";
import { createSandboxRunLogTailFactory, type SandboxRunLogTailFactory } from "./sandbox-run-log-stream.js";
import { runChildProcess } from "./server-utils.js";
import { shellQuote } from "./ssh.js";
@@ -58,14 +54,10 @@ import {
} from "./duplex-frame-codec.js";
import { DUPLEX_CHANNEL_LOST_ERROR_CODE } from "./bridge-transport-contract.js";
import {
- DUPLEX_AGGREGATE_BYTE_LEDGER_METRIC_NAMES,
- DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL,
- DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL,
DUPLEX_COUNTER_CHANNEL_OPEN_TOTAL,
DUPLEX_COUNTER_FALLBACK_TOTAL,
DUPLEX_COUNTER_LOSS_TOTAL,
DUPLEX_DIMENSION_KEYS,
- DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE,
DUPLEX_SPAN_CHANNEL_OPEN,
DUPLEX_SPAN_REQUEST,
DUPLEX_TRANSPORT_EVENT,
@@ -2675,155 +2667,6 @@ describe("sandbox adapter execution targets", () => {
}
});
- it("charges the response-body bytes against the host aggregate ledger and releases every token on success", async () => {
- // The host stamps one process-owned aggregate byte ledger on the sandbox
- // target. The forward response-body reader charges its retained bytes against
- // that ledger. A successful read charges the chunk bytes and the
- // concatenation buffer, then releases every token, so the ledger returns to
- // zero after the forward completes.
- const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-execution-target-bridge-ledger-ok-"));
- cleanupDirs.push(rootDir);
- const remoteCwd = path.join(rootDir, "workspace");
- const runtimeRootDir = path.join(remoteCwd, ".paperclip-runtime", "codex");
- await mkdir(runtimeRootDir, { recursive: true });
-
- const responseBody = JSON.stringify({ id: "issue-1" });
- const apiServer = createServer((_req, res) => {
- res.writeHead(200, {
- "content-type": "application/json",
- "content-length": String(Buffer.byteLength(responseBody, "utf8")),
- });
- res.end(responseBody);
- });
- await new Promise<void>((resolve, reject) => {
- apiServer.once("error", reject);
- apiServer.listen(0, "127.0.0.1", () => resolve());
- });
- const address = apiServer.address();
- if (!address || typeof address === "string") {
- throw new Error("Expected the bridge test API server to listen on a TCP port.");
- }
-
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1024 * 1024 });
- const target: AdapterSandboxExecutionTarget = {
- kind: "remote",
- transport: "sandbox",
- providerKey: "e2b",
- environmentId: "env-1",
- leaseId: "lease-1",
- remoteCwd,
- runner: createLocalSandboxRunner(),
- timeoutMs: 30_000,
- duplexAggregateByteLedger: ledger,
- };
-
- const bridge = await startAdapterExecutionTargetPaperclipBridge({
- runId: "run-bridge-ledger-ok",
- target,
- runtimeRootDir,
- adapterKey: "codex",
- hostApiToken: "real-run-jwt",
- hostApiUrl: `http://127.0.0.1:${address.port}`,
- maxBodyBytes: 512,
- });
- try {
- const response = await fetch(`${bridge!.env.PAPERCLIP_API_URL}/api/issues/issue-1`, {
- method: "GET",
- headers: {
- authorization: `Bearer ${bridge!.env.PAPERCLIP_API_KEY}`,
- },
- });
-
- expect(response.status).toBe(200);
- await expect(response.json()).resolves.toEqual({ id: "issue-1" });
- // The reader released every token, so the aggregate gauge and the live-token
- // registry both return to zero.
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- } finally {
- await bridge?.stop();
- await new Promise<void>((resolve) => apiServer.close(() => resolve()));
- }
- });
-
- it("fails a response-body read closed when the host aggregate ledger has no room and retains no bytes", async () => {
- // The aggregate ledger sits at a tiny ceiling, so a response body larger than
- // the ceiling cannot reserve its bytes. The reader fails closed: it cancels
- // the stream reader, retains nothing, and reports the fixed marker. The safe
- // GET maps the marker to a retryable 502. The ledger returns to zero, because
- // the reader released the tokens it held before the rejection.
- const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-execution-target-bridge-ledger-full-"));
- cleanupDirs.push(rootDir);
- const remoteCwd = path.join(rootDir, "workspace");
- const runtimeRootDir = path.join(remoteCwd, ".paperclip-runtime", "codex");
- await mkdir(runtimeRootDir, { recursive: true });
-
- // The body sits under the per-request size limit but over the aggregate
- // ceiling, so the aggregate ledger, not the per-request limit, rejects it.
- const responseBody = "x".repeat(256);
- const apiServer = createServer((_req, res) => {
- res.writeHead(200, {
- "content-type": "application/json",
- "content-length": String(Buffer.byteLength(responseBody, "utf8")),
- });
- res.end(responseBody);
- });
- await new Promise<void>((resolve, reject) => {
- apiServer.once("error", reject);
- apiServer.listen(0, "127.0.0.1", () => resolve());
- });
- const address = apiServer.address();
- if (!address || typeof address === "string") {
- throw new Error("Expected the bridge test API server to listen on a TCP port.");
- }
-
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 8 });
- const target: AdapterSandboxExecutionTarget = {
- kind: "remote",
- transport: "sandbox",
- providerKey: "e2b",
- environmentId: "env-1",
- leaseId: "lease-1",
- remoteCwd,
- runner: createLocalSandboxRunner(),
- timeoutMs: 30_000,
- duplexAggregateByteLedger: ledger,
- };
-
- const bridge = await startAdapterExecutionTargetPaperclipBridge({
- runId: "run-bridge-ledger-full",
- target,
- runtimeRootDir,
- adapterKey: "codex",
- hostApiToken: "real-run-jwt",
- hostApiUrl: `http://127.0.0.1:${address.port}`,
- maxBodyBytes: 4096,
- });
- try {
- const response = await fetch(`${bridge!.env.PAPERCLIP_API_URL}/api/issues/issue-1`, {
- method: "GET",
- headers: {
- authorization: `Bearer ${bridge!.env.PAPERCLIP_API_KEY}`,
- },
- });
-
- // The safe GET maps the aggregate rejection to a retryable 502 with no
- // indeterminate marker. The body carries only the fixed rejection marker.
- expect(response.status).toBe(502);
- expect(response.headers.get("x-paperclip-bridge-outcome")).toBeNull();
- await expect(response.json()).resolves.toEqual({
- error: DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED,
- });
- // The reader released the tokens it held before the rejection, so the
- // aggregate gauge and the live-token registry both return to zero.
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- } finally {
- await bridge?.stop();
- await new Promise<void>((resolve) => apiServer.close(() => resolve()));
- }
- });
-
it("forwards the host indeterminate-outcome header so the sandbox server maps the 504 to a non-retryable 409", async () => {
// The host marks a possibly-committed mutation with a 504 and the
// `x-paperclip-bridge-outcome: indeterminate` header. The forward must keep
@@ -3460,66 +3303,6 @@ describe("sandbox adapter execution targets", () => {
expect(control.stopCount).toBeGreaterThanOrEqual(1);
}, 20000);
- it("falls back to the file bridge when a post-READY pre-bind flood exceeds the aggregate ceiling", async () => {
- // The gateway sends a valid READY, then floods the channel before the broker
- // binds. The pre-READY buffer cap does not bound the post-READY replay buffer,
- // so the replay reservation must. The ceiling admits the small READY frame but
- // rejects the flood. The host drops the buffer, stops the channel, and selects
- // the file bridge with the aggregate marker. No request forwards, and the
- // aggregate ledger returns to zero.
- const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-duplex-replay-flood-"));
- cleanupDirs.push(rootDir);
- const remoteCwd = path.join(rootDir, "workspace");
- await mkdir(remoteCwd, { recursive: true });
- const api = await startRecordingApiServer();
- // The flood is larger than the ceiling; the READY frame is far smaller.
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 4096 });
- const { runner, control } = makeDuplexSelectionRunner((ctx) => {
- ctx.emitFrame({ version: 2, type: "ready", nonce: ctx.nonce });
- ctx.emitRaw("x".repeat(64 * 1024));
- });
- const { recorder, counters } = createRecordingDuplexRecorder();
- const target: AdapterSandboxExecutionTarget = {
- kind: "remote",
- transport: "sandbox",
- providerKey: "daytona",
- remoteCwd,
- timeoutMs: 30_000,
- runner,
- effectiveCapabilities: duplexCapabilities(true),
- duplexAggregateByteLedger: ledger,
- };
-
- const bridge = await startAdapterExecutionTargetPaperclipBridge({
- runId: "run-duplex-replay-flood",
- target,
- runtimeRootDir: path.join(remoteCwd, ".paperclip-runtime", "codex"),
- adapterKey: "codex",
- hostApiToken: "real-run-jwt",
- hostApiUrl: api.origin,
- enableSandboxDuplexBridge: true,
- duplexObservabilityRecorder: recorder,
- });
- try {
- // The file bridge serves, not the duplex transport.
- expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("queue_v1");
- // The fallback names the aggregate marker on the file transport.
- const fallback = counters.find((c) => c.metric === DUPLEX_COUNTER_FALLBACK_TOTAL);
- expect(fallback?.dimensions.fallback_reason).toBe("aggregate_bytes_exceeded");
- expect(fallback?.dimensions.transport).toBe("file");
- // The gate stopped the flooded channel.
- expect(control.stopCount).toBeGreaterThanOrEqual(1);
- // No request forwarded, because the broker never bound.
- expect(api.requests).toHaveLength(0);
- // The aggregate ledger returns to zero with no live token.
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- } finally {
- await bridge?.stop();
- await api.close();
- }
- }, 20000);
-
it("streams run logs on the http2 path under the same gate and log line as the file path", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-http2-runlog-"));
cleanupDirs.push(rootDir);
@@ -3990,24 +3773,6 @@ describe("sandbox adapter execution targets", () => {
]);
});
- it("pins the exact aggregate byte ledger metric names", () => {
- // The aggregate byte ledger metric names are closed. This test locks the
- // exact set, so a new gauge or counter name needs an explicit change here.
- // Each record carries only closed constant dimensions and no dynamic label.
- expect([...DUPLEX_AGGREGATE_BYTE_LEDGER_METRIC_NAMES]).toEqual([
- "sandbox_duplex_aggregate_bytes_in_use",
- "sandbox_duplex_aggregate_byte_reservation_rejections_total",
- "sandbox_duplex_aggregate_byte_accounting_underflow_total",
- ]);
- expect(DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE).toBe("sandbox_duplex_aggregate_bytes_in_use");
- expect(DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL).toBe(
- "sandbox_duplex_aggregate_byte_reservation_rejections_total",
- );
- expect(DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL).toBe(
- "sandbox_duplex_aggregate_byte_accounting_underflow_total",
- );
- });
-
it("records an http2 request span with latency and the fixed dimension keys", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-http2-obs-"));
cleanupDirs.push(rootDir);
@@ -4123,7 +3888,6 @@ describe("sandbox adapter execution targets", () => {
"ready_nonce_mismatch",
"ready_timeout",
"contaminated",
- "aggregate_bytes_exceeded",
];
expect(approvedReasons).toContain(fallback?.dimensions.fallback_reason);
expect(fallback?.dimensions).toMatchObject({ transport: "file", outcome: "error" });
@@ -4610,129 +4374,6 @@ describe("sandbox adapter execution targets", () => {
}
}, 20000);
- it("fails the readiness handshake closed when the host aggregate ledger has no room for the pre-READY buffer", async () => {
- const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-duplex-ready-ledger-full-"));
- cleanupDirs.push(rootDir);
- const remoteCwd = path.join(rootDir, "workspace");
- await mkdir(remoteCwd, { recursive: true });
- const api = await startRecordingApiServer();
- // The host stamps one process-owned aggregate byte ledger on the sandbox
- // target at a tiny ceiling. The fake gateway sends a pre-READY blob larger
- // than the ceiling, so the gate cannot reserve the blob bytes. The gate fails
- // closed: it retains nothing, records the aggregate fallback reason, and falls
- // back to the file bridge. The blob is smaller than the readiness buffer cap,
- // so the aggregate ledger, not the buffer cap, drives the failure.
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 256 });
- const preReadyBlob = "x".repeat(4_096);
- const { runner, control } = makeDuplexSelectionRunner((ctx) => {
- ctx.emitRaw(preReadyBlob);
- });
- const { recorder, counters } = createRecordingDuplexRecorder();
- const target: AdapterSandboxExecutionTarget = {
- kind: "remote",
- transport: "sandbox",
- providerKey: "daytona",
- remoteCwd,
- timeoutMs: 30_000,
- runner,
- effectiveCapabilities: duplexCapabilities(true),
- duplexAggregateByteLedger: ledger,
- };
-
- const bridge = await startAdapterExecutionTargetPaperclipBridge({
- runId: "run-ready-ledger-full",
- target,
- runtimeRootDir: path.join(remoteCwd, ".paperclip-runtime", "codex"),
- adapterKey: "codex",
- hostApiToken: "real-run-jwt",
- hostApiUrl: api.origin,
- enableSandboxDuplexBridge: true,
- // A long readiness timeout, so the aggregate ledger, not the timeout, drives
- // the failure.
- duplexReadinessTimeoutMs: 5_000,
- duplexObservabilityRecorder: recorder,
- });
- try {
- expect(bridge).not.toBeNull();
- expect(control.openCount).toBe(1);
- // The aggregate rejection drove the failure, so the file bridge serves.
- expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("queue_v1");
- const fallback = counters.find((c) => c.metric === DUPLEX_COUNTER_FALLBACK_TOTAL);
- expect(fallback?.dimensions.fallback_reason).toBe("aggregate_bytes_exceeded");
- // The gate retained nothing after the rejection, so the aggregate gauge and
- // the live-token registry both return to zero.
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(control.closeCount + control.stopCount).toBeGreaterThanOrEqual(1);
- } finally {
- await bridge?.stop();
- await api.close();
- }
- }, 20000);
-
- it("charges the pre-READY buffer against the injected host ledger and releases it when readiness passes", async () => {
- const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-http2-ready-ledger-ok-"));
- cleanupDirs.push(rootDir);
- const remoteCwd = path.join(rootDir, "workspace");
- await mkdir(remoteCwd, { recursive: true });
- const api = await startRecordingApiServer();
- // The host stamps one process-owned aggregate byte ledger on the sandbox
- // target at a generous ceiling. The fake gateway sends one pre-READY noise
- // line, then the valid READY frame, then a real HTTP/2 client preface. The
- // gate charges the noise bytes against the injected ledger, passes
- // readiness, and releases the pre-READY tokens. The ledger returns to zero
- // once the post-READY replay hands off to the bound HTTP/2 channel.
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1024 * 1024 });
- const reserveSpy = vi.spyOn(ledger, "reserve");
- const { runner, control } = makeHttp2SelectionRunner((ctx) => {
- ctx.emitRaw("pty-echo-noise");
- ctx.emitReady();
- ctx.connectHttp2();
- });
- const target: AdapterSandboxExecutionTarget = {
- kind: "remote",
- transport: "sandbox",
- providerKey: "daytona",
- remoteCwd,
- timeoutMs: 30_000,
- runner,
- effectiveCapabilities: duplexCapabilities(true),
- duplexAggregateByteLedger: ledger,
- };
-
- const bridge = await startAdapterExecutionTargetPaperclipBridge({
- runId: "run-ready-ledger-ok",
- target,
- runtimeRootDir: path.join(remoteCwd, ".paperclip-runtime", "codex"),
- adapterKey: "codex",
- hostApiToken: "real-run-jwt",
- hostApiUrl: api.origin,
- enableSandboxDuplexBridge: true,
- duplexReadinessTimeoutMs: 5_000,
- });
- try {
- expect(bridge).not.toBeNull();
- // Readiness passed, so the http2 transport serves.
- expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("http2_v1");
- // The gate charged the pre-READY noise against the exact injected ledger, so
- // the identity holds at this seam.
- expect(reserveSpy).toHaveBeenCalledWith("readiness_buffer", expect.any(Number));
- // The gate released every readiness-buffer token on settle, so the aggregate
- // gauge and the live-token registry both return to zero.
- await waitForCondition(
- () => ledger.bytesInUse === 0 && ledger.liveTokenCount === 0,
- "the readiness gate to release every pre-READY token",
- 4000,
- );
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- } finally {
- reserveSpy.mockRestore();
- await bridge?.stop();
- await api.close();
- }
- }, 20000);
-
it("bounds the pre-READY newline-scan work by the bytes received", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-duplex-scan-"));
cleanupDirs.push(rootDir);
@@ -5689,6 +5330,87 @@ describe("sandbox adapter execution targets", () => {
}
}, 20000);
+ it("aborts the host forward and its response-body read when the sandbox stream closes", async () => {
+ // The mock host answers with headers at once, then holds the response
+ // body open with no further chunk and no end. The read stays pending
+ // until the outbound fetch itself aborts. If the abort never reaches
+ // this connection, `hostConnectionClosed` never resolves and the test
+ // times out instead of failing fast — the assertion is: it does resolve,
+ // quickly, once the sandbox stream closes.
+ let sawRequest = false;
+ let resolveHostConnectionClosed: (() => void) | undefined;
+ const hostConnectionClosed = new Promise<void>((resolve) => {
+ resolveHostConnectionClosed = resolve;
+ });
+ const api = createServer((req, res) => {
+ sawRequest = true;
+ res.writeHead(200, { "content-type": "application/json" });
+ res.on("close", () => resolveHostConnectionClosed!());
+ });
+ await new Promise<void>((resolve, reject) => {
+ api.once("error", reject);
+ api.listen(0, "127.0.0.1", () => resolve());
+ });
+ const apiAddress = api.address();
+ if (!apiAddress || typeof apiAddress === "string") {
+ throw new Error("Expected the mock host server to listen on a TCP port.");
+ }
+ const apiOrigin = `http://127.0.0.1:${apiAddress.port}`;
+
+ const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-http2-abort-forward-"));
+ cleanupDirs.push(rootDir);
+ const remoteCwd = path.join(rootDir, "workspace");
+ await mkdir(remoteCwd, { recursive: true });
+ const sessionRef: { current: http2.ClientHttp2Session | null } = { current: null };
+ let bridgeToken = "";
+ const { runner } = makeHttp2SelectionRunner((ctx) => {
+ bridgeToken = ctx.bridgeToken;
+ ctx.emitReady();
+ sessionRef.current = ctx.connectHttp2();
+ });
+ const target: AdapterSandboxExecutionTarget = {
+ kind: "remote",
+ transport: "sandbox",
+ providerKey: "daytona",
+ remoteCwd,
+ timeoutMs: 30_000,
+ runner,
+ effectiveCapabilities: duplexCapabilities(true),
+ };
+
+ const bridge = await startAdapterExecutionTargetPaperclipBridge({
+ runId: "run-abort-forward",
+ target,
+ runtimeRootDir: path.join(remoteCwd, ".paperclip-runtime", "codex"),
+ adapterKey: "codex",
+ hostApiToken: "real-run-jwt",
+ hostApiUrl: apiOrigin,
+ enableSandboxDuplexBridge: true,
+ // Far longer than this test waits, so only the sandbox-side stream
+ // close — not this ceiling — can end the forward here.
+ forwardTimeoutMs: 60_000,
+ });
+ try {
+ expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("http2_v1");
+ await waitForCondition(() => sessionRef.current !== null, "the http2 client session to open", 4000);
+ const clientStream = sessionRef.current!.request({
+ ":method": "GET",
+ ":path": "/api/agents/me",
+ authorization: `Bearer ${bridgeToken}`,
+ });
+ clientStream.end();
+ await waitForCondition(() => sawRequest, "the mock host to receive the forwarded request", 4000);
+ // The sandbox side closes its stream while the host forward and its
+ // response-body read are both still in flight.
+ clientStream.close(http2.constants.NGHTTP2_CANCEL);
+ await hostConnectionClosed;
+ } finally {
+ sessionRef.current?.close();
+ await bridge?.stop();
+ await new Promise<void>((resolve) => api.close(() => resolve()));
+ }
+ }, 20000);
+
// ---------------------------------------------------------------------------
// Real-PTY replay.
//
@@ -6321,146 +6043,15 @@ describe("duplex readiness gate replay-buffer reservation", () => {
return { channel, control };
}
- // A ledger that counts the reservation-rejection and the accounting-underflow
- // signals, so a test proves the one-owner-one-release invariant holds.
- function makeCountingLedger(ceilingBytes: number): {
- ledger: DuplexAggregateByteLedger;
- counts: { rejections: number; underflows: number };
- } {
- const counts = { rejections: 0, underflows: 0 };
- const ledger = new DuplexAggregateByteLedger({
- ceilingBytes,
- telemetry: {
- setBytesInUse(): void {},
- recordReservationRejection(): void {
- counts.rejections += 1;
- },
- recordAccountingUnderflow(): void {
- counts.underflows += 1;
- },
- },
- });
- return { ledger, counts };
- }
-
function readyLine(): string {
return `${JSON.stringify({ version: 2, type: "ready", nonce: READY_NONCE })}\n`;
}
- it("charges the post-READY suffix and releases it after the broker handoff", async () => {
- const { channel, control } = makeFakeReadinessChannel();
- const { ledger, counts } = makeCountingLedger(1024 * 1024);
- const gate = __duplexReadinessTesting.createReadinessGate(channel, {
- nonce: READY_NONCE,
- timeoutMs: 5_000,
- ledger,
- });
- const suffix = "hello-post-ready-suffix";
- // The READY line and the suffix arrive in one chunk. The gate drops the whole
- // pre-READY buffer charge, then charges only the retained suffix.
- control.emitData(`${readyLine()}${suffix}`);
- const readiness = await gate.ready;
- expect(readiness.ok).toBe(true);
- expect(gate.replayOverflowed()).toBe(false);
- // The gate holds the suffix under one readiness_replay token before the bind.
- expect(ledger.bytesInUse).toBe(Buffer.byteLength(suffix, "utf8"));
- expect(ledger.liveTokenCount).toBe(1);
- // The broker binds and replays the suffix; the gate releases the token after
- // the synchronous handoff.
- const replayed: string[] = [];
- gate.brokerChannel.onData((chunk) => replayed.push(Buffer.from(chunk).toString("utf8")));
- expect(replayed).toEqual([suffix]);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(counts.underflows).toBe(0);
- });
-
- it("releases the suffix token on disposePendingReplay without a broker handoff", async () => {
- const { channel, control } = makeFakeReadinessChannel();
- const { ledger, counts } = makeCountingLedger(1024 * 1024);
- const gate = __duplexReadinessTesting.createReadinessGate(channel, {
- nonce: READY_NONCE,
- timeoutMs: 5_000,
- ledger,
- });
- const suffix = "abandoned-suffix";
- control.emitData(`${readyLine()}${suffix}`);
- expect((await gate.ready).ok).toBe(true);
- expect(ledger.bytesInUse).toBe(Buffer.byteLength(suffix, "utf8"));
- // A broker-construction failure abandons the buffer, so the caller disposes it.
- gate.disposePendingReplay();
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- // A second dispose is a no-op and never underflows.
- gate.disposePendingReplay();
- expect(ledger.bytesInUse).toBe(0);
- expect(counts.underflows).toBe(0);
- });
-
- it("releases the suffix token after a pre-bind exit then a broker handoff", async () => {
- const { channel, control } = makeFakeReadinessChannel();
- const { ledger, counts } = makeCountingLedger(1024 * 1024);
- const gate = __duplexReadinessTesting.createReadinessGate(channel, {
- nonce: READY_NONCE,
- timeoutMs: 5_000,
- ledger,
- });
- const suffix = "pre-bind-exit-suffix";
- control.emitData(`${readyLine()}${suffix}`);
- expect((await gate.ready).ok).toBe(true);
- // The channel exits after READY but before the broker binds. The gate holds the
- // exit and keeps the pending suffix charged.
- control.emitExit({ exitCode: 0 });
- expect(ledger.bytesInUse).toBe(Buffer.byteLength(suffix, "utf8"));
- expect(ledger.liveTokenCount).toBe(1);
- // The broker binds, replays the suffix and the exit, then the gate releases the
- // reservation.
- const replayed: string[] = [];
- const exits: Array<{ exitCode: number | null }> = [];
- gate.brokerChannel.onData((chunk) => replayed.push(Buffer.from(chunk).toString("utf8")));
- gate.brokerChannel.onExit((exit) => exits.push(exit));
- expect(replayed).toEqual([suffix]);
- expect(exits).toEqual([{ exitCode: 0 }]);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(counts.underflows).toBe(0);
- });
-
- it("fails closed when a post-READY pre-bind chunk floods past the ceiling", async () => {
- const { channel, control } = makeFakeReadinessChannel();
- // The ceiling admits the small READY line but not the flood chunk.
- const { ledger, counts } = makeCountingLedger(256);
- const gate = __duplexReadinessTesting.createReadinessGate(channel, {
- nonce: READY_NONCE,
- timeoutMs: 5_000,
- ledger,
- });
- // The READY line arrives alone, so the pending suffix starts empty.
- control.emitData(readyLine());
- expect((await gate.ready).ok).toBe(true);
- expect(ledger.bytesInUse).toBe(0);
- // A post-READY chunk larger than the ceiling floods the replay buffer before
- // the broker binds. The gate refuses the reservation and fails closed.
- control.emitData("x".repeat(512));
- expect(gate.replayOverflowed()).toBe(true);
- expect(control.stopCount).toBe(1);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(counts.rejections).toBe(1);
- expect(counts.underflows).toBe(0);
- // Binding the broker replays nothing, because the gate dropped the buffer.
- const replayed: string[] = [];
- gate.brokerChannel.onData((chunk) => replayed.push(Buffer.from(chunk).toString("utf8")));
- expect(replayed).toEqual([]);
- });
-
it("drops the retained pre-READY buffer on READY acceptance", async () => {
const { channel, control } = makeFakeReadinessChannel();
- const { ledger, counts } = makeCountingLedger(1024 * 1024);
const gate = __duplexReadinessTesting.createReadinessGate(channel, {
nonce: READY_NONCE,
timeoutMs: 5_000,
- ledger,
});
// A large pre-READY noise line and a large suffix arrive with the READY line
// in one chunk. A sandbox controls every byte here.
@@ -6469,61 +6060,37 @@ describe("duplex readiness gate replay-buffer reservation", () => {
control.emitData(`${noise}${readyLine()}${suffix}`);
expect((await gate.ready).ok).toBe(true);
// The gate drops the pre-READY buffer, so the process no longer retains the
- // noise prefix. Without this, the process holds the full sandbox string while
- // the ledger charges only the suffix, so retention passes the ceiling.
+ // noise prefix.
expect(gate.retainedReadinessBufferLength()).toBe(0);
- // The ledger charges only the retained suffix, not the dropped prefix.
- expect(ledger.bytesInUse).toBe(Buffer.byteLength(suffix, "utf8"));
- expect(ledger.liveTokenCount).toBe(1);
- // The broker binds, replays the suffix, and the gate releases the token.
+ // The broker binds and replays the retained suffix.
const replayed: string[] = [];
gate.brokerChannel.onData((chunk) => replayed.push(Buffer.from(chunk).toString("utf8")));
expect(replayed).toEqual([suffix]);
- expect(ledger.bytesInUse).toBe(0);
expect(gate.retainedReadinessBufferLength()).toBe(0);
- expect(counts.underflows).toBe(0);
});
- it("releases the readiness_replay token before the downstream listener runs, so a same-size downstream reservation for the same bytes does not double-book", async () => {
+ it("caps the post-READY replay buffer without the aggregate ledger", async () => {
const { channel, control } = makeFakeReadinessChannel();
- const suffix = "s".repeat(512);
- const suffixBytes = Buffer.byteLength(suffix, "utf8");
- // The ceiling admits the READY line plus one reservation of the suffix
- // size, but not a second, separate reservation of the suffix on top of
- // that. A downstream listener that reserves the same bytes under its own
- // owner — the real shape of the HTTP/2 preface scanner — proves the gate
- // releases its own `readiness_replay` token first: the downstream
- // reservation must still fit.
- const { ledger, counts } = makeCountingLedger(Buffer.byteLength(readyLine(), "utf8") + suffixBytes);
+ const readinessBufferCapBytes = DEFAULT_MAX_DUPLEX_FRAME_BYTES + 4_096;
+ // No ledger: only the buffer's own direct byte cap can end the channel
+ // here. This proves the cap holds even when no aggregate ledger is
+ // present, unlike the ledger-only check this replaces.
const gate = __duplexReadinessTesting.createReadinessGate(channel, {
nonce: READY_NONCE,
timeoutMs: 5_000,
- ledger,
});
- control.emitData(`${readyLine()}${suffix}`);
+ // The READY line arrives alone, so the pending suffix starts empty.
+ control.emitData(readyLine());
expect((await gate.ready).ok).toBe(true);
- expect(ledger.bytesInUse).toBe(suffixBytes);
-
- let peakBytesInUseDuringHandoff = -1;
- let downstreamToken: ReturnType<DuplexAggregateByteLedger["reserve"]> = null;
- gate.brokerChannel.onData((chunk) => {
- // The downstream listener reserves the same bytes under its own owner,
- // the same way the preface scanner charges `http2_preface_scan` for the
- // replayed chunk. The gate must have released its own token before this
- // call runs, so this reservation fits under the tight ceiling.
- downstreamToken = ledger.reserve("http2_preface_scan", chunk.byteLength);
- peakBytesInUseDuringHandoff = ledger.bytesInUse;
- });
-
- // The downstream reservation succeeded: the gate's release ran first, so
- // only one reservation for these bytes was ever live at once.
- expect(downstreamToken).not.toBeNull();
- expect(peakBytesInUseDuringHandoff).toBe(suffixBytes);
- expect(counts.rejections).toBe(0);
- expect(counts.underflows).toBe(0);
- // The gate's own token is gone; only the downstream token remains live.
- expect(ledger.liveTokenCount).toBe(1);
- expect(ledger.bytesInUse).toBe(suffixBytes);
+ // A post-READY chunk one byte over the cap floods the replay buffer
+ // before the broker binds.
+ control.emitData("x".repeat(readinessBufferCapBytes + 1));
+ expect(gate.replayOverflowed()).toBe(true);
+ expect(control.stopCount).toBe(1);
+ // Binding the broker replays nothing, because the gate dropped the buffer.
+ const replayed: string[] = [];
+ gate.brokerChannel.onData((chunk) => replayed.push(Buffer.from(chunk).toString("utf8")));
+ expect(replayed).toEqual([]);
});
});
@@ -6562,147 +6129,43 @@ describe("http2 preface scan post-preface replay buffer", () => {
return { channel, control };
}
- function makeCountingLedger(ceilingBytes: number): {
- ledger: DuplexAggregateByteLedger;
- counts: { rejections: number };
- } {
- const counts = { rejections: 0 };
- const ledger = new DuplexAggregateByteLedger({
- ceilingBytes,
- telemetry: {
- setBytesInUse(): void {},
- recordReservationRejection(): void {
- counts.rejections += 1;
- },
- recordAccountingUnderflow(): void {},
- },
- });
- return { ledger, counts };
- }
-
- it("charges the pre-preface scan buffer while it searches, across many chunks", async () => {
- const { channel, control } = makeFakeChannel();
- const { ledger, counts } = makeCountingLedger(1024 * 1024);
- const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
- capBytes: 4_096,
- timeoutMs: 5_000,
- ledger,
- });
- // Noise, then the preface, arrive as two separate chunks: the scan
- // charges each chunk against the ledger as it grows the buffer, not
- // only once the preface is found.
- const noise = "not-the-preface-yet";
- control.emitData(Buffer.from(noise));
- expect(ledger.bytesInUse).toBe(Buffer.byteLength(noise, "utf8"));
- expect(ledger.liveTokenCount).toBe(1);
- control.emitData(PREFACE);
- expect(await scan.settled).toBe("found");
- // The preface match releases the whole scan buffer (the noise prefix,
- // dropped, plus the preface) and re-charges only the retained preface
- // under the replay owner.
- expect(ledger.bytesInUse).toBe(PREFACE.byteLength);
- expect(ledger.liveTokenCount).toBe(1);
- expect(counts.rejections).toBe(0);
- });
-
- it("fails closed when the aggregate byte ledger refuses the pre-preface scan reservation", async () => {
- const { channel, control } = makeFakeChannel();
- // The ceiling admits nothing: even the 24-octet preface itself cannot
- // reserve, so the scan fails closed before it ever finds a preface.
- const { ledger, counts } = makeCountingLedger(4);
- const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
- capBytes: 4_096,
- timeoutMs: 5_000,
- ledger,
- });
- control.emitData(Buffer.concat([PREFACE, Buffer.from("12345678")]));
- expect(await scan.settled).toBe("missing");
- expect(scan.replayOverflowed()).toBe(false);
- expect(ledger.bytesInUse).toBe(0);
- expect(counts.rejections).toBe(1);
- });
-
it("rejects a single pre-preface chunk over the cap without reserving or concatenating it", async () => {
const { channel, control } = makeFakeChannel();
- // The ceiling is far larger than the cap, so a ledger refusal cannot
- // explain a rejection here: only the cap check on the chunk's
- // prospective length can.
- const { ledger, counts } = makeCountingLedger(1024 * 1024);
const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
capBytes: 64,
timeoutMs: 5_000,
- ledger,
});
// One chunk, larger than the cap on its own, with no preface inside it.
// The scan must reject it on its prospective length before the
- // `Buffer.concat` allocation and before the ledger reservation, so
- // nothing here ever charges the ledger.
+ // `Buffer.concat` allocation.
control.emitData(Buffer.from("x".repeat(128)));
expect(await scan.settled).toBe("missing");
expect(scan.replayOverflowed()).toBe(false);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(counts.rejections).toBe(0);
});
it("rejects a pre-preface chunk that tips an already-buffered scan past the cap", async () => {
const { channel, control } = makeFakeChannel();
- const { ledger, counts } = makeCountingLedger(1024 * 1024);
const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
capBytes: 64,
timeoutMs: 5_000,
- ledger,
});
- // The first chunk stays under the cap on its own, so the scan buffers
- // and charges it while it keeps searching.
+ // The first chunk stays under the cap on its own, so the scan buffers it
+ // while it keeps searching.
const firstChunk = "n".repeat(40);
control.emitData(Buffer.from(firstChunk));
- expect(ledger.bytesInUse).toBe(Buffer.byteLength(firstChunk, "utf8"));
// The second chunk, added to the first, passes the cap. The scan must
// reject it before the concat that would grow the buffer past the cap,
// and it must drop the already-buffered first chunk too.
control.emitData(Buffer.from("n".repeat(40)));
expect(await scan.settled).toBe("missing");
expect(scan.replayOverflowed()).toBe(false);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(counts.rejections).toBe(0);
- });
-
- it("charges the post-preface bytes and releases them after the downstream bind", async () => {
- const { channel, control } = makeFakeChannel();
- const { ledger } = makeCountingLedger(1024 * 1024);
- const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
- capBytes: 4_096,
- timeoutMs: 5_000,
- ledger,
- });
- const suffix = "one-http2-frame";
- // The preface and the trailing suffix arrive in one chunk. The scan
- // delivers every byte from the preface onward, inclusive, so the charged
- // and replayed bytes carry the preface itself plus the suffix.
- const fromPreface = Buffer.concat([PREFACE, Buffer.from(suffix)]).toString("utf8");
- control.emitData(Buffer.concat([PREFACE, Buffer.from(suffix)]));
- expect(await scan.settled).toBe("found");
- expect(scan.replayOverflowed()).toBe(false);
- expect(ledger.bytesInUse).toBe(Buffer.byteLength(fromPreface, "utf8"));
- expect(ledger.liveTokenCount).toBe(1);
- // The HTTP/2 server binds and replays the bytes; the scan releases the
- // token after the synchronous handoff.
- const replayed: string[] = [];
- scan.scanned.onData((chunk) => replayed.push(Buffer.from(chunk).toString("utf8")));
- expect(replayed).toEqual([fromPreface]);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
});
it("fails closed and stops the channel when the post-preface buffer floods past the cap", async () => {
const { channel, control } = makeFakeChannel();
- const { ledger } = makeCountingLedger(1024 * 1024);
const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
capBytes: 64,
timeoutMs: 5_000,
- ledger,
});
// The preface arrives alone, so the pending buffer starts empty.
control.emitData(PREFACE);
@@ -6714,8 +6177,6 @@ describe("http2 preface scan post-preface replay buffer", () => {
control.emitData(Buffer.from("x".repeat(128)));
expect(scan.replayOverflowed()).toBe(true);
expect(control.stopCount).toBe(1);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
// Binding the downstream listener replays nothing: the scan dropped the
// buffer on the overflow.
const replayed: string[] = [];
@@ -6723,62 +6184,25 @@ describe("http2 preface scan post-preface replay buffer", () => {
expect(replayed).toEqual([]);
});
- it("fails closed when the aggregate byte ledger refuses the post-preface replay reservation", async () => {
- const { channel, control } = makeFakeChannel();
- // The ceiling admits the 24-octet preface itself, but not an 8-byte
- // suffix on top of it.
- const { ledger, counts } = makeCountingLedger(30);
- const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
- capBytes: 4_096,
- timeoutMs: 5_000,
- ledger,
- });
- // The preface arrives alone, so it is the only bytes charged so far.
- control.emitData(PREFACE);
- expect(await scan.settled).toBe("found");
- expect(scan.replayOverflowed()).toBe(false);
- expect(ledger.bytesInUse).toBe(PREFACE.byteLength);
- // A post-preface suffix, on top of the retained preface, passes the
- // ceiling. The scan drops the buffer, stops the channel, and fails
- // closed — the same shape a missing preface fails closed.
- control.emitData(Buffer.from("12345678"));
- expect(scan.replayOverflowed()).toBe(true);
- expect(control.stopCount).toBe(1);
- expect(ledger.bytesInUse).toBe(0);
- expect(counts.rejections).toBe(1);
- });
-
- it("releases the scan-buffer tokens when the readiness timeout elapses with no preface found", async () => {
+ it("settles missing when the readiness timeout elapses with no preface found", async () => {
const { channel, control } = makeFakeChannel();
- const { ledger } = makeCountingLedger(1024 * 1024);
const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
capBytes: 4_096,
// A short bound, so the test does not wait out a production-sized one.
timeoutMs: 20,
- ledger,
});
- // Partial, non-matching data arrives and stays charged while the scan
- // keeps searching. No preface ever completes, so nothing else in the
- // scan releases this charge — only the timeout path can.
+ // Partial, non-matching data arrives and the scan keeps searching. No
+ // preface ever completes, so only the bound timeout settles the scan.
const partial = "not-a-preface-and-never-will-be";
control.emitData(Buffer.from(partial));
- expect(ledger.bytesInUse).toBe(Buffer.byteLength(partial, "utf8"));
- expect(ledger.liveTokenCount).toBe(1);
- // The bound readiness timeout elapses before a preface ever arrives. The
- // scan must release its held tokens here — the one terminal path that
- // has no cap or ledger refusal of its own to trigger a release.
expect(await scan.settled).toBe("missing");
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
});
it("finds a preface fragmented into many one-byte chunks", async () => {
const { channel, control } = makeFakeChannel();
- const { ledger } = makeCountingLedger(1024 * 1024);
const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
capBytes: 4_096,
timeoutMs: 5_000,
- ledger,
});
// A slow sandbox socket can deliver the preface one byte at a time. The
// scan must still find it and deliver the exact octets, the same as it
@@ -6795,7 +6219,6 @@ describe("http2 preface scan post-preface replay buffer", () => {
it("bounds the pre-preface scan search and growth-copy work by the bytes received, across many one-byte fragments", async () => {
const { channel, control } = makeFakeChannel();
- const { ledger } = makeCountingLedger(1024 * 1024);
__http2PrefaceScanTesting.resetScanSearchUnits();
__http2PrefaceScanTesting.resetScanBufferGrowthCopyUnits();
// An adversarial sandbox sends many one-byte fragments, none of them the
@@ -6808,7 +6231,6 @@ describe("http2 preface scan post-preface replay buffer", () => {
const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
capBytes,
timeoutMs: 5_000,
- ledger,
});
for (let i = 0; i < capBytes; i += 1) {
control.emitData(Buffer.from([0x2e])); // '.', never part of the preface
@@ -6832,13 +6254,11 @@ describe("http2 preface scan post-preface replay buffer", () => {
it("bounds the post-preface replay buffer growth-copy work by the bytes received, across many one-byte fragments, and still enforces the cap", async () => {
const { channel, control } = makeFakeChannel();
- const { ledger } = makeCountingLedger(1024 * 1024);
__http2PrefaceScanTesting.resetReplayBufferGrowthCopyUnits();
const capBytes = 2_048;
const scan = __http2PrefaceScanTesting.scanForHttp2ClientPreface(channel, {
capBytes,
timeoutMs: 5_000,
- ledger,
});
control.emitData(PREFACE);
expect(await scan.settled).toBe("found");
diff --git a/packages/adapter-utils/src/execution-target.ts b/packages/adapter-utils/src/execution-target.ts
index c11463841a..5d08f246fc 100644
--- a/packages/adapter-utils/src/execution-target.ts
+++ b/packages/adapter-utils/src/execution-target.ts
@@ -63,11 +63,6 @@ import {
type DuplexObservabilityRecorder,
type Http2TelemetryEventName,
} from "./duplex-observability.js";
-import {
- DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED,
- type DuplexAggregateByteLedger,
- type ReservationToken,
-} from "./duplex-aggregate-byte-ledger.js";
import { createSshCommandManagedRuntimeRunner, parseSshRemoteExecutionSpec, runSshCommand, shellQuote } from "./ssh.js";
import {
ensureCommandResolvable,
@@ -196,16 +191,6 @@ export interface AdapterSandboxExecutionTarget extends AdapterExecutionTargetWor
* duplex observability surface. Absent means the safe no-op default.
*/
duplexObservabilityRecorder?: DuplexObservabilityRecorder | null;
- /**
- * The process-owned aggregate byte ledger for the sandbox duplex channel. The
- * host stamps this same object on every sandbox target on the same seam as
- * `runner`, so one shared gauge bounds the aggregate retained bytes across all
- * live duplex routes. The live object stays on the host and never enters the
- * sandbox environment. The bridge passes it to the broker, the decoder, and the
- * response-body reader. Absent means no host ledger; a non-duplex run keeps the
- * bridge inert for this seam.
- */
- duplexAggregateByteLedger?: DuplexAggregateByteLedger | null;
}
export type AdapterExecutionTarget =
@@ -450,19 +435,6 @@ export function adapterExecutionTargetDuplexObservabilityRecorder(
: null;
}
-/**
- * Read the injected aggregate byte ledger off a target. Only a sandbox target
- * with a ledger attached returns it. Every other target returns null, so the
- * bridge stays inert for this seam. The reader never makes a fresh ledger, so a
- * host duplex run always uses the one process-owned ledger the host stamped.
- */
-export function adapterExecutionTargetDuplexAggregateByteLedger(
- target: AdapterExecutionTarget | null | undefined,
-): DuplexAggregateByteLedger | null {
- return target?.kind === "remote" && target.transport === "sandbox"
- ? target.duplexAggregateByteLedger ?? null
- : null;
-}
export function adapterExecutionTargetRemoteCwd(
target: AdapterExecutionTarget | null | undefined,
@@ -1505,26 +1477,15 @@ function bridgeResponseBodyLimitError(maxBodyBytes: number): Error {
}
/**
- * Read the forward response body into a string. The reader bounds the body with
- * two controls. The per-request `maxBodyBytes` limit rejects a body larger than
- * the configured per-request ceiling. The optional host aggregate byte ledger
- * bounds the retained bytes across all live routes.
+ * Read the forward response body into a string. The per-request `maxBodyBytes`
+ * limit rejects a body larger than the configured per-request ceiling.
*
- * The reader charges the ledger for every retained buffer before it allocates
- * that buffer. It reserves the exact chunk bytes before it copies a chunk into a
- * retained `Buffer`. It reserves the concatenation buffer before it allocates it.
- * A reservation that would pass the aggregate ceiling returns no token; the
- * reader retains nothing more, cancels the stream reader, and throws the fixed
- * marker {@link DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED}. The `finally` releases
- * every token exactly one time, so the reader charges the retained bytes only
- * while the raw buffers live and never leaves a token held after it returns or
- * throws.
+ * This function reserves no process-wide byte budget: it enforces only the
+ * one request's own ceiling. See the "Known behavior: aggregate retained
+ * body bytes" section in `doc/observability.md` for the accepted aggregate
+ * ceiling this leaves across every concurrent route.
*/
-async function readBridgeForwardResponseBody(
- response: Response,
- maxBodyBytes: number,
- ledger?: DuplexAggregateByteLedger | null,
-): Promise<string> {
+async function readBridgeForwardResponseBody(response: Response, maxBodyBytes: number): Promise<string> {
const rawContentLength = response.headers.get("content-length");
if (rawContentLength) {
const contentLength = Number.parseInt(rawContentLength, 10);
@@ -1539,54 +1500,20 @@ async function readBridgeForwardResponseBody(
const reader = response.body.getReader();
const chunks: Buffer[] = [];
- // Every response-body reservation token the reader holds. The `finally` block
- // releases each token one time, so a return, a size error, an aggregate
- // rejection, and a read error all release every token.
- const tokens: ReservationToken[] = [];
let totalBytes = 0;
- try {
- while (true) {
- const { done, value } = await reader.read();
- if (done) break;
- if (!value) continue;
- const chunkBytes = value.byteLength;
- totalBytes += chunkBytes;
- if (totalBytes > maxBodyBytes) {
- await reader.cancel().catch(() => undefined);
- throw bridgeResponseBodyLimitError(maxBodyBytes);
- }
- // Reserve the exact chunk bytes before the host copies the chunk into a
- // retained buffer. A rejection fails closed: cancel the stream reader and
- // report the fixed marker; the reader retains nothing more.
- if (ledger) {
- const token = ledger.reserve("response_body", chunkBytes);
- if (!token) {
- await reader.cancel().catch(() => undefined);
- throw new Error(DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED);
- }
- tokens.push(token);
- }
- chunks.push(Buffer.from(value));
- }
- // Reserve the concatenation buffer before the reader allocates it. The
- // concatenation buffer is a second copy of the body bytes that lives next to
- // the chunk buffers during the concatenation, so it is the peak retained
- // allocation. A rejection fails closed with the fixed marker.
- if (ledger && totalBytes > 0) {
- const concatToken = ledger.reserve("response_body", totalBytes);
- if (!concatToken) {
- throw new Error(DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED);
- }
- tokens.push(concatToken);
- }
- return Buffer.concat(chunks, totalBytes).toString("utf8");
- } finally {
- if (ledger) {
- for (const token of tokens) {
- ledger.release(token);
- }
+ while (true) {
+ const { done, value } = await reader.read();
+ if (done) break;
+ if (!value) continue;
+ const chunkBytes = value.byteLength;
+ totalBytes += chunkBytes;
+ if (totalBytes > maxBodyBytes) {
+ await reader.cancel().catch(() => undefined);
+ throw bridgeResponseBodyLimitError(maxBodyBytes);
}
+ chunks.push(Buffer.from(value));
}
+ return Buffer.concat(chunks, totalBytes).toString("utf8");
}
const PROCESS_SESSION_PROXY_SCRIPT = "paperclip-process-session-proxy.mjs";
@@ -3008,12 +2935,7 @@ export function buildDuplexGatewayLaunchArgv(input: {
}
/** The reason the duplex readiness handshake did not pass. */
-type DuplexReadinessFailure =
- | "protocol_contamination"
- | "nonce_mismatch"
- | "channel_exit"
- | "timeout"
- | "aggregate_bytes_exceeded";
+type DuplexReadinessFailure = "protocol_contamination" | "nonce_mismatch" | "channel_exit" | "timeout";
/** The outcome of the duplex readiness handshake. */
type DuplexReadinessResult =
@@ -3035,8 +2957,6 @@ function duplexReadinessFallbackReason(reason: DuplexReadinessFailure): DuplexFa
return "ready_timeout";
case "channel_exit":
return "ready_invalid";
- case "aggregate_bytes_exceeded":
- return "aggregate_bytes_exceeded";
}
}
@@ -3187,22 +3107,14 @@ function findPrefaceFrom(buffer: Buffer, from: number): number {
* found, it calls `onMissing` exactly one time and stops buffering, so the
* caller can abort the open and fall back to `queue_v1`. The function holds
* no prologue byte count: it always scans for the fixed 24-octet sequence,
- * never a length. The scan buffer holds untrusted, sandbox-controlled bytes
- * on the same footing as the readiness gate's own pre-READY buffer, so, when
- * the caller supplies a ledger, it charges each received chunk against the
- * process aggregate byte ledger under the `http2_preface_scan` owner before
- * the chunk grows the buffer. A refusal fails closed the same way the cap
- * does: the function drops the buffer and calls `onMissing`.
+ * never a length.
*
* The bytes that follow the found preface, before the HTTP/2 server binds a
* downstream listener, land in `pendingAfterPreface`. This buffer holds
* untrusted bytes on the same footing as the scan buffer, so it carries the
- * same {@link DUPLEX_READINESS_BUFFER_CAP_BYTES} cap and, when the caller
- * supplies a ledger, charges each retained chunk against it under the
- * `http2_preface_replay` owner. A chunk that would pass the cap, or that the
- * ledger refuses, fails closed: the function
- * drops the buffer, releases its ledger tokens, and stops the channel. The
- * caller reads {@link replayOverflowed} after the preface settles and, on
+ * same {@link DUPLEX_READINESS_BUFFER_CAP_BYTES} cap. A chunk that would pass
+ * the cap fails closed: the function drops the buffer and stops the channel.
+ * The caller reads {@link replayOverflowed} after the preface settles and, on
* `true`, treats the open the same as a missing preface.
*/
function createHttp2PrefaceScanningChannel(
@@ -3211,14 +3123,12 @@ function createHttp2PrefaceScanningChannel(
capBytes: number;
onFound: () => void;
onMissing: () => void;
- ledger?: DuplexAggregateByteLedger | null;
},
): {
channel: CommandManagedDuplexChannel;
replayOverflowed: () => boolean;
disposeScanBuffer: () => void;
} {
- const ledger = options.ledger ?? null;
// The pre-preface scan buffer. `scanBuf.append` grows its backing storage
// by doubling, instead of copying the whole retained buffer on every
// fragment — see {@link createGrowableByteBuffer}. `scanSearchFrom` is the
@@ -3240,42 +3150,14 @@ function createHttp2PrefaceScanningChannel(
const pendingAfterPreface = createGrowableByteBuffer((copiedBytes) => {
http2PrefaceReplayBufferGrowthCopyUnits += copiedBytes;
});
- // Every `http2_preface_replay` reservation token this buffer holds. The
- // function releases each token exactly once, on the downstream handoff or
- // on an overflow.
- const replayTokens: ReservationToken[] = [];
- // Every `http2_preface_scan` reservation token the pre-preface scan buffer
- // holds. The scan is untrusted, sandbox-controlled input on the same
- // footing as the readiness gate's own pre-READY buffer, so it charges the
- // ledger the same way: one reservation per received chunk, released in
- // full on the terminal scan outcome — the preface found, the cap passed
- // with no match, or a ledger refusal.
- const scanTokens: ReservationToken[] = [];
let replayOverflow = false;
- function releaseReplayTokens(): void {
- if (!ledger) return;
- for (const token of replayTokens) {
- ledger.release(token);
- }
- replayTokens.length = 0;
- }
-
- function releaseScanTokens(): void {
- if (!ledger) return;
- for (const token of scanTokens) {
- ledger.release(token);
- }
- scanTokens.length = 0;
- }
-
- // Drop the pending buffer, release its tokens, and stop the channel. The
- // caller reads `replayOverflowed()` after the preface settles and falls
- // back the same way it does for a missing preface.
+ // Drop the pending buffer and stop the channel. The caller reads
+ // `replayOverflowed()` after the preface settles and falls back the same
+ // way it does for a missing preface.
function overflowAndStop(): void {
replayOverflow = true;
pendingAfterPreface.reset();
- releaseReplayTokens();
channel.stop();
}
@@ -3289,14 +3171,6 @@ function createHttp2PrefaceScanningChannel(
overflowAndStop();
return;
}
- if (ledger) {
- const token = ledger.reserve("http2_preface_replay", chunk.byteLength);
- if (!token) {
- overflowAndStop();
- return;
- }
- replayTokens.push(token);
- }
pendingAfterPreface.append(chunk);
}
@@ -3318,25 +3192,9 @@ function createHttp2PrefaceScanningChannel(
failed = true;
scanBuf.reset();
scanSearchFrom = 0;
- releaseScanTokens();
options.onMissing();
return;
}
- // Charge this chunk against the aggregate ledger before it grows the
- // scan buffer. A refusal fails closed the same way the cap does: drop
- // the buffer and report a missing preface.
- if (ledger) {
- const token = ledger.reserve("http2_preface_scan", rawChunk.byteLength);
- if (!token) {
- failed = true;
- scanBuf.reset();
- scanSearchFrom = 0;
- releaseScanTokens();
- options.onMissing();
- return;
- }
- scanTokens.push(token);
- }
scanBuf.append(rawChunk);
const scanBuffer = scanBuf.view();
const offset = findPrefaceFrom(scanBuffer, scanSearchFrom);
@@ -3358,12 +3216,6 @@ function createHttp2PrefaceScanningChannel(
const fromPreface = Buffer.from(scanBuffer.subarray(offset));
scanBuf.reset();
scanSearchFrom = 0;
- // Release the scan tokens before `deliver` charges the same bytes under
- // `http2_preface_replay`. The two calls run inside one synchronous
- // callback with no `await` between them, so no other reservation can
- // observe the released state in between; releasing first keeps the
- // ledger's momentary peak at the real retained bytes, not double them.
- releaseScanTokens();
deliver(fromPreface);
});
@@ -3383,15 +3235,6 @@ function createHttp2PrefaceScanningChannel(
pendingAfterPreface.reset();
listener(replay);
}
- // Release every replay token exactly once, after the synchronous
- // handoff to the downstream listener. Order is safe here: the
- // downstream listener is the bound HTTP/2 server duplex, which holds
- // no aggregate-ledger reservation of its own for these bytes, so
- // this release cannot overlap a second reservation for the same
- // bytes. Contrast the readiness gate's own `readiness_replay`
- // handoff, which releases first because its downstream listener (the
- // preface scanner) does take its own reservation for the same bytes.
- releaseReplayTokens();
},
onExit: (listener: (exit: { exitCode: number | null }) => void) => channel.onExit(listener),
stop: () => channel.stop(),
@@ -3399,20 +3242,18 @@ function createHttp2PrefaceScanningChannel(
},
replayOverflowed: () => replayOverflow,
/**
- * Release every held `http2_preface_scan` token and drop the scan
- * buffer, for a caller-side terminal path this function itself never
- * reaches — the bound readiness timeout elapsing while the scan is
- * still searching, with no preface found and no cap or ledger refusal
- * of its own. A call after the preface already matched, or after the
- * cap or the ledger already failed the scan closed, is a no-op: both
- * paths already released the scan tokens themselves.
+ * Drop the scan buffer, for a caller-side terminal path this function
+ * itself never reaches — the bound readiness timeout elapsing while the
+ * scan is still searching, with no preface found and no cap refusal of
+ * its own. A call after the preface already matched, or after the cap
+ * already failed the scan closed, is a no-op: both paths already reset
+ * the scan buffer themselves.
*/
disposeScanBuffer: (): void => {
if (sawPreface || failed) return;
failed = true;
scanBuf.reset();
scanSearchFrom = 0;
- releaseScanTokens();
},
};
}
@@ -3430,12 +3271,12 @@ type Http2PrefaceScanResult = "found" | "missing";
* Returns the scanning channel alongside the settled result, so the caller
* binds the HTTP/2 server to it only on a `found` result. On a `found`
* result, the caller must still read `replayOverflowed()`: the post-preface
- * buffer can overflow its cap or its ledger reservation after the preface
- * settles as `found` and before the caller binds a downstream listener.
+ * buffer can overflow its cap after the preface settles as `found` and
+ * before the caller binds a downstream listener.
*/
function scanForHttp2ClientPreface(
channel: CommandManagedDuplexChannel,
- options: { capBytes: number; timeoutMs: number; ledger?: DuplexAggregateByteLedger | null },
+ options: { capBytes: number; timeoutMs: number },
): {
scanned: CommandManagedDuplexChannel;
settled: Promise<Http2PrefaceScanResult>;
@@ -3457,10 +3298,10 @@ function scanForHttp2ClientPreface(
settledOnce = true;
clearTimeout(timer);
// The bound readiness timeout can elapse while the scan still searches,
- // with no preface found and no cap or ledger refusal of its own. That
- // path holds no other cleanup, so release its `http2_preface_scan`
- // tokens here. A `found` result, or a `missing` result the scan itself
- // already failed closed, is a no-op inside `disposeScanBuffer`.
+ // with no preface found and no cap refusal of its own. That path holds
+ // no other cleanup, so drop the scan buffer here. A `found` result, or a
+ // `missing` result the scan itself already failed closed, is a no-op
+ // inside `disposeScanBuffer`.
if (result === "missing") disposeScanBuffer();
resolveSettled(result);
};
@@ -3470,7 +3311,6 @@ function scanForHttp2ClientPreface(
capBytes: options.capBytes,
onFound: () => settle("found"),
onMissing: () => settle("missing"),
- ledger: options.ledger,
});
disposeScanBuffer = scan.disposeScanBuffer;
return { scanned: scan.channel, settled, replayOverflowed: scan.replayOverflowed };
@@ -3478,13 +3318,13 @@ function scanForHttp2ClientPreface(
/**
* Test-only surface for {@link scanForHttp2ClientPreface}. A test drives the
- * post-preface replay cap and ledger charge across every terminal path
- * without the whole bridge. Production code never reads this export.
+ * post-preface replay cap across every terminal path without the whole
+ * bridge. Production code never reads this export.
*/
export const __http2PrefaceScanTesting = {
scanForHttp2ClientPreface: (
channel: CommandManagedDuplexChannel,
- options: { capBytes: number; timeoutMs: number; ledger?: DuplexAggregateByteLedger | null },
+ options: { capBytes: number; timeoutMs: number },
) => scanForHttp2ClientPreface(channel, options),
readScanSearchUnits: (): number => http2PrefaceScanSearchUnits,
resetScanSearchUnits: (): void => {
@@ -3648,12 +3488,10 @@ export const __duplexReadinessTesting = {
duplexReadinessBufferGrowthCopyUnits = 0;
},
// Build one readiness gate over a supplied channel, so a test can drive the
- // readiness-replay reservation lifecycle across every terminal path without the
- // whole bridge. Production code never reads this factory.
- createReadinessGate: (
- channel: CommandManagedDuplexChannel,
- options: { nonce: string; timeoutMs: number; ledger?: DuplexAggregateByteLedger | null },
- ) => createDuplexReadinessGate(channel, options),
+ // readiness-replay cap lifecycle across every terminal path without the whole
+ // bridge. Production code never reads this factory.
+ createReadinessGate: (channel: CommandManagedDuplexChannel, options: { nonce: string; timeoutMs: number }) =>
+ createDuplexReadinessGate(channel, options),
};
interface DuplexReadinessGate {
@@ -3667,19 +3505,18 @@ interface DuplexReadinessGate {
*/
readonly brokerChannel: CommandManagedDuplexChannel;
/**
- * Report whether a post-READY pre-bind chunk could not reserve its replay bytes
- * against the aggregate ledger. On such a refusal the gate drops the pending
- * replay buffer and stops the channel. The caller reads this after `ready`
- * resolves `ok`, and before it binds the broker. A `true` result means the caller
- * must abandon the broker and select the file bridge with the aggregate marker.
+ * Report whether a post-READY pre-bind chunk tipped the pending replay buffer
+ * past {@link DUPLEX_READINESS_BUFFER_CAP_BYTES}. On such an overflow the gate
+ * drops the pending replay buffer and stops the channel. The caller reads this
+ * after `ready` resolves `ok`, and before it binds the broker.
*/
replayOverflowed(): boolean;
/**
- * Release every held readiness-replay reservation exactly once and drop the
- * pending replay buffer. The caller runs this on a terminal path that abandons
- * the pending replay without a broker handoff: a readiness failure, a replay
- * overflow, or a broker-construction failure. The normal handoff releases the
- * reservation inside `brokerChannel.onData`, so a later call here is a no-op.
+ * Drop the pending replay buffer. The caller runs this on a terminal path that
+ * abandons the pending replay without a broker handoff: a readiness failure, a
+ * replay overflow, or a broker-construction failure. The normal handoff already
+ * drops the buffer inside `brokerChannel.onData`, so a later call here is a
+ * no-op.
*/
disposePendingReplay(): void;
/**
@@ -3696,52 +3533,15 @@ function createDuplexReadinessGate(
options: {
nonce: string;
timeoutMs: number;
- // The one host-process aggregate byte ledger. The gate charges the untrusted
- // pre-READY buffer bytes against it, so a pre-READY flood counts toward the
- // aggregate ceiling across all live routes. A gate with no ledger stays inert
- // for this seam. The gate holds the same object every other host retention
- // site holds, so the aggregate identity holds at this seam.
- ledger?: DuplexAggregateByteLedger | null;
},
): DuplexReadinessGate {
- const ledger = options.ledger ?? null;
let settled = false;
let readyOk = false;
- // Every readiness-buffer reservation token the gate holds for the pre-READY
- // bytes. The gate releases each token one time when it settles or when it
- // accepts READY. On a failed handshake the gate drops the buffer, so the release
- // frees the untrusted bytes. On READY the gate discards the whole pre-READY
- // buffer, then re-charges only the retained suffix under `readiness_replay`.
- const retainedTokens: ReservationToken[] = [];
- // Every readiness-replay reservation token the gate holds for the post-READY
- // suffix and each later pre-bind chunk. The gate releases each token one time
- // after the synchronous handoff to the broker, or on a terminal path that
- // abandons the pending replay without a broker handoff.
- const replayTokens: ReservationToken[] = [];
- // The gate sets this when a post-READY pre-bind chunk cannot reserve its replay
- // bytes. On that refusal the gate drops the pending buffer and stops the channel.
- // The caller reads it through `replayOverflowed` and selects the file bridge.
+ // The gate sets this when a post-READY pre-bind chunk tips the pending replay
+ // buffer past the cap. On that overflow the gate drops the pending buffer and
+ // stops the channel. The caller reads it through `replayOverflowed` and selects
+ // the file bridge.
let replayOverflow = false;
-
- // Release every readiness-buffer token exactly once and clear the registry. A
- // second call is a no-op, because the array is empty.
- function releaseReadinessBufferTokens(): void {
- if (!ledger) return;
- for (const token of retainedTokens) {
- ledger.release(token);
- }
- retainedTokens.length = 0;
- }
-
- // Release every readiness-replay token exactly once and clear the registry. A
- // second call is a no-op, because the array is empty.
- function releaseReplayTokens(): void {
- if (!ledger) return;
- for (const token of replayTokens) {
- ledger.release(token);
- }
- replayTokens.length = 0;
- }
// The raw bytes the host reads before the READY frame completes. `buffer` is
// append-only and always a zero-copy view over the used prefix of `storage`,
// so the O(1) cap check on `buffer.length` stays valid.
@@ -3806,11 +3606,6 @@ function createDuplexReadinessGate(
settled = true;
clearTimeout(timer);
if (result.ok) readyOk = true;
- // Release every readiness-buffer token exactly once. The gate no longer owns
- // the pre-READY bytes: a failed handshake drops the buffer. The READY-accept
- // path already released these tokens and charged the retained suffix under
- // `readiness_replay`, so this call is a no-op there.
- releaseReadinessBufferTokens();
resolveReady(result);
}
@@ -3820,23 +3615,20 @@ function createDuplexReadinessGate(
return;
}
if (readyOk) {
- // READY already passed; hold the bytes until the broker binds. Reserve the
- // exact UTF-8 bytes under `readiness_replay` before the append, so the replay
- // buffer counts toward the aggregate ceiling. A refusal fails closed: the gate
- // drops the pending buffer, releases the replay tokens, stops the channel, and
- // sets the overflow flag. The caller reads the flag and selects the file bridge
- // with the aggregate marker, because `ready` already resolved before this
- // synchronous post-READY chunk arrived.
- if (ledger) {
- const token = ledger.reserve("readiness_replay", chunk.byteLength);
- if (!token) {
- replayOverflow = true;
- pending = READINESS_EMPTY_BUFFER;
- releaseReplayTokens();
- channel.stop();
- return;
- }
- replayTokens.push(token);
+ // READY already passed; hold the bytes until the broker binds. Bound this
+ // buffer directly against {@link DUPLEX_READINESS_BUFFER_CAP_BYTES}, the
+ // same way the preface-scan gate's own post-preface replay buffer bounds
+ // itself: a worker that keeps sending bytes after READY, faster than the
+ // broker can bind, cannot grow this buffer past the cap. A chunk that
+ // would pass the cap fails closed: the gate drops the pending buffer,
+ // stops the channel, and sets the overflow flag. The caller reads the
+ // flag and selects the file bridge, because `ready` already resolved
+ // before this synchronous post-READY chunk arrived.
+ if (pending.length + chunk.byteLength > DUPLEX_READINESS_BUFFER_CAP_BYTES) {
+ replayOverflow = true;
+ pending = READINESS_EMPTY_BUFFER;
+ channel.stop();
+ return;
}
// Copy a first chunk instead of aliasing the caller's `Uint8Array`, so a
// channel that reuses its delivered buffer across calls cannot corrupt the
@@ -3850,18 +3642,6 @@ function createDuplexReadinessGate(
// never grows the buffer after the gate settles.
return;
}
- // Reserve the exact bytes of this chunk against the aggregate ledger before
- // the gate retains it. The pre-READY buffer holds untrusted bytes, so a
- // flood counts toward the process aggregate ceiling. A rejection fails
- // closed: the gate retains nothing more and falls back to the file bridge.
- if (ledger) {
- const token = ledger.reserve("readiness_buffer", chunk.byteLength);
- if (!token) {
- finish({ ok: false, reason: "aggregate_bytes_exceeded" });
- return;
- }
- retainedTokens.push(token);
- }
// Append the new bytes and continue the newline search from `scanFrom`, the
// first index not yet examined. Each byte is read at most one time for the
// search, and `appendReadinessBytes` copies at most the incoming chunk, so
@@ -3933,40 +3713,21 @@ function createDuplexReadinessGate(
return;
}
// The bytes that follow the READY line become the replay buffer for the
- // broker. Drop the whole pre-READY buffer charge first, then reserve the
- // retained suffix under `readiness_replay`. The release-before-reserve order
- // keeps the transient charge equal to the suffix, not the sum of the dropped
- // prefix and the retained suffix. The two steps run in one synchronous
- // section, so no other route can take the freed bytes in between.
+ // broker.
//
// Copy the suffix instead of slicing it off `buffer`. `buffer` is a view
// over `storage`, and `storage`'s capacity can run ahead of the bytes in
// use (the doubling growth in `appendReadinessBytes` over-provisions it).
- // A slice would keep that whole over-provisioned allocation alive, so the
- // process would retain more physical bytes than the ledger charges under
- // `readiness_replay`. The copy is exactly `suffix.length` bytes, one time,
- // not a per-fragment cost.
+ // A slice would keep that whole over-provisioned allocation alive for as
+ // long as the broker replay holds its reference. The copy is exactly
+ // `suffix.length` bytes, one time, not a per-fragment cost.
const suffix = Buffer.from(buffer.subarray(newlineIndex + 1));
// Drop the original pre-READY buffer and its backing storage now. The
- // gate keeps only the retained suffix as `pending`, and it charges that
- // suffix under `readiness_replay` below. If the gate keeps `storage`, the
- // process retains the full sandbox-controlled bytes while the ledger
- // counts only the suffix, so aggregate retention passes the ceiling. This
- // clear also covers the broker handoff and the replay disposal. Both run
- // later and read no buffer bytes.
+ // gate keeps only the retained suffix as `pending`. This clear also
+ // covers the broker handoff and the replay disposal. Both run later and
+ // read no buffer bytes.
buffer = READINESS_EMPTY_BUFFER;
storage = READINESS_EMPTY_BUFFER;
- releaseReadinessBufferTokens();
- if (ledger && suffix.length > 0) {
- const token = ledger.reserve("readiness_replay", suffix.byteLength);
- if (!token) {
- // The retained suffix passes the aggregate ceiling. Fail closed: drop
- // the suffix and fall back to the file bridge with the aggregate marker.
- finish({ ok: false, reason: "aggregate_bytes_exceeded" });
- return;
- }
- replayTokens.push(token);
- }
pending = suffix;
finish({ ok: true });
return;
@@ -4005,19 +3766,9 @@ function createDuplexReadinessGate(
if (pending.length > 0) {
const replay = pending;
pending = READINESS_EMPTY_BUFFER;
- // Release every readiness-replay token before the synchronous handoff
- // to the broker, not after. The broker (the HTTP/2 preface scanner)
- // charges its own reservation for these same bytes inside
- // `listener(replay)` below, under a different owner. Releasing first
- // keeps the ledger's momentary peak at the real retained bytes, not
- // double them: the release and the broker's reserve both run inside
- // this one synchronous call, with no `await` between them, so no
- // other route can claim the freed capacity in between.
- releaseReplayTokens();
listener(replay);
return;
}
- releaseReplayTokens();
},
onExit: (listener: (exit: { exitCode: number | null }) => void) => {
exitSink = listener;
@@ -4037,7 +3788,6 @@ function createDuplexReadinessGate(
replayOverflowed: () => replayOverflow,
disposePendingReplay: () => {
pending = READINESS_EMPTY_BUFFER;
- releaseReplayTokens();
},
retainedReadinessBufferLength: () => buffer.length,
};
@@ -4122,12 +3872,6 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: {
}
const target = input.target;
- // The process-owned aggregate byte ledger the host stamped on this sandbox
- // target. The forward response-body reader charges its retained bytes against
- // this one ledger, so the aggregate retained bytes across all live routes stay
- // under the ceiling. A target with no ledger keeps the reader inert for this
- // seam.
- const duplexAggregateByteLedger = adapterExecutionTargetDuplexAggregateByteLedger(target);
const onLog = input.onLog ?? (async () => {});
const hostApiToken = input.hostApiToken?.trim() ?? "";
if (hostApiToken.length === 0) {
@@ -4271,11 +4015,7 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: {
// to a non-retryable 409 for both the file bridge and the duplex broker.
let responseBody: string;
try {
- responseBody = await readBridgeForwardResponseBody(
- response,
- maxBodyBytes,
- duplexAggregateByteLedger,
- );
+ responseBody = await readBridgeForwardResponseBody(response, maxBodyBytes);
} catch (error) {
if (isSafeBridgeMethod(method)) {
// The method is safe, so a retry cannot double-apply a mutation. Return a
@@ -4416,10 +4156,6 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: {
const gate = createDuplexReadinessGate(channel, {
nonce,
timeoutMs: readinessTimeoutMs,
- // Inject the one host-process aggregate byte ledger, the same object the
- // broker and the response-body reader hold. The gate charges the untrusted
- // pre-READY buffer against it, so the aggregate identity holds at this seam.
- ledger: duplexAggregateByteLedger,
});
const readiness = await gate.ready;
if (!readiness.ok) {
@@ -4435,19 +4171,6 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: {
"stderr",
`[paperclip] Sandbox duplex readiness failed (${readiness.reason}). Using the file bridge.\n`,
);
- } else if (gate.replayOverflowed()) {
- // Readiness passed, but a post-READY pre-bind chunk passed the aggregate
- // byte ceiling. The gate dropped the replay buffer and stopped the channel.
- // Release any held replay reservation, close the partial channel inside the
- // cleanup budget, and select the file bridge with the aggregate marker. The
- // broker never bound, so no request reached the channel or any endpoint.
- gate.disposePendingReplay();
- await closeDuplexChannelWithinBudget(channel, DEFAULT_DUPLEX_CLEANUP_BUDGET_MS);
- duplexChannelOpen.fallback(duplexReadinessFallbackReason("aggregate_bytes_exceeded"));
- await onLog(
- "stderr",
- "[paperclip] Sandbox duplex readiness replay exceeded the aggregate byte ceiling (aggregate_bytes_exceeded). Using the file bridge.\n",
- );
} else {
// Readiness passed. The gate retained every byte that followed the
// accepted READY line. Scan those retained bytes for the HTTP/2
@@ -4460,32 +4183,20 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: {
const prefaceScan = scanForHttp2ClientPreface(gate.brokerChannel, {
capBytes: DUPLEX_READINESS_BUFFER_CAP_BYTES,
timeoutMs: readinessTimeoutMs,
- // Inject the same host-process aggregate byte ledger the readiness
- // gate charges, so the post-preface pre-bind buffer counts toward
- // the same aggregate ceiling.
- ledger: duplexAggregateByteLedger,
});
const prefaceResult = await prefaceScan.settled;
- if (prefaceResult === "missing" || prefaceScan.replayOverflowed()) {
+ if (prefaceResult === "missing") {
// Fail closed, the same shape as a readiness failure: close the
// partial channel inside the cleanup budget, then select the file
// bridge. No HTTP/2 server ever bound to this channel, so no
// request reached it or any endpoint.
gate.disposePendingReplay();
await closeDuplexChannelWithinBudget(openedChannel, DEFAULT_DUPLEX_CLEANUP_BUDGET_MS);
- if (prefaceResult === "missing") {
- duplexChannelOpen.fallback("preface_missing");
- await onLog(
- "stderr",
- "[paperclip] Sandbox HTTP/2 client preface did not appear inside the bounded readiness buffer (preface_missing). Using the file bridge.\n",
- );
- } else {
- duplexChannelOpen.fallback("aggregate_bytes_exceeded");
- await onLog(
- "stderr",
- "[paperclip] Sandbox HTTP/2 post-preface buffer exceeded the aggregate byte ceiling (aggregate_bytes_exceeded). Using the file bridge.\n",
- );
- }
+ duplexChannelOpen.fallback("preface_missing");
+ await onLog(
+ "stderr",
+ "[paperclip] Sandbox HTTP/2 client preface did not appear inside the bounded readiness buffer (preface_missing). Using the file bridge.\n",
+ );
} else {
// The run disposition latch for the http2_v1 path, in the same
// shape the retired duplex_v1 broker exposed. A loss ordered before
@@ -4532,7 +4243,7 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: {
headers: request.headers,
body: request.body.toString("utf8"),
},
- undefined,
+ request.signal,
{ suppressDebugLog: true },
);
duplexObservability.recordRequest({ latencyMs: Date.now() - dispatchStartMs, outcome: "ok" });
diff --git a/packages/adapter-utils/src/http2-bridge-server.test.ts b/packages/adapter-utils/src/http2-bridge-server.test.ts
index 113aa212e4..5613620a38 100644
--- a/packages/adapter-utils/src/http2-bridge-server.test.ts
+++ b/packages/adapter-utils/src/http2-bridge-server.test.ts
@@ -10,6 +10,7 @@ import {
createHttp2BridgeServer,
parseCanonicalBridgeRequestPath,
wrapDuplexChannelAsNodeDuplex,
+ DEFAULT_HTTP2_BRIDGE_MAX_BUFFERED_READ_BYTES,
DEFAULT_HTTP2_BRIDGE_PING_INTERVAL_MS,
DEFAULT_HTTP2_BRIDGE_PING_STALL_MS,
HTTP2_BRIDGE_ENABLE_PUSH,
@@ -28,7 +29,10 @@ import {
type Http2BridgeForwardResult,
type Http2BridgeGoawayRecord,
} from "./http2-bridge-server.js";
-import { createSandboxHttp2BridgeGateway } from "./sandbox-callback-bridge.js";
+import {
+ createSandboxHttp2BridgeGateway,
+ DEFAULT_SANDBOX_CALLBACK_BRIDGE_MAX_BODY_BYTES,
+} from "./sandbox-callback-bridge.js";
import type { CommandManagedDuplexChannel } from "./command-managed-runtime.js";
/**
@@ -346,6 +350,185 @@ describe("createHttp2BridgeServer + createSandboxHttp2BridgeGateway", () => {
}
});
+ it("test_the_server_aborts_the_forward_when_the_client_closes_the_stream", async () => {
+ let capturedRequest: Http2BridgeForwardRequest | undefined;
+ let releaseForward: (() => void) | undefined;
+ const forwardHeld = new Promise<void>((resolve) => {
+ releaseForward = resolve;
+ });
+ const { handle, bridgeToken, clientSide } = bindTestServer({
+ forwardRequest: async (request) => {
+ capturedRequest = request;
+ // Hold the handler open, so the test controls exactly when the
+ // client-side stream close happens relative to the forward.
+ await forwardHeld;
+ return { status: 200, body: "{}" };
+ },
+ });
+ const rawClient = connectRawClient(clientSide);
+ try {
+ const stream = rawClient.request({
+ ":method": "GET",
+ ":path": "/api/agents/me",
+ authorization: `Bearer ${bridgeToken}`,
+ });
+ const streamClosed = new Promise<void>((resolve) => {
+ stream.on("error", () => resolve());
+ stream.on("close", () => resolve());
+ });
+ stream.end();
+ // Give the request time to reach the server and enter forwardRequest.
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ expect(capturedRequest).toBeDefined();
+ expect(capturedRequest!.signal.aborted).toBe(false);
+ stream.close(http2.constants.NGHTTP2_CANCEL);
+ await streamClosed;
+ // Give the server's own stream `close` listener a turn to run.
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ expect(capturedRequest!.signal.aborted).toBe(true);
+ releaseForward!();
+ } finally {
+ rawClient.close();
+ await handle.close();
+ }
+ });
+
+ it("test_one_aborted_stream_leaves_the_session_and_the_other_streams_open", async () => {
+ const signalsByPath = new Map<string, AbortSignal>();
+ let releaseSurvivor: (() => void) | undefined;
+ const survivorHeld = new Promise<void>((resolve) => {
+ releaseSurvivor = resolve;
+ });
+ const { handle, bridgeToken, clientSide } = bindTestServer({
+ forwardRequest: async (request) => {
+ signalsByPath.set(request.pathname, request.signal);
+ if (request.pathname === "/api/agents/aborted") {
+ // The RST — not a manual release — is what must end this forward:
+ // it settles only when the stream's own signal fires, so the test
+ // proves the signal itself unblocks the handler.
+ await new Promise<void>((_resolve, reject) => {
+ request.signal.addEventListener("abort", () => reject(new Error("aborted")), {
+ once: true,
+ });
+ }).catch(() => undefined);
+ } else if (request.pathname === "/api/agents/survivor") {
+ await survivorHeld;
+ }
+ return { status: 200, body: JSON.stringify({ path: request.pathname }) };
+ },
+ });
+ const rawClient = connectRawClient(clientSide);
+ try {
+ const abortedStream = rawClient.request({
+ ":method": "GET",
+ ":path": "/api/agents/aborted",
+ authorization: `Bearer ${bridgeToken}`,
+ });
+ const survivorStream = rawClient.request({
+ ":method": "GET",
+ ":path": "/api/agents/survivor",
+ authorization: `Bearer ${bridgeToken}`,
+ });
+ const abortedStreamClosed = new Promise<void>((resolve) => {
+ abortedStream.on("error", () => resolve());
+ abortedStream.on("close", () => resolve());
+ });
+ // Capture the survivor stream's own response, so the test proves this
+ // exact stream — the sibling of the aborted one, on the same session —
+ // completes normally, not merely that the session admits a fresh one.
+ const survivorResponse = new Promise<{ status: number; body: string }>((resolve, reject) => {
+ let status = 0;
+ let body = "";
+ survivorStream.setEncoding("utf8");
+ survivorStream.on("response", (h) => {
+ status = Number(h[":status"]) || 0;
+ });
+ survivorStream.on("data", (chunk) => (body += chunk));
+ survivorStream.on("end", () => resolve({ status, body }));
+ survivorStream.on("error", reject);
+ });
+ abortedStream.end();
+ survivorStream.end();
+ // Give both requests time to reach the server and enter forwardRequest.
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ abortedStream.close(http2.constants.NGHTTP2_CANCEL);
+ await abortedStreamClosed;
+ // Give the server's own stream `close` listener a turn to run.
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ // The one aborted stream's signal fires. The other, still-open stream's
+ // signal stays untouched: the abort is local to its own stream.
+ expect(signalsByPath.get("/api/agents/aborted")?.aborted).toBe(true);
+ expect(signalsByPath.get("/api/agents/survivor")?.aborted).toBe(false);
+ releaseSurvivor!();
+
+ const response = await survivorResponse;
+ expect(response.status).toBe(200);
+ expect(JSON.parse(response.body)).toEqual({ path: "/api/agents/survivor" });
+ } finally {
+ rawClient.close();
+ await handle.close();
+ }
+ });
+
+ it("test_live_forward_work_never_passes_the_stream_limit", async () => {
+ // A forward that is never aborted holds open until the caller aborts it
+ // — this stands in for a forward stuck at its own long timeout. Only the
+ // per-stream abort binding this file adds can free such a forward before
+ // that timeout, so this count proves the binding, not merely the HTTP/2
+ // session's own stream-slot accounting (a stream's protocol slot frees on
+ // RST regardless of whether its forward call ever settles).
+ let liveForwards = 0;
+ let maxLiveForwards = 0;
+ const { handle, bridgeToken, clientSide } = bindTestServer({
+ forwardRequest: async (request) => {
+ liveForwards += 1;
+ maxLiveForwards = Math.max(maxLiveForwards, liveForwards);
+ try {
+ await new Promise<void>((resolve, reject) => {
+ if (request.signal.aborted) {
+ reject(new Error("aborted"));
+ return;
+ }
+ request.signal.addEventListener("abort", () => reject(new Error("aborted")), { once: true });
+ });
+ return { status: 200, body: JSON.stringify({ path: request.pathname }) };
+ } finally {
+ liveForwards -= 1;
+ }
+ },
+ });
+ const rawClient = connectRawClient(clientSide);
+ try {
+ const openAndCancelOneBatch = async (label: string) => {
+ const streams = Array.from({ length: HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS }, (_, index) =>
+ rawClient.request({
+ ":method": "GET",
+ ":path": `/api/issues/${label}-${index}`,
+ authorization: `Bearer ${bridgeToken}`,
+ }),
+ );
+ for (const stream of streams) stream.end();
+ await new Promise((resolve) => setTimeout(resolve, 30));
+ for (const stream of streams) stream.close(http2.constants.NGHTTP2_CANCEL);
+ await new Promise((resolve) => setTimeout(resolve, 30));
+ };
+
+ // One full batch of maximum-concurrency streams, closed while their
+ // forwards are still held open, then a second full batch opened right
+ // after. Without the abort binding, the first batch's forwards would
+ // still be alive when the second batch dispatches, doubling the live-
+ // forward count past the stream limit.
+ await openAndCancelOneBatch("first");
+ await openAndCancelOneBatch("second");
+
+ expect(maxLiveForwards).toBeLessThanOrEqual(HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS);
+ expect(liveForwards).toBe(0);
+ } finally {
+ rawClient.close();
+ await handle.close();
+ }
+ });
+
it("test_a_stalled_request_body_settles_instead_of_hanging_forever", async () => {
const forwarderTracker = createForwarderCallTracker();
const { handle, bridgeToken, clientSide } = bindTestServer({
@@ -630,7 +813,7 @@ describe("createHttp2BridgeServer + createSandboxHttp2BridgeGateway", () => {
streamResetBurst: HTTP2_BRIDGE_STREAM_RESET_BURST,
});
expect(HTTP2_BRIDGE_ENABLE_PUSH).toBe(false);
- expect(HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS).toBe(64);
+ expect(HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS).toBe(4);
expect(HTTP2_BRIDGE_MAX_HEADER_LIST_SIZE).toBe(16384);
expect(HTTP2_BRIDGE_HEADER_TABLE_SIZE).toBe(4096);
expect(HTTP2_BRIDGE_MAX_SESSION_MEMORY).toBe(16);
@@ -661,6 +844,20 @@ describe("createHttp2BridgeServer + createSandboxHttp2BridgeGateway", () => {
}
});
+ it("test_the_host_body_budget_matches_the_stream_limit", () => {
+ // The multiplier counts every retained `Buffer` and string copy of one
+ // live forward's request and response body: four exact `Buffer` rows,
+ // plus two string rows. Each string row applies two bytes to each UTF-16
+ // code unit of the body limit (`sandbox-callback-bridge.ts`), for an
+ // accounting peak of eight times the body limit for one live forward.
+ // `test_live_forward_work_never_passes_the_stream_limit` proves the
+ // count of live forwards never passes `HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS`,
+ // so this multiplier bounds live forwards, not merely open streams.
+ expect(
+ HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS * 8 * DEFAULT_SANDBOX_CALLBACK_BRIDGE_MAX_BODY_BYTES,
+ ).toBe(8_388_608);
+ });
+
describe("parseCanonicalBridgeRequestPath", () => {
it("parses an origin-form path with a query exactly one time", () => {
const result = parseCanonicalBridgeRequestPath({ ":path": "/api/issues/abc?foo=bar" });
@@ -979,6 +1176,36 @@ describe("createHttp2BridgeServer + createSandboxHttp2BridgeGateway", () => {
expect(stopped).toBe(true);
});
+ it("stops the channel when the read queue passes the lowered byte bound", async () => {
+ let dataListener: ((chunk: Uint8Array) => void) | undefined;
+ let stopped = false;
+ const channel: CommandManagedDuplexChannel = {
+ write: () => undefined,
+ onData: (listener) => {
+ dataListener = listener;
+ },
+ onExit: () => undefined,
+ stop: () => {
+ stopped = true;
+ },
+ close: async () => undefined,
+ };
+ // No override: this test proves the default bound itself is the direct
+ // 524,288-byte value, no longer derived from `HTTP2_BRIDGE_MAX_SESSION_MEMORY`.
+ // No consumer ever attaches, so the readable side never drains.
+ expect(DEFAULT_HTTP2_BRIDGE_MAX_BUFFERED_READ_BYTES).toBe(524_288);
+ const duplex = wrapDuplexChannelAsNodeDuplex(channel);
+ const errored = new Promise<Error>((resolve) => duplex.on("error", resolve));
+
+ dataListener?.(Buffer.alloc(70_000, "a")); // passes the default high-water mark; still pushed directly.
+ dataListener?.(Buffer.alloc(500_000, "b")); // queues most of the 524,288-byte default cap.
+ dataListener?.(Buffer.alloc(60_000, "b")); // passes the default cap.
+
+ const error = await errored;
+ expect(error.message).toMatch(/backpressure/i);
+ expect(stopped).toBe(true);
+ });
+
it("wrapDuplexChannelAsNodeDuplex fails closed on one inbound chunk larger than the bounded read backpressure buffer, before the queue holds anything to compare it against", async () => {
let dataListener: ((chunk: Uint8Array) => void) | undefined;
let stopped = false;
diff --git a/packages/adapter-utils/src/http2-bridge-server.ts b/packages/adapter-utils/src/http2-bridge-server.ts
index a42afa1703..0d07fc5ae9 100644
--- a/packages/adapter-utils/src/http2-bridge-server.ts
+++ b/packages/adapter-utils/src/http2-bridge-server.ts
@@ -48,8 +48,29 @@ import {
/** Server push. The transport never needs it. */
export const HTTP2_BRIDGE_ENABLE_PUSH = false;
-/** Open streams. This matches the current broker limit ({@link DEFAULT_DUPLEX_BROKER_MAX_IN_FLIGHT_REQUESTS} in `duplex-bridge-broker.ts`). */
-export const HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS = 64;
+/**
+ * Open streams. The host keeps one forward, its request body, and its
+ * response body alive for the life of a stream, and — before this file binds
+ * each forward to its own stream's abort signal — a forward can outlive its
+ * stream's own HTTP/2 slot until the forward's own timeout runs out. Counting
+ * every retained `Buffer` and string copy of one stream's request and
+ * response body against the {@link DEFAULT_SANDBOX_CALLBACK_BRIDGE_MAX_BODY_BYTES}
+ * body limit (`sandbox-callback-bridge.ts`) gives an accounting peak of eight
+ * times that limit for one live forward. This bound is the per-route
+ * in-flight-body budget: `HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS * 8 *
+ * DEFAULT_SANDBOX_CALLBACK_BRIDGE_MAX_BODY_BYTES` bytes = 4 * 8 * 262,144
+ * bytes = 8,388,608 bytes for one route.
+ *
+ * Known aggregate behavior: this budget applies to one route only. The host
+ * process admits up to `DEFAULT_MAX_CONCURRENT_DUPLEX_ROUTES` (128, in
+ * `plugin-worker-manager.ts`) routes at the same time, and each route holds
+ * its own 8,388,608-byte peak. The process can therefore retain up to
+ * 1,073,741,824 bytes (1 GiB) of live body data across every route at once.
+ * This document accepts that ceiling: the host tracks no process-wide byte
+ * total, so no single route can starve another route's own budget, but the
+ * host also enforces no smaller sum across every route.
+ */
+export const HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS = 4;
/** One decompressed header list. The Node default is 65535. */
export const HTTP2_BRIDGE_MAX_HEADER_LIST_SIZE = 16384;
/** The header-compression table. This keeps the Node default. */
@@ -105,8 +126,12 @@ export const HTTP2_BRIDGE_SERVER_OPTIONS: http2.ServerOptions = {
* bound. This cap also bounds one single chunk: the wrapper checks a chunk's
* own size against it before `push()` ever runs, so one oversized chunk
* cannot cross the cap on its first delivery, before the queue holds
- * anything to compare it against. */
-export const DEFAULT_HTTP2_BRIDGE_MAX_BUFFERED_READ_BYTES = HTTP2_BRIDGE_MAX_SESSION_MEMORY * 1024 * 1024;
+ * anything to compare it against. This is a fixed share of the fixed
+ * per-route byte budget the host bounds every duplex retention site
+ * against; it no longer derives from {@link HTTP2_BRIDGE_MAX_SESSION_MEMORY},
+ * which bounds the underlying `Http2Session`'s own memory, not this
+ * read-side queue. */
+export const DEFAULT_HTTP2_BRIDGE_MAX_BUFFERED_READ_BYTES = 524_288;
/** The default bound, in milliseconds, on how long the read-side queue
* {@link wrapDuplexChannelAsNodeDuplex} holds can stay non-empty with no
* chunk draining from it. The byte cap above bounds how much memory a stuck
@@ -505,6 +530,14 @@ export interface Http2BridgeForwardRequest {
query: string;
headers: Record<string, string>;
body: Buffer;
+ /**
+ * The abort signal for this one HTTP/2 stream. `handleStream` aborts it
+ * when the stream closes, aborts, or errors, so a caller that passes it
+ * through to its own outbound call (a `fetch`, for example) ends that call
+ * at once instead of leaving it to run until its own timeout. The signal
+ * never fires for any other stream or for the session.
+ */
+ signal: AbortSignal;
}
export type Http2BridgeForwardHandler = (
@@ -759,71 +792,94 @@ export function createHttp2BridgeServer(options: CreateHttp2BridgeServerOptions)
stream: http2.ServerHttp2Stream,
headers: http2.IncomingHttpHeaders,
): Promise<void> {
- // Accepted security fix 4: the constant-time bridge-token compare runs
- // before route processing and before header processing. This host check
- // is independent of the gateway's own token check on the sandbox side.
- if (!compareBridgeTokensConstantTime(options.bridgeToken, readBridgeTokenHeader(headers))) {
- denyRequest(stream, 401, { error: "Invalid bridge token." }, bodyBounds);
- return;
- }
+ // One `AbortController` for this one stream. `forwardRequest` below
+ // receives its signal, so a stream that closes, aborts, or errors ends
+ // its own forward at once instead of leaving the forward to run until its
+ // own timeout. The abort stays local to this one stream: it never
+ // touches the session or any other stream's controller.
+ const controller = new AbortController();
+ const abortForThisStream = (): void => {
+ if (!controller.signal.aborted) controller.abort();
+ };
+ stream.once("close", abortForThisStream);
+ stream.once("aborted", abortForThisStream);
+ stream.once("error", abortForThisStream);
+ try {
+ // Accepted security fix 4: the constant-time bridge-token compare runs
+ // before route processing and before header processing. This host check
+ // is independent of the gateway's own token check on the sandbox side.
+ if (!compareBridgeTokensConstantTime(options.bridgeToken, readBridgeTokenHeader(headers))) {
+ denyRequest(stream, 401, { error: "Invalid bridge token." }, bodyBounds);
+ return;
+ }
- // Accepted security fix 3: parse `:path` exactly one time; both the route
- // allowlist and the forward request below read this one result.
- const parsedPath = parseCanonicalBridgeRequestPath(headers);
- if (!parsedPath.ok) {
- denyRequest(stream, 400, { error: `Invalid request path: ${parsedPath.reason}` }, bodyBounds);
- return;
- }
- const method = normalizeStreamMethod(headers[":method"]);
-
- const denialReason = authorizeSandboxCallbackBridgeRequestWithRoutes(
- { method, path: parsedPath.value.pathname },
- routes,
- );
- if (denialReason) {
- denyRequest(stream, 403, { error: denialReason }, bodyBounds);
- return;
- }
+ // Accepted security fix 3: parse `:path` exactly one time; both the route
+ // allowlist and the forward request below read this one result.
+ const parsedPath = parseCanonicalBridgeRequestPath(headers);
+ if (!parsedPath.ok) {
+ denyRequest(stream, 400, { error: `Invalid request path: ${parsedPath.reason}` }, bodyBounds);
+ return;
+ }
+ const method = normalizeStreamMethod(headers[":method"]);
- const sanitizedHeaders = sanitizeSandboxCallbackBridgeHeaders(
- toOutboundHeaderRecord(headers),
- headerAllowlist,
- );
+ const denialReason = authorizeSandboxCallbackBridgeRequestWithRoutes(
+ { method, path: parsedPath.value.pathname },
+ routes,
+ );
+ if (denialReason) {
+ denyRequest(stream, 403, { error: denialReason }, bodyBounds);
+ return;
+ }
- let body: Buffer;
- try {
- body = await readHttp2StreamBody(stream, bodyBounds);
- } catch (error) {
- respondJson(stream, 413, { error: error instanceof Error ? error.message : String(error) });
- return;
- }
+ const sanitizedHeaders = sanitizeSandboxCallbackBridgeHeaders(
+ toOutboundHeaderRecord(headers),
+ headerAllowlist,
+ );
- let result: Http2BridgeForwardResult;
- try {
- result = await options.forwardRequest({
- method,
- pathname: parsedPath.value.pathname,
- query: parsedPath.value.query,
- headers: sanitizedHeaders,
- body,
- });
- } catch (error) {
- respondJson(stream, 502, { error: error instanceof Error ? error.message : String(error) });
- return;
- }
+ let body: Buffer;
+ try {
+ body = await readHttp2StreamBody(stream, bodyBounds);
+ } catch (error) {
+ respondJson(stream, 413, { error: error instanceof Error ? error.message : String(error) });
+ return;
+ }
- if (stream.destroyed || stream.closed) return;
- const responseHeaders: http2.OutgoingHttpHeaders = { ":status": result.status };
- for (const [key, value] of Object.entries(result.headers ?? {})) {
- if (key.toLowerCase() === "content-length") continue;
- responseHeaders[key] = value;
- }
- try {
- stream.respond(responseHeaders);
- stream.end(result.body);
- } catch {
- // The peer reset the stream (RST_STREAM) between dispatch and response.
- // One stream's write fault stays local to that stream.
+ let result: Http2BridgeForwardResult;
+ try {
+ result = await options.forwardRequest({
+ method,
+ pathname: parsedPath.value.pathname,
+ query: parsedPath.value.query,
+ headers: sanitizedHeaders,
+ body,
+ signal: controller.signal,
+ });
+ } catch (error) {
+ respondJson(stream, 502, { error: error instanceof Error ? error.message : String(error) });
+ return;
+ }
+
+ if (stream.destroyed || stream.closed) return;
+ const responseHeaders: http2.OutgoingHttpHeaders = { ":status": result.status };
+ for (const [key, value] of Object.entries(result.headers ?? {})) {
+ if (key.toLowerCase() === "content-length") continue;
+ responseHeaders[key] = value;
+ }
+ try {
+ stream.respond(responseHeaders);
+ stream.end(result.body);
+ } catch {
+ // The peer reset the stream (RST_STREAM) between dispatch and response.
+ // One stream's write fault stays local to that stream.
+ }
+ } finally {
+ // Every path above reaches this exactly once: a deny, a body-read
+ // fault, a forward fault, or a completed response. A completed stream
+ // must leak no listener; the session, not this one stream, outlives
+ // the handler.
+ stream.removeListener("close", abortForThisStream);
+ stream.removeListener("aborted", abortForThisStream);
+ stream.removeListener("error", abortForThisStream);
}
}
diff --git a/scripts/general-server-shard-durations.json b/scripts/general-server-shard-durations.json
index 9f8ff9e4fe..8f8885be24 100644
--- a/scripts/general-server-shard-durations.json
+++ b/scripts/general-server-shard-durations.json
@@ -98,7 +98,6 @@
"server/src/__tests__/document-annotations-service.test.ts": 5743,
"server/src/__tests__/documents-service.test.ts": 4497,
"server/src/__tests__/documents.test.ts": 1104,
- "server/src/__tests__/duplex-aggregate-ceiling-env.test.ts": 210,
"server/src/__tests__/duplex-observability-recorder.test.ts": 207,
"server/src/__tests__/effective-run-config-fingerprints.test.ts": 219,
"server/src/__tests__/embedded-postgres-supervisor.test.ts": 206,
@@ -267,9 +266,6 @@
"server/src/__tests__/plugin-tenant-isolation.test.ts": 4622,
"server/src/__tests__/plugin-tool-dispatcher-pluginDbId.test.ts": 795,
"server/src/__tests__/plugin-ui-static.test.ts": 1293,
- "server/src/__tests__/plugin-worker-manager-duplex-byte-ledger.test.ts": 1240,
- "server/src/__tests__/plugin-worker-manager-duplex-pending-write-ledger.test.ts": 1006,
- "server/src/__tests__/plugin-worker-manager-duplex-stdin-write-ledger.test.ts": 3918,
"server/src/__tests__/plugin-worker-manager-duplex.test.ts": 2233,
"server/src/__tests__/plugin-worker-manager.test.ts": 2669,
"server/src/__tests__/private-hostname-guard.test.ts": 300,
diff --git a/server/src/__tests__/duplex-aggregate-ceiling-env.test.ts b/server/src/__tests__/duplex-aggregate-ceiling-env.test.ts
deleted file mode 100644
index a6f1a50348..0000000000
--- a/server/src/__tests__/duplex-aggregate-ceiling-env.test.ts
+++ /dev/null
@@ -1,70 +0,0 @@
-import { describe, expect, it, vi } from "vitest";
-import { DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES } from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
-import { resolveDuplexAggregateCeilingBytesFromEnv } from "../duplex-aggregate-ceiling-env.js";
-
-describe("resolveDuplexAggregateCeilingBytesFromEnv", () => {
- it("uses the safe default and reports nothing when the variable is absent", () => {
- const onRejectedOverride = vi.fn();
-
- const resolved = resolveDuplexAggregateCeilingBytesFromEnv(undefined, onRejectedOverride);
-
- expect(resolved).toBe(DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES);
- expect(onRejectedOverride).not.toHaveBeenCalled();
- });
-
- it("treats a whitespace-only value as a present invalid override, not as absent", () => {
- const rawOverride = " ";
- const onRejectedOverride = vi.fn();
-
- // The resolver must continue with the safe default. Startup does not fail.
- const resolved = resolveDuplexAggregateCeilingBytesFromEnv(rawOverride, onRejectedOverride);
-
- expect(resolved).toBe(DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES);
- // The rejection reporter fires one time. A blank value is now visible as
- // invalid instead of silent as absent.
- expect(onRejectedOverride).toHaveBeenCalledTimes(1);
- // The reporter receives only the parsed number. The raw whitespace string
- // never reaches the reporter, so no operator-supplied text can reach a log
- // line.
- const reportedValue = onRejectedOverride.mock.calls[0]?.[0];
- expect(typeof reportedValue).toBe("number");
- expect(reportedValue).not.toBe(rawOverride);
- expect(Number.isFinite(reportedValue)).toBe(true);
- });
-
- it("treats an empty-string value as a present invalid override", () => {
- const onRejectedOverride = vi.fn();
-
- const resolved = resolveDuplexAggregateCeilingBytesFromEnv("", onRejectedOverride);
-
- expect(resolved).toBe(DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES);
- expect(onRejectedOverride).toHaveBeenCalledTimes(1);
- });
-
- it("rejects a present non-numeric value and uses the safe default", () => {
- const onRejectedOverride = vi.fn();
-
- const resolved = resolveDuplexAggregateCeilingBytesFromEnv("not-a-number", onRejectedOverride);
-
- expect(resolved).toBe(DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES);
- expect(onRejectedOverride).toHaveBeenCalledTimes(1);
- });
-
- it("passes a valid override through unchanged", () => {
- const onRejectedOverride = vi.fn();
-
- const resolved = resolveDuplexAggregateCeilingBytesFromEnv("1048576", onRejectedOverride);
-
- expect(resolved).toBe(1048576);
- expect(onRejectedOverride).not.toHaveBeenCalled();
- });
-
- it("accepts a valid override that carries surrounding whitespace", () => {
- const onRejectedOverride = vi.fn();
-
- const resolved = resolveDuplexAggregateCeilingBytesFromEnv(" 1048576 ", onRejectedOverride);
-
- expect(resolved).toBe(1048576);
- expect(onRejectedOverride).not.toHaveBeenCalled();
- });
-});
diff --git a/server/src/__tests__/environment-execution-target-duplex.test.ts b/server/src/__tests__/environment-execution-target-duplex.test.ts
index bc45c9ea58..ba30d3f860 100644
--- a/server/src/__tests__/environment-execution-target-duplex.test.ts
+++ b/server/src/__tests__/environment-execution-target-duplex.test.ts
@@ -21,11 +21,6 @@ vi.mock("../services/plugin-environment-driver.js", async (importActual) => ({
}));
import type { EffectiveExecutionCapabilities } from "@paperclipai/adapter-utils/execution-target";
-import { adapterExecutionTargetDuplexAggregateByteLedger } from "@paperclipai/adapter-utils/execution-target";
-import {
- DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES,
- DuplexAggregateByteLedger,
-} from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
import { createSshCommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/ssh";
import type { Environment, EnvironmentLease } from "@paperclipai/shared";
import { resolveEnvironmentExecutionTarget } from "../services/environment-execution-target.js";
@@ -299,50 +294,6 @@ describe("EnvironmentRuntimeService.openDuplexChannel capability gate", () => {
});
});
-describe("sandbox execution target aggregate byte ledger stamp", () => {
- it("stamps the injected ledger on the target with object identity", async () => {
- const ledger = new DuplexAggregateByteLedger({
- ceilingBytes: DEFAULT_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES,
- });
- const target = await resolveEnvironmentExecutionTarget({
- db: {} as never,
- companyId: "company-1",
- adapterType: "codex_local",
- environment: { id: "env-1", driver: "sandbox", config: { provider: "daytona" } },
- leaseId: "lease-1",
- leaseMetadata: { remoteCwd: "/work" },
- lease: { id: "lease-1", leasePolicy: "reuse_by_environment" } as never,
- duplexAggregateByteLedger: ledger,
- });
- if (!target || target.kind !== "remote" || target.transport !== "sandbox") {
- throw new Error("expected a sandbox execution target");
- }
- // The stamped field and the accessor both return the one injected object, so
- // one process-owned ledger reaches the host bridge seam.
- expect(target.duplexAggregateByteLedger).toBe(ledger);
- expect(adapterExecutionTargetDuplexAggregateByteLedger(target)).toBe(ledger);
- });
-
- it("leaves the ledger absent when the host injects none", async () => {
- const target = await resolveEnvironmentExecutionTarget({
- db: {} as never,
- companyId: "company-1",
- adapterType: "codex_local",
- environment: { id: "env-1", driver: "sandbox", config: { provider: "daytona" } },
- leaseId: "lease-1",
- leaseMetadata: { remoteCwd: "/work" },
- lease: { id: "lease-1", leasePolicy: "reuse_by_environment" } as never,
- });
- if (!target || target.kind !== "remote" || target.transport !== "sandbox") {
- throw new Error("expected a sandbox execution target");
- }
- // A run with no injected ledger keeps the seam null; the accessor never makes
- // a fresh ledger.
- expect(target.duplexAggregateByteLedger).toBeNull();
- expect(adapterExecutionTargetDuplexAggregateByteLedger(target)).toBeNull();
- });
-});
-
// Build a sandbox execution target with a fixed capability snapshot and a fake
// environment runtime whose openDuplexChannel is a spy. The helper returns the
// runner and the spy so a test reads the capability-gated member.
diff --git a/server/src/__tests__/plugin-worker-manager-duplex-byte-ledger.test.ts b/server/src/__tests__/plugin-worker-manager-duplex-byte-ledger.test.ts
deleted file mode 100644
index 12db030a4c..0000000000
--- a/server/src/__tests__/plugin-worker-manager-duplex-byte-ledger.test.ts
+++ /dev/null
@@ -1,360 +0,0 @@
-import path from "node:path";
-import { fileURLToPath } from "node:url";
-import { describe, expect, it, vi } from "vitest";
-import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
-import {
- DuplexAggregateByteLedger,
- type DuplexAggregateByteLedgerTelemetry,
-} from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
-import {
- createDuplexRouteSlotController,
- createPluginWorkerHandle,
-} from "../services/plugin-worker-manager.js";
-
-// This suite proves the plugin worker manager charges every retained duplex route
-// representation against the injected aggregate byte ledger, and releases each
-// token exactly once through the one cleanup path. The tests drive a real worker
-// fixture process, so they exercise the true frame order across the bind boundary.
-
-const FIXTURES_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), "fixtures");
-const DUPLEX_CHANNEL_WORKER_ENTRYPOINT = path.join(
- FIXTURES_DIR,
- "plugin-worker-duplex-channel.cjs",
-);
-
-const TEST_MANIFEST: PaperclipPluginManifestV1 = {
- id: "test.plugin",
- apiVersion: 1,
- version: "1.0.0",
- displayName: "Test plugin",
- description: "Test plugin",
- author: "Paperclip",
- categories: ["automation"],
- capabilities: [],
- entrypoints: { worker: "dist/worker.js" },
-};
-
-function makeDuplexHandle(extra?: Record<string, unknown>) {
- return createPluginWorkerHandle("test.plugin", {
- entrypointPath: DUPLEX_CHANNEL_WORKER_ENTRYPOINT,
- manifest: TEST_MANIFEST,
- config: {},
- instanceInfo: { instanceId: "instance-1", hostVersion: "1.0.0" },
- apiVersion: 1,
- hostHandlers: {},
- ...extra,
- });
-}
-
-// The test directive rides in `providerLeaseId`, an opaque field the manager
-// forwards to the worker unchanged.
-function duplexOpenInput(directive: unknown, companyId = "company-1") {
- return {
- driverKey: "daytona",
- companyId,
- environmentId: "env-1",
- providerLeaseId: JSON.stringify(directive),
- command: "bridge-callback",
- };
-}
-
-// A counting telemetry surface. It records how many times the ledger rejected a
-// reservation and how many accounting defects it saw, so a test asserts a
-// fail-closed rejection and proves the cleanup never double-releases.
-function countingTelemetry(): DuplexAggregateByteLedgerTelemetry & {
- rejections: number;
- underflows: number;
-} {
- const state = {
- rejections: 0,
- underflows: 0,
- setBytesInUse() {},
- recordReservationRejection() {
- state.rejections += 1;
- },
- recordAccountingUnderflow() {
- state.underflows += 1;
- },
- };
- return state;
-}
-
-describe("plugin worker manager duplex aggregate byte ledger", () => {
- it("keeps terminalized buffered bytes charged until a late listener drains them, then a worker exit is harmless", async () => {
- const telemetry = countingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1 << 20, telemetry });
- // Lower the per-chunk char bound, so the second chunk ends the route while the
- // first chunk stays buffered.
- const handle = makeDuplexHandle({
- duplexAggregateByteLedger: ledger,
- duplexChannelLimits: { maxChunkChars: 4 },
- });
- try {
- await handle.start();
- const session = await handle.openDuplexChannel(
- // No listener attaches, so "ok" buffers and charges its two raw bytes. The
- // "toolong" chunk passes the per-chunk bound, so the route terminalizes and
- // moves the still-buffered "ok" token to the terminal registry.
- duplexOpenInput({ data: [{ chunk: "ok" }, { chunk: "toolong" }] }),
- );
- // The buffered "ok" stays charged after the route leaves the live map.
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(2);
- expect(ledger.liveTokenCount).toBe(1);
- });
- // A late listener drains the terminal buffered record and releases its token.
- const chunks: string[] = [];
- // The session now streams raw `Uint8Array` chunks. Decode each one back
- // to text, so the assertion below still compares the plain-text payload
- // the fixture directive scripted.
- session.onData((chunk) => chunks.push(new TextDecoder().decode(chunk)));
- await vi.waitFor(() => {
- expect(chunks).toEqual(["ok"]);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- // The over-bound chunk retained nothing, so no reservation ever rejected.
- expect(telemetry.rejections).toBe(0);
- } finally {
- await handle.stop().catch(() => undefined);
- }
- // The worker exit sweep runs on stop. The ledger stays at zero, and the sweep
- // records no accounting defect, so the drain and the exit never double-release.
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(telemetry.underflows).toBe(0);
- });
-
- it("transfers pre-bind held bytes to the buffered representation across the bind, then releases them on drain", async () => {
- const telemetry = countingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1 << 20, telemetry });
- const handle = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- try {
- await handle.start();
- const session = await handle.openDuplexChannel(
- // The worker writes the open reply and the two data frames in one stdout
- // write, so both frames arrive before the route binds. The host holds them
- // as pre-bind events, then the bind replays and transfers each token to the
- // buffered representation.
- duplexOpenInput({ batchWithOpenReply: true, data: [{ chunk: "aa" }, { chunk: "bb" }] }),
- );
- // The two held events keep their exact reserved bytes across the transfer, so
- // the live-token count never grows and the gauge never re-admits.
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(4);
- expect(ledger.liveTokenCount).toBe(2);
- });
- const chunks: string[] = [];
- // The session now streams raw `Uint8Array` chunks. Decode each one back
- // to text, so the assertion below still compares the plain-text payload
- // the fixture directive scripted.
- session.onData((chunk) => chunks.push(new TextDecoder().decode(chunk)));
- await vi.waitFor(() => {
- expect(chunks).toEqual(["aa", "bb"]);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- } finally {
- await handle.stop().catch(() => undefined);
- }
- expect(ledger.bytesInUse).toBe(0);
- expect(telemetry.underflows).toBe(0);
- });
-
- it("fails closed and retains nothing when a write reservation would pass the ceiling", async () => {
- const telemetry = countingTelemetry();
- // A four-byte ceiling. A serialized host-to-worker write frame cannot fit.
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 4, telemetry });
- const handle = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- try {
- await handle.start();
- // Open with no scripted data, so the fixture sends only the open reply.
- // The route is bound and live by the time this line resolves.
- const session = await handle.openDuplexChannel(duplexOpenInput({}));
- // The write happens strictly after the bind, so the rejection is a genuine
- // post-bind event, never a frame that races the open reply. The host makes
- // two reservations for this write. The one raw payload byte fits the
- // ceiling, so the pending-write reservation succeeds. The host then meters
- // the serialized frame just before the stdin write. That frame is much
- // larger than four bytes, so the transport reservation rejects, the host
- // writes nothing, and the route ends fail-closed.
- session.write(new TextEncoder().encode("a"));
- await vi.waitFor(() => {
- expect(telemetry.rejections).toBeGreaterThanOrEqual(1);
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- } finally {
- await handle.stop().catch(() => undefined);
- }
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(telemetry.underflows).toBe(0);
- });
-});
-
-// A telemetry surface that also tracks the peak gauge value. A test asserts the
-// peak never passes the ceiling, so it proves aggregate admission stops at the
-// ceiling instead of overshooting it.
-function peakTrackingTelemetry(): DuplexAggregateByteLedgerTelemetry & {
- gauge: number;
- peak: number;
- rejections: number;
- underflows: number;
-} {
- const state = {
- gauge: 0,
- peak: 0,
- rejections: 0,
- underflows: 0,
- setBytesInUse(bytes: number) {
- state.gauge = bytes;
- if (bytes > state.peak) state.peak = bytes;
- },
- recordReservationRejection() {
- state.rejections += 1;
- },
- recordAccountingUnderflow() {
- state.underflows += 1;
- },
- };
- return state;
-}
-
-// The exact raw byte count of one buffered chunk each route retains.
-const LOAD_CHUNK_BYTES = 8;
-const LOAD_CHUNK = "x".repeat(LOAD_CHUNK_BYTES);
-
-describe("plugin worker manager duplex aggregate byte ledger load across workers", () => {
- it("stops aggregate byte admission at the ceiling across many routes and workers while the route-count controller stays open", async () => {
- const telemetry = peakTrackingTelemetry();
- // The ceiling holds exactly six eight-byte buffered chunks.
- const fittingRoutes = 6;
- const ceilingBytes = fittingRoutes * LOAD_CHUNK_BYTES;
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes, telemetry });
- // The route-count controller has ample room, so the byte ledger binds first.
- const routeSlots = createDuplexRouteSlotController(100);
- const handleCount = 3;
- const routesPerHandle = 3;
- const totalRoutes = handleCount * routesPerHandle;
- const rejectedRoutes = totalRoutes - fittingRoutes;
- const handles: Array<ReturnType<typeof makeDuplexHandle>> = [];
- try {
- for (let h = 0; h < handleCount; h += 1) {
- const handle = makeDuplexHandle({
- duplexAggregateByteLedger: ledger,
- duplexRouteSlots: routeSlots,
- });
- await handle.start();
- handles.push(handle);
- }
- // Open every route in order. Each worker batches one data chunk with the open
- // reply, so the host holds the chunk as a pre-bind event and reserves its exact
- // raw bytes before the route binds. No listener attaches, so a bound route holds
- // the chunk. The first six chunks fit the ceiling. Each later reservation passes
- // the ceiling, so the ledger rejects it and the route fails closed with the open
- // marker (not the route-busy marker), because the route-count controller has room.
- let opened = 0;
- let failedClosed = 0;
- for (let h = 0; h < handleCount; h += 1) {
- for (let r = 0; r < routesPerHandle; r += 1) {
- try {
- await handles[h].openDuplexChannel(
- duplexOpenInput({
- workerSessionId: `ws-${h}-${r}`,
- batchWithOpenReply: true,
- data: [{ chunk: LOAD_CHUNK }],
- }),
- );
- opened += 1;
- } catch (err) {
- // A byte-ceiling rejection fails the route closed with the open marker.
- // The route-count controller had room, so this is never the route-busy
- // marker.
- expect(String(err)).toContain("DUPLEX_CHANNEL_OPEN_FAILED");
- expect(String(err)).not.toContain("DUPLEX_CHANNEL_ROUTE_BUSY");
- failedClosed += 1;
- }
- }
- }
- // Six routes bound and hold their bytes; three failed closed at the ceiling.
- expect(opened).toBe(fittingRoutes);
- expect(failedClosed).toBe(rejectedRoutes);
- expect(opened + failedClosed).toBe(totalRoutes);
- // The aggregate gauge holds at the ceiling and the ledger recorded one rejection
- // per over-ceiling route.
- await vi.waitFor(() => {
- expect(telemetry.rejections).toBe(rejectedRoutes);
- expect(ledger.bytesInUse).toBe(ceilingBytes);
- expect(ledger.liveTokenCount).toBe(fittingRoutes);
- });
- // The gauge never passed the ceiling, so admission stopped at the ceiling and
- // never overshot it.
- expect(telemetry.peak).toBe(ceilingBytes);
- } finally {
- for (const handle of handles) await handle.stop().catch(() => undefined);
- }
- // Every worker exit sweep released its buffered tokens once, so the ledger ends
- // at zero with no accounting defect.
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- expect(telemetry.underflows).toBe(0);
- });
-
- it("still bounds the route count for fixed per-route overhead when the byte ceiling has ample room", async () => {
- const telemetry = peakTrackingTelemetry();
- // A large byte ceiling, so retained bytes never bind admission.
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1 << 20, telemetry });
- // The route-count controller holds two slots for the whole process.
- const routeSlots = createDuplexRouteSlotController(2);
- const handleA = makeDuplexHandle({
- duplexAggregateByteLedger: ledger,
- duplexRouteSlots: routeSlots,
- });
- const handleB = makeDuplexHandle({
- duplexAggregateByteLedger: ledger,
- duplexRouteSlots: routeSlots,
- });
- let opened = 0;
- let busy = 0;
- try {
- await handleA.start();
- await handleB.start();
- // Attempt four routes across the two workers. The shared controller admits the
- // first two and rejects the last two with the fixed route-busy error.
- const attempts: Array<[ReturnType<typeof makeDuplexHandle>, string]> = [
- [handleA, "ws-a1"],
- [handleA, "ws-a2"],
- [handleB, "ws-b1"],
- [handleB, "ws-b2"],
- ];
- for (const [handle, ws] of attempts) {
- try {
- await handle.openDuplexChannel(
- duplexOpenInput({ workerSessionId: ws, data: [{ chunk: "aa" }] }),
- );
- opened += 1;
- } catch (err) {
- expect(String(err)).toContain("DUPLEX_CHANNEL_ROUTE_BUSY");
- busy += 1;
- }
- }
- // The route-count controller admitted two routes and rejected the rest, so it
- // still bounds the fixed per-route overhead on its own.
- expect(opened).toBe(2);
- expect(busy).toBe(2);
- // The two open routes buffered four raw bytes, far under the byte ceiling, and
- // the byte ledger rejected nothing.
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(4);
- expect(ledger.liveTokenCount).toBe(2);
- });
- expect(telemetry.rejections).toBe(0);
- } finally {
- await handleA.stop().catch(() => undefined);
- await handleB.stop().catch(() => undefined);
- }
- expect(ledger.bytesInUse).toBe(0);
- expect(telemetry.underflows).toBe(0);
- });
-});
diff --git a/server/src/__tests__/plugin-worker-manager-duplex-pending-write-ledger.test.ts b/server/src/__tests__/plugin-worker-manager-duplex-pending-write-ledger.test.ts
deleted file mode 100644
index 983925cbce..0000000000
--- a/server/src/__tests__/plugin-worker-manager-duplex-pending-write-ledger.test.ts
+++ /dev/null
@@ -1,252 +0,0 @@
-import path from "node:path";
-import { fileURLToPath } from "node:url";
-import { describe, expect, it, vi } from "vitest";
-import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
-import {
- DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED,
- DuplexAggregateByteLedger,
- type DuplexAggregateByteLedgerTelemetry,
-} from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
-
-// Mock the shared logger, so a test reads the fixed rejection marker the manager
-// logs when a pending-write reservation fails. The child logger returns the same
-// object, so `log.warn` is this mock's `warn`.
-vi.mock("../middleware/logger.js", () => {
- const mockLogger: Record<string, unknown> = {
- trace: vi.fn(),
- debug: vi.fn(),
- info: vi.fn(),
- warn: vi.fn(),
- error: vi.fn(),
- fatal: vi.fn(),
- child: vi.fn(() => mockLogger),
- };
- return { logger: mockLogger, httpLogger: vi.fn() };
-});
-
-import { logger } from "../middleware/logger.js";
-import { createPluginWorkerHandle } from "../services/plugin-worker-manager.js";
-
-// This suite proves the plugin worker manager charges every host→worker write raw
-// payload against the injected aggregate byte ledger under the `pending_write`
-// owner, and releases each token one time when the write RPC settles. The child
-// reads its stdin here, so the separate `stdin_write` transport token flushes and
-// releases at once. Each test waits for that flush, so the assertions isolate the
-// raw-payload token. The transport token has its own suite.
-
-const FIXTURES_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), "fixtures");
-const DUPLEX_CHANNEL_WORKER_ENTRYPOINT = path.join(
- FIXTURES_DIR,
- "plugin-worker-duplex-channel.cjs",
-);
-
-const TEST_MANIFEST: PaperclipPluginManifestV1 = {
- id: "test.plugin",
- apiVersion: 1,
- version: "1.0.0",
- displayName: "Test plugin",
- description: "Test plugin",
- author: "Paperclip",
- categories: ["automation"],
- capabilities: [],
- entrypoints: { worker: "dist/worker.js" },
-};
-
-function makeDuplexHandle(extra?: Record<string, unknown>) {
- return createPluginWorkerHandle("test.plugin", {
- entrypointPath: DUPLEX_CHANNEL_WORKER_ENTRYPOINT,
- manifest: TEST_MANIFEST,
- config: {},
- instanceInfo: { instanceId: "instance-1", hostVersion: "1.0.0" },
- apiVersion: 1,
- hostHandlers: {},
- ...extra,
- });
-}
-
-// The test directive rides in `providerLeaseId`, an opaque field the manager
-// forwards to the worker unchanged.
-function duplexOpenInput(directive: unknown, companyId = "company-1") {
- return {
- driverKey: "daytona",
- companyId,
- environmentId: "env-1",
- providerLeaseId: JSON.stringify(directive),
- command: "bridge-callback",
- };
-}
-
-// A telemetry surface that tracks the peak gauge value and the two counters. A
-// test asserts the peak never passes the ceiling, so it proves aggregate admission
-// stops at the ceiling instead of overshooting it.
-function peakTrackingTelemetry(): DuplexAggregateByteLedgerTelemetry & {
- gauge: number;
- peak: number;
- rejections: number;
- underflows: number;
-} {
- const state = {
- gauge: 0,
- peak: 0,
- rejections: 0,
- underflows: 0,
- setBytesInUse(bytes: number) {
- state.gauge = bytes;
- if (bytes > state.peak) state.peak = bytes;
- },
- recordReservationRejection() {
- state.rejections += 1;
- },
- recordAccountingUnderflow() {
- state.underflows += 1;
- },
- };
- return state;
-}
-
-// Return every reason string the manager logged through the mock `warn` sink.
-function loggedWarnReasons(): string[] {
- const calls = (logger.warn as unknown as { mock: { calls: unknown[][] } }).mock.calls;
- return calls
- .map((call) => {
- const first = call[0];
- return first && typeof first === "object"
- ? (first as { reason?: unknown }).reason
- : undefined;
- })
- .filter((reason): reason is string => typeof reason === "string");
-}
-
-describe("plugin worker manager duplex pending-write byte ledger", () => {
- it("charges each held raw payload and returns to zero after worker exit", async () => {
- const telemetry = peakTrackingTelemetry();
- // The ceiling has ample room, so a transient transport reservation never
- // rejects. Each write holds its raw payload until the RPC settles.
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1 << 20, telemetry });
- const handle = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- const writeBytes = 1000;
- const writeData = "x".repeat(writeBytes);
- try {
- await handle.start();
- // The worker reads its stdin but never replies to a write, so each write RPC
- // stays pending and holds its raw payload.
- const route = await handle.openDuplexChannel(
- duplexOpenInput({ workerSessionId: "ws-a", mode: "no-write-reply" }),
- );
- const writeBytesEncoded = new TextEncoder().encode(writeData);
- route.write(writeBytesEncoded);
- route.write(writeBytesEncoded);
- route.write(writeBytesEncoded);
- // The worker reads its stdin, so each transport token flushes and releases.
- // Only the three held raw payloads remain, so the gauge settles at three
- // times the payload byte count with three live tokens.
- await vi.waitFor(() => {
- expect(ledger.liveTokenCount).toBe(3);
- expect(ledger.bytesInUse).toBe(3 * writeBytes);
- });
- expect(telemetry.peak).toBeLessThanOrEqual(ledger.ceilingBytes);
- } finally {
- await handle.stop().catch(() => undefined);
- }
- // The worker exit settles each pending write, so each token releases one time
- // and the ledger ends at zero with no accounting defect.
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- expect(telemetry.underflows).toBe(0);
- });
-
- it("releases the raw-payload token after a delayed write RPC settles, with no worker exit", async () => {
- const telemetry = peakTrackingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1 << 20, telemetry });
- const handle = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- try {
- await handle.start();
- // The worker reads its stdin and delays its write reply, so the host holds the
- // raw-payload reservation for a measurable time before the RPC settles.
- const route = await handle.openDuplexChannel(
- duplexOpenInput({ workerSessionId: "ws-d", writeReplyDelayMs: 150 }),
- );
- const writeBytes = 1000;
- route.write(new TextEncoder().encode("x".repeat(writeBytes)));
- // The transport token flushes at once, so one raw-payload token stays held for
- // the full write byte count while the RPC is in flight.
- await vi.waitFor(() => {
- expect(ledger.liveTokenCount).toBe(1);
- expect(ledger.bytesInUse).toBe(writeBytes);
- });
- // The delayed reply settles the RPC, and the `finally` releases the token.
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- } finally {
- await handle.stop().catch(() => undefined);
- }
- expect(ledger.bytesInUse).toBe(0);
- expect(telemetry.underflows).toBe(0);
- });
-
- it("reserves the exact UTF-8 byte count of the raw payload, not the character length", async () => {
- const telemetry = peakTrackingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 1 << 20, telemetry });
- const handle = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- try {
- await handle.start();
- const route = await handle.openDuplexChannel(
- duplexOpenInput({ workerSessionId: "ws-u", mode: "no-write-reply" }),
- );
- // "a€" is two characters but four UTF-8 bytes (one plus three). The raw-payload
- // reservation must charge four bytes, so it uses the UTF-8 byte count.
- const data = "a€";
- expect(data.length).toBe(2);
- expect(Buffer.byteLength(data, "utf8")).toBe(4);
- route.write(new TextEncoder().encode(data));
- // The transport token flushes, so one raw-payload token of four bytes remains.
- await vi.waitFor(() => {
- expect(ledger.liveTokenCount).toBe(1);
- expect(ledger.bytesInUse).toBe(4);
- });
- } finally {
- await handle.stop().catch(() => undefined);
- }
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- expect(telemetry.underflows).toBe(0);
- });
-
- it("ends an over-ceiling single write fail-closed with the aggregate marker, not the route-busy marker", async () => {
- vi.mocked(logger.warn).mockClear();
- const telemetry = peakTrackingTelemetry();
- // The ceiling is smaller than one raw payload, so the raw-payload reservation
- // fails before the frame reaches the transport. The manager retains nothing.
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 100, telemetry });
- const handle = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- try {
- await handle.start();
- const route = await handle.openDuplexChannel(
- duplexOpenInput({ workerSessionId: "ws-o", mode: "no-write-reply" }),
- );
- // The single write is larger than the ceiling. The raw-payload reservation
- // fails, so the manager ends the route with the aggregate marker, never the
- // route-busy marker, and never writes the frame.
- route.write(new TextEncoder().encode("x".repeat(200)));
- expect(telemetry.rejections).toBeGreaterThanOrEqual(1);
- expect(ledger.bytesInUse).toBe(0);
- expect(telemetry.peak).toBeLessThanOrEqual(ledger.ceilingBytes);
- const reasons = loggedWarnReasons();
- expect(reasons).toContain(DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED);
- expect(reasons).not.toContain("DUPLEX_CHANNEL_ROUTE_BUSY");
- } finally {
- await handle.stop().catch(() => undefined);
- }
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- expect(telemetry.underflows).toBe(0);
- });
-});
diff --git a/server/src/__tests__/plugin-worker-manager-duplex-stdin-write-ledger.test.ts b/server/src/__tests__/plugin-worker-manager-duplex-stdin-write-ledger.test.ts
deleted file mode 100644
index d20d38da32..0000000000
--- a/server/src/__tests__/plugin-worker-manager-duplex-stdin-write-ledger.test.ts
+++ /dev/null
@@ -1,300 +0,0 @@
-import path from "node:path";
-import { fileURLToPath } from "node:url";
-import { describe, expect, it, vi } from "vitest";
-import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
-import {
- DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED,
- DuplexAggregateByteLedger,
- type DuplexAggregateByteLedgerTelemetry,
-} from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
-
-// Mock the shared logger, so a test reads the fixed rejection marker the manager
-// logs when a transport reservation fails. The child logger returns the same
-// object, so `log.warn` is this mock's `warn`.
-vi.mock("../middleware/logger.js", () => {
- const mockLogger: Record<string, unknown> = {
- trace: vi.fn(),
- debug: vi.fn(),
- info: vi.fn(),
- warn: vi.fn(),
- error: vi.fn(),
- fatal: vi.fn(),
- child: vi.fn(() => mockLogger),
- };
- return { logger: mockLogger, httpLogger: vi.fn() };
-});
-
-import { logger } from "../middleware/logger.js";
-import { createPluginWorkerHandle } from "../services/plugin-worker-manager.js";
-
-// This suite proves the plugin worker manager charges the child-stdin transport
-// buffer for every host→worker duplex write, under the `stdin_write` owner. The
-// worker completes the open, then stops reading its stdin. The host write then
-// stays in the host stdin write buffer. The tests prove the reservation covers the
-// serialized frame, holds until the flush or the worker exit, survives the write
-// RPC timeout and the route terminalization, and bounds the aggregate transport
-// bytes across several routes and workers.
-
-const FIXTURES_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), "fixtures");
-const DUPLEX_CHANNEL_WORKER_ENTRYPOINT = path.join(
- FIXTURES_DIR,
- "plugin-worker-duplex-channel.cjs",
-);
-
-const TEST_MANIFEST: PaperclipPluginManifestV1 = {
- id: "test.plugin",
- apiVersion: 1,
- version: "1.0.0",
- displayName: "Test plugin",
- description: "Test plugin",
- author: "Paperclip",
- categories: ["automation"],
- capabilities: [],
- entrypoints: { worker: "dist/worker.js" },
-};
-
-function makeDuplexHandle(extra?: Record<string, unknown>) {
- return createPluginWorkerHandle("test.plugin", {
- entrypointPath: DUPLEX_CHANNEL_WORKER_ENTRYPOINT,
- manifest: TEST_MANIFEST,
- config: {},
- instanceInfo: { instanceId: "instance-1", hostVersion: "1.0.0" },
- apiVersion: 1,
- hostHandlers: {},
- ...extra,
- });
-}
-
-// The test directive rides in `providerLeaseId`, an opaque field the manager
-// forwards to the worker unchanged.
-function duplexOpenInput(directive: unknown, companyId = "company-1") {
- return {
- driverKey: "daytona",
- companyId,
- environmentId: "env-1",
- providerLeaseId: JSON.stringify(directive),
- command: "bridge-callback",
- };
-}
-
-// A telemetry surface that tracks the peak gauge value and the two counters. A
-// test asserts the peak never passes the ceiling, so it proves aggregate admission
-// stops at the ceiling instead of overshooting it.
-function peakTrackingTelemetry(): DuplexAggregateByteLedgerTelemetry & {
- gauge: number;
- peak: number;
- rejections: number;
- underflows: number;
-} {
- const state = {
- gauge: 0,
- peak: 0,
- rejections: 0,
- underflows: 0,
- setBytesInUse(bytes: number) {
- state.gauge = bytes;
- if (bytes > state.peak) state.peak = bytes;
- },
- recordReservationRejection() {
- state.rejections += 1;
- },
- recordAccountingUnderflow() {
- state.underflows += 1;
- },
- };
- return state;
-}
-
-// Return every reason string the manager logged through the mock `warn` sink.
-function loggedWarnReasons(): string[] {
- const calls = (logger.warn as unknown as { mock: { calls: unknown[][] } }).mock.calls;
- return calls
- .map((call) => {
- const first = call[0];
- return first && typeof first === "object"
- ? (first as { reason?: unknown }).reason
- : undefined;
- })
- .filter((reason): reason is string => typeof reason === "string");
-}
-
-// A frame the host keeps in its stdin write buffer must pass the operating-system
-// pipe buffer (about 64 KiB), so the write callback stays pending while the child
-// does not read. Each write here is larger than that, so the host holds the
-// transport token until the worker exit.
-const HELD_WRITE_CHARS = 200_000;
-
-describe("plugin worker manager duplex stdin transport byte ledger", () => {
- it("reserves the encoded serialized-frame size, larger than the raw payload, including the base64 wire inflation", async () => {
- const telemetry = peakTrackingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 8 << 20, telemetry });
- const handle = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- try {
- await handle.start();
- const route = await handle.openDuplexChannel(
- duplexOpenInput({
- workerSessionId: "ws-a",
- stopReadingStdinAfterOpen: true,
- exitAfterStopMs: 500,
- }),
- );
- // The JSON-RPC hop carries the write payload as a base64 string, because JSON
- // has no binary type (see `ChannelBytesWireValue` in protocol.ts). Base64
- // encodes three raw bytes as four characters, so the serialized frame is
- // larger than the raw payload by about that ratio, plus the small fixed cost
- // of the surrounding JSON envelope (the method name and the route identifiers).
- const rawBytes = 50_000;
- const data = new Uint8Array(rawBytes).fill(0x22); // an arbitrary byte value
- const encodedLength = Buffer.from(data).toString("base64").length;
- route.write(data);
- // The raw-payload token and the transport token both hold at once. The worker
- // does not read its stdin, so the transport token never flushes.
- expect(ledger.liveTokenCount).toBe(2);
- const transportBytes = ledger.bytesInUse - rawBytes;
- // The transport reservation covers the serialized frame: at least the base64
- // form of the raw payload, plus a bounded JSON envelope around it.
- expect(transportBytes).toBeGreaterThan(rawBytes);
- expect(transportBytes).toBeGreaterThanOrEqual(encodedLength);
- expect(transportBytes).toBeLessThan(encodedLength + 5_000);
- } finally {
- await handle.stop().catch(() => undefined);
- }
- // The worker exit discards the stdin buffer, so every token releases one time.
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- expect(telemetry.underflows).toBe(0);
- });
-
- it("holds the transport token across a write-RPC timeout and releases it on worker exit", async () => {
- const telemetry = peakTrackingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 8 << 20, telemetry });
- // The write RPC times out fast; the worker exits later. The gap lets a test
- // prove the timeout does not release the transport token.
- const handle = makeDuplexHandle({
- duplexAggregateByteLedger: ledger,
- duplexChannelLimits: { openTimeoutMs: 150 },
- });
- try {
- await handle.start();
- const route = await handle.openDuplexChannel(
- duplexOpenInput({
- workerSessionId: "ws-t",
- stopReadingStdinAfterOpen: true,
- exitAfterStopMs: 900,
- }),
- );
- const data = "x".repeat(HELD_WRITE_CHARS);
- const rawBytes = Buffer.byteLength(data, "utf8");
- route.write(new TextEncoder().encode(data));
- // Both tokens hold at first: the raw payload and the serialized frame.
- expect(ledger.liveTokenCount).toBe(2);
- const transportBytes = ledger.bytesInUse - rawBytes;
- expect(transportBytes).toBeGreaterThan(0);
- // The write RPC times out. The RPC settle releases the raw-payload token, so
- // one token remains: the transport token. The timeout never releases it.
- await vi.waitFor(() => {
- expect(ledger.liveTokenCount).toBe(1);
- });
- expect(ledger.bytesInUse).toBe(transportBytes);
- } finally {
- await handle.stop().catch(() => undefined);
- }
- // The worker exit releases the still-held transport token.
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- expect(telemetry.underflows).toBe(0);
- });
-
- it("does not release the transport token when the route terminalizes", async () => {
- const telemetry = peakTrackingTelemetry();
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes: 8 << 20, telemetry });
- const handle = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- try {
- await handle.start();
- const route = await handle.openDuplexChannel(
- duplexOpenInput({
- workerSessionId: "ws-c",
- stopReadingStdinAfterOpen: true,
- exitAfterStopMs: 700,
- }),
- );
- const data = "x".repeat(HELD_WRITE_CHARS);
- const rawBytes = Buffer.byteLength(data, "utf8");
- route.write(new TextEncoder().encode(data));
- expect(ledger.liveTokenCount).toBe(2);
- const transportBytes = ledger.bytesInUse - rawBytes;
- // End the route. Route terminalization releases the route-owned tokens, but it
- // must never release the transport token, so at least the serialized frame
- // stays charged.
- void route.close();
- expect(ledger.bytesInUse).toBeGreaterThanOrEqual(transportBytes);
- } finally {
- await handle.stop().catch(() => undefined);
- }
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- expect(telemetry.underflows).toBe(0);
- });
-
- it("bounds the aggregate transport bytes across several routes and workers, ending an over-ceiling near-limit write with the aggregate marker", async () => {
- vi.mocked(logger.warn).mockClear();
- const telemetry = peakTrackingTelemetry();
- // The ceiling admits only a couple of held serialized frames, so a near-limit
- // write across the routes passes it and rejects.
- const ceilingBytes = 600_000;
- const ledger = new DuplexAggregateByteLedger({ ceilingBytes, telemetry });
- const handleA = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- const handleB = makeDuplexHandle({ duplexAggregateByteLedger: ledger });
- try {
- await handleA.start();
- await handleB.start();
- const routeA = await handleA.openDuplexChannel(
- duplexOpenInput({
- workerSessionId: "ws-a",
- stopReadingStdinAfterOpen: true,
- exitAfterStopMs: 900,
- }),
- );
- const routeB = await handleB.openDuplexChannel(
- duplexOpenInput({
- workerSessionId: "ws-b",
- stopReadingStdinAfterOpen: true,
- exitAfterStopMs: 900,
- }),
- );
- // Each write is near the per-write size. The JSON-RPC hop carries it as
- // base64, so the reservation must count the encoded (larger) wire size, not
- // the raw payload size.
- const nearLimit = new Uint8Array(100_000).fill(0x22);
- const routes = [routeA, routeB, routeA, routeB];
- for (const route of routes) {
- route.write(nearLimit);
- // The aggregate bytes never pass the ceiling. Admission stops at the ceiling.
- expect(ledger.bytesInUse).toBeLessThanOrEqual(ceilingBytes);
- }
- // At least one near-limit write passed the ceiling and rejected. The manager
- // ended that route with the aggregate marker, never the route-busy marker.
- expect(telemetry.rejections).toBeGreaterThanOrEqual(1);
- expect(telemetry.peak).toBeLessThanOrEqual(ceilingBytes);
- const reasons = loggedWarnReasons();
- expect(reasons).toContain(DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED);
- expect(reasons).not.toContain("DUPLEX_CHANNEL_ROUTE_BUSY");
- } finally {
- await handleA.stop().catch(() => undefined);
- await handleB.stop().catch(() => undefined);
- }
- // Every worker exit releases its held transport tokens, so the ledger ends at
- // zero with no accounting defect.
- await vi.waitFor(() => {
- expect(ledger.bytesInUse).toBe(0);
- expect(ledger.liveTokenCount).toBe(0);
- });
- expect(telemetry.underflows).toBe(0);
- });
-});
diff --git a/server/src/__tests__/plugin-worker-manager-duplex.test.ts b/server/src/__tests__/plugin-worker-manager-duplex.test.ts
index 40d9c3f552..bf7d01ab57 100644
--- a/server/src/__tests__/plugin-worker-manager-duplex.test.ts
+++ b/server/src/__tests__/plugin-worker-manager-duplex.test.ts
@@ -423,6 +423,58 @@ describe("plugin worker manager duplex channel route", () => {
}
});
+ it("ends the route when the pending host-to-worker write bytes pass the route bound", async () => {
+ const handle = makeDuplexHandle({
+ duplexChannelLimits: { maxPendingWriteBytes: 10 },
+ });
+ try {
+ await handle.start();
+ const session = await handle.openDuplexChannel(
+ duplexOpenInput({ mode: "no-write-reply", workerSessionId: "ws-A" }),
+ );
+ const waitResult = session.wait();
+ // The worker never replies to a write, so each write's bytes stay charged
+ // against the route. The second write brings the cumulative bytes past
+ // the 10-byte bound and ends the route.
+ session.write(new TextEncoder().encode("aaaaa")); // 5 bytes → 5, under the bound
+ session.write(new TextEncoder().encode("bbbbbb")); // 6 bytes → 11 > 10, ends the route
+ await expect(waitResult).resolves.toEqual({ exitCode: null });
+ } finally {
+ await handle.stop().catch(() => undefined);
+ }
+ });
+
+ it("returns the pending write bytes to the route bound after a failed write", async () => {
+ const handle = makeDuplexHandle({
+ duplexChannelLimits: { maxPendingWriteBytes: 10, openTimeoutMs: 100 },
+ });
+ try {
+ await handle.start();
+ const session = await handle.openDuplexChannel(
+ duplexOpenInput({ mode: "no-write-reply", workerSessionId: "ws-A" }),
+ );
+ let routeEnded = false;
+ session.wait().then(() => {
+ routeEnded = true;
+ });
+ // The worker never replies, so each write's own request times out and
+ // rejects. The rejection must release this write's charged bytes, the
+ // same as a reply would. Wait past the first write's timeout before the
+ // second write sends.
+ session.write(new TextEncoder().encode("12345678")); // 8 bytes, under the 10-byte bound
+ await new Promise((resolve) => setTimeout(resolve, 200));
+ // If the first write's bytes had not released on its timeout, this
+ // second 8-byte write would bring the route to 16 bytes, past the
+ // 10-byte bound, and end the route at once, synchronously, in this call.
+ session.write(new TextEncoder().encode("87654321"));
+ await new Promise((resolve) => setTimeout(resolve, 0));
+ expect(routeEnded).toBe(false);
+ await session.close();
+ } finally {
+ await handle.stop().catch(() => undefined);
+ }
+ });
+
it("ends the route when one host-to-worker write passes the size bound", async () => {
const handle = makeDuplexHandle({
duplexChannelLimits: { maxWriteChars: 8 },
@@ -701,6 +753,36 @@ describe("plugin worker manager duplex channel route", () => {
}
});
+ it("ends the route when the pre-bind hold bytes pass the route input bound", async () => {
+ const handle = makeDuplexHandle({
+ duplexChannelLimits: { maxPreBindBufferedChars: 10 },
+ });
+ try {
+ await handle.start();
+ // The worker batches the three data frames with the open reply, so all
+ // three frames arrive before the route binds and land in the pre-bind
+ // hold, not the post-bind buffered queue. The frame-count bound stays
+ // far above three frames, so only the byte bound can end the route
+ // here: this proves the hold itself counts bytes, not only frames. The
+ // hold ends the route before the bind completes, so the open call
+ // itself fails, the same way a malformed open reply fails it.
+ await expect(
+ handle.openDuplexChannel(
+ duplexOpenInput({
+ batchWithOpenReply: true,
+ data: [
+ { chunk: "aaaaa" }, // total 5 → held
+ { chunk: "bbbbb" }, // total 10 → held
+ { chunk: "ccccc" }, // total 15 > 10 → end the route in the hold
+ ],
+ }),
+ ),
+ ).rejects.toThrow("DUPLEX_CHANNEL_OPEN_FAILED");
+ } finally {
+ await handle.stop().catch(() => undefined);
+ }
+ });
+
it("ends the route on the frame count bound even when a caller raises that bound well past the module default", async () => {
// Regression test: the pre-open hold ceiling must track
// maxDuplexChannelPreBindFrames, not a fixed value. A fixed ceiling at or
diff --git a/server/src/duplex-aggregate-ceiling-env.ts b/server/src/duplex-aggregate-ceiling-env.ts
deleted file mode 100644
index a1381c910c..0000000000
--- a/server/src/duplex-aggregate-ceiling-env.ts
+++ /dev/null
@@ -1,32 +0,0 @@
-import {
- resolveDuplexAggregateCeilingBytes,
- type DuplexAggregateCeilingOverrideRejectionReporter,
-} from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
-
-/**
- * Resolve the aggregate duplex route byte ceiling from the raw operator override
- * string. The host reads the override from
- * PAPERCLIP_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES.
- *
- * This helper distinguishes an absent variable from a present blank value. An
- * absent variable is `undefined`. The helper then uses the documented default and
- * does not report a rejection. A present variable is a string, even when the
- * string holds only whitespace. The helper sends every present string through
- * `Number`, so a blank or whitespace-only value becomes `0`. The numeric
- * validation in {@link resolveDuplexAggregateCeilingBytes} rejects `0`, reports it
- * through `onRejectedOverride`, and returns the safe default. So a present blank
- * value is visible as invalid instead of silent as absent.
- *
- * `Number` ignores surrounding whitespace, so a valid value with surrounding
- * spaces still parses. The reporter receives only the parsed number, never the raw
- * string, so no operator-supplied text reaches a log line.
- */
-export function resolveDuplexAggregateCeilingBytesFromEnv(
- rawOverride: string | undefined,
- onRejectedOverride?: DuplexAggregateCeilingOverrideRejectionReporter,
-): number {
- return resolveDuplexAggregateCeilingBytes(
- rawOverride === undefined ? undefined : Number(rawOverride),
- onRejectedOverride,
- );
-}
diff --git a/server/src/index.ts b/server/src/index.ts
index e668adf7a8..9f0dd8e3da 100644
--- a/server/src/index.ts
+++ b/server/src/index.ts
@@ -87,13 +87,6 @@ import { createFeedbackTraceShareClientFromConfig } from "./services/feedback-sh
import { buildRuntimeApiCandidateUrls, choosePrimaryRuntimeApiUrl } from "./runtime-api.js";
import { isLoopbackHost, rewriteLoopbackUrlPort } from "./url-utils.js";
import { createPluginWorkerManager } from "./services/plugin-worker-manager.js";
-import {
- createDuplexAggregateByteLedgerTelemetry,
- DuplexAggregateByteLedger,
- type DuplexAggregateByteLedgerMetricSink,
-} from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
-import { resolveDuplexAggregateCeilingBytesFromEnv } from "./duplex-aggregate-ceiling-env.js";
-import { DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL } from "@paperclipai/adapter-utils/duplex-observability";
import { createStorageServiceFromConfig } from "./storage/index.js";
import { printStartupBanner } from "./startup-banner.js";
import { getBoardClaimWarningUrl, initializeBoardClaimChallenge } from "./board-claim.js";
@@ -784,57 +777,9 @@ export async function startServer(): Promise<StartedServer> {
databaseBackupInFlight = false;
}
};
- // The process-owned aggregate byte ledger for the sandbox duplex channel. One
- // ledger per host process bounds the aggregate bytes that all live duplex routes
- // retain. The route-count controller bounds only the route count, so without this
- // ledger the per-route byte bounds multiply to many gigabytes at the maximum
- // route count. The manager injects this same object into every worker handle, so
- // one shared gauge bounds every host-side retention site.
- //
- // The optional operator override reads PAPERCLIP_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES.
- // An absent variable uses the documented default. A present invalid, blank,
- // whitespace-only, non-finite, zero, negative, non-integer, unsafe, or
- // over-maximum value does not fail startup. The host process is multi-tenant, so
- // one invalid environment value must not brick the whole host. The helper sends
- // the raw string to the resolver, so a present blank value is invalid, not
- // absent. The resolver rejects the invalid override and returns the safe default.
- // The reporter logs the rejection loudly at error, so the misconfiguration stays
- // visible while the host stays up. The log line carries only the rejected numeric
- // value, never the raw string.
- const duplexAggregateCeilingBytes = resolveDuplexAggregateCeilingBytesFromEnv(
- process.env.PAPERCLIP_MAX_AGGREGATE_DUPLEX_ROUTE_BYTES,
- (rejectedValue) => {
- logger.error(
- { rejectedValue },
- "duplex aggregate byte ceiling override rejected; using the safe default",
- );
- },
- );
- // The server has no process metric pipeline yet, so the ledger telemetry maps to
- // the structured logger. The gauge logs at debug. A reservation rejection logs at
- // warn, because it marks an availability limit hit. An accounting-underflow defect
- // logs at error, because it marks a real cleanup bug. Each record carries only the
- // fixed metric name and the numeric value; no route, company, run, or payload
- // value reaches a log line.
- const duplexAggregateByteLedgerMetricSink: DuplexAggregateByteLedgerMetricSink = {
- setGauge(name, value) {
- logger.debug({ metric: name, value }, "duplex aggregate byte ledger gauge");
- },
- incrementCounter(name) {
- if (name === DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL) {
- logger.error({ metric: name }, "duplex aggregate byte ledger accounting underflow");
- return;
- }
- logger.warn({ metric: name }, "duplex aggregate byte ledger reservation rejected");
- },
- };
- const duplexAggregateByteLedger = new DuplexAggregateByteLedger({
- ceilingBytes: duplexAggregateCeilingBytes,
- telemetry: createDuplexAggregateByteLedgerTelemetry(duplexAggregateByteLedgerMetricSink),
- });
- const pluginWorkerManager = createPluginWorkerManager({ duplexAggregateByteLedger });
+ const pluginWorkerManager = createPluginWorkerManager();
const heartbeat = config.heartbeatSchedulerEnabled
- ? heartbeatService(db as any, { pluginWorkerManager, duplexAggregateByteLedger })
+ ? heartbeatService(db as any, { pluginWorkerManager })
: null;
const decisionServiceOptions = {
wakeOriginAgent: createDecisionWakeOriginAgent(heartbeat?.wakeup ?? null),
diff --git a/server/src/services/environment-execution-target.ts b/server/src/services/environment-execution-target.ts
index 5ad4642b1b..04c9f66435 100644
--- a/server/src/services/environment-execution-target.ts
+++ b/server/src/services/environment-execution-target.ts
@@ -6,7 +6,6 @@ import {
type AdapterExecutionTarget,
} from "@paperclipai/adapter-utils/execution-target";
import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability";
-import type { DuplexAggregateByteLedger } from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
import {
clampSpanLabel,
getActiveStepContext,
@@ -208,10 +207,6 @@ export async function resolveEnvironmentExecutionTarget(input: {
// enters the sandbox environment. Absent keeps the safe no-op default in the
// bridge, so the surface stays inert until the host injects a real recorder.
duplexObservabilityRecorder?: DuplexObservabilityRecorder | null;
- // The process-owned aggregate byte ledger. The seam stamps it onto the sandbox
- // target next to the runner, so the live object stays on the host and never
- // enters the sandbox environment. Absent keeps the bridge inert for this seam.
- duplexAggregateByteLedger?: DuplexAggregateByteLedger | null;
}): Promise<AdapterExecutionTarget | null> {
if (input.environment.driver === "local") {
return {
@@ -340,10 +335,6 @@ export async function resolveEnvironmentExecutionTarget(input: {
// binds it to the fixed observability surface. Absent keeps the no-op
// default, so the surface stays inert on a run with no injected recorder.
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
- // Attach the process-owned aggregate byte ledger next to the runner. The
- // bridge passes it to the broker, the decoder, and the response-body reader.
- // Absent keeps the bridge inert for this seam.
- duplexAggregateByteLedger: input.duplexAggregateByteLedger ?? null,
...(effectiveCapabilities ? { effectiveCapabilities: Object.freeze({ ...effectiveCapabilities }) } : {}),
environmentId: input.environment.id ?? null,
leaseId: input.leaseId ?? null,
diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts
index 9cf984f3b6..ccabc7e51d 100644
--- a/server/src/services/environment-run-orchestrator.ts
+++ b/server/src/services/environment-run-orchestrator.ts
@@ -44,7 +44,6 @@ import {
type AdapterWorkspaceRealization,
} from "@paperclipai/adapter-utils/execution-target";
import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability";
-import type { DuplexAggregateByteLedger } from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
import { buildWorkspaceRealizationRequest } from "./workspace-realization.js";
import { executionWorkspaceService } from "./execution-workspaces.js";
import { logActivity } from "./activity-log.js";
@@ -156,14 +155,6 @@ export function environmentRunOrchestrator(
options: {
pluginWorkerManager?: PluginWorkerManager;
environmentRuntime?: EnvironmentRuntimeService;
- /**
- * The process-owned aggregate byte ledger for the sandbox duplex channel.
- * The server root creates one ledger per host process and injects the same
- * object here. The orchestrator stamps it onto the sandbox execution target,
- * so one shared gauge bounds the aggregate retained bytes across all live
- * duplex routes. Absent keeps the bridge inert for this seam.
- */
- duplexAggregateByteLedger?: DuplexAggregateByteLedger | null;
} = {},
) {
const environmentsSvc = environmentService(db);
@@ -529,7 +520,6 @@ export function environmentRunOrchestrator(
lease,
environmentRuntime,
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
- duplexAggregateByteLedger: options.duplexAggregateByteLedger ?? null,
});
const realizationMode = workspaceRealization.mode === "in_place" ? "in_place" : "copy";
const authoritativeRoot =
diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts
index 69419dad9d..ca32a904cf 100644
--- a/server/src/services/heartbeat.ts
+++ b/server/src/services/heartbeat.ts
@@ -73,7 +73,6 @@ import {
import { conflict, HttpError, notFound } from "../errors.js";
import { getStartupTraceContext, getStartupTracer } from "../instrumentation.js";
import { createHostDuplexObservabilityRecorder } from "./duplex-observability-recorder.js";
-import type { DuplexAggregateByteLedger } from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
import { incrementToolRuntimeMetricCounter } from "./tool-runtime-metrics.js";
import { logger } from "../middleware/logger.js";
import {
@@ -6707,14 +6706,6 @@ export interface HeartbeatServiceOptions {
pluginWorkerManager?: PluginWorkerManager;
environmentRuntime?: HeartbeatEnvironmentRuntime;
runtimeEnv?: Record<string, string | undefined>;
- /**
- * The process-owned aggregate byte ledger for the sandbox duplex channel. The
- * server root creates one ledger per host process and injects the same object
- * here. The heartbeat threads it into the environment run orchestrator, which
- * stamps it onto the sandbox execution target. Absent keeps the bridge inert
- * for this seam.
- */
- duplexAggregateByteLedger?: DuplexAggregateByteLedger | null;
}
type WorkspaceReadyCommentWriter = {
@@ -6838,7 +6829,6 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
const envOrchestrator = environmentRunOrchestrator(db, {
pluginWorkerManager: options.pluginWorkerManager,
environmentRuntime,
- duplexAggregateByteLedger: options.duplexAggregateByteLedger,
});
const workspaceOperationsSvc = workspaceOperationService(db);
const liveRunExecutions = {
diff --git a/server/src/services/plugin-worker-manager.ts b/server/src/services/plugin-worker-manager.ts
index 0b533a3385..9d2d8b8ab3 100644
--- a/server/src/services/plugin-worker-manager.ts
+++ b/server/src/services/plugin-worker-manager.ts
@@ -59,12 +59,6 @@ import type {
InitializeParams,
} from "@paperclipai/plugin-sdk";
import { getActiveStepContext } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
-import {
- DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED,
- type DuplexAggregateByteLedger,
- type DuplexAggregateTokenOwner,
- type ReservationToken,
-} from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger";
import {
isLoginCommandKey,
validateLoginSessionHome,
@@ -73,21 +67,6 @@ import {
import { logger } from "../middleware/logger.js";
import { traceparentFromContextToken } from "../instrumentation.js";
-/**
- * The host raises this error when the child-stdin transport reservation for a
- * duplex write fails against the aggregate byte ledger. The host does not write
- * the frame. The write path throws it, `callInternal` rejects the RPC with it
- * unwrapped, and the duplex write caller ends the route fail-closed with the
- * {@link DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED} marker. Only a duplex write
- * meters the transport, so this error never reaches a non-duplex control message.
- */
-class DuplexAggregateBytesExceededError extends Error {
- constructor() {
- super(DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED);
- this.name = "DuplexAggregateBytesExceededError";
- }
-}
-
// ---------------------------------------------------------------------------
// Constants
// ---------------------------------------------------------------------------
@@ -193,14 +172,21 @@ const LOGIN_PTY_OPEN_FAILED = "LOGIN_PTY_OPEN_FAILED";
// login pseudo-terminal route, but it carries no command allowlist and adds seven
// explicit bounds the pseudo-terminal route lacks. Each bound ends the route when
// it passes the limit, so a faulty or hostile worker cannot flood the host.
-/** The default maximum characters for one duplex channel data notification. */
-const MAX_DUPLEX_CHANNEL_CHUNK_CHARS = 1_000_000;
+/** The default maximum characters for one duplex channel data notification. This
+ * matches the host duplex frame bound ({@link DEFAULT_MAX_DUPLEX_FRAME_BYTES} in
+ * `duplex-frame-codec.ts`), so one chunk always fits inside one frame. */
+const MAX_DUPLEX_CHANNEL_CHUNK_CHARS = 262_144;
/**
- * The default maximum cumulative characters the host buffers for one duplex
- * channel route before a data listener attaches. A worker that streams data
- * before the consumer binds cannot grow the host buffer without limit.
+ * The default maximum cumulative bytes the host retains, on one duplex channel
+ * route, for input the worker has not yet delivered: the pre-bind hold (before
+ * the worker session id binds), the post-bind buffered queue (bound, before a
+ * data listener attaches), and the terminal buffer (a route that ended before
+ * a listener attached). All three representations share this one route-local
+ * bound, so a worker cannot grow any of them past it. This bound is a fixed
+ * share of the fixed per-route byte budget the host now bounds every duplex
+ * retention site against; it holds no measurement of real traffic.
*/
-const MAX_DUPLEX_CHANNEL_PRE_BIND_CHARS = 8 * 1024 * 1024;
+export const DUPLEX_ROUTE_INPUT_MAX_BYTES = 393_216;
/**
* The default maximum number of data frames the host buffers for one duplex
* channel route before a data listener attaches.
@@ -229,8 +215,21 @@ const DUPLEX_CHANNEL_PRE_BIND_HOLD_MARGIN_FRAMES = 1;
* unbounded number of pending requests.
*/
const MAX_DUPLEX_CHANNEL_PENDING_REQUESTS = 256;
-/** The default maximum characters for one host→worker duplex channel write. */
-const MAX_DUPLEX_CHANNEL_WRITE_CHARS = 1_000_000;
+/** The default maximum characters for one host→worker duplex channel write. This
+ * matches the host duplex frame bound, so one write always fits inside one
+ * frame. */
+const MAX_DUPLEX_CHANNEL_WRITE_CHARS = 262_144;
+/**
+ * The default maximum cumulative bytes the host retains, on one duplex channel
+ * route, for a host→worker write the worker has not yet acknowledged: the raw
+ * write payload and the serialized child-stdin frame the host built from it.
+ * This bound is route-local and independent of the pending-request count
+ * bound above; a worker that never replies still cannot make one route hold
+ * an unbounded number of write bytes, even under a raised pending-request
+ * count. This bound is a fixed share of the fixed per-route byte budget; it
+ * holds no measurement of real traffic.
+ */
+export const DUPLEX_ROUTE_PENDING_WRITE_MAX_BYTES = 393_216;
/**
* The default maximum number of protocol errors for one duplex channel route.
* A protocol error is one malformed or mismatched data frame. The route ends
@@ -428,14 +427,6 @@ export interface WorkerStartOptions {
* constructs a handle this way).
*/
duplexRouteSlots?: DuplexRouteSlotController | null;
- /**
- * The process-owned aggregate byte ledger for the duplex channel. The manager
- * injects one shared instance into every worker handle, so one gauge bounds the
- * aggregate retained bytes across every route in the process. When it is absent,
- * the worker retains duplex bytes unbounded (a unit test constructs a handle
- * this way). The manager never makes a fresh per-handle default.
- */
- duplexAggregateByteLedger?: DuplexAggregateByteLedger | null;
/**
* Companies this worker may act on from proactive (no-invocation) worker→host
* calls — the plugin's configured companies. Seeded onto the handle at
@@ -504,6 +495,9 @@ export interface WorkerStartOptions {
maxPendingRequests?: number;
/** Max characters for one host→worker duplex channel write. */
maxWriteChars?: number;
+ /** Max cumulative bytes the host retains, for one route, for a
+ * host→worker write pending the worker's acknowledgement. */
+ maxPendingWriteBytes?: number;
/** Max number of protocol errors for one route before the route ends. */
maxProtocolErrors?: number;
/** Max cumulative bytes the host forwards for one route over its whole life. */
@@ -935,7 +929,7 @@ export function createPluginWorkerHandle(
options.duplexChannelLimits?.maxChunkChars ?? MAX_DUPLEX_CHANNEL_CHUNK_CHARS;
const maxDuplexChannelPreBindChars =
options.duplexChannelLimits?.maxPreBindBufferedChars ??
- MAX_DUPLEX_CHANNEL_PRE_BIND_CHARS;
+ DUPLEX_ROUTE_INPUT_MAX_BYTES;
const maxDuplexChannelPreBindFrames =
options.duplexChannelLimits?.maxPreBindBufferedFrames ??
MAX_DUPLEX_CHANNEL_PRE_BIND_FRAMES;
@@ -949,6 +943,8 @@ export function createPluginWorkerHandle(
MAX_DUPLEX_CHANNEL_PENDING_REQUESTS;
const maxDuplexChannelWriteChars =
options.duplexChannelLimits?.maxWriteChars ?? MAX_DUPLEX_CHANNEL_WRITE_CHARS;
+ const maxDuplexRoutePendingWriteBytes =
+ options.duplexChannelLimits?.maxPendingWriteBytes ?? DUPLEX_ROUTE_PENDING_WRITE_MAX_BYTES;
const maxDuplexChannelProtocolErrors =
options.duplexChannelLimits?.maxProtocolErrors ??
MAX_DUPLEX_CHANNEL_PROTOCOL_ERRORS;
@@ -1025,42 +1021,11 @@ export function createPluginWorkerHandle(
// JSON-RPC message sending
// -----------------------------------------------------------------------
- function sendMessage(message: unknown, meterDuplexWrite = false): void {
+ function sendMessage(message: unknown): void {
if (!childProcess?.stdin?.writable) {
throw new Error(`Worker process for plugin "${pluginId}" is not writable`);
}
const serialized = serializeMessage(message as any);
- const ledger = duplexAggregateByteLedger;
- if (meterDuplexWrite && ledger) {
- // Charge the child-stdin transport buffer for a duplex write. The host writes
- // the serialized frame to the child stdin without backpressure. When the child
- // stops reading its stdin, the frame stays in the host stdin write buffer.
- // Reserve the exact serialized-frame byte count. That count includes the JSON
- // escaping and the newline framing, so the ledger covers the retained transport
- // bytes, not only the raw payload. The RPC separately holds the raw payload
- // under a `pending_write` token, so the two tokens cover the peak of both
- // retentions at the same time.
- const bytes = Buffer.byteLength(serialized);
- const token = ledger.reserve("stdin_write", bytes);
- if (!token) {
- // The reservation would pass the aggregate ceiling. Fail closed before the
- // enqueue: do not write the frame. The duplex write caller ends the route.
- throw new DuplexAggregateBytesExceededError();
- }
- // Hold the token until the stream flushes the chunk. The write callback fires
- // when the stream hands the chunk to the operating system, so the bytes then
- // leave the host stream buffer. The RPC settle and the RPC timeout never
- // release this token. Only the flush, the stream error, the stream close, or
- // the worker exit releases it. Release through the outstanding-token set, so a
- // later stream-error or worker-exit sweep never double-releases the same token.
- pendingStdinWriteTokens.add(token);
- childProcess.stdin.write(serialized, () => {
- if (pendingStdinWriteTokens.delete(token)) {
- ledger.release(token);
- }
- });
- return;
- }
childProcess.stdin.write(serialized);
}
@@ -1784,27 +1749,22 @@ export function createPluginWorkerHandle(
// 7. route lifetime — the milliseconds from the open to the terminal end.
// One buffered data chunk retained for a late listener drain. It carries the raw
- // chunk bytes and the aggregate byte token that reserved its raw bytes. The
- // token is `null` when no ledger is injected.
+ // chunk bytes.
interface BufferedDuplexChunk {
chunk: Uint8Array;
- token: ReservationToken | null;
}
// One pre-bind data event, normalized to the narrow duplex-event schema. The host
- // retains only these bounded scalar fields plus the reservation token, never the
- // original arbitrary notification graph. The bind re-resolves the pair from
- // `workerSessionId`, so a frame whose pair does not match the bound pair still
- // fails closed.
+ // retains only these bounded scalar fields, never the original arbitrary
+ // notification graph. The bind re-resolves the pair from `workerSessionId`, so a
+ // frame whose pair does not match the bound pair still fails closed.
interface HeldDuplexEvent {
workerSessionId: string;
chunk: Uint8Array;
- token: ReservationToken | null;
}
// One pre-bind exit event, normalized to the narrow duplex-event schema.
interface HeldDuplexExitEvent {
workerSessionId: string;
exitCode: number | null;
- token: ReservationToken | null;
}
interface DuplexChannelRoute {
@@ -1825,18 +1785,32 @@ export function createPluginWorkerHandle(
// lost and a frame whose pair does not match the bound pair still fails closed.
// The hold ceiling is `maxDuplexChannelPreBindHoldFrames`, one frame above the
// buffered bound, so the replay's buffered-bound check ends the route, not the
- // hold. Each held event carries the aggregate byte token that reserved its raw
- // bytes; the host never retains the original arbitrary notification graph.
+ // hold. The host never retains the original arbitrary notification graph.
preBind: HeldDuplexEvent[];
+ // The cumulative raw bytes `preBind` holds. `DUPLEX_ROUTE_INPUT_MAX_BYTES`
+ // bounds this route-local counter, so a worker cannot flood the pre-bind
+ // hold with bytes the frame-count ceiling alone does not catch (a handful
+ // of large frames, each well under the frame-count limit). This counter
+ // covers only the hold; the post-bind buffered queue counts its own bytes
+ // in `bufferedChars`, against the same named bound.
+ preBindBytes: number;
// The single bounded exit event that arrived before the bind. An exit never
// consumes a data hold slot, so a worker that batches an exit among enough data
// frames to fill the hold cannot crowd out a data frame. The bind replays the
// held data events first, then this exit last.
preBindExit: HeldDuplexExitEvent | null;
- // The authoritative registry of every aggregate byte token this route still
- // holds, across the pre-bind, buffered, and terminal-buffered representations.
- // The byte cleanup releases every token here exactly once.
- retainedTokens: Set<ReservationToken>;
+ // The cumulative raw payload bytes of every host→worker write this route
+ // has sent but the worker has not yet acknowledged. The window this
+ // counter tracks spans both retained representations of a pending write:
+ // the raw payload the pending request object holds, and the serialized
+ // child-stdin frame the host builds from it and writes before the request
+ // settles. `DUPLEX_ROUTE_PENDING_WRITE_MAX_BYTES` bounds this route-local
+ // counter, independent of `pendingRequests`: a worker that never replies
+ // cannot make one route hold an unbounded number of write bytes, even
+ // under a raised pending-request count. `sendBoundedRequest` increments it
+ // before the send and decrements it once the request settles, on every
+ // path: a reply, a rejection, a timeout, a worker exit, or a shutdown.
+ pendingWriteBytes: number;
}
// The live duplex routes on this worker, keyed by the exact
// `{ hostRouteId, workerSessionId }` pair. The host binds one pair once, at
@@ -1862,89 +1836,11 @@ export function createPluginWorkerHandle(
// time per route, so a double terminalize never releases two slots.
const duplexRouteSlotHolders = new Set<DuplexChannelRoute>();
- // The process-owned aggregate byte ledger. The manager injects it, so one gauge
- // bounds the aggregate retained bytes across every route in the process. When it
- // is absent, the worker retains duplex bytes unbounded (a unit test constructs it
- // this way).
- const duplexAggregateByteLedger = options.duplexAggregateByteLedger ?? null;
- // The outstanding child-stdin transport tokens for duplex writes. Each token
- // covers one serialized frame the host stdin write buffer still retains. The
- // write callback releases a token on the flush; a stream error, a stream close,
- // or a worker exit releases every remaining token, because each of those
- // discards the stdin write buffer. The set is the release guard, so a token
- // releases one time across the two paths.
- const pendingStdinWriteTokens = new Set<ReservationToken>();
- // Release every outstanding child-stdin transport token. A stream error, a
- // stream close, or a worker exit calls this, because each discards the stdin
- // write buffer. The `delete` guard drops each token one time, so a later flush
- // callback or a second sweep releases nothing again.
- function releaseAllPendingStdinWriteTokens(): void {
- if (!duplexAggregateByteLedger) return;
- for (const token of pendingStdinWriteTokens) {
- duplexAggregateByteLedger.release(token);
- }
- pendingStdinWriteTokens.clear();
- }
// The terminalized routes that still hold buffered bytes for a late listener
// drain. A terminalized route leaves the opening and live maps, so this registry
- // keeps the worker-exit sweep able to release its still-charged buffered tokens.
+ // keeps the worker-exit sweep able to clear its still-buffered representations.
const terminalDuplexRoutes = new Set<DuplexChannelRoute>();
- // Reserve `bytes` for one route retention against the aggregate ledger. Return
- // the held token, or `null` when the reservation would pass the ceiling. When no
- // ledger is present, return `"no-ledger"`, so the caller admits the retention
- // with no token. The helper adds a real token to `route.retainedTokens`.
- function reserveRouteBytes(
- route: DuplexChannelRoute,
- owner: DuplexAggregateTokenOwner,
- bytes: number,
- ): ReservationToken | "no-ledger" | null {
- if (!duplexAggregateByteLedger) return "no-ledger";
- const token = duplexAggregateByteLedger.reserve(owner, bytes);
- if (!token) return null;
- route.retainedTokens.add(token);
- return token;
- }
-
- // Transfer a held route token to a new owner label. The token identity and the
- // reserved bytes stay the same, so no admission gap opens between two
- // representations of the same retained bytes.
- function transferRouteToken(
- token: ReservationToken | null,
- owner: DuplexAggregateTokenOwner,
- ): void {
- if (!token) return;
- duplexAggregateByteLedger?.transfer(token, owner);
- }
-
- // Release one held route token and drop it from the route registry, in one
- // synchronous step. A `null` token (no ledger) releases nothing.
- function releaseRouteToken(route: DuplexChannelRoute, token: ReservationToken | null): void {
- if (!token) return;
- route.retainedTokens.delete(token);
- duplexAggregateByteLedger?.release(token);
- }
-
- // Release every token a route still holds and clear its retained representations,
- // exactly once. A later call finds an empty registry and releases nothing, so the
- // helper is idempotent. Every terminal-discard, open-failure, bind-replay
- // failure, close-acknowledgement failure, worker-exit, and shutdown path calls
- // it. Terminal map deletion is never treated as proof that retained bytes are
- // gone; this registry is authoritative for byte cleanup.
- function discardRouteRetained(route: DuplexChannelRoute): void {
- if (duplexAggregateByteLedger) {
- for (const token of route.retainedTokens) {
- duplexAggregateByteLedger.release(token);
- }
- }
- route.retainedTokens.clear();
- route.preBind = [];
- route.preBindExit = null;
- route.buffered = [];
- route.bufferedChars = 0;
- terminalDuplexRoutes.delete(route);
- }
-
// Try to reserve one aggregate route slot for a route. Return true when the
// route holds a slot after the call. When no controller is present, the route
// always holds a slot.
@@ -2042,34 +1938,16 @@ export function createPluginWorkerHandle(
route.terminalized = true;
route.state = "closed";
route.listener = null;
- // Stop admission first. Then move the byte cleanup. Keep the buffered chunks the
- // host accepted before the route ended, so a listener that attaches after the
- // end still drains them. A frame can end the route during the pre-bind replay,
- // before a listener attaches, and the chunks the host accepted before that frame
- // are valid data the listener must still receive. The buffered bytes stay
- // bounded by the pre-bind buffered bound, and `onData` clears and releases them
- // once it drains them.
- //
- // The buffered records keep their tokens; move each token to the terminal owner
- // label and register the route in the terminal registry, so the worker-exit
- // sweep can release the still-charged bytes later. Do not release a buffered
- // token at map deletion; terminal map deletion is never proof the bytes are
- // gone. Release every other token the route still holds (the pre-bind events,
- // the held exit, and any stranded token), because they never reach a listener.
- const bufferedTokens = new Set<ReservationToken>();
- for (const record of route.buffered) {
- if (record.token) {
- bufferedTokens.add(record.token);
- transferRouteToken(record.token, "terminal_buffered");
- }
- }
- for (const token of [...route.retainedTokens]) {
- if (!bufferedTokens.has(token)) {
- route.retainedTokens.delete(token);
- duplexAggregateByteLedger?.release(token);
- }
- }
+ // Stop admission first. Keep the buffered chunks the host accepted before the
+ // route ended, so a listener that attaches after the end still drains them. A
+ // frame can end the route during the pre-bind replay, before a listener
+ // attaches, and the chunks the host accepted before that frame are valid data
+ // the listener must still receive. The buffered bytes stay bounded by the
+ // pre-bind buffered bound, and `onData` clears them once it drains them.
+ // Register the route in the terminal registry, so the worker-exit sweep can
+ // clear the still-buffered chunks later.
route.preBind = [];
+ route.preBindBytes = 0;
route.preBindExit = null;
if (route.buffered.length > 0) {
terminalDuplexRoutes.add(route);
@@ -2156,16 +2034,8 @@ export function createPluginWorkerHandle(
}
// Route one duplex channel data frame. On a live route with a listener, deliver
- // the chunk transiently. On a live route with no listener, buffer the chunk and
- // charge its raw bytes against the aggregate ledger. The `carried` argument marks
- // a bind-replay frame: it carries the pre-bind token the buffered record must
- // reuse, so the replay transfers the token to the buffered representation with no
- // decrement and no re-reserve. A live frame passes no `carried`, so it reserves a
- // fresh buffered token. A `carried.token` of `null` means no ledger is present.
- function routeDuplexChannelData(
- notification: JsonRpcNotification,
- carried?: { token: ReservationToken | null },
- ): void {
+ // the chunk transiently. On a live route with no listener, buffer the chunk.
+ function routeDuplexChannelData(notification: JsonRpcNotification): void {
const params = isRecord(notification.params) ? notification.params : {};
const hostRouteId = readNonEmptyString(params.hostRouteId);
const workerSessionId = readNonEmptyString(params.workerSessionId);
@@ -2200,15 +2070,13 @@ export function createPluginWorkerHandle(
const chunk = rawChunk instanceof Uint8Array ? rawChunk : decodeChannelBytes(rawChunk);
if (chunk === null || chunk.byteLength === 0) {
// The exact pair matches, but the chunk is malformed or empty. Count one
- // per-route protocol error. Release a carried replay token first.
- releaseRouteToken(route, carried?.token ?? null);
+ // per-route protocol error.
recordDuplexChannelProtocolError(route);
return;
}
if (chunk.byteLength > maxDuplexChannelChunkChars) {
// One inbound chunk is larger than the per-chunk limit. End the route at
// once. Do not count the chunk as a protocol error.
- releaseRouteToken(route, carried?.token ?? null);
void terminalizeDuplexChannelRoute(route);
return;
}
@@ -2217,16 +2085,13 @@ export function createPluginWorkerHandle(
// data past the cap.
const chunkBytes = chunk.byteLength;
if (route.totalDataBytes + chunkBytes > maxDuplexChannelTotalDataBytes) {
- releaseRouteToken(route, carried?.token ?? null);
void terminalizeDuplexChannelRoute(route);
return;
}
route.totalDataBytes += chunkBytes;
if (route.listener) {
- // A listener is attached. Deliver the chunk transiently and release a carried
- // replay token after the synchronous delivery boundary.
+ // A listener is attached. Deliver the chunk transiently.
deliverDuplexChannelChunk(route.listener, chunk);
- releaseRouteToken(route, carried?.token ?? null);
return;
}
// No listener attached yet. Buffer the frame under the pre-bind bounds. End
@@ -2235,28 +2100,10 @@ export function createPluginWorkerHandle(
route.buffered.length + 1 > maxDuplexChannelPreBindFrames ||
route.bufferedChars + chunk.length > maxDuplexChannelPreBindChars
) {
- releaseRouteToken(route, carried?.token ?? null);
void terminalizeDuplexChannelRoute(route);
return;
}
- let token: ReservationToken | null;
- if (carried) {
- // The bind replay transfers the pre-bind token to the buffered
- // representation. No decrement and no re-reserve, so no admission gap opens.
- token = carried.token;
- transferRouteToken(token, "buffered_chunk");
- } else {
- // A live frame reserves a fresh buffered token for its exact raw bytes.
- const reserved = reserveRouteBytes(route, "buffered_chunk", chunkBytes);
- if (reserved === null) {
- // The aggregate ceiling rejected the reservation. Retain nothing and fail
- // closed with the fixed marker.
- void terminalizeDuplexChannelRoute(route);
- return;
- }
- token = reserved === "no-ledger" ? null : reserved;
- }
- route.buffered.push({ chunk, token });
+ route.buffered.push({ chunk });
route.bufferedChars += chunk.length;
}
@@ -2319,23 +2166,10 @@ export function createPluginWorkerHandle(
return;
}
if (notification.method === DUPLEX_CHANNEL_EXIT_NOTIFICATION) {
- // Normalize the exit to the narrow duplex-event schema. An exit retains only
- // a bounded scalar, so reserve a zero-byte token that carries the record
- // through the one cleanup path. Release any earlier held exit token first, so
- // a replaced exit never leaks its reservation.
+ // Normalize the exit to the narrow duplex-event schema. A replaced exit
+ // simply overwrites the earlier held exit.
const exitCode = typeof params.exitCode === "number" ? params.exitCode : null;
- if (route.preBindExit) releaseRouteToken(route, route.preBindExit.token);
- const reserved = reserveRouteBytes(route, "pre_bind_event", 0);
- if (reserved === null) {
- // The aggregate ceiling rejected the reservation. Fail closed.
- void terminalizeDuplexChannelRoute(route);
- return;
- }
- route.preBindExit = {
- workerSessionId,
- exitCode,
- token: reserved === "no-ledger" ? null : reserved,
- };
+ route.preBindExit = { workerSessionId, exitCode };
return;
}
// A data event. Validate and normalize it to the narrow duplex-event schema
@@ -2350,20 +2184,16 @@ export function createPluginWorkerHandle(
recordDuplexChannelProtocolError(route);
return;
}
- // Reserve the exact retained raw byte count before the host holds the event.
- const reserved = reserveRouteBytes(route, "pre_bind_event", chunk.byteLength);
- if (reserved === null) {
- // The aggregate ceiling rejected the reservation. The caller retains nothing
- // and the route fails closed with the fixed marker.
- log.warn({ pluginId, reason: DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED }, "duplex pre-bind hold rejected");
+ // The route-local input bound covers the pre-bind hold directly. A handful
+ // of large frames can pass the frame-count ceiling above while still
+ // flooding the host with bytes, so this route ends at once on the byte
+ // bound, the same way the post-bind buffered queue already does.
+ if (route.preBindBytes + chunk.byteLength > maxDuplexChannelPreBindChars) {
void terminalizeDuplexChannelRoute(route);
return;
}
- route.preBind.push({
- workerSessionId,
- chunk,
- token: reserved === "no-ledger" ? null : reserved,
- });
+ route.preBind.push({ workerSessionId, chunk });
+ route.preBindBytes += chunk.byteLength;
}
// Replay the frames a route held before it bound. The route is live now, so the
@@ -2376,36 +2206,29 @@ export function createPluginWorkerHandle(
function replayPreBindDuplexFrames(route: DuplexChannelRoute): void {
const held = route.preBind;
route.preBind = [];
+ route.preBindBytes = 0;
for (const event of held) {
if (route.terminalized) {
- // The route ended mid-replay. Release the remaining held tokens, so the
- // unreplayed events retain nothing.
- releaseRouteToken(route, event.token);
+ // The route ended mid-replay. The remaining held events are dropped.
continue;
}
// Reconstruct a transient data notification for the exact-pair routing. The
// host holds only the bounded event, so it builds this notification for the
- // routing step and discards it at once. The carried token moves to the
- // buffered representation without a decrement or a re-reserve.
- routeDuplexChannelData(
- {
- jsonrpc: "2.0",
- method: DUPLEX_CHANNEL_DATA_NOTIFICATION,
- params: {
- hostRouteId: route.hostRouteId,
- workerSessionId: event.workerSessionId,
- chunk: event.chunk,
- },
+ // routing step and discards it at once.
+ routeDuplexChannelData({
+ jsonrpc: "2.0",
+ method: DUPLEX_CHANNEL_DATA_NOTIFICATION,
+ params: {
+ hostRouteId: route.hostRouteId,
+ workerSessionId: event.workerSessionId,
+ chunk: event.chunk,
},
- { token: event.token },
- );
+ });
}
const heldExit = route.preBindExit;
route.preBindExit = null;
if (heldExit) {
- // The exit retains only a bounded scalar. Release its placeholder token, then
- // resolve the wait through the exact-pair routing when the route still lives.
- releaseRouteToken(route, heldExit.token);
+ // Resolve the wait through the exact-pair routing when the route still lives.
if (!route.terminalized) {
routeDuplexChannelExit({
jsonrpc: "2.0",
@@ -2448,10 +2271,14 @@ export function createPluginWorkerHandle(
releaseDuplexRouteSlot(route);
settleRouteWait(route, { exitCode: null });
}
- // Release every token the route still holds and clear its retained
- // representations, exactly once. A route already drained by a late listener
- // holds no token, so this is harmless and leaves the ledger at zero.
- discardRouteRetained(route);
+ // Clear the route's retained buffers, exactly once. A route already drained
+ // by a late listener holds nothing, so this is harmless.
+ route.preBind = [];
+ route.preBindBytes = 0;
+ route.preBindExit = null;
+ route.buffered = [];
+ route.bufferedChars = 0;
+ terminalDuplexRoutes.delete(route);
}
// Clear the terminal registry. Every route in it was just discarded above.
terminalDuplexRoutes.clear();
@@ -2486,8 +2313,9 @@ export function createPluginWorkerHandle(
terminalized: false,
settleWait,
preBind: [],
+ preBindBytes: 0,
preBindExit: null,
- retainedTokens: new Set<ReservationToken>(),
+ pendingWriteBytes: 0,
};
// Reserve one aggregate route slot before any work. When the process-scoped
// ceiling is full, reject with the fixed route-busy error and open nothing, so
@@ -2579,58 +2407,30 @@ export function createPluginWorkerHandle(
void terminalizeDuplexChannelRoute(route);
return;
}
- // Reserve the exact raw byte count of a host→worker write against the
- // aggregate ledger before `callInternal` retains the payload. A pending write
- // RPC holds `params.data` until it settles, so this reservation bounds the
- // aggregate host→worker pending-write bytes across every route. A stop
- // request carries no payload, so it reserves nothing. When no ledger is
- // present, admit the write with no token (a unit test constructs the handle
- // this way).
- let pendingWriteToken: ReservationToken | null = null;
- if (method === "duplexChannelWrite" && duplexAggregateByteLedger) {
- pendingWriteToken = duplexAggregateByteLedger.reserve("pending_write", writeByteCount ?? 0);
- if (!pendingWriteToken) {
- // The reservation would pass the aggregate ceiling. Retain nothing, do
- // not enqueue the RPC, and end the route fail-closed with the aggregate
- // marker, not the route-busy marker.
- log.warn(
- { pluginId, reason: DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED },
- "duplex pending write reservation rejected",
- );
- void terminalizeDuplexChannelRoute(route);
- return;
- }
+ // The route-local pending-write byte bound: a worker that never replies
+ // cannot make one route hold an unbounded number of write bytes, even
+ // under a raised pending-request count. A stop request carries no
+ // payload, so it never counts against this bound.
+ const pendingWriteBytesToCharge = method === "duplexChannelWrite" ? writeByteCount ?? 0 : 0;
+ if (route.pendingWriteBytes + pendingWriteBytesToCharge > maxDuplexRoutePendingWriteBytes) {
+ void terminalizeDuplexChannelRoute(route);
+ return;
}
route.pendingRequests += 1;
- // Meter the child-stdin transport buffer only for a duplex write. A stop
- // request carries a tiny fixed frame that the host never lets grow, so it
- // does not meter or reject. The write path reserves the serialized frame in
- // `sendMessage` right before the stdin write.
- const meterDuplexWrite = method === "duplexChannelWrite";
- void callInternal(method, params, duplexChannelOpenTimeoutMs, undefined, meterDuplexWrite)
- .catch((err: unknown) => {
- if (err instanceof DuplexAggregateBytesExceededError) {
- // The transport reservation failed. The host did not write the frame.
- // End the route fail-closed with the aggregate marker, not the
- // route-busy marker.
- log.warn(
- { pluginId, reason: DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED },
- "duplex stdin write reservation rejected",
- );
- void terminalizeDuplexChannelRoute(route);
- }
- })
+ route.pendingWriteBytes += pendingWriteBytesToCharge;
+ void callInternal(method, params, duplexChannelOpenTimeoutMs, undefined)
+ // A send failure, an RPC rejection, a timeout, or a worker exit ends up
+ // here. The route already handles a lost write through its own state
+ // (a worker exit or a stop rejects every pending request through the
+ // normal call machinery), so this fire-and-forget call swallows the
+ // rejection instead of leaving it unhandled.
+ .catch(() => {})
.finally(() => {
route.pendingRequests -= 1;
- // Release the pending-write token one time, after the RPC settles on any
- // path: success, error, timeout, worker exit, or shutdown. The token is
- // not in `route.retainedTokens`, so route terminalization never releases
- // it; only this settlement releases it. The separate `stdin_write` token
- // covers the serialized frame and releases on the stream flush, the
- // stream error, the stream close, or the worker exit, never here.
- if (pendingWriteToken) {
- duplexAggregateByteLedger?.release(pendingWriteToken);
- }
+ // Release the route-local pending-write bytes one time, after the RPC
+ // settles on any path: success, error, timeout, worker exit, or
+ // shutdown.
+ route.pendingWriteBytes -= pendingWriteBytesToCharge;
});
};
@@ -2642,11 +2442,9 @@ export function createPluginWorkerHandle(
route.buffered = [];
route.bufferedChars = 0;
for (const record of pending) {
- // Deliver each buffered record, drop its retained buffer, and release
- // its exact token before the drain proceeds to the next record. A
- // terminal route drains through this same code.
+ // Deliver each buffered record before the drain proceeds to the next
+ // record. A terminal route drains through this same code.
deliverDuplexChannelChunk(listener, record.chunk);
- releaseRouteToken(route, record.token);
}
// The buffered records are gone, so the route no longer holds terminal
// bytes for a late listener.
@@ -2664,8 +2462,8 @@ export function createPluginWorkerHandle(
}
// Encode the raw bytes to the wire-safe base64 form (JSON carries no
// binary type) and pass the exact raw byte count separately, so the
- // pending-write ledger reservation charges the real payload bytes, not
- // the inflated base64 string length.
+ // pending-write byte bound charges the real payload bytes, not the
+ // inflated base64 string length.
sendBoundedRequest(
"duplexChannelWrite",
{
@@ -2961,16 +2759,11 @@ export function createPluginWorkerHandle(
readline.on("line", handleLine);
}
- // Release the outstanding child-stdin transport tokens when the stdin stream
- // errors or closes, because each discards the stdin write buffer. The `error`
- // listener also stops an unhandled EPIPE from a child that closed its stdin.
+ // The `error` listener stops an unhandled EPIPE from a child that closed its
+ // stdin.
if (child.stdin) {
- child.stdin.on("error", () => {
- releaseAllPendingStdinWriteTokens();
- });
- child.stdin.on("close", () => {
- releaseAllPendingStdinWriteTokens();
- });
+ child.stdin.on("error", () => {});
+ child.stdin.on("close", () => {});
}
// Capture stderr for logging
@@ -3023,12 +2816,6 @@ export function createPluginWorkerHandle(
childProcess = null;
startedAt = null;
- // The worker exit discards the child-stdin write buffer, so release every
- // outstanding transport token. The RPC rejections below never release these
- // tokens; only this sweep, a stream flush, a stream error, or a stream close
- // releases them.
- releaseAllPendingStdinWriteTokens();
-
// Reject all pending requests
rejectAllPending(
new Error(formatWorkerFailureMessage(
@@ -3364,7 +3151,6 @@ export function createPluginWorkerHandle(
params: HostToWorkerMethods[M][0],
timeoutMs?: number,
executeLogSink?: ExecuteLogSink,
- meterDuplexWrite = false,
): Promise<HostToWorkerMethods[M][1]> {
const rpcPromise = new Promise<HostToWorkerMethods[M][1]>((resolve, reject) => {
if (!childProcess?.stdin?.writable) {
@@ -3438,26 +3224,19 @@ export function createPluginWorkerHandle(
...createRequest(method, params, id),
...(invocation ? { paperclipInvocation: invocation } : {}),
};
- sendMessage(request, meterDuplexWrite);
+ sendMessage(request);
} catch (err) {
clearTimeout(timer);
pendingRequests.delete(id);
clearInvocation(invocation);
clearExecuteRoute(invocation?.id);
- if (err instanceof DuplexAggregateBytesExceededError) {
- // The transport reservation failed before the write. Reject with the
- // typed error unwrapped, so the duplex write caller ends the route
- // fail-closed with the aggregate marker.
- reject(err);
- } else {
- reject(
- new Error(
- `Failed to send "${method}" to worker: ${
- err instanceof Error ? err.message : String(err)
- }`,
- ),
- );
- }
+ reject(
+ new Error(
+ `Failed to send "${method}" to worker: ${
+ err instanceof Error ? err.message : String(err)
+ }`,
+ ),
+ );
}
});
@@ -3630,20 +3409,20 @@ export interface PluginWorkerManagerOptions {
* stays upstream admission only.
*/
maxConcurrentDuplexRoutes?: number | null;
- /**
- * The process-owned aggregate byte ledger. The process root creates one ledger
- * from validated configuration and passes it here. The manager injects the same
- * object into every worker handle, so one gauge bounds the aggregate retained
- * bytes across every route in the process. When it is absent, the worker retains
- * duplex bytes unbounded. The manager never makes a fresh default ledger.
- */
- duplexAggregateByteLedger?: DuplexAggregateByteLedger | null;
}
/**
* The default process-scoped aggregate ceiling for concurrent duplex channel
* routes. It caps the manager-wide resource, not one agent's run budget. The host
* reports an explicit route-busy outcome when the ceiling is full.
+ *
+ * Known aggregate behavior: this ceiling bounds route count only, not
+ * retained bytes. Each HTTP/2 bridge route bounds its own retained body
+ * bytes to 8,388,608 bytes (see `HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS` in
+ * `http2-bridge-server.ts`), so this route ceiling caps the process's
+ * aggregate retained body bytes at 128 * 8,388,608 = 1,073,741,824 bytes
+ * (1 GiB). This is accepted, known behavior, not a defect: the process
+ * tracks no aggregate byte ledger across routes.
*/
export const DEFAULT_MAX_CONCURRENT_DUPLEX_ROUTES = 128;
@@ -3709,11 +3488,6 @@ export function createPluginWorkerManager(
const duplexRouteSlots = createDuplexRouteSlotController(
managerOptions?.maxConcurrentDuplexRoutes,
);
- // The one shared, process-owned aggregate byte ledger. The manager injects the
- // same object into every worker handle, so one gauge bounds the aggregate
- // retained bytes across every route in the process. It is `null` when the
- // process root injected no ledger.
- const duplexAggregateByteLedger = managerOptions?.duplexAggregateByteLedger ?? null;
return {
async startWorker(
@@ -3735,11 +3509,9 @@ export function createPluginWorkerManager(
}
const handle = createPluginWorkerHandle(pluginId, {
- // Inject the shared process-scoped route-slot controller and the shared
- // process-owned aggregate byte ledger, unless the caller already supplied
- // its own (a test may inject its own).
+ // Inject the shared process-scoped route-slot controller, unless the
+ // caller already supplied its own (a test may inject its own).
duplexRouteSlots,
- duplexAggregateByteLedger,
...options,
});
workers.set(pluginId, handle);