Skip to content

armgit / rea

Public Imported from GitHub
0

Reverse engineer anything with agents, from app behavior down to native binaries.

feat(captures): make the HAR/SAZ/qlog decoder heap configurable (#6299)

morluto306ed799320e
diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md
index adf475e0f..a75b279f0 100644
--- a/.agents/skills/reverse-engineer-anything/SKILL.md
+++ b/.agents/skills/reverse-engineer-anything/SKILL.md
@@ -126,7 +126,10 @@ explicit path or endpoint and do not need it.
connect Everywhere's separate MCP server for live traffic.
- Retained HAR/SAZ/native mitmproxy/aioquic qlog capture: `inspect_web_network_capture`. Select
original record ordinals when useful; inspect producer fields and byte/number
- sidecars without fetching recorded URLs or claiming live attribution. See
+ sidecars without fetching recorded URLs or claiming live attribution. HAR/SAZ/qlog
+ old-generation heap is `REA_CAPTURE_DECODER_OLD_GENERATION_MIB` (default 192 MiB,
+ floor 8, maximum 16384), set where REA starts. Host memory and `NODE_OPTIONS` do not
+ raise it. See
[historical captures](https://github.com/morluto/rea/blob/main/docs/web-network-captures.md) for the exact upstream
profile and credential exclusions. Qlog support starts with aioquic 1.5.0
JSON 0.3; stream coverage and endpoint loss declarations do not recover bodies
diff --git a/docs/web-network-captures.md b/docs/web-network-captures.md
index a998083c8..e4afbde3f 100644
--- a/docs/web-network-captures.md
+++ b/docs/web-network-captures.md
@@ -716,8 +716,17 @@ budget. Those budgets have no runtime override. `capture_timeout_ms` does not
raise them. Exceeding a budget returns no partial success.
Oversized selected captures return an input error identifying `capture_path`
and the byte budget; owned reply/output failures remain output or provider errors.
-HAR, SAZ and qlog run in an owned
-192 MiB Node old-generation heap with one V8 worker. Mitmproxy decoding applies
+HAR, SAZ and qlog run in an owned Node old-generation heap with one V8 worker.
+`REA_CAPTURE_DECODER_OLD_GENERATION_MIB` is a decimal integer of mebibytes from
+8 to 16384. Unset uses 192, the previous constant. Invalid values fail
+configuration and name the setting; they do not fall back to 192. The setting
+is read from the environment that starts REA, so a change needs a restart.
+A memory failure names the effective mebibytes, whether they came from the
+default or the setting, the default, the floor, and the maximum. The decoder
+passes that value as `--max-old-space-size` and removes inherited
+`NODE_OPTIONS`, so additional host memory and `NODE_OPTIONS` do not raise the
+heap. The 8 MiB semi-space stays fixed; a smaller old generation aborts Node
+during startup instead of enforcing a limit. Mitmproxy decoding applies
Linux limits of 768 MiB address space, 30 seconds CPU and 96 MiB file output;
both commands use `capture_timeout_ms` (default 30000) and independently supervised
process cleanup. Owned decoder diagnostics retain at most 1 MiB across stdout
@@ -725,15 +734,18 @@ and stderr; the collector drops overflow as it arrives, reports the limit
failure and verifies cleanup. Heap limits alone do not establish aggregate RSS limits.
SAZ and qlog refuse decoding when neither a Node heap limit nor a Windows Job Object bound can be enforced. Existing HAR availability is preserved.
When REA runs on Bun, the HAR/SAZ/qlog decoder runs on Bun, which ignores these V8 heap
-flags; its limitations and resource failures then omit the heap sizes. On
+flags; its limitations and resource failures then omit the V8 heap sizes. On
Windows, the decoder's owned Job Object instead limits its committed memory to
-704 MiB, the 192 MiB budget plus a 512 MiB reserve for Bun itself. The
-limitations state that limit, and reaching it returns `resource_constraint`
-with `process_memory_limit_bytes`, whatever exit status Bun reports.
+the effective `REA_CAPTURE_DECODER_OLD_GENERATION_MIB` value plus a 512 MiB
+reserve for Bun itself. The default 192 MiB setting therefore yields 704 MiB.
+The limitations state that limit, and reaching it returns `resource_constraint`
+with `process_memory_limit_bytes`, whatever exit status Bun reports. Other Bun
+hosts enforce no decoder heap limit, so raising the setting does not bound them.
Observed decoder memory exhaustion returns `resource_constraint`, with the
-selected capture, applicable limits and retained process diagnostics. HAR/SAZ/qlog's
-old-generation heap is fixed; inherited `NODE_OPTIONS` cannot raise it.
+selected capture, applicable limits and retained process diagnostics. Raising
+host memory or `NODE_OPTIONS` does not raise the old-generation heap. Set
+`REA_CAPTURE_DECODER_OLD_GENERATION_MIB` and restart REA.
Record selection happens after complete decoding and does not reduce that
workload. Use a smaller capture exported by its producer or another decoder
with sufficient capacity, retaining the original for provenance. Subsets do
diff --git a/src/application/Doctor.test.ts b/src/application/Doctor.test.ts
index 03957e636..3187a4b88 100644
--- a/src/application/Doctor.test.ts
+++ b/src/application/Doctor.test.ts
@@ -165,6 +165,7 @@ describe("doctor native analysis configuration", () => {
it.each([
["REA_ANALYSIS_PROVIDER", "Hopper"],
["REA_REGEX_MATCHING_DEADLINE_MS", "0"],
+ ["REA_CAPTURE_DECODER_OLD_GENERATION_MIB", "7"],
["REA_REGEX_WORKER_HEAP_REQUEST_MIB", "6"],
["REA_GHIDRA_STARTUP_TIMEOUT_MS", "abc"],
["REA_SWIFTC_COMPILE_TIMEOUT_MS", "0"],
diff --git a/src/application/SetupRegistrationEnvironment.ts b/src/application/SetupRegistrationEnvironment.ts
index 027acb77e..864c788d5 100644
--- a/src/application/SetupRegistrationEnvironment.ts
+++ b/src/application/SetupRegistrationEnvironment.ts
@@ -3,6 +3,7 @@ import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
const SELECTED_REGISTRATION_SETTINGS = [
"REA_ANALYSIS_PROVIDER",
+ "REA_CAPTURE_DECODER_OLD_GENERATION_MIB",
"REA_GHIDRA_STARTUP_TIMEOUT_MS",
"REA_HOPPER_STARTUP_TIMEOUT_MS",
"REA_JEB_MCP_TIMEOUT_MS",
diff --git a/src/browser/history/CaptureFailures.test.ts b/src/browser/history/CaptureFailures.test.ts
index d6268645a..7508da5b0 100644
--- a/src/browser/history/CaptureFailures.test.ts
+++ b/src/browser/history/CaptureFailures.test.ts
@@ -36,10 +36,12 @@ it.each([
expect(projected).toMatchObject({
code: "resource_constraint",
retryable: false,
- message: expect.stringContaining("fixed V8 heap"),
+ message: expect.stringContaining(
+ "exhausted its 192 MiB V8 old-generation heap (source default)",
+ ),
remediation: {
action: expect.stringContaining(
- "smaller capture exported by its producer",
+ "The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 192 MiB (source default; default 192 MiB, floor 8 MiB, maximum 16384 MiB).",
),
},
details: {
@@ -52,8 +54,13 @@ it.each([
},
});
expect(JSON.stringify(projected)).not.toContain("rea doctor");
+ expect(JSON.stringify(projected)).not.toContain("no runtime override");
// Bun ignores the decoder's V8 heap flags, so no fixed heap is reported.
- const heapLimits = { old_generation_heap_mib: 192, semi_space_heap_mib: 8 };
+ const heapLimits = {
+ old_generation_heap_mib: 192,
+ old_generation_heap_source: "default",
+ semi_space_heap_mib: 8,
+ };
if (process.versions.bun === undefined)
expect(projected).toMatchObject({
details: { reported_limits: heapLimits },
diff --git a/src/browser/history/CaptureFailures.ts b/src/browser/history/CaptureFailures.ts
index 4fc854501..34600a7a6 100644
--- a/src/browser/history/CaptureFailures.ts
+++ b/src/browser/history/CaptureFailures.ts
@@ -21,11 +21,16 @@ import {
webNetworkCaptureTimeoutRecovery,
type InspectWebNetworkCaptureInput,
} from "../../domain/webNetworkCapture.js";
+import {
+ captureDecoderHeapRecovery,
+ DEFAULT_CAPTURE_DECODER_HEAP,
+ type CaptureDecoderHeap,
+} from "../../config/captureDecoderHeap.js";
import { OwnedCommandFailure } from "../../process/OwnedCommand.js";
import {
NODE_CAPTURE_HEAP_ENFORCED,
NODE_CAPTURE_HEAP_LIMITS,
- NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES,
+ captureDecoderProcessMemoryLimitBytes,
} from "./CaptureRelease.js";
const OPERATION = "inspect_web_network_capture";
@@ -39,6 +44,7 @@ export const historicalCaptureFailure = (
cause: unknown,
phase: "capture-read" | "descriptor-read" | "decoder",
options?: ExecutionOptions,
+ heap: CaptureDecoderHeap = DEFAULT_CAPTURE_DECODER_HEAP,
): AnalysisError => {
if (cause instanceof AnalysisError) return cause;
const descriptorRead = phase === "descriptor-read";
@@ -100,6 +106,7 @@ export const historicalCaptureFailure = (
input,
CAPTURE_PROCESS_MEMORY_LIMIT_REASON,
cause,
+ heap,
);
if (
phase === "decoder" &&
@@ -111,8 +118,9 @@ export const historicalCaptureFailure = (
)
return captureMemoryFailure(
input,
- `The ${input.format} decoder exhausted its fixed V8 heap`,
+ `The ${input.format} decoder exhausted its ${String(heap.oldGenerationMiB)} MiB V8 old-generation heap (source ${heap.source})`,
cause,
+ heap,
);
}
if (
@@ -224,8 +232,13 @@ export const captureMemoryFailure = (
input: InspectWebNetworkCaptureInput,
reason: string,
cause?: unknown,
-): AnalysisResourceConstraintError =>
- new AnalysisResourceConstraintError(
+ heap: CaptureDecoderHeap = DEFAULT_CAPTURE_DECODER_HEAP,
+): AnalysisResourceConstraintError => {
+ const processMemoryLimitBytes = captureDecoderProcessMemoryLimitBytes(
+ heap.oldGenerationMiB,
+ );
+ const recovery = captureDecoderHeapRecovery(heap);
+ return new AnalysisResourceConstraintError(
OPERATION,
"memory",
`${reason}: ${input.capture_path}. No complete capture result is available.`,
@@ -237,14 +250,14 @@ export const captureMemoryFailure = (
maximum_reply_bytes: WEB_NETWORK_CAPTURE_LIMITS.outputBytes,
...(input.format !== "mitmproxy" && NODE_CAPTURE_HEAP_ENFORCED
? {
- old_generation_heap_mib: NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB,
+ old_generation_heap_mib: heap.oldGenerationMiB,
+ old_generation_heap_source: heap.source,
semi_space_heap_mib: NODE_CAPTURE_HEAP_LIMITS.semiSpaceMiB,
}
: {}),
- ...(input.format !== "mitmproxy" &&
- NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES !== null
+ ...(input.format !== "mitmproxy" && processMemoryLimitBytes !== null
? {
- process_memory_limit_bytes: NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES,
+ process_memory_limit_bytes: processMemoryLimitBytes,
}
: {}),
...(cause instanceof OwnedCommandFailure
@@ -256,7 +269,7 @@ export const captureMemoryFailure = (
},
{
cause,
- remediationAction: `Use a smaller capture exported by its producer, or another decoder with sufficient capacity for the original capture. Preserve the original for provenance; a subset does not establish complete-capture coverage.${input.format !== "mitmproxy" && NODE_CAPTURE_HEAP_ENFORCED ? ` REA's Node capture old-generation heap is fixed at ${NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB} MiB; inherited NODE_OPTIONS cannot raise it.` : ""}${input.format !== "mitmproxy" && NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES !== null ? ` REA's Node capture decoder process commit limit is fixed at ${String(NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES)} bytes.` : ""}`,
+ remediationAction: `Use a smaller capture exported by its producer, or another decoder with sufficient capacity for the original capture. Preserve the original for provenance; a subset does not establish complete-capture coverage.${input.format !== "mitmproxy" ? ` ${recovery}` : ""}${input.format !== "mitmproxy" && processMemoryLimitBytes !== null ? ` The Windows Job Object commit limit for this decoder is ${String(processMemoryLimitBytes)} bytes, ${String(heap.oldGenerationMiB)} MiB plus a 512 MiB Bun reserve.` : ""}`,
...(cause instanceof OwnedCommandFailure && cause.snapshot !== null
? {
capturedOutput: {
@@ -268,6 +281,7 @@ export const captureMemoryFailure = (
: {}),
},
);
+};
/** Preserve the exact file owner when capture or descriptor acquisition needs cleanup. */
export const withCaptureReaderCleanup = (
diff --git a/src/browser/history/CaptureRelease.ts b/src/browser/history/CaptureRelease.ts
index 77f29df51..62dfecee4 100644
--- a/src/browser/history/CaptureRelease.ts
+++ b/src/browser/history/CaptureRelease.ts
@@ -1,3 +1,7 @@
+import {
+ CAPTURE_DECODER_SEMI_SPACE_MIB,
+ DEFAULT_CAPTURE_DECODER_OLD_GENERATION_MIB,
+} from "../../config/captureDecoderHeap.js";
import { javaScriptRuntime } from "../../domain/runtimeVersion.js";
import { javaScriptWorkerMemoryLimitBytes } from "../../process/JavaScriptWorkerMemoryLimit.js";
@@ -8,19 +12,30 @@ export const HAR_CAPTURE_PROVIDER_IDENTITY = {
} as const;
-/** Fixed V8 heap settings used by the owned Node capture decoder process. */
+/**
+ * Default V8 heap settings for the owned Node capture decoder process.
+ * The old generation is `REA_CAPTURE_DECODER_OLD_GENERATION_MIB`; the
+ * semi-space stays fixed.
+ */
export const NODE_CAPTURE_HEAP_LIMITS = {
- oldGenerationMiB: 192,
- semiSpaceMiB: 8,
+ oldGenerationMiB: DEFAULT_CAPTURE_DECODER_OLD_GENERATION_MIB,
+ semiSpaceMiB: CAPTURE_DECODER_SEMI_SPACE_MIB,
} as const;
+/** Windows Job Object commit limit for one resolved old-generation budget. */
+export const captureDecoderProcessMemoryLimitBytes = (
+ oldGenerationMiB: number,
+): number | null => javaScriptWorkerMemoryLimitBytes(oldGenerationMiB);
+
/** The Node capture decoder runs on this executable; Bun accepts but ignores V8 heap flags. */
export const NODE_CAPTURE_HEAP_ENFORCED =
javaScriptRuntime(process.versions).id === "node";
-/** Windows Job Object commit limit replacing the ignored heap flags under Bun; otherwise null. */
+/** Commit limit for the default heap. A configured heap uses `captureDecoderProcessMemoryLimitBytes`. */
export const NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES =
- javaScriptWorkerMemoryLimitBytes(NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB);
+ captureDecoderProcessMemoryLimitBytes(
+ NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB,
+ );
/** Native executable profile verified with unchanged mitmproxy raw-state decoding. */
export const MITMPROXY_CAPTURE_PROVIDER_IDENTITY = {
diff --git a/src/browser/history/HistoricalCaptureDecoder.ts b/src/browser/history/HistoricalCaptureDecoder.ts
index 6ec81366d..bb0c61cb6 100644
--- a/src/browser/history/HistoricalCaptureDecoder.ts
+++ b/src/browser/history/HistoricalCaptureDecoder.ts
@@ -9,10 +9,14 @@ import {
captureMemoryFailure,
withCaptureReaderCleanup,
} from "./CaptureFailures.js";
+import {
+ captureDecoderHeapRecovery,
+ resolveCaptureDecoderHeap,
+ type CaptureDecoderHeap,
+} from "../../config/captureDecoderHeap.js";
import {
NODE_CAPTURE_HEAP_ENFORCED,
- NODE_CAPTURE_HEAP_LIMITS,
- NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES,
+ captureDecoderProcessMemoryLimitBytes,
} from "./CaptureRelease.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import type { HistoricalCaptureFormatAdapter } from "./HistoricalCaptureFormatAdapter.js";
@@ -76,6 +80,7 @@ const replySchema = z.discriminatedUnion("ok", [
/** One owned snapshot/process/root lifecycle for replaceable historical format decoders. */
export class HistoricalCaptureDecoder {
readonly #resources: ArtifactResourceScope;
+ readonly #heap: CaptureDecoderHeap;
constructor(
readonly adapters: readonly HistoricalCaptureFormatAdapter[],
@@ -86,6 +91,18 @@ export class HistoricalCaptureDecoder {
resources = new ArtifactResourceScope(),
) {
this.#resources = resources;
+ // Adapters that launch the Node decoder already carry its parsed heap.
+ // Reuse that fact. This environment is only the fallback when none do,
+ // or when those facts disagree.
+ const shared = sharedCaptureDecoderHeap(adapters);
+ this.#heap = shared ?? resolveCaptureDecoderHeap(environment);
+ }
+
+ /** Heap the selected adapter launches, or the shared fallback when it has none. */
+ #reportedHeap(
+ adapter: HistoricalCaptureFormatAdapter | undefined,
+ ): CaptureDecoderHeap {
+ return adapter?.heap ?? this.#heap;
}
/** Retry retained readers and private roots at service shutdown. */
@@ -108,6 +125,7 @@ export class HistoricalCaptureDecoder {
cause,
"capture-read",
options,
+ this.#heap,
);
return err(
redactExplicitFailure(
@@ -126,10 +144,12 @@ export class HistoricalCaptureDecoder {
let result: Result<WebNetworkCapture, AnalysisError>;
let phase: "capture-read" | "decoder" = "capture-read";
let encodedReplyBytes = 0;
+ let reportedHeap = this.#heap;
try {
const adapter = this.adapters.find(
(candidate) => candidate.format === input.format,
);
+ reportedHeap = this.#reportedHeap(adapter);
if (adapter === undefined)
throw new AnalysisCapabilityUnavailableError(
input.format,
@@ -186,7 +206,12 @@ export class HistoricalCaptureDecoder {
// A decoder that survived a failed allocation may still reply; its
// reply is not trusted as complete once the limit was reached.
if (completed.memoryLimitReached === true)
- throw captureMemoryFailure(input, CAPTURE_PROCESS_MEMORY_LIMIT_REASON);
+ throw captureMemoryFailure(
+ input,
+ CAPTURE_PROCESS_MEMORY_LIMIT_REASON,
+ undefined,
+ reportedHeap,
+ );
let reply: z.output<typeof replySchema>;
try {
const replyFile = await readStableArtifact(
@@ -248,7 +273,12 @@ export class HistoricalCaptureDecoder {
},
]);
if (reply.reason === "resource-limit")
- throw captureMemoryFailure(input, reply.message);
+ throw captureMemoryFailure(
+ input,
+ reply.message,
+ undefined,
+ reportedHeap,
+ );
if (reply.reason === "decoder")
throw new ProviderAdapterError(input.format, OPERATION, {
diagnostics: {
@@ -288,7 +318,7 @@ export class HistoricalCaptureDecoder {
"This is retained producer evidence, without live browser transaction IDs, scenario provenance, execution attribution or deployment authenticity. Recorded URLs are never fetched.",
"Every record is decoded before selection; no partial success is returned on malformed input or a resource limit.",
"Known authentication headers/cookies and explicitly marked literal values are excluded. Unknown extension names do not establish sensitivity.",
- `Complete evidence is bounded by a ${String(WEB_NETWORK_CAPTURE_LIMITS.inputBytes)}-byte input budget, a ${String(WEB_NETWORK_CAPTURE_LIMITS.outputBytes)}-byte reply budget, and a ${String(WEB_NETWORK_CAPTURE_LIMITS.depth)}-level nesting cap. Those have no runtime override. capture_timeout_ms is ${String(selectedWebNetworkCaptureTimeoutMs(input.capture_timeout_ms))} ms (omitted means ${String(WEB_NETWORK_CAPTURE_LIMITS.timeoutMs)}). ${nodeMemoryLimitation()}; mitmproxy decoding uses a 768 MiB Linux address-space limit.`,
+ `Complete evidence is bounded by a ${String(WEB_NETWORK_CAPTURE_LIMITS.inputBytes)}-byte input budget, a ${String(WEB_NETWORK_CAPTURE_LIMITS.outputBytes)}-byte reply budget, and a ${String(WEB_NETWORK_CAPTURE_LIMITS.depth)}-level nesting cap. Those have no runtime override. capture_timeout_ms is ${String(selectedWebNetworkCaptureTimeoutMs(input.capture_timeout_ms))} ms (omitted means ${String(WEB_NETWORK_CAPTURE_LIMITS.timeoutMs)}). ${nodeMemoryLimitation(reportedHeap)} Mitmproxy decoding uses a 768 MiB Linux address-space limit.`,
],
};
if (input.format === "qlog") {
@@ -348,13 +378,25 @@ export class HistoricalCaptureDecoder {
message: cause.message,
},
])
- : historicalCaptureFailure(input, cause, phase, options);
+ : historicalCaptureFailure(
+ input,
+ cause,
+ phase,
+ options,
+ reportedHeap,
+ );
if (safePartialSource) error.retainPartialObservation(result.value);
throw error;
}
}
} catch (cause: unknown) {
- const error = historicalCaptureFailure(input, cause, phase, options);
+ const error = historicalCaptureFailure(
+ input,
+ cause,
+ phase,
+ options,
+ reportedHeap,
+ );
result = err(
redactExplicitFailure(
withCaptureReaderCleanup(error, cause),
@@ -418,13 +460,36 @@ export class HistoricalCaptureDecoder {
}
}
+/** One shared parsed heap, when every Node adapter agrees. Disagreement stays unresolved here. */
+const sharedCaptureDecoderHeap = (
+ adapters: readonly HistoricalCaptureFormatAdapter[],
+): CaptureDecoderHeap | undefined => {
+ const heaps = adapters.flatMap((adapter) =>
+ adapter.heap === undefined ? [] : [adapter.heap],
+ );
+ const first = heaps[0];
+ if (first === undefined) return undefined;
+ return heaps.every(
+ (heap) =>
+ heap === first ||
+ (heap.oldGenerationMiB === first.oldGenerationMiB &&
+ heap.source === first.source),
+ )
+ ? first
+ : undefined;
+};
+
/** The Node capture decoder runs on this executable; only Node.js enforces its V8 heap flags. */
-const nodeMemoryLimitation = (): string => {
+const nodeMemoryLimitation = (heap: CaptureDecoderHeap): string => {
+ const recovery = captureDecoderHeapRecovery(heap);
if (NODE_CAPTURE_HEAP_ENFORCED)
- return `HAR/SAZ/qlog use a ${String(NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB)} MiB Node old-generation heap`;
- return NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null
- ? "HAR/SAZ/qlog run on Bun, which does not enforce REA's V8 heap flags"
- : `HAR/SAZ/qlog run on Bun, which does not enforce REA's V8 heap flags, under a ${String(NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES)}-byte Windows Job Object commit limit`;
+ return `HAR/SAZ/qlog use a ${String(heap.oldGenerationMiB)} MiB Node old-generation heap (source ${heap.source}). ${recovery}`;
+ const commitLimit = captureDecoderProcessMemoryLimitBytes(
+ heap.oldGenerationMiB,
+ );
+ return commitLimit === null
+ ? `HAR/SAZ/qlog run on Bun, which does not enforce REA's V8 heap flags. ${recovery}`
+ : `HAR/SAZ/qlog run on Bun, which does not enforce REA's V8 heap flags, under a ${String(commitLimit)}-byte Windows Job Object commit limit. ${recovery}`;
};
const replyFailureReason = (cause: unknown): string => {
diff --git a/src/browser/history/HistoricalCaptureFormatAdapter.ts b/src/browser/history/HistoricalCaptureFormatAdapter.ts
index 9660fd579..88e811b73 100644
--- a/src/browser/history/HistoricalCaptureFormatAdapter.ts
+++ b/src/browser/history/HistoricalCaptureFormatAdapter.ts
@@ -1,9 +1,15 @@
+import type { CaptureDecoderHeap } from "../../config/captureDecoderHeap.js";
import type { InspectWebNetworkCaptureInput } from "../../domain/webNetworkCapture.js";
/** Format-specific process profile keeps mature upstream code behind a replaceable adapter seam. */
export interface HistoricalCaptureFormatAdapter {
readonly format: InspectWebNetworkCaptureInput["format"];
readonly limitations?: readonly string[];
+ /**
+ * Parsed heap this adapter launches. Reports reuse this fact. Adapters that
+ * do not launch the Node decoder omit it.
+ */
+ readonly heap?: CaptureDecoderHeap;
readonly identity: {
readonly id: string;
readonly name: string;
diff --git a/src/browser/history/NodeCaptureAdapter.test.ts b/src/browser/history/NodeCaptureAdapter.test.ts
index bf2528f5b..3e47e0973 100644
--- a/src/browser/history/NodeCaptureAdapter.test.ts
+++ b/src/browser/history/NodeCaptureAdapter.test.ts
@@ -5,7 +5,7 @@ import { NodeCaptureAdapter } from "./NodeCaptureAdapter.js";
vi.mock("./CaptureRelease.js", async (original) => ({
...(await original<typeof import("./CaptureRelease.js")>()),
NODE_CAPTURE_HEAP_ENFORCED: false,
- NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES: null,
+ captureDecoderProcessMemoryLimitBytes: () => null,
}));
it.each(["saz", "qlog"] as const)(
diff --git a/src/browser/history/NodeCaptureAdapter.ts b/src/browser/history/NodeCaptureAdapter.ts
index a366459a4..249572c26 100644
--- a/src/browser/history/NodeCaptureAdapter.ts
+++ b/src/browser/history/NodeCaptureAdapter.ts
@@ -1,12 +1,17 @@
import { fileURLToPath } from "node:url";
import type { HistoricalCaptureFormatAdapter } from "./HistoricalCaptureFormatAdapter.js";
+import {
+ CAPTURE_DECODER_SEMI_SPACE_MIB,
+ DEFAULT_CAPTURE_DECODER_HEAP,
+ resolveCaptureDecoderHeap,
+ type CaptureDecoderHeap,
+} from "../../config/captureDecoderHeap.js";
import {
NODE_CAPTURE_HEAP_ENFORCED,
HAR_CAPTURE_PROVIDER_IDENTITY,
SAZ_CAPTURE_PROVIDER_IDENTITY,
QLOG_CAPTURE_PROVIDER_IDENTITY,
- NODE_CAPTURE_HEAP_LIMITS,
- NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES,
+ captureDecoderProcessMemoryLimitBytes,
} from "./CaptureRelease.js";
import { AnalysisCapabilityUnavailableError } from "../../domain/analysisErrorCore.js";
@@ -14,7 +19,22 @@ import { AnalysisCapabilityUnavailableError } from "../../domain/analysisErrorCo
export class NodeCaptureAdapter implements HistoricalCaptureFormatAdapter {
readonly identity;
readonly limitations;
- constructor(readonly format: "har" | "saz" | "qlog") {
+ readonly heap: CaptureDecoderHeap;
+ /**
+ * `environment` is the selected startup environment. Omitted means the
+ * default heap, not ambient `process.env`. An invalid setting is refused.
+ * `heap` is an already parsed fact and is not read from `environment` again.
+ */
+ constructor(
+ readonly format: "har" | "saz" | "qlog",
+ environment?: Readonly<Record<string, string | undefined>>,
+ heap?: CaptureDecoderHeap,
+ ) {
+ this.heap =
+ heap ??
+ (environment === undefined
+ ? DEFAULT_CAPTURE_DECODER_HEAP
+ : resolveCaptureDecoderHeap(environment));
this.identity =
format === "har"
? HAR_CAPTURE_PROVIDER_IDENTITY
@@ -32,10 +52,13 @@ export class NodeCaptureAdapter implements HistoricalCaptureFormatAdapter {
: [];
}
async command(requestPath: string) {
+ const memoryLimitBytes = captureDecoderProcessMemoryLimitBytes(
+ this.heap.oldGenerationMiB,
+ );
if (
this.format !== "har" &&
!NODE_CAPTURE_HEAP_ENFORCED &&
- NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null
+ memoryLimitBytes === null
)
throw new AnalysisCapabilityUnavailableError(
this.identity.id,
@@ -45,17 +68,15 @@ export class NodeCaptureAdapter implements HistoricalCaptureFormatAdapter {
return {
command: process.execPath,
arguments: [
- `--max-old-space-size=${NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB}`,
- `--max-semi-space-size=${NODE_CAPTURE_HEAP_LIMITS.semiSpaceMiB}`,
+ `--max-old-space-size=${String(this.heap.oldGenerationMiB)}`,
+ `--max-semi-space-size=${String(CAPTURE_DECODER_SEMI_SPACE_MIB)}`,
"--v8-pool-size=1",
"--single-threaded",
fileURLToPath(new URL("./NodeCaptureProcess.js", import.meta.url)),
requestPath,
this.format,
],
- ...(NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null
- ? {}
- : { memoryLimitBytes: NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES }),
+ ...(memoryLimitBytes === null ? {} : { memoryLimitBytes }),
};
}
}
diff --git a/src/cli/webNetworkCaptureCommands.ts b/src/cli/webNetworkCaptureCommands.ts
index 5a3d82b96..659d9c023 100644
--- a/src/cli/webNetworkCaptureCommands.ts
+++ b/src/cli/webNetworkCaptureCommands.ts
@@ -23,7 +23,6 @@ export const registerWebNetworkCaptureCommands = (
logger: Logger,
environment: Readonly<NodeJS.ProcessEnv>,
): void => {
- const service = createWebNetworkCaptureService(environment);
cli.command(CLI_COMMANDS.inspectWebNetworkCapture, {
description:
"Inspect retained HAR, native mitmproxy, Fiddler SAZ or aioquic qlog records without fetching recorded URLs",
@@ -65,6 +64,7 @@ export const registerWebNetworkCaptureCommands = (
logger,
CLI_COMMANDS.inspectWebNetworkCapture,
async () => {
+ const service = createWebNetworkCaptureService(environment);
return deliverCliAnalysisEvidence({
run: () =>
service.inspect(
diff --git a/src/composition/webNetworkCaptures.ts b/src/composition/webNetworkCaptures.ts
index 9d9e8c6cc..1653f644a 100644
--- a/src/composition/webNetworkCaptures.ts
+++ b/src/composition/webNetworkCaptures.ts
@@ -2,19 +2,22 @@ import { WebNetworkCaptureService } from "../application/WebNetworkCaptureServic
import { HistoricalCaptureDecoder } from "../browser/history/HistoricalCaptureDecoder.js";
import { NodeCaptureAdapter } from "../browser/history/NodeCaptureAdapter.js";
import { MitmproxyCaptureAdapter } from "../browser/history/MitmproxyCaptureAdapter.js";
+import { resolveCaptureDecoderHeap } from "../config/captureDecoderHeap.js";
/** Compose offline format adapters without import-time reads, installation or process startup. */
export const createWebNetworkCaptureService = (
environment: Readonly<NodeJS.ProcessEnv> = process.env,
-): WebNetworkCaptureService =>
- new WebNetworkCaptureService(
+): WebNetworkCaptureService => {
+ const heap = resolveCaptureDecoderHeap(environment);
+ return new WebNetworkCaptureService(
new HistoricalCaptureDecoder(
[
- new NodeCaptureAdapter("har"),
- new NodeCaptureAdapter("saz"),
- new NodeCaptureAdapter("qlog"),
+ new NodeCaptureAdapter("har", environment, heap),
+ new NodeCaptureAdapter("saz", environment, heap),
+ new NodeCaptureAdapter("qlog", environment, heap),
new MitmproxyCaptureAdapter(environment),
],
environment,
),
);
+};
diff --git a/src/config/captureDecoderHeap.test.ts b/src/config/captureDecoderHeap.test.ts
new file mode 100644
index 000000000..e6bb48f5e
--- /dev/null
+++ b/src/config/captureDecoderHeap.test.ts
@@ -0,0 +1,260 @@
+import { mkdtemp, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { expect, it, vi } from "vitest";
+
+import { NodeCaptureAdapter } from "../browser/history/NodeCaptureAdapter.js";
+import { HistoricalCaptureDecoder } from "../browser/history/HistoricalCaptureDecoder.js";
+import { decodeHarCapture } from "../browser/history/HarCapture.js";
+import { historicalCaptureFailure } from "../browser/history/CaptureFailures.js";
+import { ConfigurationError } from "../domain/configurationErrors.js";
+import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
+import { inspectWebNetworkCaptureInputSchema } from "../domain/webNetworkCapture.js";
+import { OwnedCommandFailure } from "../process/OwnedCommand.js";
+import { createWebNetworkCaptureService } from "../composition/webNetworkCaptures.js";
+import { createCli } from "../cli.js";
+import { historicalHar } from "../../tests/fixtures/historicalHar.js";
+import { parseEnvironment } from "./environment.js";
+import { parseConfig } from "./parseConfig.js";
+import {
+ CAPTURE_DECODER_OLD_GENERATION_SETTING,
+ captureDecoderHeapConstraint,
+ captureDecoderHeapRecovery,
+ DEFAULT_CAPTURE_DECODER_HEAP,
+ MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB,
+ parseCaptureDecoderHeap,
+} from "./captureDecoderHeap.js";
+
+const configured = (oldGenerationMiB: number) =>
+ ({ oldGenerationMiB, source: "setting" }) as const;
+
+const heapExhaustion = () => {
+ const stderr =
+ "FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory\n";
+ return new OwnedCommandFailure("process", "Decoder failed", {
+ stdout: { text: "", bytes: 0, observedBytes: 0 },
+ stderr: {
+ text: stderr,
+ bytes: Buffer.byteLength(stderr),
+ observedBytes: Buffer.byteLength(stderr),
+ },
+ exitCode: null,
+ signal: "SIGABRT",
+ });
+};
+
+it("keeps the 192 MiB decoder heap when the setting is unset", async () => {
+ expect(parseCaptureDecoderHeap(undefined)).toEqual({
+ ok: true,
+ value: DEFAULT_CAPTURE_DECODER_HEAP,
+ });
+ const parsed = parseConfig({});
+ if (!parsed.ok) throw new Error("unset heap was rejected");
+ const recovery = captureDecoderHeapRecovery(DEFAULT_CAPTURE_DECODER_HEAP);
+ expect(recovery).toContain(
+ "The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 192 MiB (source default; default 192 MiB, floor 8 MiB, maximum 16384 MiB).",
+ );
+ expect(recovery).toContain(
+ "Set REA_CAPTURE_DECODER_OLD_GENERATION_MIB in the environment that starts REA to a decimal integer from 8 to 16384, then restart REA.",
+ );
+ expect(recovery).not.toContain("no runtime override");
+ const command = await new NodeCaptureAdapter("har").command(
+ "/owned/request.json",
+ );
+ expect(command.arguments).toContain("--max-old-space-size=192");
+ expect(command.arguments).toContain("--max-semi-space-size=8");
+});
+
+it("uses a valid REA_CAPTURE_DECODER_OLD_GENERATION_MIB override", async () => {
+ expect(parseCaptureDecoderHeap("256")).toEqual({
+ ok: true,
+ value: configured(256),
+ });
+ expect(
+ parseCaptureDecoderHeap(String(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)),
+ ).toEqual({
+ ok: true,
+ value: configured(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB),
+ });
+ const environment = { REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "256" };
+ const parsed = parseConfig(environment);
+ if (!parsed.ok) throw new Error("valid heap was rejected");
+ const adapter = new NodeCaptureAdapter("har", environment);
+ expect(adapter.heap).toEqual(configured(256));
+ const command = await adapter.command("/owned/request.json");
+ expect(command.arguments).toContain("--max-old-space-size=256");
+ expect(command.arguments).toContain("--max-semi-space-size=8");
+ const input = inspectWebNetworkCaptureInputSchema.parse({
+ capture_path: "/selected/input.har",
+ format: "har",
+ });
+ const projected = projectAnalysisError(
+ historicalCaptureFailure(
+ input,
+ heapExhaustion(),
+ "decoder",
+ undefined,
+ configured(256),
+ ),
+ );
+ const recovery = captureDecoderHeapRecovery(configured(256));
+ expect(projected.message).toContain(
+ "exhausted its 256 MiB V8 old-generation heap (source setting)",
+ );
+ expect(projected.remediation.action).toContain(recovery);
+ expect(projected.message).not.toContain("no runtime override");
+ if (process.versions.bun === undefined)
+ expect(projected.details).toMatchObject({
+ reported_limits: {
+ old_generation_heap_mib: 256,
+ old_generation_heap_source: "setting",
+ },
+ });
+});
+
+it("keeps a parsed heap instead of reading the adapter environment again", async () => {
+ const heap = configured(256);
+ const adapter = new NodeCaptureAdapter(
+ "har",
+ { REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "512" },
+ heap,
+ );
+ expect(adapter.heap).toBe(heap);
+ const command = await adapter.command("/owned/request.json");
+ expect(command.arguments).toContain("--max-old-space-size=256");
+ expect(command.arguments).not.toContain("--max-old-space-size=512");
+});
+
+it("reports the heap the selected adapter launches when decoder environment differs", async () => {
+ const launched = new NodeCaptureAdapter("har", {
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "256",
+ });
+ const other = new NodeCaptureAdapter("saz", {
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "512",
+ });
+ const root = await mkdtemp(join(tmpdir(), "rea-capture-heap-report-"));
+ const capturePath = join(root, "input.har");
+ const text = JSON.stringify(historicalHar());
+ await writeFile(capturePath, text);
+ const input = inspectWebNetworkCaptureInputSchema.parse({
+ capture_path: capturePath,
+ format: "har",
+ });
+ const decoder = new HistoricalCaptureDecoder([launched, other], {});
+ try {
+ vi.spyOn(launched, "command").mockImplementation(async (requestPath) => {
+ const runtimePath = join(requestPath, "..");
+ await writeFile(
+ join(runtimePath, "reply.json"),
+ JSON.stringify({ ok: true, value: decodeHarCapture(text, []) }),
+ );
+ return {
+ command: process.execPath,
+ arguments: ["-e", "process.exit(0)"],
+ };
+ });
+ const reported = await decoder.inspect(input);
+ if (!reported.ok)
+ throw new Error(projectAnalysisError(reported.error).message);
+ const limitations = reported.value.limitations.join("\n");
+ expect(limitations).toContain(
+ "The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 256 MiB (source setting",
+ );
+ expect(limitations).not.toContain("192 MiB (source default)");
+ expect(limitations).not.toContain("512 MiB (source setting)");
+
+ vi.spyOn(launched, "command").mockRejectedValue(heapExhaustion());
+ const exhausted = await decoder.inspect(input);
+ if (exhausted.ok) throw new Error("Heap exhaustion must fail inspection");
+ const projected = projectAnalysisError(exhausted.error);
+ expect(projected.message).toContain(
+ "exhausted its 256 MiB V8 old-generation heap (source setting)",
+ );
+ expect(projected.remediation.action).toContain(
+ "The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 256 MiB (source setting",
+ );
+ expect(projected.message).not.toContain("192 MiB");
+ expect(projected.remediation.action).not.toContain(
+ "512 MiB (source setting)",
+ );
+ } finally {
+ await decoder.close();
+ await rm(root, { recursive: true, force: true });
+ }
+});
+
+it("registers CLI commands when the capture decoder heap setting is invalid", async () => {
+ const cli = createCli({
+ REA_LOG_LEVEL: "silent",
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "7",
+ });
+ let text = "";
+ await cli.serve(["inspect-web-network-capture", "--help"], {
+ env: {},
+ exit: () => undefined,
+ stdout: (chunk: string) => {
+ text += chunk;
+ },
+ });
+ expect(text).toContain("inspect-web-network-capture");
+});
+
+it.each([
+ "",
+ "0",
+ "7",
+ "-1",
+ "0192",
+ "192.5",
+ " 192",
+ "192 ",
+ "1e3",
+ String(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB + 1),
+ "9".repeat(40),
+])("rejects invalid %j as a configuration error", (value) => {
+ const parsed = parseCaptureDecoderHeap(value);
+ if (parsed.ok) throw new Error(`Accepted ${JSON.stringify(value)}`);
+ expect(parsed.error).toBeInstanceOf(ConfigurationError);
+ expect(parsed.error.settings).toEqual([
+ {
+ setting: CAPTURE_DECODER_OLD_GENERATION_SETTING,
+ constraint: captureDecoderHeapConstraint,
+ },
+ ]);
+ expect(parsed.error.message).toBe(captureDecoderHeapConstraint);
+ const startup = parseEnvironment({
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
+ });
+ if (startup.ok) throw new Error(`Startup accepted ${JSON.stringify(value)}`);
+ expect(startup.error.settings).toContainEqual({
+ setting: "REA_CAPTURE_DECODER_OLD_GENERATION_MIB",
+ constraint: expect.stringContaining("from 8 to 16384 MiB"),
+ });
+ const config = parseConfig({
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
+ });
+ if (config.ok)
+ throw new Error(`parseConfig accepted ${JSON.stringify(value)}`);
+ expect(projectAnalysisError(config.error)).toMatchObject({
+ code: "configuration_invalid",
+ message: expect.stringContaining("REA_CAPTURE_DECODER_OLD_GENERATION_MIB"),
+ });
+ expect(
+ () =>
+ new NodeCaptureAdapter("har", {
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
+ }),
+ ).toThrow(ConfigurationError);
+ expect(
+ () =>
+ new HistoricalCaptureDecoder([], {
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
+ }),
+ ).toThrow(ConfigurationError);
+ expect(() =>
+ createWebNetworkCaptureService({
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
+ }),
+ ).toThrow(ConfigurationError);
+});
diff --git a/src/config/captureDecoderHeap.ts b/src/config/captureDecoderHeap.ts
new file mode 100644
index 000000000..aed3b0619
--- /dev/null
+++ b/src/config/captureDecoderHeap.ts
@@ -0,0 +1,105 @@
+import { z } from "zod";
+
+import { ConfigurationError } from "../domain/configurationErrors.js";
+import { err, ok, type Result } from "../domain/result.js";
+
+export const CAPTURE_DECODER_OLD_GENERATION_SETTING =
+ "REA_CAPTURE_DECODER_OLD_GENERATION_MIB";
+
+/** Old-generation mebibytes used when the setting is unset. Today's constant. */
+export const DEFAULT_CAPTURE_DECODER_OLD_GENERATION_MIB = 192;
+
+/**
+ * Semi-space the owned decoder always passes. It is not a setting. An old
+ * generation smaller than this semi-space aborts Node during isolate startup
+ * instead of enforcing a heap limit, so the setting's floor is this value.
+ */
+export const CAPTURE_DECODER_SEMI_SPACE_MIB = 8;
+
+/** Smallest old generation that still starts the decoder with that semi-space. */
+export const MINIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB =
+ CAPTURE_DECODER_SEMI_SPACE_MIB;
+
+/**
+ * Largest old generation one owned decoder may request. 16 GiB is the same
+ * ceiling REA already uses for an isolated JavaScript analysis worker. Node
+ * will start with a larger flag; this maximum is REA's refusal, so one
+ * decoder cannot ask for an unbounded old generation.
+ */
+export const MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB = 16_384;
+
+/** Effective HAR/SAZ/qlog old-generation heap and where its value came from. */
+export interface CaptureDecoderHeap {
+ readonly oldGenerationMiB: number;
+ readonly source: "default" | "setting";
+}
+
+export const DEFAULT_CAPTURE_DECODER_HEAP: CaptureDecoderHeap = {
+ oldGenerationMiB: DEFAULT_CAPTURE_DECODER_OLD_GENERATION_MIB,
+ source: "default",
+};
+
+export const captureDecoderHeapConstraint = `${CAPTURE_DECODER_OLD_GENERATION_SETTING} must be a decimal integer from ${String(MINIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)} to ${String(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)} MiB; the default is ${String(DEFAULT_CAPTURE_DECODER_OLD_GENERATION_MIB)} MiB. An old generation below the decoder's ${String(CAPTURE_DECODER_SEMI_SPACE_MIB)} MiB semi-space aborts Node during startup instead of enforcing a limit, and a larger value is refused.`;
+
+const decimalInteger = /^[1-9]\d*$/u;
+
+const withinHeapRange = (value: string): boolean => {
+ // The pattern check reports a non-decimal value. Refine only bounds integers.
+ if (!decimalInteger.test(value)) return true;
+ const parsed = BigInt(value);
+ return (
+ parsed >= BigInt(MINIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB) &&
+ parsed <= BigInt(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)
+ );
+};
+
+/** Optional decimal mebibyte count for the HAR/SAZ/qlog old-generation heap. */
+export const captureDecoderOldGenerationMiBSchema = z
+ .string()
+ .regex(decimalInteger, { error: captureDecoderHeapConstraint })
+ .refine(withinHeapRange, { error: captureDecoderHeapConstraint })
+ .transform(Number);
+
+/** Resolve the effective heap from one environment value. */
+export const parseCaptureDecoderHeap = (
+ value: string | undefined,
+): Result<CaptureDecoderHeap, ConfigurationError> => {
+ if (value === undefined) return ok(DEFAULT_CAPTURE_DECODER_HEAP);
+ const parsed = captureDecoderOldGenerationMiBSchema.safeParse(value);
+ return parsed.success
+ ? ok({ oldGenerationMiB: parsed.data, source: "setting" })
+ : err(
+ new ConfigurationError(captureDecoderHeapConstraint, {
+ settings: [
+ {
+ setting: CAPTURE_DECODER_OLD_GENERATION_SETTING,
+ constraint: captureDecoderHeapConstraint,
+ },
+ ],
+ }),
+ );
+};
+
+/**
+ * Resolve the heap from a composition root's selected environment. An invalid
+ * setting is refused, not ignored. Callers that already resolved it at startup
+ * pass that value through instead of reading the environment again.
+ */
+export const resolveCaptureDecoderHeap = (
+ environment: Readonly<Record<string, string | undefined>>,
+): CaptureDecoderHeap => {
+ const parsed = parseCaptureDecoderHeap(
+ environment[CAPTURE_DECODER_OLD_GENERATION_SETTING],
+ );
+ if (!parsed.ok) throw parsed.error;
+ return parsed.value;
+};
+
+/** How to read and change the effective HAR/SAZ/qlog old-generation heap. */
+export const captureDecoderHeapRecovery = (heap: CaptureDecoderHeap): string =>
+ [
+ `The effective ${CAPTURE_DECODER_OLD_GENERATION_SETTING} value is ${String(heap.oldGenerationMiB)} MiB (source ${heap.source}; default ${String(DEFAULT_CAPTURE_DECODER_OLD_GENERATION_MIB)} MiB, floor ${String(MINIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)} MiB, maximum ${String(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)} MiB).`,
+ `Set ${CAPTURE_DECODER_OLD_GENERATION_SETTING} in the environment that starts REA to a decimal integer from ${String(MINIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)} to ${String(MAXIMUM_CAPTURE_DECODER_OLD_GENERATION_MIB)}, then restart REA.`,
+ `On Node.js the decoder passes --max-old-space-size=${String(heap.oldGenerationMiB)} and removes inherited NODE_OPTIONS, so additional host memory and NODE_OPTIONS do not raise this heap. The ${String(CAPTURE_DECODER_SEMI_SPACE_MIB)} MiB semi-space stays fixed.`,
+ "On Bun the V8 flags are ignored. Windows limits the decoder's committed memory to this value plus a 512 MiB reserve. Other Bun hosts enforce no decoder heap limit.",
+ ].join(" ");
diff --git a/src/config/environment.ts b/src/config/environment.ts
index d90715c8d..09ca39fb5 100644
--- a/src/config/environment.ts
+++ b/src/config/environment.ts
@@ -19,6 +19,7 @@ import { mcpInputSchemaProfileSchema } from "./mcpInputSchemaProfile.js";
import { mcpOutputSchemaProfileSchema } from "./mcpOutputSchemaProfile.js";
import { mcpResponseBudgetSchema } from "./mcpResponseBudget.js";
import { asarMaxHeaderBytesSchema } from "./asarHeaderAdmission.js";
+import { captureDecoderOldGenerationMiBSchema } from "./captureDecoderHeap.js";
import { regexMatchingDeadlineMsSchema } from "./regexMatchingDeadline.js";
import { regexWorkerHeapRequestMiBSchema } from "./regexWorkerHeapRequest.js";
import { parseLoaderArgs, parseStringArray } from "./parsers.js";
@@ -27,6 +28,8 @@ const environmentSchema = z.object({
REA_ANALYSIS_PROVIDER: analysisProviderSelectorSchema.default("auto"),
REA_MCP_MAX_RESPONSE_BYTES: mcpResponseBudgetSchema.optional(),
REA_ASAR_MAX_HEADER_BYTES: asarMaxHeaderBytesSchema.optional(),
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB:
+ captureDecoderOldGenerationMiBSchema.optional(),
REA_REGEX_MATCHING_DEADLINE_MS: regexMatchingDeadlineMsSchema.optional(),
REA_REGEX_WORKER_HEAP_REQUEST_MIB: regexWorkerHeapRequestMiBSchema.optional(),
REA_MCP_INPUT_SCHEMA_PROFILE: mcpInputSchemaProfileSchema.optional(),
diff --git a/src/setupEnvironment.test.ts b/src/setupEnvironment.test.ts
index d24b4e574..015e1badb 100644
--- a/src/setupEnvironment.test.ts
+++ b/src/setupEnvironment.test.ts
@@ -185,6 +185,7 @@ it.each([
REA_GHIDRA_STARTUP_TIMEOUT_MS: "2700000",
REA_HOPPER_STARTUP_TIMEOUT_MS: "120000",
REA_MCP_INPUT_SCHEMA_PROFILE: "full",
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "256",
REA_REGEX_MATCHING_DEADLINE_MS: "10000",
REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256",
},
@@ -219,6 +220,7 @@ it.each([
REA_GHIDRA_STARTUP_TIMEOUT_MS: "2700000",
REA_HOPPER_STARTUP_TIMEOUT_MS: "120000",
REA_MCP_INPUT_SCHEMA_PROFILE: "full",
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "256",
REA_REGEX_MATCHING_DEADLINE_MS: "10000",
REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256",
});
diff --git a/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts b/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts
index cfe958909..142426834 100644
--- a/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts
+++ b/tests/acceptance/setup/registrationRuntimeEnvironmentCli.test.ts
@@ -486,6 +486,7 @@ cliTest(
REA_HOPPER_STARTUP_TIMEOUT_MS: "2800002",
REA_JEB_MCP_TIMEOUT_MS: "2900003",
REA_MCP_INPUT_SCHEMA_PROFILE: "full",
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "256",
REA_REGEX_MATCHING_DEADLINE_MS: "10000",
REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256",
REA_UNRELATED_MARKER: "do-not-persist",
@@ -516,6 +517,9 @@ cliTest(
);
expect(codexRegistration).toMatchObject({ tool_timeout_sec: 3021 });
expect(codexConfig).toContain('REA_JEB_MCP_TIMEOUT_MS = "2900003"');
+ expect(codexConfig).toContain(
+ 'REA_CAPTURE_DECODER_OLD_GENERATION_MIB = "256"',
+ );
expect(codexConfig).toContain('REA_REGEX_MATCHING_DEADLINE_MS = "10000"');
expect(codexConfig).toContain('REA_REGEX_WORKER_HEAP_REQUEST_MIB = "256"');
expect(codexConfig).not.toContain("REA_UNRELATED_MARKER");
@@ -533,6 +537,7 @@ cliTest(
REA_HOPPER_STARTUP_TIMEOUT_MS: "2800002",
REA_JEB_MCP_TIMEOUT_MS: "2900003",
REA_MCP_INPUT_SCHEMA_PROFILE: "full",
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "256",
REA_REGEX_MATCHING_DEADLINE_MS: "10000",
REA_REGEX_WORKER_HEAP_REQUEST_MIB: "256",
},
diff --git a/tests/acceptance/tools/historicalCaptureResources.test.ts b/tests/acceptance/tools/historicalCaptureResources.test.ts
index 64a7b6638..0073fc606 100644
--- a/tests/acceptance/tools/historicalCaptureResources.test.ts
+++ b/tests/acceptance/tools/historicalCaptureResources.test.ts
@@ -64,13 +64,14 @@ it.skipIf(process.versions.bun !== undefined && jobLimit === null)(
sensitive_values: ["caller-marked"],
});
expect(response.isError).toBe(true);
- expect(parseMcpToolError(response)).toMatchObject({
+ const exhausted = parseMcpToolError(response);
+ expect(exhausted).toMatchObject({
error: {
code: "resource_constraint",
retryable: false,
message: expect.stringContaining(
jobLimit === null
- ? "fixed V8 heap"
+ ? "192 MiB V8 old-generation heap (source default)"
: "Windows Job Object memory limit",
),
remediation: {
@@ -101,6 +102,14 @@ it.skipIf(process.versions.bun !== undefined && jobLimit === null)(
});
expect(JSON.stringify(response)).not.toContain("caller-marked");
expect(JSON.stringify(response)).not.toContain("rea doctor");
+ const remediation = String(exhausted.error.remediation?.action ?? "");
+ expect(remediation).toContain(
+ "The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 192 MiB (source default; default 192 MiB, floor 8 MiB, maximum 16384 MiB).",
+ );
+ expect(remediation).toContain(
+ "Set REA_CAPTURE_DECODER_OLD_GENERATION_MIB in the environment that starts REA",
+ );
+ expect(remediation).not.toContain("no runtime override");
const failed: unknown = await execute(process.execPath, [
cli,
"inspect-web-network-capture",
@@ -117,6 +126,9 @@ it.skipIf(process.versions.bun !== undefined && jobLimit === null)(
? String(failed.stdout)
: "";
expect(JSON.parse(stdout)).toMatchObject({ code: "resource_constraint" });
+ expect(stdout).toContain("REA_CAPTURE_DECODER_OLD_GENERATION_MIB");
+ expect(stdout).toContain("source default");
+ expect(stdout).not.toContain("no runtime override");
const small = join(root, "small.har");
await writeFile(small, capture(2));
await client.ping();
@@ -128,6 +140,16 @@ it.skipIf(process.versions.bun !== undefined && jobLimit === null)(
expect(control.structuredContent).toMatchObject({
normalized_result: { total_records: 2 },
});
+ const controlText = JSON.stringify(control.structuredContent);
+ expect(controlText).toContain(
+ "The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 192 MiB (source default; default 192 MiB, floor 8 MiB, maximum 16384 MiB).",
+ );
+ expect(controlText).toContain("source default");
+ expect(controlText).toContain("floor 8 MiB");
+ expect(controlText).toContain("maximum 16384 MiB");
+ expect(controlText).toContain(
+ "Set REA_CAPTURE_DECODER_OLD_GENERATION_MIB in the environment that starts REA",
+ );
await truncate(path, WEB_NETWORK_CAPTURE_LIMITS.inputBytes + 1);
const oversized = await call("inspect_web_network_capture", {
capture_path: path,
diff --git a/tests/boundary/cli/captureDecoderHeapDoctor.test.ts b/tests/boundary/cli/captureDecoderHeapDoctor.test.ts
new file mode 100644
index 000000000..94c4ebb42
--- /dev/null
+++ b/tests/boundary/cli/captureDecoderHeapDoctor.test.ts
@@ -0,0 +1,28 @@
+import { expect } from "vitest";
+
+import { cliTest } from "../../support/cli/cliFixture.js";
+
+cliTest(
+ "reports an invalid capture decoder heap from doctor without failing registration",
+ async ({ cli }) => {
+ const doctor = await cli.run({
+ arguments: ["doctor", "--json"],
+ environment: {
+ REA_LOG_LEVEL: "silent",
+ REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "7",
+ },
+ });
+ expect(doctor.exitCode, doctor.stderr).toBe(1);
+ expect(doctor.stderr).toBe("");
+ expect(doctor.json).toMatchObject({
+ healthy: false,
+ scope_checks: expect.arrayContaining([
+ expect.objectContaining({
+ name: "runtime-configuration:REA_CAPTURE_DECODER_OLD_GENERATION_MIB",
+ ok: false,
+ classification: "config_drift",
+ }),
+ ]),
+ });
+ },
+);
diff --git a/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts b/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts
index 53aca18e9..c13d2615f 100644
--- a/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts
+++ b/tests/boundary/filesystem/clientConfigurationEnvironment.test.ts
@@ -496,6 +496,11 @@ it.each([
{ REA_REGEX_MATCHING_DEADLINE_MS: "0" },
"REA_REGEX_MATCHING_DEADLINE_MS",
],
+ [
+ "capture decoder heap below its floor",
+ { REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "7" },
+ "REA_CAPTURE_DECODER_OLD_GENERATION_MIB",
+ ],
[
"regex worker heap request below its floor",
{ REA_REGEX_WORKER_HEAP_REQUEST_MIB: "6" },