Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(deploy): give codex-llm-proxy 768Mi and a 384 MiB heap cap (#739)

Alfredo Lopezf4c03552beaf
diff --git a/codex-llm-proxy/src/requestLimits.ts b/codex-llm-proxy/src/requestLimits.ts
index 667c1104c..b16221deb 100644
--- a/codex-llm-proxy/src/requestLimits.ts
+++ b/codex-llm-proxy/src/requestLimits.ts
@@ -15,13 +15,16 @@ export const DEFAULT_MAX_BODY_BYTES = CONTRACT_LIMITS.maxRequestBodyBytes + ENVE
/**
* #731 R3-2 — how many maximum-size bodies may be read and parsed at once.
- * About five copies of a body are alive while it is parsed and hashed (the raw
- * buffer, the decoded string, the parsed object, the contract copy and the
- * canonical serialization), so three 8 MiB bodies hold about 120 MiB of the
- * pod's 256Mi. Without this bound the stream gate would let 24 bodies in.
+ * Several copies of a body are alive while it is parsed, hashed and forwarded
+ * (the raw buffer, the decoded string, the parsed object, the contract copy and
+ * more than one serialization of it). Without this bound the stream gate would
+ * let 24 bodies in.
* Bodies over the ordinary cap never take this budget: they are V2 visual
* envelopes, bounded by `visualStreamGate` instead, so the declared bodies a pod
- * can hold at once are 3 x 8 MiB here plus 2 x 24 MiB there.
+ * can hold at once are 3 x 8 MiB here plus 2 x 24 MiB there. Measured with all
+ * five in flight, the process peaked at 714 MiB of RSS with an uncapped heap
+ * and at 574 MiB with `--max-old-space-size=384`, which is why the deployment
+ * sets that cap and a 768Mi memory limit.
*/
export const IN_FLIGHT_BODY_BUDGET_BODIES = 3
@@ -36,10 +39,11 @@ export const STREAM_LIMITS = {
/**
* Admission for a body whose Content-Length exceeds the ordinary cap.
- * The 256Mi pod cannot hold the ordinary 8-stream gate across a 24 MiB image,
- * so those requests are a tighter sibling and a V2 request keeps the slot
- * until the stream ends. Small bodies, including every valid V1, must not
- * enter this gate. Do not raise proxy memory to widen it.
+ * The pod cannot hold the ordinary 8-stream gate across a 24 MiB image, so
+ * those requests are a tighter sibling and a V2 request keeps the slot until
+ * the stream ends. Small bodies, including every valid V1, must not enter this
+ * gate. The 768Mi limit is sized for these two slots plus the ordinary body
+ * budget; widening either one needs a new memory measurement first.
*/
export const VISUAL_STREAM_LIMITS = {
maxConcurrentStreams: 2,
diff --git a/codex-llm-proxy/test/bodyAdmission.test.ts b/codex-llm-proxy/test/bodyAdmission.test.ts
index a5d6b40d2..08c62ab34 100644
--- a/codex-llm-proxy/test/bodyAdmission.test.ts
+++ b/codex-llm-proxy/test/bodyAdmission.test.ts
@@ -1,9 +1,10 @@
/**
* #731 R3-2 - body admission before `express.json`.
*
- * At an 8 MiB request cap, parsing every body the stream gate lets in (8
- * running plus 16 queued) holds about five copies of each in memory, far past
- * the proxy's 256Mi. These tests drive the real runtime app over HTTP and use
+ * At an 8 MiB request cap, the stream gate alone would let 24 bodies in (8
+ * running plus 16 queued). Five bodies in flight already peaked at 574-714 MiB
+ * of RSS, so 24 would not fit the proxy's 768Mi limit by that ratio (not
+ * measured at 24). These tests drive the real runtime app over HTTP and use
* the control-api `redeem` call as the witness: it runs only after the whole
* body was read, JSON-parsed, contract-parsed and hash-checked, so the number
* of attempts held there is the number of bodies in memory.
diff --git a/deploy/base/control-plane/codex-llm-proxy.yaml b/deploy/base/control-plane/codex-llm-proxy.yaml
index 4dc6f1d1d..4ddebeb85 100644
--- a/deploy/base/control-plane/codex-llm-proxy.yaml
+++ b/deploy/base/control-plane/codex-llm-proxy.yaml
@@ -78,6 +78,12 @@ spec:
secretKeyRef:
name: codex-llm-proxy-secrets
key: CODEX_LLM_PROXY_CONTROL_API_TOKEN
+ # Caps V8's old space well under the 768Mi limit so the heap is
+ # collected before the kernel kills the pod. Measured peak with
+ # three 8 MiB and two 24 MiB bodies in flight: 574 MiB of RSS with
+ # this cap, 714 MiB without it.
+ - name: NODE_OPTIONS
+ value: "--max-old-space-size=384"
ports:
- name: runtime
containerPort: 8080
@@ -101,10 +107,10 @@ spec:
resources:
requests:
cpu: 50m
- memory: 64Mi
+ memory: 256Mi
limits:
cpu: 250m
- memory: 256Mi
+ memory: 768Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true