Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(deploy): raise the proxy heap cap to 512 MiB and the memory limit to 1536Mi

Alfredo Lopezbd707a59e906
diff --git a/codex-llm-proxy/src/config.ts b/codex-llm-proxy/src/config.ts
index 43178bec3..92359bb07 100644
--- a/codex-llm-proxy/src/config.ts
+++ b/codex-llm-proxy/src/config.ts
@@ -91,7 +91,7 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): CodexLlmProxyC
`CODEX_LLM_PROXY_MAX_VISUAL_BODY_BYTES must be at least ${LIMITS.maxVisualRequestBodyBytes}, the contract maxVisualRequestBodyBytes`
)
}
- // The 1Gi memory limit was measured with two visual streams at the contract
+ // The 1536Mi memory limit was measured with two visual streams at the contract
// ceiling only (see IN_FLIGHT_BODY_BUDGET_BODIES). A larger value would admit
// bodies that measurement never covered.
if (maxVisualBodyBytes > LIMITS.maxVisualRequestBodyBytes) {
diff --git a/codex-llm-proxy/src/requestLimits.ts b/codex-llm-proxy/src/requestLimits.ts
index 8a3944325..d2e36913a 100644
--- a/codex-llm-proxy/src/requestLimits.ts
+++ b/codex-llm-proxy/src/requestLimits.ts
@@ -29,7 +29,12 @@ export const DEFAULT_MAX_BODY_BYTES = CONTRACT_LIMITS.maxRequestBodyBytes + ENVE
* full load the gates admit (eight 8 MiB streams, two 24 MiB visual streams
* and three queued 8 MiB bodies, #739 D5), the process peaked at 790 MiB of
* RSS with `--max-old-space-size=384` and at 886-1009 MiB with an uncapped
- * heap, which is why the deployment sets that cap and a 1Gi memory limit.
+ * heap, which is why the deployment sets a heap cap. The budget counts bytes,
+ * not structure: bodies at the worst structure the contract admits (262 144
+ * containers, 262 144 members) are 4.7 MiB each, so it admits about five of
+ * them parsing at once, each a ~39 MiB tree. That aborted the process at a
+ * 384 MiB heap cap; the deployment uses 512 MiB and a 1536Mi memory limit,
+ * the peak of that load (1095-1129 MiB of RSS) plus 25 % (#806 Q1).
*
* Bytes alone do not bound the parse: JSON.parse allocates one heap
* object per container, so the contracts cap containers at
@@ -83,7 +88,7 @@ export const STREAM_LIMITS = {
* The pod cannot hold the ordinary 8-stream gate across a 24 MiB image, so
* those requests are a tighter sibling and a V2 request keeps the slot until
* the stream ends. Small bodies, including every valid V1, must not enter this
- * gate. The 1Gi limit is sized for these two slots plus the ordinary body
+ * gate. The 1536Mi limit is sized for these two slots plus the ordinary body
* budget; widening either one needs a new memory measurement first.
*/
export const VISUAL_STREAM_LIMITS = {
diff --git a/codex-llm-proxy/test/bodyAdmission.test.ts b/codex-llm-proxy/test/bodyAdmission.test.ts
index 8803b4214..4959a139f 100644
--- a/codex-llm-proxy/test/bodyAdmission.test.ts
+++ b/codex-llm-proxy/test/bodyAdmission.test.ts
@@ -4,7 +4,7 @@
* At an 8 MiB request cap, the stream gate alone would let 24 bodies in (8
* running plus 16 queued). Thirteen bodies in flight (eight streams, two
* visual, three queued; #739 D5) already peaked at 790 MiB of RSS with the
- * heap capped, so 24 would not fit the proxy's 1Gi limit by that ratio (not
+ * heap capped, so 24 would not fit the proxy's 1536Mi limit by that ratio (not
* measured at 24). These tests drive the real runtime app over HTTP and use
* the control-api `redeem` call as the witness: it runs only after the whole
* body was read, JSON-parsed, contract-parsed and hash-checked, so the number
diff --git a/codex-llm-proxy/test/server.security.test.ts b/codex-llm-proxy/test/server.security.test.ts
index bb2d2590c..4a2481097 100644
--- a/codex-llm-proxy/test/server.security.test.ts
+++ b/codex-llm-proxy/test/server.security.test.ts
@@ -638,7 +638,7 @@ describe('codex-llm-proxy security surface', () => {
expect(loadConfig(required).maxVisualBodyBytes).toBe(24 * 1024 * 1024)
})
- // The 1Gi memory limit was measured with two visual streams at the contract
+ // The 1536Mi memory limit was measured with two visual streams at the contract
// ceiling. A larger visual limit would admit bodies that measurement never
// covered, so the proxy refuses to start with one.
it('refuses a visual body limit above the contract visual ceiling', () => {
diff --git a/deploy/base/control-plane/codex-llm-proxy.yaml b/deploy/base/control-plane/codex-llm-proxy.yaml
index 4190dd30e..011ff7998 100644
--- a/deploy/base/control-plane/codex-llm-proxy.yaml
+++ b/deploy/base/control-plane/codex-llm-proxy.yaml
@@ -84,12 +84,16 @@ spec:
secretKeyRef:
name: codex-llm-proxy-secrets
key: CODEX_LLM_PROXY_CONTROL_API_TOKEN
- # Caps V8's old space under the 1Gi limit so the heap is collected
- # before the kernel kills the pod. Measured peak with eight 8 MiB
- # streams, two 24 MiB visual streams and three queued 8 MiB bodies
- # (#739 D5): 790 MiB of RSS with this cap, 886-1009 MiB without it.
+ # Caps V8's old space under the 1536Mi limit so the heap is
+ # collected before the kernel kills the pod. Measured peak with
+ # eight 8 MiB streams, two 24 MiB visual streams and three queued
+ # 8 MiB bodies (#739 D5): 790 MiB of RSS at a 384 MiB cap, 886-1009
+ # MiB without one. When the three ordinary bodies carry the worst
+ # structure the contract admits (#806 Q1), 384 MiB aborts while
+ # parsing them; 512 MiB is the smallest cap that survives, and that
+ # load peaked at 1095-1129 MiB of RSS.
- name: NODE_OPTIONS
- value: "--max-old-space-size=384"
+ value: "--max-old-space-size=512"
ports:
- name: runtime
containerPort: 8080
@@ -116,7 +120,7 @@ spec:
memory: 768Mi
limits:
cpu: 250m
- memory: 1Gi
+ memory: 1536Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
diff --git a/deploy/base/control-plane/grok-llm-proxy.yaml b/deploy/base/control-plane/grok-llm-proxy.yaml
index 420aa0b23..f40acb507 100644
--- a/deploy/base/control-plane/grok-llm-proxy.yaml
+++ b/deploy/base/control-plane/grok-llm-proxy.yaml
@@ -84,14 +84,18 @@ spec:
secretKeyRef:
name: grok-llm-proxy-secrets
key: GROK_LLM_PROXY_CONTROL_API_TOKEN
- # Caps V8's old space well under the 1Gi limit, so heap growth
- # is bounded and collected before the kernel kills the pod. The
- # cap does not lower the peak: with eight 8 MiB streams and three
- # queued 8 MiB bodies (#739 D5), RSS measured 511 MiB with it and
- # 480-511 MiB without it. One ~36 MB V2 stream in the visual slot
- # on top of that load (#784) raised the capped peak to 775 MiB.
+ # Caps V8's old space under the 1536Mi limit, so heap growth is
+ # bounded and collected before the kernel kills the pod. The cap
+ # does not lower the peak: with eight 8 MiB streams and three
+ # queued 8 MiB bodies (#739 D5), RSS measured 511 MiB at a 384 MiB
+ # cap and 480-511 MiB without it. One ~36 MB V2 stream in the
+ # visual slot on top of that load (#784) raised it to 775 MiB. When
+ # the three ordinary bodies carry the worst structure the contract
+ # admits (#806 Q1), 384 MiB aborts while parsing them; 512 MiB is
+ # the smallest cap that survives, and that load peaked at
+ # 1097-1111 MiB of RSS.
- name: NODE_OPTIONS
- value: "--max-old-space-size=384"
+ value: "--max-old-space-size=512"
ports:
- name: runtime
containerPort: 8080
@@ -118,7 +122,7 @@ spec:
memory: 768Mi
limits:
cpu: 250m
- memory: 1Gi
+ memory: 1536Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
diff --git a/docs/architecture/codex-subscription-transport-contract.md b/docs/architecture/codex-subscription-transport-contract.md
index a1c6fdbf6..41aabbe99 100644
--- a/docs/architecture/codex-subscription-transport-contract.md
+++ b/docs/architecture/codex-subscription-transport-contract.md
@@ -241,8 +241,9 @@ window and its source (`catalog` or `default`).
### Body structure before parse (#806)
`JSON.parse` allocates one heap object per container before any contract
-check runs, so a body within the byte limit can exhaust a heap capped at
-384 MiB. A 25165824-byte body of `[],` padding aborted `codex-llm-proxy`, and
+check runs, so a body within the byte limit can exhaust a capped heap (384 MiB
+in the measurements below; the manifest now caps it at 512 MiB, see the end of
+this section). A 25165824-byte body of `[],` padding aborted `codex-llm-proxy`, and
the contracts accepted 4117647 containers in an 8 MiB request. Both contracts
therefore bound the structure as well as the bytes, and every parser checks the
raw body before parsing it.
@@ -291,6 +292,20 @@ keep every run at exit 0 with five of five admitted, at most 200 MiB after GC
and at most 280 MiB sampled peak. These figures are lower bounds: the
serializations made while forwarding are not included.
+Worst-structure load (r16, macOS, Node v24.18.0, tsc build, real proxy with
+undici to a loopback upstream, 29 runs across both proxies). The bodies above
+carry only containers. A body at all three limits (262144 containers, 262144
+members, 1048576 elements) parses into about 39 MiB, and the byte budget
+admits about five of them at once. Codex with two visual bodies at that shape:
+384 MiB heap aborted out of memory with zero upstream hits, 512 MiB passed
+(resident after GC 224.8 MiB, peak RSS 1095-1129 MiB). Scales 1.5 and 2 of the
+same shape passed at 384 MiB, so the abort is specific to the densest bodies.
+The manifest therefore sets `--max-old-space-size=512` and a 1536Mi limit
+(`ceil(1129.2 × 1.25)` = 1412 MiB, pinned by `deployManifest.test.ts`), with
+the request kept at 768Mi. The resident heap at 512 MiB is still above the
+200 MiB target, so a structure-weighted body budget and parsing after the
+stream slot is granted remain open as a code change.
+
The bound does not refuse realistic requests. Tool results and message
content are strings, which count only as structural bytes for their quotes; a
2 MiB tool catalog is about 100000 containers. Realistic JSON reaches about
diff --git a/docs/architecture/grok-subscription-transport-contract.md b/docs/architecture/grok-subscription-transport-contract.md
index 2b5b4bc96..d9d657010 100644
--- a/docs/architecture/grok-subscription-transport-contract.md
+++ b/docs/architecture/grok-subscription-transport-contract.md
@@ -159,13 +159,23 @@ disconnect earlier would mean reading queued bodies into memory, which is
what the gate exists to avoid. `grok-llm-proxy/test/streamGate.handoff.test.ts`
pins both cases.
-Memory. Measured on macOS (Node v24.18.0, tsc build, one process, heap capped
-at 384 MiB, upstream calls through undici): #739's D5 load (eight 8 MiB streams
-held and three 8 MiB bodies queued) peaked at 511 MiB, and D5 plus one
-36.3 MB V2 stream in the visual gate peaked at 775.4 MiB. The manifest
-therefore sets a 1Gi limit and a 768Mi request, the Codex shape, and
+Memory. Measured on macOS (Node v24.18.0, tsc build, one process, upstream
+calls through undici, 29 runs): #739's D5 load (eight 8 MiB streams held and
+three 8 MiB bodies queued) peaked at 511 MiB, and D5 plus one 36.3 MB V2 stream
+in the visual gate peaked at 775.4 MiB. A body at the structure limits (262144
+containers, 262144 members, 1048576 elements) is a worse load: the byte budget
+admits about five dense 4.7 MiB bodies at once, and each parse allocates a tree
+of about 39 MiB plus transient copies. At a 384 MiB heap cap that load aborted
+with out-of-memory before any upstream call; at 512 MiB it passed with a peak
+RSS of 1097-1111 MiB. The manifest therefore sets `--max-old-space-size=512`, a
+1536Mi limit and a 768Mi request, and
`grok-llm-proxy/test/deployManifest.test.ts` requires the limit to be at least
-1.25 × the peak. The in-cluster peak is not measured.
+1.25 × that peak (`ceil(1110.7 × 1.25)` = 1389 MiB). The in-cluster peak is
+not measured. The heap cap is a resource change only: the resident heap after
+GC at that load was 176.6-208.4 MiB, above the 200 MiB target recorded for
+`BODY_STRUCTURE_LIMITS`, so lowering the resident set needs a code change
+(a structure-weighted body budget, and parsing after the stream slot is
+granted), which is tracked separately.
Upstream projection. The proxy maps a user message with parts to
`content: [{ type: 'input_text', text }, { type: 'input_image', image_url:
@@ -449,8 +459,8 @@ Proxy robustness (both proxies):
### Body structure before parse (#806)
`JSON.parse` allocates one heap object per container before any contract
-check runs, so a body within the byte limit can exhaust the proxy's 384 MiB
-heap. A 36700158-byte body of `[],` padding aborted `grok-llm-proxy`, and so
+check runs, so a body within the byte limit can exhaust the proxy's heap cap
+(512 MiB, measured; see "Memory" above). A 36700158-byte body of `[],` padding aborted `grok-llm-proxy`, and so
did three ordinary 8 MiB bodies of 4117647 empty containers each, which the
contract accepted. The fix is the same in both proxies and both contracts, and
is described in full, with the measurement, under "Body structure before
diff --git a/grok-llm-proxy/src/config.ts b/grok-llm-proxy/src/config.ts
index e21885c06..1d4ed00f5 100644
--- a/grok-llm-proxy/src/config.ts
+++ b/grok-llm-proxy/src/config.ts
@@ -93,7 +93,7 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): GrokLlmProxyCo
`GROK_LLM_PROXY_MAX_VISUAL_BODY_BYTES must be at least ${LIMITS.maxVisualRequestBodyBytes}, the contract maxVisualRequestBodyBytes`
)
}
- // The 1Gi memory limit was measured with one visual stream at the contract
+ // The 1536Mi memory limit was measured with one visual stream at the contract
// ceiling only (see VISUAL_STREAM_LIMITS). A larger value would admit bodies
// that measurement never covered.
if (maxVisualBodyBytes > LIMITS.maxVisualRequestBodyBytes) {
diff --git a/grok-llm-proxy/src/requestLimits.ts b/grok-llm-proxy/src/requestLimits.ts
index fe6792103..024e028fc 100644
--- a/grok-llm-proxy/src/requestLimits.ts
+++ b/grok-llm-proxy/src/requestLimits.ts
@@ -27,9 +27,13 @@ export const DEFAULT_MAX_BODY_BYTES = CONTRACT_LIMITS.maxRequestBodyBytes + ENVE
* visual slot. With eight 8 MiB streams and three queued 8 MiB bodies (#739
* D5) the process peaked at 511 MiB of RSS with `--max-old-space-size=384`
* (#739 measured 509.8) and at 480-511 MiB with an uncapped heap, which is
- * why the deployment sets that heap cap. The full load the gates admit adds
- * one visual stream; `VISUAL_STREAM_LIMITS` records that peak and the 1Gi
- * memory limit it sets.
+ * why the deployment sets a heap cap. The full load the gates admit adds
+ * one visual stream; `VISUAL_STREAM_LIMITS` records that peak and the memory
+ * limit it sets. The budget counts bytes, not structure: bodies at the worst
+ * structure the contract admits (262 144 containers, 262 144 members) are
+ * 4.7 MiB each, so it admits about five of them parsing at once, each a
+ * ~39 MiB tree. That aborted the process at a 384 MiB heap cap; the
+ * deployment uses 512 MiB (#806 Q1).
*/
export const IN_FLIGHT_BODY_BUDGET_BODIES = 3
@@ -74,9 +78,11 @@ export const STREAM_LIMITS = {
* Measured with the full load the gates admit (the D5 load above plus one
* ~36 MB V2 stream: a 20 MiB PNG and 8 MiB of text; tsc build, one process,
* `--max-old-space-size=384`, upstream request through undici), the process
- * peaked at 775 MiB of RSS, against 511 MiB for D5 alone. The deployment's
- * 1Gi memory limit is that peak plus 25 %, rounded up; widening this gate or
- * the ordinary body budget needs a new memory measurement first.
+ * peaked at 775 MiB of RSS, against 511 MiB for D5 alone. With the ordinary
+ * bodies at the worst structure and a 512 MiB heap cap, the same load peaked
+ * at 1097-1111 MiB. The deployment's 1536Mi memory limit is that peak plus
+ * 25 %, rounded up; widening this gate or the ordinary body budget needs a
+ * new memory measurement first.
*/
export const VISUAL_STREAM_LIMITS = {
maxConcurrentStreams: 1,
diff --git a/grok-llm-proxy/test/bodyAdmission.test.ts b/grok-llm-proxy/test/bodyAdmission.test.ts
index 7e4ff9bc9..d5ecdfd3e 100644
--- a/grok-llm-proxy/test/bodyAdmission.test.ts
+++ b/grok-llm-proxy/test/bodyAdmission.test.ts
@@ -4,7 +4,7 @@
* At an 8 MiB request cap, the stream gate alone would let 24 bodies in (8
* running plus 16 queued). Eleven bodies in flight (eight streams, three
* queued; #739 D5) already peaked at 511 MiB of RSS with the heap capped, so
- * 24 would not fit the proxy's 1Gi limit by that ratio (not measured at 24).
+ * 24 would not fit the proxy's 1536Mi limit by that ratio (not measured at 24).
* These tests drive the real runtime app over HTTP and use the control-api
* `redeem` call as the witness: it runs only after the whole
* body was read, JSON-parsed, contract-parsed and hash-checked, so the number
diff --git a/grok-llm-proxy/test/requestLimits.test.ts b/grok-llm-proxy/test/requestLimits.test.ts
index f1ddc8aba..077f6c8b3 100644
--- a/grok-llm-proxy/test/requestLimits.test.ts
+++ b/grok-llm-proxy/test/requestLimits.test.ts
@@ -398,7 +398,7 @@ describe('StreamGate', () => {
// A V2 body above the ordinary cap keeps its image bytes resident for the
// whole upstream stream, so it takes a 1-wide sibling of the 8-wide gate.
- // Widening it needs a new memory measurement against the 1Gi limit.
+ // Widening it needs a new memory measurement against the 1536Mi limit.
it('pins the visual 1/4 sibling, its per-host share, and the ordinary 8/16 gate', () => {
expect(VISUAL_STREAM_LIMITS).toEqual({ maxConcurrentStreams: 1, maxQueuedRequests: 4 })
expect(VISUAL_PER_HOST_MAX_ADMITTED).toBe(2)
diff --git a/grok-llm-proxy/test/server.security.test.ts b/grok-llm-proxy/test/server.security.test.ts
index 797f799d8..2600fb7f9 100644
--- a/grok-llm-proxy/test/server.security.test.ts
+++ b/grok-llm-proxy/test/server.security.test.ts
@@ -831,7 +831,7 @@ describe('grok-llm-proxy startup config', () => {
).toBe(LIMITS.maxVisualRequestBodyBytes)
})
- // The 1Gi memory limit was measured with one visual stream at the contract
+ // The 1536Mi memory limit was measured with one visual stream at the contract
// ceiling. A larger visual limit would admit bodies that measurement never
// covered, so the proxy refuses to start with one.
it('refuses a visual body limit above the contract visual ceiling', () => {