Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(deploy): give grok-llm-proxy 768Mi and a 384 MiB heap cap (#739)

Alfredo Lopezaa506853b882
diff --git a/deploy/base/control-plane/grok-llm-proxy.yaml b/deploy/base/control-plane/grok-llm-proxy.yaml
index 831653870..f22147197 100644
--- a/deploy/base/control-plane/grok-llm-proxy.yaml
+++ b/deploy/base/control-plane/grok-llm-proxy.yaml
@@ -73,6 +73,12 @@ spec:
secretKeyRef:
name: grok-llm-proxy-secrets
key: GROK_LLM_PROXY_CONTROL_API_TOKEN
+ # Caps V8's old space well under the 768Mi limit so the heap is
+ # collected before the kernel kills the pod. Measured peak with
+ # three 8 MiB bodies in flight: 230-252 MiB of RSS without this
+ # cap and 249-292 MiB with it (one run per mode).
+ - name: NODE_OPTIONS
+ value: "--max-old-space-size=384"
ports:
- name: runtime
containerPort: 8080
@@ -96,10 +102,10 @@ spec:
resources:
requests:
cpu: 50m
- memory: 64Mi
+ memory: 256Mi
limits:
cpu: 250m
- memory: 256Mi
+ memory: 768Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
diff --git a/grok-llm-proxy/src/requestLimits.ts b/grok-llm-proxy/src/requestLimits.ts
index d76a0d975..fda0c39fa 100644
--- a/grok-llm-proxy/src/requestLimits.ts
+++ b/grok-llm-proxy/src/requestLimits.ts
@@ -15,10 +15,13 @@ export const DEFAULT_MAX_BODY_BYTES = CONTRACT_LIMITS.maxRequestBodyBytes + ENVE
/**
* #731 R3-2 — how many maximum-size bodies may be read and parsed at once.
- * About five copies of a body are alive while it is parsed and hashed (the raw
+ * Several copies of a body are alive while it is parsed and hashed (the raw
* buffer, the decoded string, the parsed object, the contract copy and the
- * canonical serialization), so three 8 MiB bodies hold about 120 MiB of the
- * pod's 256Mi. Without this bound the stream gate would let 24 bodies in.
+ * canonical serialization). Without this bound the stream gate would let 24
+ * bodies in. Measured with three 8 MiB bodies in flight, the process peaked at
+ * 230-252 MiB of RSS with an uncapped heap and at 249-292 MiB with
+ * `--max-old-space-size=384` (one run per mode), past the former 256Mi limit,
+ * which is why the deployment sets that cap and a 768Mi memory limit.
*/
export const IN_FLIGHT_BODY_BUDGET_BODIES = 3
diff --git a/grok-llm-proxy/test/bodyAdmission.test.ts b/grok-llm-proxy/test/bodyAdmission.test.ts
index d15bdec75..ec5fbb0a7 100644
--- a/grok-llm-proxy/test/bodyAdmission.test.ts
+++ b/grok-llm-proxy/test/bodyAdmission.test.ts
@@ -1,9 +1,10 @@
/**
* #731 R3-2 - body admission before `express.json`.
*
- * At an 8 MiB request cap, parsing every body the stream gate lets in (8
- * running plus 16 queued) holds about five copies of each in memory, far past
- * the proxy's 256Mi. These tests drive the real runtime app over HTTP and use
+ * At an 8 MiB request cap, the stream gate alone would let 24 bodies in (8
+ * running plus 16 queued). Three bodies in flight already peaked at 230-292 MiB
+ * of RSS, so 24 would not fit the proxy's 768Mi limit by that ratio (not
+ * measured at 24). These tests drive the real runtime app over HTTP and use
* the control-api `redeem` call as the witness: it runs only after the whole
* body was read, JSON-parsed, contract-parsed and hash-checked, so the number
* of attempts held there is the number of bodies in memory.