Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(llm-proxy): document rate_limited, upstream_rejected and control_plane_unavailable

Alfredo Lopez5ccd0ee8d49d
diff --git a/codex-llm-proxy/test/server.security.test.ts b/codex-llm-proxy/test/server.security.test.ts
index bf8fb88cb..a1ff63014 100644
--- a/codex-llm-proxy/test/server.security.test.ts
+++ b/codex-llm-proxy/test/server.security.test.ts
@@ -1404,6 +1404,29 @@ describe('codex-llm-proxy attempt telemetry', () => {
expect(metricsText).not.toContain(rawCode)
})
+ // G1-5 (#720): the rpc gateway answers a redeem it could not deliver with
+ // JSON `control_plane_unavailable`; the proxy passes it through with its own
+ // status and metric label.
+ it('(g1-5) passes a control_plane_unavailable redeem denial through as 503 with its own label', async () => {
+ const { res, lines, metricsText } = await run(
+ 'att-cp-json',
+ 0,
+ 0,
+ undefined,
+ 'control_plane_unavailable'
+ )
+ expect(res.status).toBe(503)
+ expect(res.body).toEqual({ error: 'control_plane_unavailable' })
+ expect(lines).toHaveLength(1)
+ expect(lines[0]).toMatchObject({
+ outcome: 'failed',
+ code: 'control_plane_unavailable',
+ httpStatus: 503,
+ })
+ expect(failureCount(metricsText, 'control_plane_unavailable')).toBe(1)
+ expect(failureCount(metricsText, 'other')).toBe(0)
+ })
+
// `ATTEMPT_ERROR_STATUS` is an object literal, so an inherited name resolves
// to a function or an object instead of undefined. A plain index read would
// hand that value to `res.status()`, which throws ERR_HTTP_INVALID_STATUS_CODE
diff --git a/docs/architecture/codex-subscription-transport-contract.md b/docs/architecture/codex-subscription-transport-contract.md
index 16fb4304f..803cd88da 100644
--- a/docs/architecture/codex-subscription-transport-contract.md
+++ b/docs/architecture/codex-subscription-transport-contract.md
@@ -374,7 +374,9 @@ behavior changes:
`deliveredAs: 'sse_done'` and `usage` when present. On a thrown failure,
`outcome` is `failed` and the event adds `code`, the transport `reason`,
`details` (for example `{limit, observed}` on
- `tool_call_limit_exceeded`) and `deliveredAs`: `http_status` with
+ `tool_call_limit_exceeded`), `causeCode` when the failure is a rejected
+ `fetch` (the undici cause code, such as `ECONNREFUSED`, and nothing else
+ from the error) and `deliveredAs`: `http_status` with
`httpStatus` when no SSE byte had been sent, or `sse_error` when the
failure went out as an SSE error frame.
- Once the redeem succeeds, the proxy writes a `: keepalive` SSE comment
@@ -421,6 +423,7 @@ behavior changes:
Stable codes: `insufficient_scope`, `no_grant`, `model_not_allowed`,
`budget_denied`, `connection_unavailable`, `provider_unavailable`,
+`rate_limited`, `upstream_rejected`, `control_plane_unavailable`,
`origin_denied`, `ticket_invalid`, `ticket_replayed`, `request_hash_mismatch`,
`invalid_request`, `tool_call_limit_exceeded`, `sse_buffer_exceeded`,
`stream_duration_exceeded`, `context_length_exceeded`, `invalid_tool_arguments`,
@@ -455,6 +458,22 @@ code the proxy constructs, and every code it refuses a request with
- `Unauthorized`: HTTP 401. The platform JWT is missing or invalid.
- `not_found`: HTTP 404. Unknown route on the runtime, admin or probe listener.
- `internal_error`: HTTP 500. An unhandled error in the request pipeline.
+- `rate_limited`: HTTP 429. Either the upstream answered the completion with
+ 429 (a subscription quota or rate limit), or the proxy's own limiter on the
+ completion endpoint (60 requests per minute, counted before authorization
+ and body parsing) refused the request. For an upstream 429 the proxy
+ forwards `Retry-After` only when it is delta-seconds from 1 to 3600; any
+ other value is dropped, never guessed. The header travels only on a JSON
+ reply: once SSE bytes are on the wire, the error frame carries the code
+ alone. The limiter's reply carries the `Retry-After` and draft-7
+ `RateLimit`/`RateLimit-Policy` headers that express-rate-limit sets.
+- `upstream_rejected`: HTTP 502. The upstream answered the completion with a
+ 4xx that no narrower code covers (for example 404, 409 or 422). The same
+ request would get the same answer, so it is not a provider outage. The
+ status is in the log reason. An upstream 408 is transient and stays
+ `provider_unavailable`.
+- `control_plane_unavailable`: HTTP 503. The redeem could not reach
+ control-api; see "Control-plane outage" below.
- `context_length_exceeded`: the upstream refused the request because it
exceeds the model's context window. The upstream sends an SSE `error` event
@@ -464,7 +483,8 @@ code the proxy constructs, and every code it refuses a request with
non-success HTTP reply other than 401/403 is mapped the same way; the proxy
reads at most `UPSTREAM_ERROR_BODY_MAX_BYTES` (16 KiB) of that body and
otherwise keeps the status mapping (400 `invalid_request`, 401/403
- `connection_unavailable`, any other `provider_unavailable`). It is the only
+ `connection_unavailable`, 429 `rate_limited`, any other 4xx except 408
+ `upstream_rejected`, anything else `provider_unavailable`). It is the only
upstream code the proxy forwards: a streamed failure with any other upstream
code, or none, stays `provider_unavailable`. The Host maps it to
`LLM_CONTEXT_LENGTH_EXCEEDED`,
@@ -598,6 +618,30 @@ code the proxy constructs, and every code it refuses a request with
whole-body row applies only when the request carries an image; without one
the refusal is `payload_too_large` above.
+### Control-plane outage
+
+control-api runs one replica with a `Recreate` strategy, so a rollout or a
+cluster update leaves it with no ready endpoint for a few seconds; the same
+happens when `control-api-rpc-gateway` restarts. The proxy answers
+`control_plane_unavailable` only when the redeem certainly reached no live
+control-plane process:
+
+- the redeem `fetch` rejected before any response, the 15 s request timeout
+ had not fired, and the undici cause code is `ECONNREFUSED`, `ENOTFOUND`,
+ `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`. A
+ Service with no ready endpoint refuses the connection;
+- the redeem was denied with JSON `{"error":"control_plane_unavailable"}`,
+ which passes through like any other control-api code.
+
+These stay `provider_unavailable`, because control-api may have received the
+request: `ECONNRESET` or another socket error after the request was sent, the
+15 s timeout, a non-JSON 502/503/504 from a gateway, and a JSON
+`provider_unavailable`. Finalize failures do not change: they are logged and
+never reach the caller.
+
+The client logs `codex_proxy_control_api_unreachable` with `causeCode` and
+`path`, and the attempt line carries the same `causeCode`.
+
## Evidence
Origins are frozen from the public Codex CLI OAuth registration observed at
diff --git a/docs/architecture/grok-subscription-transport-contract.md b/docs/architecture/grok-subscription-transport-contract.md
index 019237faa..c5efe106c 100644
--- a/docs/architecture/grok-subscription-transport-contract.md
+++ b/docs/architecture/grok-subscription-transport-contract.md
@@ -288,8 +288,9 @@ For both providers, a stream counts as a completion only when it ends with
Stable codes: `insufficient_scope`, `no_grant`, `model_not_allowed`,
`budget_denied` (Host-side only), `connection_unavailable`,
-`provider_unavailable`, `origin_denied`, `ticket_invalid`, `ticket_replayed`,
-`request_hash_mismatch`, `client_upgrade_required`, `tool_call_limit_exceeded`,
+`provider_unavailable`, `rate_limited`, `upstream_rejected`,
+`control_plane_unavailable`, `origin_denied`, `ticket_invalid`,
+`ticket_replayed`, `request_hash_mismatch`, `client_upgrade_required`, `tool_call_limit_exceeded`,
`tool_call_arguments_exceeded`, `invalid_tool_arguments`, `invalid_request`,
`sse_buffer_exceeded`, `stream_duration_exceeded`, `payload_too_large`,
`context_length_exceeded`, `request_limit_exceeded` (Host-side only),
@@ -324,6 +325,25 @@ code the proxy constructs, and every code it refuses a request with
- `Unauthorized`: HTTP 401. The platform JWT is missing or invalid.
- `not_found`: HTTP 404. Unknown route on the runtime, admin or probe listener.
- `internal_error`: HTTP 500. An unhandled error in the request pipeline.
+- `rate_limited`: HTTP 429. Either the upstream answered the completion with
+ 429 (a subscription quota or rate limit), or the proxy's own limiter on the
+ completion endpoint (60 requests per minute, counted before authorization
+ and body parsing) refused the request. For an upstream 429 the proxy
+ forwards `Retry-After` only when it is delta-seconds from 1 to 3600; any
+ other value is dropped, never guessed. The header travels only on a JSON
+ reply: once SSE bytes are on the wire, the error frame carries the code
+ alone. The limiter's reply carries the `Retry-After` and draft-7
+ `RateLimit`/`RateLimit-Policy` headers that express-rate-limit sets.
+- `upstream_rejected`: HTTP 502. The upstream answered the completion with a
+ 4xx that no narrower code covers, including a 402/403 entitlement refusal.
+ The same request would get the same answer, so it is not a provider outage.
+ The status is in the log reason. The upstream status mapping is: 400
+ `invalid_request`, 401 `connection_unavailable`, 426
+ `client_upgrade_required`, 429 `rate_limited`, any other 4xx except 408
+ `upstream_rejected`, anything else (408 and 5xx included)
+ `provider_unavailable`.
+- `control_plane_unavailable`: HTTP 503. The redeem could not reach
+ control-api; see "Control-plane outage" below.
- `invalid_request`: the request body failed the transport schema (HTTP 400
before redeem), or the upstream answered the completion with HTTP 400.
@@ -497,6 +517,32 @@ queue wait exceeded, invalid deadline) reaches the Host as
`provider_unavailable`. The metric labels it `request_limit` to keep it apart
from upstream outages, and the log line carries the limit's fixed `reason`.
+### Control-plane outage
+
+control-api runs one replica with a `Recreate` strategy, so a rollout or a
+cluster update leaves it with no ready endpoint for a few seconds; the same
+happens when `control-api-rpc-gateway` restarts. The proxy answers
+`control_plane_unavailable` only when the redeem certainly reached no live
+control-plane process:
+
+- the redeem `fetch` rejected before any response, the 15 s request timeout
+ had not fired, and the undici cause code is `ECONNREFUSED`, `ENOTFOUND`,
+ `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`. A
+ Service with no ready endpoint refuses the connection;
+- the redeem was denied with JSON `{"error":"control_plane_unavailable"}`,
+ which passes through like any other control-api code.
+
+These stay `provider_unavailable`, because control-api may have received the
+request: `ECONNRESET` or another socket error after the request was sent, the
+15 s timeout, a non-JSON 502/503/504 from a gateway, and a JSON
+`provider_unavailable`. Finalize failures do not change: they are logged and
+never reach the caller.
+
+The client logs `grok_proxy_control_api_unreachable` with `causeCode` and
+`path`. The `grok_proxy_attempt_finished` line carries `causeCode` whenever
+the failure is a rejected `fetch`: the undici cause code, such as
+`ECONNREFUSED`, and nothing else from the error.
+
## Feature flags
All flags default to off.
diff --git a/grok-llm-proxy/test/server.security.test.ts b/grok-llm-proxy/test/server.security.test.ts
index 524beb08f..dc1ddc98c 100644
--- a/grok-llm-proxy/test/server.security.test.ts
+++ b/grok-llm-proxy/test/server.security.test.ts
@@ -1415,6 +1415,26 @@ describe('grok-llm-proxy attempt telemetry', () => {
expect(metricsText).not.toContain(rawCode)
})
+ // G1-5 (#720): the rpc gateway answers a redeem it could not deliver with
+ // JSON `control_plane_unavailable`; the proxy passes it through with its own
+ // status and metric label.
+ it('(g1-5) passes a control_plane_unavailable redeem denial through as 503 with its own label', async () => {
+ const { res, lines, metricsText } = await run({
+ providerAttemptId: 'att-cp-json',
+ deniedCode: 'control_plane_unavailable',
+ })
+ expect(res.status).toBe(503)
+ expect(res.body).toEqual({ error: 'control_plane_unavailable' })
+ expect(lines).toHaveLength(1)
+ expect(lines[0]).toMatchObject({
+ outcome: 'failed',
+ code: 'control_plane_unavailable',
+ httpStatus: 503,
+ })
+ expect(failureCount(metricsText, 'control_plane_unavailable')).toBe(1)
+ expect(failureCount(metricsText, 'other')).toBe(0)
+ })
+
// `ATTEMPT_ERROR_STATUS` is an object literal, so an inherited name resolves
// to a function or an object instead of undefined. A plain index read would
// hand that value to `res.status()`, which throws ERR_HTTP_INVALID_STATUS_CODE