Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs: narrow the control-plane outage wording and add the deploy order (#720)

Alfredo Lopez1bcab16216bf
diff --git a/codex-llm-proxy/src/server.ts b/codex-llm-proxy/src/server.ts
index 5a902616f..638500e61 100644
--- a/codex-llm-proxy/src/server.ts
+++ b/codex-llm-proxy/src/server.ts
@@ -640,7 +640,8 @@ export function createProxyApps(
res.removeHeader('cache-control')
res.setHeader('content-type', 'application/json; charset=utf-8')
// G1-1: an upstream Retry-After reaches the Host only on this path; an
- // SSE error frame carries the code alone.
+ // SSE error frame carries the code and, for upstream_rejected, the
+ // upstream status, never Retry-After.
const retryAfter =
err instanceof CodexTransportError ? err.details?.retryAfterSeconds : undefined
if (typeof retryAfter === 'number') res.setHeader('retry-after', String(retryAfter))
@@ -806,7 +807,8 @@ const ATTEMPT_ERROR_STATUS: Record<string, number> = {
// An upstream 4xx the same request would get again (G1-3, #720). Not 502:
// the gateways answer 502 when nothing behind them answered (R1-H2).
upstream_rejected: 422,
- // A control-api call no live process received (G1-4, #720).
+ // A redeem no control-plane process answered (G1-4, #720); control-api may
+ // still have received it when the gateway's 502 mapping produced the code.
control_plane_unavailable: 503,
invalid_tool_arguments: 422,
// The attempt ran for its whole stream budget. Retrying the same request
diff --git a/deploy/base/control-plane/configmaps.yaml b/deploy/base/control-plane/configmaps.yaml
index 146eef919..c3740fe09 100644
--- a/deploy/base/control-plane/configmaps.yaml
+++ b/deploy/base/control-plane/configmaps.yaml
@@ -152,7 +152,9 @@ data:
# G1-10 (#720): a 502 generated here means control-api produced no
# valid response (connect refused, closed before a response, invalid
- # header); a 502 control-api returns passes through, because
+ # header). In the last two cases control-api may already have
+ # processed the request, so the code means "no response", not "never
+ # received". A 502 control-api returns passes through, because
# `proxy_intercept_errors` is off. Only the redeem locations, whose
# failure reaches the user, map it. A 504 is not mapped: control-api
# may be alive and slow.
@@ -406,7 +408,9 @@ data:
# G1-10 (#720): a 502 generated here means control-api produced no
# valid response (connect refused, closed before a response, invalid
- # header); a 502 control-api returns passes through, because
+ # header). In the last two cases control-api may already have
+ # processed the request, so the code means "no response", not "never
+ # received". A 502 control-api returns passes through, because
# `proxy_intercept_errors` is off. Only the authorize location, whose
# failure reaches the user, maps it. A 504 is not mapped: control-api
# may be alive and slow.
diff --git a/docs/architecture/codex-subscription-transport-contract.md b/docs/architecture/codex-subscription-transport-contract.md
index 496e1ded9..94b8dfd87 100644
--- a/docs/architecture/codex-subscription-transport-contract.md
+++ b/docs/architecture/codex-subscription-transport-contract.md
@@ -476,8 +476,8 @@ code the proxy constructs, and every code it refuses a request with
the SSE error frame) and in the log line's `details`; the Host exposes it as
the classified error's `httpStatus`. An upstream 408 is transient and stays
`provider_unavailable`.
-- `control_plane_unavailable`: HTTP 503. The redeem could not reach
- control-api; see "Control-plane outage" below.
+- `control_plane_unavailable`: HTTP 503. No control-plane process answered
+ the redeem; see "Control-plane outage" below.
- `context_length_exceeded`: the upstream refused the request because it
exceeds the model's context window. The upstream sends an SSE `error` event
@@ -625,24 +625,47 @@ code the proxy constructs, and every code it refuses a request with
### Control-plane outage
control-api runs one replica with a `Recreate` strategy, so a rollout or a
-cluster update leaves it with no ready endpoint for a few seconds; the same
-happens when `control-api-rpc-gateway` restarts. The proxy answers
-`control_plane_unavailable` only when the redeem certainly reached no live
-control-plane process:
+cluster update leaves it with no ready endpoint; the same happens when
+`control-api-rpc-gateway` restarts. In the minikube check of #720 a real
+restart left control-api without an endpoint for about 15 s, after about 30 s
+in which the terminating pod still answered. The proxy answers
+`control_plane_unavailable` only when no control-plane process produced a
+response to the redeem:
- the redeem `fetch` rejected before any response, the 15 s request timeout
had not fired, and the undici cause code is `ECONNREFUSED`, `ENOTFOUND`,
- `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`. A
- Service with no ready endpoint refuses the connection;
-- the redeem was denied with JSON `{"error":"control_plane_unavailable"}`,
- which passes through like any other control-api code.
-
-These stay `provider_unavailable`, because control-api may have received the
-request: `ECONNRESET` or another socket error after the request was sent, the
-15 s timeout, a non-JSON 502/503/504 from a gateway, and a JSON
+ `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`.
+ The connection is refused only in the short window where the endpoint still
+ lists a pod that has exited. Once the endpoint is gone the SYN is dropped,
+ and undici gives up after its 10 s connect timeout with
+ `UND_ERR_CONNECT_TIMEOUT`;
+- the rpc gateway answered JSON `{"error":"control_plane_unavailable"}`. The
+ gateway writes that body itself, from an `error_page 502` named location,
+ when nginx generated the 502: the connection to control-api was refused, or
+ control-api closed it or sent an invalid header before a response. In the
+ last two cases control-api may already have processed the redeem, so the
+ code means that no response came back, not that the request never arrived.
+ The body passes through the proxy's JSON-`error` branch like a code
+ control-api returns.
+
+These stay `provider_unavailable`: `ECONNRESET` or another socket error after
+the request was sent, the 15 s timeout, a non-JSON 502/503/504 from a gateway
+(the rpc gateway's connect timeout is nginx's default 60 s, so during an
+outage the proxy's 15 s timeout usually fires first), and a JSON
`provider_unavailable`. Finalize failures do not change: they are logged and
never reach the caller.
+Deployment order: an mcp-host from before #720 has no arm for
+`control_plane_unavailable` and classifies it as `LLM_API_CALL_FAILED`, not
+retryable and without failover, where the same outage used to be a retryable
+`provider_unavailable`. Roll out the mcp-host images first, including the one
+HCC sets as `CONTEXT_MAPPER_HOST_IMAGE`, and wait until every Host pod runs
+the new image; then roll out the proxies and the control-plane ConfigMap,
+restarting `control-api-rpc-gateway` and `nginx-workflow-approval-gateway`,
+which mount `nginx.conf` through `subPath`. `rate_limited` and
+`upstream_rejected` need no order: an old Host already classifies them as a
+new one does, without `httpStatus`.
+
The client logs `codex_proxy_control_api_unreachable` with `path` only. The
cause code is logged on one line only: the attempt line for a redeem, and,
when both finalize tries fail, the `err` of `codex_proxy_finalize_failed`.
diff --git a/docs/architecture/grok-subscription-transport-contract.md b/docs/architecture/grok-subscription-transport-contract.md
index d50085e09..99e85c301 100644
--- a/docs/architecture/grok-subscription-transport-contract.md
+++ b/docs/architecture/grok-subscription-transport-contract.md
@@ -346,8 +346,8 @@ code the proxy constructs, and every code it refuses a request with
`client_upgrade_required`, 429 `rate_limited`, any other 4xx except 408
`upstream_rejected`, anything else (408 and 5xx included)
`provider_unavailable`.
-- `control_plane_unavailable`: HTTP 503. The redeem could not reach
- control-api; see "Control-plane outage" below.
+- `control_plane_unavailable`: HTTP 503. No control-plane process answered
+ the redeem; see "Control-plane outage" below.
- `invalid_request`: the request body failed the transport schema (HTTP 400
before redeem), or the upstream answered the completion with HTTP 400.
@@ -524,24 +524,47 @@ from upstream outages, and the log line carries the limit's fixed `reason`.
### Control-plane outage
control-api runs one replica with a `Recreate` strategy, so a rollout or a
-cluster update leaves it with no ready endpoint for a few seconds; the same
-happens when `control-api-rpc-gateway` restarts. The proxy answers
-`control_plane_unavailable` only when the redeem certainly reached no live
-control-plane process:
+cluster update leaves it with no ready endpoint; the same happens when
+`control-api-rpc-gateway` restarts. In the minikube check of #720 a real
+restart left control-api without an endpoint for about 15 s, after about 30 s
+in which the terminating pod still answered. The proxy answers
+`control_plane_unavailable` only when no control-plane process produced a
+response to the redeem:
- the redeem `fetch` rejected before any response, the 15 s request timeout
had not fired, and the undici cause code is `ECONNREFUSED`, `ENOTFOUND`,
- `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`. A
- Service with no ready endpoint refuses the connection;
-- the redeem was denied with JSON `{"error":"control_plane_unavailable"}`,
- which passes through like any other control-api code.
-
-These stay `provider_unavailable`, because control-api may have received the
-request: `ECONNRESET` or another socket error after the request was sent, the
-15 s timeout, a non-JSON 502/503/504 from a gateway, and a JSON
+ `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`.
+ The connection is refused only in the short window where the endpoint still
+ lists a pod that has exited. Once the endpoint is gone the SYN is dropped,
+ and undici gives up after its 10 s connect timeout with
+ `UND_ERR_CONNECT_TIMEOUT`;
+- the rpc gateway answered JSON `{"error":"control_plane_unavailable"}`. The
+ gateway writes that body itself, from an `error_page 502` named location,
+ when nginx generated the 502: the connection to control-api was refused, or
+ control-api closed it or sent an invalid header before a response. In the
+ last two cases control-api may already have processed the redeem, so the
+ code means that no response came back, not that the request never arrived.
+ The body passes through the proxy's JSON-`error` branch like a code
+ control-api returns.
+
+These stay `provider_unavailable`: `ECONNRESET` or another socket error after
+the request was sent, the 15 s timeout, a non-JSON 502/503/504 from a gateway
+(the rpc gateway's connect timeout is nginx's default 60 s, so during an
+outage the proxy's 15 s timeout usually fires first), and a JSON
`provider_unavailable`. Finalize failures do not change: they are logged and
never reach the caller.
+Deployment order: an mcp-host from before #720 has no arm for
+`control_plane_unavailable` and classifies it as `LLM_API_CALL_FAILED`, not
+retryable and without failover, where the same outage used to be a retryable
+`provider_unavailable`. Roll out the mcp-host images first, including the one
+HCC sets as `CONTEXT_MAPPER_HOST_IMAGE`, and wait until every Host pod runs
+the new image; then roll out the proxies and the control-plane ConfigMap,
+restarting `control-api-rpc-gateway` and `nginx-workflow-approval-gateway`,
+which mount `nginx.conf` through `subPath`. `rate_limited` and
+`upstream_rejected` need no order: an old Host already classifies them as a
+new one does, without `httpStatus`.
+
The client logs `grok_proxy_control_api_unreachable` with `path` only. The
`grok_proxy_attempt_finished` line carries `causeCode` whenever the failure
is a rejected `fetch`: the undici cause code, such as `ECONNREFUSED`, and
diff --git a/grok-llm-proxy/src/server.ts b/grok-llm-proxy/src/server.ts
index c5f710e74..a2dcd38dc 100644
--- a/grok-llm-proxy/src/server.ts
+++ b/grok-llm-proxy/src/server.ts
@@ -513,7 +513,8 @@ export function createProxyApps(
res.removeHeader('cache-control')
res.setHeader('content-type', 'application/json; charset=utf-8')
// G1-1: an upstream Retry-After reaches the Host only on this path; an
- // SSE error frame carries the code alone.
+ // SSE error frame carries the code and, for upstream_rejected, the
+ // upstream status, never Retry-After.
const retryAfter =
err instanceof GrokTransportError ? err.details?.retryAfterSeconds : undefined
if (typeof retryAfter === 'number') res.setHeader('retry-after', String(retryAfter))
@@ -679,7 +680,8 @@ const ATTEMPT_ERROR_STATUS: Record<string, number> = {
// An upstream 4xx the same request would get again (G1-3, #720). Not 502:
// the gateways answer 502 when nothing behind them answered (R1-H2).
upstream_rejected: 422,
- // A control-api call no live process received (G1-4, #720).
+ // A redeem no control-plane process answered (G1-4, #720); control-api may
+ // still have received it when the gateway's 502 mapping produced the code.
control_plane_unavailable: 503,
tool_call_arguments_exceeded: 422,
invalid_tool_arguments: 422,