Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs: narrow the control-plane outage wording and add the deploy order (#720)
Alfredo Lopez1bcab16216bf
diff --git a/codex-llm-proxy/src/server.ts b/codex-llm-proxy/src/server.tsindex 5a902616f..638500e61 100644--- a/codex-llm-proxy/src/server.ts+++ b/codex-llm-proxy/src/server.ts@@ -640,7 +640,8 @@ export function createProxyApps(res.removeHeader('cache-control')res.setHeader('content-type', 'application/json; charset=utf-8')// G1-1: an upstream Retry-After reaches the Host only on this path; an- // SSE error frame carries the code alone.+ // SSE error frame carries the code and, for upstream_rejected, the+ // upstream status, never Retry-After.const retryAfter =err instanceof CodexTransportError ? err.details?.retryAfterSeconds : undefinedif (typeof retryAfter === 'number') res.setHeader('retry-after', String(retryAfter))@@ -806,7 +807,8 @@ const ATTEMPT_ERROR_STATUS: Record<string, number> = {// An upstream 4xx the same request would get again (G1-3, #720). Not 502:// the gateways answer 502 when nothing behind them answered (R1-H2).upstream_rejected: 422,- // A control-api call no live process received (G1-4, #720).+ // A redeem no control-plane process answered (G1-4, #720); control-api may+ // still have received it when the gateway's 502 mapping produced the code.control_plane_unavailable: 503,invalid_tool_arguments: 422,// The attempt ran for its whole stream budget. Retrying the same requestdiff --git a/deploy/base/control-plane/configmaps.yaml b/deploy/base/control-plane/configmaps.yamlindex 146eef919..c3740fe09 100644--- a/deploy/base/control-plane/configmaps.yaml+++ b/deploy/base/control-plane/configmaps.yaml@@ -152,7 +152,9 @@ data:# G1-10 (#720): a 502 generated here means control-api produced no# valid response (connect refused, closed before a response, invalid- # header); a 502 control-api returns passes through, because+ # header). In the last two cases control-api may already have+ # processed the request, so the code means "no response", not "never+ # received". A 502 control-api returns passes through, because# `proxy_intercept_errors` is off. Only the redeem locations, whose# failure reaches the user, map it. A 504 is not mapped: control-api# may be alive and slow.@@ -406,7 +408,9 @@ data:# G1-10 (#720): a 502 generated here means control-api produced no# valid response (connect refused, closed before a response, invalid- # header); a 502 control-api returns passes through, because+ # header). In the last two cases control-api may already have+ # processed the request, so the code means "no response", not "never+ # received". A 502 control-api returns passes through, because# `proxy_intercept_errors` is off. Only the authorize location, whose# failure reaches the user, maps it. A 504 is not mapped: control-api# may be alive and slow.diff --git a/docs/architecture/codex-subscription-transport-contract.md b/docs/architecture/codex-subscription-transport-contract.mdindex 496e1ded9..94b8dfd87 100644--- a/docs/architecture/codex-subscription-transport-contract.md+++ b/docs/architecture/codex-subscription-transport-contract.md@@ -476,8 +476,8 @@ code the proxy constructs, and every code it refuses a request withthe SSE error frame) and in the log line's `details`; the Host exposes it asthe classified error's `httpStatus`. An upstream 408 is transient and stays`provider_unavailable`.-- `control_plane_unavailable`: HTTP 503. The redeem could not reach- control-api; see "Control-plane outage" below.+- `control_plane_unavailable`: HTTP 503. No control-plane process answered+ the redeem; see "Control-plane outage" below.- `context_length_exceeded`: the upstream refused the request because itexceeds the model's context window. The upstream sends an SSE `error` event@@ -625,24 +625,47 @@ code the proxy constructs, and every code it refuses a request with### Control-plane outagecontrol-api runs one replica with a `Recreate` strategy, so a rollout or a-cluster update leaves it with no ready endpoint for a few seconds; the same-happens when `control-api-rpc-gateway` restarts. The proxy answers-`control_plane_unavailable` only when the redeem certainly reached no live-control-plane process:+cluster update leaves it with no ready endpoint; the same happens when+`control-api-rpc-gateway` restarts. In the minikube check of #720 a real+restart left control-api without an endpoint for about 15 s, after about 30 s+in which the terminating pod still answered. The proxy answers+`control_plane_unavailable` only when no control-plane process produced a+response to the redeem:- the redeem `fetch` rejected before any response, the 15 s request timeouthad not fired, and the undici cause code is `ECONNREFUSED`, `ENOTFOUND`,- `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`. A- Service with no ready endpoint refuses the connection;-- the redeem was denied with JSON `{"error":"control_plane_unavailable"}`,- which passes through like any other control-api code.--These stay `provider_unavailable`, because control-api may have received the-request: `ECONNRESET` or another socket error after the request was sent, the-15 s timeout, a non-JSON 502/503/504 from a gateway, and a JSON+ `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`.+ The connection is refused only in the short window where the endpoint still+ lists a pod that has exited. Once the endpoint is gone the SYN is dropped,+ and undici gives up after its 10 s connect timeout with+ `UND_ERR_CONNECT_TIMEOUT`;+- the rpc gateway answered JSON `{"error":"control_plane_unavailable"}`. The+ gateway writes that body itself, from an `error_page 502` named location,+ when nginx generated the 502: the connection to control-api was refused, or+ control-api closed it or sent an invalid header before a response. In the+ last two cases control-api may already have processed the redeem, so the+ code means that no response came back, not that the request never arrived.+ The body passes through the proxy's JSON-`error` branch like a code+ control-api returns.++These stay `provider_unavailable`: `ECONNRESET` or another socket error after+the request was sent, the 15 s timeout, a non-JSON 502/503/504 from a gateway+(the rpc gateway's connect timeout is nginx's default 60 s, so during an+outage the proxy's 15 s timeout usually fires first), and a JSON`provider_unavailable`. Finalize failures do not change: they are logged andnever reach the caller.+Deployment order: an mcp-host from before #720 has no arm for+`control_plane_unavailable` and classifies it as `LLM_API_CALL_FAILED`, not+retryable and without failover, where the same outage used to be a retryable+`provider_unavailable`. Roll out the mcp-host images first, including the one+HCC sets as `CONTEXT_MAPPER_HOST_IMAGE`, and wait until every Host pod runs+the new image; then roll out the proxies and the control-plane ConfigMap,+restarting `control-api-rpc-gateway` and `nginx-workflow-approval-gateway`,+which mount `nginx.conf` through `subPath`. `rate_limited` and+`upstream_rejected` need no order: an old Host already classifies them as a+new one does, without `httpStatus`.+The client logs `codex_proxy_control_api_unreachable` with `path` only. Thecause code is logged on one line only: the attempt line for a redeem, and,when both finalize tries fail, the `err` of `codex_proxy_finalize_failed`.diff --git a/docs/architecture/grok-subscription-transport-contract.md b/docs/architecture/grok-subscription-transport-contract.mdindex d50085e09..99e85c301 100644--- a/docs/architecture/grok-subscription-transport-contract.md+++ b/docs/architecture/grok-subscription-transport-contract.md@@ -346,8 +346,8 @@ code the proxy constructs, and every code it refuses a request with`client_upgrade_required`, 429 `rate_limited`, any other 4xx except 408`upstream_rejected`, anything else (408 and 5xx included)`provider_unavailable`.-- `control_plane_unavailable`: HTTP 503. The redeem could not reach- control-api; see "Control-plane outage" below.+- `control_plane_unavailable`: HTTP 503. No control-plane process answered+ the redeem; see "Control-plane outage" below.- `invalid_request`: the request body failed the transport schema (HTTP 400before redeem), or the upstream answered the completion with HTTP 400.@@ -524,24 +524,47 @@ from upstream outages, and the log line carries the limit's fixed `reason`.### Control-plane outagecontrol-api runs one replica with a `Recreate` strategy, so a rollout or a-cluster update leaves it with no ready endpoint for a few seconds; the same-happens when `control-api-rpc-gateway` restarts. The proxy answers-`control_plane_unavailable` only when the redeem certainly reached no live-control-plane process:+cluster update leaves it with no ready endpoint; the same happens when+`control-api-rpc-gateway` restarts. In the minikube check of #720 a real+restart left control-api without an endpoint for about 15 s, after about 30 s+in which the terminating pod still answered. The proxy answers+`control_plane_unavailable` only when no control-plane process produced a+response to the redeem:- the redeem `fetch` rejected before any response, the 15 s request timeouthad not fired, and the undici cause code is `ECONNREFUSED`, `ENOTFOUND`,- `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`. A- Service with no ready endpoint refuses the connection;-- the redeem was denied with JSON `{"error":"control_plane_unavailable"}`,- which passes through like any other control-api code.--These stay `provider_unavailable`, because control-api may have received the-request: `ECONNRESET` or another socket error after the request was sent, the-15 s timeout, a non-JSON 502/503/504 from a gateway, and a JSON+ `EAI_AGAIN`, `EHOSTUNREACH`, `ENETUNREACH` or `UND_ERR_CONNECT_TIMEOUT`.+ The connection is refused only in the short window where the endpoint still+ lists a pod that has exited. Once the endpoint is gone the SYN is dropped,+ and undici gives up after its 10 s connect timeout with+ `UND_ERR_CONNECT_TIMEOUT`;+- the rpc gateway answered JSON `{"error":"control_plane_unavailable"}`. The+ gateway writes that body itself, from an `error_page 502` named location,+ when nginx generated the 502: the connection to control-api was refused, or+ control-api closed it or sent an invalid header before a response. In the+ last two cases control-api may already have processed the redeem, so the+ code means that no response came back, not that the request never arrived.+ The body passes through the proxy's JSON-`error` branch like a code+ control-api returns.++These stay `provider_unavailable`: `ECONNRESET` or another socket error after+the request was sent, the 15 s timeout, a non-JSON 502/503/504 from a gateway+(the rpc gateway's connect timeout is nginx's default 60 s, so during an+outage the proxy's 15 s timeout usually fires first), and a JSON`provider_unavailable`. Finalize failures do not change: they are logged andnever reach the caller.+Deployment order: an mcp-host from before #720 has no arm for+`control_plane_unavailable` and classifies it as `LLM_API_CALL_FAILED`, not+retryable and without failover, where the same outage used to be a retryable+`provider_unavailable`. Roll out the mcp-host images first, including the one+HCC sets as `CONTEXT_MAPPER_HOST_IMAGE`, and wait until every Host pod runs+the new image; then roll out the proxies and the control-plane ConfigMap,+restarting `control-api-rpc-gateway` and `nginx-workflow-approval-gateway`,+which mount `nginx.conf` through `subPath`. `rate_limited` and+`upstream_rejected` need no order: an old Host already classifies them as a+new one does, without `httpStatus`.+The client logs `grok_proxy_control_api_unreachable` with `path` only. The`grok_proxy_attempt_finished` line carries `causeCode` whenever the failureis a rejected `fetch`: the undici cause code, such as `ECONNREFUSED`, anddiff --git a/grok-llm-proxy/src/server.ts b/grok-llm-proxy/src/server.tsindex c5f710e74..a2dcd38dc 100644--- a/grok-llm-proxy/src/server.ts+++ b/grok-llm-proxy/src/server.ts@@ -513,7 +513,8 @@ export function createProxyApps(res.removeHeader('cache-control')res.setHeader('content-type', 'application/json; charset=utf-8')// G1-1: an upstream Retry-After reaches the Host only on this path; an- // SSE error frame carries the code alone.+ // SSE error frame carries the code and, for upstream_rejected, the+ // upstream status, never Retry-After.const retryAfter =err instanceof GrokTransportError ? err.details?.retryAfterSeconds : undefinedif (typeof retryAfter === 'number') res.setHeader('retry-after', String(retryAfter))@@ -679,7 +680,8 @@ const ATTEMPT_ERROR_STATUS: Record<string, number> = {// An upstream 4xx the same request would get again (G1-3, #720). Not 502:// the gateways answer 502 when nothing behind them answered (R1-H2).upstream_rejected: 422,- // A control-api call no live process received (G1-4, #720).+ // A redeem no control-plane process answered (G1-4, #720); control-api may+ // still have received it when the gateway's 502 mapping produced the code.control_plane_unavailable: 503,tool_call_arguments_exceeded: 422,invalid_tool_arguments: 422,