Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(grok): document visual requests and the live image probe

Alfredo Lopez846ad8001744
diff --git a/docs/architecture/grok-subscription-transport-contract.md b/docs/architecture/grok-subscription-transport-contract.md
index 019237faa..288c028e8 100644
--- a/docs/architecture/grok-subscription-transport-contract.md
+++ b/docs/architecture/grok-subscription-transport-contract.md
@@ -4,6 +4,136 @@ Starting freeze: `grok-subscription-transport.v1`.
This is the Phase 0 architecture freeze for provider `grok-subscription`.
+## Visual requests (issue #784)
+
+The subscription endpoint `cli-chat-proxy.grok.com/v1/responses` is
+**unmeasured for images**. Every limit and wire shape in this section comes
+from the xAI API documentation for `api.x.ai/v1` (docs.x.ai, read
+2026-09-23), not from a call to the subscription endpoint. No Grok
+subscription was available to probe it. The live probe in
+[`docs/testing/grok-subscription-validation.md`](../testing/grok-subscription-validation.md)
+is the check that closes this gap; until it runs, a passing conformance suite
+certifies only what Evenfire sends, not what the endpoint accepts.
+
+Request shape. Schema `grok-completion-request.v2` adds one message field,
+`contentParts`, on `user` messages only. Parts are ordered text parts or
+inline image parts. An image part carries `mimeType` (`image/png` or
+`image/jpeg`), canonical base64 `data`, and a closed `source` identity:
+`{ kind: 'attachment', attachmentId, messageId }` or
+`{ kind: 'tool', attachmentId, toolCallId }`. The text parts joined with a
+newline must equal the message's `content`. The source is part of the
+canonical request hash and never reaches the upstream. No URL is accepted. A
+request with no parts is still sent as V1, with its existing bytes and hash.
+The part structure, the source union and the image validator are copies of
+the Codex V2 contract (issue #650); the Grok contract package does not import
+the Codex one, and a parity test runs one image corpus through both copies.
+
+Limits, from the xAI documentation (grok-4.5, 4.6 and 4.7 list
+"Text, Image → Text"):
+
+- at most 20 images per request (xAI sets no count limit; 20 matches the
+ shared ingress);
+- at most 20 MiB (20971520 bytes) decoded per image;
+- at most 20 MiB decoded across all images of the request, whole history
+ included;
+- JPEG and PNG only. WebP and GIF are refused.
+
+There is **no dimension or pixel limit**. The validator reads the PNG or
+JPEG header to check the container, and never refuses on width or height; a
+declared 9000×9000 PNG is accepted and reaches the upstream. This differs from
+Codex, whose 2048 px bound comes from its measured endpoint. The validator
+checks canonical base64, MIME/container framing and header dimensions; it does
+not decode pixels.
+
+Envelope. `LIMITS.maxVisualRequestBodyBytes` is 36700160 (35 MiB): the
+encoded image budget (`4 × ceil(20971520 / 3)` = 27962028 bytes) plus the
+8 MiB non-image share, rounded up to a whole MiB. It is the cap for the whole
+serialized V2 request and its HTTP envelope, signed ticket included, with no
+envelope allowance on top, as for Codex. V1 keeps `maxRequestBodyBytes` plus
+the 16 KiB `ENVELOPE_ALLOWANCE_BYTES`. Text, tools and other non-image fields
+of a V2 request stay bounded to `maxRequestBodyBytes`, measured with the image
+data blanked in a temporary projection. A request over 35 MiB is reachable
+only with images already over their 20 MiB budget, so the image refusal fires
+first. The value is a runtime limit, not a published fixture limit: the
+`grok-llm-proxy/test/contractFreeze.test.ts` fixture describes the measured
+upstream, and the upstream image limits are not measured.
+
+Where the envelope is enforced:
+
+- the Host builds the V2 proxy envelope with `buildGrokProxyEnvelope` before
+ it streams, and its authorizer bounds the authorize body by the Grok
+ contract, not the Codex one;
+- the control-api authorize route parses up to 36700160 bytes (the larger of
+ the Codex and Grok visual caps), and the authorizer builds the exact V2
+ proxy envelope inside its transaction after signing and before commit, so
+ an envelope over the cap rolls back the attempt, the ticket and the
+ reservation;
+- the workflow-approval gateway's authorize location has
+ `client_max_body_size 36700160`;
+- `grok-llm-proxy` reads a V2 body with a separate parser whose limit is
+ `GROK_LLM_PROXY_MAX_VISUAL_BODY_BYTES`, 36700160 in the base manifest; the
+ proxy refuses to start when it is configured below the contract cap.
+
+V2 has no outer deadline. Its deadline is `request.deadlineMs`, part of the
+authorized hash. The Host refuses to stream with a deadline other than the
+request's, and the proxy refuses a V2 envelope that carries an outer
+`deadlineMs`.
+
+Proxy admission. A request whose declared `Content-Length` is above the
+ordinary cap needs a valid platform identity, then takes a slot in a visual
+stream gate of 1 running and 4 queued (`VISUAL_STREAM_LIMITS`) before the body
+is read, so queued visual bodies are not held in memory. Visual bodies do not
+take the ordinary in-flight byte budget. Chunked bodies stay on the ordinary
+path, and a declared length above the visual cap is refused 413 before
+reading. Anonymous, wrong-scope and admin requests keep the ordinary limit.
+The visual wait uses the request's single admission clock and ends at the
+ticket's `exp`, so a ticket that expires while a visual body waits yields
+`ticket_expired`.
+
+Memory. Measured on macOS (Node v24.18.0, tsc build, one process, heap capped
+at 384 MiB, upstream calls through undici): #739's D5 load (eight 8 MiB streams
+held and three 8 MiB bodies queued) peaked at 511 MiB, and D5 plus one
+36.3 MB V2 stream in the visual gate peaked at 775.4 MiB. The manifest
+therefore sets a 1Gi limit and a 768Mi request, the Codex shape, and
+`grok-llm-proxy/test/deployManifest.test.ts` requires the limit to be at least
+1.25 × the peak. The in-cluster peak is not measured.
+
+Upstream projection. The proxy maps a user message with parts to
+`content: [{ type: 'input_text', text }, { type: 'input_image', image_url:
+'data:<mime>;base64,<data>', detail: 'high' }]` in the original order.
+`detail` is always `high`. The Responses shape is the one the xAI
+documentation gives for `api.x.ai/v1`.
+
+Errors:
+
+- An image over its per-image or total budget, or a V2 body over 35 MiB, is
+ refused by the Host before authorize as `attachment_too_large`, mapped to
+ `LLM_INVALID_ATTACHMENT`, not retryable. The user sees messages such as
+ "An attached image is too large: it exceeds 20 MiB." No provider attempt is
+ spent.
+- More than 20 images is `invalid_request` at the Host, control-api (HTTP 400)
+ and the proxy.
+- An image part whose source identity is missing or malformed is
+ `image_source_invalid` at the Host, mapped to `LLM_API_CALL_FAILED`, not
+ retryable.
+- A non-image share over 8 MiB is `request_limit_exceeded` at the Host
+ ("Conversation Too Long"). control-api answers any size refusal with HTTP
+ 413 `payload_too_large`, for V1 as well as V2 (V1 was HTTP 400
+ `invalid_request` before #784), and the proxy refuses size with 413
+ `payload_too_large`. The Host maps `payload_too_large` to
+ `LLM_CONTEXT_LENGTH_EXCEEDED`, not retryable.
+- Images are never stripped silently and never trigger a provider fallback.
+
+Desktop. The composer budget for `grok-subscription` is 16 MiB per image and
+16 MiB decoded in total, with no dimension bound. That is the shared rpc-proxy
+and Host ingress cap for one chat message, not the xAI limit. Only tool
+screenshots, which do not cross that ingress, can reach the contract's
+20 MiB per image.
+
+Enablement. There is no image-specific flag. Image input follows the existing
+Grok flags, and tool-screenshot source identity is on because the Grok
+descriptor sets `requiresImageSourceIdentity`.
+
## Probe gate
The following facts are a starting freeze taken from Grok Build docs and
@@ -159,10 +289,13 @@ All three enforcement points read this module — the control-api authorizer,
`grok-llm-proxy` and the Host — so a deployment that mixes versions rejects
requests that fall between the old and the new bounds. Which code the caller
sees depends on where the rejection happens: the control-api authorizer and
-the proxy both surface the contract parser's failure as `invalid_request`,
-while the Host raises `request_limit_exceeded` before it authorizes at all —
-for the four size refusals listed under that code below, and `invalid_request`
-for the other four, which no amount of compaction would fix.
+the proxy both surface a size refusal (`kind: 'size'`) as HTTP 413
+`payload_too_large` and every other parser failure as `invalid_request`
+(#784; the control-api authorizer answered `invalid_request` for size too
+before it), while the Host raises `request_limit_exceeded` before it
+authorizes at all — for the four size refusals listed under that code below,
+and `invalid_request` for the other four, which no amount of compaction would
+fix.
That symmetry holds for a request and not for a response, which is why the
rollout order below is not interchangeable. A proxy carrying the new bound in
@@ -293,7 +426,8 @@ Stable codes: `insufficient_scope`, `no_grant`, `model_not_allowed`,
`tool_call_arguments_exceeded`, `invalid_tool_arguments`, `invalid_request`,
`sse_buffer_exceeded`, `stream_duration_exceeded`, `payload_too_large`,
`context_length_exceeded`, `request_limit_exceeded` (Host-side only),
-`request_timeout`, `length_required`, `ticket_expired`,
+`attachment_too_large` (Host-side only), `image_source_invalid` (Host-side
+only), `request_timeout`, `length_required`, `ticket_expired`,
`unsupported_media_type`, `unknown_field`, `host_binding_mismatch`, `disabled`,
`Unauthorized`, `not_found`, `internal_error`. The freeze gate checks that every
code the proxy constructs, and every code it refuses a request with
@@ -468,6 +602,20 @@ code the proxy constructs, and every code it refuses a request with
authorize raises it too, and so does the proxy's 413, with or without a
JSON body. The Host maps it to `LLM_CONTEXT_LENGTH_EXCEEDED`, not
retryable. No provider attempt is spent when authorize refused it.
+- `attachment_too_large` (Host-side only, #784): an image broke the
+ per-image budget (20 MiB decoded), the request's total image budget
+ (20 MiB decoded) or the 35 MiB V2 envelope. The Host raises it before
+ authorization and maps it to `LLM_INVALID_ATTACHMENT`, not retryable and not
+ failover-eligible. Compaction cannot shrink an image, so it is never
+ labelled a context-length failure. The table is built from the Grok
+ contract limits by `buildAttachmentBudgetRefusals`
+ (`mcp-host/src/llm/attachmentBudgetRefusal.ts`), the builder the Codex
+ provider also uses; Grok has no dimension or pixel row because its contract
+ never emits those refusals. The image-count refusal stays `invalid_request`.
+- `image_source_invalid` (Host-side only, #784): an image part has no valid
+ `attachment` or `tool` source identity, which means a Host producer defect.
+ The Host refuses it before authorization and maps it to
+ `LLM_API_CALL_FAILED`, not retryable.
- `context_length_exceeded`: the upstream refused a prompt over the model's
context window. Recorded on 2026-09-23 against `/v1/responses` with
`grok-4.6`, the refusal is an HTTP 400 before any stream starts, with the
@@ -486,9 +634,11 @@ code the proxy constructs, and every code it refuses a request with
allowlist is `ATTEMPT_ERROR_STATUS` in `grok-llm-proxy/src/server.ts` — the
same table that maps a code to its HTTP status — not the list above. The two
differ: the table also carries two control-api codes the list does not publish
-(`invalid_receipt`, `conflict`), and it leaves out the two Host-side codes above
-and the proxy's own request refusals (`payload_too_large`, `length_required`,
+(`invalid_receipt`, `conflict`), and it leaves out the Host-side codes above
+and the proxy's own request refusals (`length_required`,
`unsupported_media_type`, `unknown_field`, `not_found`, `internal_error`).
+`payload_too_large` is in the table (HTTP 413) since #784, because the
+transport now raises it for a contract size refusal.
Anything outside the table is recorded as `other`, because a control-api error
body is not bounded by the proxy. The raw code stays in the
`grok_proxy_attempt_finished` log line.
diff --git a/docs/testing/grok-subscription-validation.md b/docs/testing/grok-subscription-validation.md
new file mode 100644
index 000000000..94eef8075
--- /dev/null
+++ b/docs/testing/grok-subscription-validation.md
@@ -0,0 +1,89 @@
+# Grok subscription validation
+
+Evidence lanes for `grok-subscription` image input (issue #784). One lane does
+not stand in for another.
+
+## Purpose
+
+`cli-chat-proxy.grok.com/v1/responses`, the endpoint `grok-llm-proxy` calls,
+is **unmeasured for images**. The limits and the wire shape in
+[the transport contract](../architecture/grok-subscription-transport-contract.md#visual-requests-issue-784)
+come from the xAI API documentation for `api.x.ai/v1` (docs.x.ai, read
+2026-09-23): 20 MiB per image, JPEG and PNG, any text/image order, and
+`{ "type": "input_image", "image_url": "data:<mime>;base64,<data>", "detail": "high" }`
+next to `input_text` parts. The contract, proxy, control-api, Host and Desktop
+suites prove what Evenfire sends and refuses. None of them proves that the
+subscription endpoint accepts that body or that the model sees the pixels.
+
+The live probe below is the only check that does. Run it when a Grok
+subscription is available. Until it has run, report the endpoint as
+unmeasured for images.
+
+## Preconditions
+
+- Separate authorization for the run: it calls the real upstream and spends
+ subscription quota.
+- A branch-owned local profile, bootstrapped by the canonical Minikube entry
+ points, with the images built from the commit under test. Do not touch
+ another branch's profile or port-forwards.
+- `GROK_LLM_PROXY_EXECUTION_ENABLED` on, a connected Grok grant, and a Host
+ whose Grok projection mints `llm:grok:execute` (HCC then injects
+ `MCP_HOST_GROK_SUBSCRIPTION_ENABLED`). There is no image-specific flag.
+- A catalog model that the xAI documentation lists with image input
+ (grok-4.5, grok-4.6 or grok-4.7). Record the canonical model id the runtime
+ returns, not an alias.
+- Never read `~/.grok/auth.json`, and never read the `api_key` fields of
+ `~/.grok/models_cache.json`. The probe runs through the real
+ Host → control-api → `grok-llm-proxy` path and needs neither.
+
+## Steps
+
+Send every request through the Host, from the Desktop composer or as a tool
+screenshot, never by calling the upstream directly. Use a fresh 64-bit
+hexadecimal challenge per image, rendered only into the pixels. The generators
+in `desktop-app/test/e2e-playwright/codexImageChallenge.ts`
+(`challengeImage`, `paddedChallengeImage`, `challengeImageAt`) produce such
+images; the filename and the prompt must not carry the answer.
+
+1. PNG challenge. One small PNG. Ask the model for the hexadecimal code in
+ the image.
+2. JPEG challenge. The same with a JPEG.
+3. Order. One user message of text, image, text, where the answer depends on
+ the order (for example: "the code in the image, then the word after it").
+4. Limits. Each case in its own request:
+ - one image padded close to the ingress cap of 16 MiB decoded from the
+ composer, and one tool screenshot close to 20 MiB decoded, the contract's
+ per-image limit, which only tool screenshots can reach;
+ - 20 small images in one request, each with its own challenge;
+ - one PNG whose long side is 9000 px. The contract sets no pixel limit, so
+ this case measures whether the upstream does.
+
+## Expected
+
+- Steps 1 to 3: the answer contains the challenge (hexadecimal case ignored),
+ the turn ends without an error, no tool step substitutes OCR or file loading
+ for direct image input, and no fallback badge appears.
+- Step 4: every case is accepted by Evenfire, since each is inside the
+ contract. Whatever the upstream answers is the measurement. An upstream
+ refusal is a limit mismatch between the xAI documentation and the
+ subscription endpoint. Report it, and change the contract only in a new
+ pull request that cites the recorded status.
+- A local refusal in any step (`attachment_too_large`, `invalid_request`,
+ `image_source_invalid`, HTTP 413) is a defect in Evenfire, not an upstream
+ measurement.
+
+## Recording
+
+For each step record:
+
+- the commit, the profile, the model id and the date;
+- the image format, the decoded size in bytes, the dimensions and the image
+ count;
+- the upstream HTTP status and, on failure, the bounded `upstreamHint` from
+ the proxy's `grok_upstream_http` log line and the error code from its
+ `grok_proxy_attempt_finished` line;
+- whether the answer contained the challenge.
+
+Never record tickets, tokens, grant credentials or image bytes. Put the
+results in the pull request or issue that closes the measurement, and update
+the "unmeasured" statement in the transport contract only with that evidence.