Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs(grok): document visual requests and the live image probe
Alfredo Lopez846ad8001744
diff --git a/docs/architecture/grok-subscription-transport-contract.md b/docs/architecture/grok-subscription-transport-contract.mdindex 019237faa..288c028e8 100644--- a/docs/architecture/grok-subscription-transport-contract.md+++ b/docs/architecture/grok-subscription-transport-contract.md@@ -4,6 +4,136 @@ Starting freeze: `grok-subscription-transport.v1`.This is the Phase 0 architecture freeze for provider `grok-subscription`.+## Visual requests (issue #784)++The subscription endpoint `cli-chat-proxy.grok.com/v1/responses` is+**unmeasured for images**. Every limit and wire shape in this section comes+from the xAI API documentation for `api.x.ai/v1` (docs.x.ai, read+2026-09-23), not from a call to the subscription endpoint. No Grok+subscription was available to probe it. The live probe in+[`docs/testing/grok-subscription-validation.md`](../testing/grok-subscription-validation.md)+is the check that closes this gap; until it runs, a passing conformance suite+certifies only what Evenfire sends, not what the endpoint accepts.++Request shape. Schema `grok-completion-request.v2` adds one message field,+`contentParts`, on `user` messages only. Parts are ordered text parts or+inline image parts. An image part carries `mimeType` (`image/png` or+`image/jpeg`), canonical base64 `data`, and a closed `source` identity:+`{ kind: 'attachment', attachmentId, messageId }` or+`{ kind: 'tool', attachmentId, toolCallId }`. The text parts joined with a+newline must equal the message's `content`. The source is part of the+canonical request hash and never reaches the upstream. No URL is accepted. A+request with no parts is still sent as V1, with its existing bytes and hash.+The part structure, the source union and the image validator are copies of+the Codex V2 contract (issue #650); the Grok contract package does not import+the Codex one, and a parity test runs one image corpus through both copies.++Limits, from the xAI documentation (grok-4.5, 4.6 and 4.7 list+"Text, Image → Text"):++- at most 20 images per request (xAI sets no count limit; 20 matches the+ shared ingress);+- at most 20 MiB (20971520 bytes) decoded per image;+- at most 20 MiB decoded across all images of the request, whole history+ included;+- JPEG and PNG only. WebP and GIF are refused.++There is **no dimension or pixel limit**. The validator reads the PNG or+JPEG header to check the container, and never refuses on width or height; a+declared 9000×9000 PNG is accepted and reaches the upstream. This differs from+Codex, whose 2048 px bound comes from its measured endpoint. The validator+checks canonical base64, MIME/container framing and header dimensions; it does+not decode pixels.++Envelope. `LIMITS.maxVisualRequestBodyBytes` is 36700160 (35 MiB): the+encoded image budget (`4 × ceil(20971520 / 3)` = 27962028 bytes) plus the+8 MiB non-image share, rounded up to a whole MiB. It is the cap for the whole+serialized V2 request and its HTTP envelope, signed ticket included, with no+envelope allowance on top, as for Codex. V1 keeps `maxRequestBodyBytes` plus+the 16 KiB `ENVELOPE_ALLOWANCE_BYTES`. Text, tools and other non-image fields+of a V2 request stay bounded to `maxRequestBodyBytes`, measured with the image+data blanked in a temporary projection. A request over 35 MiB is reachable+only with images already over their 20 MiB budget, so the image refusal fires+first. The value is a runtime limit, not a published fixture limit: the+`grok-llm-proxy/test/contractFreeze.test.ts` fixture describes the measured+upstream, and the upstream image limits are not measured.++Where the envelope is enforced:++- the Host builds the V2 proxy envelope with `buildGrokProxyEnvelope` before+ it streams, and its authorizer bounds the authorize body by the Grok+ contract, not the Codex one;+- the control-api authorize route parses up to 36700160 bytes (the larger of+ the Codex and Grok visual caps), and the authorizer builds the exact V2+ proxy envelope inside its transaction after signing and before commit, so+ an envelope over the cap rolls back the attempt, the ticket and the+ reservation;+- the workflow-approval gateway's authorize location has+ `client_max_body_size 36700160`;+- `grok-llm-proxy` reads a V2 body with a separate parser whose limit is+ `GROK_LLM_PROXY_MAX_VISUAL_BODY_BYTES`, 36700160 in the base manifest; the+ proxy refuses to start when it is configured below the contract cap.++V2 has no outer deadline. Its deadline is `request.deadlineMs`, part of the+authorized hash. The Host refuses to stream with a deadline other than the+request's, and the proxy refuses a V2 envelope that carries an outer+`deadlineMs`.++Proxy admission. A request whose declared `Content-Length` is above the+ordinary cap needs a valid platform identity, then takes a slot in a visual+stream gate of 1 running and 4 queued (`VISUAL_STREAM_LIMITS`) before the body+is read, so queued visual bodies are not held in memory. Visual bodies do not+take the ordinary in-flight byte budget. Chunked bodies stay on the ordinary+path, and a declared length above the visual cap is refused 413 before+reading. Anonymous, wrong-scope and admin requests keep the ordinary limit.+The visual wait uses the request's single admission clock and ends at the+ticket's `exp`, so a ticket that expires while a visual body waits yields+`ticket_expired`.++Memory. Measured on macOS (Node v24.18.0, tsc build, one process, heap capped+at 384 MiB, upstream calls through undici): #739's D5 load (eight 8 MiB streams+held and three 8 MiB bodies queued) peaked at 511 MiB, and D5 plus one+36.3 MB V2 stream in the visual gate peaked at 775.4 MiB. The manifest+therefore sets a 1Gi limit and a 768Mi request, the Codex shape, and+`grok-llm-proxy/test/deployManifest.test.ts` requires the limit to be at least+1.25 × the peak. The in-cluster peak is not measured.++Upstream projection. The proxy maps a user message with parts to+`content: [{ type: 'input_text', text }, { type: 'input_image', image_url:+'data:<mime>;base64,<data>', detail: 'high' }]` in the original order.+`detail` is always `high`. The Responses shape is the one the xAI+documentation gives for `api.x.ai/v1`.++Errors:++- An image over its per-image or total budget, or a V2 body over 35 MiB, is+ refused by the Host before authorize as `attachment_too_large`, mapped to+ `LLM_INVALID_ATTACHMENT`, not retryable. The user sees messages such as+ "An attached image is too large: it exceeds 20 MiB." No provider attempt is+ spent.+- More than 20 images is `invalid_request` at the Host, control-api (HTTP 400)+ and the proxy.+- An image part whose source identity is missing or malformed is+ `image_source_invalid` at the Host, mapped to `LLM_API_CALL_FAILED`, not+ retryable.+- A non-image share over 8 MiB is `request_limit_exceeded` at the Host+ ("Conversation Too Long"). control-api answers any size refusal with HTTP+ 413 `payload_too_large`, for V1 as well as V2 (V1 was HTTP 400+ `invalid_request` before #784), and the proxy refuses size with 413+ `payload_too_large`. The Host maps `payload_too_large` to+ `LLM_CONTEXT_LENGTH_EXCEEDED`, not retryable.+- Images are never stripped silently and never trigger a provider fallback.++Desktop. The composer budget for `grok-subscription` is 16 MiB per image and+16 MiB decoded in total, with no dimension bound. That is the shared rpc-proxy+and Host ingress cap for one chat message, not the xAI limit. Only tool+screenshots, which do not cross that ingress, can reach the contract's+20 MiB per image.++Enablement. There is no image-specific flag. Image input follows the existing+Grok flags, and tool-screenshot source identity is on because the Grok+descriptor sets `requiresImageSourceIdentity`.+## Probe gateThe following facts are a starting freeze taken from Grok Build docs and@@ -159,10 +289,13 @@ All three enforcement points read this module — the control-api authorizer,`grok-llm-proxy` and the Host — so a deployment that mixes versions rejectsrequests that fall between the old and the new bounds. Which code the callersees depends on where the rejection happens: the control-api authorizer and-the proxy both surface the contract parser's failure as `invalid_request`,-while the Host raises `request_limit_exceeded` before it authorizes at all —-for the four size refusals listed under that code below, and `invalid_request`-for the other four, which no amount of compaction would fix.+the proxy both surface a size refusal (`kind: 'size'`) as HTTP 413+`payload_too_large` and every other parser failure as `invalid_request`+(#784; the control-api authorizer answered `invalid_request` for size too+before it), while the Host raises `request_limit_exceeded` before it+authorizes at all — for the four size refusals listed under that code below,+and `invalid_request` for the other four, which no amount of compaction would+fix.That symmetry holds for a request and not for a response, which is why therollout order below is not interchangeable. A proxy carrying the new bound in@@ -293,7 +426,8 @@ Stable codes: `insufficient_scope`, `no_grant`, `model_not_allowed`,`tool_call_arguments_exceeded`, `invalid_tool_arguments`, `invalid_request`,`sse_buffer_exceeded`, `stream_duration_exceeded`, `payload_too_large`,`context_length_exceeded`, `request_limit_exceeded` (Host-side only),-`request_timeout`, `length_required`, `ticket_expired`,+`attachment_too_large` (Host-side only), `image_source_invalid` (Host-side+only), `request_timeout`, `length_required`, `ticket_expired`,`unsupported_media_type`, `unknown_field`, `host_binding_mismatch`, `disabled`,`Unauthorized`, `not_found`, `internal_error`. The freeze gate checks that everycode the proxy constructs, and every code it refuses a request with@@ -468,6 +602,20 @@ code the proxy constructs, and every code it refuses a request withauthorize raises it too, and so does the proxy's 413, with or without aJSON body. The Host maps it to `LLM_CONTEXT_LENGTH_EXCEEDED`, notretryable. No provider attempt is spent when authorize refused it.+- `attachment_too_large` (Host-side only, #784): an image broke the+ per-image budget (20 MiB decoded), the request's total image budget+ (20 MiB decoded) or the 35 MiB V2 envelope. The Host raises it before+ authorization and maps it to `LLM_INVALID_ATTACHMENT`, not retryable and not+ failover-eligible. Compaction cannot shrink an image, so it is never+ labelled a context-length failure. The table is built from the Grok+ contract limits by `buildAttachmentBudgetRefusals`+ (`mcp-host/src/llm/attachmentBudgetRefusal.ts`), the builder the Codex+ provider also uses; Grok has no dimension or pixel row because its contract+ never emits those refusals. The image-count refusal stays `invalid_request`.+- `image_source_invalid` (Host-side only, #784): an image part has no valid+ `attachment` or `tool` source identity, which means a Host producer defect.+ The Host refuses it before authorization and maps it to+ `LLM_API_CALL_FAILED`, not retryable.- `context_length_exceeded`: the upstream refused a prompt over the model'scontext window. Recorded on 2026-09-23 against `/v1/responses` with`grok-4.6`, the refusal is an HTTP 400 before any stream starts, with the@@ -486,9 +634,11 @@ code the proxy constructs, and every code it refuses a request withallowlist is `ATTEMPT_ERROR_STATUS` in `grok-llm-proxy/src/server.ts` — thesame table that maps a code to its HTTP status — not the list above. The twodiffer: the table also carries two control-api codes the list does not publish-(`invalid_receipt`, `conflict`), and it leaves out the two Host-side codes above-and the proxy's own request refusals (`payload_too_large`, `length_required`,+(`invalid_receipt`, `conflict`), and it leaves out the Host-side codes above+and the proxy's own request refusals (`length_required`,`unsupported_media_type`, `unknown_field`, `not_found`, `internal_error`).+`payload_too_large` is in the table (HTTP 413) since #784, because the+transport now raises it for a contract size refusal.Anything outside the table is recorded as `other`, because a control-api errorbody is not bounded by the proxy. The raw code stays in the`grok_proxy_attempt_finished` log line.diff --git a/docs/testing/grok-subscription-validation.md b/docs/testing/grok-subscription-validation.mdnew file mode 100644index 000000000..94eef8075--- /dev/null+++ b/docs/testing/grok-subscription-validation.md@@ -0,0 +1,89 @@+# Grok subscription validation++Evidence lanes for `grok-subscription` image input (issue #784). One lane does+not stand in for another.++## Purpose++`cli-chat-proxy.grok.com/v1/responses`, the endpoint `grok-llm-proxy` calls,+is **unmeasured for images**. The limits and the wire shape in+[the transport contract](../architecture/grok-subscription-transport-contract.md#visual-requests-issue-784)+come from the xAI API documentation for `api.x.ai/v1` (docs.x.ai, read+2026-09-23): 20 MiB per image, JPEG and PNG, any text/image order, and+`{ "type": "input_image", "image_url": "data:<mime>;base64,<data>", "detail": "high" }`+next to `input_text` parts. The contract, proxy, control-api, Host and Desktop+suites prove what Evenfire sends and refuses. None of them proves that the+subscription endpoint accepts that body or that the model sees the pixels.++The live probe below is the only check that does. Run it when a Grok+subscription is available. Until it has run, report the endpoint as+unmeasured for images.++## Preconditions++- Separate authorization for the run: it calls the real upstream and spends+ subscription quota.+- A branch-owned local profile, bootstrapped by the canonical Minikube entry+ points, with the images built from the commit under test. Do not touch+ another branch's profile or port-forwards.+- `GROK_LLM_PROXY_EXECUTION_ENABLED` on, a connected Grok grant, and a Host+ whose Grok projection mints `llm:grok:execute` (HCC then injects+ `MCP_HOST_GROK_SUBSCRIPTION_ENABLED`). There is no image-specific flag.+- A catalog model that the xAI documentation lists with image input+ (grok-4.5, grok-4.6 or grok-4.7). Record the canonical model id the runtime+ returns, not an alias.+- Never read `~/.grok/auth.json`, and never read the `api_key` fields of+ `~/.grok/models_cache.json`. The probe runs through the real+ Host → control-api → `grok-llm-proxy` path and needs neither.++## Steps++Send every request through the Host, from the Desktop composer or as a tool+screenshot, never by calling the upstream directly. Use a fresh 64-bit+hexadecimal challenge per image, rendered only into the pixels. The generators+in `desktop-app/test/e2e-playwright/codexImageChallenge.ts`+(`challengeImage`, `paddedChallengeImage`, `challengeImageAt`) produce such+images; the filename and the prompt must not carry the answer.++1. PNG challenge. One small PNG. Ask the model for the hexadecimal code in+ the image.+2. JPEG challenge. The same with a JPEG.+3. Order. One user message of text, image, text, where the answer depends on+ the order (for example: "the code in the image, then the word after it").+4. Limits. Each case in its own request:+ - one image padded close to the ingress cap of 16 MiB decoded from the+ composer, and one tool screenshot close to 20 MiB decoded, the contract's+ per-image limit, which only tool screenshots can reach;+ - 20 small images in one request, each with its own challenge;+ - one PNG whose long side is 9000 px. The contract sets no pixel limit, so+ this case measures whether the upstream does.++## Expected++- Steps 1 to 3: the answer contains the challenge (hexadecimal case ignored),+ the turn ends without an error, no tool step substitutes OCR or file loading+ for direct image input, and no fallback badge appears.+- Step 4: every case is accepted by Evenfire, since each is inside the+ contract. Whatever the upstream answers is the measurement. An upstream+ refusal is a limit mismatch between the xAI documentation and the+ subscription endpoint. Report it, and change the contract only in a new+ pull request that cites the recorded status.+- A local refusal in any step (`attachment_too_large`, `invalid_request`,+ `image_source_invalid`, HTTP 413) is a defect in Evenfire, not an upstream+ measurement.++## Recording++For each step record:++- the commit, the profile, the model id and the date;+- the image format, the decoded size in bytes, the dimensions and the image+ count;+- the upstream HTTP status and, on failure, the bounded `upstreamHint` from+ the proxy's `grok_upstream_http` log line and the error code from its+ `grok_proxy_attempt_finished` line;+- whether the answer contained the challenge.++Never record tickets, tokens, grant credentials or image bytes. Put the+results in the pull request or issue that closes the measurement, and update+the "unmeasured" statement in the transport contract only with that evidence.