Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
Codex subscription validation
Evidence lanes for codex-subscription. One lane does not stand in for another.
Image input validation (#650)
The shared package tests V1 golden-hash stability and V2 parsing, source identity,
ordered text/image content and local image/envelope budgets. Its
fixtures/visual-requests.json contains real 2x2 PNG/JPEG samples;
fixtures/canonical-request-hashes.v2.json freezes the new projection separately.
The image parser validates container structure and dimensions, not decoded pixels.
Capacity checks cover the usual 5 / 9 / 14 MiB target and the hard 16 MiB
per-image / 16 MiB aggregate / 24 MiB envelope ceiling, plus the 2048 px
model bound. Exercise 5 MiB, 12 MiB (exceptional, above 10 MiB), 10+5 MiB
and 3x5 MiB acceptance, plus overflow above 16 MiB. V1 and V2 non-image fields keep an 8 MiB ceiling
(#731), and beside a hard-ceiling image the 24 MiB envelope leaves about 2.7 MiB of non-image data;
chat text and envelope bytes stay at 6 MiB, while qualifying kind:'file' attachments (at most 11 MiB decoded each)
have their own 16 MiB base64 quota under the same 24 MiB body; unrelated rpc-proxy and Host control routes keep the 10 MB ordinary JSON cap. The larger proxy parser must not admit an
anonymous, wrong-scope or admin request, or enlarge V1's effective body limit.
Changing these budgets does not change OAuth, grants, ticket binding, origins,
fallback, or connection selection. Local capacity evidence is separate from
successful image interpretation by a real model.
For an already bootstrapped local profile whose required public images are
present, MINIKUBE_BUILD_IMAGE_ARGS=--skip-public forwards the existing builder
option through the public Make entry point. This avoids pulling unrelated demo
images during a full source rebuild. Keep the normal image verification and
runtime readiness gates; this option is not a substitute for bootstrap.
When reconciling an existing profile configured through Control UI, use
MINIKUBE_REAPPLY_INSTANCES=false to preserve its Host and other bootstrap
instance configuration. This does not skip core manifests, CRDs, image checks,
or readiness. Leave the default enabled for initial bootstrap. Verify the
Host's provider/model/connection binding before and after the update.
Provider tests must cover both completion methods, direct/tool origins, prompt context, redacted text parts and repeated bytes from distinct tool calls. Proxy conformance inspects the final upstream body, asserts provenance does not become model input, and retains tools/cancel/SSE checks. Authorizer tests require a pre-commit exact-envelope check; its PostgreSQL case must prove rollback against a real transaction. A mocked transaction result does not satisfy that PostgreSQL gate.
Visual input is on by default for every codex-subscription model. No test
fixture or passing conformance suite certifies that the real endpoint or
selected model sees images.
Keep browser/Electron evidence separate: visible login and Host selection, upload through Add context / Upload Files, preview, send, correlated task completion and visible answer. Assert a business signal tied to the submitted image and current task; an answer saying it saw an image is insufficient. Do not use SDK text-only journeys or GFS file upload as substitutes for composer image input. Run the E2E static auditor on changed browser specs and record external prerequisites rather than silently skipping them.
Opt-in Desktop image lane
desktop-app/test/e2e-playwright/codex-image-input.spec.ts covers direct PNG/JPEG
uploads through the visible composer, including a 5 MiB JPEG, a 12 MiB PNG
(exceptional), a 10+5 MiB pair, three 5 MiB images, and composer refusals over
16 MiB per image, a 21st image, and a non-PNG/JPEG type. The 16 MiB aggregate is
enforced on the hop, not the composer. It creates a fresh 64-bit hexadecimal
challenge rendered only into image pixels. Large cases pad that same image so
the answer stays in pixels while the decoded size matches the hop budget. The
filename and prompt do not carry the answer. A passing run requires that
answer (hexadecimal case is ignored), a non-error response, no tool steps
substituting OCR/loading for direct image input, and no fallback badge.
The selector's stable data attributes verify the actual selected Host, provider
and model. Composer budget and resolution refusals never send.
This is a real upstream lane, not a mock. Before running it, obtain separate authorization for runtime, upstream access and the existing login fixture's stored-session reset. Provision the owned Host and consumers first. Set:
E2E_CODEX_IMAGE_INPUT=1andCODEX_REAL_UPSTREAM_CONFIRM=1only for that authorized run;E2E_CODEX_ALLOW_SESSION_RESET=1acknowledges the fixture's session reset.E2E_HOST_REFto the actual owned Host andE2E_CODEX_HOST_LABELto its exact visible label in the Agents list; there is no shared-host default.E2E_CODEX_IMAGE_MODELandE2E_CODEX_IMAGE_MODEL_LABELto its model ID and visible picker label. The test does not change deployments or grants.
Install Host dependencies as well as Desktop dependencies: the neutral challenge
uses the existing Host canvas dependency. Node 24 and verify:electron remain
mandatory. After loading the approved branch-owned service URLs and prerequisites,
the command from desktop-app is:
node node_modules/@playwright/test/cli.js test \
--config test/e2e-playwright/playwright.codex-image.config.ts --project codex-image-input
The normal Desktop project excludes this spec. Its dedicated config rejects missing authority or Host/model identity before global setup; it cannot return a successful all-skipped run. The model ID must be the canonical ID returned by the runtime, not an alias. Static audit/typecheck and local challenge-image decoding do not count as browser or real-upstream execution. Typed tool-result images remain a separate integration/runtime journey; this direct-input test does not certify GFS producers or retained image history.
Lanes
- T0 — unit, schema, hash, contracts, typecheck, lint, build. The
aggregator is
make test-codex-subscription-t0. It fails when a required suite is missing, executes zero tests, reports skipped/todo cases, or exits non-zero. Flag-off compatibility still usesmake test-unit-all,make test-contracts, andmake build-preflight. - T1 — real PostgreSQL.
CONTROL_API_REAL_PG_REQUIRED=1. A missing DSN, skipped suite, or zero tests is FAIL. - Manifest — Kustomize, images, ServiceAccount/RBAC, secrets, gateways, NetworkPolicy. No cluster.
- T2 — exact-HEAD on a branch-owned Minikube profile. Full reconcile for this change (CRDs, manifests, policies, image matrix).
- Browser/Electron — visible login and journeys. Node 24 and
npm run verify:electronfirst. - Upstream real — opt-in only with
CODEX_REAL_UPSTREAM_CONFIRM=1and an approved account. Never satisfied by a mock. - Load/compromise — limits, backpressure, cancel, and contained pivot probes.
Freeze fixture
tests/e2e/fixtures/codex-subscription/sanitized-upstream-contract.json and
docs/architecture/codex-subscription-transport-contract.md are the Phase 0
contract. The freeze test is
tests/e2e/integration/codex-subscription-contract-freeze.test.ts.
Commands
make test-codex-subscription-t0
cd tests/e2e
npm test -- --run integration/codex-subscription-contract-freeze.test.ts
Live upstream is the Playwright connection lane (codex-subscription-connection.spec.ts)
against a signed-in Control UI. There is no standalone
e2e-codex-subscription-real-upstream.sh script.
Subscription catalog reconciliation cron
control-api/src/services/subscriptionCatalogSyncCron.ts re-reads both
subscription catalogs on an interval, so a model an upstream publishes after a
grant's handshake reaches the runtime allowlist without an operator pressing
anything. It is off by default: SUBSCRIPTION_CATALOG_SYNC_CRON_ENABLED and
SUBSCRIPTION_CATALOG_SYNC_INTERVAL_MS in
deploy/base/control-plane/configmaps.yaml carry the deployment's choice, and
the interval has a 15-minute floor because each tick costs one upstream call
per connection and a Grok refresh rotates the refresh token.
Two suites, and neither stands in for the other:
- Unit —
control-api/src/services/__tests__/subscriptionCatalogSyncCron.test.ts. Owns the tick's decision table against doubles: the advisory lock, the broker/status skips, per-connection failure isolation, thenever_syncedrule that decides whether the ConfigMap is republished, and the first-run jitter. - T1 —
control-api/test/services.subscriptionCatalogSyncCron.realPostgres.integration.test.ts, underCONTROL_API_REAL_PG_REQUIRED=1. Owns what doubles cannot reach: rows landing ingrok_catalog_models, thegrok-subscriptionunion rebuilt inllm_allowed_models, a model that vanished upstream stale-flagged in the catalog and disabled in the union, and an identical second tick recording a fresh outcome without duplicating rows. The grant is created through the device flow rather than by writing credential rows, so the tick's own refresh-token rotation runs against a fingerprint it can match. Only xAI's HTTP endpoints and the Kubernetes ConfigMap writer are doubled.
Run the T1 suite the way the lane does, against a real PostgreSQL 16:
cd control-api && CONTROL_API_REAL_PG_ADMIN_URL="$DSN" CONTROL_API_REAL_PG_REQUIRED=1 npx vitest run --no-file-parallelism --maxWorkers=1 realPostgres
On-demand catalog re-sync
The same re-read is available to an operator from the grant modal's Sync catalog button. Three lanes own three different claims about it, and none substitutes for another.
- Proxy, hermetic —
codex-llm-proxy/test/runtimePath.hermetic.e2e.test.ts, describehermetic catalog re-read. Owns "the re-read returns a model the upstream published after the handshake". This is the only place in the repository where that is provable:CODEX_CATALOG_ORIGINis frozen incodex-llm-proxy/src/originPolicy.tsas part of the SSRF boundary, and the in-processfetchFn/lookupseam in that file is the one thing that redirects it at the fixture. The fixture serves a second model whenCODEX_TEST_UPSTREAM_EXTRA_MODELis set; unset keeps the single-model reply every other case expects. The fixture'smodelscounter is the liveness witness. - T1 — the cron suite above. Owns the rows and the union allowlist.
- Control UI Playwright —
tests/e2e/playwright/control-ui/codex-subscription-catalog-resync.spec.ts. Owns the affordance and the operator's verdict: the button exists on a connected grant and nowhere else, the click issues exactly onePOST .../catalog/sync, a 200 advancescatalogRevision, and the modal ends on a success toast or a non-empty error banner rather than a spinner. It cannot prove a new model appears: that lane deploys no fake upstream, so the grant talks to the frozen origin.
Plugin Workload SDK Desktop lanes
Three Desktop/Electron journeys exercise codex-subscription through the
Plugin Workload SDK. Each has its own spec and its own Make target, because
their preconditions are mutually exclusive. None of them starts Minikube or
creates grants — the T2/T3 profile lane must have provisioned those already.
Happy path
E2E_PLUGIN_SDK_WRITE_CONFIRM=1 \
E2E_PLUGIN_SDK_EXPECT_PROVIDER=codex-subscription \
CONTROL_UI_BASE_URL=http://127.0.0.1:<random-control-ui-port> \
CONTROL_API_BASE_URL=http://127.0.0.1:<random-control-api-port> \
EXTERNAL_REST_API_BASE_URL=http://127.0.0.1:<random-external-rest-port> \
RPC_PROXY_BASE_URL=http://127.0.0.1:<random-rpc-port> \
KUBECONTEXT=<branch-profile-context> \
make test-e2e-plugin-workload-sdk-desktop
E2E_PLUGIN_SDK_EXPECT_PROVIDER is mandatory and has no default. It
declares which provider the run is meant to exercise, and the precondition
asserts the live recipe declares exactly that one. Without it the spec used to
accept openai, claude or codex-subscription and skip its Codex assertions
when the recipe was not Codex — so a run against the OpenAI recipe that
seed-e2e-data.sh provisions by default went green while proving nothing about
Codex. Declare the provider you intend to evidence; a mismatch now fails the
precondition instead of quietly narrowing the run.
No-grant guard lane
plugin-workload-sdk-no-grant-guard.spec.ts is the regression test for the
symptom of issue #533: a Codex recipe whose execution binding is missing must
report awaiting_policy, never validated. It is mutually exclusive with the
happy path — that one requires the recipe to reach validated, this one
requires it not to.
E2E_PLUGIN_SDK_WRITE_CONFIRM=1 \
E2E_PLUGIN_SDK_NO_GRANT_RECIPE_NAME=<codex-recipe-without-binding> \
E2E_PLUGIN_SDK_NO_GRANT_APP_TITLE=<its-app-title> \
CONTROL_UI_BASE_URL=http://127.0.0.1:<random-control-ui-port> \
CONTROL_API_BASE_URL=http://127.0.0.1:<random-control-api-port> \
EXTERNAL_REST_API_BASE_URL=http://127.0.0.1:<random-external-rest-port> \
RPC_PROXY_BASE_URL=http://127.0.0.1:<random-rpc-port> \
KUBECONTEXT=<branch-profile-context> \
make test-e2e-plugin-workload-sdk-desktop-no-grant
Both variables above are mandatory: the spec has no default recipe and fails
loudly when either is missing, and it refuses to run against the happy path's
recipe even if pointed at it. E2E_PLUGIN_SDK_NO_GRANT_RECIPE_NAMESPACE is the
one optional knob and defaults to sandbox-recipes — set it explicitly when the
ungranted fixture lives elsewhere, or the lane will look for the right recipe
name in the wrong namespace. This target takes no
E2E_PLUGIN_SDK_EXPECT_PROVIDER: no provider is supposed to be reached, so
there is none to declare. Provisioning a Codex recipe with no execution
binding is operator work — the lane will not create one, and a lane that
cannot find its fixture fails rather than skipping.
Codex fallback lane
plugin-workload-sdk-codex-fallback.spec.ts closes acceptance criterion 8 of
issue #533: a real journey where an eligible Codex failure reaches the
authorized non-Codex fallback. It runs against the granted happy-path
recipe, because the fallback needs a working ordered target list.
E2E_PLUGIN_SDK_WRITE_CONFIRM=1 \
E2E_PLUGIN_SDK_EXPECT_PROVIDER=codex-subscription \
CONTROL_UI_BASE_URL=http://127.0.0.1:<random-control-ui-port> \
CONTROL_API_BASE_URL=http://127.0.0.1:<random-control-api-port> \
EXTERNAL_REST_API_BASE_URL=http://127.0.0.1:<random-external-rest-port> \
RPC_PROXY_BASE_URL=http://127.0.0.1:<random-rpc-port> \
KUBECONTEXT=<branch-profile-context> \
make test-e2e-plugin-workload-sdk-desktop-codex-fallback
This lane mutates cluster state. It scales control-plane/codex-llm-proxy
to zero replicas so the primary target fails with a class the failover engine
accepts, then restores it to one. The restore runs twice — in the spec's
finally and again in a Make-level EXIT trap — and both report loudly on
failure. If you ever see [E2E-GUARD] FAILED to restore control-plane/codex-llm-proxy, restore it before running any other lane on
that profile: every Codex journey after it would fail for the wrong reason.
E2E_PLUGIN_SDK_EXPECT_PROVIDER must be exactly codex-subscription; the
target rejects any other value, because a non-Codex primary would make the
journey vacuous. The recipe knobs are shared with the happy path and carry
silent defaults — E2E_PLUGIN_SDK_RECIPE_NAME defaults to
evenfire-prompt-notify-app, E2E_PLUGIN_SDK_RECIPE_NAMESPACE to
sandbox-recipes, and E2E_PLUGIN_SDK_APP_TITLE to Prompt & Notify. Set
them explicitly whenever your fixture differs, or the lane looks for the wrong
app in the Desktop catalog and fails at the card locator rather than at a
precondition.
The grant must expose at least two ordered prompt targets, Codex first and a non-Codex provider second. The spec asserts that before injecting the fault: a single-target grant would let the journey pass while proving nothing, so it fails as a precondition instead.