Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

test: guard subscription T0 lanes and document Grok rollout

Jose Gabriel Ramirez Encinasd0ee13909921
diff --git a/.github/workflows/ci-public.yml b/.github/workflows/ci-public.yml
index 88835d502..bc1e52655 100644
--- a/.github/workflows/ci-public.yml
+++ b/.github/workflows/ci-public.yml
@@ -264,7 +264,10 @@ jobs:
pluginWorkloadSdkCodexDualLedger.realPostgres.integration \
pluginWorkloadSdkGrokDualLedger.realPostgres.integration \
services.grokSubscriptionConnection.realPostgres.integration \
- services.grokProviderAttemptRedemption.realPostgres.integration
+ services.grokProviderAttemptRedemption.realPostgres.integration \
+ services.grokProviderAttemptRedemption.refresh.realPostgres.integration \
+ services.codexSubscriptionLifecycle.realPostgres.integration \
+ db.llmProviderAttemptConnectionIntegrity.realPostgres.integration
do
grep -Fq "$suite" "$normalized_log" || {
echo "real-Postgres control-api lane omitted Codex suite ${suite}" >&2
diff --git a/docs/architecture/codex-subscription-transport-contract.md b/docs/architecture/codex-subscription-transport-contract.md
index 5ce49ad09..c22645472 100644
--- a/docs/architecture/codex-subscription-transport-contract.md
+++ b/docs/architecture/codex-subscription-transport-contract.md
@@ -148,6 +148,69 @@ Deployment readiness checks after a combined apply do not enforce the ordering
above. The infrastructure rollout barrier remains a separate release dependency;
this document alone does not certify mixed-version deployment safety.
+### Codex changes shipped with the Grok broker
+
+The Grok subscription broker (see
+`docs/architecture/grok-subscription-transport-contract.md`) shares the
+contract, authorizer, proxy robustness and recipe-grant code with Codex. The V1
+wire and hash projection are unchanged. Operators should know about these Codex
+behavior changes:
+
+- **Revoke and reconnect.** Codex connection keys stay reusable after revoke:
+ `deployment-default` and named keys can be revoked and then reconnected with
+ a new OAuth flow.
+ - Revoke also cancels every pending browser and device OAuth state for that
+ key, atomically in the same statement.
+ - A flow that started before the revoke cannot persist a grant. It fails
+ with `state_cancelled`.
+ - When two first grants race on the same key, the losing active-key insert
+ maps to a stale-revision conflict instead of a 500.
+- **Attempt integrity (migration 0114).** A constraint trigger requires a
+ non-null `llm_provider_attempts.connection_id` of a Codex attempt to exist in
+ `codex_subscription_connections`. This replaces the FK that 0112 dropped.
+ Historic rows are not revalidated, and the migration is forward-only.
+- **Catalog bounds.** Catalog sync stores at most 256 discovered models, each
+ with an id of at most 128 characters.
+ - Extra entries are dropped with a count-only warning.
+ - The catalog record, row mutations and union rebuild commit in one
+ transaction.
+ - `codex-llm-proxy` admin catalog and connection-test calls have a 15 s
+ deadline and an 8 MiB streamed body cap.
+- **Request identity.** `mcp-host` hashes with `hashCanonicalCodexRequest(raw)`
+ (parse, then hash the canonical value), so empty `generation`, `tools` or
+ `transportHints` objects hash the same as on the server. Contract trees are
+ capped at a nesting depth of 64. The authorizer rejects deeper bodies with
+ `invalid_request`.
+- **Terminal outcomes.** Only `success` is a completion. `canceled`, `error`,
+ and `unknown` with partial text or tool calls are errors. The proxy maps an
+ unrecognized finalize outcome to `unknown`.
+- **Proxy robustness.**
+ - `codex-llm-proxy` fails at startup when its control-api URL or service
+ token is empty.
+ - It respects SSE write backpressure.
+ - It drops queued stream-gate waiters on abort and checks the abort signal
+ before redeeming a ticket.
+ - It requires `maxStreamDurationMs` greater than 0.
+- **Live-target attestation.** Codex authorize attests the live Host or recipe
+ target. The allowed providers come only from the spec's model, allowed
+ models and fallbacks (Hosts) or agent providers (recipes). A target that
+ reached Codex only through a grant annotation or `connectionRef` now gets
+ `host_binding_mismatch`.
+- **Recipe grant annotations.** control-api now writes both
+ `codex-connection-ref` and `subscription-connection-ref` for Codex recipes.
+ It rejects explicit disagreeing pairs with 422
+ `subscriptionAnnotationsDisagree`. Broker changes that omit the annotations
+ return 422 `providerChangeRequiresGrant`.
+- **Rollout.** Roll out control-api first, and wait until every pod runs the
+ new image. An older control-api rejects the entire workflow-control token
+ issue when HCC or WRC request the unknown `llm:grok:execute` scope.
+- **Rollback.** Turn the Grok flags off and wait for HCC and WRC to drop
+ `llm:grok:execute` before rolling back control-api. While an older
+ control-api serves, a reassigned Codex recipe grant updates only
+ `codex-connection-ref`. The new WRC then reads the disagreeing pair as
+ `unassigned`. After rolling forward again, re-assign those Codex recipe
+ grants with explicit annotations.
+
## Errors
Stable codes: `insufficient_scope`, `no_grant`, `model_not_allowed`,
diff --git a/docs/architecture/grok-subscription-transport-contract.md b/docs/architecture/grok-subscription-transport-contract.md
index 85162162e..986a30c1d 100644
--- a/docs/architecture/grok-subscription-transport-contract.md
+++ b/docs/architecture/grok-subscription-transport-contract.md
@@ -3,9 +3,31 @@
Starting freeze: `grok-subscription-transport.v1`.
This is the Phase 0 architecture freeze for provider `grok-subscription`.
-Catalog origin and CLI identity headers remain probe-gated (see
-`docs/superpowers/specs/2026-09-15-llm-coding-plan-subscriptions-design.md`
-§9 and R2-M3). Live SuperGrok confirm may tighten, never widen, this allowlist.
+
+## Probe gate
+
+The following facts are a starting freeze taken from Grok Build docs and
+third-party probes. No live SuperGrok probe has confirmed them yet:
+
+- the catalog origin,
+- the CLI identity headers,
+- the numeric limits,
+- the public CLI `client_id`,
+- the OAuth scope list,
+- whether the upstream accepts `temperature`.
+
+Rule: before the production flag flip, commit a redacted probe record beside
+this document. Regenerate the sanitized fixture
+(`tests/e2e/fixtures/grok-subscription/sanitized-upstream-contract.json`) from
+that record. The freeze test must assert that the fixture matches the enforced
+origin-policy constants. A recorded `GROK_REAL_UPSTREAM_CONFIRM=1` run must also
+exist.
+
+A live probe may tighten this allowlist but must never widen it. If the probe
+finds a different catalog endpoint, or no list API at all, the origin may be
+replaced once. After that, it cannot be widened. Until the probe record exists,
+the fixture's `approvedTestAccountFingerprint` stays
+`redacted:grok-subscription-pending-live-confirm`.
## Ownership
@@ -22,13 +44,61 @@ Exact HTTPS origins (no caller-supplied URL):
- Device authorization: `https://auth.x.ai/oauth2/device/code`
- Token: `https://auth.x.ai/oauth2/token`
+- Revoke: `https://auth.x.ai/oauth2/revoke`
- Completions: `https://cli-chat-proxy.grok.com/v1/responses`
- Catalog (starting candidate): `https://cli-chat-proxy.grok.com/v1/models`
Forbidden: `https://api.x.ai`, HTTP, loopback, link-local, metadata.
Redirects: at most one, HTTPS, same origin and same path and same query.
-OAuth `device/code` and `token` use `redirect: 'manual'`; any 3xx is an error.
+OAuth `device/code`, `token` and `revoke` use `redirect: 'manual'`; any 3xx is
+an error.
+
+## OAuth
+
+- `client_id`: `b1a00492-073a-47ea-816f-4c329264a828`. This is the public
+ Grok CLI registration, not a secret. `CONTROL_API_GROK_OAUTH_CLIENT_ID` may
+ override it, but only to pin a documented registration.
+- Grant: `urn:ietf:params:oauth:grant-type:device_code`.
+- Scopes (`GROK_OAUTH_SCOPES` in
+ `control-api/src/services/grokSubscriptionOAuth.ts`): `openid`, `profile`,
+ `email`, `offline_access`, `grok-cli:access`, `api:access`,
+ `conversations:read`, `conversations:write`.
+ - This list is **pending the live probe trim**. `api:access` and
+ `conversations:read/write` are broader than inference needs: a leaked
+ access token could read or write Grok conversations. The probe must record
+ the minimum set the CLI proxy accepts, and the list must shrink to that
+ set before the production flag flip.
+- The refresh token rotates on every refresh. The new ciphertext is persisted
+ before any fallible post-processing.
+- Revoke revokes the refresh token upstream. Access tokens that were already
+ issued stay valid until they expire (at most one access-token lifetime).
+
+## Connection lifecycle
+
+- **Grok connection keys are terminal after revoke (migration 0113).**
+ - A revoked key cannot be reconnected. Create a new connection key instead.
+ - A device flow that started before the revoke cannot revive the key: it
+ fails with `revoked grant cannot be reused`.
+ - 0113 archives superseded revoked rows for the same key as
+ `<key>~revoked~<id>`. It also cancels pending OAuth states for revoked
+ keys and replaces the partial active-key index with a unique index on
+ `connection_key`.
+ - Archived `~revoked~` rows fall outside the key grammar. Admin list and GET
+ hide them. The key's own terminal tombstone stays listed as `revoked`.
+- **Attempt integrity (migration 0114).** A constraint trigger on
+ `llm_provider_attempts` INSERT, and on UPDATE of `provider` or
+ `connection_id`, requires a non-null `connection_id` to exist in the
+ provider's own connection table:
+ - `grok-subscription` → `grok_subscription_connections`
+ - `codex-subscription` → `codex_subscription_connections`
+
+ This replaces the FK that 0112 dropped, so it covers Codex too. It does not
+ revalidate historic rows: the migration logs only a count of dangling rows.
+- Catalog sync stores at most 256 discovered models, each with an id of at most
+ 128 characters. Extra entries are dropped with a structured count-only
+ warning. The catalog record, row mutations and union rebuild commit in one
+ transaction.
## Ticket
@@ -36,11 +106,37 @@ OAuth `device/code` and `token` use `redirect: 'manual'`; any 3xx is an error.
- `aud`: `grok-llm-proxy`
- Admin permit `typ`/`aud`: `grok-admin-permit` / `grok-llm-proxy-admin`
+## Request identity
+
+- `mcp-host` hashes the request with `hashCanonicalGrokRequest(raw)` from
+ `@clerum/grok-provider-attempt-contract`. This function parses the request
+ first and then hashes the parsed canonical value, so empty `generation: {}`,
+ `tools: []` or `transportHints: {}` hash the same on the client and the
+ server. Codex uses the twin `hashCanonicalCodexRequest`.
+- Contract trees are capped at a nesting depth of 64 (`fail('limit')`). The
+ control-api authorizer rejects deeper bodies with `invalid_request` before it
+ calls `JSON.stringify`.
+
## Limits
Owned by `@clerum/grok-provider-attempt-contract`. Independent of Codex
`LIMITS` (`maxToolCalls` is 64, not 32).
+Proxy robustness (both proxies):
+
+- Admin catalog and connection-test upstream calls have a 15 s deadline that
+ covers headers and body, and a streamed body cap: 1 MiB for Grok, 8 MiB for
+ Codex. Normalized models are capped at 256, and each id at 128 characters,
+ matching control-api.
+- The SSE writer respects `res.write` backpressure: it waits for `drain` and
+ stops waiting if the client closes.
+- Queued stream-gate waiters stop when the request aborts, and the proxy
+ checks the abort signal before redeeming a ticket.
+- The proxy fails at startup when `GROK_LLM_PROXY_CONTROL_API_URL` or
+ `GROK_LLM_PROXY_CONTROL_API_TOKEN` is empty.
+- An unrecognized finalize outcome maps to `unknown`, never `success`.
+ `maxStreamDurationMs` must be greater than 0.
+
## Identity headers
Stamped in `grok-llm-proxy`, never taken from mcp-host or the hashed request:
@@ -50,24 +146,130 @@ Stamped in `grok-llm-proxy`, never taken from mcp-host or the hashed request:
CLI impersonation headers (`x-xai-token-auth`, `x-grok-client-identifier`) stay
off until a recorded SuperGrok probe requires them. Do not send `OpenAI-Beta`
-or `service_tier`. `max_output_tokens` is bindable on this wire.
+or `service_tier`.
+
+`max_output_tokens` is bindable on this wire and is sent.
+
+`temperature` is **withheld upstream** while
+`GROK_UPSTREAM_TEMPERATURE_PROBE_CONFIRMED` (in
+`grok-llm-proxy/src/grokTransport.ts`) is `false`. The Codex Responses wire
+rejects sampling fields, and no live probe has shown that Grok accepts them.
+`generation.temperature` still counts toward the authorize hash. Set the
+constant to `true` only with recorded live-probe evidence.
Tool payloads send `strict: false` so optional MCP fields stay optional on the
Responses-shaped Grok wire (sibling of Codex #648). Agent tool presentation
uses the same `CODEX_TOOL_PRESENTATION` knob as Codex and defaults to `direct`
when Grok is the primary or a fallback (sibling of Codex #644).
+## Terminal outcomes
+
+For both providers, a stream counts as a completion only when it ends with
+`success`. `mcp-host` reports these outcomes as errors, never as a completion:
+
+- `canceled`,
+- `error`,
+- `unknown` with partial text or tool calls.
+
+## Feature flags
+
+All flags default to off.
+
+| Flag | Owner | Gates |
+| --- | --- | --- |
+| `CONTROL_API_GROK_SUBSCRIPTION_ENABLED` | control-api ConfigMap | Admin Grok routes, Host/recipe acceptance of `grok-subscription`, and the `clerum.io/grok-enabled` annotation that control-api publishes on `clerum-llm-allowed-models`. |
+| `WRC_GROK_SUBSCRIPTION_ENABLED` | workflow-recipes Deployment | Folded into the recipe verdict. Gates the `llm:grok:execute` scope, the `<recipe>-mcp-host-to-grok-proxy` egress NetworkPolicy, the bootstrap binding, the pod env, and `grantRedeemable`. |
+| `GROK_LLM_PROXY_EXECUTION_ENABLED` | grok-llm-proxy ConfigMap | Kill switch: while it is off, runtime and admin requests return 404 `disabled`, even with a valid JWT, ticket or permit. |
+| `MCP_HOST_GROK_SUBSCRIPTION_ENABLED` | **injected, not a manual key** | HCC injects it on Host pods whose Grok projection mints `llm:grok:execute`. WRC injects it, with `GROK_LLM_PROXY_RUNTIME_URL`, on recipe pods that declare Grok when `WRC_GROK_SUBSCRIPTION_ENABLED` is on. Do not add it to `mcp-host-config`. |
+
+HCC has no local Grok switch. Its Grok projection follows the
+`clerum.io/grok-enabled` annotation together with the Host's own Grok
+references.
+
+## Recipe grant annotations
+
+Annotation write shapes:
+
+- Codex: `{codex-connection-ref: k, subscription-connection-ref: k}`
+- Grok: `{codex-connection-ref: '', subscription-connection-ref: k}`
+
+The server (`control-api/src/services/recipeGrantTransition.ts`) enforces
+these transitions. None of them may return a 500:
+
+| Case | Behavior |
+| --- | --- |
+| Explicit alias and canonical both non-empty and unequal (any spec) | 422 `subscriptionAnnotationsDisagree` |
+| Create, or static → broker | Omitted annotations: 422 (grant required). Explicit: validate, then write the broker's shape. |
+| Same broker, annotations omitted | Keep the stored annotations |
+| Same broker, annotations explicit | Validate the key, then write that broker's shape |
+| Codex → Grok | Omitted annotations: 422 `providerChangeRequiresGrant`. Non-empty alias: 422. Otherwise validate the canonical key and write the Grok shape. |
+| Grok → Codex | Omitted annotations: 422 `providerChangeRequiresGrant`. Otherwise validate (canonical-only is allowed) and write the Codex shape. Never clear silently. |
+| Broker → static | Without an SDK: clear both annotations. With an SDK: keep them (the SDK route owns them). |
+| Static → static | Keep the stored annotations (the SDK route owns static-recipe identity) |
+
## Allowlist ConfigMap republish
-control-api is `replicas: 1`. After rolling a new control-api that writes
+control-api is `replicas: 1`. After rolling out a new control-api that writes
`clerum.io/grok-*` annotations, republish `clerum-llm-allowed-models` once the
-new pod is Ready so a mutation on the old pod cannot strip Grok annotations.
+new pod is Ready. Otherwise a mutation served by the old pod could strip the
+Grok annotations. A new control-api also republishes on boot.
+
+## Rollout order
+
+1. **evenfire-infra first.** Pin the `grok-llm-proxy` image for `gcp-dev` and
+ `gcp-prod`. Provision `grok-llm-proxy-secrets`
+ (`GROK_LLM_PROXY_CONTROL_API_TOKEN`) with
+ `deploy/scripts/apply-inter-service-tokens.sh`. Both must be in place before
+ the merge to `dev` triggers the automatic deploy.
+2. **control-api** with migrations 0109–0114. Wait until every control-api pod
+ runs the new image. An old control-api rejects the whole workflow-control
+ token issue when the request includes the unknown `llm:grok:execute` scope
+ (`invalid_workflow_control_scopes`). HCC or WRC deployed first could
+ therefore break token issuance for every pod they mint for, not only Grok
+ pods (C-RP-016).
+3. **HCC, WRC, mcp-host and grok-llm-proxy.** Grok flags stay off.
+4. Republish `clerum-llm-allowed-models` (see above).
+5. Turn on the flags only after the control-api rollout is complete:
+ `CONTROL_API_GROK_SUBSCRIPTION_ENABLED`, `WRC_GROK_SUBSCRIPTION_ENABLED` and
+ `GROK_LLM_PROXY_EXECUTION_ENABLED`. `MCP_HOST_GROK_SUBSCRIPTION_ENABLED`
+ follows automatically through HCC and WRC.
-Rollback: drop `llm:grok:execute` and Grok proxy NetworkPolicies in HCC/WRC
-before rolling control-api back. A new control-api republishes on boot.
+## Rollback order
+
+1. Turn the Grok flags off. control-api republishes
+ `clerum.io/grok-enabled=false`.
+2. Wait for HCC and WRC to reconcile, so that no Host or recipe pod still mints
+ `llm:grok:execute` and the Grok proxy NetworkPolicies are gone. Only then
+ roll back control-api. An old control-api rejects the entire token issue
+ for the unknown scope (C-RP-016).
+3. After a control-api rollback, the old control-api writes only
+ `codex-connection-ref` when a Codex recipe grant is reassigned. The stale
+ `subscription-connection-ref` then disagrees with it. The new WRC reads the
+ recipe as `unassigned`, and authorize returns a mismatch (B-M5b).
+ - Once a control-api that writes both keys is serving again, list the Codex
+ recipes whose two annotations are both non-empty and unequal.
+ - Re-assign each grant with explicit annotations, which write the Codex
+ shape. In the UI, re-pick the connection.
+ - A save that omits the annotations keeps the stored disagreeing pair, so
+ it does not repair the grant.
+4. Migrations 0113 (terminal Grok keys) and 0114 (attempt connection
+ integrity) are forward-only. An older control-api still runs against them:
+ - The 0114 trigger only rejects connection ids that do not exist.
+ - Under 0113, a revoked Grok key can no longer be reconnected by the old
+ code path. That attempt fails on the unique key index instead of reviving
+ the key.
## SDK bootstrap
-Grok writers emit `subscriptionBinding` only. Readers accept `subscriptionBinding`
-or `codexBinding`. Grok missing-binding reason is `execution_binding_missing`.
-Codex keeps `codexBinding` and `codex_execution_binding_missing`.
+Binding slots are strict:
+
+- Grok writers emit `subscriptionBinding`, and Grok readers (mcp-host
+ `bootstrapIdentity`, WRC provisioner and bootstrap proof parsers) read only
+ `subscriptionBinding`.
+- Codex writes and reads only `codexBinding`. A proof in the other slot is
+ ignored.
+
+Missing-binding reasons:
+
+- Grok: `execution_binding_missing`
+- Codex: `codex_execution_binding_missing`
diff --git a/scripts/tests/test-codex-subscription-t0.sh b/scripts/tests/test-codex-subscription-t0.sh
index 25594c85f..2a2db3224 100755
--- a/scripts/tests/test-codex-subscription-t0.sh
+++ b/scripts/tests/test-codex-subscription-t0.sh
@@ -4,6 +4,9 @@
# Fails when a required suite is missing, executes zero tests, reports
# skipped/todo cases, or exits non-zero. Exit 0 alone is never enough: the
# script parses machine-readable counts from Vitest or node:test.
+#
+# Lane separation: Grok-only suites run in test-grok-subscription-t0.sh, never
+# here. Provider-neutral suites shared by both brokers may appear in both.
set -u
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
@@ -221,17 +224,17 @@ run_group "control-api" "control-api" \
"test/routes.adminRecipes.test.ts" \
"test/routes.adminPluginWorkloadSdk.test.ts" \
"test/services.recipeCodexGrantIdentity.test.ts" \
+ "test/services.recipeGrantTransition.test.ts" \
"test/crd.llmProviderEnums.test.ts" \
"test/hostSpecValidation.codexSubscription.test.ts" \
- "test/hostSpecValidation.grokSubscription.test.ts" \
"test/llmProviders.test.ts" \
"test/services.codexSubscriptionConnection.test.ts" \
"test/services.llmAllowedModelsConfigMap.test.ts" \
"test/services.codexSubscriptionOAuth.test.ts" \
- "test/services.grokSubscriptionOAuth.test.ts" \
"test/services.codexSubscriptionCatalog.test.ts" \
+ "test/services.subscriptionCatalogBounds.test.ts" \
"test/services.llmProviderAttemptAuthorizer.test.ts" \
- "test/services.llmProviderAttemptAuthorizer.grok.test.ts" \
+ "test/services.llmProviderAttemptAuthorizer.depth.test.ts" \
"test/services.llmProviderAttemptTicket.test.ts" \
"test/services.llmProviderAttemptRedemption.test.ts" \
"test/services.llmProviderAttemptFinalization.test.ts" \
@@ -244,11 +247,14 @@ run_group "control-api" "control-api" \
run_group "codex-llm-proxy" "codex-llm-proxy" \
"test/approvedToolsUpstream.test.ts" \
+ "test/catalogBounds.test.ts" \
"test/codexTransport.conformance.test.ts" \
"test/controlApiClient.test.ts" \
"test/originPolicy.test.ts" \
"test/redaction.test.ts" \
- "test/server.security.test.ts"
+ "test/requestLimits.test.ts" \
+ "test/server.security.test.ts" \
+ "test/sseBackpressure.test.ts"
run_group "mcp-host" "mcp-host" \
"src/capabilities/toolCatalogTools.test.ts" \
@@ -263,6 +269,7 @@ run_group "mcp-host" "mcp-host" \
"src/agent/__tests__/taskExecutor.test.ts" \
"src/llm/__tests__/codexSubscription.test.ts" \
"src/llm/__tests__/codexLlmProxyClient.test.ts" \
+ "src/llm/__tests__/subscriptionRequestHash.test.ts" \
"src/llm/__tests__/providerAttemptAuthorizer.test.ts" \
"src/llm/hostLlmBinding.test.ts" \
"src/config/configStore.test.ts" \
@@ -291,9 +298,7 @@ run_group "workflow-recipes" "workflow-recipes" \
"src/workflow/llmAllowedModelsSnapshot.test.ts" \
"src/workflow/networkPolicyFactory.codex.test.ts" \
"src/workflow/sdkOnlyCodexBinding.test.ts" \
- "src/workflow/sdkOnlyGrokBinding.test.ts" \
"src/workflow/pluginWorkloadSdkProvisioner.codexPolicy.test.ts" \
- "src/workflow/pluginWorkloadSdkProvisioner.grokPolicy.test.ts" \
"src/reconciler/pluginWorkloadSdkValidator.test.ts" \
"tests/unit/workflow/modelConfigHandler.test.ts" \
"tests/unit/workflow/modelConfigHandler.pluginSdkBroker.test.ts"
diff --git a/scripts/tests/test-grok-subscription-t0.sh b/scripts/tests/test-grok-subscription-t0.sh
index cc2f526fd..cf43e81c4 100755
--- a/scripts/tests/test-grok-subscription-t0.sh
+++ b/scripts/tests/test-grok-subscription-t0.sh
@@ -1,12 +1,23 @@
#!/usr/bin/env bash
# Wave-2 Grok subscription T0 aggregator.
+#
# Fails when a required suite is missing, executes zero tests, reports
-# skipped/todo cases, or exits non-zero.
+# skipped/todo cases, runs fewer Vitest files than listed, or exits non-zero.
+# Exit 0 alone is never enough: the script parses machine-readable counts from
+# Vitest or node:test. Every listed suite is guarded by require_file before its
+# group runs, because `vitest run <present> <missing>` still exits 0.
+#
+# Lane separation: Codex-only suites live in test-codex-subscription-t0.sh.
+# Real-Postgres suites need CONTROL_API_REAL_PG_ADMIN_URL and run in the CI
+# real-PG lane; this script checks they exist and that the lane lists them.
set -u
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
FAIL=0
GROUPS_RUN=0
+EXPECTED_GROUPS=9
+REGISTERED=()
+COUNT_SUMMARY=""
pass() { echo "PASS: $1"; }
fail() { echo "FAIL: $1"; FAIL=1; }
@@ -24,20 +35,32 @@ strip_ansi() {
sed $'s/\033\\[[0-9;]*m//g'
}
+# Return 0 when the captured reporter proves at least one passing test, zero
+# skipped/todo cases and, for Vitest, exactly the expected number of files.
+# Sets COUNT_SUMMARY on success.
assert_executed_counts() {
local name="$1"
local log="$2"
+ local expected_files="$3"
local normalized
normalized="$(mktemp)"
strip_ansi <"$log" >"$normalized"
local vitest_passed
vitest_passed="$(
- grep -Eo 'Tests[[:space:]]+[1-9][0-9]* passed' "$normalized" | head -n 1 || true
+ grep -Eo 'Tests[[:space:]]+[1-9][0-9]* passed' "$normalized" | head -n 1 | awk '{print $2}' || true
+ )"
+ local vitest_files
+ vitest_files="$(
+ grep -Eo 'Test Files[[:space:]]+[0-9]+ passed' "$normalized" | head -n 1 | awk '{print $3}' || true
+ )"
+ local node_tests
+ node_tests="$(
+ grep -E '^(#|ℹ) tests[[:space:]]+[0-9]+$' "$normalized" | awk '{print $NF}' | tail -n 1 || true
)"
local node_pass
node_pass="$(
- grep -E 'pass[[:space:]]+[1-9][0-9]*$' "$normalized" | awk '{print $NF}' | tail -n 1 || true
+ grep -E '^(#|ℹ) pass[[:space:]]+[1-9][0-9]*$' "$normalized" | awk '{print $NF}' | tail -n 1 || true
)"
if [[ -z "${vitest_passed}" && -z "${node_pass}" ]]; then
@@ -48,6 +71,19 @@ assert_executed_counts() {
return 1
fi
+ if [[ -n "${node_pass}" && "${node_tests}" != "${node_pass}" ]]; then
+ fail "${name}: node:test pass count ${node_pass} does not match tests ${node_tests:-missing}"
+ rm -f "$normalized"
+ return 1
+ fi
+
+ if [[ -n "${vitest_passed}" && "${vitest_files}" != "${expected_files}" ]]; then
+ fail "${name}: Vitest passed ${vitest_files:-0} test files, expected ${expected_files}"
+ grep -E 'Test Files|No test files found' "$normalized" || true
+ rm -f "$normalized"
+ return 1
+ fi
+
if grep -Eq '(Test Files|Tests).*[1-9][0-9]* skipped' "$normalized"; then
fail "${name}: Vitest reported skipped tests"
rm -f "$normalized"
@@ -58,6 +94,11 @@ assert_executed_counts() {
rm -f "$normalized"
return 1
fi
+ if grep -Eq '^(#|ℹ) skipped[[:space:]]+[1-9]' "$normalized"; then
+ fail "${name}: node:test reported skipped tests"
+ rm -f "$normalized"
+ return 1
+ fi
if grep -Eq '^(#|ℹ) todo[[:space:]]+[1-9]' "$normalized"; then
fail "${name}: node:test reported todo tests"
rm -f "$normalized"
@@ -69,123 +110,262 @@ assert_executed_counts() {
return 1
fi
+ if [[ -n "${vitest_passed}" ]]; then
+ COUNT_SUMMARY="${vitest_files} files, ${vitest_passed} tests"
+ else
+ COUNT_SUMMARY="${node_pass}/${node_tests} node:test tests"
+ fi
rm -f "$normalized"
return 0
}
+# run_group <name> <package dir> <suite files relative to the package...>
+# Runs the package's own `npm test -- <files>`; the grok-llm-proxy test script
+# runs its test typecheck before Vitest.
run_group() {
local name="$1"
- shift
- GROUPS_RUN=$((GROUPS_RUN + 1))
+ local prefix="$2"
+ shift 2
+ if [[ "$#" -eq 0 ]]; then
+ fail "${name}: group listed no suite files"
+ return 1
+ fi
+ local files=("$@")
+ local rel
+ for rel in "${files[@]}"; do
+ REGISTERED+=("${prefix}/${rel}")
+ done
+ for rel in "${files[@]}"; do
+ require_file "${prefix}/${rel}" || return 1
+ done
+
local log
log="$(mktemp)"
- if "$@" >"$log" 2>&1; then
- if assert_executed_counts "$name" "$log"; then
- pass "$name"
- fi
- else
- fail "$name"
- echo "----- ${name} -----"
+ local npm_args=(--prefix "${ROOT}/${prefix}" test --)
+ npm_args+=("${files[@]}")
+
+ echo "── ${name} ──"
+ if ! (
+ cd "${ROOT}"
+ FORCE_COLOR=0 NO_COLOR=1 npm "${npm_args[@]}"
+ ) >"$log" 2>&1; then
+ fail "${name}: command failed"
cat "$log"
+ rm -f "$log"
+ return 1
+ fi
+
+ COUNT_SUMMARY=""
+ if ! assert_executed_counts "$name" "$log" "${#files[@]}"; then
+ rm -f "$log"
+ return 1
fi
+
+ GROUPS_RUN=$((GROUPS_RUN + 1))
+ pass "${name} (${COUNT_SUMMARY})"
rm -f "$log"
}
-require_file packages/grok-provider-attempt-contract/index.test.cjs
-require_file grok-llm-proxy/test/contractFreeze.test.ts
-require_file grok-llm-proxy/test/originPolicy.test.ts
-require_file control-api/test/services.grokSubscriptionOAuth.test.ts
-require_file control-api/test/services.grokProviderAttemptTicket.test.ts
-require_file control-api/test/services.grokSubscriptionConnection.realPostgres.integration.test.ts
-require_file control-api/test/pluginWorkloadSdkGrokDualLedger.realPostgres.integration.test.ts
-require_file control-api/test/services.grokProviderAttemptRedemption.realPostgres.integration.test.ts
-require_file control-api/test/services.grokProviderAttemptRedemption.test.ts
-require_file control-api/test/db.grokSubscriptionMigration.test.ts
-require_file control-api/test/services.llmProviderAttemptAuthorizer.grok.test.ts
-require_file control-api/test/subscriptionGrantIdentity.test.ts
-require_file control-api/test/routes.adminPluginWorkloadSdk.test.ts
-require_file control-api/test/services.recipeCodexGrantIdentity.test.ts
-require_file control-api/test/hostSpecValidation.grokSubscription.test.ts
-require_file control-api/test/crd.llmProviderEnums.test.ts
-require_file control-api/test/routes.mcp-host.plugin-workload-sdk.test.ts
-require_file mcp-host/src/pluginWorkloadSdk/promptBridge/controlApiClient.test.ts
-require_file mcp-host/src/pluginWorkloadSdk/promptBridge/llmBridge.failover.test.ts
-require_file control-ui/components/__tests__/HostWizard.test.tsx
-require_file control-ui/components/__tests__/HostDetailsPage.identity.test.tsx
-require_file scripts/tests/test-grok-llm-proxy-deploy-contract.sh
-require_file tests/e2e/fixtures/grok-subscription/sanitized-upstream-contract.json
-require_file workflow-recipes/src/workflow/codexRecipeVerdict.test.ts
-require_file workflow-recipes/src/workflow/sdkOnlyGrokBinding.test.ts
-require_file workflow-recipes/src/workflow/pluginWorkloadSdkProvisioner.grokPolicy.test.ts
-require_file workflow-recipes/tests/unit/workflow/modelConfigHandler.pluginSdkBroker.test.ts
-require_file mcp-host/src/pluginWorkloadSdk/bootstrapIdentity.test.ts
-require_file packages/codex-catalog-projection/index.test.cjs
-require_file control-api/test/services.usageEvents.test.ts
-require_file control-api/test/services.llmProviderAttemptFinalization.test.ts
-require_file workflow-recipes/tests/unit/workflow/podFactory.test.ts
-require_file control-ui/components/__tests__/RecipeEditor.test.tsx
-require_file control-ui/lib/__tests__/llm.test.ts
-require_file control-ui/components/__tests__/PluginWorkloadSdkPage.test.tsx
+require_ci_matrix_entry() {
+ local entry="$1"
+ if ! grep -Eq "^[[:space:]]+- ${entry}$" "${ROOT}/.github/workflows/ci-public.yml"; then
+ fail "ci-public.yml matrix is missing ${entry}"
+ return 1
+ fi
+ pass "ci-public.yml matrix includes ${entry}"
+}
+
+# Real-Postgres suites are env-gated (skipped without a real PG), so T0 only
+# proves they exist and that the CI real-PG lane asserts each one ran.
+require_real_pg_suite() {
+ local rel="$1"
+ REGISTERED+=("${rel}")
+ require_file "${rel}" || return 1
+ local suite
+ suite="$(basename "${rel}" .test.ts)"
+ if ! sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*\\$//' "${ROOT}/.github/workflows/ci-public.yml" |
+ grep -Fxq "${suite}"; then
+ fail "ci-public.yml real-PG lane does not list ${suite}"
+ return 1
+ fi
+ pass "real-PG suite present and listed in CI: ${suite}"
+}
+
+echo "Grok subscription T0 aggregator"
+require_ci_matrix_entry "grok-llm-proxy"
+require_ci_matrix_entry "packages/grok-provider-attempt-contract"
+
+oauth_source="${ROOT}/control-api/src/services/grokSubscriptionOAuth.ts"
expected_device_url='https://auth.x.ai/oauth2/device/code'
expected_token_url='https://auth.x.ai/oauth2/token'
+expected_revoke_url='https://auth.x.ai/oauth2/revoke'
expected_client_id='b1a00492-073a-47ea-816f-4c329264a828'
-api_device_url="$(
- sed -n "s/^export const GROK_OAUTH_DEVICE_URL = '\\(.*\\)'$/\\1/p" \
- "${ROOT}/control-api/src/services/grokSubscriptionOAuth.ts"
-)"
-api_token_url="$(
- sed -n "s/^export const GROK_OAUTH_TOKEN_URL = '\\(.*\\)'$/\\1/p" \
- "${ROOT}/control-api/src/services/grokSubscriptionOAuth.ts"
-)"
+api_device_url="$(sed -n "s/^export const GROK_OAUTH_DEVICE_URL = '\\(.*\\)'$/\\1/p" "${oauth_source}")"
+api_token_url="$(sed -n "s/^export const GROK_OAUTH_TOKEN_URL = '\\(.*\\)'$/\\1/p" "${oauth_source}")"
+api_revoke_url="$(sed -n "s/^export const GROK_OAUTH_REVOKE_URL = '\\(.*\\)'$/\\1/p" "${oauth_source}")"
api_client_id="$(
sed -n "s/.*CONTROL_API_GROK_OAUTH_CLIENT_ID || '\\(.*\\)',$/\\1/p" \
"${ROOT}/control-api/src/config.ts"
)"
if [[ "${api_device_url}" == "${expected_device_url}" &&
"${api_token_url}" == "${expected_token_url}" &&
+ "${api_revoke_url}" == "${expected_revoke_url}" &&
"${api_client_id}" == "${expected_client_id}" ]]; then
pass "Grok OAuth origin/client lock"
else
- fail "Grok OAuth origin/client lock (device=${api_device_url:-missing} token=${api_token_url:-missing} client=${api_client_id:-missing})"
+ fail "Grok OAuth origin/client lock (device=${api_device_url:-missing} token=${api_token_url:-missing} revoke=${api_revoke_url:-missing} client=${api_client_id:-missing})"
fi
-run_group "grok-provider-attempt-contract" \
- bash -lc "cd '${ROOT}/packages/grok-provider-attempt-contract' && node --test index.test.cjs"
+run_group "grok-provider-attempt-contract" "packages/grok-provider-attempt-contract" \
+ "index.test.cjs"
-run_group "grok-catalog-projection" \
- bash -lc "cd '${ROOT}/packages/codex-catalog-projection' && node --test index.test.cjs"
+# Shared projection package; its suite covers the Grok ConfigMap annotations.
+run_group "grok-catalog-projection" "packages/codex-catalog-projection" \
+ "index.test.cjs"
-run_group "grok-llm-proxy freeze+origin" \
- bash -lc "cd '${ROOT}/grok-llm-proxy' && npx vitest run test/contractFreeze.test.ts test/originPolicy.test.ts test/grokTransport.conformance.test.ts --no-file-parallelism"
+# Every proxy suite, including the hermetic runtime e2e. The unlisted-suite
+# check below fails when a proxy test file is added without being listed here.
+run_group "grok-llm-proxy" "grok-llm-proxy" \
+ "test/abortWhenClientDisconnects.test.ts" \
+ "test/approvedToolsUpstream.test.ts" \
+ "test/catalogBounds.test.ts" \
+ "test/contractFreeze.test.ts" \
+ "test/controlApiClient.test.ts" \
+ "test/grokTransport.conformance.test.ts" \
+ "test/grokUpstreamHeaders.test.ts" \
+ "test/originPolicy.test.ts" \
+ "test/redaction.test.ts" \
+ "test/requestLimits.test.ts" \
+ "test/runtimePath.hermetic.e2e.test.ts" \
+ "test/server.security.test.ts" \
+ "test/sseBackpressure.test.ts" \
+ "test/toolNameMap.test.ts"
-run_group "grok-llm-proxy tsc" \
- bash -lc "cd '${ROOT}/grok-llm-proxy' && npx tsc --noEmit && echo 'pass 1'"
+run_group "control-api grok" "control-api" \
+ "test/db.grokSubscriptionMigration.test.ts" \
+ "test/db.llmProviderAttemptMigration.test.ts" \
+ "test/services.grokSubscriptionConnection.test.ts" \
+ "test/services.grokSubscriptionOAuth.test.ts" \
+ "test/services.grokSubscriptionCatalog.test.ts" \
+ "test/services.subscriptionCatalogBounds.test.ts" \
+ "test/services.grokProviderAttemptTicket.test.ts" \
+ "test/services.grokProviderAttemptRedemption.test.ts" \
+ "test/services.llmProviderAttemptAuthorizer.grok.test.ts" \
+ "test/services.llmProviderAttemptAuthorizer.depth.test.ts" \
+ "test/services.llmProviderAttemptFinalization.test.ts" \
+ "test/services.usageEvents.test.ts" \
+ "test/services.recipeGrantTransition.test.ts" \
+ "test/services.recipeCodexGrantIdentity.test.ts" \
+ "test/subscriptionGrantIdentity.test.ts" \
+ "test/llmProviders.test.ts" \
+ "test/routes.admin.grokSubscription.test.ts" \
+ "test/routes.internal.llmProviderAttempts.grok.test.ts" \
+ "test/routes.adminPluginWorkloadSdk.test.ts" \
+ "test/routes.adminRecipes.test.ts" \
+ "test/routes.mcp-host.plugin-workload-sdk.test.ts" \
+ "test/hostSpecValidation.grokSubscription.test.ts" \
+ "test/hostSpecValidation.limits.test.ts" \
+ "test/crd.hostCel.test.ts" \
+ "test/crd.llmProviderEnums.test.ts"
-run_group "control-api grok grant/oauth/redeem/authorize" \
- bash -lc "cd '${ROOT}/control-api' && npx vitest run test/db.grokSubscriptionMigration.test.ts test/services.grokSubscriptionConnection.test.ts test/services.grokSubscriptionOAuth.test.ts test/services.grokSubscriptionCatalog.test.ts test/services.grokProviderAttemptRedemption.test.ts test/services.grokProviderAttemptTicket.test.ts test/services.llmProviderAttemptAuthorizer.grok.test.ts test/subscriptionGrantIdentity.test.ts test/routes.adminPluginWorkloadSdk.test.ts test/services.recipeCodexGrantIdentity.test.ts test/hostSpecValidation.grokSubscription.test.ts test/crd.llmProviderEnums.test.ts test/routes.mcp-host.plugin-workload-sdk.test.ts test/services.usageEvents.test.ts test/services.llmProviderAttemptFinalization.test.ts --no-file-parallelism"
+require_real_pg_suite "control-api/test/services.grokSubscriptionConnection.realPostgres.integration.test.ts"
+require_real_pg_suite "control-api/test/services.grokProviderAttemptRedemption.realPostgres.integration.test.ts"
+require_real_pg_suite "control-api/test/services.grokProviderAttemptRedemption.refresh.realPostgres.integration.test.ts"
+require_real_pg_suite "control-api/test/pluginWorkloadSdkGrokDualLedger.realPostgres.integration.test.ts"
+require_real_pg_suite "control-api/test/db.llmProviderAttemptConnectionIntegrity.realPostgres.integration.test.ts"
-run_group "workflow-recipes grok SDK" \
- bash -lc "cd '${ROOT}/workflow-recipes' && npx vitest run src/workflow/codexRecipeVerdict.test.ts src/workflow/sdkOnlyGrokBinding.test.ts src/workflow/pluginWorkloadSdkProvisioner.grokPolicy.test.ts src/reconciler/pluginWorkloadSdkValidator.test.ts tests/unit/workflow/modelConfigHandler.pluginSdkBroker.test.ts tests/unit/workflow/podFactory.test.ts --no-file-parallelism"
+run_group "mcp-host grok" "mcp-host" \
+ "src/llm/__tests__/grokSubscription.test.ts" \
+ "src/llm/__tests__/grokLlmProxyClient.test.ts" \
+ "src/llm/__tests__/grokPolicyBinding.test.ts" \
+ "src/llm/__tests__/subscriptionRequestHash.test.ts" \
+ "src/llm/__tests__/registry.test.ts" \
+ "src/llm/__tests__/makeProvider.throws.test.ts" \
+ "src/config/configStore.test.ts" \
+ "src/core/orchestration/__tests__/toolPresentationPolicy.test.ts" \
+ "src/pluginWorkloadSdk/bootstrapIdentity.test.ts" \
+ "src/pluginWorkloadSdk/bootstrapServer.test.ts" \
+ "src/pluginWorkloadSdk/promptBridge/controlApiClient.test.ts" \
+ "src/pluginWorkloadSdk/promptBridge/llmBridge.failover.test.ts"
-run_group "mcp-host grok bootstrap" \
- bash -lc "cd '${ROOT}/mcp-host' && npx vitest run src/pluginWorkloadSdk/bootstrapIdentity.test.ts src/pluginWorkloadSdk/promptBridge/controlApiClient.test.ts src/pluginWorkloadSdk/promptBridge/llmBridge.failover.test.ts --no-file-parallelism"
+run_group "workflow-recipes grok" "workflow-recipes" \
+ "src/workflow/codexRecipeVerdict.test.ts" \
+ "src/workflow/sdkOnlyGrokBinding.test.ts" \
+ "src/workflow/pluginWorkloadSdkProvisioner.grokPolicy.test.ts" \
+ "src/workflow/workflowReconciler.grokWrcFlag.test.ts" \
+ "src/reconciler/workflowRecipeReconciler.grokFlagWiring.test.ts" \
+ "src/reconciler/pluginWorkloadSdkValidator.test.ts" \
+ "src/mcp/server.pluginSdkBroker.test.ts" \
+ "tests/unit/workflow/modelConfigHandler.test.ts" \
+ "tests/unit/workflow/modelConfigHandler.pluginSdkBroker.test.ts" \
+ "tests/unit/workflow/podFactory.test.ts"
-run_group "control-ui grok SDK picker" \
- bash -lc "cd '${ROOT}/control-ui' && npx vitest run components/__tests__/PluginWorkloadSdkPage.test.tsx components/__tests__/HostWizard.test.tsx components/__tests__/HostDetailsPage.identity.test.tsx components/__tests__/RecipeEditor.test.tsx lib/__tests__/llm.test.ts --no-file-parallelism"
+run_group "host-context-controller grok" "host-context-controller" \
+ "src/hostReconciler.grokHostRuntime.test.ts" \
+ "test/hostReconciler.test.ts"
-if ! bash "${ROOT}/scripts/tests/test-grok-llm-proxy-deploy-contract.sh"; then
- fail "grok-llm-proxy deploy contract"
-else
- GROUPS_RUN=$((GROUPS_RUN + 1))
- pass "grok-llm-proxy deploy contract"
+run_group "control-ui grok" "control-ui" \
+ "components/__tests__/CodexSubscriptionHub.test.tsx" \
+ "components/__tests__/LlmCatalogFormsGrokGate.test.tsx" \
+ "components/__tests__/RecipeEditor.test.tsx" \
+ "components/__tests__/HostWizard.test.tsx" \
+ "components/__tests__/HostDetailsPage.identity.test.tsx" \
+ "components/__tests__/PluginWorkloadSdkPage.test.tsx" \
+ "components/__tests__/LlmProviderConfig.test.tsx" \
+ "components/__tests__/LlmPolicyEditor.test.tsx" \
+ "components/__tests__/LlmModelForm.test.tsx" \
+ "lib/__tests__/grokSubscriptionFeature.test.ts" \
+ "lib/__tests__/llm.test.ts" \
+ "lib/hooks/__tests__/useGrokSubscriptionEnabled.test.tsx"
+
+echo "── grok-llm-proxy deploy contract ──"
+if require_file "scripts/tests/test-grok-llm-proxy-deploy-contract.sh" &&
+ require_file "tests/e2e/fixtures/grok-subscription/sanitized-upstream-contract.json"; then
+ if bash "${ROOT}/scripts/tests/test-grok-llm-proxy-deploy-contract.sh"; then
+ GROUPS_RUN=$((GROUPS_RUN + 1))
+ pass "grok-llm-proxy deploy contract"
+ else
+ fail "grok-llm-proxy deploy contract"
+ fi
+fi
+
+# A Grok suite that exists but is not listed above is lost coverage. Every
+# proxy/contract test file and every *grok* test file in the Grok-touching
+# packages must be registered in a group or as a real-PG presence check.
+is_registered() {
+ local candidate="$1" entry
+ for entry in "${REGISTERED[@]}"; do
+ [[ "${entry}" == "${candidate}" ]] && return 0
+ done
+ return 1
+}
+unlisted=0
+while IFS= read -r rel; do
+ [[ -n "${rel}" ]] || continue
+ if ! is_registered "${rel}"; then
+ fail "unlisted Grok suite ${rel}"
+ unlisted=1
+ fi
+done < <(
+ cd "${ROOT}" &&
+ {
+ find grok-llm-proxy/test packages/grok-provider-attempt-contract \
+ -name node_modules -prune -o -type f \( -name '*.test.ts' -o -name '*.test.cjs' \) -print
+ find control-api mcp-host workflow-recipes host-context-controller control-ui \
+ \( -name node_modules -o -name dist -o -name .next -o -name coverage \) -prune -o \
+ -type f -iname '*grok*' \( -name '*.test.ts' -o -name '*.test.tsx' \) -print
+ } | sort -u
+)
+if [[ "${unlisted}" -eq 0 ]]; then
+ pass "every Grok suite is registered in T0"
fi
-if [[ "$GROUPS_RUN" -lt 9 ]]; then
- fail "T0 ran too few groups (${GROUPS_RUN})"
+if [[ "${GROUPS_RUN}" -ne "${EXPECTED_GROUPS}" ]]; then
+ fail "expected all ${EXPECTED_GROUPS} T0 groups, ran ${GROUPS_RUN}"
fi
-if [[ "$FAIL" -ne 0 ]]; then
+if [[ "${FAIL}" -ne 0 ]]; then
echo "FAIL: grok subscription T0"
exit 1
fi