Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

test(e2e): cover the Codex to authorized-fallback journey end to end

Alfredo Lopez4052e462b6e2
diff --git a/Makefile b/Makefile
index e4b9efa91..487f3903f 100644
--- a/Makefile
+++ b/Makefile
@@ -1165,6 +1165,29 @@ test-e2e-plugin-workload-sdk-desktop-no-grant: ## Run the no-grant Codex guard (
E2E_PLUGIN_SDK_NO_GRANT_APP_TITLE="$$E2E_PLUGIN_SDK_NO_GRANT_APP_TITLE" \
./node_modules/.bin/playwright test --config test/e2e-playwright/playwright.config.ts plugin-workload-sdk-no-grant-guard.spec.ts
+.PHONY: test-e2e-plugin-workload-sdk-desktop-codex-fallback
+test-e2e-plugin-workload-sdk-desktop-codex-fallback: ## Run the Codex -> authorized fallback journey (#533 acceptance criterion 8); stops codex-llm-proxy and restores it
+ @set -eu; \
+ ctx="$${KUBECONTEXT:-$(E2E_KUBECONTEXT)}"; \
+ test -n "$$ctx" || { echo "Refusing Desktop Plugin Workload SDK fallback journey: explicit Kubernetes context is required" >&2; exit 1; }; \
+ test "$${E2E_PLUGIN_SDK_WRITE_CONFIRM:-}" = 1 || { echo "Refusing Desktop Plugin Workload SDK fallback journey: set E2E_PLUGIN_SDK_WRITE_CONFIRM=1" >&2; exit 1; }; \
+ test -n "$${CONTROL_UI_BASE_URL:-}" || { echo "CONTROL_UI_BASE_URL must be set to the branch-owned random port" >&2; exit 1; }; \
+ test -n "$${CONTROL_API_BASE_URL:-}" || { echo "CONTROL_API_BASE_URL must be set to the branch-owned random port" >&2; exit 1; }; \
+ test -n "$${EXTERNAL_REST_API_BASE_URL:-}" || { echo "EXTERNAL_REST_API_BASE_URL must be set to the branch-owned random port" >&2; exit 1; }; \
+ test -n "$${RPC_PROXY_BASE_URL:-}" || { echo "RPC_PROXY_BASE_URL must be set to the branch-owned random port" >&2; exit 1; }; \
+ test "$${E2E_PLUGIN_SDK_EXPECT_PROVIDER:-}" = codex-subscription || { echo "Refusing Desktop Plugin Workload SDK fallback journey: set E2E_PLUGIN_SDK_EXPECT_PROVIDER=codex-subscription; this journey exists only for the Codex primary" >&2; exit 1; }; \
+ . scripts/e2e/e2e-lib.sh; \
+ require_branch_profile_urls "$$ctx" "$${CLERUM_PROFILE_PORTS_ENV:-$${E2E_PROFILE_PORTS_ENV:-}}"; \
+ echo "[E2E-GUARD] Desktop Plugin Workload SDK fallback journey: context=$$ctx; recipe=$${E2E_PLUGIN_SDK_RECIPE_NAMESPACE:-sandbox-recipes}/$${E2E_PLUGIN_SDK_RECIPE_NAME:-evenfire-prompt-notify-app}; codex-llm-proxy will be stopped and restored"; \
+ trap 'kubectl --context "$$ctx" -n control-plane scale deployment/codex-llm-proxy --replicas=1 >/dev/null || echo "[E2E-GUARD] FAILED to restore control-plane/codex-llm-proxy to replicas=1 — restore it before running any other lane on this profile" >&2' EXIT; \
+ cd desktop-app && KUBECONTEXT="$$ctx" E2E_K8S_CONTEXT="$$ctx" E2E_PLUGIN_SDK_DESKTOP=1 E2E_PLUGIN_SDK_CODEX_FALLBACK=1 npm run build && \
+ KUBECONTEXT="$$ctx" E2E_K8S_CONTEXT="$$ctx" E2E_PLUGIN_SDK_DESKTOP=1 E2E_PLUGIN_SDK_CODEX_FALLBACK=1 \
+ E2E_PLUGIN_SDK_EXPECT_PROVIDER="$$E2E_PLUGIN_SDK_EXPECT_PROVIDER" \
+ E2E_PLUGIN_SDK_RECIPE_NAME="$${E2E_PLUGIN_SDK_RECIPE_NAME:-evenfire-prompt-notify-app}" \
+ E2E_PLUGIN_SDK_RECIPE_NAMESPACE="$${E2E_PLUGIN_SDK_RECIPE_NAMESPACE:-sandbox-recipes}" \
+ E2E_PLUGIN_SDK_APP_TITLE="$${E2E_PLUGIN_SDK_APP_TITLE:-Prompt & Notify}" \
+ ./node_modules/.bin/playwright test --config test/e2e-playwright/playwright.config.ts plugin-workload-sdk-codex-fallback.spec.ts
+
.PHONY: test-e2e-cron-tab-validation
test-e2e-cron-tab-validation: ## Run recipe cron tab E2E (set E2E_CRON_TAB_FIX_REQUIRED=1 when the fix must be present)
@echo "Running cron tab validation E2E gate..."
diff --git a/desktop-app/test/e2e-playwright/plugin-workload-sdk-codex-fallback.spec.ts b/desktop-app/test/e2e-playwright/plugin-workload-sdk-codex-fallback.spec.ts
new file mode 100644
index 000000000..8eed48c30
--- /dev/null
+++ b/desktop-app/test/e2e-playwright/plugin-workload-sdk-codex-fallback.spec.ts
@@ -0,0 +1,244 @@
+import { type ElectronApplication, type Page, expect, test } from '@playwright/test'
+import {
+ type EmbeddedContents,
+ type SandboxUiFixture,
+ activateEmbedded,
+ assertSandboxUiFixture,
+ assertSteplessSdkRecipePrecondition,
+ embeddedText,
+ findSandboxUiContents,
+ latestSdkInvocationStatus,
+ promptBridgeAttemptsForRun,
+ promptBridgeGrantTargets,
+ readPluginWorkloadSdkStatus,
+ sandboxUiViewUrlPrefix,
+ sdkInvocationCount,
+ typeEmbedded,
+} from './pluginWorkloadSdkSandboxUi'
+import {
+ EXTERNAL_REST_API_BASE_URL,
+ RPC_PROXY_BASE_URL,
+ assertAllowedTarget,
+ desktopCredentials,
+ finalizeRecording,
+ launchDesktopApp,
+ login,
+ requireRecorderConfirm,
+} from './qa-recorder-helpers'
+import { expectDeploymentReady, kubectlOut } from './workflow-approval-quadrants/cluster'
+
+/*
+ * E2E_GUARDIAN_IPC_FLOW: Desktop Apps opens the Plugin Workload SDK sandbox UI
+ * through the main-process WebContentsView and EmbeddedContents bridge. The
+ * renderer has no HTTP response to await for that transition; the prompt
+ * output and the persisted dual ledger are the visible/business signals.
+ *
+ * Acceptance criterion 8 of issue #533: "a real user-journey E2E covers
+ * sdk-only -> codex-subscription primary -> successful prompt, PLUS an eligible
+ * failure that reaches a Z.ai fallback". The happy half lives in
+ * plugin-workload-sdk-sandbox-ui.spec.ts. This file is the other half, and it
+ * is the one that matters most, because the missing fallback WAS the original
+ * bug: #533's root cause records that a local no_grant was wrapped as
+ * provider_unavailable, classified as a non-retryable authentication failure,
+ * and so "the ordered Z.ai fallback is never attempted".
+ *
+ * Fault injection: `codex-llm-proxy` is scaled to zero. The resulting fetch
+ * rejection carries no provider code, so mcp-host's classifyError falls through
+ * to classifyUnknown -> {ApiCallFailed, retryable:true} -> failover class
+ * `provider_unavailable`, which is in the default triggerOn set. A more
+ * specific fault (revoked credential, disallowed model) would classify as
+ * non-eligible and propagate WITHOUT failover, proving nothing. control-api's
+ * authorize path never contacts the proxy, so attempt 0 is still reserved and
+ * the displaced-attempt bookkeeping is exercised for real.
+ *
+ * This journey therefore also covers `settlePriorProviderAttemptFloors`, the
+ * server-side prior-attempt discovery: attempt 0 never reaches finalize when a
+ * later target wins, so its spend floor exists only if that path ran.
+ *
+ * Its own file, its own Make target: the two sandbox journeys previously shared
+ * module-scope state, and this one additionally mutates cluster state.
+ */
+const RUN_ENABLED =
+ process.env.E2E_PLUGIN_SDK_DESKTOP === '1' && process.env.E2E_PLUGIN_SDK_CODEX_FALLBACK === '1'
+
+test.skip(
+ !RUN_ENABLED,
+ 'Set E2E_PLUGIN_SDK_DESKTOP=1 and E2E_PLUGIN_SDK_CODEX_FALLBACK=1 for the Codex fallback journey.'
+)
+
+const PROXY_NAMESPACE = 'control-plane'
+const PROXY_DEPLOYMENT = 'codex-llm-proxy'
+
+/**
+ * The granted happy-path fixture. The fallback needs a working grant with an
+ * ordered target list, which is exactly what that recipe already has — unlike
+ * the no-grant guard, this journey has no reason to install a second recipe.
+ */
+function grantedFixture(): SandboxUiFixture {
+ return assertSandboxUiFixture(
+ {
+ recipeName: process.env.E2E_PLUGIN_SDK_RECIPE_NAME || 'evenfire-prompt-notify-app',
+ recipeNamespace: process.env.E2E_PLUGIN_SDK_RECIPE_NAMESPACE || 'sandbox-recipes',
+ appTitle: process.env.E2E_PLUGIN_SDK_APP_TITLE || 'Prompt & Notify',
+ },
+ 'E2E_PLUGIN_SDK'
+ )
+}
+
+function scaleProxy(replicas: number): void {
+ kubectlOut([
+ '-n',
+ PROXY_NAMESPACE,
+ 'scale',
+ `deployment/${PROXY_DEPLOYMENT}`,
+ `--replicas=${replicas}`,
+ ])
+}
+
+/**
+ * Wait until no proxy Pod is left serving. Scaling returns immediately, so
+ * prompting before termination completes would race the fault into a normal
+ * success and silently turn this into a second happy-path run.
+ */
+async function expectProxyStopped(): Promise<void> {
+ await expect
+ .poll(
+ () =>
+ kubectlOut([
+ '-n',
+ PROXY_NAMESPACE,
+ 'get',
+ 'pods',
+ '-l',
+ `app=${PROXY_DEPLOYMENT}`,
+ '-o',
+ 'jsonpath={.items[*].metadata.name}',
+ ]).trim(),
+ { timeout: 120_000 }
+ )
+ .toBe('')
+}
+
+test('Desktop Apps falls back to the authorized non-Codex target when Codex is unavailable', async ({}, testInfo) => {
+ requireRecorderConfirm(
+ 'E2E_PLUGIN_SDK_WRITE_CONFIRM',
+ 'This journey stops codex-llm-proxy, performs one paid promptBridge call served by the fallback provider, and restores the proxy.'
+ )
+ const fixture = grantedFixture()
+ // Hard preconditions, never branches. RP-539-002 was exactly this shape: the
+ // happy path kept its only Codex assertions behind a guard, so an OpenAI
+ // recipe passed it green without touching Codex, and the static auditor
+ // reported PASS because it validates form and cannot reach semantics.
+ assertSteplessSdkRecipePrecondition(fixture, 'codex-subscription')
+ await expect
+ .poll(() => readPluginWorkloadSdkStatus(fixture).state, { timeout: 180_000 })
+ .toBe('validated')
+ expect(readPluginWorkloadSdkStatus(fixture).bootstrapContractVersion).toBe(3)
+
+ // Without a second authorized target there is nowhere to fall to, and the
+ // journey would pass while proving nothing.
+ const targets = promptBridgeGrantTargets(fixture)
+ expect(targets.length).toBeGreaterThanOrEqual(2)
+ expect(targets[0]?.provider).toBe('codex-subscription')
+ expect(targets[1]?.provider).not.toBe('codex-subscription')
+ expect(targets[1]?.provider).not.toBe('')
+ const fallbackProvider = targets[1]!.provider
+
+ await assertAllowedTarget('EXTERNAL_REST_API_BASE_URL', EXTERNAL_REST_API_BASE_URL)
+ await assertAllowedTarget('RPC_PROXY_BASE_URL', RPC_PROXY_BASE_URL)
+
+ let app: ElectronApplication | undefined
+ let page: Page | undefined
+ let proxyStopped = false
+ try {
+ const credentials = desktopCredentials()
+ const launched = await launchDesktopApp(testInfo)
+ app = launched.app
+ page = launched.page
+ await login(page, credentials)
+ await page.getByTestId('nav-sandbox-ui').click()
+ await expect(page.getByRole('heading', { name: 'Apps', exact: true })).toBeVisible()
+ const appCard = page
+ .getByRole('main')
+ .getByRole('button', { name: `Open ${fixture.appTitle}`, exact: true })
+ await expect(appCard).toHaveCount(1)
+ await expect(appCard).toBeVisible({ timeout: 30_000 })
+ await appCard.click()
+ await expect(page.getByRole('button', { name: 'Back to apps' })).toBeVisible({
+ timeout: 30_000,
+ })
+ let embedded: EmbeddedContents | null = null
+ await expect
+ .poll(async () => {
+ embedded = await findSandboxUiContents(app!, fixture)
+ return embedded?.url ?? ''
+ })
+ .toContain(sandboxUiViewUrlPrefix(fixture))
+ const webContentsId = embedded!.id
+
+ await test.step('stop the Codex proxy so the primary target fails eligibly', async () => {
+ scaleProxy(0)
+ proxyStopped = true
+ await expectProxyStopped()
+ })
+
+ const runStartedAt = new Date().toISOString()
+ const before = sdkInvocationCount(fixture, 'promptBridge')
+ await typeEmbedded(
+ app,
+ webContentsId,
+ '#prompt',
+ 'Reply with the single word OK. Codex is down; the authorized fallback must serve this.'
+ )
+ await activateEmbedded(app, webContentsId, '#run')
+ await expect
+ .poll(() => embeddedText(app!, webContentsId, '#prompt-out'), { timeout: 180_000 })
+ .not.toMatch(/Running…/)
+
+ const promptResult = await embeddedText(app, webContentsId, '#prompt-out')
+ expect(promptResult.trim()).not.toBe('')
+ expect(sdkInvocationCount(fixture, 'promptBridge')).toBe(before + 1)
+ expect(latestSdkInvocationStatus(fixture, 'promptBridge')).toBe('complete')
+
+ // The dual ledger is the business/audit signal: ordering, attempt limits
+ // and spend fencing are all row-level facts here, not inferences from the
+ // rendered text.
+ const attempts = promptBridgeAttemptsForRun(fixture, runStartedAt)
+ await testInfo.attach('promptBridge-attempts.json', {
+ body: JSON.stringify(attempts, null, 2),
+ contentType: 'application/json',
+ })
+ expect(attempts).toHaveLength(2)
+
+ const [codexAttempt, fallbackAttempt] = attempts
+ expect(codexAttempt!.attemptIndex).toBe(0)
+ expect(codexAttempt!.provider).toBe('codex-subscription')
+ expect(codexAttempt!.status).toBe('provider_unavailable')
+ // The floor for a displaced Codex attempt is honest uncertainty. The fetch
+ // rejection leaves providerDispatched undefined, which is read as
+ // dispatched, so the ledger records `unknown` rather than claiming the
+ // subscription was never touched. `exact` would mean a spend was proven.
+ expect(codexAttempt!.spendOutcome).toBe('unknown')
+ expect(codexAttempt!.spendOutcome).not.toBe('exact')
+
+ expect(fallbackAttempt!.attemptIndex).toBe(1)
+ expect(fallbackAttempt!.provider).toBe(fallbackProvider)
+ expect(fallbackAttempt!.provider).not.toBe('codex-subscription')
+ // `complete` is the success terminal for a physical attempt; the schema's
+ // CHECK allows only reserved/in_progress/complete/failed/
+ // provider_unavailable/skipped.
+ expect(fallbackAttempt!.status).toBe('complete')
+ expect(fallbackAttempt!.codexAttemptId).toBe('')
+
+ await page.getByRole('button', { name: 'Back to apps' }).click()
+ await expect(page.getByRole('heading', { name: 'Apps', exact: true })).toBeVisible()
+ } finally {
+ if (proxyStopped) {
+ // Fail loudly rather than leave the cluster degraded. A silent restore
+ // failure would poison every later lane on this profile.
+ scaleProxy(1)
+ expectDeploymentReady(PROXY_NAMESPACE, PROXY_DEPLOYMENT)
+ }
+ await finalizeRecording(app, page)
+ }
+})
diff --git a/desktop-app/test/e2e-playwright/pluginWorkloadSdkSandboxUi.ts b/desktop-app/test/e2e-playwright/pluginWorkloadSdkSandboxUi.ts
index bf51358c0..14587e744 100644
--- a/desktop-app/test/e2e-playwright/pluginWorkloadSdkSandboxUi.ts
+++ b/desktop-app/test/e2e-playwright/pluginWorkloadSdkSandboxUi.ts
@@ -572,6 +572,107 @@ export function promptBridgeLedgerForRun(
}
}
+export type PromptBridgeAttemptRow = {
+ attemptIndex: number
+ targetRef: string
+ provider: string
+ model: string
+ status: string
+ spendOutcome: string
+ codexAttemptId: string
+}
+
+/**
+ * Every physical attempt of the run's first promptBridge invocation, ordered by
+ * `attempt_index`.
+ *
+ * `promptBridgeLedgerForRun` answers "what did the served attempt record", so
+ * it stops at `LIMIT 1`. A failover journey needs the opposite: the displaced
+ * attempt is the evidence. Attempt 0 never reaches `finalize` when a later
+ * target wins, so its spend floor is written by
+ * `settlePriorProviderAttemptFloors` — the server-side discovery that replaced
+ * a client-declared `priorAttempts` field. Reading every row is what proves
+ * that path ran at all.
+ */
+export function promptBridgeAttemptsForRun(
+ fixture: SandboxUiFixture,
+ startedAt: string
+): PromptBridgeAttemptRow[] {
+ const raw = profilesSql(`
+ SELECT sdk.attempt_index::text || '|' || sdk.target_ref || '|' || sdk.provider || '|' ||
+ sdk.model || '|' || sdk.status || '|' || coalesce(spend.outcome, '') || '|' ||
+ coalesce(codex.id::text, '')
+ FROM plugin_workload_sdk_invocations inv
+ JOIN plugin_workload_sdk_provider_attempts sdk
+ ON sdk.invocation_id = inv.id
+ LEFT JOIN llm_provider_attempts codex
+ ON codex.plugin_workload_sdk_provider_attempt_id = sdk.id
+ LEFT JOIN plugin_workload_sdk_spend_outcomes spend
+ ON spend.provider_attempt_id = sdk.id
+ WHERE inv.recipe_namespace = ${sqlLiteral(fixture.recipeNamespace)}
+ AND inv.recipe_name = ${sqlLiteral(fixture.recipeName)}
+ AND inv.method = 'promptBridge'
+ AND inv.created_at >= ${sqlLiteral(startedAt)}::timestamptz
+ AND inv.id = (
+ SELECT id FROM plugin_workload_sdk_invocations
+ WHERE recipe_namespace = ${sqlLiteral(fixture.recipeNamespace)}
+ AND recipe_name = ${sqlLiteral(fixture.recipeName)}
+ AND method = 'promptBridge'
+ AND created_at >= ${sqlLiteral(startedAt)}::timestamptz
+ ORDER BY created_at ASC
+ LIMIT 1
+ )
+ ORDER BY sdk.attempt_index ASC;
+ `)
+ if (!raw.trim()) return []
+ return raw
+ .trim()
+ .split('\n')
+ .map(line => {
+ const [attemptIndex, targetRef, provider, model, status, spendOutcome, codexAttemptId] = line
+ .trim()
+ .split('|')
+ return {
+ attemptIndex: Number.parseInt(attemptIndex ?? '', 10),
+ targetRef: targetRef ?? '',
+ provider: provider ?? '',
+ model: model ?? '',
+ status: status ?? '',
+ spendOutcome: spendOutcome ?? '',
+ codexAttemptId: codexAttemptId ?? '',
+ }
+ })
+}
+
+/**
+ * The ordered promptBridge targets on the live grant.
+ *
+ * A failover journey that runs against a single-target grant would pass while
+ * proving nothing — there would be nowhere to fall to. This is asserted before
+ * the fault is injected so an unprovisioned fixture fails as a precondition,
+ * not as a mysteriously green run.
+ */
+export function promptBridgeGrantTargets(
+ fixture: SandboxUiFixture
+): Array<{ targetRef: string; provider: string }> {
+ const raw = profilesSql(`
+ SELECT string_agg(t->>'targetRef' || '|' || (t->>'provider'), E'\\n' ORDER BY ord)
+ FROM plugin_workload_sdk_grants g
+ CROSS JOIN LATERAL jsonb_array_elements(g.prompt_targets) WITH ORDINALITY AS a(t, ord)
+ WHERE g.recipe_namespace = ${sqlLiteral(fixture.recipeNamespace)}
+ AND g.recipe_name = ${sqlLiteral(fixture.recipeName)}
+ AND g.capability_family = 'promptBridge';
+ `)
+ if (!raw.trim()) return []
+ return raw
+ .trim()
+ .split('\n')
+ .map(line => {
+ const [targetRef, provider] = line.trim().split('|')
+ return { targetRef: targetRef ?? '', provider: provider ?? '' }
+ })
+}
+
export function latestSdkInvocationStatus(
fixture: SandboxUiFixture,
method: 'promptBridge' | 'clientNotifications'
diff --git a/docs/testing/optional-playwright-qa-recorder.md b/docs/testing/optional-playwright-qa-recorder.md
index fc3876cb7..0a577bd70 100644
--- a/docs/testing/optional-playwright-qa-recorder.md
+++ b/docs/testing/optional-playwright-qa-recorder.md
@@ -155,6 +155,47 @@ there is none to declare. **Provisioning a Codex recipe with no execution
binding is operator work** — the lane will not create one, and a lane that
cannot find its fixture fails rather than skipping.
+### Codex fallback lane
+
+`plugin-workload-sdk-codex-fallback.spec.ts` closes acceptance criterion 8 of
+issue #533: a real journey where an eligible Codex failure reaches the
+authorized non-Codex fallback. It runs against the **granted** happy-path
+recipe, because the fallback needs a working ordered target list.
+
+```bash
+E2E_PLUGIN_SDK_WRITE_CONFIRM=1 \
+E2E_PLUGIN_SDK_EXPECT_PROVIDER=codex-subscription \
+CONTROL_UI_BASE_URL=http://127.0.0.1:<random-control-ui-port> \
+CONTROL_API_BASE_URL=http://127.0.0.1:<random-control-api-port> \
+EXTERNAL_REST_API_BASE_URL=http://127.0.0.1:<random-external-rest-port> \
+RPC_PROXY_BASE_URL=http://127.0.0.1:<random-rpc-port> \
+KUBECONTEXT=<branch-profile-context> \
+make test-e2e-plugin-workload-sdk-desktop-codex-fallback
+```
+
+**This lane mutates cluster state.** It scales `control-plane/codex-llm-proxy`
+to zero replicas so the primary target fails with a class the failover engine
+accepts, then restores it to one. The restore runs twice — in the spec's
+`finally` and again in a Make-level `EXIT` trap — and both report loudly on
+failure. If you ever see `[E2E-GUARD] FAILED to restore
+control-plane/codex-llm-proxy`, restore it before running any other lane on
+that profile: every Codex journey after it would fail for the wrong reason.
+
+`E2E_PLUGIN_SDK_EXPECT_PROVIDER` must be exactly `codex-subscription`; the
+target rejects any other value, because a non-Codex primary would make the
+journey vacuous. The recipe knobs are shared with the happy path and carry
+**silent defaults** — `E2E_PLUGIN_SDK_RECIPE_NAME` defaults to
+`evenfire-prompt-notify-app`, `E2E_PLUGIN_SDK_RECIPE_NAMESPACE` to
+`sandbox-recipes`, and `E2E_PLUGIN_SDK_APP_TITLE` to `Prompt & Notify`. Set
+them explicitly whenever your fixture differs, or the lane looks for the wrong
+app in the Desktop catalog and fails at the card locator rather than at a
+precondition.
+
+The grant must expose **at least two ordered prompt targets**, Codex first and
+a non-Codex provider second. The spec asserts that before injecting the fault:
+a single-target grant would let the journey pass while proving nothing, so it
+fails as a precondition instead.
+
## Repository isolation
The recorder adds new commands; it does not alter any existing command: