Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
test(e2e): cover the Codex to authorized-fallback journey end to end
Alfredo Lopez4052e462b6e2
diff --git a/Makefile b/Makefileindex e4b9efa91..487f3903f 100644--- a/Makefile+++ b/Makefile@@ -1165,6 +1165,29 @@ test-e2e-plugin-workload-sdk-desktop-no-grant: ## Run the no-grant Codex guard (E2E_PLUGIN_SDK_NO_GRANT_APP_TITLE="$$E2E_PLUGIN_SDK_NO_GRANT_APP_TITLE" \./node_modules/.bin/playwright test --config test/e2e-playwright/playwright.config.ts plugin-workload-sdk-no-grant-guard.spec.ts+.PHONY: test-e2e-plugin-workload-sdk-desktop-codex-fallback+test-e2e-plugin-workload-sdk-desktop-codex-fallback: ## Run the Codex -> authorized fallback journey (#533 acceptance criterion 8); stops codex-llm-proxy and restores it+ @set -eu; \+ ctx="$${KUBECONTEXT:-$(E2E_KUBECONTEXT)}"; \+ test -n "$$ctx" || { echo "Refusing Desktop Plugin Workload SDK fallback journey: explicit Kubernetes context is required" >&2; exit 1; }; \+ test "$${E2E_PLUGIN_SDK_WRITE_CONFIRM:-}" = 1 || { echo "Refusing Desktop Plugin Workload SDK fallback journey: set E2E_PLUGIN_SDK_WRITE_CONFIRM=1" >&2; exit 1; }; \+ test -n "$${CONTROL_UI_BASE_URL:-}" || { echo "CONTROL_UI_BASE_URL must be set to the branch-owned random port" >&2; exit 1; }; \+ test -n "$${CONTROL_API_BASE_URL:-}" || { echo "CONTROL_API_BASE_URL must be set to the branch-owned random port" >&2; exit 1; }; \+ test -n "$${EXTERNAL_REST_API_BASE_URL:-}" || { echo "EXTERNAL_REST_API_BASE_URL must be set to the branch-owned random port" >&2; exit 1; }; \+ test -n "$${RPC_PROXY_BASE_URL:-}" || { echo "RPC_PROXY_BASE_URL must be set to the branch-owned random port" >&2; exit 1; }; \+ test "$${E2E_PLUGIN_SDK_EXPECT_PROVIDER:-}" = codex-subscription || { echo "Refusing Desktop Plugin Workload SDK fallback journey: set E2E_PLUGIN_SDK_EXPECT_PROVIDER=codex-subscription; this journey exists only for the Codex primary" >&2; exit 1; }; \+ . scripts/e2e/e2e-lib.sh; \+ require_branch_profile_urls "$$ctx" "$${CLERUM_PROFILE_PORTS_ENV:-$${E2E_PROFILE_PORTS_ENV:-}}"; \+ echo "[E2E-GUARD] Desktop Plugin Workload SDK fallback journey: context=$$ctx; recipe=$${E2E_PLUGIN_SDK_RECIPE_NAMESPACE:-sandbox-recipes}/$${E2E_PLUGIN_SDK_RECIPE_NAME:-evenfire-prompt-notify-app}; codex-llm-proxy will be stopped and restored"; \+ trap 'kubectl --context "$$ctx" -n control-plane scale deployment/codex-llm-proxy --replicas=1 >/dev/null || echo "[E2E-GUARD] FAILED to restore control-plane/codex-llm-proxy to replicas=1 — restore it before running any other lane on this profile" >&2' EXIT; \+ cd desktop-app && KUBECONTEXT="$$ctx" E2E_K8S_CONTEXT="$$ctx" E2E_PLUGIN_SDK_DESKTOP=1 E2E_PLUGIN_SDK_CODEX_FALLBACK=1 npm run build && \+ KUBECONTEXT="$$ctx" E2E_K8S_CONTEXT="$$ctx" E2E_PLUGIN_SDK_DESKTOP=1 E2E_PLUGIN_SDK_CODEX_FALLBACK=1 \+ E2E_PLUGIN_SDK_EXPECT_PROVIDER="$$E2E_PLUGIN_SDK_EXPECT_PROVIDER" \+ E2E_PLUGIN_SDK_RECIPE_NAME="$${E2E_PLUGIN_SDK_RECIPE_NAME:-evenfire-prompt-notify-app}" \+ E2E_PLUGIN_SDK_RECIPE_NAMESPACE="$${E2E_PLUGIN_SDK_RECIPE_NAMESPACE:-sandbox-recipes}" \+ E2E_PLUGIN_SDK_APP_TITLE="$${E2E_PLUGIN_SDK_APP_TITLE:-Prompt & Notify}" \+ ./node_modules/.bin/playwright test --config test/e2e-playwright/playwright.config.ts plugin-workload-sdk-codex-fallback.spec.ts+.PHONY: test-e2e-cron-tab-validationtest-e2e-cron-tab-validation: ## Run recipe cron tab E2E (set E2E_CRON_TAB_FIX_REQUIRED=1 when the fix must be present)@echo "Running cron tab validation E2E gate..."diff --git a/desktop-app/test/e2e-playwright/plugin-workload-sdk-codex-fallback.spec.ts b/desktop-app/test/e2e-playwright/plugin-workload-sdk-codex-fallback.spec.tsnew file mode 100644index 000000000..8eed48c30--- /dev/null+++ b/desktop-app/test/e2e-playwright/plugin-workload-sdk-codex-fallback.spec.ts@@ -0,0 +1,244 @@+import { type ElectronApplication, type Page, expect, test } from '@playwright/test'+import {+ type EmbeddedContents,+ type SandboxUiFixture,+ activateEmbedded,+ assertSandboxUiFixture,+ assertSteplessSdkRecipePrecondition,+ embeddedText,+ findSandboxUiContents,+ latestSdkInvocationStatus,+ promptBridgeAttemptsForRun,+ promptBridgeGrantTargets,+ readPluginWorkloadSdkStatus,+ sandboxUiViewUrlPrefix,+ sdkInvocationCount,+ typeEmbedded,+} from './pluginWorkloadSdkSandboxUi'+import {+ EXTERNAL_REST_API_BASE_URL,+ RPC_PROXY_BASE_URL,+ assertAllowedTarget,+ desktopCredentials,+ finalizeRecording,+ launchDesktopApp,+ login,+ requireRecorderConfirm,+} from './qa-recorder-helpers'+import { expectDeploymentReady, kubectlOut } from './workflow-approval-quadrants/cluster'++/*+ * E2E_GUARDIAN_IPC_FLOW: Desktop Apps opens the Plugin Workload SDK sandbox UI+ * through the main-process WebContentsView and EmbeddedContents bridge. The+ * renderer has no HTTP response to await for that transition; the prompt+ * output and the persisted dual ledger are the visible/business signals.+ *+ * Acceptance criterion 8 of issue #533: "a real user-journey E2E covers+ * sdk-only -> codex-subscription primary -> successful prompt, PLUS an eligible+ * failure that reaches a Z.ai fallback". The happy half lives in+ * plugin-workload-sdk-sandbox-ui.spec.ts. This file is the other half, and it+ * is the one that matters most, because the missing fallback WAS the original+ * bug: #533's root cause records that a local no_grant was wrapped as+ * provider_unavailable, classified as a non-retryable authentication failure,+ * and so "the ordered Z.ai fallback is never attempted".+ *+ * Fault injection: `codex-llm-proxy` is scaled to zero. The resulting fetch+ * rejection carries no provider code, so mcp-host's classifyError falls through+ * to classifyUnknown -> {ApiCallFailed, retryable:true} -> failover class+ * `provider_unavailable`, which is in the default triggerOn set. A more+ * specific fault (revoked credential, disallowed model) would classify as+ * non-eligible and propagate WITHOUT failover, proving nothing. control-api's+ * authorize path never contacts the proxy, so attempt 0 is still reserved and+ * the displaced-attempt bookkeeping is exercised for real.+ *+ * This journey therefore also covers `settlePriorProviderAttemptFloors`, the+ * server-side prior-attempt discovery: attempt 0 never reaches finalize when a+ * later target wins, so its spend floor exists only if that path ran.+ *+ * Its own file, its own Make target: the two sandbox journeys previously shared+ * module-scope state, and this one additionally mutates cluster state.+ */+const RUN_ENABLED =+ process.env.E2E_PLUGIN_SDK_DESKTOP === '1' && process.env.E2E_PLUGIN_SDK_CODEX_FALLBACK === '1'++test.skip(+ !RUN_ENABLED,+ 'Set E2E_PLUGIN_SDK_DESKTOP=1 and E2E_PLUGIN_SDK_CODEX_FALLBACK=1 for the Codex fallback journey.'+)++const PROXY_NAMESPACE = 'control-plane'+const PROXY_DEPLOYMENT = 'codex-llm-proxy'++/**+ * The granted happy-path fixture. The fallback needs a working grant with an+ * ordered target list, which is exactly what that recipe already has — unlike+ * the no-grant guard, this journey has no reason to install a second recipe.+ */+function grantedFixture(): SandboxUiFixture {+ return assertSandboxUiFixture(+ {+ recipeName: process.env.E2E_PLUGIN_SDK_RECIPE_NAME || 'evenfire-prompt-notify-app',+ recipeNamespace: process.env.E2E_PLUGIN_SDK_RECIPE_NAMESPACE || 'sandbox-recipes',+ appTitle: process.env.E2E_PLUGIN_SDK_APP_TITLE || 'Prompt & Notify',+ },+ 'E2E_PLUGIN_SDK'+ )+}++function scaleProxy(replicas: number): void {+ kubectlOut([+ '-n',+ PROXY_NAMESPACE,+ 'scale',+ `deployment/${PROXY_DEPLOYMENT}`,+ `--replicas=${replicas}`,+ ])+}++/**+ * Wait until no proxy Pod is left serving. Scaling returns immediately, so+ * prompting before termination completes would race the fault into a normal+ * success and silently turn this into a second happy-path run.+ */+async function expectProxyStopped(): Promise<void> {+ await expect+ .poll(+ () =>+ kubectlOut([+ '-n',+ PROXY_NAMESPACE,+ 'get',+ 'pods',+ '-l',+ `app=${PROXY_DEPLOYMENT}`,+ '-o',+ 'jsonpath={.items[*].metadata.name}',+ ]).trim(),+ { timeout: 120_000 }+ )+ .toBe('')+}++test('Desktop Apps falls back to the authorized non-Codex target when Codex is unavailable', async ({}, testInfo) => {+ requireRecorderConfirm(+ 'E2E_PLUGIN_SDK_WRITE_CONFIRM',+ 'This journey stops codex-llm-proxy, performs one paid promptBridge call served by the fallback provider, and restores the proxy.'+ )+ const fixture = grantedFixture()+ // Hard preconditions, never branches. RP-539-002 was exactly this shape: the+ // happy path kept its only Codex assertions behind a guard, so an OpenAI+ // recipe passed it green without touching Codex, and the static auditor+ // reported PASS because it validates form and cannot reach semantics.+ assertSteplessSdkRecipePrecondition(fixture, 'codex-subscription')+ await expect+ .poll(() => readPluginWorkloadSdkStatus(fixture).state, { timeout: 180_000 })+ .toBe('validated')+ expect(readPluginWorkloadSdkStatus(fixture).bootstrapContractVersion).toBe(3)++ // Without a second authorized target there is nowhere to fall to, and the+ // journey would pass while proving nothing.+ const targets = promptBridgeGrantTargets(fixture)+ expect(targets.length).toBeGreaterThanOrEqual(2)+ expect(targets[0]?.provider).toBe('codex-subscription')+ expect(targets[1]?.provider).not.toBe('codex-subscription')+ expect(targets[1]?.provider).not.toBe('')+ const fallbackProvider = targets[1]!.provider++ await assertAllowedTarget('EXTERNAL_REST_API_BASE_URL', EXTERNAL_REST_API_BASE_URL)+ await assertAllowedTarget('RPC_PROXY_BASE_URL', RPC_PROXY_BASE_URL)++ let app: ElectronApplication | undefined+ let page: Page | undefined+ let proxyStopped = false+ try {+ const credentials = desktopCredentials()+ const launched = await launchDesktopApp(testInfo)+ app = launched.app+ page = launched.page+ await login(page, credentials)+ await page.getByTestId('nav-sandbox-ui').click()+ await expect(page.getByRole('heading', { name: 'Apps', exact: true })).toBeVisible()+ const appCard = page+ .getByRole('main')+ .getByRole('button', { name: `Open ${fixture.appTitle}`, exact: true })+ await expect(appCard).toHaveCount(1)+ await expect(appCard).toBeVisible({ timeout: 30_000 })+ await appCard.click()+ await expect(page.getByRole('button', { name: 'Back to apps' })).toBeVisible({+ timeout: 30_000,+ })+ let embedded: EmbeddedContents | null = null+ await expect+ .poll(async () => {+ embedded = await findSandboxUiContents(app!, fixture)+ return embedded?.url ?? ''+ })+ .toContain(sandboxUiViewUrlPrefix(fixture))+ const webContentsId = embedded!.id++ await test.step('stop the Codex proxy so the primary target fails eligibly', async () => {+ scaleProxy(0)+ proxyStopped = true+ await expectProxyStopped()+ })++ const runStartedAt = new Date().toISOString()+ const before = sdkInvocationCount(fixture, 'promptBridge')+ await typeEmbedded(+ app,+ webContentsId,+ '#prompt',+ 'Reply with the single word OK. Codex is down; the authorized fallback must serve this.'+ )+ await activateEmbedded(app, webContentsId, '#run')+ await expect+ .poll(() => embeddedText(app!, webContentsId, '#prompt-out'), { timeout: 180_000 })+ .not.toMatch(/Running…/)++ const promptResult = await embeddedText(app, webContentsId, '#prompt-out')+ expect(promptResult.trim()).not.toBe('')+ expect(sdkInvocationCount(fixture, 'promptBridge')).toBe(before + 1)+ expect(latestSdkInvocationStatus(fixture, 'promptBridge')).toBe('complete')++ // The dual ledger is the business/audit signal: ordering, attempt limits+ // and spend fencing are all row-level facts here, not inferences from the+ // rendered text.+ const attempts = promptBridgeAttemptsForRun(fixture, runStartedAt)+ await testInfo.attach('promptBridge-attempts.json', {+ body: JSON.stringify(attempts, null, 2),+ contentType: 'application/json',+ })+ expect(attempts).toHaveLength(2)++ const [codexAttempt, fallbackAttempt] = attempts+ expect(codexAttempt!.attemptIndex).toBe(0)+ expect(codexAttempt!.provider).toBe('codex-subscription')+ expect(codexAttempt!.status).toBe('provider_unavailable')+ // The floor for a displaced Codex attempt is honest uncertainty. The fetch+ // rejection leaves providerDispatched undefined, which is read as+ // dispatched, so the ledger records `unknown` rather than claiming the+ // subscription was never touched. `exact` would mean a spend was proven.+ expect(codexAttempt!.spendOutcome).toBe('unknown')+ expect(codexAttempt!.spendOutcome).not.toBe('exact')++ expect(fallbackAttempt!.attemptIndex).toBe(1)+ expect(fallbackAttempt!.provider).toBe(fallbackProvider)+ expect(fallbackAttempt!.provider).not.toBe('codex-subscription')+ // `complete` is the success terminal for a physical attempt; the schema's+ // CHECK allows only reserved/in_progress/complete/failed/+ // provider_unavailable/skipped.+ expect(fallbackAttempt!.status).toBe('complete')+ expect(fallbackAttempt!.codexAttemptId).toBe('')++ await page.getByRole('button', { name: 'Back to apps' }).click()+ await expect(page.getByRole('heading', { name: 'Apps', exact: true })).toBeVisible()+ } finally {+ if (proxyStopped) {+ // Fail loudly rather than leave the cluster degraded. A silent restore+ // failure would poison every later lane on this profile.+ scaleProxy(1)+ expectDeploymentReady(PROXY_NAMESPACE, PROXY_DEPLOYMENT)+ }+ await finalizeRecording(app, page)+ }+})diff --git a/desktop-app/test/e2e-playwright/pluginWorkloadSdkSandboxUi.ts b/desktop-app/test/e2e-playwright/pluginWorkloadSdkSandboxUi.tsindex bf51358c0..14587e744 100644--- a/desktop-app/test/e2e-playwright/pluginWorkloadSdkSandboxUi.ts+++ b/desktop-app/test/e2e-playwright/pluginWorkloadSdkSandboxUi.ts@@ -572,6 +572,107 @@ export function promptBridgeLedgerForRun(}}+export type PromptBridgeAttemptRow = {+ attemptIndex: number+ targetRef: string+ provider: string+ model: string+ status: string+ spendOutcome: string+ codexAttemptId: string+}++/**+ * Every physical attempt of the run's first promptBridge invocation, ordered by+ * `attempt_index`.+ *+ * `promptBridgeLedgerForRun` answers "what did the served attempt record", so+ * it stops at `LIMIT 1`. A failover journey needs the opposite: the displaced+ * attempt is the evidence. Attempt 0 never reaches `finalize` when a later+ * target wins, so its spend floor is written by+ * `settlePriorProviderAttemptFloors` — the server-side discovery that replaced+ * a client-declared `priorAttempts` field. Reading every row is what proves+ * that path ran at all.+ */+export function promptBridgeAttemptsForRun(+ fixture: SandboxUiFixture,+ startedAt: string+): PromptBridgeAttemptRow[] {+ const raw = profilesSql(`+ SELECT sdk.attempt_index::text || '|' || sdk.target_ref || '|' || sdk.provider || '|' ||+ sdk.model || '|' || sdk.status || '|' || coalesce(spend.outcome, '') || '|' ||+ coalesce(codex.id::text, '')+ FROM plugin_workload_sdk_invocations inv+ JOIN plugin_workload_sdk_provider_attempts sdk+ ON sdk.invocation_id = inv.id+ LEFT JOIN llm_provider_attempts codex+ ON codex.plugin_workload_sdk_provider_attempt_id = sdk.id+ LEFT JOIN plugin_workload_sdk_spend_outcomes spend+ ON spend.provider_attempt_id = sdk.id+ WHERE inv.recipe_namespace = ${sqlLiteral(fixture.recipeNamespace)}+ AND inv.recipe_name = ${sqlLiteral(fixture.recipeName)}+ AND inv.method = 'promptBridge'+ AND inv.created_at >= ${sqlLiteral(startedAt)}::timestamptz+ AND inv.id = (+ SELECT id FROM plugin_workload_sdk_invocations+ WHERE recipe_namespace = ${sqlLiteral(fixture.recipeNamespace)}+ AND recipe_name = ${sqlLiteral(fixture.recipeName)}+ AND method = 'promptBridge'+ AND created_at >= ${sqlLiteral(startedAt)}::timestamptz+ ORDER BY created_at ASC+ LIMIT 1+ )+ ORDER BY sdk.attempt_index ASC;+ `)+ if (!raw.trim()) return []+ return raw+ .trim()+ .split('\n')+ .map(line => {+ const [attemptIndex, targetRef, provider, model, status, spendOutcome, codexAttemptId] = line+ .trim()+ .split('|')+ return {+ attemptIndex: Number.parseInt(attemptIndex ?? '', 10),+ targetRef: targetRef ?? '',+ provider: provider ?? '',+ model: model ?? '',+ status: status ?? '',+ spendOutcome: spendOutcome ?? '',+ codexAttemptId: codexAttemptId ?? '',+ }+ })+}++/**+ * The ordered promptBridge targets on the live grant.+ *+ * A failover journey that runs against a single-target grant would pass while+ * proving nothing — there would be nowhere to fall to. This is asserted before+ * the fault is injected so an unprovisioned fixture fails as a precondition,+ * not as a mysteriously green run.+ */+export function promptBridgeGrantTargets(+ fixture: SandboxUiFixture+): Array<{ targetRef: string; provider: string }> {+ const raw = profilesSql(`+ SELECT string_agg(t->>'targetRef' || '|' || (t->>'provider'), E'\\n' ORDER BY ord)+ FROM plugin_workload_sdk_grants g+ CROSS JOIN LATERAL jsonb_array_elements(g.prompt_targets) WITH ORDINALITY AS a(t, ord)+ WHERE g.recipe_namespace = ${sqlLiteral(fixture.recipeNamespace)}+ AND g.recipe_name = ${sqlLiteral(fixture.recipeName)}+ AND g.capability_family = 'promptBridge';+ `)+ if (!raw.trim()) return []+ return raw+ .trim()+ .split('\n')+ .map(line => {+ const [targetRef, provider] = line.trim().split('|')+ return { targetRef: targetRef ?? '', provider: provider ?? '' }+ })+}+export function latestSdkInvocationStatus(fixture: SandboxUiFixture,method: 'promptBridge' | 'clientNotifications'diff --git a/docs/testing/optional-playwright-qa-recorder.md b/docs/testing/optional-playwright-qa-recorder.mdindex fc3876cb7..0a577bd70 100644--- a/docs/testing/optional-playwright-qa-recorder.md+++ b/docs/testing/optional-playwright-qa-recorder.md@@ -155,6 +155,47 @@ there is none to declare. **Provisioning a Codex recipe with no executionbinding is operator work** — the lane will not create one, and a lane thatcannot find its fixture fails rather than skipping.+### Codex fallback lane++`plugin-workload-sdk-codex-fallback.spec.ts` closes acceptance criterion 8 of+issue #533: a real journey where an eligible Codex failure reaches the+authorized non-Codex fallback. It runs against the **granted** happy-path+recipe, because the fallback needs a working ordered target list.++```bash+E2E_PLUGIN_SDK_WRITE_CONFIRM=1 \+E2E_PLUGIN_SDK_EXPECT_PROVIDER=codex-subscription \+CONTROL_UI_BASE_URL=http://127.0.0.1:<random-control-ui-port> \+CONTROL_API_BASE_URL=http://127.0.0.1:<random-control-api-port> \+EXTERNAL_REST_API_BASE_URL=http://127.0.0.1:<random-external-rest-port> \+RPC_PROXY_BASE_URL=http://127.0.0.1:<random-rpc-port> \+KUBECONTEXT=<branch-profile-context> \+make test-e2e-plugin-workload-sdk-desktop-codex-fallback+```++**This lane mutates cluster state.** It scales `control-plane/codex-llm-proxy`+to zero replicas so the primary target fails with a class the failover engine+accepts, then restores it to one. The restore runs twice — in the spec's+`finally` and again in a Make-level `EXIT` trap — and both report loudly on+failure. If you ever see `[E2E-GUARD] FAILED to restore+control-plane/codex-llm-proxy`, restore it before running any other lane on+that profile: every Codex journey after it would fail for the wrong reason.++`E2E_PLUGIN_SDK_EXPECT_PROVIDER` must be exactly `codex-subscription`; the+target rejects any other value, because a non-Codex primary would make the+journey vacuous. The recipe knobs are shared with the happy path and carry+**silent defaults** — `E2E_PLUGIN_SDK_RECIPE_NAME` defaults to+`evenfire-prompt-notify-app`, `E2E_PLUGIN_SDK_RECIPE_NAMESPACE` to+`sandbox-recipes`, and `E2E_PLUGIN_SDK_APP_TITLE` to `Prompt & Notify`. Set+them explicitly whenever your fixture differs, or the lane looks for the wrong+app in the Desktop catalog and fails at the card locator rather than at a+precondition.++The grant must expose **at least two ordered prompt targets**, Codex first and+a non-Codex provider second. The spec asserts that before injecting the fault:+a single-target grant would let the journey pass while proving nothing, so it+fails as a precondition instead.+## Repository isolationThe recorder adds new commands; it does not alter any existing command: