π OpenHands: AI-Driven Development
fix(e2e): let the live Agent Server E2E job run and finish (#18084)
Engel Nyst64fe658d51a7
diff --git a/.agents/skills/e2e-testing/references/guide.md b/.agents/skills/e2e-testing/references/guide.mdindex 8078fb2215..23e472ae7d 100644--- a/.agents/skills/e2e-testing/references/guide.md+++ b/.agents/skills/e2e-testing/references/guide.md@@ -5,8 +5,8 @@- `npm run test:e2e:live` loads `.env` through Node's `--env-file-if-exists` flag and invokes `tests/e2e/live/scripts/run-live-e2e.mjs`. The runner validates the required local environment, explains missing credentials/prerequisites, and then runs `playwright test --config=playwright.live.config.ts`. Use `npm run test:e2e:live -- --check` to validate local setup without running the test, and pass Playwright flags after `--` (for example `npm run test:e2e:live -- --headed`).- Local live E2E requires one LLM credential: `LIVE_E2E_LLM_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `LLM_API_KEY`. Optional overrides are `LIVE_E2E_LLM_BASE_URL`, `LIVE_E2E_LLM_MODEL`, `LIVE_E2E_SESSION_API_KEY`, `LIVE_E2E_BACKEND_URL`, and `LIVE_E2E_FRONTEND_PORT`. The local runner prints which variables are missing without printing secret values.- Live-test-only helpers belong under `tests/e2e/live/utils/`. The current helper module is `tests/e2e/live/utils/agent-server-conversation.ts`; do not put live-only helpers in the shared `tests/e2e/support/` directory.-- `playwright.live.config.ts` starts the real local Agent Server/UI stack via `npm run dev:minimal`, not MSW mocks. It uses `LIVE_E2E_SESSION_API_KEY` when set, otherwise generates a per-run random session key and passes it through `SESSION_API_KEY`, `OH_SESSION_API_KEYS_0`, and `VITE_SESSION_API_KEY`; specs that need direct backend requests must inject `X-Session-API-Key` only for the configured backend origin through `routeBackendSessionApiKey(page)`, never through global Playwright `extraHTTPHeaders`. Live tests default to frontend port `3101` and Agent Server `http://127.0.0.1:18100` so they do not accidentally reuse a normal local dev stack.-- `tests/e2e/live/utils/agent-server-conversation.ts` configures the running Agent Server before each live conversation by PATCHing `${LIVE_E2E_BACKEND_URL ?? "http://127.0.0.1:18100"}/api/settings` with LLM settings and low-risk conversation settings. LLM credentials are read from `LIVE_E2E_LLM_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `LLM_API_KEY`; CI defaults use `LIVE_E2E_LLM_BASE_URL` (default `https://llm-proxy.app.all-hands.dev`) and `LIVE_E2E_LLM_MODEL` (default `openhands/claude-haiku-4-5-20251001`).+- `playwright.live.config.ts` starts the real local Agent Server/UI stack via `npm run dev:minimal`, not MSW mocks. It uses `LIVE_E2E_SESSION_API_KEY` when set, otherwise generates a per-run random session key and passes it through `SESSION_API_KEY`, `OH_SESSION_API_KEYS_0`, and `VITE_SESSION_API_KEY`; specs that need direct backend requests must inject `X-Session-API-Key` only for the configured backend origin through `routeBackendSessionApiKey(page)`, never through global Playwright `extraHTTPHeaders`. Live tests default to frontend port `3101` and Agent Server `http://127.0.0.1:18100` so they do not accidentally reuse a normal local dev stack. The config also sets `TMUX_TMPDIR=/tmp/oh-live-e2e-tmux-<uid>`: tmux binds `<TMUX_TMPDIR>/tmux-<uid>/openhands-<32 hex>`, and under the checkout's `.tmp/live-e2e-state` that path exceeds the 104/108-byte Unix socket limit on CI runners, which makes every conversation's terminal fail to start.+- `tests/e2e/live/utils/agent-server-conversation.ts` configures the running Agent Server before each live conversation by PATCHing `${LIVE_E2E_BACKEND_URL ?? "http://127.0.0.1:18100"}/api/settings` with LLM settings and low-risk conversation settings. LLM credentials are read from `LIVE_E2E_LLM_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `LLM_API_KEY`; CI defaults use `LIVE_E2E_LLM_BASE_URL` (default `https://llm-proxy.app.all-hands.dev`) and `LIVE_E2E_LLM_MODEL` (default `openhands/claude-haiku-4-5-20251001`). The critic spec passes only `critic_api_key` and keeps the Agent Server's default critic endpoint and model (the proxy's `/vllm` classification service); do not point `critic_server_url` at the chat base URL, which has no `/classify` route.- The live conversation test should stay cheap and as deterministic as possible while still exercising one real tool call: it asks the model to run the exact `EXPECTED_BASH_COMMAND`, waits for the bash output token to appear outside the user's message in the UI, confirms a successful `ExecuteBashObservation`/`TerminalObservation` through the real Agent Server events API, and then waits for the final `EXPECTED_REPLY_TOKEN`. This exercises the real UI, Agent Server settings API, conversation creation, websocket/event path, terminal tool execution, and LLM response path. Because LLM behavior is not perfectly deterministic even at temperature 0, CI keeps one retry for live E2E; future live tests should document any expected variance and avoid prompts that require unnecessary formatting obedience.- Live E2E must not pollute analytics. `playwright.live.config.ts` starts the app with `VITE_DO_NOT_TRACK=1`; the live helper seeds local storage with telemetry/analytics opt-out values before app code runs; and each live spec should install `guardAgainstPostHogRequests(page)` before navigation so any attempted request to `*.posthog.com` or `z.openhands.dev` is blocked locally and fails the test.- Live Playwright videos are intentionally recorded for CI QA debugging when `LIVE_E2E_RECORD_VIDEO=on` is set; local default video mode is `retain-on-failure`. Do not add live tests that render API keys, tokens, secret values, or credential-bearing error messages in the browser. Screenshots should target a safe app/chat region such as `data-testid="chat-interface"` instead of `page.screenshot({ fullPage: true })`, and should apply `getLiveArtifactMask(page)` for text/field redaction; if a future live test must exercise sensitive UI, change that test/media path to redact the sensitive output or retain video only on failure.diff --git a/playwright.live.config.ts b/playwright.live.config.tsindex 4312430f01..6ec543568d 100644--- a/playwright.live.config.ts+++ b/playwright.live.config.ts@@ -18,6 +18,10 @@ try {const liveE2EFrontendURL = `http://localhost:${liveE2EFrontendPort}/`;const liveE2EVideoMode =process.env.LIVE_E2E_RECORD_VIDEO === "on" ? "on" : "retain-on-failure";+// tmux binds <TMUX_TMPDIR>/tmux-<uid>/openhands-<32 hex>. Under this+// checkout's .tmp state dir that exceeds the 104/108-byte Unix socket path+// limit on CI runners, and every conversation then fails to start its terminal.+const liveE2ETmuxTmpDir = `/tmp/oh-live-e2e-tmux-${process.getuid?.() ?? 0}`;function shellQuote(value: string) {return `'${value.replaceAll("'", "'\\''")}'`;@@ -60,6 +64,7 @@ export default defineConfig({"node -e \"const fs=require('node:fs'); for (const p of ['.tmp/live-e2e-state','node_modules/.vite']) fs.rmSync(p,{recursive:true,force:true});\" && " +["OH_CANVAS_SAFE_STATE_DIR=.tmp/live-e2e-state",+ envAssignment("TMUX_TMPDIR", liveE2ETmuxTmpDir),envAssignment("LOCAL_BACKEND_API_KEY", liveE2ESessionApiKey),envAssignment("OH_CANVAS_SAFE_BACKEND_PORT", liveE2EBackendPort),"VITE_DO_NOT_TRACK=1",@@ -70,5 +75,10 @@ export default defineConfig({url: liveE2EFrontendURL,timeout: 120_000,reuseExistingServer: false,+ // Playwright's default teardown SIGKILLs the shell's process group, but+ // dev-safe.mjs spawns the Agent Server detached, so it survives holding+ // the webServer's stdout and Playwright waits on it until the CI job+ // times out. SIGTERM lets dev-safe.mjs stop its services first.+ gracefulShutdown: { signal: "SIGTERM", timeout: 15_000 },},});diff --git a/tests/e2e/live/utils/agent-server-conversation.ts b/tests/e2e/live/utils/agent-server-conversation.tsindex fce332ab31..bf11b9f303 100644--- a/tests/e2e/live/utils/agent-server-conversation.ts+++ b/tests/e2e/live/utils/agent-server-conversation.ts@@ -197,24 +197,21 @@ function buildLiveLlmSettings(): Record<string, string | number> {function buildLiveVerificationSettings(options: ConfigureLiveAgentServerOptions = {},-): Record<string, string | number | boolean> {- const verificationSettings: Record<string, string | number | boolean> =- options.enableCritic- ? {- critic_enabled: true,- critic_mode: "finish_and_message",- enable_iterative_refinement: false,- critic_api_key: llmApiKey,- critic_model_name: llmModel,- }- : {- critic_enabled: false,- enable_iterative_refinement: false,- };- if (options.enableCritic && llmBaseUrl) {- verificationSettings.critic_server_url = llmBaseUrl;- }- return verificationSettings;+): Record<string, string | boolean> {+ // The critic is a separate classification service, not the chat model:+ // keep the Agent Server's default endpoint and model, which serve+ // /vllm/classify on the LLM proxy (its root has no /classify route).+ return options.enableCritic+ ? {+ critic_enabled: true,+ critic_mode: "finish_and_message",+ enable_iterative_refinement: false,+ critic_api_key: llmApiKey,+ }+ : {+ critic_enabled: false,+ enable_iterative_refinement: false,+ };}export async function createLiveConversation(