Skip to content

πŸ™Œ OpenHands: AI-Driven Development

fix(e2e): let the live Agent Server E2E job run and finish (#18084)

Engel Nyst64fe658d51a7
diff --git a/.agents/skills/e2e-testing/references/guide.md b/.agents/skills/e2e-testing/references/guide.md
index 8078fb2215..23e472ae7d 100644
--- a/.agents/skills/e2e-testing/references/guide.md
+++ b/.agents/skills/e2e-testing/references/guide.md
@@ -5,8 +5,8 @@
- `npm run test:e2e:live` loads `.env` through Node's `--env-file-if-exists` flag and invokes `tests/e2e/live/scripts/run-live-e2e.mjs`. The runner validates the required local environment, explains missing credentials/prerequisites, and then runs `playwright test --config=playwright.live.config.ts`. Use `npm run test:e2e:live -- --check` to validate local setup without running the test, and pass Playwright flags after `--` (for example `npm run test:e2e:live -- --headed`).
- Local live E2E requires one LLM credential: `LIVE_E2E_LLM_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `LLM_API_KEY`. Optional overrides are `LIVE_E2E_LLM_BASE_URL`, `LIVE_E2E_LLM_MODEL`, `LIVE_E2E_SESSION_API_KEY`, `LIVE_E2E_BACKEND_URL`, and `LIVE_E2E_FRONTEND_PORT`. The local runner prints which variables are missing without printing secret values.
- Live-test-only helpers belong under `tests/e2e/live/utils/`. The current helper module is `tests/e2e/live/utils/agent-server-conversation.ts`; do not put live-only helpers in the shared `tests/e2e/support/` directory.
-- `playwright.live.config.ts` starts the real local Agent Server/UI stack via `npm run dev:minimal`, not MSW mocks. It uses `LIVE_E2E_SESSION_API_KEY` when set, otherwise generates a per-run random session key and passes it through `SESSION_API_KEY`, `OH_SESSION_API_KEYS_0`, and `VITE_SESSION_API_KEY`; specs that need direct backend requests must inject `X-Session-API-Key` only for the configured backend origin through `routeBackendSessionApiKey(page)`, never through global Playwright `extraHTTPHeaders`. Live tests default to frontend port `3101` and Agent Server `http://127.0.0.1:18100` so they do not accidentally reuse a normal local dev stack.
-- `tests/e2e/live/utils/agent-server-conversation.ts` configures the running Agent Server before each live conversation by PATCHing `${LIVE_E2E_BACKEND_URL ?? "http://127.0.0.1:18100"}/api/settings` with LLM settings and low-risk conversation settings. LLM credentials are read from `LIVE_E2E_LLM_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `LLM_API_KEY`; CI defaults use `LIVE_E2E_LLM_BASE_URL` (default `https://llm-proxy.app.all-hands.dev`) and `LIVE_E2E_LLM_MODEL` (default `openhands/claude-haiku-4-5-20251001`).
+- `playwright.live.config.ts` starts the real local Agent Server/UI stack via `npm run dev:minimal`, not MSW mocks. It uses `LIVE_E2E_SESSION_API_KEY` when set, otherwise generates a per-run random session key and passes it through `SESSION_API_KEY`, `OH_SESSION_API_KEYS_0`, and `VITE_SESSION_API_KEY`; specs that need direct backend requests must inject `X-Session-API-Key` only for the configured backend origin through `routeBackendSessionApiKey(page)`, never through global Playwright `extraHTTPHeaders`. Live tests default to frontend port `3101` and Agent Server `http://127.0.0.1:18100` so they do not accidentally reuse a normal local dev stack. The config also sets `TMUX_TMPDIR=/tmp/oh-live-e2e-tmux-<uid>`: tmux binds `<TMUX_TMPDIR>/tmux-<uid>/openhands-<32 hex>`, and under the checkout's `.tmp/live-e2e-state` that path exceeds the 104/108-byte Unix socket limit on CI runners, which makes every conversation's terminal fail to start.
+- `tests/e2e/live/utils/agent-server-conversation.ts` configures the running Agent Server before each live conversation by PATCHing `${LIVE_E2E_BACKEND_URL ?? "http://127.0.0.1:18100"}/api/settings` with LLM settings and low-risk conversation settings. LLM credentials are read from `LIVE_E2E_LLM_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `LLM_API_KEY`; CI defaults use `LIVE_E2E_LLM_BASE_URL` (default `https://llm-proxy.app.all-hands.dev`) and `LIVE_E2E_LLM_MODEL` (default `openhands/claude-haiku-4-5-20251001`). The critic spec passes only `critic_api_key` and keeps the Agent Server's default critic endpoint and model (the proxy's `/vllm` classification service); do not point `critic_server_url` at the chat base URL, which has no `/classify` route.
- The live conversation test should stay cheap and as deterministic as possible while still exercising one real tool call: it asks the model to run the exact `EXPECTED_BASH_COMMAND`, waits for the bash output token to appear outside the user's message in the UI, confirms a successful `ExecuteBashObservation`/`TerminalObservation` through the real Agent Server events API, and then waits for the final `EXPECTED_REPLY_TOKEN`. This exercises the real UI, Agent Server settings API, conversation creation, websocket/event path, terminal tool execution, and LLM response path. Because LLM behavior is not perfectly deterministic even at temperature 0, CI keeps one retry for live E2E; future live tests should document any expected variance and avoid prompts that require unnecessary formatting obedience.
- Live E2E must not pollute analytics. `playwright.live.config.ts` starts the app with `VITE_DO_NOT_TRACK=1`; the live helper seeds local storage with telemetry/analytics opt-out values before app code runs; and each live spec should install `guardAgainstPostHogRequests(page)` before navigation so any attempted request to `*.posthog.com` or `z.openhands.dev` is blocked locally and fails the test.
- Live Playwright videos are intentionally recorded for CI QA debugging when `LIVE_E2E_RECORD_VIDEO=on` is set; local default video mode is `retain-on-failure`. Do not add live tests that render API keys, tokens, secret values, or credential-bearing error messages in the browser. Screenshots should target a safe app/chat region such as `data-testid="chat-interface"` instead of `page.screenshot({ fullPage: true })`, and should apply `getLiveArtifactMask(page)` for text/field redaction; if a future live test must exercise sensitive UI, change that test/media path to redact the sensitive output or retain video only on failure.
diff --git a/playwright.live.config.ts b/playwright.live.config.ts
index 4312430f01..6ec543568d 100644
--- a/playwright.live.config.ts
+++ b/playwright.live.config.ts
@@ -18,6 +18,10 @@ try {
const liveE2EFrontendURL = `http://localhost:${liveE2EFrontendPort}/`;
const liveE2EVideoMode =
process.env.LIVE_E2E_RECORD_VIDEO === "on" ? "on" : "retain-on-failure";
+// tmux binds <TMUX_TMPDIR>/tmux-<uid>/openhands-<32 hex>. Under this
+// checkout's .tmp state dir that exceeds the 104/108-byte Unix socket path
+// limit on CI runners, and every conversation then fails to start its terminal.
+const liveE2ETmuxTmpDir = `/tmp/oh-live-e2e-tmux-${process.getuid?.() ?? 0}`;
function shellQuote(value: string) {
return `'${value.replaceAll("'", "'\\''")}'`;
@@ -60,6 +64,7 @@ export default defineConfig({
"node -e \"const fs=require('node:fs'); for (const p of ['.tmp/live-e2e-state','node_modules/.vite']) fs.rmSync(p,{recursive:true,force:true});\" && " +
[
"OH_CANVAS_SAFE_STATE_DIR=.tmp/live-e2e-state",
+ envAssignment("TMUX_TMPDIR", liveE2ETmuxTmpDir),
envAssignment("LOCAL_BACKEND_API_KEY", liveE2ESessionApiKey),
envAssignment("OH_CANVAS_SAFE_BACKEND_PORT", liveE2EBackendPort),
"VITE_DO_NOT_TRACK=1",
@@ -70,5 +75,10 @@ export default defineConfig({
url: liveE2EFrontendURL,
timeout: 120_000,
reuseExistingServer: false,
+ // Playwright's default teardown SIGKILLs the shell's process group, but
+ // dev-safe.mjs spawns the Agent Server detached, so it survives holding
+ // the webServer's stdout and Playwright waits on it until the CI job
+ // times out. SIGTERM lets dev-safe.mjs stop its services first.
+ gracefulShutdown: { signal: "SIGTERM", timeout: 15_000 },
},
});
diff --git a/tests/e2e/live/utils/agent-server-conversation.ts b/tests/e2e/live/utils/agent-server-conversation.ts
index fce332ab31..bf11b9f303 100644
--- a/tests/e2e/live/utils/agent-server-conversation.ts
+++ b/tests/e2e/live/utils/agent-server-conversation.ts
@@ -197,24 +197,21 @@ function buildLiveLlmSettings(): Record<string, string | number> {
function buildLiveVerificationSettings(
options: ConfigureLiveAgentServerOptions = {},
-): Record<string, string | number | boolean> {
- const verificationSettings: Record<string, string | number | boolean> =
- options.enableCritic
- ? {
- critic_enabled: true,
- critic_mode: "finish_and_message",
- enable_iterative_refinement: false,
- critic_api_key: llmApiKey,
- critic_model_name: llmModel,
- }
- : {
- critic_enabled: false,
- enable_iterative_refinement: false,
- };
- if (options.enableCritic && llmBaseUrl) {
- verificationSettings.critic_server_url = llmBaseUrl;
- }
- return verificationSettings;
+): Record<string, string | boolean> {
+ // The critic is a separate classification service, not the chat model:
+ // keep the Agent Server's default endpoint and model, which serve
+ // /vllm/classify on the LLM proxy (its root has no /classify route).
+ return options.enableCritic
+ ? {
+ critic_enabled: true,
+ critic_mode: "finish_and_message",
+ enable_iterative_refinement: false,
+ critic_api_key: llmApiKey,
+ }
+ : {
+ critic_enabled: false,
+ enable_iterative_refinement: false,
+ };
}
export async function createLiveConversation(