π OpenHands: AI-Driven Development
mock-llm e2e: use bin/agent-canvas.mjs (production binary path) (#906)
Rohit Malhotracaef236c1037
diff --git a/.github/workflows/mock-llm-e2e.yml b/.github/workflows/mock-llm-e2e.ymlindex 9d136a7077..09f8b12d97 100644--- a/.github/workflows/mock-llm-e2e.yml+++ b/.github/workflows/mock-llm-e2e.yml@@ -95,6 +95,9 @@ jobs:-d '{"model":"test","messages":[]}' | python3 -m json.toolkill $SERVER_PID+ - name: Build frontend (for agent-canvas binary)+ run: npm run build:app+- name: Run mock-LLM E2E testsid: run_testsenv:diff --git a/AGENTS.md b/AGENTS.mdindex 0398b79f78..5769b73141 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -142,6 +142,18 @@ you are running inside of β NOT the automation backend.- The live reporting scripts live beside the live tests under `tests/e2e/live/scripts/`: `run-live-e2e.mjs`, `extract-live-e2e-media.mjs`, `render-live-e2e-report.mjs`, and `upsert-pr-comment.mjs`. Keep report/comment/local-runner logic there rather than in top-level `scripts/`, because these scripts are part of the live E2E framework.- When changing any part of this framework β live workflow triggers, artifact publishing, `.pr` cleanup, live Playwright config, live test file layout, helper locations, local runner behavior, or report/comment scripts β update this `AGENTS.md` section in the same PR so future agents have the current operating model.+## Mock-LLM E2E Tests++- Mock-LLM tests live under `tests/e2e/mock-llm/` and exercise the complete stack β from the browser through the real agent-server to a scripted mock LLM server β without any real LLM credentials. Run locally with `npm run test:e2e:mock-llm`.+- **Production-fidelity launch**: The Playwright config (`playwright.mock-llm.config.ts`) starts the full `agent-canvas` stack via `bin/agent-canvas.mjs` β the same binary that `npx @openhands/agent-canvas` executes when users install the npm package. This means mock-LLM tests exercise the actual production path: pre-built static frontend + static-server.mjs + agent-server via uvx + automation backend via uvx + ingress proxy, all behind a single port.+- A pre-built `build/` directory is required. The Playwright webServer command runs `npm run build:app` when `build/index.html` is absent, but CI should run the build step explicitly for caching (`npm run build:app` in `.github/workflows/mock-llm-e2e.yml`).+- **Single ingress URL**: Tests use one URL for both the browser (`baseURL`) and backend API assertions (`BACKEND_URL`). The ingress proxy routes `/api/*` to the agent-server, `/api/automation/*` to the automation backend, and `/*` to the static frontend. Default ingress port for tests is `18300` (override via `MOCK_LLM_INGRESS_PORT` env var).+- **State isolation**: `OH_CANVAS_SAFE_STATE_DIR=.tmp/mock-llm-state` isolates test state from the user's real `~/.openhands/agent-canvas/` directory. The directory is cleaned before each test run.+- **Session API key**: A random key is generated per test run and passed to the stack via `SESSION_API_KEY` / `OH_SESSION_API_KEYS_0` / `VITE_SESSION_API_KEY`. The static server injects it into `index.html` at serve time so the frontend authenticates automatically.+- The mock LLM server (`tests/e2e/mock-llm/scripts/mock-llm-server.py`) uses openhands-sdk's `TestLLM` and runs on port 9999 (override via `MOCK_LLM_PORT`). It returns scripted responses (a tool call followed by a text reply) regardless of the prompt.+- CI workflow: `.github/workflows/mock-llm-e2e.yml` runs on PRs with the `e2e-tests` label or on manual dispatch. It builds the frontend, starts the mock LLM server, runs the tests, and posts a PR comment with results.+- The custom `DoneMarkerReporter` writes `.mock-llm-markers/.tests-done` after all tests complete (before webServer teardown) so the CI wrapper can detect completion and kill the lingering teardown process.+## Additional Notes- **Published binary auth fix**: When users install the npm package globally (`npm install -g @openhands/agent-canvas`) and run `agent-canvas`, the pre-built static frontend has a `VITE_SESSION_API_KEY` baked in at publish time that differs from the user's persisted runtime key (`~/.openhands/agent-canvas/session-api-key.txt`). The fix is to inject the runtime session key into `index.html` responses at serve time (not build time). `scripts/static-server.mjs` accepts a `--session-api-key <key>` flag and injects a tiny inline `<script>` before `</head>` that seeds the key into `localStorage['openhands-agent-server-config'].sessionApiKey` β only if no key is already stored (preserving user-set overrides). `scripts/dev-with-automation.mjs` and `scripts/dev-static.mjs` both pass `--session-api-key ${config.sessionApiKey}` when starting the static server.diff --git a/playwright.mock-llm.config.ts b/playwright.mock-llm.config.tsindex 77d0352f3a..e77d043e22 100644--- a/playwright.mock-llm.config.ts+++ b/playwright.mock-llm.config.ts@@ -3,19 +3,31 @@** Starts two processes:* 1. Mock LLM server (Python, using openhands-sdk TestLLM)- * 2. Agent-server + frontend (via npm run dev:minimal)+ * 2. Full agent-canvas stack via bin/agent-canvas.mjs (agent-server ++ * automation backend + static frontend + ingress proxy), matching the+ * production npm-published binary.** The test creates an LLM profile via the UI that points at the mock server,* so no real LLM credentials are needed.+ *+ * A pre-built `build/` directory is required β the Playwright webServer+ * command runs `npm run build:app` when `build/index.html` is absent.+ * CI should run the build step explicitly before the tests for caching.*/import { defineConfig, devices } from "@playwright/test";import { randomBytes } from "node:crypto";-// ββ Port allocation (separate from live E2E to avoid collisions) βββββββ+// ββ Port allocation (separate from live E2E / dev to avoid collisions) βconst MOCK_LLM_PORT = process.env.MOCK_LLM_PORT ?? "9999";-const BACKEND_PORT = process.env.MOCK_LLM_BACKEND_PORT ?? "18200";-const FRONTEND_PORT = process.env.MOCK_LLM_FRONTEND_PORT ?? "3102";++// The agent-canvas binary exposes a single ingress port that routes:+// /api/automation/* β automation backend+// /api/*, /sockets β agent-server+// /* β static frontend+// Tests use this single URL for both the browser (baseURL) and backend API+// calls (the ingress proxies /api/* transparently).+const INGRESS_PORT = process.env.MOCK_LLM_INGRESS_PORT ?? "18300";// ββ Session API key ββββββββββββββββββββββββββββββββββββββββββββββββββββconst sessionApiKey =@@ -23,9 +35,11 @@ const sessionApiKey =randomBytes(32).toString("hex");process.env.MOCK_LLM_SESSION_API_KEY = sessionApiKey;+// ββ State directory (isolated per test run) ββββββββββββββββββββββββββββ+const STATE_DIR = ".tmp/mock-llm-state";+// ββ URLs βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ-const FRONTEND_URL = `http://localhost:${FRONTEND_PORT}/`;-const BACKEND_URL = `http://127.0.0.1:${BACKEND_PORT}`;+const INGRESS_URL = `http://localhost:${INGRESS_PORT}/`;const MOCK_LLM_URL = `http://127.0.0.1:${MOCK_LLM_PORT}`;// Python binary for the mock server β defaults to "python3" but CI can@@ -33,8 +47,9 @@ const MOCK_LLM_URL = `http://127.0.0.1:${MOCK_LLM_PORT}`;// PEP 668 "externally managed" errors on Ubuntu 24.04+.const MOCK_LLM_PYTHON = process.env.MOCK_LLM_PYTHON ?? "python3";-// Export for the test helpers-process.env.MOCK_LLM_BACKEND_URL = BACKEND_URL;+// Export for the test helpers β BACKEND_URL points to the ingress (API+// calls are proxied to the agent-server, so no direct backend port needed).+process.env.MOCK_LLM_BACKEND_URL = `http://localhost:${INGRESS_PORT}`;process.env.MOCK_LLM_PORT = MOCK_LLM_PORT;process.env.VITE_SESSION_API_KEY = sessionApiKey;@@ -63,7 +78,7 @@ export default defineConfig({],outputDir: "test-results-mock-llm",use: {- baseURL: FRONTEND_URL,+ baseURL: INGRESS_URL,screenshot: "only-on-failure",trace: "on-first-retry",video: "retain-on-failure",@@ -84,36 +99,38 @@ export default defineConfig({stdout: "pipe",stderr: "pipe",},- // 2. Agent-server + Vite frontend via dev-safe.mjs+ // 2. Full agent-canvas stack via bin/agent-canvas.mjs//- // `exec` replaces the shell so Playwright's tracked PID IS the node- // process. SIGTERM goes directly to dev-safe.mjs's signal handler,- // which kills children via process groups and exits cleanly.+ // This mirrors the production `npx @openhands/agent-canvas` path:+ // - Pre-built static frontend served via static-server.mjs+ // - Agent-server via uvx+ // - Automation backend via uvx+ // - Ingress proxy unifying all routes on a single port//- // Previously this used `exec ... npm run dev:minimal`, but npm does- // NOT forward SIGTERM to its child processes β the node process and- // its spawned agent-server/vite survived as orphans, causing the- // webServer teardown to hang indefinitely.+ // `exec` replaces the shell so Playwright's tracked PID IS the node+ // process. SIGTERM goes directly to the shutdown handler, which+ // kills children via process groups and exits cleanly.{command:// Clean state dir to avoid stale profile/conversation data between runs- "node -e \"const fs=require('node:fs'); for (const p of ['.tmp/mock-llm-state','node_modules/.vite']) fs.rmSync(p,{recursive:true,force:true});\" && " ++ `node -e "const fs=require('node:fs'); fs.rmSync('${STATE_DIR}',{recursive:true,force:true});" && ` ++ // Build frontend if not already built (CI should pre-build for caching)+ '[ -f build/index.html ] || npm run build:app && ' +["exec env",- "OH_CANVAS_SAFE_STATE_DIR=.tmp/mock-llm-state",+ envAssignment("OH_CANVAS_SAFE_STATE_DIR", STATE_DIR),+ envAssignment("PORT", INGRESS_PORT),envAssignment("SESSION_API_KEY", sessionApiKey),envAssignment("OH_SESSION_API_KEYS_0", sessionApiKey),- envAssignment("OH_CANVAS_SAFE_BACKEND_PORT", BACKEND_PORT),envAssignment("VITE_SESSION_API_KEY", sessionApiKey),"VITE_DO_NOT_TRACK=1","VITE_ENABLE_BROWSER_TOOLS=false",- envAssignment("VITE_FRONTEND_PORT", FRONTEND_PORT),// Bypass npm β exec directly into node so SIGTERM reaches- // dev-safe.mjs's shutdown handler (npm swallows it).- "node --env-file-if-exists=.env scripts/dev-safe.mjs",+ // the shutdown handler (npm swallows it).+ "node --env-file-if-exists=.env bin/agent-canvas.mjs",].join(" "),- url: FRONTEND_URL,- timeout: 120_000,+ url: INGRESS_URL,+ timeout: 180_000, // allow extra time for build + agent-server + automation startupreuseExistingServer: !process.env.CI,},],diff --git a/scripts/dev-with-automation.mjs b/scripts/dev-with-automation.mjsindex cf41bd1461..7c61ef9ca4 100644--- a/scripts/dev-with-automation.mjs+++ b/scripts/dev-with-automation.mjs@@ -330,7 +330,9 @@ async function buildConfig(args, env = process.env) {// Session API key β shared by both agent-server and automation backend.// Both validate it via the `X-Session-API-Key` header.- const stateDir = join(homedir(), ".openhands", "agent-canvas");+ const stateDir =+ env.OH_CANVAS_SAFE_STATE_DIR ||+ join(homedir(), ".openhands", "agent-canvas");const safeConfig = buildSafeDevConfig(projectRoot, {...env,OH_CANVAS_SAFE_STATE_DIR: stateDir,diff --git a/tests/e2e/mock-llm/utils/mock-llm-helpers.ts b/tests/e2e/mock-llm/utils/mock-llm-helpers.tsindex f8f14e0448..97e3b0216a 100644--- a/tests/e2e/mock-llm/utils/mock-llm-helpers.ts+++ b/tests/e2e/mock-llm/utils/mock-llm-helpers.ts@@ -17,11 +17,13 @@ export const REPLY_TOKEN = "MOCK_LLM_E2E_REPLY_OK";export const BASH_COMMAND = `printf '${BASH_TOKEN}\\n'`;// Ports / URLs β set via env or defaults matching playwright.mock-llm.config.ts.+// The agent-canvas binary exposes a single ingress port; API calls are proxied+// through it, so BACKEND_URL = ingress URL (no separate backend port).export const MOCK_LLM_PORT =process.env.MOCK_LLM_PORT ?? "9999";export const MOCK_LLM_BASE_URL = `http://127.0.0.1:${MOCK_LLM_PORT}`;export const BACKEND_URL =- process.env.MOCK_LLM_BACKEND_URL ?? "http://127.0.0.1:18200";+ process.env.MOCK_LLM_BACKEND_URL ?? "http://localhost:18300";export const SESSION_API_KEY = (() => {const key =process.env.MOCK_LLM_SESSION_API_KEY ??