Skip to content

πŸ™Œ OpenHands: AI-Driven Development

mock-llm e2e: use bin/agent-canvas.mjs (production binary path) (#906)

Rohit Malhotracaef236c1037
diff --git a/.github/workflows/mock-llm-e2e.yml b/.github/workflows/mock-llm-e2e.yml
index 9d136a7077..09f8b12d97 100644
--- a/.github/workflows/mock-llm-e2e.yml
+++ b/.github/workflows/mock-llm-e2e.yml
@@ -95,6 +95,9 @@ jobs:
-d '{"model":"test","messages":[]}' | python3 -m json.tool
kill $SERVER_PID
+ - name: Build frontend (for agent-canvas binary)
+ run: npm run build:app
+
- name: Run mock-LLM E2E tests
id: run_tests
env:
diff --git a/AGENTS.md b/AGENTS.md
index 0398b79f78..5769b73141 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -142,6 +142,18 @@ you are running inside of β€” NOT the automation backend.
- The live reporting scripts live beside the live tests under `tests/e2e/live/scripts/`: `run-live-e2e.mjs`, `extract-live-e2e-media.mjs`, `render-live-e2e-report.mjs`, and `upsert-pr-comment.mjs`. Keep report/comment/local-runner logic there rather than in top-level `scripts/`, because these scripts are part of the live E2E framework.
- When changing any part of this framework β€” live workflow triggers, artifact publishing, `.pr` cleanup, live Playwright config, live test file layout, helper locations, local runner behavior, or report/comment scripts β€” update this `AGENTS.md` section in the same PR so future agents have the current operating model.
+## Mock-LLM E2E Tests
+
+- Mock-LLM tests live under `tests/e2e/mock-llm/` and exercise the complete stack β€” from the browser through the real agent-server to a scripted mock LLM server β€” without any real LLM credentials. Run locally with `npm run test:e2e:mock-llm`.
+- **Production-fidelity launch**: The Playwright config (`playwright.mock-llm.config.ts`) starts the full `agent-canvas` stack via `bin/agent-canvas.mjs` β€” the same binary that `npx @openhands/agent-canvas` executes when users install the npm package. This means mock-LLM tests exercise the actual production path: pre-built static frontend + static-server.mjs + agent-server via uvx + automation backend via uvx + ingress proxy, all behind a single port.
+- A pre-built `build/` directory is required. The Playwright webServer command runs `npm run build:app` when `build/index.html` is absent, but CI should run the build step explicitly for caching (`npm run build:app` in `.github/workflows/mock-llm-e2e.yml`).
+- **Single ingress URL**: Tests use one URL for both the browser (`baseURL`) and backend API assertions (`BACKEND_URL`). The ingress proxy routes `/api/*` to the agent-server, `/api/automation/*` to the automation backend, and `/*` to the static frontend. Default ingress port for tests is `18300` (override via `MOCK_LLM_INGRESS_PORT` env var).
+- **State isolation**: `OH_CANVAS_SAFE_STATE_DIR=.tmp/mock-llm-state` isolates test state from the user's real `~/.openhands/agent-canvas/` directory. The directory is cleaned before each test run.
+- **Session API key**: A random key is generated per test run and passed to the stack via `SESSION_API_KEY` / `OH_SESSION_API_KEYS_0` / `VITE_SESSION_API_KEY`. The static server injects it into `index.html` at serve time so the frontend authenticates automatically.
+- The mock LLM server (`tests/e2e/mock-llm/scripts/mock-llm-server.py`) uses openhands-sdk's `TestLLM` and runs on port 9999 (override via `MOCK_LLM_PORT`). It returns scripted responses (a tool call followed by a text reply) regardless of the prompt.
+- CI workflow: `.github/workflows/mock-llm-e2e.yml` runs on PRs with the `e2e-tests` label or on manual dispatch. It builds the frontend, starts the mock LLM server, runs the tests, and posts a PR comment with results.
+- The custom `DoneMarkerReporter` writes `.mock-llm-markers/.tests-done` after all tests complete (before webServer teardown) so the CI wrapper can detect completion and kill the lingering teardown process.
+
## Additional Notes
- **Published binary auth fix**: When users install the npm package globally (`npm install -g @openhands/agent-canvas`) and run `agent-canvas`, the pre-built static frontend has a `VITE_SESSION_API_KEY` baked in at publish time that differs from the user's persisted runtime key (`~/.openhands/agent-canvas/session-api-key.txt`). The fix is to inject the runtime session key into `index.html` responses at serve time (not build time). `scripts/static-server.mjs` accepts a `--session-api-key <key>` flag and injects a tiny inline `<script>` before `</head>` that seeds the key into `localStorage['openhands-agent-server-config'].sessionApiKey` β€” only if no key is already stored (preserving user-set overrides). `scripts/dev-with-automation.mjs` and `scripts/dev-static.mjs` both pass `--session-api-key ${config.sessionApiKey}` when starting the static server.
diff --git a/playwright.mock-llm.config.ts b/playwright.mock-llm.config.ts
index 77d0352f3a..e77d043e22 100644
--- a/playwright.mock-llm.config.ts
+++ b/playwright.mock-llm.config.ts
@@ -3,19 +3,31 @@
*
* Starts two processes:
* 1. Mock LLM server (Python, using openhands-sdk TestLLM)
- * 2. Agent-server + frontend (via npm run dev:minimal)
+ * 2. Full agent-canvas stack via bin/agent-canvas.mjs (agent-server +
+ * automation backend + static frontend + ingress proxy), matching the
+ * production npm-published binary.
*
* The test creates an LLM profile via the UI that points at the mock server,
* so no real LLM credentials are needed.
+ *
+ * A pre-built `build/` directory is required β€” the Playwright webServer
+ * command runs `npm run build:app` when `build/index.html` is absent.
+ * CI should run the build step explicitly before the tests for caching.
*/
import { defineConfig, devices } from "@playwright/test";
import { randomBytes } from "node:crypto";
-// ── Port allocation (separate from live E2E to avoid collisions) ───────
+// ── Port allocation (separate from live E2E / dev to avoid collisions) ─
const MOCK_LLM_PORT = process.env.MOCK_LLM_PORT ?? "9999";
-const BACKEND_PORT = process.env.MOCK_LLM_BACKEND_PORT ?? "18200";
-const FRONTEND_PORT = process.env.MOCK_LLM_FRONTEND_PORT ?? "3102";
+
+// The agent-canvas binary exposes a single ingress port that routes:
+// /api/automation/* β†’ automation backend
+// /api/*, /sockets β†’ agent-server
+// /* β†’ static frontend
+// Tests use this single URL for both the browser (baseURL) and backend API
+// calls (the ingress proxies /api/* transparently).
+const INGRESS_PORT = process.env.MOCK_LLM_INGRESS_PORT ?? "18300";
// ── Session API key ────────────────────────────────────────────────────
const sessionApiKey =
@@ -23,9 +35,11 @@ const sessionApiKey =
randomBytes(32).toString("hex");
process.env.MOCK_LLM_SESSION_API_KEY = sessionApiKey;
+// ── State directory (isolated per test run) ────────────────────────────
+const STATE_DIR = ".tmp/mock-llm-state";
+
// ── URLs ───────────────────────────────────────────────────────────────
-const FRONTEND_URL = `http://localhost:${FRONTEND_PORT}/`;
-const BACKEND_URL = `http://127.0.0.1:${BACKEND_PORT}`;
+const INGRESS_URL = `http://localhost:${INGRESS_PORT}/`;
const MOCK_LLM_URL = `http://127.0.0.1:${MOCK_LLM_PORT}`;
// Python binary for the mock server β€” defaults to "python3" but CI can
@@ -33,8 +47,9 @@ const MOCK_LLM_URL = `http://127.0.0.1:${MOCK_LLM_PORT}`;
// PEP 668 "externally managed" errors on Ubuntu 24.04+.
const MOCK_LLM_PYTHON = process.env.MOCK_LLM_PYTHON ?? "python3";
-// Export for the test helpers
-process.env.MOCK_LLM_BACKEND_URL = BACKEND_URL;
+// Export for the test helpers β€” BACKEND_URL points to the ingress (API
+// calls are proxied to the agent-server, so no direct backend port needed).
+process.env.MOCK_LLM_BACKEND_URL = `http://localhost:${INGRESS_PORT}`;
process.env.MOCK_LLM_PORT = MOCK_LLM_PORT;
process.env.VITE_SESSION_API_KEY = sessionApiKey;
@@ -63,7 +78,7 @@ export default defineConfig({
],
outputDir: "test-results-mock-llm",
use: {
- baseURL: FRONTEND_URL,
+ baseURL: INGRESS_URL,
screenshot: "only-on-failure",
trace: "on-first-retry",
video: "retain-on-failure",
@@ -84,36 +99,38 @@ export default defineConfig({
stdout: "pipe",
stderr: "pipe",
},
- // 2. Agent-server + Vite frontend via dev-safe.mjs
+ // 2. Full agent-canvas stack via bin/agent-canvas.mjs
//
- // `exec` replaces the shell so Playwright's tracked PID IS the node
- // process. SIGTERM goes directly to dev-safe.mjs's signal handler,
- // which kills children via process groups and exits cleanly.
+ // This mirrors the production `npx @openhands/agent-canvas` path:
+ // - Pre-built static frontend served via static-server.mjs
+ // - Agent-server via uvx
+ // - Automation backend via uvx
+ // - Ingress proxy unifying all routes on a single port
//
- // Previously this used `exec ... npm run dev:minimal`, but npm does
- // NOT forward SIGTERM to its child processes β€” the node process and
- // its spawned agent-server/vite survived as orphans, causing the
- // webServer teardown to hang indefinitely.
+ // `exec` replaces the shell so Playwright's tracked PID IS the node
+ // process. SIGTERM goes directly to the shutdown handler, which
+ // kills children via process groups and exits cleanly.
{
command:
// Clean state dir to avoid stale profile/conversation data between runs
- "node -e \"const fs=require('node:fs'); for (const p of ['.tmp/mock-llm-state','node_modules/.vite']) fs.rmSync(p,{recursive:true,force:true});\" && " +
+ `node -e "const fs=require('node:fs'); fs.rmSync('${STATE_DIR}',{recursive:true,force:true});" && ` +
+ // Build frontend if not already built (CI should pre-build for caching)
+ '[ -f build/index.html ] || npm run build:app && ' +
[
"exec env",
- "OH_CANVAS_SAFE_STATE_DIR=.tmp/mock-llm-state",
+ envAssignment("OH_CANVAS_SAFE_STATE_DIR", STATE_DIR),
+ envAssignment("PORT", INGRESS_PORT),
envAssignment("SESSION_API_KEY", sessionApiKey),
envAssignment("OH_SESSION_API_KEYS_0", sessionApiKey),
- envAssignment("OH_CANVAS_SAFE_BACKEND_PORT", BACKEND_PORT),
envAssignment("VITE_SESSION_API_KEY", sessionApiKey),
"VITE_DO_NOT_TRACK=1",
"VITE_ENABLE_BROWSER_TOOLS=false",
- envAssignment("VITE_FRONTEND_PORT", FRONTEND_PORT),
// Bypass npm β€” exec directly into node so SIGTERM reaches
- // dev-safe.mjs's shutdown handler (npm swallows it).
- "node --env-file-if-exists=.env scripts/dev-safe.mjs",
+ // the shutdown handler (npm swallows it).
+ "node --env-file-if-exists=.env bin/agent-canvas.mjs",
].join(" "),
- url: FRONTEND_URL,
- timeout: 120_000,
+ url: INGRESS_URL,
+ timeout: 180_000, // allow extra time for build + agent-server + automation startup
reuseExistingServer: !process.env.CI,
},
],
diff --git a/scripts/dev-with-automation.mjs b/scripts/dev-with-automation.mjs
index cf41bd1461..7c61ef9ca4 100644
--- a/scripts/dev-with-automation.mjs
+++ b/scripts/dev-with-automation.mjs
@@ -330,7 +330,9 @@ async function buildConfig(args, env = process.env) {
// Session API key β€” shared by both agent-server and automation backend.
// Both validate it via the `X-Session-API-Key` header.
- const stateDir = join(homedir(), ".openhands", "agent-canvas");
+ const stateDir =
+ env.OH_CANVAS_SAFE_STATE_DIR ||
+ join(homedir(), ".openhands", "agent-canvas");
const safeConfig = buildSafeDevConfig(projectRoot, {
...env,
OH_CANVAS_SAFE_STATE_DIR: stateDir,
diff --git a/tests/e2e/mock-llm/utils/mock-llm-helpers.ts b/tests/e2e/mock-llm/utils/mock-llm-helpers.ts
index f8f14e0448..97e3b0216a 100644
--- a/tests/e2e/mock-llm/utils/mock-llm-helpers.ts
+++ b/tests/e2e/mock-llm/utils/mock-llm-helpers.ts
@@ -17,11 +17,13 @@ export const REPLY_TOKEN = "MOCK_LLM_E2E_REPLY_OK";
export const BASH_COMMAND = `printf '${BASH_TOKEN}\\n'`;
// Ports / URLs β€” set via env or defaults matching playwright.mock-llm.config.ts.
+// The agent-canvas binary exposes a single ingress port; API calls are proxied
+// through it, so BACKEND_URL = ingress URL (no separate backend port).
export const MOCK_LLM_PORT =
process.env.MOCK_LLM_PORT ?? "9999";
export const MOCK_LLM_BASE_URL = `http://127.0.0.1:${MOCK_LLM_PORT}`;
export const BACKEND_URL =
- process.env.MOCK_LLM_BACKEND_URL ?? "http://127.0.0.1:18200";
+ process.env.MOCK_LLM_BACKEND_URL ?? "http://localhost:18300";
export const SESSION_API_KEY = (() => {
const key =
process.env.MOCK_LLM_SESSION_API_KEY ??