π OpenHands: AI-Driven Development
fix(e2e): preserve mock validation trajectories (#17876)
kr1shna-exe660fee9ef25d
diff --git a/.agents/skills/e2e-testing/references/guide.md b/.agents/skills/e2e-testing/references/guide.mdindex 23e472ae7d..3b6cd05460 100644--- a/.agents/skills/e2e-testing/references/guide.md+++ b/.agents/skills/e2e-testing/references/guide.md@@ -26,12 +26,14 @@- **Single ingress URL**: Tests use one URL for both the browser (`baseURL`) and backend API assertions (`BACKEND_URL`). The ingress proxy routes `/api/*` to the agent-server, `/api/automation/*` to the automation backend, and `/*` to the static frontend. Default ingress port for tests is `18300` (override via `MOCK_LLM_INGRESS_PORT` env var).- **State isolation**: `OH_CANVAS_SAFE_STATE_DIR=.tmp/mock-llm-state` isolates test state from the user's real `~/.openhands/agent-canvas/` directory. Both `STATE_DIR` (`.tmp/mock-llm-state`) and the automation DB dir (`.tmp/automation/`) are cleaned before each test run β the automation DB now lives outside STATE_DIR at `dirname(STATE_DIR)/automation/automations.db`, mirroring Docker's `~/.openhands/automation/automations.db`.- **Session API key**: A random key is generated per test run and passed to the stack via `SESSION_API_KEY` / `OH_SESSION_API_KEYS_0` / `VITE_SESSION_API_KEY`. The static server injects it into `index.html` at serve time so the frontend authenticates automatically.+- **User skill fixtures**: The npm launcher sets `OH_PERSISTENCE_DIR` to `dirname(STATE_DIR)`, so `skill-test-helpers.ts` writes user skills to `.tmp/skills/`. Keep this aligned with the launcher's persistence root. `MOCK_LLM_USER_SKILLS_HOST_DIR` still overrides the host path for Docker's volume mount.- **Mock LLM server** (`tests/e2e/mock-llm/scripts/mock-llm-server.py`): Python HTTP server using openhands-sdk's `TestLLM` to return scripted tool-call + text trajectories. Supports admin API endpoints for dynamic trajectory management:- `POST /admin/reset` β reset to the default trajectory (terminal printf + text reply); also clears the stored completion-request history- `POST /admin/trajectory/register` β register a named trajectory (JSON body: `{name, turns}` where each turn is `{tool_call: {name, arguments}}` or `{text: "..."}`)- `POST /admin/trajectory/activate` β activate a previously registered trajectory- `GET /admin/requests` β return the list of all `/v1/chat/completions` request bodies captured since the last reset (used by the image-upload test to verify the image was forwarded to the LLM)- - Profile pre-flight ping: agent-server β₯ 1.43 sends a 1-token `ping` completion whenever an LLM profile is saved (`POST /api/profiles/{name}/validate`, 30 s budget in the canvas). The mock answers it with a canned `pong` instead of feeding it to `TestLLM`, so it neither consumes a scripted turn nor 500s-and-retries past the canvas timeout when the trajectory is exhausted; it is also left out of the `/admin/requests` history.+ - Profile pre-flight ping: agent-server β₯ 1.43 sends a 1-token `ping` completion whenever an LLM profile is saved (`POST /api/profiles/{name}/validate`, 30 s budget in the canvas). The mock recognizes both the older user-only request and the pinned server's `Reply with one token.` system message followed by the user ping. It answers with a canned `pong` without consuming a scripted turn or entering `/admin/requests` history. The pytest regression tests in `scripts/test_mock_llm_server.py` verify both shapes and preserve ordinary conversation requests. They run in the post-merge `Mock-LLM E2E Tests` workflow (push to `main` or `workflow_dispatch`), not on PRs, so run them locally when changing the matcher: `.mock-llm-venv/bin/python3 -m pytest tests/e2e/mock-llm/scripts -q`.+ - Trajectory ownership: profile validation must not switch the active trajectory. Each scenario spec owns its calls to `registerTrajectory()`, `activateTrajectory()` and `resetMockLLM()`. Onboarding-specific activation timing belongs in `onboarding/mock-llm-onboarding-happy-path.spec.ts`; it is separate from recognizing validation pings in the mock server.- **Real automation backend**: The automation test uses the production automation backend (started by `bin/agent-canvas.mjs`), NOT a mock server. Terminal `curl` commands from the agent hit the automation API through the ingress proxy at the test's `BACKEND_URL` (default `http://localhost:18300`). Auth uses the `X-Session-API-Key` header matching the stack's session key.- **Test helpers** (`tests/e2e/mock-llm/utils/mock-llm-helpers.ts`): Exports `registerTrajectory()`, `activateTrajectory()`, `resetMockLLM()`, `ensureMockLLMProfile()`, `getMockLLMRequests()` (fetches captured completion bodies from `GET /admin/requests`), `IMAGE_REPLY_TOKEN` + `MINIMAL_PNG_BASE64` (constants for the image-upload spec), ACP helpers (`configureAcpAgent()`, `verifyAcpAgentSettings()`, `resetToOpenHandsAgent()`, `ACP_REPLY_TOKEN`, `MOCK_ACP_SERVER_PATH`), and more.- **Padding response for internal LLM call**: The agent-server makes an internal LLM call (condenser/skill-analysis) before the agent's main loop starts when skills are activated. This consumes one trajectory response. Automation tests prepend a throwaway `{ text: "" }` response as padding. The conversation test does NOT need this because its user message doesn't trigger skill activation. Since `openhands-automation==1.10.0` (openhands/automation#405), the automation lifecycle spec scripts the required `finish` tool turn in its run-conversation budget β that release requires preset automation conversations to callthe `finish` tool before the run reaches COMPLETED; blank turns alone would loop and exhaust the trajectory. The mock server logs "Mock LLM exhausted after N calls" pinpoint the exact count if it drifts again.diff --git a/.github/workflows/mock-llm-e2e.yml b/.github/workflows/mock-llm-e2e.ymlindex 24059d5dd7..a08fa6b68f 100644--- a/.github/workflows/mock-llm-e2e.yml+++ b/.github/workflows/mock-llm-e2e.yml@@ -75,7 +75,7 @@ jobs:AGENT_SERVER_VERSION: ${{ steps.defaults.outputs.agent_server_version }}run: |uv venv .mock-llm-venv- uv pip install -p .mock-llm-venv "openhands-sdk==$AGENT_SERVER_VERSION"+ uv pip install -p .mock-llm-venv "openhands-sdk==$AGENT_SERVER_VERSION" pytest- name: Pre-warm automation backend (uvx cache)env:@@ -87,6 +87,9 @@ jobs:# before the automation backend finishes installing (~60-90s).uvx --from "openhands-automation==$AUTOMATION_VERSION" python -c "from openhands.automation.app import app; print('automation package cached')"+ - name: Test mock LLM server+ run: .mock-llm-venv/bin/python3 -m pytest tests/e2e/mock-llm/scripts -q+- name: Verify mock LLM server startsrun: |.mock-llm-venv/bin/python3 tests/e2e/mock-llm/scripts/mock-llm-server.py --port 9998 &diff --git a/__tests__/e2e/mock-llm-skill-paths.test.ts b/__tests__/e2e/mock-llm-skill-paths.test.tsnew file mode 100644index 0000000000..67e76ce95c--- /dev/null+++ b/__tests__/e2e/mock-llm-skill-paths.test.ts@@ -0,0 +1,42 @@+// @vitest-environment node+import path from "node:path";+import { afterEach, describe, expect, it, vi } from "vitest";++import {+ buildAgentServerEnv,+ buildSafeDevConfig,+} from "../../scripts/dev-safe.mjs";++afterEach(() => {+ vi.unstubAllEnvs();+ vi.resetModules();+});++describe("mock-LLM user skill paths", () => {+ it("writes npm fixtures where the isolated Agent Server loads user skills", async () => {+ vi.stubEnv("MOCK_LLM_USER_SKILLS_HOST_DIR", undefined);+ const { STATE_DIR, USER_SKILLS_DIR } =+ await import("../../tests/e2e/mock-llm/utils/skill-test-helpers");+ const config = buildSafeDevConfig(process.cwd(), {+ OH_CANVAS_SAFE_STATE_DIR: STATE_DIR,+ LOCAL_BACKEND_API_KEY: "test-session-key",+ OH_SECRET_KEY: "test-secret-key",+ });++ const serverEnv = buildAgentServerEnv(config, { env: {} });++ expect(USER_SKILLS_DIR).toBe(+ path.join(serverEnv.OH_PERSISTENCE_DIR, "skills"),+ );+ });++ it("keeps Docker fixture writes on the configured host mount", async () => {+ const mountPath = path.resolve(".tmp", "mock-llm-user-skills");+ vi.stubEnv("MOCK_LLM_USER_SKILLS_HOST_DIR", mountPath);++ const { USER_SKILLS_DIR } =+ await import("../../tests/e2e/mock-llm/utils/skill-test-helpers");++ expect(USER_SKILLS_DIR).toBe(mountPath);+ });+});diff --git a/tests/e2e/mock-llm/scripts/mock-llm-server.py b/tests/e2e/mock-llm/scripts/mock-llm-server.pyindex 39f0f5df85..d62468e342 100644--- a/tests/e2e/mock-llm/scripts/mock-llm-server.py+++ b/tests/e2e/mock-llm/scripts/mock-llm-server.py@@ -363,18 +363,33 @@ class MockLLMHandler(BaseHTTPRequestHandler):def _is_preflight_ping(body: dict) -> bool:"""Match the agent-server's profile pre-flight check.- It is a single user message saying "ping" with ``max_tokens=1`` (see- ``profiles_router.validate_profile`` in openhands-agent-server). The text- arrives either as a plain string or as OpenAI content parts.+ It sends "ping" with ``max_tokens=1`` (see ``profiles_router.validate_profile``+ in openhands-agent-server). The pinned agent-server also sends a+ "Reply with one token." system instruction; older versions sent only the+ user message. Match+ that exact instruction so ordinary one-token conversations still consume+ their scripted turns. Text can be a string or OpenAI content parts."""if body.get("max_tokens") != 1:return Falsemessages = body.get("messages")- if not isinstance(messages, list) or len(messages) != 1:+ if not isinstance(messages, list) or len(messages) not in (1, 2):return False- message = messages[0]+ if len(messages) == 2:+ system = messages[0]+ if (+ not isinstance(system, dict)+ or system.get("role") != "system"+ or _message_text(system) != "Reply with one token."+ ):+ return False+ message = messages[-1]if not isinstance(message, dict) or message.get("role") != "user":return False+ return _message_text(message) == PREFLIGHT_PING_TEXT+++def _message_text(message: dict) -> str | None:content = message.get("content")if isinstance(content, list):content = "".join(@@ -382,7 +397,7 @@ def _is_preflight_ping(body: dict) -> bool:for part in contentif isinstance(part, dict) and part.get("type") == "text")- return isinstance(content, str) and content.strip() == PREFLIGHT_PING_TEXT+ return content.strip() if isinstance(content, str) else Nonedef _preflight_pong(model: str | None) -> dict:diff --git a/tests/e2e/mock-llm/scripts/test_mock_llm_server.py b/tests/e2e/mock-llm/scripts/test_mock_llm_server.pynew file mode 100644index 0000000000..f6727305e8--- /dev/null+++ b/tests/e2e/mock-llm/scripts/test_mock_llm_server.py@@ -0,0 +1,78 @@+"""Regression tests for the mock server's profile-validation boundary."""++import importlib.util+import json+from pathlib import Path+import threading+from urllib.request import Request, urlopen++import pytest+++spec = importlib.util.spec_from_file_location(+ "mock_llm_server", Path(__file__).with_name("mock-llm-server.py")+)+server = importlib.util.module_from_spec(spec)+spec.loader.exec_module(server)+++def url():+ handler = server.MockLLMHandler+ handler.test_llm = server.TestLLM.from_messages(server.build_trajectory())+ handler._completion_requests = []+ httpd = server.HTTPServer(("127.0.0.1", 0), handler)+ thread = threading.Thread(target=httpd.serve_forever)+ thread.start()+ yield f"http://127.0.0.1:{httpd.server_port}"+ httpd.shutdown()+ httpd.server_close()+ thread.join()+++def text(value, as_parts):+ return [{"type": "text", "text": value}] if as_parts else value+++def complete(url, messages, max_tokens=100):+ request = Request(+ f"{url}/v1/chat/completions",+ data=json.dumps(+ {"model": "test", "messages": messages, "max_tokens": max_tokens}+ ).encode(),+ headers={"Content-Type": "application/json"},+ )+ with urlopen(request, timeout=5) as response:+ return json.load(response)["choices"][0]["message"]+++def recorded(url):+ with urlopen(f"{url}/admin/requests", timeout=5) as response:+ return json.load(response)["requests"]++[email protected]("as_parts", [False, True], ids=["string", "parts"])[email protected]("with_system", [False, True], ids=["user-only", "system+user"])+def test_validation_ping_leaves_the_trajectory_intact(url, as_parts, with_system):+ messages = [{"role": "user", "content": text("ping", as_parts)}]+ if with_system:+ system = {"role": "system", "content": text("Reply with one token.", as_parts)}+ messages.insert(0, system)++ assert complete(url, messages, max_tokens=1)["content"] == "pong"+ assert recorded(url) == []++ conversation = [{"role": "user", "content": "Run the scripted command."}]+ tool_call = complete(url, conversation)["tool_calls"][0]["function"]+ assert server.BASH_TOKEN in tool_call["arguments"]+ assert complete(url, conversation)["content"] == server.REPLY_TOKEN+++def test_one_token_conversation_is_not_mistaken_for_validation(url):+ messages = [+ {"role": "system", "content": "You are an assistant."},+ {"role": "user", "content": "ping"},+ ]++ assert "tool_calls" in complete(url, messages, max_tokens=1)+ assert recorded(url)[0]["messages"] == messagesdiff --git a/tests/e2e/mock-llm/skills/mock-llm-skills.spec.ts b/tests/e2e/mock-llm/skills/mock-llm-skills.spec.tsindex a398fa51eb..ec2ee7eb95 100644--- a/tests/e2e/mock-llm/skills/mock-llm-skills.spec.ts+++ b/tests/e2e/mock-llm/skills/mock-llm-skills.spec.ts@@ -11,7 +11,7 @@* agent-server creates a worktree from the repo, and since the skill* is committed, `load_project_skills` finds it in the worktree.*- * 2. **User skills** from `~/.openhands/skills/` are loaded and trigger+ * 2. **User skills** from the server's persistence directory are loaded and trigger* on matching keywords.** 3. **Skill deletion**: removing a skill file means it is NOT loaded@@ -251,9 +251,9 @@ test.describe("skill loading: project, user, and deletion", () => {});});- // ββ Test 2: User skill loaded from ~/.openhands/skills/ ββββββββββ+ // Test 2: User skill loaded from the server's persistence directory- test("user skill in ~/.openhands/skills/ triggers on matching keyword", async ({+ test("user skill in the persistence directory triggers on matching keyword", async ({page,request,}) => {diff --git a/tests/e2e/mock-llm/utils/skill-test-helpers.ts b/tests/e2e/mock-llm/utils/skill-test-helpers.tsindex 6a8d1d96cf..67ed3786e9 100644--- a/tests/e2e/mock-llm/utils/skill-test-helpers.ts+++ b/tests/e2e/mock-llm/utils/skill-test-helpers.ts@@ -12,10 +12,9 @@* with the agent-server while creating files on the host (volume-mounted).*/-import { resolve, join } from "path";+import { resolve, join, dirname } from "path";import { mkdirSync, writeFileSync, rmSync, existsSync } from "fs";import { execSync } from "child_process";-import { homedir } from "os";// ββ Paths ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ@@ -39,13 +38,14 @@ export const SKILL_REPOS_AGENT_DIR =process.env.MOCK_LLM_SKILL_REPOS_CONTAINER_DIR ?? SKILL_REPOS_DIR;/**- * User-level skills directory β HOST-side (for file creation/removal).+ * User-level skills directory on the host (for file creation/removal).+ * The npm launcher sets OH_PERSISTENCE_DIR to the parent of STATE_DIR.* In Docker mode, we use a local temp dir that is volume-mounted into the* container at the agent-server's expected `~/.openhands/skills/` path.*/export const USER_SKILLS_DIR = process.env.MOCK_LLM_USER_SKILLS_HOST_DIR? resolve(process.env.MOCK_LLM_USER_SKILLS_HOST_DIR)- : join(homedir(), ".openhands", "skills");+ : join(dirname(STATE_DIR), "skills");// ββ Skill content builders βββββββββββββββββββββββββββββββββββββββββββ