Skip to content

πŸ™Œ OpenHands: AI-Driven Development

fix(e2e): preserve mock validation trajectories (#17876)

kr1shna-exe660fee9ef25d
diff --git a/.agents/skills/e2e-testing/references/guide.md b/.agents/skills/e2e-testing/references/guide.md
index 23e472ae7d..3b6cd05460 100644
--- a/.agents/skills/e2e-testing/references/guide.md
+++ b/.agents/skills/e2e-testing/references/guide.md
@@ -26,12 +26,14 @@
- **Single ingress URL**: Tests use one URL for both the browser (`baseURL`) and backend API assertions (`BACKEND_URL`). The ingress proxy routes `/api/*` to the agent-server, `/api/automation/*` to the automation backend, and `/*` to the static frontend. Default ingress port for tests is `18300` (override via `MOCK_LLM_INGRESS_PORT` env var).
- **State isolation**: `OH_CANVAS_SAFE_STATE_DIR=.tmp/mock-llm-state` isolates test state from the user's real `~/.openhands/agent-canvas/` directory. Both `STATE_DIR` (`.tmp/mock-llm-state`) and the automation DB dir (`.tmp/automation/`) are cleaned before each test run β€” the automation DB now lives outside STATE_DIR at `dirname(STATE_DIR)/automation/automations.db`, mirroring Docker's `~/.openhands/automation/automations.db`.
- **Session API key**: A random key is generated per test run and passed to the stack via `SESSION_API_KEY` / `OH_SESSION_API_KEYS_0` / `VITE_SESSION_API_KEY`. The static server injects it into `index.html` at serve time so the frontend authenticates automatically.
+- **User skill fixtures**: The npm launcher sets `OH_PERSISTENCE_DIR` to `dirname(STATE_DIR)`, so `skill-test-helpers.ts` writes user skills to `.tmp/skills/`. Keep this aligned with the launcher's persistence root. `MOCK_LLM_USER_SKILLS_HOST_DIR` still overrides the host path for Docker's volume mount.
- **Mock LLM server** (`tests/e2e/mock-llm/scripts/mock-llm-server.py`): Python HTTP server using openhands-sdk's `TestLLM` to return scripted tool-call + text trajectories. Supports admin API endpoints for dynamic trajectory management:
- `POST /admin/reset` β€” reset to the default trajectory (terminal printf + text reply); also clears the stored completion-request history
- `POST /admin/trajectory/register` β€” register a named trajectory (JSON body: `{name, turns}` where each turn is `{tool_call: {name, arguments}}` or `{text: "..."}`)
- `POST /admin/trajectory/activate` β€” activate a previously registered trajectory
- `GET /admin/requests` β€” return the list of all `/v1/chat/completions` request bodies captured since the last reset (used by the image-upload test to verify the image was forwarded to the LLM)
- - Profile pre-flight ping: agent-server β‰₯ 1.43 sends a 1-token `ping` completion whenever an LLM profile is saved (`POST /api/profiles/{name}/validate`, 30 s budget in the canvas). The mock answers it with a canned `pong` instead of feeding it to `TestLLM`, so it neither consumes a scripted turn nor 500s-and-retries past the canvas timeout when the trajectory is exhausted; it is also left out of the `/admin/requests` history.
+ - Profile pre-flight ping: agent-server β‰₯ 1.43 sends a 1-token `ping` completion whenever an LLM profile is saved (`POST /api/profiles/{name}/validate`, 30 s budget in the canvas). The mock recognizes both the older user-only request and the pinned server's `Reply with one token.` system message followed by the user ping. It answers with a canned `pong` without consuming a scripted turn or entering `/admin/requests` history. The pytest regression tests in `scripts/test_mock_llm_server.py` verify both shapes and preserve ordinary conversation requests. They run in the post-merge `Mock-LLM E2E Tests` workflow (push to `main` or `workflow_dispatch`), not on PRs, so run them locally when changing the matcher: `.mock-llm-venv/bin/python3 -m pytest tests/e2e/mock-llm/scripts -q`.
+ - Trajectory ownership: profile validation must not switch the active trajectory. Each scenario spec owns its calls to `registerTrajectory()`, `activateTrajectory()` and `resetMockLLM()`. Onboarding-specific activation timing belongs in `onboarding/mock-llm-onboarding-happy-path.spec.ts`; it is separate from recognizing validation pings in the mock server.
- **Real automation backend**: The automation test uses the production automation backend (started by `bin/agent-canvas.mjs`), NOT a mock server. Terminal `curl` commands from the agent hit the automation API through the ingress proxy at the test's `BACKEND_URL` (default `http://localhost:18300`). Auth uses the `X-Session-API-Key` header matching the stack's session key.
- **Test helpers** (`tests/e2e/mock-llm/utils/mock-llm-helpers.ts`): Exports `registerTrajectory()`, `activateTrajectory()`, `resetMockLLM()`, `ensureMockLLMProfile()`, `getMockLLMRequests()` (fetches captured completion bodies from `GET /admin/requests`), `IMAGE_REPLY_TOKEN` + `MINIMAL_PNG_BASE64` (constants for the image-upload spec), ACP helpers (`configureAcpAgent()`, `verifyAcpAgentSettings()`, `resetToOpenHandsAgent()`, `ACP_REPLY_TOKEN`, `MOCK_ACP_SERVER_PATH`), and more.
- **Padding response for internal LLM call**: The agent-server makes an internal LLM call (condenser/skill-analysis) before the agent's main loop starts when skills are activated. This consumes one trajectory response. Automation tests prepend a throwaway `{ text: "" }` response as padding. The conversation test does NOT need this because its user message doesn't trigger skill activation. Since `openhands-automation==1.10.0` (openhands/automation#405), the automation lifecycle spec scripts the required `finish` tool turn in its run-conversation budget β€” that release requires preset automation conversations to callthe `finish` tool before the run reaches COMPLETED; blank turns alone would loop and exhaust the trajectory. The mock server logs "Mock LLM exhausted after N calls" pinpoint the exact count if it drifts again.
diff --git a/.github/workflows/mock-llm-e2e.yml b/.github/workflows/mock-llm-e2e.yml
index 24059d5dd7..a08fa6b68f 100644
--- a/.github/workflows/mock-llm-e2e.yml
+++ b/.github/workflows/mock-llm-e2e.yml
@@ -75,7 +75,7 @@ jobs:
AGENT_SERVER_VERSION: ${{ steps.defaults.outputs.agent_server_version }}
run: |
uv venv .mock-llm-venv
- uv pip install -p .mock-llm-venv "openhands-sdk==$AGENT_SERVER_VERSION"
+ uv pip install -p .mock-llm-venv "openhands-sdk==$AGENT_SERVER_VERSION" pytest
- name: Pre-warm automation backend (uvx cache)
env:
@@ -87,6 +87,9 @@ jobs:
# before the automation backend finishes installing (~60-90s).
uvx --from "openhands-automation==$AUTOMATION_VERSION" python -c "from openhands.automation.app import app; print('automation package cached')"
+ - name: Test mock LLM server
+ run: .mock-llm-venv/bin/python3 -m pytest tests/e2e/mock-llm/scripts -q
+
- name: Verify mock LLM server starts
run: |
.mock-llm-venv/bin/python3 tests/e2e/mock-llm/scripts/mock-llm-server.py --port 9998 &
diff --git a/__tests__/e2e/mock-llm-skill-paths.test.ts b/__tests__/e2e/mock-llm-skill-paths.test.ts
new file mode 100644
index 0000000000..67e76ce95c
--- /dev/null
+++ b/__tests__/e2e/mock-llm-skill-paths.test.ts
@@ -0,0 +1,42 @@
+// @vitest-environment node
+import path from "node:path";
+import { afterEach, describe, expect, it, vi } from "vitest";
+
+import {
+ buildAgentServerEnv,
+ buildSafeDevConfig,
+} from "../../scripts/dev-safe.mjs";
+
+afterEach(() => {
+ vi.unstubAllEnvs();
+ vi.resetModules();
+});
+
+describe("mock-LLM user skill paths", () => {
+ it("writes npm fixtures where the isolated Agent Server loads user skills", async () => {
+ vi.stubEnv("MOCK_LLM_USER_SKILLS_HOST_DIR", undefined);
+ const { STATE_DIR, USER_SKILLS_DIR } =
+ await import("../../tests/e2e/mock-llm/utils/skill-test-helpers");
+ const config = buildSafeDevConfig(process.cwd(), {
+ OH_CANVAS_SAFE_STATE_DIR: STATE_DIR,
+ LOCAL_BACKEND_API_KEY: "test-session-key",
+ OH_SECRET_KEY: "test-secret-key",
+ });
+
+ const serverEnv = buildAgentServerEnv(config, { env: {} });
+
+ expect(USER_SKILLS_DIR).toBe(
+ path.join(serverEnv.OH_PERSISTENCE_DIR, "skills"),
+ );
+ });
+
+ it("keeps Docker fixture writes on the configured host mount", async () => {
+ const mountPath = path.resolve(".tmp", "mock-llm-user-skills");
+ vi.stubEnv("MOCK_LLM_USER_SKILLS_HOST_DIR", mountPath);
+
+ const { USER_SKILLS_DIR } =
+ await import("../../tests/e2e/mock-llm/utils/skill-test-helpers");
+
+ expect(USER_SKILLS_DIR).toBe(mountPath);
+ });
+});
diff --git a/tests/e2e/mock-llm/scripts/mock-llm-server.py b/tests/e2e/mock-llm/scripts/mock-llm-server.py
index 39f0f5df85..d62468e342 100644
--- a/tests/e2e/mock-llm/scripts/mock-llm-server.py
+++ b/tests/e2e/mock-llm/scripts/mock-llm-server.py
@@ -363,18 +363,33 @@ class MockLLMHandler(BaseHTTPRequestHandler):
def _is_preflight_ping(body: dict) -> bool:
"""Match the agent-server's profile pre-flight check.
- It is a single user message saying "ping" with ``max_tokens=1`` (see
- ``profiles_router.validate_profile`` in openhands-agent-server). The text
- arrives either as a plain string or as OpenAI content parts.
+ It sends "ping" with ``max_tokens=1`` (see ``profiles_router.validate_profile``
+ in openhands-agent-server). The pinned agent-server also sends a
+ "Reply with one token." system instruction; older versions sent only the
+ user message. Match
+ that exact instruction so ordinary one-token conversations still consume
+ their scripted turns. Text can be a string or OpenAI content parts.
"""
if body.get("max_tokens") != 1:
return False
messages = body.get("messages")
- if not isinstance(messages, list) or len(messages) != 1:
+ if not isinstance(messages, list) or len(messages) not in (1, 2):
return False
- message = messages[0]
+ if len(messages) == 2:
+ system = messages[0]
+ if (
+ not isinstance(system, dict)
+ or system.get("role") != "system"
+ or _message_text(system) != "Reply with one token."
+ ):
+ return False
+ message = messages[-1]
if not isinstance(message, dict) or message.get("role") != "user":
return False
+ return _message_text(message) == PREFLIGHT_PING_TEXT
+
+
+def _message_text(message: dict) -> str | None:
content = message.get("content")
if isinstance(content, list):
content = "".join(
@@ -382,7 +397,7 @@ def _is_preflight_ping(body: dict) -> bool:
for part in content
if isinstance(part, dict) and part.get("type") == "text"
)
- return isinstance(content, str) and content.strip() == PREFLIGHT_PING_TEXT
+ return content.strip() if isinstance(content, str) else None
def _preflight_pong(model: str | None) -> dict:
diff --git a/tests/e2e/mock-llm/scripts/test_mock_llm_server.py b/tests/e2e/mock-llm/scripts/test_mock_llm_server.py
new file mode 100644
index 0000000000..f6727305e8
--- /dev/null
+++ b/tests/e2e/mock-llm/scripts/test_mock_llm_server.py
@@ -0,0 +1,78 @@
+"""Regression tests for the mock server's profile-validation boundary."""
+
+import importlib.util
+import json
+from pathlib import Path
+import threading
+from urllib.request import Request, urlopen
+
+import pytest
+
+
+spec = importlib.util.spec_from_file_location(
+ "mock_llm_server", Path(__file__).with_name("mock-llm-server.py")
+)
+server = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(server)
+
+
+def url():
+ handler = server.MockLLMHandler
+ handler.test_llm = server.TestLLM.from_messages(server.build_trajectory())
+ handler._completion_requests = []
+ httpd = server.HTTPServer(("127.0.0.1", 0), handler)
+ thread = threading.Thread(target=httpd.serve_forever)
+ thread.start()
+ yield f"http://127.0.0.1:{httpd.server_port}"
+ httpd.shutdown()
+ httpd.server_close()
+ thread.join()
+
+
+def text(value, as_parts):
+ return [{"type": "text", "text": value}] if as_parts else value
+
+
+def complete(url, messages, max_tokens=100):
+ request = Request(
+ f"{url}/v1/chat/completions",
+ data=json.dumps(
+ {"model": "test", "messages": messages, "max_tokens": max_tokens}
+ ).encode(),
+ headers={"Content-Type": "application/json"},
+ )
+ with urlopen(request, timeout=5) as response:
+ return json.load(response)["choices"][0]["message"]
+
+
+def recorded(url):
+ with urlopen(f"{url}/admin/requests", timeout=5) as response:
+ return json.load(response)["requests"]
+
+
[email protected]("as_parts", [False, True], ids=["string", "parts"])
[email protected]("with_system", [False, True], ids=["user-only", "system+user"])
+def test_validation_ping_leaves_the_trajectory_intact(url, as_parts, with_system):
+ messages = [{"role": "user", "content": text("ping", as_parts)}]
+ if with_system:
+ system = {"role": "system", "content": text("Reply with one token.", as_parts)}
+ messages.insert(0, system)
+
+ assert complete(url, messages, max_tokens=1)["content"] == "pong"
+ assert recorded(url) == []
+
+ conversation = [{"role": "user", "content": "Run the scripted command."}]
+ tool_call = complete(url, conversation)["tool_calls"][0]["function"]
+ assert server.BASH_TOKEN in tool_call["arguments"]
+ assert complete(url, conversation)["content"] == server.REPLY_TOKEN
+
+
+def test_one_token_conversation_is_not_mistaken_for_validation(url):
+ messages = [
+ {"role": "system", "content": "You are an assistant."},
+ {"role": "user", "content": "ping"},
+ ]
+
+ assert "tool_calls" in complete(url, messages, max_tokens=1)
+ assert recorded(url)[0]["messages"] == messages
diff --git a/tests/e2e/mock-llm/skills/mock-llm-skills.spec.ts b/tests/e2e/mock-llm/skills/mock-llm-skills.spec.ts
index a398fa51eb..ec2ee7eb95 100644
--- a/tests/e2e/mock-llm/skills/mock-llm-skills.spec.ts
+++ b/tests/e2e/mock-llm/skills/mock-llm-skills.spec.ts
@@ -11,7 +11,7 @@
* agent-server creates a worktree from the repo, and since the skill
* is committed, `load_project_skills` finds it in the worktree.
*
- * 2. **User skills** from `~/.openhands/skills/` are loaded and trigger
+ * 2. **User skills** from the server's persistence directory are loaded and trigger
* on matching keywords.
*
* 3. **Skill deletion**: removing a skill file means it is NOT loaded
@@ -251,9 +251,9 @@ test.describe("skill loading: project, user, and deletion", () => {
});
});
- // ── Test 2: User skill loaded from ~/.openhands/skills/ ──────────
+ // Test 2: User skill loaded from the server's persistence directory
- test("user skill in ~/.openhands/skills/ triggers on matching keyword", async ({
+ test("user skill in the persistence directory triggers on matching keyword", async ({
page,
request,
}) => {
diff --git a/tests/e2e/mock-llm/utils/skill-test-helpers.ts b/tests/e2e/mock-llm/utils/skill-test-helpers.ts
index 6a8d1d96cf..67ed3786e9 100644
--- a/tests/e2e/mock-llm/utils/skill-test-helpers.ts
+++ b/tests/e2e/mock-llm/utils/skill-test-helpers.ts
@@ -12,10 +12,9 @@
* with the agent-server while creating files on the host (volume-mounted).
*/
-import { resolve, join } from "path";
+import { resolve, join, dirname } from "path";
import { mkdirSync, writeFileSync, rmSync, existsSync } from "fs";
import { execSync } from "child_process";
-import { homedir } from "os";
// ── Paths ────────────────────────────────────────────────────────────
@@ -39,13 +38,14 @@ export const SKILL_REPOS_AGENT_DIR =
process.env.MOCK_LLM_SKILL_REPOS_CONTAINER_DIR ?? SKILL_REPOS_DIR;
/**
- * User-level skills directory β€” HOST-side (for file creation/removal).
+ * User-level skills directory on the host (for file creation/removal).
+ * The npm launcher sets OH_PERSISTENCE_DIR to the parent of STATE_DIR.
* In Docker mode, we use a local temp dir that is volume-mounted into the
* container at the agent-server's expected `~/.openhands/skills/` path.
*/
export const USER_SKILLS_DIR = process.env.MOCK_LLM_USER_SKILLS_HOST_DIR
? resolve(process.env.MOCK_LLM_USER_SKILLS_HOST_DIR)
- : join(homedir(), ".openhands", "skills");
+ : join(dirname(STATE_DIR), "skills");
// ── Skill content builders ───────────────────────────────────────────