Skip to content

🙌 OpenHands: AI-Driven Development

Default onboarding LLM to OpenAI GPT-5.5 (#1103)

Engel Nyst00062bdfad2d
diff --git a/AGENTS.md b/AGENTS.md
index d56aad4951..191df3ad36 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -208,6 +208,45 @@ you are running inside of — NOT the automation backend.
- **State isolation**: The Docker container uses its internal state directory (no host mount needed for tests). Each test run starts a fresh container.
- CI workflow: `.github/workflows/mock-llm-docker-e2e.yml` has three triggers — all pull the already-built image from GHCR (no rebuild): (1) `workflow_run` fires automatically after the `Docker` workflow completes on main; (2) `pull_request` with the `e2e-tests` label polls the Docker workflow until it finishes for the PR's head SHA, then pulls the image (needed because `workflow_run` only fires for workflow files already on the default branch); (3) `workflow_dispatch` accepts a custom `docker_image` input. The image tag is derived from the commit SHA (`ghcr.io/openhands/agent-canvas:sha-<short>-amd64`). Fork PRs are skipped (no GHCR push). Report artifacts go to `test-results-mock-llm-docker/` and `playwright-report-mock-llm-docker/`.
+## Debugging E2E Test Failures
+
+When an E2E test fails in CI, use this workflow to diagnose the root cause efficiently:
+
+### 1. Read the PR comment first
+The mock-LLM E2E workflow posts a structured comment on the PR with a test results table, pass/fail status, and collapsible failure details including the Playwright error message. **Start here** — the error message usually reveals whether the failure is a locator mismatch, a timeout, or a missing element.
+
+### 2. Download CI artifacts
+Every failing test run uploads artifacts (`mock-llm-e2e-results` for npm, `test-results-mock-llm-docker` for Docker). Download them with:
+```bash
+gh run download <run_id> --repo OpenHands/agent-canvas --name mock-llm-e2e-results --dir /tmp/artifacts
+```
+Artifacts contain:
+- `test-results-mock-llm/` — per-test directories with `test-failed-N.png` (screenshot at failure) and `error-context.md` (Playwright page snapshot as YAML accessibility tree + test source with the failing line marked)
+- `playwright-report-mock-llm/` — full HTML report (`npx playwright show-report /tmp/artifacts/playwright-report-mock-llm`)
+
+### 3. Inspect the error-context.md page snapshot
+The `error-context.md` file contains a YAML accessibility tree of the entire page at the moment of failure. This is the single most useful artifact — it shows exactly what DOM elements exist, which tabs are selected, what text is in inputs, and whether a component rendered at all. Search for the element your test expects (e.g. `llm-provider-input`) to see if it's present or absent, and check surrounding context (tab selection state, form view mode, etc.) to understand why.
+
+### 4. Common failure patterns
+
+**"element(s) not found"** — The locator matched zero elements. The component either:
+- Didn't render (conditional rendering path not taken — check the page snapshot for what DID render)
+- Has a different `name`/`data-testid` than expected
+- Is behind a lazy-load boundary that hasn't resolved
+
+**Stale state from earlier serial tests** — Mock-LLM tests run serially (`workers: 1`) against a real agent-server. Earlier tests (conversation, automation) persist settings on the server. If your test depends on "clean" state but a prior test configured `llm_base_url`, `llm_model`, etc., the form may render in a different view mode. Use Playwright `page.route()` to intercept and normalize the settings response. Example: `routeOnboardingLlmCatalog` in `tests/e2e/support/onboarding-helpers.ts` intercepts `GET /api/settings` to clear `llm_base_url` so the LLM form always opens in "Basic" view.
+
+**View mode mismatch (Basic vs Advanced)** — `LlmSettingsScreen` switches between "Basic" (renders `ModelSelector` with provider/model dropdowns) and "Advanced" (renders plain text inputs). The view is determined by `getInitialView()` which checks `currentSettings.llm_base_url` — a non-default base URL triggers "Advanced" view. If your test expects `input[name="llm-provider-input"]` but sees text inputs instead, the settings have a stale `base_url`.
+
+**Playwright route interception vs real server** — In mock-LLM tests, routes registered with `page.route()` intercept at the browser level before requests reach the real agent-server. However, `page.route()` must be set up BEFORE `page.goto()`. The `showOnboarding` helper handles this correctly (routes are registered before navigation). Non-GET methods should use `route.fallback()` to pass through to the real server.
+
+### 5. Running locally
+```bash
+npm run test:e2e:mock-llm # full suite
+npm run test:e2e:mock-llm -- --headed # watch in browser
+npm run test:e2e:mock-llm -- -g "test name" # run single test by name
+```
+
## Additional Notes
- **Published binary auth fix**: When users install the npm package globally (`npm install -g @openhands/agent-canvas`) and run `agent-canvas`, the pre-built static frontend has NO `VITE_SESSION_API_KEY` baked in (npm publish runs `npm run build` with no such env var). The runtime session key is generated when the CLI launches and reaches the frontend via `scripts/static-server.mjs --session-api-key <key>`, which injects a `<head>` script that does two things: (a) sets `window.__AGENT_CANVAS_SESSION_API_KEY__ = <key>` — read by `getBakedSessionApiKey()` in `src/api/agent-server-config.ts` as a fallback when the env var is empty, symmetric with `__AGENT_CANVAS_AUTH_REQUIRED__` / `isAuthRequired()`; (b) writes the same key into `localStorage['openhands-agent-server-config'].sessionApiKey`, always overwriting when the value differs, so any code path that still reads the legacy storage key (e.g. e2e fixtures) sees the live key. The window-global path is the load-bearing one — without it, `makeDefaultLocalBackend()` returns null on a fresh install, the backend registry seeds empty, and `root.tsx` traps the user behind the Manage Backends modal instead of onboarding. `scripts/dev-with-automation.mjs` and `scripts/dev-static.mjs` both pass `--session-api-key ${config.sessionApiKey}` when starting the static server.
diff --git a/__tests__/api/option-service.test.ts b/__tests__/api/option-service.test.ts
index a8fe9ff317..b8ba9fe857 100644
--- a/__tests__/api/option-service.test.ts
+++ b/__tests__/api/option-service.test.ts
@@ -94,8 +94,14 @@ describe("OptionService", () => {
const models = await OptionService.getModels();
expect(models.models).toContain("openhands/claude-opus-4-5-20251101");
+ expect(models.models).toContain("openai/gpt-5.5");
expect(models.verified_models).toContain("claude-opus-4-5-20251101");
- expect(models.verified_providers).toEqual(["anthropic", "openhands"]);
+ expect(models.verified_models).toContain("gpt-5.5");
+ expect(models.verified_providers).toEqual([
+ "anthropic",
+ "openai",
+ "openhands",
+ ]);
expect(models.default_model).toBeTruthy();
});
diff --git a/__tests__/components/onboarding/onboarding-modal.test.tsx b/__tests__/components/onboarding/onboarding-modal.test.tsx
index e0a4cff877..f938cfb9e6 100644
--- a/__tests__/components/onboarding/onboarding-modal.test.tsx
+++ b/__tests__/components/onboarding/onboarding-modal.test.tsx
@@ -7,6 +7,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { __resetActiveStoreForTests } from "#/api/backend-registry/active-store";
import { ActiveBackendProvider } from "#/contexts/active-backend-context";
import { OnboardingModal } from "#/components/features/onboarding/onboarding-modal";
+import { ONBOARDING_DEFAULT_LLM_MODEL } from "#/components/features/onboarding/steps/setup-llm-step";
import { NavigationProvider } from "#/context/navigation-context";
import SettingsService from "#/api/settings-service/settings-service.api";
import { SecretsService } from "#/api/secrets-service";
@@ -233,14 +234,14 @@ describe("OnboardingModal", () => {
);
});
- it("pre-fills the LLM step with the Anthropic provider's Claude Opus model", () => {
+ it("pre-fills the LLM step with OpenAI GPT-5.5", () => {
renderModal();
expect(llmSettingsScreenMock).toHaveBeenCalledTimes(1);
expect(llmSettingsScreenMock).toHaveBeenCalledWith(
expect.objectContaining({
initialValueOverrides: {
- "llm.model": "anthropic/claude-opus-4-8",
+ "llm.model": ONBOARDING_DEFAULT_LLM_MODEL,
},
}),
);
diff --git a/src/components/features/onboarding/steps/setup-llm-step.tsx b/src/components/features/onboarding/steps/setup-llm-step.tsx
index b470ea922f..dda6ce765e 100644
--- a/src/components/features/onboarding/steps/setup-llm-step.tsx
+++ b/src/components/features/onboarding/steps/setup-llm-step.tsx
@@ -15,14 +15,16 @@ interface SetupLlmStepProps {
}
/**
- * Pre-fills the LLM form with the Anthropic provider's Claude Opus
- * model. Most users won't have an OpenHands account, so onboarding
- * routes directly through Anthropic by default. Keeping this as an
- * explicit override marks the model dirty so the Next button persists
- * the suggested default immediately.
+ * Pre-fills the LLM form with the OpenAI GPT-5.5 model. The SDK's
+ * bare default model is `gpt-5.5`; Canvas stores provider-qualified
+ * LiteLLM model ids, so the onboarding override uses this OpenAI-prefixed
+ * model id. Keeping this as an explicit override marks the model dirty so
+ * the Next button persists the suggested default immediately.
*/
+export const ONBOARDING_DEFAULT_LLM_MODEL = "openai/gpt-5.5";
+
const ONBOARDING_LLM_OVERRIDES = {
- "llm.model": "anthropic/claude-opus-4-8",
+ "llm.model": ONBOARDING_DEFAULT_LLM_MODEL,
} as const;
/**
diff --git a/src/i18n/translation.json b/src/i18n/translation.json
index 0f091ab62f..59fe00ae48 100644
--- a/src/i18n/translation.json
+++ b/src/i18n/translation.json
@@ -27778,21 +27778,21 @@
"uk": "Налаштуйте свій LLM"
},
"ONBOARDING$LLM_SUBTITLE": {
- "en": "Pick a provider and model. We've pre-selected Anthropic Claude Opus.",
- "ja": "プロバイダーとモデルを選択してください。Anthropic Claude Opus を事前に選択しています。",
- "zh-CN": "选择提供商和模型。我们已为你预先选择 Anthropic Claude Opus。",
- "zh-TW": "選擇供應商與模型。我們已預先選好 Anthropic Claude Opus。",
- "ko-KR": "공급자와 모델을 선택하세요. 기본값으로 Anthropic Claude Opus가 선택되어 있습니다.",
- "no": "Velg en leverandør og modell. Vi har forhåndsvalgt Anthropic Claude Opus.",
- "ar": "اختر مزوّدًا ونموذجًا. لقد اخترنا مسبقًا Anthropic Claude Opus.",
- "de": "Wählen Sie Anbieter und Modell. Wir haben Anthropic Claude Opus vorausgewählt.",
- "fr": "Choisissez un fournisseur et un modèle. Nous avons présélectionné Anthropic Claude Opus.",
- "it": "Scegli un provider e un modello. Abbiamo preselezionato Anthropic Claude Opus.",
- "pt": "Escolha um provedor e modelo. Já selecionamos o Anthropic Claude Opus.",
- "es": "Elige un proveedor y un modelo. Hemos preseleccionado Anthropic Claude Opus.",
- "ca": "Trieu un proveïdor i model. Hem preseleccionat Anthropic Claude Opus.",
- "tr": "Bir sağlayıcı ve model seçin. Anthropic Claude Opus'u önceden seçtik.",
- "uk": "Виберіть провайдера та модель. Anthropic Claude Opus уже обрано."
+ "en": "Pick a provider and model. We've pre-selected OpenAI GPT-5.5.",
+ "ja": "プロバイダーとモデルを選択してください。OpenAI GPT-5.5 を事前に選択しています。",
+ "zh-CN": "选择提供商和模型。我们已为你预先选择 OpenAI GPT-5.5。",
+ "zh-TW": "選擇供應商與模型。我們已預先選好 OpenAI GPT-5.5。",
+ "ko-KR": "공급자와 모델을 선택하세요. 기본값으로 OpenAI GPT-5.5가 선택되어 있습니다.",
+ "no": "Velg en leverandør og modell. Vi har forhåndsvalgt OpenAI GPT-5.5.",
+ "ar": "اختر مزوّدًا ونموذجًا. لقد اخترنا مسبقًا OpenAI GPT-5.5.",
+ "de": "Wählen Sie Anbieter und Modell. Wir haben OpenAI GPT-5.5 vorausgewählt.",
+ "fr": "Choisissez un fournisseur et un modèle. Nous avons présélectionné OpenAI GPT-5.5.",
+ "it": "Scegli un provider e un modello. Abbiamo preselezionato OpenAI GPT-5.5.",
+ "pt": "Escolha um provedor e modelo. Já selecionamos o OpenAI GPT-5.5.",
+ "es": "Elige un proveedor y modelo. Hemos preseleccionado OpenAI GPT-5.5.",
+ "ca": "Trieu un proveïdor i model. Hem preseleccionat OpenAI GPT-5.5.",
+ "tr": "Bir sağlayıcı ve model seçin. OpenAI GPT-5.5'i önceden seçtik.",
+ "uk": "Виберіть провайдера та модель. OpenAI GPT-5.5 уже обрано."
},
"ONBOARDING$HELLO_TITLE": {
"en": "Say hello 👋",
diff --git a/src/mocks/settings-handlers.ts b/src/mocks/settings-handlers.ts
index 05e41dcdfa..c3d8daf856 100644
--- a/src/mocks/settings-handlers.ts
+++ b/src/mocks/settings-handlers.ts
@@ -439,6 +439,7 @@ const MOCK_MODELS = [
"anthropic/claude-opus-4-5-20251101",
"anthropic/claude-opus-4-8",
"openai/gpt-3.5-turbo",
+ "openai/gpt-5.5",
"openai/gpt-4o",
"openai/gpt-4o-mini",
"openhands/claude-sonnet-4-20250514",
@@ -453,6 +454,7 @@ const MOCK_VERIFIED_MODELS = new Set([
"anthropic/claude-opus-4-5-20251101",
"anthropic/claude-opus-4-8",
"anthropic/claude-sonnet-4-5-20250929",
+ "openai/gpt-5.5",
"openhands/claude-opus-4-5-20251101",
"openhands/claude-sonnet-4-5-20250929",
"openhands/minimax-m2.7",
diff --git a/tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts b/tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts
index 88184f8da1..fcfe90c69c 100644
--- a/tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts
+++ b/tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts
@@ -63,12 +63,30 @@ test.describe("onboarding recent regressions", () => {
// Regression coverage for #1077 / PR #1089: first-run LLM setup
// should not default users to the OpenHands provider.
- test("defaults the LLM setup step to Anthropic Claude Opus", async ({
+ test("defaults the LLM setup step to OpenAI GPT-5.5", async ({
page,
}) => {
await showOnboarding(page, {
apiKey: SESSION_API_KEY,
- beforeGoto: () => routeSessionApiKey(page),
+ beforeGoto: async () => {
+ await routeSessionApiKey(page);
+ // Intercept GET /api/settings so the LLM form sees a clean
+ // base_url, forcing basic view mode regardless of what earlier
+ // specs configured. Registered AFTER routeSessionApiKey so
+ // Playwright's LIFO matching picks this up first for settings.
+ await page.route("**/api/settings", async (route, req) => {
+ if (req.method() !== "GET") {
+ await route.fallback();
+ return;
+ }
+ const response = await route.fetch();
+ const body = await response.json();
+ if (body?.agent_settings?.llm) {
+ body.agent_settings.llm.base_url = null;
+ }
+ await route.fulfill({ response, json: body });
+ });
+ },
});
await advanceOnboardingToLlmStep(page);
@@ -77,19 +95,18 @@ test.describe("onboarding recent regressions", () => {
await expect(
providerInput,
- "first-run onboarding should default to the Anthropic provider",
- ).toHaveValue("Anthropic", { timeout: 10_000 });
- // The model input displays the Anthropic model ID without the provider
- // prefix.
+ "first-run onboarding should default to the OpenAI provider",
+ ).toHaveValue("OpenAI", { timeout: 10_000 });
+ // The model input displays the model ID without the provider prefix.
await expect(
modelInput,
- "first-run onboarding should default to Claude Opus",
- ).toHaveValue("claude-opus-4-8", {
+ "first-run onboarding should default to GPT-5.5",
+ ).toHaveValue("gpt-5.5", {
timeout: 10_000,
});
await expect(
page.getByTestId("openhands-account-help"),
- "OpenHands account helper should stay hidden for Anthropic defaults",
+ "OpenHands account helper should stay hidden for OpenAI defaults",
).toHaveCount(0);
});
});
diff --git a/tests/e2e/support/onboarding-helpers.ts b/tests/e2e/support/onboarding-helpers.ts
index c14f6ec31e..0a0d0a31fe 100644
--- a/tests/e2e/support/onboarding-helpers.ts
+++ b/tests/e2e/support/onboarding-helpers.ts
@@ -13,6 +13,7 @@ export async function routeOnboardingLlmCatalog(page: Page) {
body: JSON.stringify({
models: {
anthropic: ["claude-opus-4-8"],
+ openai: ["gpt-5.5"],
openhands: ["claude-opus-4-5-20251101"],
},
}),
@@ -26,6 +27,7 @@ export async function routeOnboardingLlmCatalog(page: Page) {
body: JSON.stringify({
models: [
"anthropic/claude-opus-4-8",
+ "openai/gpt-5.5",
"openhands/claude-opus-4-5-20251101",
],
}),
@@ -36,7 +38,9 @@ export async function routeOnboardingLlmCatalog(page: Page) {
await route.fulfill({
status: 200,
contentType: "application/json",
- body: JSON.stringify({ providers: ["anthropic", "openhands"] }),
+ body: JSON.stringify({
+ providers: ["anthropic", "openai", "openhands"],
+ }),
});
});
}