🙌 OpenHands: AI-Driven Development
Default onboarding LLM to OpenAI GPT-5.5 (#1103)
Engel Nyst00062bdfad2d
diff --git a/AGENTS.md b/AGENTS.mdindex d56aad4951..191df3ad36 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -208,6 +208,45 @@ you are running inside of — NOT the automation backend.- **State isolation**: The Docker container uses its internal state directory (no host mount needed for tests). Each test run starts a fresh container.- CI workflow: `.github/workflows/mock-llm-docker-e2e.yml` has three triggers — all pull the already-built image from GHCR (no rebuild): (1) `workflow_run` fires automatically after the `Docker` workflow completes on main; (2) `pull_request` with the `e2e-tests` label polls the Docker workflow until it finishes for the PR's head SHA, then pulls the image (needed because `workflow_run` only fires for workflow files already on the default branch); (3) `workflow_dispatch` accepts a custom `docker_image` input. The image tag is derived from the commit SHA (`ghcr.io/openhands/agent-canvas:sha-<short>-amd64`). Fork PRs are skipped (no GHCR push). Report artifacts go to `test-results-mock-llm-docker/` and `playwright-report-mock-llm-docker/`.+## Debugging E2E Test Failures++When an E2E test fails in CI, use this workflow to diagnose the root cause efficiently:++### 1. Read the PR comment first+The mock-LLM E2E workflow posts a structured comment on the PR with a test results table, pass/fail status, and collapsible failure details including the Playwright error message. **Start here** — the error message usually reveals whether the failure is a locator mismatch, a timeout, or a missing element.++### 2. Download CI artifacts+Every failing test run uploads artifacts (`mock-llm-e2e-results` for npm, `test-results-mock-llm-docker` for Docker). Download them with:+```bash+gh run download <run_id> --repo OpenHands/agent-canvas --name mock-llm-e2e-results --dir /tmp/artifacts+```+Artifacts contain:+- `test-results-mock-llm/` — per-test directories with `test-failed-N.png` (screenshot at failure) and `error-context.md` (Playwright page snapshot as YAML accessibility tree + test source with the failing line marked)+- `playwright-report-mock-llm/` — full HTML report (`npx playwright show-report /tmp/artifacts/playwright-report-mock-llm`)++### 3. Inspect the error-context.md page snapshot+The `error-context.md` file contains a YAML accessibility tree of the entire page at the moment of failure. This is the single most useful artifact — it shows exactly what DOM elements exist, which tabs are selected, what text is in inputs, and whether a component rendered at all. Search for the element your test expects (e.g. `llm-provider-input`) to see if it's present or absent, and check surrounding context (tab selection state, form view mode, etc.) to understand why.++### 4. Common failure patterns++**"element(s) not found"** — The locator matched zero elements. The component either:+- Didn't render (conditional rendering path not taken — check the page snapshot for what DID render)+- Has a different `name`/`data-testid` than expected+- Is behind a lazy-load boundary that hasn't resolved++**Stale state from earlier serial tests** — Mock-LLM tests run serially (`workers: 1`) against a real agent-server. Earlier tests (conversation, automation) persist settings on the server. If your test depends on "clean" state but a prior test configured `llm_base_url`, `llm_model`, etc., the form may render in a different view mode. Use Playwright `page.route()` to intercept and normalize the settings response. Example: `routeOnboardingLlmCatalog` in `tests/e2e/support/onboarding-helpers.ts` intercepts `GET /api/settings` to clear `llm_base_url` so the LLM form always opens in "Basic" view.++**View mode mismatch (Basic vs Advanced)** — `LlmSettingsScreen` switches between "Basic" (renders `ModelSelector` with provider/model dropdowns) and "Advanced" (renders plain text inputs). The view is determined by `getInitialView()` which checks `currentSettings.llm_base_url` — a non-default base URL triggers "Advanced" view. If your test expects `input[name="llm-provider-input"]` but sees text inputs instead, the settings have a stale `base_url`.++**Playwright route interception vs real server** — In mock-LLM tests, routes registered with `page.route()` intercept at the browser level before requests reach the real agent-server. However, `page.route()` must be set up BEFORE `page.goto()`. The `showOnboarding` helper handles this correctly (routes are registered before navigation). Non-GET methods should use `route.fallback()` to pass through to the real server.++### 5. Running locally+```bash+npm run test:e2e:mock-llm # full suite+npm run test:e2e:mock-llm -- --headed # watch in browser+npm run test:e2e:mock-llm -- -g "test name" # run single test by name+```+## Additional Notes- **Published binary auth fix**: When users install the npm package globally (`npm install -g @openhands/agent-canvas`) and run `agent-canvas`, the pre-built static frontend has NO `VITE_SESSION_API_KEY` baked in (npm publish runs `npm run build` with no such env var). The runtime session key is generated when the CLI launches and reaches the frontend via `scripts/static-server.mjs --session-api-key <key>`, which injects a `<head>` script that does two things: (a) sets `window.__AGENT_CANVAS_SESSION_API_KEY__ = <key>` — read by `getBakedSessionApiKey()` in `src/api/agent-server-config.ts` as a fallback when the env var is empty, symmetric with `__AGENT_CANVAS_AUTH_REQUIRED__` / `isAuthRequired()`; (b) writes the same key into `localStorage['openhands-agent-server-config'].sessionApiKey`, always overwriting when the value differs, so any code path that still reads the legacy storage key (e.g. e2e fixtures) sees the live key. The window-global path is the load-bearing one — without it, `makeDefaultLocalBackend()` returns null on a fresh install, the backend registry seeds empty, and `root.tsx` traps the user behind the Manage Backends modal instead of onboarding. `scripts/dev-with-automation.mjs` and `scripts/dev-static.mjs` both pass `--session-api-key ${config.sessionApiKey}` when starting the static server.diff --git a/__tests__/api/option-service.test.ts b/__tests__/api/option-service.test.tsindex a8fe9ff317..b8ba9fe857 100644--- a/__tests__/api/option-service.test.ts+++ b/__tests__/api/option-service.test.ts@@ -94,8 +94,14 @@ describe("OptionService", () => {const models = await OptionService.getModels();expect(models.models).toContain("openhands/claude-opus-4-5-20251101");+ expect(models.models).toContain("openai/gpt-5.5");expect(models.verified_models).toContain("claude-opus-4-5-20251101");- expect(models.verified_providers).toEqual(["anthropic", "openhands"]);+ expect(models.verified_models).toContain("gpt-5.5");+ expect(models.verified_providers).toEqual([+ "anthropic",+ "openai",+ "openhands",+ ]);expect(models.default_model).toBeTruthy();});diff --git a/__tests__/components/onboarding/onboarding-modal.test.tsx b/__tests__/components/onboarding/onboarding-modal.test.tsxindex e0a4cff877..f938cfb9e6 100644--- a/__tests__/components/onboarding/onboarding-modal.test.tsx+++ b/__tests__/components/onboarding/onboarding-modal.test.tsx@@ -7,6 +7,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";import { __resetActiveStoreForTests } from "#/api/backend-registry/active-store";import { ActiveBackendProvider } from "#/contexts/active-backend-context";import { OnboardingModal } from "#/components/features/onboarding/onboarding-modal";+import { ONBOARDING_DEFAULT_LLM_MODEL } from "#/components/features/onboarding/steps/setup-llm-step";import { NavigationProvider } from "#/context/navigation-context";import SettingsService from "#/api/settings-service/settings-service.api";import { SecretsService } from "#/api/secrets-service";@@ -233,14 +234,14 @@ describe("OnboardingModal", () => {);});- it("pre-fills the LLM step with the Anthropic provider's Claude Opus model", () => {+ it("pre-fills the LLM step with OpenAI GPT-5.5", () => {renderModal();expect(llmSettingsScreenMock).toHaveBeenCalledTimes(1);expect(llmSettingsScreenMock).toHaveBeenCalledWith(expect.objectContaining({initialValueOverrides: {- "llm.model": "anthropic/claude-opus-4-8",+ "llm.model": ONBOARDING_DEFAULT_LLM_MODEL,},}),);diff --git a/src/components/features/onboarding/steps/setup-llm-step.tsx b/src/components/features/onboarding/steps/setup-llm-step.tsxindex b470ea922f..dda6ce765e 100644--- a/src/components/features/onboarding/steps/setup-llm-step.tsx+++ b/src/components/features/onboarding/steps/setup-llm-step.tsx@@ -15,14 +15,16 @@ interface SetupLlmStepProps {}/**- * Pre-fills the LLM form with the Anthropic provider's Claude Opus- * model. Most users won't have an OpenHands account, so onboarding- * routes directly through Anthropic by default. Keeping this as an- * explicit override marks the model dirty so the Next button persists- * the suggested default immediately.+ * Pre-fills the LLM form with the OpenAI GPT-5.5 model. The SDK's+ * bare default model is `gpt-5.5`; Canvas stores provider-qualified+ * LiteLLM model ids, so the onboarding override uses this OpenAI-prefixed+ * model id. Keeping this as an explicit override marks the model dirty so+ * the Next button persists the suggested default immediately.*/+export const ONBOARDING_DEFAULT_LLM_MODEL = "openai/gpt-5.5";+const ONBOARDING_LLM_OVERRIDES = {- "llm.model": "anthropic/claude-opus-4-8",+ "llm.model": ONBOARDING_DEFAULT_LLM_MODEL,} as const;/**diff --git a/src/i18n/translation.json b/src/i18n/translation.jsonindex 0f091ab62f..59fe00ae48 100644--- a/src/i18n/translation.json+++ b/src/i18n/translation.json@@ -27778,21 +27778,21 @@"uk": "Налаштуйте свій LLM"},"ONBOARDING$LLM_SUBTITLE": {- "en": "Pick a provider and model. We've pre-selected Anthropic Claude Opus.",- "ja": "プロバイダーとモデルを選択してください。Anthropic Claude Opus を事前に選択しています。",- "zh-CN": "选择提供商和模型。我们已为你预先选择 Anthropic Claude Opus。",- "zh-TW": "選擇供應商與模型。我們已預先選好 Anthropic Claude Opus。",- "ko-KR": "공급자와 모델을 선택하세요. 기본값으로 Anthropic Claude Opus가 선택되어 있습니다.",- "no": "Velg en leverandør og modell. Vi har forhåndsvalgt Anthropic Claude Opus.",- "ar": "اختر مزوّدًا ونموذجًا. لقد اخترنا مسبقًا Anthropic Claude Opus.",- "de": "Wählen Sie Anbieter und Modell. Wir haben Anthropic Claude Opus vorausgewählt.",- "fr": "Choisissez un fournisseur et un modèle. Nous avons présélectionné Anthropic Claude Opus.",- "it": "Scegli un provider e un modello. Abbiamo preselezionato Anthropic Claude Opus.",- "pt": "Escolha um provedor e modelo. Já selecionamos o Anthropic Claude Opus.",- "es": "Elige un proveedor y un modelo. Hemos preseleccionado Anthropic Claude Opus.",- "ca": "Trieu un proveïdor i model. Hem preseleccionat Anthropic Claude Opus.",- "tr": "Bir sağlayıcı ve model seçin. Anthropic Claude Opus'u önceden seçtik.",- "uk": "Виберіть провайдера та модель. Anthropic Claude Opus уже обрано."+ "en": "Pick a provider and model. We've pre-selected OpenAI GPT-5.5.",+ "ja": "プロバイダーとモデルを選択してください。OpenAI GPT-5.5 を事前に選択しています。",+ "zh-CN": "选择提供商和模型。我们已为你预先选择 OpenAI GPT-5.5。",+ "zh-TW": "選擇供應商與模型。我們已預先選好 OpenAI GPT-5.5。",+ "ko-KR": "공급자와 모델을 선택하세요. 기본값으로 OpenAI GPT-5.5가 선택되어 있습니다.",+ "no": "Velg en leverandør og modell. Vi har forhåndsvalgt OpenAI GPT-5.5.",+ "ar": "اختر مزوّدًا ونموذجًا. لقد اخترنا مسبقًا OpenAI GPT-5.5.",+ "de": "Wählen Sie Anbieter und Modell. Wir haben OpenAI GPT-5.5 vorausgewählt.",+ "fr": "Choisissez un fournisseur et un modèle. Nous avons présélectionné OpenAI GPT-5.5.",+ "it": "Scegli un provider e un modello. Abbiamo preselezionato OpenAI GPT-5.5.",+ "pt": "Escolha um provedor e modelo. Já selecionamos o OpenAI GPT-5.5.",+ "es": "Elige un proveedor y modelo. Hemos preseleccionado OpenAI GPT-5.5.",+ "ca": "Trieu un proveïdor i model. Hem preseleccionat OpenAI GPT-5.5.",+ "tr": "Bir sağlayıcı ve model seçin. OpenAI GPT-5.5'i önceden seçtik.",+ "uk": "Виберіть провайдера та модель. OpenAI GPT-5.5 уже обрано."},"ONBOARDING$HELLO_TITLE": {"en": "Say hello 👋",diff --git a/src/mocks/settings-handlers.ts b/src/mocks/settings-handlers.tsindex 05e41dcdfa..c3d8daf856 100644--- a/src/mocks/settings-handlers.ts+++ b/src/mocks/settings-handlers.ts@@ -439,6 +439,7 @@ const MOCK_MODELS = ["anthropic/claude-opus-4-5-20251101","anthropic/claude-opus-4-8","openai/gpt-3.5-turbo",+ "openai/gpt-5.5","openai/gpt-4o","openai/gpt-4o-mini","openhands/claude-sonnet-4-20250514",@@ -453,6 +454,7 @@ const MOCK_VERIFIED_MODELS = new Set(["anthropic/claude-opus-4-5-20251101","anthropic/claude-opus-4-8","anthropic/claude-sonnet-4-5-20250929",+ "openai/gpt-5.5","openhands/claude-opus-4-5-20251101","openhands/claude-sonnet-4-5-20250929","openhands/minimax-m2.7",diff --git a/tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts b/tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.tsindex 88184f8da1..fcfe90c69c 100644--- a/tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts+++ b/tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts@@ -63,12 +63,30 @@ test.describe("onboarding recent regressions", () => {// Regression coverage for #1077 / PR #1089: first-run LLM setup// should not default users to the OpenHands provider.- test("defaults the LLM setup step to Anthropic Claude Opus", async ({+ test("defaults the LLM setup step to OpenAI GPT-5.5", async ({page,}) => {await showOnboarding(page, {apiKey: SESSION_API_KEY,- beforeGoto: () => routeSessionApiKey(page),+ beforeGoto: async () => {+ await routeSessionApiKey(page);+ // Intercept GET /api/settings so the LLM form sees a clean+ // base_url, forcing basic view mode regardless of what earlier+ // specs configured. Registered AFTER routeSessionApiKey so+ // Playwright's LIFO matching picks this up first for settings.+ await page.route("**/api/settings", async (route, req) => {+ if (req.method() !== "GET") {+ await route.fallback();+ return;+ }+ const response = await route.fetch();+ const body = await response.json();+ if (body?.agent_settings?.llm) {+ body.agent_settings.llm.base_url = null;+ }+ await route.fulfill({ response, json: body });+ });+ },});await advanceOnboardingToLlmStep(page);@@ -77,19 +95,18 @@ test.describe("onboarding recent regressions", () => {await expect(providerInput,- "first-run onboarding should default to the Anthropic provider",- ).toHaveValue("Anthropic", { timeout: 10_000 });- // The model input displays the Anthropic model ID without the provider- // prefix.+ "first-run onboarding should default to the OpenAI provider",+ ).toHaveValue("OpenAI", { timeout: 10_000 });+ // The model input displays the model ID without the provider prefix.await expect(modelInput,- "first-run onboarding should default to Claude Opus",- ).toHaveValue("claude-opus-4-8", {+ "first-run onboarding should default to GPT-5.5",+ ).toHaveValue("gpt-5.5", {timeout: 10_000,});await expect(page.getByTestId("openhands-account-help"),- "OpenHands account helper should stay hidden for Anthropic defaults",+ "OpenHands account helper should stay hidden for OpenAI defaults",).toHaveCount(0);});});diff --git a/tests/e2e/support/onboarding-helpers.ts b/tests/e2e/support/onboarding-helpers.tsindex c14f6ec31e..0a0d0a31fe 100644--- a/tests/e2e/support/onboarding-helpers.ts+++ b/tests/e2e/support/onboarding-helpers.ts@@ -13,6 +13,7 @@ export async function routeOnboardingLlmCatalog(page: Page) {body: JSON.stringify({models: {anthropic: ["claude-opus-4-8"],+ openai: ["gpt-5.5"],openhands: ["claude-opus-4-5-20251101"],},}),@@ -26,6 +27,7 @@ export async function routeOnboardingLlmCatalog(page: Page) {body: JSON.stringify({models: ["anthropic/claude-opus-4-8",+ "openai/gpt-5.5","openhands/claude-opus-4-5-20251101",],}),@@ -36,7 +38,9 @@ export async function routeOnboardingLlmCatalog(page: Page) {await route.fulfill({status: 200,contentType: "application/json",- body: JSON.stringify({ providers: ["anthropic", "openhands"] }),+ body: JSON.stringify({+ providers: ["anthropic", "openai", "openhands"],+ }),});});}