🙌 OpenHands: AI-Driven Development
fix(verify-openhands): point the default agent profile at the preset's model (#18209)
Engel Nyste726c0620ed4
diff --git a/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md b/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.mdindex e282e83c2f..7a60fc4f37 100644--- a/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md+++ b/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md@@ -69,7 +69,7 @@ Local run (`launch --new`):- **Skip checklist box (`F01.onboarding-skip-checklist`).** Run `control-openhands browser goto /conversations` and `control-openhands browser count 'testid=sidebar-onboarding-checklist'` (`1`). Run `control-openhands browser goto '/?previewOnboardingStep=3'`, then `control-openhands browser click 'text=Skip Getting Started checklist'`. `control-openhands browser eval "document.querySelector('[data-testid=onboarding-skip-getting-started-checklist]').checked"` is `true`. Run `control-openhands browser goto /conversations` and `control-openhands browser reload`; the checklist count is `0`. On `/settings/app`, `control-openhands browser eval "document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked"` is `false`. Restore it: go back to `/?previewOnboardingStep=3`, where the box now reads `true`, click the same text again, and the checklist count on `/conversations` is `1`.- **Phone (`F01.onboarding-phone`).** Run `control-openhands browser viewport phone`. Then for N in 1, 2 and 3 run `control-openhands browser goto '/?previewOnboardingStep=N'`, `control-openhands browser bbox 'testid=onboarding-modal'` and `control-openhands browser screenshot --feature F01.onboarding-phone --name step-N`. The modal is 351 px wide with `insideViewport` `true` and `pageHorizontalOverflow` `false`, and Back/Next or Back/Close are visible. Return with `control-openhands browser viewport desktop`.- **Error page (`F01.route-error-boundary`).** Run `control-openhands browser errors --clear`, `control-openhands browser goto /this-route-does-not-exist` and `control-openhands browser snapshot 'testid=not-found-screen'`. It shows `heading "Page not found"`, the paragraph `This address does not match any page. Check the URL, or go back to the home page.` and `link "Home"` (`/url: /`). `control-openhands browser count 'aside[data-collapsed]'` is `1`: the sidebar stays. `control-openhands browser screenshot --feature F01.route-error-boundary --name not-found` shows the message and the Home button centered beside the sidebar. `control-openhands browser errors --app-only` reports `pageErrors` `0` and `appErrors` `0`. Run `control-openhands browser click 'testid=not-found-home-link' --expect-url '/$'`; `control-openhands browser count 'testid=home-screen'` is `1`.-- **Onboarding again with the saved endpoint (`F01.onboarding-repeat-endpoint`).** Two more first runs on this stack, both through the All view, which shows the Base URL field. Pass 1 saves a custom endpoint: `control-openhands browser reset`, `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-agent-next'`, `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="1"]'`, `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-all-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` (empty). Run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' openai/deepseek-chat`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=base-url-input' https://api.deepseek.com/v1`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="2"]'`. `control-openhands llm show` lists `deepseek-chat` (`openai/deepseek-chat`, `base_url` `https://api.deepseek.com/v1`) as `active_profile`. Say hello as above (`browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'`, `browser press Enter --selector 'testid=onboarding-hello-input'`, `browser wait-url '/conversations/[0-9a-f-]+'`), then `control-openhands conversation wait <id> --timeout 180` is `finished` and the agent `MessageEvent` is `hello`. Pass 2 types the same endpoint again: repeat the steps up to the All view. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` is now `https://api.deepseek.com/v1`, the saved value. Fill the model `openai/deepseek-v4-flash` (a DeepSeek alias, so the new profile gets its own name), the same Base URL and the key, then Next and wait for step 2 (`control-openhands browser screenshot 'testid=onboarding-step-setup-llm' --feature F01.onboarding-repeat-endpoint --name second-pass-form` before Next). Expected: `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`, and the say-hello conversation finishes with `hello`. Known failure (reproduced 2026-10-08 at `53c8b4d`): the profile is saved with `base_url` `null`. The say-hello conversation then ends in `error`, and `control-openhands conversation events <id> --last 10` shows `ConversationErrorEvent` `LLMAuthenticationError: litellm.AuthenticationError: ... OpenAIException - Incorrect API key provided`: the DeepSeek key went to OpenAI. The chat shows `Your LLM API key appears to be invalid or has expired.` (`browser screenshot --feature F01.onboarding-repeat-endpoint --name second-pass-error`). Issue #17884, fix in #17889. Restore: run `control-openhands llm preset deepseek`, then point the `default` agent profile back at it. Run `control-openhands browser goto /settings/agents`, `control-openhands browser click 'testid=agent-profile-row >> has-text=default >> testid=agent-profile-menu-trigger'` and `control-openhands browser click 'testid=agent-profile-actions-menu >> text=Edit'`. Then run `control-openhands browser choose 'testid=agent-profile-llm-selector' 'deepseek-flash (deepseek/deepseek-flash)'` and `control-openhands browser click 'testid=save-agent-profile-btn'`; the toast reads `Profile "default" updated`. Only then do `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps.+- **Onboarding again with the saved endpoint (`F01.onboarding-repeat-endpoint`).** Two more first runs on this stack, both through the All view, which shows the Base URL field. Pass 1 saves a custom endpoint: `control-openhands browser reset`, `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-agent-next'`, `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="1"]'`, `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-all-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` (empty). Run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' openai/deepseek-chat`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=base-url-input' https://api.deepseek.com/v1`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="2"]'`. `control-openhands llm show` lists `deepseek-chat` (`openai/deepseek-chat`, `base_url` `https://api.deepseek.com/v1`) as `active_profile`. Say hello as above (`browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'`, `browser press Enter --selector 'testid=onboarding-hello-input'`, `browser wait-url '/conversations/[0-9a-f-]+'`), then `control-openhands conversation wait <id> --timeout 180` is `finished` and the agent `MessageEvent` is `hello`. Pass 2 types the same endpoint again: repeat the steps up to the All view. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` is now `https://api.deepseek.com/v1`, the saved value. Fill the model `openai/deepseek-v4-flash` (a DeepSeek alias, so the new profile gets its own name), the same Base URL and the key, then Next and wait for step 2 (`control-openhands browser screenshot 'testid=onboarding-step-setup-llm' --feature F01.onboarding-repeat-endpoint --name second-pass-form` before Next). Expected: `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`, and the say-hello conversation finishes with `hello`. Known failure (reproduced 2026-10-08 at `53c8b4d`): the profile is saved with `base_url` `null`. The say-hello conversation then ends in `error`, and `control-openhands conversation events <id> --last 10` shows `ConversationErrorEvent` `LLMAuthenticationError: litellm.AuthenticationError: ... OpenAIException - Incorrect API key provided`: the DeepSeek key went to OpenAI. The chat shows `Your LLM API key appears to be invalid or has expired.` (`browser screenshot --feature F01.onboarding-repeat-endpoint --name second-pass-error`). Issue #17884, fix in #17889. Restore: run `control-openhands llm preset deepseek`. It also points the `default` agent profile back at `deepseek-flash`: its output has `repointed` from `deepseek-v4-flash` to `deepseek-flash`. Then `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps.- **Clean up.** Delete the hello conversation from its header menu: `control-openhands browser goto /conversations/<id>` (the hello conversation's `<id>` from Say hello; the Error page bullet left `/`), `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'`, `control-openhands browser click 'role=button[name="Confirm Delete"]'`. `control-openhands conversation list` then shows `count` `0`.Public run A (`launch --new --public`), backend step walked:@@ -102,7 +102,7 @@ Not reachable locally:- Slide indices renumber. After the public backend step succeeds, "Choose your agent" becomes index `0` and has no Back. `previewOnboardingStep` counts phases with the backend step included, so `0` and `1` look identical on a healthy backend.- The consent modal (z-70) sits over the onboarding modal, and clicks at the screen center land on agent tiles once it closes. A forced click on `first-run-onboarding-screen` selected Codex. Check backdrop non-dismissal with `browser mouse-click 40 40`, a corner outside the modal.- The say-hello default message ("Create a basic webpage…") makes the agent work for a while. Replace it with a tiny prompt to keep model cost down.-- On a local backend, Next on the LLM step also points the `default` agent profile at the profile it just created. Running `llm preset deepseek` later does not move that pointer back. Deleting the onboarding profile then answers `409` `LLM profile is referenced by 1 agent profile(s): default`. Re-point it first in Settings → Agents (the Restore steps of `F01.onboarding-repeat-endpoint`).+- On a local backend, Next on the LLM step also points the `default` agent profile at the profile it just created. Activating another LLM profile does not move that pointer, and while it stays, deleting the onboarding profile answers `409` `LLM profile is referenced by 1 agent profile(s): default`. `control-openhands llm preset deepseek` moves it to `deepseek-flash` and reports `repointed`; `llm set` does not. In the UI, edit `default` in Settings → Agents and pick the profile in `testid=agent-profile-llm-selector`.- The LLM step saves only the fields that differ from the saved settings, and the profile it creates is built from that same diff. That is the cause of the `F01.onboarding-repeat-endpoint` failure (#17884, fix in #17889). It is also why the mock-LLM onboarding e2e failed after earlier specs had saved the mock endpoint. The first pass of that bullet passes on a run whose saved Base URL is still empty.- The tab title prefixes a status emoji to the stored title. A generated title that already starts with an emoji therefore shows twice (`✅ ✅ Reply hello without tools | OpenHands`). Right after launch the title is `Conversation <id prefix>` until a reload picks up the generated one.- Right after Next, the ACP step reads "Checking for an existing Claude Code login…" with Next disabled. Wait for `testid=onboarding-acp-auth-detected` before reading the banner. The "already signed in" banner depends on the host's CLI login. On a clean CI machine the fields become required, and Next is blocked until they are filled.diff --git a/.agents/skills/verify-openhands/scripts/control-openhands.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.mjsindex b21f47d640..7f63db1b1c 100755--- a/.agents/skills/verify-openhands/scripts/control-openhands.mjs+++ b/.agents/skills/verify-openhands/scripts/control-openhands.mjs@@ -42,6 +42,10 @@ import {} from "./lib/testids.mjs";import { tmuxPathFor } from "./lib/tmux-path.mjs";import { browserCallLimit } from "./lib/call-limit.mjs";+import {+ DEFAULT_AGENT_PROFILE,+ agentProfileRepoint,+} from "./lib/agent-profile-repoint.mjs";import {PAGE_LIMIT,collectEvents,@@ -1492,6 +1496,30 @@ async function activateProfile(run, name) {);}+// Onboarding pins the `default` agent profile to the LLM profile it created;+// `llm preset` moves it to the one it activates (see+// lib/agent-profile-repoint.mjs). `llm set` leaves it: its throwaway profiles+// would otherwise become `default`'s and refuse deletion.+async function repointDefaultAgentProfile(run, target) {+ const path = `/api/agent-profiles/${DEFAULT_AGENT_PROFILE}`;+ const detail = await http(run, "GET", path);+ if (detail.status === 404) return undefined;+ if (!detail.ok)+ throw new CliError(+ `Reading agent profile ${DEFAULT_AGENT_PROFILE} failed: ${detail.status} ${detail.text.slice(0, 300)}`,+ );+ const llm = await http(run, "GET", "/api/profiles");+ const names = (llm.json?.profiles ?? []).map((p) => p.name);+ const plan = agentProfileRepoint(detail.json?.profile, names, target);+ if (!plan) return undefined;+ const saved = await http(run, "POST", path, { body: plan.body });+ if (!saved.ok)+ throw new CliError(+ `Pointing agent profile ${DEFAULT_AGENT_PROFILE} at ${target} failed: ${saved.status} ${saved.text.slice(0, 300)}`,+ );+ return { agentProfile: DEFAULT_AGENT_PROFILE, from: plan.from, to: target };+}+const PRESETS = {deepseek: {envVar: "DEEPSEEK_API_KEY",@@ -1571,7 +1599,11 @@ async function cmdLlm({ positional, flags }) {created.push(profile.name);}const active = preset.profiles.find((p) => p.activate);- if (active) await activateProfile(run, active.name);+ let repointed;+ if (active) {+ await activateProfile(run, active.name);+ repointed = await repointDefaultAgentProfile(run, active.name);+ }const settings = await http(run, "GET", "/api/settings");out({ok: true,@@ -1579,6 +1611,7 @@ async function cmdLlm({ positional, flags }) {profiles: created,active: active?.name,activeModel: settings.json?.agent_settings?.llm?.model,+ ...(repointed ? { repointed } : {}),});return;}@@ -4072,6 +4105,10 @@ set/preset validate with a 1-token completion first (skip with --no-validate).Keys are read from an environment variable or file, never from argv.'preset deepseek' saves deepseek-flash (deepseek/deepseek-flash, activated) anddeepseek-pro (deepseek/deepseek-v4-pro). Prefer flash; it is cheaper.+'preset' also points the 'default' agent profile at deepseek-flash when it+references another LLM profile that exists, as onboarding leaves it; the output+then has 'repointed'. A reference to a missing profile (a fresh run's seed),+named agent profiles and 'set' leave agent profiles as they are.Examples:DEEPSEEK_API_KEY=... control-openhands llm preset deepseekdiff --git a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjsindex c7dd417fc5..c5a7dd959b 100755--- a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs+++ b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs@@ -36,6 +36,7 @@ import { routePattern } from "./lib/route-pattern.mjs";import { resolveTestids } from "./lib/testids.mjs";import { tmuxPathFor } from "./lib/tmux-path.mjs";import { browserCallLimit } from "./lib/call-limit.mjs";+import { agentProfileRepoint } from "./lib/agent-profile-repoint.mjs";import {canvasSize,encodeRecording,@@ -1444,6 +1445,72 @@ test("fixture skill --commit records the project skill once and reports a re-runassert.match(personal.json.error, /needs --repo/);});+test("llm preset repoints default only off another live LLM profile", () => {+ const seeded = {+ id: "a1",+ name: "default",+ revision: 3,+ schema_version: 1,+ agent_kind: "openhands",+ llm_profile_ref: "deepseek-chat",+ enable_sub_agents: true,+ mcp_server_refs: ["qa_mcp_a"],+ };+ const names = ["deepseek-chat", "deepseek-flash", "deepseek-pro"];+ // Onboarding left default on its own profile: move it, keep every other field.+ assert.deepEqual(agentProfileRepoint(seeded, names, "deepseek-flash"), {+ from: "deepseek-chat",+ body: {+ schema_version: 1,+ agent_kind: "openhands",+ llm_profile_ref: "deepseek-flash",+ enable_sub_agents: true,+ mcp_server_refs: ["qa_mcp_a"],+ },+ });+ // Already there, a fresh run's missing `default` ref, or no ref: unchanged.+ assert.equal(agentProfileRepoint(seeded, names, "deepseek-chat"), null);+ assert.equal(+ agentProfileRepoint(+ { ...seeded, llm_profile_ref: "default" },+ names,+ "deepseek-flash",+ ),+ null,+ );+ assert.equal(+ agentProfileRepoint(+ { ...seeded, llm_profile_ref: null },+ names,+ "deepseek-flash",+ ),+ null,+ );+ // An ACP default owns its own model; an older backend has no profile.+ assert.equal(+ agentProfileRepoint(+ { ...seeded, agent_kind: "acp" },+ names,+ "deepseek-flash",+ ),+ null,+ );+ assert.equal(agentProfileRepoint(undefined, names, "deepseek-flash"), null);+});++test("llm help says when preset repoints the default agent profile", () => {+ const help = spawnSync(process.execPath, [cli, "help", "llm"], {+ encoding: "utf8",+ });+ assert.equal(help.status, 0);+ assert.match(+ help.stdout,+ /'preset' also points the 'default' agent profile at deepseek-flash/,+ );+ assert.match(help.stdout, /and 'set' leave agent profiles as they are/);+ assert.match(help.stdout, /'repointed'/);+});+test("a recording's timeline cuts the lead-in and pauses, and holds the end", () => {// Changes at 0, 3, 3.2 and 10 s; stopped at 10.5 s.const frames = [{ t: 0 }, { t: 3000 }, { t: 3200 }, { t: 10000 }];diff --git a/.agents/skills/verify-openhands/scripts/lib/agent-profile-repoint.mjs b/.agents/skills/verify-openhands/scripts/lib/agent-profile-repoint.mjsnew file mode 100644index 0000000000..c4c4f6be9e--- /dev/null+++ b/.agents/skills/verify-openhands/scripts/lib/agent-profile-repoint.mjs@@ -0,0 +1,36 @@+// Whether `llm preset` must also move the seeded `default` agent profile onto+// the LLM profile it activates.+//+// On a local backend, onboarding points `default` at the LLM profile it+// creates (useApplyOnboardingAgentProfile). Activating another LLM profile+// leaves that reference behind. The Agents page then still shows the+// onboarding model on `default`, a launch that picks `default` explicitly+// uses it, and deleting the onboarding profile answers 409 ("referenced by+// 1 agent profile(s)").+//+// Only `default` moves, and only when it is an OpenHands profile whose+// reference names another LLM profile that exists. Named agent profiles are+// deliberate picks. `llm set` never moves it: recipes activate throwaway+// profiles with it and delete them afterwards. A reference to a missing+// profile is the fresh-run seed: the app already falls back to the active LLM+// profile for it, and F13.stale-llm-ref drives that state.++export const DEFAULT_AGENT_PROFILE = "default";++/**+ * @param {object | undefined} agentProfile `profile` from+ * `GET /api/agent-profiles/default`+ * @param {string[]} llmProfileNames names from `GET /api/profiles`+ * @param {string} target the LLM profile just activated+ * @returns {{ from: string, body: object } | null} the save body for+ * `POST /api/agent-profiles/default`, or null when nothing should move+ */+export function agentProfileRepoint(agentProfile, llmProfileNames, target) {+ if (agentProfile?.agent_kind !== "openhands") return null;+ const from = agentProfile.llm_profile_ref;+ if (!from || from === target || !llmProfileNames.includes(from)) return null;+ // The save is a whole-profile overwrite: keep every stored field and drop+ // only the identity the server owns (as the Agents editor does).+ const { id, name, revision, ...stored } = agentProfile;+ return { from, body: { ...stored, llm_profile_ref: target } };+}