Skip to content

🙌 OpenHands: AI-Driven Development

fix(verify-openhands): point the default agent profile at the preset's model (#18209)

Engel Nyste726c0620ed4
diff --git a/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md b/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md
index e282e83c2f..7a60fc4f37 100644
--- a/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md
+++ b/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md
@@ -69,7 +69,7 @@ Local run (`launch --new`):
- **Skip checklist box (`F01.onboarding-skip-checklist`).** Run `control-openhands browser goto /conversations` and `control-openhands browser count 'testid=sidebar-onboarding-checklist'` (`1`). Run `control-openhands browser goto '/?previewOnboardingStep=3'`, then `control-openhands browser click 'text=Skip Getting Started checklist'`. `control-openhands browser eval "document.querySelector('[data-testid=onboarding-skip-getting-started-checklist]').checked"` is `true`. Run `control-openhands browser goto /conversations` and `control-openhands browser reload`; the checklist count is `0`. On `/settings/app`, `control-openhands browser eval "document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked"` is `false`. Restore it: go back to `/?previewOnboardingStep=3`, where the box now reads `true`, click the same text again, and the checklist count on `/conversations` is `1`.
- **Phone (`F01.onboarding-phone`).** Run `control-openhands browser viewport phone`. Then for N in 1, 2 and 3 run `control-openhands browser goto '/?previewOnboardingStep=N'`, `control-openhands browser bbox 'testid=onboarding-modal'` and `control-openhands browser screenshot --feature F01.onboarding-phone --name step-N`. The modal is 351 px wide with `insideViewport` `true` and `pageHorizontalOverflow` `false`, and Back/Next or Back/Close are visible. Return with `control-openhands browser viewport desktop`.
- **Error page (`F01.route-error-boundary`).** Run `control-openhands browser errors --clear`, `control-openhands browser goto /this-route-does-not-exist` and `control-openhands browser snapshot 'testid=not-found-screen'`. It shows `heading "Page not found"`, the paragraph `This address does not match any page. Check the URL, or go back to the home page.` and `link "Home"` (`/url: /`). `control-openhands browser count 'aside[data-collapsed]'` is `1`: the sidebar stays. `control-openhands browser screenshot --feature F01.route-error-boundary --name not-found` shows the message and the Home button centered beside the sidebar. `control-openhands browser errors --app-only` reports `pageErrors` `0` and `appErrors` `0`. Run `control-openhands browser click 'testid=not-found-home-link' --expect-url '/$'`; `control-openhands browser count 'testid=home-screen'` is `1`.
-- **Onboarding again with the saved endpoint (`F01.onboarding-repeat-endpoint`).** Two more first runs on this stack, both through the All view, which shows the Base URL field. Pass 1 saves a custom endpoint: `control-openhands browser reset`, `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-agent-next'`, `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="1"]'`, `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-all-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` (empty). Run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' openai/deepseek-chat`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=base-url-input' https://api.deepseek.com/v1`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="2"]'`. `control-openhands llm show` lists `deepseek-chat` (`openai/deepseek-chat`, `base_url` `https://api.deepseek.com/v1`) as `active_profile`. Say hello as above (`browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'`, `browser press Enter --selector 'testid=onboarding-hello-input'`, `browser wait-url '/conversations/[0-9a-f-]+'`), then `control-openhands conversation wait <id> --timeout 180` is `finished` and the agent `MessageEvent` is `hello`. Pass 2 types the same endpoint again: repeat the steps up to the All view. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` is now `https://api.deepseek.com/v1`, the saved value. Fill the model `openai/deepseek-v4-flash` (a DeepSeek alias, so the new profile gets its own name), the same Base URL and the key, then Next and wait for step 2 (`control-openhands browser screenshot 'testid=onboarding-step-setup-llm' --feature F01.onboarding-repeat-endpoint --name second-pass-form` before Next). Expected: `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`, and the say-hello conversation finishes with `hello`. Known failure (reproduced 2026-10-08 at `53c8b4d`): the profile is saved with `base_url` `null`. The say-hello conversation then ends in `error`, and `control-openhands conversation events <id> --last 10` shows `ConversationErrorEvent` `LLMAuthenticationError: litellm.AuthenticationError: ... OpenAIException - Incorrect API key provided`: the DeepSeek key went to OpenAI. The chat shows `Your LLM API key appears to be invalid or has expired.` (`browser screenshot --feature F01.onboarding-repeat-endpoint --name second-pass-error`). Issue #17884, fix in #17889. Restore: run `control-openhands llm preset deepseek`, then point the `default` agent profile back at it. Run `control-openhands browser goto /settings/agents`, `control-openhands browser click 'testid=agent-profile-row >> has-text=default >> testid=agent-profile-menu-trigger'` and `control-openhands browser click 'testid=agent-profile-actions-menu >> text=Edit'`. Then run `control-openhands browser choose 'testid=agent-profile-llm-selector' 'deepseek-flash (deepseek/deepseek-flash)'` and `control-openhands browser click 'testid=save-agent-profile-btn'`; the toast reads `Profile "default" updated`. Only then do `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps.
+- **Onboarding again with the saved endpoint (`F01.onboarding-repeat-endpoint`).** Two more first runs on this stack, both through the All view, which shows the Base URL field. Pass 1 saves a custom endpoint: `control-openhands browser reset`, `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-agent-next'`, `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="1"]'`, `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-all-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` (empty). Run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' openai/deepseek-chat`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=base-url-input' https://api.deepseek.com/v1`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="2"]'`. `control-openhands llm show` lists `deepseek-chat` (`openai/deepseek-chat`, `base_url` `https://api.deepseek.com/v1`) as `active_profile`. Say hello as above (`browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'`, `browser press Enter --selector 'testid=onboarding-hello-input'`, `browser wait-url '/conversations/[0-9a-f-]+'`), then `control-openhands conversation wait <id> --timeout 180` is `finished` and the agent `MessageEvent` is `hello`. Pass 2 types the same endpoint again: repeat the steps up to the All view. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` is now `https://api.deepseek.com/v1`, the saved value. Fill the model `openai/deepseek-v4-flash` (a DeepSeek alias, so the new profile gets its own name), the same Base URL and the key, then Next and wait for step 2 (`control-openhands browser screenshot 'testid=onboarding-step-setup-llm' --feature F01.onboarding-repeat-endpoint --name second-pass-form` before Next). Expected: `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`, and the say-hello conversation finishes with `hello`. Known failure (reproduced 2026-10-08 at `53c8b4d`): the profile is saved with `base_url` `null`. The say-hello conversation then ends in `error`, and `control-openhands conversation events <id> --last 10` shows `ConversationErrorEvent` `LLMAuthenticationError: litellm.AuthenticationError: ... OpenAIException - Incorrect API key provided`: the DeepSeek key went to OpenAI. The chat shows `Your LLM API key appears to be invalid or has expired.` (`browser screenshot --feature F01.onboarding-repeat-endpoint --name second-pass-error`). Issue #17884, fix in #17889. Restore: run `control-openhands llm preset deepseek`. It also points the `default` agent profile back at `deepseek-flash`: its output has `repointed` from `deepseek-v4-flash` to `deepseek-flash`. Then `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps.
- **Clean up.** Delete the hello conversation from its header menu: `control-openhands browser goto /conversations/<id>` (the hello conversation's `<id>` from Say hello; the Error page bullet left `/`), `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'`, `control-openhands browser click 'role=button[name="Confirm Delete"]'`. `control-openhands conversation list` then shows `count` `0`.
Public run A (`launch --new --public`), backend step walked:
@@ -102,7 +102,7 @@ Not reachable locally:
- Slide indices renumber. After the public backend step succeeds, "Choose your agent" becomes index `0` and has no Back. `previewOnboardingStep` counts phases with the backend step included, so `0` and `1` look identical on a healthy backend.
- The consent modal (z-70) sits over the onboarding modal, and clicks at the screen center land on agent tiles once it closes. A forced click on `first-run-onboarding-screen` selected Codex. Check backdrop non-dismissal with `browser mouse-click 40 40`, a corner outside the modal.
- The say-hello default message ("Create a basic webpage…") makes the agent work for a while. Replace it with a tiny prompt to keep model cost down.
-- On a local backend, Next on the LLM step also points the `default` agent profile at the profile it just created. Running `llm preset deepseek` later does not move that pointer back. Deleting the onboarding profile then answers `409` `LLM profile is referenced by 1 agent profile(s): default`. Re-point it first in Settings → Agents (the Restore steps of `F01.onboarding-repeat-endpoint`).
+- On a local backend, Next on the LLM step also points the `default` agent profile at the profile it just created. Activating another LLM profile does not move that pointer, and while it stays, deleting the onboarding profile answers `409` `LLM profile is referenced by 1 agent profile(s): default`. `control-openhands llm preset deepseek` moves it to `deepseek-flash` and reports `repointed`; `llm set` does not. In the UI, edit `default` in Settings → Agents and pick the profile in `testid=agent-profile-llm-selector`.
- The LLM step saves only the fields that differ from the saved settings, and the profile it creates is built from that same diff. That is the cause of the `F01.onboarding-repeat-endpoint` failure (#17884, fix in #17889). It is also why the mock-LLM onboarding e2e failed after earlier specs had saved the mock endpoint. The first pass of that bullet passes on a run whose saved Base URL is still empty.
- The tab title prefixes a status emoji to the stored title. A generated title that already starts with an emoji therefore shows twice (`✅ ✅ Reply hello without tools | OpenHands`). Right after launch the title is `Conversation <id prefix>` until a reload picks up the generated one.
- Right after Next, the ACP step reads "Checking for an existing Claude Code login…" with Next disabled. Wait for `testid=onboarding-acp-auth-detected` before reading the banner. The "already signed in" banner depends on the host's CLI login. On a clean CI machine the fields become required, and Next is blocked until they are filled.
diff --git a/.agents/skills/verify-openhands/scripts/control-openhands.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.mjs
index b21f47d640..7f63db1b1c 100755
--- a/.agents/skills/verify-openhands/scripts/control-openhands.mjs
+++ b/.agents/skills/verify-openhands/scripts/control-openhands.mjs
@@ -42,6 +42,10 @@ import {
} from "./lib/testids.mjs";
import { tmuxPathFor } from "./lib/tmux-path.mjs";
import { browserCallLimit } from "./lib/call-limit.mjs";
+import {
+ DEFAULT_AGENT_PROFILE,
+ agentProfileRepoint,
+} from "./lib/agent-profile-repoint.mjs";
import {
PAGE_LIMIT,
collectEvents,
@@ -1492,6 +1496,30 @@ async function activateProfile(run, name) {
);
}
+// Onboarding pins the `default` agent profile to the LLM profile it created;
+// `llm preset` moves it to the one it activates (see
+// lib/agent-profile-repoint.mjs). `llm set` leaves it: its throwaway profiles
+// would otherwise become `default`'s and refuse deletion.
+async function repointDefaultAgentProfile(run, target) {
+ const path = `/api/agent-profiles/${DEFAULT_AGENT_PROFILE}`;
+ const detail = await http(run, "GET", path);
+ if (detail.status === 404) return undefined;
+ if (!detail.ok)
+ throw new CliError(
+ `Reading agent profile ${DEFAULT_AGENT_PROFILE} failed: ${detail.status} ${detail.text.slice(0, 300)}`,
+ );
+ const llm = await http(run, "GET", "/api/profiles");
+ const names = (llm.json?.profiles ?? []).map((p) => p.name);
+ const plan = agentProfileRepoint(detail.json?.profile, names, target);
+ if (!plan) return undefined;
+ const saved = await http(run, "POST", path, { body: plan.body });
+ if (!saved.ok)
+ throw new CliError(
+ `Pointing agent profile ${DEFAULT_AGENT_PROFILE} at ${target} failed: ${saved.status} ${saved.text.slice(0, 300)}`,
+ );
+ return { agentProfile: DEFAULT_AGENT_PROFILE, from: plan.from, to: target };
+}
+
const PRESETS = {
deepseek: {
envVar: "DEEPSEEK_API_KEY",
@@ -1571,7 +1599,11 @@ async function cmdLlm({ positional, flags }) {
created.push(profile.name);
}
const active = preset.profiles.find((p) => p.activate);
- if (active) await activateProfile(run, active.name);
+ let repointed;
+ if (active) {
+ await activateProfile(run, active.name);
+ repointed = await repointDefaultAgentProfile(run, active.name);
+ }
const settings = await http(run, "GET", "/api/settings");
out({
ok: true,
@@ -1579,6 +1611,7 @@ async function cmdLlm({ positional, flags }) {
profiles: created,
active: active?.name,
activeModel: settings.json?.agent_settings?.llm?.model,
+ ...(repointed ? { repointed } : {}),
});
return;
}
@@ -4072,6 +4105,10 @@ set/preset validate with a 1-token completion first (skip with --no-validate).
Keys are read from an environment variable or file, never from argv.
'preset deepseek' saves deepseek-flash (deepseek/deepseek-flash, activated) and
deepseek-pro (deepseek/deepseek-v4-pro). Prefer flash; it is cheaper.
+'preset' also points the 'default' agent profile at deepseek-flash when it
+references another LLM profile that exists, as onboarding leaves it; the output
+then has 'repointed'. A reference to a missing profile (a fresh run's seed),
+named agent profiles and 'set' leave agent profiles as they are.
Examples:
DEEPSEEK_API_KEY=... control-openhands llm preset deepseek
diff --git a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs
index c7dd417fc5..c5a7dd959b 100755
--- a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs
+++ b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs
@@ -36,6 +36,7 @@ import { routePattern } from "./lib/route-pattern.mjs";
import { resolveTestids } from "./lib/testids.mjs";
import { tmuxPathFor } from "./lib/tmux-path.mjs";
import { browserCallLimit } from "./lib/call-limit.mjs";
+import { agentProfileRepoint } from "./lib/agent-profile-repoint.mjs";
import {
canvasSize,
encodeRecording,
@@ -1444,6 +1445,72 @@ test("fixture skill --commit records the project skill once and reports a re-run
assert.match(personal.json.error, /needs --repo/);
});
+test("llm preset repoints default only off another live LLM profile", () => {
+ const seeded = {
+ id: "a1",
+ name: "default",
+ revision: 3,
+ schema_version: 1,
+ agent_kind: "openhands",
+ llm_profile_ref: "deepseek-chat",
+ enable_sub_agents: true,
+ mcp_server_refs: ["qa_mcp_a"],
+ };
+ const names = ["deepseek-chat", "deepseek-flash", "deepseek-pro"];
+ // Onboarding left default on its own profile: move it, keep every other field.
+ assert.deepEqual(agentProfileRepoint(seeded, names, "deepseek-flash"), {
+ from: "deepseek-chat",
+ body: {
+ schema_version: 1,
+ agent_kind: "openhands",
+ llm_profile_ref: "deepseek-flash",
+ enable_sub_agents: true,
+ mcp_server_refs: ["qa_mcp_a"],
+ },
+ });
+ // Already there, a fresh run's missing `default` ref, or no ref: unchanged.
+ assert.equal(agentProfileRepoint(seeded, names, "deepseek-chat"), null);
+ assert.equal(
+ agentProfileRepoint(
+ { ...seeded, llm_profile_ref: "default" },
+ names,
+ "deepseek-flash",
+ ),
+ null,
+ );
+ assert.equal(
+ agentProfileRepoint(
+ { ...seeded, llm_profile_ref: null },
+ names,
+ "deepseek-flash",
+ ),
+ null,
+ );
+ // An ACP default owns its own model; an older backend has no profile.
+ assert.equal(
+ agentProfileRepoint(
+ { ...seeded, agent_kind: "acp" },
+ names,
+ "deepseek-flash",
+ ),
+ null,
+ );
+ assert.equal(agentProfileRepoint(undefined, names, "deepseek-flash"), null);
+});
+
+test("llm help says when preset repoints the default agent profile", () => {
+ const help = spawnSync(process.execPath, [cli, "help", "llm"], {
+ encoding: "utf8",
+ });
+ assert.equal(help.status, 0);
+ assert.match(
+ help.stdout,
+ /'preset' also points the 'default' agent profile at deepseek-flash/,
+ );
+ assert.match(help.stdout, /and 'set' leave agent profiles as they are/);
+ assert.match(help.stdout, /'repointed'/);
+});
+
test("a recording's timeline cuts the lead-in and pauses, and holds the end", () => {
// Changes at 0, 3, 3.2 and 10 s; stopped at 10.5 s.
const frames = [{ t: 0 }, { t: 3000 }, { t: 3200 }, { t: 10000 }];
diff --git a/.agents/skills/verify-openhands/scripts/lib/agent-profile-repoint.mjs b/.agents/skills/verify-openhands/scripts/lib/agent-profile-repoint.mjs
new file mode 100644
index 0000000000..c4c4f6be9e
--- /dev/null
+++ b/.agents/skills/verify-openhands/scripts/lib/agent-profile-repoint.mjs
@@ -0,0 +1,36 @@
+// Whether `llm preset` must also move the seeded `default` agent profile onto
+// the LLM profile it activates.
+//
+// On a local backend, onboarding points `default` at the LLM profile it
+// creates (useApplyOnboardingAgentProfile). Activating another LLM profile
+// leaves that reference behind. The Agents page then still shows the
+// onboarding model on `default`, a launch that picks `default` explicitly
+// uses it, and deleting the onboarding profile answers 409 ("referenced by
+// 1 agent profile(s)").
+//
+// Only `default` moves, and only when it is an OpenHands profile whose
+// reference names another LLM profile that exists. Named agent profiles are
+// deliberate picks. `llm set` never moves it: recipes activate throwaway
+// profiles with it and delete them afterwards. A reference to a missing
+// profile is the fresh-run seed: the app already falls back to the active LLM
+// profile for it, and F13.stale-llm-ref drives that state.
+
+export const DEFAULT_AGENT_PROFILE = "default";
+
+/**
+ * @param {object | undefined} agentProfile `profile` from
+ * `GET /api/agent-profiles/default`
+ * @param {string[]} llmProfileNames names from `GET /api/profiles`
+ * @param {string} target the LLM profile just activated
+ * @returns {{ from: string, body: object } | null} the save body for
+ * `POST /api/agent-profiles/default`, or null when nothing should move
+ */
+export function agentProfileRepoint(agentProfile, llmProfileNames, target) {
+ if (agentProfile?.agent_kind !== "openhands") return null;
+ const from = agentProfile.llm_profile_ref;
+ if (!from || from === target || !llmProfileNames.includes(from)) return null;
+ // The save is a whole-profile overwrite: keep every stored field and drop
+ // only the identity the server owns (as the Agents editor does).
+ const { id, name, revision, ...stored } = agentProfile;
+ return { from, body: { ...stored, llm_profile_ref: target } };
+}