Skip to content

🙌 OpenHands: AI-Driven Development

docs(verify-openhands): record every known bug in the feature map (#18304)

Engel Nystcd46cf9944b1
diff --git a/.agents/skills/verify-openhands/SKILL.md b/.agents/skills/verify-openhands/SKILL.md
index 4b24fec546..e9da8c2c14 100644
--- a/.agents/skills/verify-openhands/SKILL.md
+++ b/.agents/skills/verify-openhands/SKILL.md
@@ -211,7 +211,11 @@ prerequisite; never substitute a mock and call it a pass.
- **Product bug**: the app is broken. Keep the evidence, search existing issues,
report it separately (right repository: Canvas UI here, Agent Server/SDK in
`OpenHands/software-agent-sdk`, scheduling/dispatch in `OpenHands/automation`).
- Never rewrite an expected result to bless broken behavior.
+ Never rewrite an expected result to bless broken behavior. Then record it
+ in the map, which should know every open bug: a `Known failure (reproduced
+ <date>, #<issue>)` note on the bullet that reproduces it, next to the
+ expected result, or a `Known issue #<issue>` Gotchas entry when no recipe can
+ drive it. Passes re-drive these notes once the issue closes.
- **Blocked**: name the missing prerequisite (account, entitlement, OS, binary)
and the route you attempted. Unreachable is never a pass.
diff --git a/.agents/skills/verify-openhands/references/daily.md b/.agents/skills/verify-openhands/references/daily.md
index 88481f60aa..27d4962eea 100644
--- a/.agents/skills/verify-openhands/references/daily.md
+++ b/.agents/skills/verify-openhands/references/daily.md
@@ -73,7 +73,9 @@ cheaper than the next and catches a different kind of rot.
pass has a second half: every bullet whose Gotchas or Known failure note
links an issue that closed since `BASE` (check each linked issue's state)
is re-driven too, its row recorded and its note updated. A fix that landed
- in the Agent Server or the automation service shows up only this way.
+ in the Agent Server or the automation service shows up only this way. So is
+ every open `bug` issue opened since `BASE` that names a mapped feature: it
+ is reproduced and gets its note, so bugs filed outside a pass reach the map.
3. **Smoke (every family, cheaply).** For each family not already driven in
tier 2: open its first entry point, run the family's `errors --app-only`
sweep, and drive the one bullet that proves the page's main state (a list
@@ -111,7 +113,8 @@ ledger>` when that ledger is at hand. Fail and blocked rows come first; the
Triage as the skill does: map drift (fix, with live evidence), harness gap
(extend the CLI, re-drive), product defect (keep the evidence, search issues,
-file in the owning repository, keep it out of the map PR), blocked (name the
+file in the owning repository, and add its Known failure note to the map PR;
+the fix stays out), blocked (name the
prerequisite and the route). At most one PR of proven map and CLI corrections
per pass, from current `main` (or from the previous pass's branch while its
PR is open), carrying the moved baseline line and saying "delta" in its
@@ -135,4 +138,5 @@ names no run, revision or entry point does not count; a gap is not a pass.
behavior without touching this repository.
- Not CI: a green Playwright run or a passing `map check` is an input to a
verdict, never the verdict.
-- Not a bug-fix lane: product defects are filed, not fixed, inside the pass.
+- Not a bug-fix lane: product defects are filed and mapped, not fixed, inside
+ the pass.
diff --git a/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md b/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md
index e403033799..cdc60b02a1 100644
--- a/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md
+++ b/.agents/skills/verify-openhands/references/feature-map/F01-first-run-and-sign-in.md
@@ -69,7 +69,7 @@ Local run (`launch --new`):
- **Skip checklist box (`F01.onboarding-skip-checklist`).** Run `control-openhands browser goto /conversations` and `control-openhands browser count 'testid=sidebar-onboarding-checklist'` (`1`). Run `control-openhands browser goto '/?previewOnboardingStep=3'`, then `control-openhands browser click 'text=Skip Getting Started checklist'`. `control-openhands browser eval "document.querySelector('[data-testid=onboarding-skip-getting-started-checklist]').checked"` is `true`. Run `control-openhands browser goto /conversations` and `control-openhands browser reload`; the checklist count is `0`. On `/settings/app`, `control-openhands browser eval "document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked"` is `false`. Restore it: go back to `/?previewOnboardingStep=3`, where the box now reads `true`, click the same text again, and the checklist count on `/conversations` is `1`.
- **Phone (`F01.onboarding-phone`).** Run `control-openhands browser viewport phone`. Then for N in 1, 2 and 3 run `control-openhands browser goto '/?previewOnboardingStep=N'`, `control-openhands browser bbox 'testid=onboarding-modal'` and `control-openhands browser screenshot --feature F01.onboarding-phone --name step-N`. The modal is 351 px wide with `insideViewport` `true` and `pageHorizontalOverflow` `false`, and Back/Next or Back/Close are visible. Return with `control-openhands browser viewport desktop`.
- **Error page (`F01.route-error-boundary`).** Run `control-openhands browser errors --clear`, `control-openhands browser goto /this-route-does-not-exist` and `control-openhands browser snapshot 'testid=not-found-screen'`. It shows `heading "Page not found"`, the paragraph `This address does not match any page. Check the URL, or go back to the home page.` and `link "Home"` (`/url: /`). `control-openhands browser count 'aside[data-collapsed]'` is `1`: the sidebar stays. `control-openhands browser screenshot --feature F01.route-error-boundary --name not-found` shows the message and the Home button centered beside the sidebar. `control-openhands browser errors --app-only` reports `pageErrors` `0` and `appErrors` `0`. Run `control-openhands browser click 'testid=not-found-home-link' --expect-url '/$'`; `control-openhands browser count 'testid=home-screen'` is `1`.
-- **Onboarding again with the saved endpoint (`F01.onboarding-repeat-endpoint`).** Two more first runs on this stack, both through the All view, which shows the Base URL field. Pass 1 saves a custom endpoint: `control-openhands browser reset`, `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-agent-next'`, `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="1"]'`, `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-all-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` (empty). Run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' openai/deepseek-chat`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=base-url-input' https://api.deepseek.com/v1`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="2"]'`. `control-openhands llm show` lists `deepseek-chat` (`openai/deepseek-chat`, `base_url` `https://api.deepseek.com/v1`) as `active_profile`. Say hello as above (`browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'`, `browser press Enter --selector 'testid=onboarding-hello-input'`, `browser wait-url '/conversations/[0-9a-f-]+'`), then `control-openhands conversation wait <id> --timeout 180` is `finished` and the agent `MessageEvent` is `hello`. Pass 2 types the same endpoint again: repeat the steps up to the All view. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` is now `https://api.deepseek.com/v1`, the saved value. Fill the model `openai/deepseek-v4-flash` (a DeepSeek alias, so the new profile gets its own name), the same Base URL and the key, then Next and wait for step 2 (`control-openhands browser screenshot 'testid=onboarding-step-setup-llm' --feature F01.onboarding-repeat-endpoint --name second-pass-form` before Next). Expected: `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`, and the say-hello conversation finishes with `hello`. Raw LLM settings that disagree with the active profile do not reach the form. To check it, arrange them before pass 2 with `control-openhands api PATCH /api/settings --write --data '{"agent_settings_diff":{"llm":{"base_url":"http://127.0.0.1:9/stale-raw-settings"}}}'`. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` still reads `https://api.deepseek.com/v1`. Fill only the model, run `browser network --clear` before Next and `browser network --filter '/api/(settings|profiles)'` after it: the rows are the profile `POST` and its `/activate`, with no `PATCH /api/settings`. `deepseek-v4-flash` keeps the DeepSeek Base URL, and the say-hello conversation finishes with `hello` (verified 2026-10-09). Restore: run `control-openhands llm preset deepseek`. It also points the `default` agent profile back at `deepseek-flash`: its output has `repointed` from `deepseek-v4-flash` to `deepseek-flash`. Then `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps.
+- **Onboarding again with the saved endpoint (`F01.onboarding-repeat-endpoint`).** Two more first runs on this stack, both through the All view, which shows the Base URL field. Pass 1 saves a custom endpoint: `control-openhands browser reset`, `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-agent-next'`, `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="1"]'`, `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-all-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` (empty). Run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' openai/deepseek-chat`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=base-url-input' https://api.deepseek.com/v1`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step="2"]'`. `control-openhands llm show` lists `deepseek-chat` (`openai/deepseek-chat`, `base_url` `https://api.deepseek.com/v1`) as `active_profile`. Say hello as above (`browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'`, `browser press Enter --selector 'testid=onboarding-hello-input'`, `browser wait-url '/conversations/[0-9a-f-]+'`), then `control-openhands conversation wait <id> --timeout 180` is `finished` and the agent `MessageEvent` is `hello`. Pass 2 types the same endpoint again: repeat the steps up to the All view. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` is now `https://api.deepseek.com/v1`, the saved value. Fill the model `openai/deepseek-v4-flash` (a DeepSeek alias, so the new profile gets its own name), the same Base URL and the key, then Next and wait for step 2 (`control-openhands browser screenshot 'testid=onboarding-step-setup-llm' --feature F01.onboarding-repeat-endpoint --name second-pass-form` before Next). Expected: `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`, and the say-hello conversation finishes with `hello`. Raw LLM settings that disagree with the active profile do not reach the form. To check it, arrange them before pass 2 with `control-openhands api PATCH /api/settings --write --data '{"agent_settings_diff":{"llm":{"base_url":"http://127.0.0.1:9/stale-raw-settings"}}}'`. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` still reads `https://api.deepseek.com/v1`. Fill only the model, run `browser network --clear` before Next and `browser network --filter '/api/(settings|profiles)'` after it: the rows are the profile `POST` and its `/activate`, with no `PATCH /api/settings`. `deepseek-v4-flash` keeps the DeepSeek Base URL, and the say-hello conversation finishes with `hello` (verified 2026-10-09). Known failure (reproduced 2026-10-12, #18302): Next replaces a profile that already has the model-derived name. Arrange one before pass 2 with `control-openhands llm set --profile deepseek-v4-flash --model openai/deepseek-v4-flash --base-url http://127.0.0.1:9/other-endpoint --api-key-env DEEPSEEK_API_KEY --no-activate --no-validate`. After pass 2's Next, `browser network --filter '/api/profiles'` shows `POST /api/profiles/deepseek-v4-flash`, and `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`: its own Base URL is gone, and nothing warned. Expected: the existing profile is kept, under a name that does not collide or after a confirmation. Restore: run `control-openhands llm preset deepseek`. It also points the `default` agent profile back at `deepseek-flash`: its output has `repointed` from `deepseek-v4-flash` to `deepseek-flash`. Then `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps.
- **Clean up.** Delete the hello conversation from its header menu: `control-openhands browser goto /conversations/<id>` (the hello conversation's `<id>` from Say hello; the Error page bullet left `/`), `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'`, `control-openhands browser click 'role=button[name="Confirm Delete"]'`. `control-openhands conversation list` then shows `count` `0`.
Public run A (`launch --new --public`), backend step walked:
@@ -103,7 +103,7 @@ Not reachable locally:
- The consent modal (z-70) sits over the onboarding modal, and clicks at the screen center land on agent tiles once it closes. A forced click on `first-run-onboarding-screen` selected Codex. Check backdrop non-dismissal with `browser mouse-click 40 40`, a corner outside the modal.
- The say-hello default message ("Create a basic webpage…") makes the agent work for a while. Replace it with a tiny prompt to keep model cost down.
- On a local backend, Next on the LLM step also points the `default` agent profile at the profile it just created. Activating another LLM profile does not move that pointer, and while it stays, deleting the onboarding profile answers `409` `LLM profile is referenced by 1 agent profile(s): default`. `control-openhands llm preset deepseek` moves it to `deepseek-flash` and reports `repointed`; `llm set` does not. In the UI, edit `default` in Settings → Agents and pick the profile in `testid=agent-profile-llm-selector`.
-- The LLM step saves only the fields that differ from the saved settings, and the profile it creates is built from that same diff. That is the cause of the `F01.onboarding-repeat-endpoint` failure (#17884, fix in #17889). It is also why the mock-LLM onboarding e2e failed after earlier specs had saved the mock endpoint. The first pass of that bullet passes on a run whose saved Base URL is still empty.
+- On a local backend the LLM step edits an LLM profile and never reads or writes the raw LLM settings (#18218, after #17889 fixed #17884). It takes the active profile, else the one the `default` agent profile points at (the Agent Server's backfill of an older install's raw settings), else a fresh one with the default model. It reads the profile only once the backend's health check reports it connected, so a step mounted while the Agent Server was down fills in after it comes back (verified 2026-10-12 with `control-openhands service stop agent-server`, then `restart`). If the profiles cannot be read it shows `Failed to load existing profile. Please try again.` (`testid=onboarding-llm-load-error`) with Retry (`testid=onboarding-llm-load-retry`) and Next disabled. Next without changes only sends `POST /api/profiles/<name>/activate`.
- The tab title prefixes a status emoji to the stored title. A generated title that already starts with an emoji therefore shows twice (`✅ ✅ Reply hello without tools | OpenHands`). Right after launch the title is `Conversation <id prefix>` until a reload picks up the generated one.
- Right after Next, the ACP step reads "Checking for an existing Claude Code login…" with Next disabled. Wait for `testid=onboarding-acp-auth-detected` before reading the banner. The "already signed in" banner depends on the host's CLI login. On a clean CI machine the fields become required, and Next is blocked until they are filled.
- Choosing an ACP agent saves `agent_kind: acp` immediately. Re-choose OpenHands before leaving, or later conversations in the run use the ACP agent.
diff --git a/.agents/skills/verify-openhands/references/feature-map/F10-llm-profiles.md b/.agents/skills/verify-openhands/references/feature-map/F10-llm-profiles.md
index b68134db07..1779dd5138 100644
--- a/.agents/skills/verify-openhands/references/feature-map/F10-llm-profiles.md
+++ b/.agents/skills/verify-openhands/references/feature-map/F10-llm-profiles.md
@@ -92,7 +92,7 @@ Preconditions:
- **Link to a provider connection (`F10.connection-link`, `F10.list-grouped`).** Arrange a connection through the F11 form: `control-openhands browser click 'testid=add-provider-connection'`, `control-openhands browser fill 'testid=provider-connection-name-input' QA_ds_conn`, `control-openhands browser click 'testid=provider-connection-provider-input'`, `control-openhands browser fill 'testid=provider-connection-provider-input' deepseek`, `control-openhands browser click 'testid=provider-item-deepseek'`, `control-openhands browser fill 'testid=provider-connection-api-key-input' --value-file <key file>`, `control-openhands browser click 'testid=provider-connection-submit'` and `control-openhands browser wait-text 'Connection "QA_ds_conn" created'`. Edit `QA_adv`, click `testid=sdk-section-basic-toggle`, run `control-openhands browser click 'testid=llm-provider-connection-input'` (options `None`, `QA_ds_conn`) and `control-openhands browser click 'role=option[name="QA_ds_conn"]'`; `browser count 'testid=llm-api-key-input'` is `0`, and on the Advanced tab `browser count 'testid=base-url-input'` is `0`. Picking `role=option[name="None"]` brings the key field back (`1`). Re-select `QA_ds_conn`, then `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 3000`: states `Save`, `Saving...` (no `Validating...`), and `wait-text 'Profile "QA_adv" updated'`. After a reload `control-openhands browser eval "[...document.querySelectorAll('[data-testid=profile-group-header]')].map(e=>e.innerText).join(' | ')"` is `QA_DS_CONN | NOT LINKED` (CSS upper-case) and `control-openhands browser count 'div:has(> [data-testid=profile-group-header]:text-is("QA_ds_conn")) [data-testid=profile-row]:has([title="QA_adv"])'` is `1` (screenshot `--feature F10.list-grouped --name grouped`).
- **Add models from a provider (`F10.add-models`).** Run `control-openhands browser click 'testid=add-models-from-provider'`; `browser enabled 'testid=add-models-submit'` is `false`. Keyless: `control-openhands browser select 'testid=add-models-provider' moonshot` and `control-openhands browser wait 'testid=add-models-select-all'`; `browser text 'testid=add-models-keyless-note'` is `No connection — profiles will need a key added later`, rows start unchecked (`Add 0 profiles`), and `control-openhands browser check 'testid=add-models-select-all'` makes `browser text 'testid=add-models-submit'` read `Add 12 profiles`; uncheck it again. Linked: `control-openhands browser select 'testid=add-models-provider' deepseek`; the connection field (`testid=add-models-connection-field`, options `No connection (keyless)` and `QA_ds_conn`) is pre-bound to `QA_ds_conn`. Run `control-openhands browser check 'testid=add-models-check-deepseek/deepseek-chat'`, `control-openhands browser check 'testid=add-models-check-deepseek/deepseek-v3'`, `control-openhands browser click 'testid=add-models-submit'` and `control-openhands browser wait-text 'Added 2'` (toast `Added 2 profiles`). The modal closes; after a reload `deepseek-chat` and `deepseek-v3` sit under the `QA_DS_CONN` header and `api GET /api/profiles` shows both with a `provider_connection_id`. Preselect entry: `control-openhands browser click 'testid=provider-connection-row >> has-text=QA_ds_conn >> testid=provider-connection-menu-trigger'` (menu: Bulk add, Edit, Delete), `control-openhands browser click 'testid=provider-connection-actions-menu >> testid=provider-connection-add-models'`; provider `deepseek` and the connection are preset, every remaining row is checked (`Add 10 profiles`), and `browser count 'testid=add-models-row-deepseek/deepseek-chat'` is `0` (already added). Close with `control-openhands browser click 'role=dialog[name="Add models as profiles"] >> role=button[name="Cancel"]'`.
- **Broken link (`F10.broken-link`).** The local server refuses to delete a referenced connection (row menu → `testid=provider-connection-delete` → `testid=delete-provider-connection-confirm` toasts `Provider connection cannot be deleted while it is referenced by LLM profile(s): ...`), so arrange the state directly: `control-openhands api POST /api/profiles/QA_broken --write --data '{"llm":{"model":"deepseek/deepseek-chat","provider_connection_id":"00000000000000000000000000000000"},"include_secrets":false}'`. Then `control-openhands browser reload` and `control-openhands browser text '[data-testid=profile-row]:has([title="QA_broken"])'` reads `QA_broken deepseek/deepseek-chat Broken link`; `control-openhands browser attr '[data-testid=profile-row]:has([title="QA_broken"]) >> testid=profile-broken-connection-badge' title` is `The linked provider connection was deleted. Edit this profile to re-link it.` (screenshot `--feature F10.broken-link --name badge`). Edit it: `browser value 'testid=llm-provider-connection-input'` is the orphaned id and the listbox offers `None`, `QA_ds_conn` and that id. The row's group header shows the raw id, not a name (see Gotchas).
-- **ChatGPT subscription, pre-auth (`F10.subscription`).** In a fresh editor run `control-openhands browser click 'testid=sdk-section-basic-toggle'` first (after the schema step the Default profile has a temperature, so a new editor opens on All; see `F10.create-prefill`), then `control-openhands browser click 'testid=llm-auth-type-input'` (options `API key`, `ChatGPT subscription`) and `control-openhands browser click 'role=option[name="ChatGPT subscription"]'`. `browser text 'testid=subscription-status'` is `ChatGPT subscription not connected`, `browser value 'testid=llm-subscription-model-input'` is `gpt-5.5` (list: gpt-5.5, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-6-astra), and `llm-provider-input` and `llm-api-key-input` are gone (count `0`). `control-openhands browser click 'testid=subscription-connect'` and `control-openhands browser wait 'testid=subscription-user-code'`: the card shows the code, `Open sign-in page`, `I've finished signing in` (`subscription-poll`) and Cancel, and `control-openhands browser tabs` lists a second page on `auth.openai.com` (it opens a second or two after the code appears; rerun `browser tabs` if it lists one page). The card polls on its own, so `Sign-in is not complete yet. ...` can already show before you click poll. Close it with `control-openhands browser close-tab 1` and `control-openhands browser tab 0`. `control-openhands browser click 'testid=subscription-poll'` leaves `Sign-in is not complete yet. Finish the browser flow and try again.` on the card, and `control-openhands browser click 'testid=subscription-cancel'` removes the code (`count 'testid=subscription-user-code'` `0`). Switching back with `role=option[name="API key"]` restores provider `OpenAI` and model `gpt-5.6-sol`. Completing sign-in is blocked (needs a ChatGPT Plus/Pro account).
+- **ChatGPT subscription, pre-auth (`F10.subscription`).** In a fresh editor run `control-openhands browser click 'testid=sdk-section-basic-toggle'` first (after the schema step the Default profile has a temperature, so a new editor opens on All; see `F10.create-prefill`), then `control-openhands browser click 'testid=llm-auth-type-input'` (options `API key`, `ChatGPT subscription`) and `control-openhands browser click 'role=option[name="ChatGPT subscription"]'`. `browser text 'testid=subscription-status'` is `ChatGPT subscription not connected`, `browser value 'testid=llm-subscription-model-input'` is `gpt-5.5` (list: gpt-5.5, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-6-astra), and `llm-provider-input` and `llm-api-key-input` are gone (count `0`). Known failure (reproduced 2026-10-12, #17135): run `control-openhands browser fill 'testid=profile-name-input' QA_sub`, `control-openhands browser network --clear`, `control-openhands browser click 'testid=save-profile-btn'` and `control-openhands browser wait-text 'Model is required'`. The toast is `Model is required`, and `control-openhands browser network --filter '/api/profiles'` lists no request, although the dropdown shows `gpt-5.5`: it shows the first subscription model without writing it to the form. Expected: Save validates and saves `gpt-5.5`. Picking a model in `testid=llm-subscription-model-input`, even `gpt-5.5` again, makes the next Save send `POST /api/profiles/QA_sub/validate`, which answers `OpenAI subscription login is required` here (`browser toasts --history`), because no subscription is connected. `control-openhands browser click 'testid=subscription-connect'` and `control-openhands browser wait 'testid=subscription-user-code'`: the card shows the code, `Open sign-in page`, `I've finished signing in` (`subscription-poll`) and Cancel, and `control-openhands browser tabs` lists a second page on `auth.openai.com` (it opens a second or two after the code appears; rerun `browser tabs` if it lists one page). The card polls on its own, so `Sign-in is not complete yet. ...` can already show before you click poll. Close it with `control-openhands browser close-tab 1` and `control-openhands browser tab 0`. `control-openhands browser click 'testid=subscription-poll'` leaves `Sign-in is not complete yet. Finish the browser flow and try again.` on the card, and `control-openhands browser click 'testid=subscription-cancel'` removes the code (`count 'testid=subscription-user-code'` `0`). Switching back with `role=option[name="API key"]` restores provider `OpenAI` and model `gpt-5.6-sol`. Completing sign-in is blocked (needs a ChatGPT Plus/Pro account).
- **Delete (`F10.delete`, `F10.delete-default`).** Run `control-openhands browser click '[data-testid=profile-row]:has([title="QA_pro-copy"]) >> testid=profile-menu-trigger'` and `control-openhands browser click 'testid=profile-actions-menu >> testid=profile-delete'`. `control-openhands browser snapshot 'role=dialog'` shows `Delete Profile` with `Are you sure you want to delete the profile "QA_pro-copy"? This action cannot be undone.`, and `browser eval "document.activeElement.innerText"` is `Cancel`. `control-openhands browser click 'role=dialog[name="Delete Profile"] >> role=button[name="Cancel"]'` keeps the row (count `1`). Repeat, then `control-openhands browser click 'testid=delete-profile-confirm'`, `control-openhands browser wait-text 'Profile "QA_pro-copy" deleted'`, reload, and the count is `0`. Deleting the Default row the same way leaves exactly one `profile-active-badge` on another profile (`browser text '[data-testid=profile-row]:has([data-testid=profile-active-badge])'`), and `api GET /api/profiles` names it in `active_profile`.
- **Command menu and direct URL (`F10.list`).** Run `control-openhands browser goto /` (the bullets above end on `/settings/llm`, where the URL wait would pass at once), `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' LLM` and `control-openhands browser snapshot 'testid=command-menu'`: two options, `LLM profiles` ("Manage models, providers, and API keys.") and `Model Router` (F12; its keywords include "llm"). Hovering an option selects it, so which one Enter runs depends on where the mouse rests: click the option instead, `control-openhands browser click 'testid=command-menu >> role=option[name="LLM profiles"]' --expect-url '/settings/llm(\?|$)'`. `browser count 'testid=add-llm-profile'` is `1` on arrival. `control-openhands browser goto /settings/llm` shows the same list (`browser count 'testid=profile-row'`, one `profile-active-badge`; screenshot `--feature F10.list --name list`).
- **Phone layout (`F10.phone`).** Run `control-openhands browser viewport phone`, `control-openhands browser goto /settings/llm`, `control-openhands browser bbox 'testid=add-llm-profile'`, `control-openhands browser bbox 'testid=profile-row >> nth=0'` and `control-openhands browser screenshot --feature F10.phone --name list`: `insideViewport` `true`, `pageHorizontalOverflow` `false`. Open a row menu: `browser bbox 'testid=profile-actions-menu'` is inside the viewport. Open Edit: `browser bbox 'testid=profile-name-input'` is 348 px wide inside the 390 px viewport and `pageHorizontalOverflow` is `false` (`llm-settings-screen` may report `insideViewport` `false` when the edited profile's form is taller than the screen; only the overflow flag matters). Return with `control-openhands browser click 'testid=cancel-profile-btn'` and `control-openhands browser viewport desktop`.
@@ -120,3 +120,4 @@ Preconditions:
- The list follows the server's order, which sorts by profile file name: `QA_pro-copy-1` lists before `QA_pro-copy` (`-` sorts before `.json`'s `.`). Do not assert a strict alphabetical order on names that are prefixes of each other.
- Toasts last only a few seconds. A `browser screenshot` taken after a slow command can miss them, so assert with `browser wait-text` or `browser toasts`.
- Known issue #17955: a new profile is saved without the inherited LLM options the editor shows, such as temperature (`F10.create-prefill`).
+- Known issue #18303: `Subscription models are not loaded yet.` is English in every language. It shows when a subscription profile is saved before `GET /api/llm/subscription/openai/models` answers, in the settings save (`src/routes/llm-settings.tsx`) and in onboarding's LLM step. No recipe drives it, since it needs that request to be slow or to fail.
diff --git a/.agents/skills/verify-openhands/references/maintenance.md b/.agents/skills/verify-openhands/references/maintenance.md
index a45af2a04d..ff5397e698 100644
--- a/.agents/skills/verify-openhands/references/maintenance.md
+++ b/.agents/skills/verify-openhands/references/maintenance.md
@@ -127,7 +127,14 @@ automatically a bug and a zero difference does not prove behavior.
- **Product defect**: keep the evidence, search existing issues, then file in the
owning repository (Canvas here; Agent Server/SDK in
`OpenHands/software-agent-sdk`; scheduling/dispatch in `OpenHands/automation`),
- following that repository's issue template. Keep it out of the maintenance PR.
+ following that repository's issue template. The fix stays out of the
+ maintenance PR, but the defect goes in: the same PR adds a `Known failure
+ (reproduced <date>, #<issue>)` note to the bullet that reproduces it, next to
+ the expected result, or a `Known issue #<issue>` Gotchas entry when no recipe
+ can drive it.
+- **Bugs filed outside the pass**: open `bug` issues opened since BASE that
+ name a mapped feature are repro candidates too. Reproduce each and add its
+ note; when one does not reproduce, comment on the issue with the run instead.
- **Undocumented or contradictory change**: cite the PR and the observed
difference; ask the maintainer instead of rewriting the expectation.
- **Known issues** linked in Gotchas are repro candidates: retest, link fresh
diff --git a/.agents/skills/verify-openhands/references/mapping.md b/.agents/skills/verify-openhands/references/mapping.md
index 3cffaa7681..421cb40c2e 100644
--- a/.agents/skills/verify-openhands/references/mapping.md
+++ b/.agents/skills/verify-openhands/references/mapping.md
@@ -119,7 +119,10 @@ sections, in this order:
4. `## Gotchas`: traps that waste or invalidate a run: debounces, focus traps,
items that only appear after the agent runs, features hidden behind a
prerequisite, and known open issues (link them; they are repro candidates, not
- exemptions).
+ exemptions). The map should know every open bug. Record one on the bullet
+ that reproduces it, next to the expected result, as `Known failure
+ (reproduced <date>, #<issue>): <what happens>`; list it here as
+ `Known issue #<issue>: ...` only when no recipe can drive it.
Treat commands as literal: quote selectors, keep names unique to the run (prefix
fixtures with `QA_`), and restore shared state after mutations (delete the secret,
diff --git a/.pr/17884/01-onboarding-before.png b/.pr/17884/01-onboarding-before.png
deleted file mode 100644
index 80a4404c03..0000000000
Binary files a/.pr/17884/01-onboarding-before.png and /dev/null differ
diff --git a/.pr/17884/02-onboarding-after.png b/.pr/17884/02-onboarding-after.png
deleted file mode 100644
index 42b004bbc3..0000000000
Binary files a/.pr/17884/02-onboarding-after.png and /dev/null differ