π OpenHands: AI-Driven Development
docs(verify): use existing OpenHands profiles for critic checks (#18611)
πΎ smolpaws852132d38dbc
diff --git a/.agents/skills/verify-openhands/references/feature-map/F06-agent-activity.md b/.agents/skills/verify-openhands/references/feature-map/F06-agent-activity.mdindex 7788577794..b7efe858e4 100644--- a/.agents/skills/verify-openhands/references/feature-map/F06-agent-activity.md+++ b/.agents/skills/verify-openhands/references/feature-map/F06-agent-activity.md@@ -67,7 +67,7 @@ Preconditions:- Baseline state (launched, doctored, `onboard --skip` done) and `control-openhands llm preset deepseek` (deepseek-flash active). Every prompt below is tiny and stays in the conversation's own workspace.- Conversation ids are printed as `"id"` by `control-openhands conversation start`, or read from `control-openhands browser url` (`/conversations/<id>`). `<id>` below is always the conversation the step just created or opened.- `F06.llm-not-configured-banner` needs a second, fresh `control-openhands launch --new --build never` with no `llm preset` (export its run dir as `OH_VERIFY_RUN` for those commands, then `control-openhands stop` it).-- `F06.critic-result` is blocked: it needs a model and an OpenHands Cloud critic API key (Settings β Verification β Critic API Key); its bullet says what a scored run shows. The automatic-fetch half of `F06.load-older-history` (its 16-command conversation) needs a model; the scroll-to-top half is driven without one, on the dummy profile below, with thirty refused user messages. `F06.hook-events` needs a configured hook; none is set in a fresh run.+- `F06.critic-result`: if an authorized OpenHands (`openhands/β¦`) or compatible OpenHands LLM-proxy profile is available, use it for the critic check, regardless of the profile used for the rest of the pass. The SDK reuses that profile's LLM key when Critic API Key is empty; a separate critic key is not required. For a non-default proxy deployment, use its compatible critic endpoint. Restore the previous profile after the check. Record `blocked` only when no authorized working critic configuration is available, not merely because the main profile uses another provider. The automatic-fetch half of `F06.load-older-history` (its 16-command conversation) needs a model; the scroll-to-top half is driven without one, on the dummy profile below, with thirty refused user messages. `F06.hook-events` needs a configured hook; none is set in a fresh run.- `F06.clock-skew-send` and `F06.failed-send-persists` need no model reply: a conversation that exists is enough (without a key the agent errors at once, and the user rows still land). Without a key, create the family's conversations with a dummy profile (`QA_DUMMY_KEY=qa control-openhands llm set --profile qa-zeta --model openai/gpt-4o --api-key-env QA_DUMMY_KEY --no-validate`) and `control-openhands conversation start --prompt ...` without `--wait`. `F06.corrective-nudge` is blocked: the nudge needs a model turn with neither text nor a tool call, which a normal prompt to deepseek-flash does not produce. When one happens, `control-openhands browser count 'testid=corrective-nudge-message'` is `1` and `browser count 'testid=user-message >> has-text=Your last response did not include'` is `0`.- The harness browser grants clipboard access: `control-openhands browser clipboard` reads what a Copy button wrote and `browser clipboard --write qa-empty` resets it first.- A terminal call can come back as "The terminal session was reset..." after this run's agent server restarts; the model reruns it, which turns one action into a group of two (see Gotchas).@@ -99,7 +99,7 @@ Preconditions:- **Error events (`F06.error-events`).** Any run that ends in a `ConversationErrorEvent` (today: any message sent in a branched conversation) leaves `conversation wait <id>` at `"status": "error"`; `control-openhands browser testids 'testid=chat-interface'` lists `error-message-banner` with `warning-message-banner-icon` and the error's text, and the composer status reads `Error`. Read the code with `control-openhands api GET "/api/conversations/<id>/events/search?limit=3&sort_order=TIMESTAMP_DESC"`.- **Skill installed banner (`F06.skill-install-banner`).** The banner keys on the installer's success line in terminal output. Put this one line in a prompt file: `Run exactly this one terminal command, verbatim, then reply DONE: mkdir -p .agents/skills/qa-skill && printf '# qa-skill\n' > .agents/skills/qa-skill/SKILL.md && echo "β Successfully installed 'qa-skill' to $PWD/.agents/skills/qa-skill"`, then run `control-openhands conversation start --prompt "$(cat <prompt file>)" --wait --timeout 150`. `control-openhands browser text 'testid=skill-install-restart-banner'` reads `Installed to this workspace: qa-skill. Skills load when a conversation starts, so this conversation can't use them yet.` Run `control-openhands browser click 'testid=skill-install-restart-action' --expect-url '/conversations/(?!<id>)'`; `control-openhands api GET /api/conversations/<new id>` has the same `workspace.working_dir` as the source. Back on the source (`control-openhands browser goto /conversations/<id>`), `control-openhands browser click 'testid=skill-install-restart-dismiss'` hides it (count `0`); after `browser reload` it is back (dismissal is session-only by design).- **No LLM (`F06.llm-not-configured-banner`).** On the fresh no-LLM run: `control-openhands onboard --skip`, `control-openhands browser click 'testid=conversation-panel-new-thread-picker'`, `control-openhands browser click 'testid=launch-no-workspace' --expect-url '/conversations/'`. `control-openhands browser text 'testid=home-llm-not-configured-banner'` reads `Your LLM isn't set up yet, so conversations won't run. Finish setup to get started. Set up LLM`; `count 'testid=chat-suggestions'` is `0`; the composer cannot take text (`control-openhands browser attr 'testid=chat-input' contenteditable` is `false`, so `browser fill` fails) and `control-openhands browser enabled 'testid=submit-button'` is `false`. `control-openhands browser click 'testid=home-llm-not-configured-action'` then `browser url` ends in `/settings/llm`.-- **Critic (`F06.critic-result`, blocked).** Toggle the hidden switch through its label on `/settings/verification`: `control-openhands browser goto /settings/verification`, `control-openhands browser click 'label:has([data-testid="sdk-settings-verification.critic_enabled"])'`, save, `api GET /api/settings` shows `"critic_enabled": true`. Without a Critic API key a `Reply with only the word OK.` conversation renders no critic block. With a key and a model, run `control-openhands conversation start --prompt "Create qa_critic.txt containing hi, then finish." --wait --timeout 180`: every scored event carries a block, so `control-openhands browser count 'text=Critic: agent success likelihood'` is at least `1` and a finishing action (`finish` tool row) shows its own block next to the agent's message; the stars carry the score as their label, which an ARIA snapshot does not list: `control-openhands browser count '[aria-label^="Score: "]'` is at least `2` and `control-openhands browser attr '[aria-label^="Score: "] >> nth=0' aria-label` reads `Score: <N.N>%` with one decimal (`Score: 82.0%`), the same number as the `(82.0%)` text beside the stars. `control-openhands browser click 'role=button[name="Expand details"] >> nth=0'` opens the categories `Potential Issues:`, `Infrastructure:` and `Likely Follow-up:` with their named issues in `control-openhands browser text 'testid=chat-scroll-container'`, and the button's label flips to `Collapse details`. Blocked here: DEEPSEEK_API_KEY and a Cloud critic key. Turn it off again the same way.+- **Critic (`F06.critic-result`).** Toggle the hidden switch through its label on `/settings/verification`: `control-openhands browser goto /settings/verification`, `control-openhands browser click 'label:has([data-testid="sdk-settings-verification.critic_enabled"])'`, save, `api GET /api/settings` shows `"critic_enabled": true`. Use the authorized OpenHands/proxy-backed profile described above for this check; leave Critic API Key empty to reuse its LLM key. With that profile and the critic enabled, run `control-openhands conversation start --prompt "Create qa_critic.txt containing hi, then finish." --wait --timeout 180`: every scored event carries a block, so `control-openhands browser count 'text=Critic: agent success likelihood'` is at least `1` and a finishing action (`finish` tool row) shows its own block next to the agent's message; the stars carry the score as their label, which an ARIA snapshot does not list: `control-openhands browser count '[aria-label^="Score: "]'` is at least `2` and `control-openhands browser attr '[aria-label^="Score: "] >> nth=0' aria-label` reads `Score: <N.N>%` with one decimal (`Score: 82.0%`), the same number as the `(82.0%)` text beside the stars. `control-openhands browser click 'role=button[name="Expand details"] >> nth=0'` opens the categories `Potential Issues:`, `Infrastructure:` and `Likely Follow-up:` with their named issues in `control-openhands browser text 'testid=chat-scroll-container'`, and the button's label flips to `Collapse details`. Turn it off again the same way and restore the previous profile. Profile availability is a precondition, not proof that the service accepted the credential: require real scored events before recording a pass.- **Phone (`F06.phone`).** Run `control-openhands browser viewport phone`, open the table conversation (`control-openhands browser goto /conversations/<stream id>`), `control-openhands browser bbox 'testid=chat-interface'` (`insideViewport` `true`, `pageHorizontalOverflow` `false`) and `control-openhands browser bbox 'testid=markdown-table-scroll'` (`scrollWidth` larger than `clientWidth`: the table scrolls inside itself). `control-openhands browser screenshot --feature F06.phone --name conversation`, then `control-openhands browser viewport desktop`.- After each group run `control-openhands browser errors --app-only`. Outages driven with `service stop` log connection-refused console errors by design; `pageErrors` must stay `0`.@@ -215,5 +215,5 @@ Preconditions:- Known issue [#18523](https://github.com/OpenHands/OpenHands/issues/18523) (`F06.tool-visualizers`): a timed-out terminal action's clock (`testid=status-icon`) measures 1.91:1 on the Light+ chat panel and 1.77:1 on Solarized Light, below 3:1. Reproduce with a follow-up `Call the terminal tool exactly once with command 'sleep 3' and its timeout parameter set to 1 (seconds). Do not run anything else. Then reply DONE.`- Known issue [#18522](https://github.com/OpenHands/OpenHands/issues/18522) (`F06.markdown-rendering`): chat Markdown links have no underline before hover, and a reply `See [QA docs](https://example.com/qa) for the verification page.` shows the link at 3.76:1 on the Light+ chat panel and 3.49:1 on Solarized Light (2.99:1 and 2.83:1 in the user bubble), below 4.5:1.- Known issue [#18521](https://github.com/OpenHands/OpenHands/issues/18521) (`F06.hook-events`): with real `user_prompt_submit` hooks in `<workspace>/.openhands/hooks.json` (stderr then exit 0; `sleep 2` with timeout 1; a JSON deny with exit 2), expanded hook rows in Light+ show the ok/failed/blocked badges at 1.84:1, 1.50:1 and 1.80:1, stderr at 1.30:1, the timeout error at 2.89:1 and the blocked reason at 1.72:1; Solarized Light gives 1.18:1 to 2.68:1.-- Known issue [#18524](https://github.com/OpenHands/OpenHands/issues/18524) (`F06.critic-result`): a real score-0 critic block measures header 1.48:1, stars 2.89:1 and percentage 4.74:1 on the Light+ chat panel (1.37:1, 2.68:1 and 4.40:1 on Solarized Light). Without a Cloud critic key, arrange it with `control-openhands api POST /api/conversations --write` using a direct `agent` with `critic:{kind:"AgentFinishedCritic"}`, `tools:[]`, `include_default_tools:["FinishTool"]` and a finish-immediately prompt; that critic scores 0 locally.+- Known issue [#18524](https://github.com/OpenHands/OpenHands/issues/18524) (`F06.critic-result`): a real score-0 critic block measures header 1.48:1, stars 2.89:1 and percentage 4.74:1 on the Light+ chat panel (1.37:1, 2.68:1 and 4.40:1 on Solarized Light). For a local rendering-only check without an authorized remote critic profile, arrange it with `control-openhands api POST /api/conversations --write` using a direct `agent` with `critic:{kind:"AgentFinishedCritic"}`, `tools:[]`, `include_default_tools:["FinishTool"]` and a finish-immediately prompt; that critic scores 0 locally.- Known issue [#18541](https://github.com/OpenHands/OpenHands/issues/18541) (`F06.markdown-rendering`): in a fenced `toml` block, section headers like `[build-system]` render on three lines (`[`, name, `]`) because the highlighter's `token table` span takes `display: table`; `browser eval "getComputedStyle(document.querySelector('[data-testid=agent-message] .token.table')).display"` returns `table` instead of `inline`. Reproduce with `Write a short markdown answer on Python packaging with pyproject.toml. Include one toml code block with [build-system] and [project] sections. Do not run any tools.` The Files tab preview of a `.toml` file (F08) shows the same break.diff --git a/.agents/skills/verify-openhands/references/feature-map/README.md b/.agents/skills/verify-openhands/references/feature-map/README.mdindex 37dd47083f..d3d79d419d 100644--- a/.agents/skills/verify-openhands/references/feature-map/README.md+++ b/.agents/skills/verify-openhands/references/feature-map/README.md@@ -22,6 +22,7 @@ Maintenance baseline: main@d3a58cd1cc045bbb1b24f2c8c0d79a8404ef239b (2026-10-16)rate limit), `llm preset deepseek --no-validate` still saves the profiles:recipes that only need a conversation to exist can run on, and everyrecipe that reads a model reply is `blocked`, never a pass.+- For critic checks, use an available, authorized OpenHands (`openhands/β¦`) or compatible OpenHands LLM-proxy profile, regardless of the main profile used for the pass. Its LLM key is reused when Critic API Key is empty. For a non-default proxy deployment, use its compatible critic endpoint. Restore the previous profile afterwards; see [F06](F06-agent-activity.md). Do not declare the critic blocked merely because the main model uses another provider. This does not authorize new credentials, additional budget, or sending private test data to another service.- The browser is at the desktop viewport (1440Γ1000) unless a recipe saysotherwise; `control-openhands browser viewport phone` is 390Γ844.- Never drive an instance that this verification run did not start.