π OpenHands: AI-Driven Development
docs: define valid live review evidence (#17726)
Graham Neubig16b16fe1fd18
diff --git a/.agents/skills/custom-codereview-guide.md b/.agents/skills/custom-codereview-guide.mdindex 2c8ec8fe38..8d7760d77c 100644--- a/.agents/skills/custom-codereview-guide.md+++ b/.agents/skills/custom-codereview-guide.md@@ -201,33 +201,49 @@ when cleanup and dependencies are explicit.- Every PR that touches frontend code needs an authentic screenshot or videocaptured while exercising the PR code. This includes non-visual frontend logicwhose effect appears in a generated or downloaded user-facing artifact.+- Require evidence proportional to the behavior changed. UI changes need a+ screenshot or video from the real app; CLI, API, and script changes need the+ exact runtime command and observed result.- Authentic evidence comes from the real Agent Canvas app, browser, OS dialog,generated or downloaded artifact, or actual terminal/runtime output. It mustinclude enough surrounding context and reproduction steps to establish whatproduced it. Mockups, Figma or design images, diagrams, manually recreatedterminal output, synthetic before/after cards, and other illustrative graphicsdo not prove that the code ran.+- "Real app" evidence must exercise the production integration path with a real+ backend and, when the behavior depends on model execution, a real LLM. A+ mock-LLM or mocked backend run is E2E regression coverage, not live evidence.+ The artifact and description must identify the backend and model used so the+ reviewer can distinguish live evidence from a mock fixture.- Runtime and user-visible bug fixes require the same production-facing setupbefore and after the change. The base or released version must reproduce thebug; the PR head must show the corrected behavior. If the claimed state cannotbe produced through the real product, require the issue and PR to say so anddescribe the change as defensive handling rather than a reproduced productionbug.+- Match the artifact type to the behavior. A screenshot can prove a static render+ state, but not duration, ordering, disappearance, refresh, navigation, or any+ other temporal behavior. Those changes require a video that visibly shows the+ trigger, the relevant transition, and the final corrected state. A still image+ of a supposedly stuck or stale state does not establish how long it persisted.+- Visual evidence must leave the changed behavior and relevant controls readable.+ Dismiss privacy, consent, onboarding, cookie, tooltip, and other overlays before+ capture; an obscured target is not evidence even if the underlying app is real.- When unsure what genuine product evidence looks like, compare it with theseofficial OpenHands documentation captures: [first-time setup](https://github.com/OpenHands/docs/blob/2409e927af05a40acc27190383a424615fc0f807/openhands/static/img/agent-canvas-setup-step-1.png),[LLM profiles manager](https://github.com/OpenHands/docs/blob/2409e927af05a40acc27190383a424615fc0f807/openhands/static/img/agent-canvas-llm-profiles-manager.png),and [`/model` interaction](https://github.com/OpenHands/docs/blob/2409e927af05a40acc27190383a424615fc0f807/openhands/static/img/model-command-agent-canvas.png).-- CLI, API, and script changes need the exact runtime command and observed result.- Lifecycle fixes must also verify resulting process or resource state, such as+- Lifecycle fixes must also verify resulting process or resource state, such asthe parent exit code and remaining child services or listening ports.- Tests must exercise real logic and observable state. A claimed regression testmust reach the target behavior and fail when that behavior regresses; mocksthat only prove another mock was called are insufficient.- Do not duplicate library behavior or add brittle presentation-only snapshots.-Tests are regression proof, not a substitute for required live evidence. Submit-**COMMENT** when authentic production-facing evidence is missing or ambiguous,-and name the exact capture or verification still needed before approval.+Tests, mock-LLM runs, and mocked-backend runs are regression proof, not a+substitute for required live evidence. Submit **COMMENT** when production-facing+evidence is required but absent or ambiguous, and name the exact capture or+verification still needed before approval.Follow the test routing in `AGENTS.md`. Mock-LLM, Docker mock-LLM, and liveLLM-backed E2E suites run after changes reach `main`, not from PR labels. For