Skip to content

πŸ™Œ OpenHands: AI-Driven Development

docs: define valid live review evidence (#17726)

Graham Neubig16b16fe1fd18
diff --git a/.agents/skills/custom-codereview-guide.md b/.agents/skills/custom-codereview-guide.md
index 2c8ec8fe38..8d7760d77c 100644
--- a/.agents/skills/custom-codereview-guide.md
+++ b/.agents/skills/custom-codereview-guide.md
@@ -201,33 +201,49 @@ when cleanup and dependencies are explicit.
- Every PR that touches frontend code needs an authentic screenshot or video
captured while exercising the PR code. This includes non-visual frontend logic
whose effect appears in a generated or downloaded user-facing artifact.
+- Require evidence proportional to the behavior changed. UI changes need a
+ screenshot or video from the real app; CLI, API, and script changes need the
+ exact runtime command and observed result.
- Authentic evidence comes from the real Agent Canvas app, browser, OS dialog,
generated or downloaded artifact, or actual terminal/runtime output. It must
include enough surrounding context and reproduction steps to establish what
produced it. Mockups, Figma or design images, diagrams, manually recreated
terminal output, synthetic before/after cards, and other illustrative graphics
do not prove that the code ran.
+- "Real app" evidence must exercise the production integration path with a real
+ backend and, when the behavior depends on model execution, a real LLM. A
+ mock-LLM or mocked backend run is E2E regression coverage, not live evidence.
+ The artifact and description must identify the backend and model used so the
+ reviewer can distinguish live evidence from a mock fixture.
- Runtime and user-visible bug fixes require the same production-facing setup
before and after the change. The base or released version must reproduce the
bug; the PR head must show the corrected behavior. If the claimed state cannot
be produced through the real product, require the issue and PR to say so and
describe the change as defensive handling rather than a reproduced production
bug.
+- Match the artifact type to the behavior. A screenshot can prove a static render
+ state, but not duration, ordering, disappearance, refresh, navigation, or any
+ other temporal behavior. Those changes require a video that visibly shows the
+ trigger, the relevant transition, and the final corrected state. A still image
+ of a supposedly stuck or stale state does not establish how long it persisted.
+- Visual evidence must leave the changed behavior and relevant controls readable.
+ Dismiss privacy, consent, onboarding, cookie, tooltip, and other overlays before
+ capture; an obscured target is not evidence even if the underlying app is real.
- When unsure what genuine product evidence looks like, compare it with these
official OpenHands documentation captures: [first-time setup](https://github.com/OpenHands/docs/blob/2409e927af05a40acc27190383a424615fc0f807/openhands/static/img/agent-canvas-setup-step-1.png),
[LLM profiles manager](https://github.com/OpenHands/docs/blob/2409e927af05a40acc27190383a424615fc0f807/openhands/static/img/agent-canvas-llm-profiles-manager.png),
and [`/model` interaction](https://github.com/OpenHands/docs/blob/2409e927af05a40acc27190383a424615fc0f807/openhands/static/img/model-command-agent-canvas.png).
-- CLI, API, and script changes need the exact runtime command and observed result.
- Lifecycle fixes must also verify resulting process or resource state, such as
+- Lifecycle fixes must also verify resulting process or resource state, such as
the parent exit code and remaining child services or listening ports.
- Tests must exercise real logic and observable state. A claimed regression test
must reach the target behavior and fail when that behavior regresses; mocks
that only prove another mock was called are insufficient.
- Do not duplicate library behavior or add brittle presentation-only snapshots.
-Tests are regression proof, not a substitute for required live evidence. Submit
-**COMMENT** when authentic production-facing evidence is missing or ambiguous,
-and name the exact capture or verification still needed before approval.
+Tests, mock-LLM runs, and mocked-backend runs are regression proof, not a
+substitute for required live evidence. Submit **COMMENT** when production-facing
+evidence is required but absent or ambiguous, and name the exact capture or
+verification still needed before approval.
Follow the test routing in `AGENTS.md`. Mock-LLM, Docker mock-LLM, and live
LLM-backed E2E suites run after changes reach `main`, not from PR labels. For