🙌 OpenHands: AI-Driven Development
docs: clarify Canvas triage and review evidence boundaries (#17990)
Graham Neubig6801911d8d43
diff --git a/.agents/skills/custom-codereview-guide.md b/.agents/skills/custom-codereview-guide.mdindex 3e13cbece5..c6be40f372 100644--- a/.agents/skills/custom-codereview-guide.md+++ b/.agents/skills/custom-codereview-guide.md@@ -1,6 +1,6 @@---name: custom-codereview-guide-description: Repository-specific review rules for the OpenHands Agent Canvas frontend.+description: Repository-specific triage and review rules for the OpenHands Agent Canvas frontend.triggers:- /codereview---@@ -31,6 +31,44 @@ integration and depends on public interfaces from the owning repository. When achange belongs in one of the repositories above, say so and ask a maintainer toconfirm before reviewing the rest.+## Triage: Ownership and Scope++During triage, check ownership, supported behavior, and acceptance criteria.+Issue readiness means the work is clear enough to start—not that the fix is+complete. Do not require a PR, passing implementation tests, or before-and-after+fix evidence. Reconsider readiness when new information leaves scope or expected+behavior unresolved.++Trace the failing boundary before assigning an owner:++- Check the SDK for events, tools, ACP, profiles, conversation lifecycle, and persistence.+- Check automation for form contracts, run state, and pagination.+- Check producers and consumers at installed or supported versions using locks,+ runtime versions, and `config/defaults.json`, not upstream main alone.+ Cite versions and sources when ownership or compatibility depends on them.++Prefer fixes in the owning provider. Separate provider work from Canvas integration+and state release/dependency constraints. Distinguish new regressions from independent,+preexisting upstream issues. Recommend material follow-ups without expanding scope,+editing other repositories, or creating tickets without authorization.+Reuse answers in the issue and linked discussions.++## Triage: Behavioral Acceptance and Readiness++Write acceptance criteria as observable outcomes, affected modes, and relevant+failure/recovery cases, not implementation requirements or PR artifacts. Keep verification separate.+Bugs need expected/actual behavior, environment/version, and reproduction steps;+features need the desired outcome and scope, not evidence of an unbuilt feature.++For Canvas functional bugs, capture the failure in the running app; logs, API+output, and reproducers can supplement the capture. Accept evidence linked in+comments without duplicate uploads. State uncertainty when the failure has not+been reproduced. Triage does not require an after-fix demonstration.++## Implementation and Merge Review++The following checks apply to PRs, not triage.+## Review Sequence and DecisionReview the current PR head in this order:@@ -58,6 +96,12 @@ lightweight evaluation. Use COMMENT when an acceptance criterion or requiredcheck calls for specific eval evidence and that evidence is missing or failing,or when available results show a regression.+Separate code quality from merge readiness. Use **COMMENT** for a failing required+check, an unmergeable branch, or missing required evidence, even if the code is sound. Name the unmet gate.+Keep the summary, final verdict/footer, and submitted review state consistent;+never say `APPROVED` with an unresolved material finding or merge gate.+Optional evaluation follow-up remains nonblocking.+Include a compact checklist for every linked acceptance criterion. Meeting thechecklist is necessary but does not replace review for regressions, security, ormaintainability.@@ -109,6 +153,10 @@ versioned.Require an increase to `minimumAgentServer` to the first compatible released version.+Before requiring or removing a fallback, show the SDK producer's payloads and behavior+at the supported release floor and affected deployed versions.+Old fixtures or hypothetical legacy fields alone do not justify compatibility code.+Verify the compatibility boundary. Adding a TypeScript-client method does notmake older Agent Servers support it. Submit **COMMENT** if a supported backendcan reach the new code and fail because the required server behavior is absent.@@ -172,6 +220,12 @@ Telemetry has stricter named owners:`setTelemetryConsent` is the only consent controller.- A business milestone has one canonical capture.+### Docker user and permission workarounds++For Docker `--user` or `HOME` workarounds, test the full path under that UID/GID:+home, config/cache directories, temporary files, and subprocess/browser startup.+Check ownership and permissions outside the persistence mount too.+## Design ReviewPrefer named hooks, services, stores, and feature modules over shared-root@@ -243,12 +297,16 @@ correctness, security, compatibility, or architecture defect.## Testing and Production Evidence-- Every PR that touches frontend code needs an authentic screenshot or video- captured while exercising the PR code. This includes non-visual frontend logic- whose effect appears in a generated or downloaded user-facing artifact.-- Require evidence proportional to the behavior changed. UI changes need a- screenshot or video from the real app; CLI, API, and script changes need the- exact runtime command and observed result.+- Nonvisual evidence alone is acceptable only for non-functional changes, such+ as documentation or test-only edits that do not alter product behavior. Use+ relevant commands and output; docs-only changes need no runtime tests or media.+- Check rendering in Canvas. Token, class-name, and source-string assertions do+ not prove computed styling, isolation, layout, or readability, or replace captures.+- Every functional change needs screenshots or video of the running Canvas+ exercising the changed behavior, including backend integration, launcher,+ API, and lifecycle changes. Logs, commands, and tests supplement—not replace—+ that demonstration. Bug fixes must show failure before and success after;+ new features must demonstrate the working behavior.- Authentic evidence comes from the real Agent Canvas app, browser, OS dialog,generated or downloaded artifact, or actual terminal/runtime output. It mustinclude enough surrounding context and reproduction steps to establish what@@ -286,9 +344,15 @@ correctness, security, compatibility, or architecture defect.- Do not duplicate library behavior or add brittle presentation-only snapshots.Tests, mock-LLM runs, and mocked-backend runs are regression proof, not a-substitute for required live evidence. Submit **COMMENT** when production-facing-evidence is required but absent or ambiguous, and name the exact capture or-verification still needed before approval.+substitute for live evidence when the changed behavior requires it. Submit+**COMMENT** when production-facing evidence is required but absent or ambiguous,+and name the exact capture or verification still needed before approval.++The PR-description checker requires media for frontend paths and declared bug+fixes or features, including nonfrontend changes. It checks presence, not whether+Canvas actually ran or the capture proves before/after behavior. Reviewers enforce+the functional-change evidence rule above even when a PR is labeled as a chore;+do not fabricate media or relabel a functional change to avoid evidence.Follow the test routing in `AGENTS.md`. Mock-LLM, Docker mock-LLM, and liveLLM-backed E2E suites run after changes reach `main`, not from PR labels. Fordiff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.ymlindex 78f0549170..bd4e82682d 100644--- a/.github/ISSUE_TEMPLATE/bug_report.yml+++ b/.github/ISSUE_TEMPLATE/bug_report.yml@@ -76,10 +76,10 @@ body:attributes:label: Actual Behaviordescription: |- What actually happened? To qualify for the `ready-for-dev` label,- describe the observed behavior and attach a screenshot or video of the- bug in this section.- Drag a screenshot/video into this field, or paste a link.+ Describe what happened. For functional Canvas bugs, include a screenshot+ or video of the failure in the running app. Logs, API output, and+ reproducers can supplement it. Say if you have not reproduced it.+ Links in comments count; no after-fix evidence is needed for triage.placeholder: |The button moved to the next step unexpectedly.@@ -109,11 +109,11 @@ body:attributes:label: Acceptance Criteriadescription: |- Required for the `ready-for-dev` label. Testable, checkable bullets a- reviewer can verify. Describe the *fixed* state, not the bug.+ For `ready-for-dev`, list observable fixed outcomes, not PR artifacts.+ Tests and before/after fix evidence are not needed for triage.placeholder: |- - [ ] A screenshot/video demonstrating the bug can be replicated on main is attached to the PR.- - [ ] A screenshot/video demonstrating that the bug is fixed on the PR branch is attached to the PR.+ - [ ] Following the reproduction steps produces the expected result.+ - [ ] The affected mode handles the failure without losing user state.- [ ] …validations:required: falsediff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.ymlindex c127811762..164026c3ae 100644--- a/.github/ISSUE_TEMPLATE/feature_request.yml+++ b/.github/ISSUE_TEMPLATE/feature_request.yml@@ -35,10 +35,10 @@ body:attributes:label: Acceptance Criteriadescription: |- Required for the `ready-for-dev` label. Testable, checkable bullets a- reviewer can verify to confirm the feature is done.+ For `ready-for-dev`, list observable outcomes that show the feature is+ done, not PR artifacts or tests for an unbuilt feature.placeholder: |- - [ ] A screenshot/video demonstrating the feature behavior is attached to the PR.+ - [ ] As a user, I can complete the requested action in the affected mode.- [ ] …validations:required: truediff --git a/.github/pull_request_template.md b/.github/pull_request_template.mdindex e4d00e78e3..291196afb1 100644--- a/.github/pull_request_template.md+++ b/.github/pull_request_template.md@@ -24,7 +24,7 @@ exactly what command you ran and include logs, screenshots, or reproduction note## Summary-<!-- 1-3 bullets describing what changed. -->+<!-- 1-3 bullets describing what changed. For bugs, state the failure and fix. -->-## Issue Number@@ -39,17 +39,21 @@ Fixes #<!--Required. Share the steps for the reviewer to be able to test your PR. e.g. You can test by running `npm install` then `npm build dev`.+For bugs, include reproduction steps and observed before/after results.+For functional changes, logs and tests supplement the running-Canvas demonstration below.+Nonvisual evidence alone is sufficient only for non-functional changes.+If you could not test this, say why.-->## Video/Screenshots<!---Provide a video or screenshots of testing your PR. e.g. you added a new feature to the gui, show us the video of you testing it successfully.--For bug fixes: reproduction evidence is required. Show the bug reproduced (the-error state) and then the result after your fix. A terminal screenshot or video-is fine for non-UI bugs.+Functional changes require screenshots/video of the running Canvas exercising+the changed behavior, even outside frontend files. For bugs, demonstrate failure+before and success after using the same setup. For new features, show the working+behavior. Use video for timing or transitions. Logs/tests alone do not suffice.+Non-functional changes may use relevant commands and results under How to Test.-->## Design Docdiff --git a/.github/scripts/check_pr_description.py b/.github/scripts/check_pr_description.pyindex 54e1ec7388..a28ee83505 100644--- a/.github/scripts/check_pr_description.py+++ b/.github/scripts/check_pr_description.py@@ -7,16 +7,14 @@ Additional checks:checked.- If frontend code was touched, the description must include a screenshot orvideo.-- If the PR is marked as a Bug fix, the description must include a screenshot- or video showing reproduction evidence — regardless of whether frontend- code was touched.+- Bug-fix and feature PRs require running-Canvas screenshots or video even+ outside frontend paths. Reviewers verify before/after behavior and authenticity.- The body must reference at least one issue (e.g. `Fixes #123`) and at leastone referenced issue must carry the `ready-for-dev` label. The API lookup isonly performed in CI (when GITHUB_EVENT_PATH and GITHUB_TOKEN are available).- The PR's Type checkbox must match the linked issue's labels: a "Bug fix" PRshould link an issue with the `bug` label; a "Feature" PR should link one- with the `enhancement` label. This prevents a contributor from bypassing- bug-specific requirements by mislabeling the PR type.+ with the `enhancement` label.Local usage example:python .github/scripts/check_pr_description.py --body-file /tmp/pr-body.md \@@ -220,36 +218,17 @@ def validate_human_tested_checkbox(body: str) -> list[str]:def validate_frontend_screenshot(body: str, files: list[str]) -> list[str]:- """Require a screenshot/video in the body when frontend code was touched."""- if not touches_frontend(files):+ """Require media for frontend edits and declared functional changes."""+ frontend = touches_frontend(files)+ if not frontend and extract_pr_type(body) not in (BUG_LABEL, ENHANCEMENT_LABEL):return []if has_screenshot_or_video(body):return []+ change = "touches frontend code" if frontend else "declares a functional change"return [- "This PR touches frontend code but the description has no screenshot or "- "video. Add one under `## Video/Screenshots` (drag a file into the editor "- "or paste a video link)."- ]---def validate_bug_fix_evidence(body: str) -> list[str]:- """Require reproduction evidence when the PR is marked as a Bug fix.-- A bug-fix PR must include a screenshot or video showing the bug reproduced- and then fixed. This applies regardless of whether frontend code was touched- — a terminal screenshot showing the error before and the fix after is just- as valid as a UI screenshot.- """- pr_type = extract_pr_type(body)- if pr_type != BUG_LABEL:- return []- if has_screenshot_or_video(body):- return []- return [- "This PR is marked as a Bug fix but the description has no screenshot or "- "video. Add reproduction evidence under `## Video/Screenshots` showing the "- "bug before the fix and the result after (drag a file into the editor or "- "paste a video link)."+ f"This PR {change} but the description has no screenshot or video. "+ "Add evidence from a running Canvas under `## Video/Screenshots`; "+ "for bug fixes, demonstrate failure before and success after the change."]@@ -411,7 +390,6 @@ def validate_pr_body(body: str, files: list[str] | None = None) -> list[str]:errors.extend(validate_human_tested_checkbox(body))errors.extend(validate_frontend_screenshot(body, files or []))- errors.extend(validate_bug_fix_evidence(body))return errorsdiff --git a/.github/scripts/tests/test_pr_description.py b/.github/scripts/tests/test_pr_description.pyindex 4891578ba8..fa52234b36 100644--- a/.github/scripts/tests/test_pr_description.py+++ b/.github/scripts/tests/test_pr_description.py@@ -16,7 +16,6 @@ from check_pr_description import (extract_linked_issue_numbers,extract_pr_type,validate_linked_issue_ready,- validate_bug_fix_evidence,BUG_LABEL,ENHANCEMENT_LABEL,)@@ -210,71 +209,83 @@ def test_type_check_no_type_checked_no_type_error():# ----------------------------------------------------------------------------# validate_bug_fix_evidence+# Behavioral evidence through the complete validator# ----------------------------------------------------------------------------BUG_FIX_BODY_NO_EVIDENCE = """## Type+NONVISUAL_BUG_BODY = """HUMAN:-- [x] Bug fix-- [ ] Feature-"""--BUG_FIX_BODY_WITH_SCREENSHOT = """## Type+I reproduced the shutdown failure and verified the corrected exit status.-- [x] Bug fix-- [ ] Feature+AGENT:-## Video/Screenshots+## Why+The launcher left a child process alive after shutdown.- → -"""+## Summary+Stop the child process before exiting the launcher.-FEATURE_BODY = """## Type+## How to Test+Run `node scripts/launcher.cjs`, interrupt it, then inspect the child PID.+Before: child PID still running, exit status 1.+After: no remaining child process, exit status 0.-- [ ] Bug fix-- [x] Feature+## Type+- [x] Bug fix"""-def test_bug_fix_no_screenshot_errors():- errors = validate_bug_fix_evidence(BUG_FIX_BODY_NO_EVIDENCE)++def test_functional_launcher_bug_requires_running_canvas_media():+ errors = validate_pr_body(NONVISUAL_BUG_BODY, ["scripts/launcher.cjs"])assert len(errors) == 1- assert "reproduction evidence" in errors[0].lower()+ assert "functional change" in errors[0]-def test_bug_fix_with_screenshot_no_errors():- errors = validate_bug_fix_evidence(BUG_FIX_BODY_WITH_SCREENSHOT)- assert errors == []-def test_feature_no_bug_evidence_check():- errors = validate_bug_fix_evidence(FEATURE_BODY)- assert errors == []+def test_nonfrontend_feature_requires_running_canvas_media():+ body = NONVISUAL_BUG_BODY.replace("[x] Bug fix", "[x] Feature")+ assert len(validate_pr_body(body, ["scripts/launcher.cjs"])) == 1-def test_no_type_no_bug_evidence_check():- errors = validate_bug_fix_evidence("## Summary\n\nNo type\n")- assert errors == []-def test_bug_fix_with_github_attachment_no_errors():- body = """## Type+def test_functional_launcher_bug_accepts_canvas_recording():+ body = NONVISUAL_BUG_BODY + "\n## Video/Screenshots\nhttps://github.com/user-attachments/assets/abc123"+ assert validate_pr_body(body, ["scripts/launcher.cjs"]) == []-- [x] Bug fix-## Video/Screenshots+def test_nonfunctional_change_accepts_text_evidence():+ body = NONVISUAL_BUG_BODY.replace("[x] Bug fix", "[x] Docs / chore")+ assert validate_pr_body(body, ["docs/README.md"]) == []-https://github.com/user-attachments/assets/abc123-"""- errors = validate_bug_fix_evidence(body)- assert errors == []-def test_bug_fix_with_video_link_no_errors():- body = """## Type+def test_nonvisual_bug_still_requires_summary_and_test_details():+ for section in ("Summary", "How to Test"):+ content = extract_sections(NONVISUAL_BUG_BODY)[section]+ body = NONVISUAL_BUG_BODY.replace(content, "\n<!-- Fill this in -->\n")+ assert f"Fill in the `## {section}` section of the PR template." in validate_pr_body(+ body, ["scripts/launcher.cjs"]+ )-- [x] Bug fix-## Video/Screenshots+def test_frontend_bug_reproduction_logs_do_not_replace_media():+ errors = validate_pr_body(NONVISUAL_BUG_BODY, ["src/app.tsx"])+ assert len(errors) == 1+ assert "touches frontend code" in errors[0]+-https://youtube.com/watch?v=abc123-"""- errors = validate_bug_fix_evidence(body)- assert errors == []+def test_mixed_bug_change_still_requires_frontend_media():+ errors = validate_pr_body(+ NONVISUAL_BUG_BODY, ["scripts/launcher.cjs", "src/styles/main.css"]+ )+ assert len(errors) == 1+ assert "touches frontend code" in errors[0]+++def test_frontend_bug_accepts_screenshot_or_video():+ for media in (+ " → ",+ "https://github.com/user-attachments/assets/abc123",+ "https://youtube.com/watch?v=abc123",+ ):+ body = NONVISUAL_BUG_BODY + "\n## Video/Screenshots\n" + media+ assert validate_pr_body(body, ["src/app.tsx"]) == []def test_markdown_under_frontend_prefix_is_not_frontend():