Skip to content

🙌 OpenHands: AI-Driven Development

docs: clarify Canvas triage and review evidence boundaries (#17990)

Graham Neubig6801911d8d43
diff --git a/.agents/skills/custom-codereview-guide.md b/.agents/skills/custom-codereview-guide.md
index 3e13cbece5..c6be40f372 100644
--- a/.agents/skills/custom-codereview-guide.md
+++ b/.agents/skills/custom-codereview-guide.md
@@ -1,6 +1,6 @@
---
name: custom-codereview-guide
-description: Repository-specific review rules for the OpenHands Agent Canvas frontend.
+description: Repository-specific triage and review rules for the OpenHands Agent Canvas frontend.
triggers:
- /codereview
---
@@ -31,6 +31,44 @@ integration and depends on public interfaces from the owning repository. When a
change belongs in one of the repositories above, say so and ask a maintainer to
confirm before reviewing the rest.
+## Triage: Ownership and Scope
+
+During triage, check ownership, supported behavior, and acceptance criteria.
+Issue readiness means the work is clear enough to start—not that the fix is
+complete. Do not require a PR, passing implementation tests, or before-and-after
+fix evidence. Reconsider readiness when new information leaves scope or expected
+behavior unresolved.
+
+Trace the failing boundary before assigning an owner:
+
+- Check the SDK for events, tools, ACP, profiles, conversation lifecycle, and persistence.
+- Check automation for form contracts, run state, and pagination.
+- Check producers and consumers at installed or supported versions using locks,
+ runtime versions, and `config/defaults.json`, not upstream main alone.
+ Cite versions and sources when ownership or compatibility depends on them.
+
+Prefer fixes in the owning provider. Separate provider work from Canvas integration
+and state release/dependency constraints. Distinguish new regressions from independent,
+preexisting upstream issues. Recommend material follow-ups without expanding scope,
+editing other repositories, or creating tickets without authorization.
+Reuse answers in the issue and linked discussions.
+
+## Triage: Behavioral Acceptance and Readiness
+
+Write acceptance criteria as observable outcomes, affected modes, and relevant
+failure/recovery cases, not implementation requirements or PR artifacts. Keep verification separate.
+Bugs need expected/actual behavior, environment/version, and reproduction steps;
+features need the desired outcome and scope, not evidence of an unbuilt feature.
+
+For Canvas functional bugs, capture the failure in the running app; logs, API
+output, and reproducers can supplement the capture. Accept evidence linked in
+comments without duplicate uploads. State uncertainty when the failure has not
+been reproduced. Triage does not require an after-fix demonstration.
+
+## Implementation and Merge Review
+
+The following checks apply to PRs, not triage.
+
## Review Sequence and Decision
Review the current PR head in this order:
@@ -58,6 +96,12 @@ lightweight evaluation. Use COMMENT when an acceptance criterion or required
check calls for specific eval evidence and that evidence is missing or failing,
or when available results show a regression.
+Separate code quality from merge readiness. Use **COMMENT** for a failing required
+check, an unmergeable branch, or missing required evidence, even if the code is sound. Name the unmet gate.
+Keep the summary, final verdict/footer, and submitted review state consistent;
+never say `APPROVED` with an unresolved material finding or merge gate.
+Optional evaluation follow-up remains nonblocking.
+
Include a compact checklist for every linked acceptance criterion. Meeting the
checklist is necessary but does not replace review for regressions, security, or
maintainability.
@@ -109,6 +153,10 @@ versioned.
Require an increase to `minimumAgentServer` to the first compatible released version.
+Before requiring or removing a fallback, show the SDK producer's payloads and behavior
+at the supported release floor and affected deployed versions.
+Old fixtures or hypothetical legacy fields alone do not justify compatibility code.
+
Verify the compatibility boundary. Adding a TypeScript-client method does not
make older Agent Servers support it. Submit **COMMENT** if a supported backend
can reach the new code and fail because the required server behavior is absent.
@@ -172,6 +220,12 @@ Telemetry has stricter named owners:
`setTelemetryConsent` is the only consent controller.
- A business milestone has one canonical capture.
+### Docker user and permission workarounds
+
+For Docker `--user` or `HOME` workarounds, test the full path under that UID/GID:
+home, config/cache directories, temporary files, and subprocess/browser startup.
+Check ownership and permissions outside the persistence mount too.
+
## Design Review
Prefer named hooks, services, stores, and feature modules over shared-root
@@ -243,12 +297,16 @@ correctness, security, compatibility, or architecture defect.
## Testing and Production Evidence
-- Every PR that touches frontend code needs an authentic screenshot or video
- captured while exercising the PR code. This includes non-visual frontend logic
- whose effect appears in a generated or downloaded user-facing artifact.
-- Require evidence proportional to the behavior changed. UI changes need a
- screenshot or video from the real app; CLI, API, and script changes need the
- exact runtime command and observed result.
+- Nonvisual evidence alone is acceptable only for non-functional changes, such
+ as documentation or test-only edits that do not alter product behavior. Use
+ relevant commands and output; docs-only changes need no runtime tests or media.
+- Check rendering in Canvas. Token, class-name, and source-string assertions do
+ not prove computed styling, isolation, layout, or readability, or replace captures.
+- Every functional change needs screenshots or video of the running Canvas
+ exercising the changed behavior, including backend integration, launcher,
+ API, and lifecycle changes. Logs, commands, and tests supplement—not replace—
+ that demonstration. Bug fixes must show failure before and success after;
+ new features must demonstrate the working behavior.
- Authentic evidence comes from the real Agent Canvas app, browser, OS dialog,
generated or downloaded artifact, or actual terminal/runtime output. It must
include enough surrounding context and reproduction steps to establish what
@@ -286,9 +344,15 @@ correctness, security, compatibility, or architecture defect.
- Do not duplicate library behavior or add brittle presentation-only snapshots.
Tests, mock-LLM runs, and mocked-backend runs are regression proof, not a
-substitute for required live evidence. Submit **COMMENT** when production-facing
-evidence is required but absent or ambiguous, and name the exact capture or
-verification still needed before approval.
+substitute for live evidence when the changed behavior requires it. Submit
+**COMMENT** when production-facing evidence is required but absent or ambiguous,
+and name the exact capture or verification still needed before approval.
+
+The PR-description checker requires media for frontend paths and declared bug
+fixes or features, including nonfrontend changes. It checks presence, not whether
+Canvas actually ran or the capture proves before/after behavior. Reviewers enforce
+the functional-change evidence rule above even when a PR is labeled as a chore;
+do not fabricate media or relabel a functional change to avoid evidence.
Follow the test routing in `AGENTS.md`. Mock-LLM, Docker mock-LLM, and live
LLM-backed E2E suites run after changes reach `main`, not from PR labels. For
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
index 78f0549170..bd4e82682d 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.yml
+++ b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -76,10 +76,10 @@ body:
attributes:
label: Actual Behavior
description: |
- What actually happened? To qualify for the `ready-for-dev` label,
- describe the observed behavior and attach a screenshot or video of the
- bug in this section.
- Drag a screenshot/video into this field, or paste a link.
+ Describe what happened. For functional Canvas bugs, include a screenshot
+ or video of the failure in the running app. Logs, API output, and
+ reproducers can supplement it. Say if you have not reproduced it.
+ Links in comments count; no after-fix evidence is needed for triage.
placeholder: |
The button moved to the next step unexpectedly.
@@ -109,11 +109,11 @@ body:
attributes:
label: Acceptance Criteria
description: |
- Required for the `ready-for-dev` label. Testable, checkable bullets a
- reviewer can verify. Describe the *fixed* state, not the bug.
+ For `ready-for-dev`, list observable fixed outcomes, not PR artifacts.
+ Tests and before/after fix evidence are not needed for triage.
placeholder: |
- - [ ] A screenshot/video demonstrating the bug can be replicated on main is attached to the PR.
- - [ ] A screenshot/video demonstrating that the bug is fixed on the PR branch is attached to the PR.
+ - [ ] Following the reproduction steps produces the expected result.
+ - [ ] The affected mode handles the failure without losing user state.
- [ ] …
validations:
required: false
diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml
index c127811762..164026c3ae 100644
--- a/.github/ISSUE_TEMPLATE/feature_request.yml
+++ b/.github/ISSUE_TEMPLATE/feature_request.yml
@@ -35,10 +35,10 @@ body:
attributes:
label: Acceptance Criteria
description: |
- Required for the `ready-for-dev` label. Testable, checkable bullets a
- reviewer can verify to confirm the feature is done.
+ For `ready-for-dev`, list observable outcomes that show the feature is
+ done, not PR artifacts or tests for an unbuilt feature.
placeholder: |
- - [ ] A screenshot/video demonstrating the feature behavior is attached to the PR.
+ - [ ] As a user, I can complete the requested action in the affected mode.
- [ ] …
validations:
required: true
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index e4d00e78e3..291196afb1 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -24,7 +24,7 @@ exactly what command you ran and include logs, screenshots, or reproduction note
## Summary
-<!-- 1-3 bullets describing what changed. -->
+<!-- 1-3 bullets describing what changed. For bugs, state the failure and fix. -->
-
## Issue Number
@@ -39,17 +39,21 @@ Fixes #
<!--
Required. Share the steps for the reviewer to be able to test your PR. e.g. You can test by running `npm install` then `npm build dev`.
+For bugs, include reproduction steps and observed before/after results.
+For functional changes, logs and tests supplement the running-Canvas demonstration below.
+Nonvisual evidence alone is sufficient only for non-functional changes.
+
If you could not test this, say why.
-->
## Video/Screenshots
<!--
-Provide a video or screenshots of testing your PR. e.g. you added a new feature to the gui, show us the video of you testing it successfully.
-
-For bug fixes: reproduction evidence is required. Show the bug reproduced (the
-error state) and then the result after your fix. A terminal screenshot or video
-is fine for non-UI bugs.
+Functional changes require screenshots/video of the running Canvas exercising
+the changed behavior, even outside frontend files. For bugs, demonstrate failure
+before and success after using the same setup. For new features, show the working
+behavior. Use video for timing or transitions. Logs/tests alone do not suffice.
+Non-functional changes may use relevant commands and results under How to Test.
-->
## Design Doc
diff --git a/.github/scripts/check_pr_description.py b/.github/scripts/check_pr_description.py
index 54e1ec7388..a28ee83505 100644
--- a/.github/scripts/check_pr_description.py
+++ b/.github/scripts/check_pr_description.py
@@ -7,16 +7,14 @@ Additional checks:
checked.
- If frontend code was touched, the description must include a screenshot or
video.
-- If the PR is marked as a Bug fix, the description must include a screenshot
- or video showing reproduction evidence — regardless of whether frontend
- code was touched.
+- Bug-fix and feature PRs require running-Canvas screenshots or video even
+ outside frontend paths. Reviewers verify before/after behavior and authenticity.
- The body must reference at least one issue (e.g. `Fixes #123`) and at least
one referenced issue must carry the `ready-for-dev` label. The API lookup is
only performed in CI (when GITHUB_EVENT_PATH and GITHUB_TOKEN are available).
- The PR's Type checkbox must match the linked issue's labels: a "Bug fix" PR
should link an issue with the `bug` label; a "Feature" PR should link one
- with the `enhancement` label. This prevents a contributor from bypassing
- bug-specific requirements by mislabeling the PR type.
+ with the `enhancement` label.
Local usage example:
python .github/scripts/check_pr_description.py --body-file /tmp/pr-body.md \
@@ -220,36 +218,17 @@ def validate_human_tested_checkbox(body: str) -> list[str]:
def validate_frontend_screenshot(body: str, files: list[str]) -> list[str]:
- """Require a screenshot/video in the body when frontend code was touched."""
- if not touches_frontend(files):
+ """Require media for frontend edits and declared functional changes."""
+ frontend = touches_frontend(files)
+ if not frontend and extract_pr_type(body) not in (BUG_LABEL, ENHANCEMENT_LABEL):
return []
if has_screenshot_or_video(body):
return []
+ change = "touches frontend code" if frontend else "declares a functional change"
return [
- "This PR touches frontend code but the description has no screenshot or "
- "video. Add one under `## Video/Screenshots` (drag a file into the editor "
- "or paste a video link)."
- ]
-
-
-def validate_bug_fix_evidence(body: str) -> list[str]:
- """Require reproduction evidence when the PR is marked as a Bug fix.
-
- A bug-fix PR must include a screenshot or video showing the bug reproduced
- and then fixed. This applies regardless of whether frontend code was touched
- — a terminal screenshot showing the error before and the fix after is just
- as valid as a UI screenshot.
- """
- pr_type = extract_pr_type(body)
- if pr_type != BUG_LABEL:
- return []
- if has_screenshot_or_video(body):
- return []
- return [
- "This PR is marked as a Bug fix but the description has no screenshot or "
- "video. Add reproduction evidence under `## Video/Screenshots` showing the "
- "bug before the fix and the result after (drag a file into the editor or "
- "paste a video link)."
+ f"This PR {change} but the description has no screenshot or video. "
+ "Add evidence from a running Canvas under `## Video/Screenshots`; "
+ "for bug fixes, demonstrate failure before and success after the change."
]
@@ -411,7 +390,6 @@ def validate_pr_body(body: str, files: list[str] | None = None) -> list[str]:
errors.extend(validate_human_tested_checkbox(body))
errors.extend(validate_frontend_screenshot(body, files or []))
- errors.extend(validate_bug_fix_evidence(body))
return errors
diff --git a/.github/scripts/tests/test_pr_description.py b/.github/scripts/tests/test_pr_description.py
index 4891578ba8..fa52234b36 100644
--- a/.github/scripts/tests/test_pr_description.py
+++ b/.github/scripts/tests/test_pr_description.py
@@ -16,7 +16,6 @@ from check_pr_description import (
extract_linked_issue_numbers,
extract_pr_type,
validate_linked_issue_ready,
- validate_bug_fix_evidence,
BUG_LABEL,
ENHANCEMENT_LABEL,
)
@@ -210,71 +209,83 @@ def test_type_check_no_type_checked_no_type_error():
# ---------------------------------------------------------------------------
-# validate_bug_fix_evidence
+# Behavioral evidence through the complete validator
# ---------------------------------------------------------------------------
-BUG_FIX_BODY_NO_EVIDENCE = """## Type
+NONVISUAL_BUG_BODY = """HUMAN:
-- [x] Bug fix
-- [ ] Feature
-"""
-
-BUG_FIX_BODY_WITH_SCREENSHOT = """## Type
+I reproduced the shutdown failure and verified the corrected exit status.
-- [x] Bug fix
-- [ ] Feature
+AGENT:
-## Video/Screenshots
+## Why
+The launcher left a child process alive after shutdown.
-![before](https://example.com/before.png) → ![after](https://example.com/after.png)
-"""
+## Summary
+Stop the child process before exiting the launcher.
-FEATURE_BODY = """## Type
+## How to Test
+Run `node scripts/launcher.cjs`, interrupt it, then inspect the child PID.
+Before: child PID still running, exit status 1.
+After: no remaining child process, exit status 0.
-- [ ] Bug fix
-- [x] Feature
+## Type
+- [x] Bug fix
"""
-def test_bug_fix_no_screenshot_errors():
- errors = validate_bug_fix_evidence(BUG_FIX_BODY_NO_EVIDENCE)
+
+def test_functional_launcher_bug_requires_running_canvas_media():
+ errors = validate_pr_body(NONVISUAL_BUG_BODY, ["scripts/launcher.cjs"])
assert len(errors) == 1
- assert "reproduction evidence" in errors[0].lower()
+ assert "functional change" in errors[0]
-def test_bug_fix_with_screenshot_no_errors():
- errors = validate_bug_fix_evidence(BUG_FIX_BODY_WITH_SCREENSHOT)
- assert errors == []
-def test_feature_no_bug_evidence_check():
- errors = validate_bug_fix_evidence(FEATURE_BODY)
- assert errors == []
+def test_nonfrontend_feature_requires_running_canvas_media():
+ body = NONVISUAL_BUG_BODY.replace("[x] Bug fix", "[x] Feature")
+ assert len(validate_pr_body(body, ["scripts/launcher.cjs"])) == 1
-def test_no_type_no_bug_evidence_check():
- errors = validate_bug_fix_evidence("## Summary\n\nNo type\n")
- assert errors == []
-def test_bug_fix_with_github_attachment_no_errors():
- body = """## Type
+def test_functional_launcher_bug_accepts_canvas_recording():
+ body = NONVISUAL_BUG_BODY + "\n## Video/Screenshots\nhttps://github.com/user-attachments/assets/abc123"
+ assert validate_pr_body(body, ["scripts/launcher.cjs"]) == []
-- [x] Bug fix
-## Video/Screenshots
+def test_nonfunctional_change_accepts_text_evidence():
+ body = NONVISUAL_BUG_BODY.replace("[x] Bug fix", "[x] Docs / chore")
+ assert validate_pr_body(body, ["docs/README.md"]) == []
-https://github.com/user-attachments/assets/abc123
-"""
- errors = validate_bug_fix_evidence(body)
- assert errors == []
-def test_bug_fix_with_video_link_no_errors():
- body = """## Type
+def test_nonvisual_bug_still_requires_summary_and_test_details():
+ for section in ("Summary", "How to Test"):
+ content = extract_sections(NONVISUAL_BUG_BODY)[section]
+ body = NONVISUAL_BUG_BODY.replace(content, "\n<!-- Fill this in -->\n")
+ assert f"Fill in the `## {section}` section of the PR template." in validate_pr_body(
+ body, ["scripts/launcher.cjs"]
+ )
-- [x] Bug fix
-## Video/Screenshots
+def test_frontend_bug_reproduction_logs_do_not_replace_media():
+ errors = validate_pr_body(NONVISUAL_BUG_BODY, ["src/app.tsx"])
+ assert len(errors) == 1
+ assert "touches frontend code" in errors[0]
+
-https://youtube.com/watch?v=abc123
-"""
- errors = validate_bug_fix_evidence(body)
- assert errors == []
+def test_mixed_bug_change_still_requires_frontend_media():
+ errors = validate_pr_body(
+ NONVISUAL_BUG_BODY, ["scripts/launcher.cjs", "src/styles/main.css"]
+ )
+ assert len(errors) == 1
+ assert "touches frontend code" in errors[0]
+
+
+def test_frontend_bug_accepts_screenshot_or_video():
+ for media in (
+ "![before](https://example.com/before.png) → ![after](https://example.com/after.png)",
+ "https://github.com/user-attachments/assets/abc123",
+ "https://youtube.com/watch?v=abc123",
+ ):
+ body = NONVISUAL_BUG_BODY + "\n## Video/Screenshots\n" + media
+ assert validate_pr_body(body, ["src/app.tsx"]) == []
def test_markdown_under_frontend_prefix_is_not_frontend():