Skip to content

πŸ™Œ OpenHands: AI-Driven Development

test: rename sidebar nav label "New" β†’ "Chats" to verify snapshot CI diff comment (#497)

Rohit Malhotra63704f65e5f4
diff --git a/.github/workflows/snapshot-tests.yml b/.github/workflows/snapshot-tests.yml
index fafed6c5c2..faf9c01e97 100644
--- a/.github/workflows/snapshot-tests.yml
+++ b/.github/workflows/snapshot-tests.yml
@@ -160,6 +160,20 @@ jobs:
run: npm run test:e2e:snapshots
continue-on-error: true
+ # Playwright clears test-results/ at the start of every new run, so the
+ # diff files produced by the comparison step above would be wiped by the
+ # update step below before post-snapshot-comment.mjs can read them.
+ # Save the comparison output now so the comment script can still find the
+ # diff/actual/expected images.
+ - name: Save comparison test-results
+ if: github.event_name == 'pull_request'
+ run: |
+ if [ -d "test-results" ]; then
+ cp -r test-results /tmp/comparison-results
+ else
+ mkdir -p /tmp/comparison-results
+ fi
+
- name: Generate current PR snapshots
if: github.event_name == 'pull_request'
run: npm run test:e2e:snapshots:update
@@ -173,6 +187,7 @@ jobs:
RUN_ID: ${{ github.run_id }}
HEAD_REF: ${{ github.head_ref }}
MAIN_BASELINES_DIR: /tmp/main-baselines
+ COMPARISON_RESULTS_DIR: /tmp/comparison-results
SNAPSHOTS_APPROVED: ${{ contains(github.event.pull_request.labels.*.name, 'update-snapshots') }}
run: node tests/e2e/snapshots/scripts/post-snapshot-comment.mjs
diff --git a/AGENTS.md b/AGENTS.md
index 618882ff57..d9e02baea6 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -29,6 +29,7 @@
- **On PRs**: Downloads the latest `snapshot-baselines` artifact, runs `test:e2e:snapshots` against it, generates current snapshots via `test:e2e:snapshots:update`, then posts a fresh PR comment (old comment is deleted first so image URLs always point to the current run). Changed snapshots are shown in a side-by-side expected/actual/diff table; new snapshots show the full screenshot. Images are force-pushed to a dedicated `snapshot-artifacts/pr-<N>` orphan branch (NOT the PR branch) so required CI checks are never invalidated. URLs are `raw.githubusercontent.com/<owner>/<repo>/<sha>/changed/...` or `.../new/...`. Triggers: `opened`, `synchronize`, `reopened`, `labeled`, `unlabeled`.
- **Force-regenerate baselines**: Trigger the `Snapshot Tests` workflow manually with `force_update=true`.
- **PR comment**: `tests/e2e/snapshots/scripts/post-snapshot-comment.mjs` posts a fresh comment (`<!-- snapshot-test-report -->` marker) with collapsed `<details>` sections β€” πŸ”΄ Changed (side-by-side expected/actual/diff), πŸ†• New (full screenshot), βœ… Unchanged (list of names).
+- **Critical ordering note**: Playwright clears `test-results/` at the start of every new run. The workflow runs two Playwright passes: (1) `test:e2e:snapshots` (comparison, writes `*-diff.png`), then (2) `test:e2e:snapshots:update` (regenerates baselines, clears `test-results/`, no diffs written). The "Save comparison test-results" step copies `test-results/` to `/tmp/comparison-results` between these two passes and passes it as `COMPARISON_RESULTS_DIR` to the comment script, which reads `TEST_RESULTS_DIR` from that env var. Without this step all snapshots appear "unchanged" because the diff files are gone.
- **Acknowledging intentional changes**: If snapshots changed on purpose (UI redesign, etc.), add the `update-snapshots` label to the PR. This causes: (1) the CI failure step to be skipped so the check passes, (2) the comment status to flip to βœ… with a note that changes are acknowledged, (3) the `labeled` trigger fires a fresh CI run automatically so no manual re-run is needed. Removing the label re-enables the failure. When the PR merges, the main-branch run uploads the new screenshots as the updated baseline β€” no separate "regenerate on main" step required.
- **Viewing diffs**: On failure, Playwright generates `*-actual.png`, `*-expected.png`, and `*-diff.png` in `test-results/`. Run `npx playwright show-report` to view the HTML report. The PR comment also embeds these images directly.
- **Bootstrap**: When no `snapshot-baselines` artifact exists on main yet, all snapshots are classified as "πŸ†• New" and CI passes. The first main-branch run after this state uploads the initial artifact.
diff --git a/tests/e2e/snapshots/scripts/post-snapshot-comment.mjs b/tests/e2e/snapshots/scripts/post-snapshot-comment.mjs
index be4bc7cd90..414512ac4d 100644
--- a/tests/e2e/snapshots/scripts/post-snapshot-comment.mjs
+++ b/tests/e2e/snapshots/scripts/post-snapshot-comment.mjs
@@ -41,7 +41,11 @@ const MAIN_BASELINES_DIR =
const SNAPSHOTS_APPROVED = process.env.SNAPSHOTS_APPROVED === "true";
const SNAPSHOTS_DIR = "tests/e2e/__snapshots__";
-const TEST_RESULTS_DIR = "test-results";
+// The workflow saves comparison test-results to this path before the
+// --update-snapshots pass wipes test-results/. Fall back to the default
+// Playwright output directory when running outside CI.
+const TEST_RESULTS_DIR =
+ process.env.COMPARISON_RESULTS_DIR ?? "test-results";
// Images are pushed to this dedicated branch, NOT to the PR branch.
// Pushing to the PR branch with [skip ci] was blocking required checks on the HEAD commit.
const ARTIFACTS_BRANCH = `snapshot-artifacts/pr-${PR_NUMBER}`;