π OpenHands: AI-Driven Development
test: rename sidebar nav label "New" β "Chats" to verify snapshot CI diff comment (#497)
Rohit Malhotra63704f65e5f4
diff --git a/.github/workflows/snapshot-tests.yml b/.github/workflows/snapshot-tests.ymlindex fafed6c5c2..faf9c01e97 100644--- a/.github/workflows/snapshot-tests.yml+++ b/.github/workflows/snapshot-tests.yml@@ -160,6 +160,20 @@ jobs:run: npm run test:e2e:snapshotscontinue-on-error: true+ # Playwright clears test-results/ at the start of every new run, so the+ # diff files produced by the comparison step above would be wiped by the+ # update step below before post-snapshot-comment.mjs can read them.+ # Save the comparison output now so the comment script can still find the+ # diff/actual/expected images.+ - name: Save comparison test-results+ if: github.event_name == 'pull_request'+ run: |+ if [ -d "test-results" ]; then+ cp -r test-results /tmp/comparison-results+ else+ mkdir -p /tmp/comparison-results+ fi+- name: Generate current PR snapshotsif: github.event_name == 'pull_request'run: npm run test:e2e:snapshots:update@@ -173,6 +187,7 @@ jobs:RUN_ID: ${{ github.run_id }}HEAD_REF: ${{ github.head_ref }}MAIN_BASELINES_DIR: /tmp/main-baselines+ COMPARISON_RESULTS_DIR: /tmp/comparison-resultsSNAPSHOTS_APPROVED: ${{ contains(github.event.pull_request.labels.*.name, 'update-snapshots') }}run: node tests/e2e/snapshots/scripts/post-snapshot-comment.mjsdiff --git a/AGENTS.md b/AGENTS.mdindex 618882ff57..d9e02baea6 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -29,6 +29,7 @@- **On PRs**: Downloads the latest `snapshot-baselines` artifact, runs `test:e2e:snapshots` against it, generates current snapshots via `test:e2e:snapshots:update`, then posts a fresh PR comment (old comment is deleted first so image URLs always point to the current run). Changed snapshots are shown in a side-by-side expected/actual/diff table; new snapshots show the full screenshot. Images are force-pushed to a dedicated `snapshot-artifacts/pr-<N>` orphan branch (NOT the PR branch) so required CI checks are never invalidated. URLs are `raw.githubusercontent.com/<owner>/<repo>/<sha>/changed/...` or `.../new/...`. Triggers: `opened`, `synchronize`, `reopened`, `labeled`, `unlabeled`.- **Force-regenerate baselines**: Trigger the `Snapshot Tests` workflow manually with `force_update=true`.- **PR comment**: `tests/e2e/snapshots/scripts/post-snapshot-comment.mjs` posts a fresh comment (`<!-- snapshot-test-report -->` marker) with collapsed `<details>` sections β π΄ Changed (side-by-side expected/actual/diff), π New (full screenshot), β Unchanged (list of names).+- **Critical ordering note**: Playwright clears `test-results/` at the start of every new run. The workflow runs two Playwright passes: (1) `test:e2e:snapshots` (comparison, writes `*-diff.png`), then (2) `test:e2e:snapshots:update` (regenerates baselines, clears `test-results/`, no diffs written). The "Save comparison test-results" step copies `test-results/` to `/tmp/comparison-results` between these two passes and passes it as `COMPARISON_RESULTS_DIR` to the comment script, which reads `TEST_RESULTS_DIR` from that env var. Without this step all snapshots appear "unchanged" because the diff files are gone.- **Acknowledging intentional changes**: If snapshots changed on purpose (UI redesign, etc.), add the `update-snapshots` label to the PR. This causes: (1) the CI failure step to be skipped so the check passes, (2) the comment status to flip to β with a note that changes are acknowledged, (3) the `labeled` trigger fires a fresh CI run automatically so no manual re-run is needed. Removing the label re-enables the failure. When the PR merges, the main-branch run uploads the new screenshots as the updated baseline β no separate "regenerate on main" step required.- **Viewing diffs**: On failure, Playwright generates `*-actual.png`, `*-expected.png`, and `*-diff.png` in `test-results/`. Run `npx playwright show-report` to view the HTML report. The PR comment also embeds these images directly.- **Bootstrap**: When no `snapshot-baselines` artifact exists on main yet, all snapshots are classified as "π New" and CI passes. The first main-branch run after this state uploads the initial artifact.diff --git a/tests/e2e/snapshots/scripts/post-snapshot-comment.mjs b/tests/e2e/snapshots/scripts/post-snapshot-comment.mjsindex be4bc7cd90..414512ac4d 100644--- a/tests/e2e/snapshots/scripts/post-snapshot-comment.mjs+++ b/tests/e2e/snapshots/scripts/post-snapshot-comment.mjs@@ -41,7 +41,11 @@ const MAIN_BASELINES_DIR =const SNAPSHOTS_APPROVED = process.env.SNAPSHOTS_APPROVED === "true";const SNAPSHOTS_DIR = "tests/e2e/__snapshots__";-const TEST_RESULTS_DIR = "test-results";+// The workflow saves comparison test-results to this path before the+// --update-snapshots pass wipes test-results/. Fall back to the default+// Playwright output directory when running outside CI.+const TEST_RESULTS_DIR =+ process.env.COMPARISON_RESULTS_DIR ?? "test-results";// Images are pushed to this dedicated branch, NOT to the PR branch.// Pushing to the PR branch with [skip ci] was blocking required checks on the HEAD commit.const ARTIFACTS_BRANCH = `snapshot-artifacts/pr-${PR_NUMBER}`;