Skip to content

πŸ™Œ OpenHands: AI-Driven Development

chore: defer PR artifact cleanup until after merge to main (#18469)

Engel Nyst6dda32edd7dc
diff --git a/.agents/skills/custom-codereview-guide.md b/.agents/skills/custom-codereview-guide.md
index c6be40f372..bf5fd68f2f 100644
--- a/.agents/skills/custom-codereview-guide.md
+++ b/.agents/skills/custom-codereview-guide.md
@@ -267,10 +267,10 @@ Adequate design context states:
Put it in the PR description, or link a `.pr/` design doc that covers it from
the `pr-design-doc` skill. Count a `.pr/` doc only when the link is pinned to a
-commit SHA, not the branch name. Approving a same-repository PR runs the
-`PR Artifacts` cleanup, which removes `.pr/` from the branch, so a branch link
-stops resolving before a human maintainer reads it, while a SHA link keeps
-working.
+full commit SHA, not the branch name. After either a same-repository or fork PR
+merges, the `PR Artifacts` workflow opens or updates a cleanup PR that removes
+`.pr/` from `main`. A SHA-pinned link keeps working after post-merge cleanup or
+PR branch deletion.
Scale the response to the risk assessment:
diff --git a/.agents/skills/e2e-testing/references/guide.md b/.agents/skills/e2e-testing/references/guide.md
index 3b6cd05460..d2b098a261 100644
--- a/.agents/skills/e2e-testing/references/guide.md
+++ b/.agents/skills/e2e-testing/references/guide.md
@@ -14,7 +14,7 @@
- Keep live E2E secrets out of job-level `env`. The workflow should check whether credentials exist before checkout, but inject the LLM key only into the trusted step that actually runs the live test.
- The live job uploads the Playwright HTML report plus screenshot/video output as a GitHub Actions artifact, and also extracts the primary screenshot/video attachments. It converts the WebM recording to a GIF preview with `ffmpeg` so GitHub PR comments can inline the preview. Keep Playwright trace capture disabled for live tests because the setup flow sends LLM credentials to the Agent Server settings API, and traces can record request bodies. Failure messages around live Agent Server settings must not print response bodies from credential-bearing requests.
- Inline PR-comment media is stored as PR-only files under `.pr/live-e2e/<github_run_id>/` on the PR branch, not on a long-lived orphan media branch. The comment uses `raw.githubusercontent.com/<repo>/<artifact_commit>/.pr/live-e2e/...` URLs for the GIF and PNG so GitHub can render them inline. The WebM is linked as the full recording because GitHub comments do not reliably inline WebM.
-- `.github/workflows/pr-artifacts.yml` owns cleanup for `.pr/live-e2e/`: it comments when `.pr/` artifacts exist, removes them after PR approval for same-repo PRs, and opens or updates a cleanup PR against `main` if artifacts reach `main` through a fork PR or a missed approval cleanup.
+- `.github/workflows/pr-artifacts.yml` owns cleanup for `.pr/live-e2e/`: it comments when `.pr/` artifacts exist and opens or updates a cleanup PR against `main` after either a same-repository or fork PR merges. Approval leaves review artifacts available without pushing another cleanup commit to the PR branch.
- The live reporting scripts live beside the live tests under `tests/e2e/live/scripts/`: `run-live-e2e.mjs`, `extract-live-e2e-media.mjs`, `render-live-e2e-report.mjs`, and `upsert-pr-comment.mjs`. Keep report/comment/local-runner logic there rather than in top-level `scripts/`, because these scripts are part of the live E2E framework.
- When changing any part of this framework β€” live workflow triggers, artifact publishing, `.pr` cleanup, live Playwright config, live test file layout, helper locations, local runner behavior, or report/comment scripts β€” update this `AGENTS.md` section in the same PR so future agents have the current operating model.
diff --git a/.agents/skills/pr-design-doc/SKILL.md b/.agents/skills/pr-design-doc/SKILL.md
index e5ef5066b0..b123248024 100644
--- a/.agents/skills/pr-design-doc/SKILL.md
+++ b/.agents/skills/pr-design-doc/SKILL.md
@@ -41,17 +41,18 @@ non-trivial PR easy to review.
Use the temporary **`.pr/`** directory for PR-only artifacts. Before relying on automatic
cleanup, verify that the target repository has an enabled
-`.github/workflows/pr-artifacts.yml` workflow that removes `.pr/` after approval.
+`.github/workflows/pr-artifacts.yml` workflow that opens or updates a cleanup PR after
+artifacts reach `main`.
-- Same-repository PR with verified cleanup workflow: the workflow removes `.pr/` after
- approval.
-- Fork PR, or repository without a verified cleanup workflow: remove `.pr/` manually before
- merge.
+- Same-repository or fork PR with a verified cleanup workflow: `.pr/` is removed
+ automatically through a cleanup PR after the PR merges to `main`.
+- Repository without a verified cleanup workflow: remove `.pr/` manually before merge.
-The design doc is a temporary review aid and must not ship in the merged tree. Automated
-approval can trigger cleanup before human review. Keep the essential design summary in the
-PR description: intent, important before/after behavior or API shape, compatibility, risk,
-and code references. Use commit-pinned document links so cleanup does not break access.
+The design doc is a temporary review aid. In repositories with verified post-merge cleanup,
+it remains available throughout review and is removed from `main` after merge. Keep the
+essential design summary in the PR description: intent, important before/after behavior or
+API shape, compatibility, risk, and code references. Use commit-pinned document links so
+cleanup or branch deletion does not break access.
## Workflow
@@ -156,9 +157,9 @@ and code references. Use commit-pinned document links so cleanup does not break
5. **Self-contained & offline.** One HTML file, inline CSS/SVG, no external scripts or
assets, opens by double-click, and survives being copied to another machine.
6. **`.pr/` only, and temporary.** The doc is a review aid, not project docs. Keep it in
- `.pr/` and ensure it is removed before merge. Rely on automatic cleanup only when the
- repository's workflow has been verified; otherwise remove it manually. Do not move design
- HTML into `docs/` or ship it in the merged tree.
+ `.pr/` and rely on post-merge cleanup only when the repository's workflow has been
+ verified; otherwise remove it manually before merge. Do not move design HTML into
+ `docs/` or keep it permanently in the merged tree.
## Anti-patterns
@@ -168,7 +169,7 @@ and code references. Use commit-pinned document links so cleanup does not break
location.
- ❌ Only the after-state when something changed - reviewers want the *contrast*.
- ❌ A design doc on a trivial PR - noise. Skip it.
-- ❌ Committing the HTML outside `.pr/` (e.g. `docs/`), where it would merge into `main`.
+- ❌ Committing the HTML outside `.pr/` (e.g. `docs/`), where post-merge cleanup will not remove it.
- ❌ Publishing a private-repository design doc through htmlpreview, GitHub Pages, or another
public host. Use the private/local preview path in the craft reference. Use GitHub Pages
only with explicit user authorization after verifying private Pages access control.
diff --git a/.agents/skills/pr-design-doc/references/html-craft.md b/.agents/skills/pr-design-doc/references/html-craft.md
index 58cf8c4d84..47c308694c 100644
--- a/.agents/skills/pr-design-doc/references/html-craft.md
+++ b/.agents/skills/pr-design-doc/references/html-craft.md
@@ -299,8 +299,8 @@ Example shape:
`https://htmlpreview.github.io/?https://github.com/FORK_OWNER/REPO/blob/DOC_COMMIT_SHA/.pr/design.html`
Point the URL at the **PR head repository and full commit SHA containing the document**.
-A branch-based link breaks when `.pr/` is removed after approval; a commit-pinned link
-continues to reference the document before cleanup. Refresh the document and its link when
+A branch-based link can break after post-merge cleanup or branch deletion; a commit-pinned
+link continues to reference the document before cleanup. Refresh the document and its link when
substantive changes affect the design.
For public repositories, anyone can open the rendered page without a download or local server.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index 291196afb1..edc938b55f 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -65,8 +65,8 @@ before/after of your change, then link it here via htmlpreview at the commit tha
https://htmlpreview.github.io/?https://github.com/<your-fork>/<repo>/blob/<commit-sha>/.pr/design.html
-Use the commit SHA, not the branch name: `.pr/` is removed from the branch when a
-same-repository PR is approved, and a branch link stops working then.
+Use the commit SHA, not the branch name: `.pr/` is removed automatically through
+a cleanup PR after this PR merges to `main`, and branch links may stop working.
See docs/DEVELOPMENT.md ("Design doc for non-trivial PRs") for details.
-->
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index efb1222eb6..68d380bed5 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -338,7 +338,7 @@ jobs:
{
echo "# PR Artifacts"
echo
- echo "This directory contains generated PR-only QA artifacts. The PR Artifacts workflow removes it after approval so these files do not enter the final squash merge."
+ echo "This directory contains generated PR-only QA artifacts. After the PR merges to main, the PR Artifacts workflow opens or updates a cleanup PR that removes this directory from main."
} > .pr/README.md
if [ -n "$SCREENSHOT_PATH" ]; then
diff --git a/.github/workflows/pr-artifacts.yml b/.github/workflows/pr-artifacts.yml
index c0f28f17e8..25c28b2d21 100644
--- a/.github/workflows/pr-artifacts.yml
+++ b/.github/workflows/pr-artifacts.yml
@@ -6,149 +6,8 @@ on:
types: [opened, synchronize, reopened, closed]
branches:
- main
- pull_request_review:
- types: [submitted]
jobs:
- cleanup-on-approval:
- if: github.event_name == 'pull_request_review' && github.event.review.state == 'approved'
- runs-on: ubuntu-latest
- concurrency:
- group: pr-artifacts-live-e2e-${{ github.event.pull_request.number }}
- cancel-in-progress: false
- permissions:
- contents: write
- issues: write
- pull-requests: write
- steps:
- - name: Check if fork PR
- id: check_fork
- run: |
- if [ "${{ github.event.pull_request.head.repo.full_name }}" != "${{ github.event.pull_request.base.repo.full_name }}" ]; then
- echo "is_fork=true" >> "$GITHUB_OUTPUT"
- echo "::notice::Fork PR detected - skipping automatic .pr cleanup."
- else
- echo "is_fork=false" >> "$GITHUB_OUTPUT"
- fi
-
- - name: Resolve PR context
- id: pr_context
- if: steps.check_fork.outputs.is_fork == 'false'
- env:
- GH_TOKEN: ${{ github.token }}
- run: |
- set -euo pipefail
- pr_api="repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}"
- head_ref="$(gh api "$pr_api" --jq '.head.ref')"
- head_sha="$(gh api "$pr_api" --jq '.head.sha')"
- echo "head_ref=$head_ref" >> "$GITHUB_OUTPUT"
- echo "head_sha=$head_sha" >> "$GITHUB_OUTPUT"
- if gh api "repos/${{ github.repository }}/contents/.pr?ref=$head_sha" --silent >/dev/null 2>&1; then
- echo "has_pr_dir=true" >> "$GITHUB_OUTPUT"
- else
- echo "has_pr_dir=false" >> "$GITHUB_OUTPUT"
- echo "::notice::No .pr directory exists at the approved PR head; skipping artifact cleanup."
- fi
-
- - name: Check out PR branch
- if: steps.check_fork.outputs.is_fork == 'false' && steps.pr_context.outputs.has_pr_dir == 'true'
- env:
- PR_ARTIFACT_PUSH_TOKEN: ${{ secrets.OPENHANDS_BOT_GITHUB_PAT_PUBLIC || github.token }}
- PR_HEAD_SHA: ${{ steps.pr_context.outputs.head_sha }}
- run: |
- set -euo pipefail
- git init .
- git remote add origin "https://github.com/${GITHUB_REPOSITORY}.git"
- git fetch origin "$PR_HEAD_SHA"
- git checkout --detach FETCH_HEAD
- actual_sha="$(git rev-parse HEAD)"
- if [ "$actual_sha" != "$PR_HEAD_SHA" ]; then
- echo "::error::SHA mismatch: expected $PR_HEAD_SHA, got $actual_sha"
- exit 1
- fi
- git config credential.helper '!f() { echo "username=x-access-token"; echo "password=$PR_ARTIFACT_PUSH_TOKEN"; }; f'
-
- - name: Remove .pr directory
- id: remove
- if: steps.check_fork.outputs.is_fork == 'false' && steps.pr_context.outputs.has_pr_dir == 'true'
- env:
- PR_ARTIFACT_PUSH_TOKEN: ${{ secrets.OPENHANDS_BOT_GITHUB_PAT_PUBLIC || github.token }}
- PR_HEAD_REF: ${{ steps.pr_context.outputs.head_ref }}
- PR_HEAD_SHA: ${{ steps.pr_context.outputs.head_sha }}
- run: |
- set -euo pipefail
- if [ ! -d ".pr" ]; then
- echo "removed=false" >> "$GITHUB_OUTPUT"
- echo "::notice::No .pr directory to remove."
- exit 0
- fi
-
- git config user.name "allhands-bot"
- git config user.email "[email protected]"
- git rm -rf --ignore-unmatch .pr/
- if git diff --cached --quiet; then
- echo "removed=false" >> "$GITHUB_OUTPUT"
- echo "::notice::No tracked .pr files to remove."
- exit 0
- fi
- git commit -m "chore: Remove PR-only artifacts"
- push_succeeded=false
- for attempt in 1 2 3; do
- if git push origin "HEAD:refs/heads/$PR_HEAD_REF"; then
- push_succeeded=true
- break
- fi
-
- if [ "$attempt" -lt 3 ]; then
- echo "::notice::Failed to push cleanup commit, rebasing and retrying."
- git fetch origin "$PR_HEAD_REF"
- if ! git rebase FETCH_HEAD; then
- echo "::error::Failed to rebase .pr cleanup commit. Manual resolution required."
- exit 1
- fi
- if ! git merge-base --is-ancestor "$PR_HEAD_SHA" HEAD; then
- echo "::error::Rebased .pr cleanup commit no longer descends from the approved PR head $PR_HEAD_SHA."
- exit 1
- fi
- sleep 2
- fi
- done
- if [ "$push_succeeded" != "true" ]; then
- echo "::error::Failed to push .pr cleanup commit after retries."
- exit 1
- fi
- echo "removed=true" >> "$GITHUB_OUTPUT"
-
- - name: Update PR artifacts comment
- if: steps.check_fork.outputs.is_fork == 'false' && steps.pr_context.outputs.has_pr_dir == 'true' && steps.remove.outputs.removed == 'true'
- uses: actions/github-script@v9
- with:
- script: |
- const marker = '<!-- pr-artifacts-notice -->';
- const body = [
- marker,
- '**PR Artifacts Cleaned Up**',
- '',
- 'The `.pr/` directory has been removed after approval.',
- '',
- ].join('\n');
-
- const { data: comments } = await github.rest.issues.listComments({
- owner: context.repo.owner,
- repo: context.repo.repo,
- issue_number: context.issue.number,
- });
- const existing = comments.find((comment) => comment.body.includes(marker));
-
- if (existing) {
- await github.rest.issues.updateComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- comment_id: existing.id,
- body,
- });
- }
-
check-pr-artifacts:
if: github.event_name == 'pull_request_target' && github.event.action != 'closed'
runs-on: ubuntu-latest
@@ -210,10 +69,7 @@ jobs:
return;
}
- const isFork = headRepository.full_name !== context.payload.repository.full_name;
- const cleanup = isFork
- ? 'Because this is a fork PR, the workflow will **open or update a cleanup PR against `main` after merge**.'
- : 'The directory will be **automatically removed when the PR is approved**.';
+ const cleanup = 'The directory will be **automatically removed after this PR merges to `main`**, through a cleanup PR opened or updated by this workflow.';
const body = [
marker,
'**PR Artifacts Notice**',
@@ -239,8 +95,8 @@ jobs:
});
}
- # Fork branches cannot be modified reliably. If artifacts reach the trusted base,
- # remove them through a pull request so branch protections remain enforced.
+ # Remove artifacts from all merged PRs through a pull request against the trusted
+ # base, so branch protections remain enforced without another approval-time commit.
cleanup-after-merge:
concurrency:
group: pr-artifacts-live-e2e-${{ github.event.pull_request.base.ref }}
diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md
index 60c84b8afa..e864c4d42d 100644
--- a/docs/DEVELOPMENT.md
+++ b/docs/DEVELOPMENT.md
@@ -388,10 +388,10 @@ The convention:
the code/API design and a **before/after** of your change, grounded to the actual code.
Keep it static, with no scripts: htmlpreview runs the page in the reviewer's browser.
2. Commit it under the temporary **`.pr/`** directory, e.g. `.pr/design.html`. This directory
- is for PR-only artifacts and must not land in `main`. For a same-repository PR it is
- **removed automatically when the PR is approved** (`.github/workflows/pr-artifacts.yml`).
- For a fork PR the workflow cannot push to your branch; it posts a notice and, if `.pr/`
- reaches `main` after merge, opens or updates a cleanup PR. Delete `.pr/` yourself before merge.
+ is for temporary PR-only artifacts. For both same-repository and fork PRs, it is
+ **removed automatically after the PR merges to `main`**, through a cleanup PR opened
+ or updated by `.github/workflows/pr-artifacts.yml`. Approval leaves review artifacts
+ available and avoids rerunning CI solely for their removal.
3. Link it near the top of the PR description via htmlpreview, at the commit that contains
the page (`git rev-parse HEAD` after you push it):
@@ -399,8 +399,8 @@ The convention:
https://htmlpreview.github.io/?https://github.com/<your-fork>/<repo>/blob/<commit-sha>/.pr/design.html
```
- Use the commit SHA, not the branch name. Approval cleanup removes `.pr/` from the branch,
- so a branch link stops working while a SHA link keeps resolving. Refresh the link when the
+ Use the commit SHA, not the branch name. Post-merge cleanup removes `.pr/` from `main`,
+ and PR branches may be deleted; a SHA link keeps resolving. Refresh the link when the
page changes substantively.
Keep the essentials in the PR description as well: the intent, the important before/after