3D engine that AI agents drive
Make the PR check friendlier and less wrong (#865)
Nguyen Xuan An71ca1020c9e3
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.mdindex 014f5e20..cf658c36 100644--- a/.github/pull_request_template.md+++ b/.github/pull_request_template.md@@ -25,7 +25,7 @@ Closes ### Tool schema or skill bench-<!-- Only when you change an MCP tool, its schema or a skill. Paste the table from `bench/agent-modeling/compare-bench.mjs`. Without a Claude Code login, write that and a maintainer runs it. CONTRIBUTING.md#tool-schema-and-skill-changes. Delete this section otherwise. -->+<!-- Only when you change an MCP tool, its schema or a skill. Paste the table from `bench/agent-modeling/compare-bench.mjs`. Without a Claude Code login, write "I have no Claude Code login; a maintainer runs the bench." If the tool schemas did not change, write "The tool schemas are unchanged." CONTRIBUTING.md#tool-schema-and-skill-changes. Delete this section otherwise. -->## Size or tool costdiff --git a/.github/workflows/pr-metadata.yml b/.github/workflows/pr-metadata.ymlindex 021cf145..05057f71 100644--- a/.github/workflows/pr-metadata.yml+++ b/.github/workflows/pr-metadata.yml@@ -3,8 +3,8 @@ name: PR metadata# A friendly check of the pull request description, files and labels against CONTRIBUTING.md. It is not a required check.# It runs on pull_request_target so that it can comment on pull requests from forks. For that reason it NEVER checks out,# builds or runs the pull request's code: the checkout is the base branch (the default for this event), and the pull request-# is read through the API only. Keep it that way. The hourly schedule sends the 24 hour reminder and closes pull requests-# that stayed unfixed for 72 hours. How it works and how to test it: CONTRIBUTING.md#the-pull-request-check.+# is read through the API only. Keep it that way. The hourly schedule sends the 5 day reminder and closes pull requests+# that stayed unfixed for 7 days. How it works and how to test it: CONTRIBUTING.md#the-pull-request-check.on:pull_request_target:types: [opened, edited, synchronize, reopened, labeled, unlabeled, milestoned, demilestoned, ready_for_review]@@ -23,7 +23,7 @@ permissions:issues: writeconcurrency:- group: pr-metadata-${{ github.event.pull_request.number || 'sweep' }}+ group: pr-metadatacancel-in-progress: falsejobs:diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex ff0a8f7f..ec6297db 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -39,7 +39,7 @@ The `Test` workflow runs this check on every pull request, so `git push --no-ver### Describe how you checked it-Under **How I checked it**, paste the commands you ran and what they printed. A line such as `npm run preflight: 14 checks ok` is enough. This applies to agents too: show the commands and their results.+Under **How I checked it** (or **Ran**, **Checks**, **Tests**, **Verification**), paste the commands you ran and what they printed. A line such as `npm run preflight: 14 checks ok` is enough. This applies to agents too: show the commands and their results.### Keep private data out@@ -89,7 +89,7 @@ node bench/agent-modeling/bench.mjs --models haiku --runs 3 --label <branch> --bnode bench/agent-modeling/compare-bench.mjs bench/agent-modeling/results/bench-baseline.json bench/agent-modeling/results/bench-<branch>.json```-Paste the table into the pull request. The bench runs through a Claude Code login. Without one, say so in the pull request and a maintainer runs it. [bench/agent-modeling/README.md](bench/agent-modeling/README.md) has the details.+Paste the table into the pull request. The bench runs through a Claude Code login. Without one, write "I have no Claude Code login; a maintainer runs the bench." in the pull request and a maintainer runs it. If the tool schemas did not change, write "The tool schemas are unchanged." Only `skills/` and `scripts/mcp-server.mjs` trigger this check. [bench/agent-modeling/README.md](bench/agent-modeling/README.md) has the details.### Look changes@@ -127,8 +127,8 @@ The `PR metadata` workflow checks each pull request against this guide. It postsDocs-only, CI-only and test-only pull requests skip everything except the local path check.-- The comment updates itself when you fix an item. One reminder follows when 24 hours are left.-- If an item is still open 72 hours after the first comment, the bot closes the pull request. It invites you to reopen it once the item is fixed, and reopening starts the time again.+- The comment updates itself when you fix an item. One reminder follows after 5 days.+- A maintainer gives a first answer within about 7 days. That reply time is separate from the bot clock, which counts only items on your side. If an item is still open 7 days after the first comment (the end of that UTC day, weekends included), the bot closes the pull request. It invites you to reopen it once the item is fixed, and reopening starts the time again.- Pull requests from the owner, members, collaborators and bots never close by themselves. Drafts and pull requests labelled `needs-owner-decision` do not close either.- Labels and the milestone never count against you. The bot adds `needs-triage` when they are missing and `status:needs-author` while something of yours is missing.diff --git a/scripts/pr-metadata-check.mjs b/scripts/pr-metadata-check.mjsindex 6e085ca4..601b5d2a 100644--- a/scripts/pr-metadata-check.mjs+++ b/scripts/pr-metadata-check.mjs@@ -16,7 +16,9 @@ const OWNER_LOGIN = 'anxuanng';const TRUSTED_ASSOCIATIONS = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);const shippedPathPattern = /^(src|bin|packages|skills)\//;-const toolOrSkillPathPattern = /^(src\/(agent-|mcp)|bin\/|skills\/)/;+// Only skills and the file that defines the MCP tool schemas need a bench table. Detail text in src/agent-scene-capabilities.js+// is not a schema, and the tool cost gate already covers schema growth.+const toolOrSkillPathPattern = /^(skills\/|scripts\/mcp-server\.mjs$)/;const lightPathPattern = /^(docs\/|\.github\/|tests\/|changes\/)|\.md$/;// A look change touches shaders, lighting, materials, the default render or a post step. A name part must stand alone, so// "flight" and "highlight" do not match "light".@@ -30,7 +32,13 @@ const localPathPatterns = [/(?:^|[\s`"'(=])\/Users\/[A-Za-z0-9._-]+/,];const imagePattern = /!\[[^\]]*\]\([^)]+\)|<img\b|github\.com\/user-attachments\//i;-const benchOptOutPattern = /(?:no|without)\b[^.\n]*claude code login|maintainer\b[^.\n]*\b(?:run|runs)\b[^.\n]*\bbench/i;+// Accepts the natural ways to say "I cannot run the bench": no login, no access, cannot run it, a maintainer runs it.+const benchOptOutPattern = /(?:\bno|\bwithout|not have|n't have|\bcannot|can't|can not|\bunable to)\b[^.\n]*\b(?:claude code|bench)\b|\bmaintainers?\b[^.\n]*\b(?:run|runs|running)\b[^.\n]*\bbench|\bbench\b[^.\n]*\bmaintainers?\b[^.\n]*\brun/i;+// A body that says the tool schemas did not change needs no bench table.+const schemasUnchangedPattern = /\bschemas?\b[^.\n]*\b(?:unchanged|not changed|did not change|do not change|don't change|are the same|stay the same)\b|\bno (?:tool )?schema changes?\b|\bunchanged\b[^.\n]*\bschemas?\b/i;+const checkedHeadings = ['How I checked it', 'Ran', 'Checks', 'Tests', 'Verification', 'How I tested it'];+const fencedBlockPattern = /```/;+const commandLinePattern = /^\s*(?:[$>]\s*)?(?:npm|node|npx)\s+\S/m;function stripHtmlComments(text) {return text.replace(/<!--[\s\S]*?-->/g, '');@@ -60,6 +68,12 @@ function hasContent(sectionText) {return meaningful.length >= 10;}+// The checks are described when a section under an equivalent heading has content, or the body shows a fenced block or a command line.+function describesChecks(body, visibleBody) {+ if (checkedHeadings.some(heading => hasContent(sectionOf(body, heading)))) return true;+ return fencedBlockPattern.test(visibleBody) || commandLinePattern.test(visibleBody);+}+function normalizePath(path) {return path.replace(/\\/g, '/');}@@ -110,10 +124,10 @@ function contributorFixesFor(pr, files, light) {'link-the-issue',));}- if (!hasContent(sectionOf(body, 'How I checked it'))) {+ if (!describesChecks(body, visibleBody)) {fixes.push(fix('empty-how-checked',- 'Under **How I checked it**, paste the commands you ran and what they printed, such as `npm run preflight` and its last lines.',+ 'Under **How I checked it** (or **Ran**), paste the commands you ran and what they printed, such as `npm run preflight` and its last lines.','describe-how-you-checked-it',));}@@ -131,11 +145,11 @@ function contributorFixesFor(pr, files, light) {const touchesToolOrSkill = paths.some(path => toolOrSkillPathPattern.test(path));const hasBenchTable = /^\s*\|\s*:?-{3,}/m.test(visibleBody) && /bench|haiku|sonnet/i.test(visibleBody);- const benchOptOut = benchOptOutPattern.test(visibleBody);+ const benchOptOut = benchOptOutPattern.test(visibleBody) || schemasUnchangedPattern.test(visibleBody);if (touchesToolOrSkill && !hasBenchTable && !benchOptOut) {fixes.push(fix('missing-bench-table',- 'This changes an MCP tool or a skill, so paste the agent bench table in the description. Without a Claude Code login, say so and a maintainer will run it.',+ 'This changes an MCP tool or a skill, so paste the agent bench table in the description. Without a Claude Code login, write "I have no Claude Code login; a maintainer runs the bench." and a maintainer will run it. If the tool schemas did not change, write "The tool schemas are unchanged."','tool-schema-and-skill-changes',));}diff --git a/scripts/pr-metadata-flow.mjs b/scripts/pr-metadata-flow.mjsindex a82f6bd1..ce855ce2 100644--- a/scripts/pr-metadata-flow.mjs+++ b/scripts/pr-metadata-flow.mjs@@ -1,14 +1,14 @@// The time rules and the wording of the `PR metadata` bot. Pure, so tests call it without GitHub.// The bot keeps one status comment per pull request and finds it by a hidden marker. The marker also holds the time of the-// first failing comment, which starts the 72 hour clock, and the time of the one reminder.+// first failing comment, which starts the 7 day clock, and the time of the one reminder.import { NEEDS_OWNER_DECISION_LABEL } from './pr-metadata-check.mjs';export const STATUS_MARKER = 'pr-metadata-status';export const REMINDER_MARKER = 'pr-metadata-reminder';export const CLOSE_MARKER = 'pr-metadata-closed';export const HOUR_MS = 60 * 60 * 1000;-export const REMINDER_AFTER_HOURS = 48;-export const CLOSE_AFTER_HOURS = 72;+export const REMINDER_AFTER_HOURS = 120;+export const CLOSE_AFTER_HOURS = 168;const guideUrl = 'https://github.com/codepawl/67.js/blob/main/CONTRIBUTING.md';@@ -49,7 +49,7 @@ export function renderStatusComment({ contributorFixes, firstFailedAt, reminderSconst closeDate = formatDate(Date.parse(firstFailedAt) + CLOSE_AFTER_HOURS * HOUR_MS);const deadline = exempt? 'This is a maintainer pull request, so it will not close by itself.'- : `If these are still open on ${closeDate}, we close the pull request so the queue stays clear. You can reopen it any time after you fix them.`;+ : `There is no rush. A maintainer gives a first answer within about 7 days, and the clock counts only the items above, which are on your side. If they are still open at the end of ${closeDate} (UTC, weekends count), we close the pull request so the queue stays clear. You can reopen it any time after you fix them.`;return [marker,'Thank you for the pull request. A few small things are missing before a maintainer can review it:',@@ -65,7 +65,7 @@ export function renderStatusComment({ contributorFixes, firstFailedAt, reminderSexport function renderReminderComment({ contributorFixes, closeDate }) {return [buildMarker(REMINDER_MARKER),- `A small reminder: these items are still open, and we close the pull request on ${closeDate} if they stay that way.`,+ `A small reminder: these items are still open, and we close the pull request at the end of ${closeDate} (UTC) if they stay that way.`,'',bulletsOf(contributorFixes),'',@@ -76,7 +76,7 @@ export function renderReminderComment({ contributorFixes, closeDate }) {export function renderCloseComment() {return [buildMarker(CLOSE_MARKER),- 'Thank you for the work on this. We are closing the pull request because the items above stayed open for 3 days. Nothing is lost.',+ 'Thank you for the work on this. We are closing the pull request because the items above stayed open for 7 days. Nothing is lost and there is no rush: reopen it whenever you are ready.','',`Once they are fixed, reopen it and the check runs again. Details are in the [contributing guide](${guideUrl}).`,].join('\n');diff --git a/tests/pr-metadata-check.test.js b/tests/pr-metadata-check.test.jsindex 4b6743ad..1aa543f8 100644--- a/tests/pr-metadata-check.test.js+++ b/tests/pr-metadata-check.test.js@@ -94,7 +94,7 @@ test('a local path in the body is a contributor fix, for every kind of path', ()});test('a tool schema or skill change needs a bench table, or a note that there is no Claude Code login', () => {- const files = [file('src/agent-tools.js'), file('changes/12.md')];+ const files = [file('scripts/mcp-server.mjs'), file('changes/12.md')];assert.deepEqual(fixIds({ pr: pullRequest(), files }), ['missing-bench-table']);const table = `${goodBody}\n| task | before | after |\n| --- | --- | --- |\n| haiku bench | 20/28 | 22/28 |\n`;assert.deepEqual(fixIds({ pr: pullRequest({ body: table }), files }), []);@@ -105,6 +105,37 @@ test('a tool schema or skill change needs a bench table, or a note that there isassert.deepEqual(fixIds({ pr: pullRequest(), files: [file('skills/x/SKILL.md'), file('changes/12.md')] }), ['missing-bench-table']);});+test('natural ways to say there is no bench run are accepted', () => {+ const files = [file('scripts/mcp-server.mjs'), file('changes/12.md')];+ const sentences = [+ 'I have no Claude Code login; a maintainer runs the bench.',+ 'I do not have a Claude Code login, please run the bench.',+ "I don't have a Claude Code login.",+ 'I cannot run the bench, a maintainer can run it.',+ "I can't run the bench.",+ 'No access to Claude Code.',+ 'A maintainer can run the bench for me.',+ 'The tool schemas are unchanged.',+ ];+ for (const sentence of sentences) {+ const body = [goodBody, sentence, ''].join("\n");+ assert.deepEqual(fixIds({ pr: pullRequest({ body }), files }), [], sentence);+ }+});++test('detail text in src/agent-scene-capabilities.js alone needs no bench table', () => {+ const files = [file('src/agent-scene-capabilities.js'), file('changes/12.md')];+ assert.deepEqual(fixIds({ pr: pullRequest(), files }), []);+});++test('a section named Ran, or a fenced block, counts as how I checked it', () => {+ const ran = ['## What changed', '', 'Adds a thing.', '', 'Closes #12', '', '## Ran', '', 'npm run preflight passed, 14 checks ok.', ''].join('\n');+ assert.deepEqual(fixIds({ pr: pullRequest({ body: ran }), files: shippedFiles }), []);+ const fence = '`'.repeat(3);+ const fenced = ['## What changed', '', 'Closes #12', '', fence, 'node --test tests/a.test.js', '# pass 4', fence, ''].join('\n');+ assert.deepEqual(fixIds({ pr: pullRequest({ body: fenced }), files: shippedFiles }), []);+});+test('look paths are detected by whole name parts', () => {for (const path of ['src/shaders.js', 'src/lighting.js', 'src/material-pbr.js', 'src/environment.js', 'src/post/bloom.js', 'packages/a/fog.js', 'src/shadow-map.glsl.js']) {assert.ok(isLookPath(path), path);@@ -172,7 +203,7 @@ test('sectionOf reads a section up to the next heading', () => {const someFix = [{ id: 'x', message: 'Fix x.', docsAnchor: 'link-the-issue' }];-test('the clock starts at the first failure, reminds once at 48 hours and closes at 72', () => {+test('the clock starts at the first failure, reminds once at 5 days and closes at 7', () => {const start = Date.parse('2026-11-01T00:00:00.000Z');const base = { contributorFixes: someFix, exempt: false, draft: false, labels: [] };const first = decideActions({ ...base, now: start, previousState: null });@@ -183,8 +214,8 @@ test('the clock starts at the first failure, reminds once at 48 hours and closesassert.ok(!early.remind);const reminder = decideActions({ ...base, now: start + REMINDER_AFTER_HOURS * HOUR_MS, previousState: state });assert.ok(reminder.remind && !reminder.close);- assert.equal(reminder.closeDate, '2026-11-04');- const again = decideActions({ ...base, now: start + 50 * HOUR_MS, previousState: reminder.statusState });+ assert.equal(reminder.closeDate, '2026-11-08');+ const again = decideActions({ ...base, now: start + (REMINDER_AFTER_HOURS + 2) * HOUR_MS, previousState: reminder.statusState });assert.ok(!again.remind);const closing = decideActions({ ...base, now: start + CLOSE_AFTER_HOURS * HOUR_MS, previousState: reminder.statusState });assert.ok(closing.close);@@ -220,7 +251,8 @@ test('the marker round trips and the comment stays warm and short', () => {assert.match(comment, /^<!-- pr-metadata-status firstFailedAt=2026-11-01T00:00:00.000Z -->/);assert.match(comment, /Thank you/);assert.match(comment, /CONTRIBUTING\.md#link-the-issue/);- assert.match(comment, /2026-11-04/);+ assert.match(comment, /2026-11-08/);+ assert.match(comment, /first answer within about 7 days/);});function fakeGitHub({ pullRequest: pr, files = [], comments = [], events = [] }) {@@ -271,10 +303,10 @@ test('the runner posts one friendly comment for a new pull request and labels itassert.ok(!names.includes('closePullRequest'));});-test('the runner closes a pull request that stayed unfixed for 72 hours, and never an owner one', async () => {+test('the runner closes a pull request that stayed unfixed for 7 days, and never an owner one', async () => {const marker = buildMarker('pr-metadata-status', { firstFailedAt: '2026-11-01T00:00:00.000Z' });const comments = [{ id: 9, user: { login: 'github-actions[bot]' }, body: `${marker}\nold` }];- const now = Date.parse('2026-11-05T00:00:00Z');+ const now = Date.parse('2026-11-09T00:00:00Z');const contributor = fakeGitHub({ pullRequest: apiPullRequest(), files: [{ filename: 'src/scene.js' }], comments });await checkOnePullRequest({ github: contributor, repo: {}, number: 5, now, reopened: false, dryRun: false, log() {} });assert.ok(contributor.calls.some(call => call.name === 'closePullRequest'));