Zulip server and web application. Open-source team chat that helps teams stay productive and focused.
commit-message: Add evals for the skill.
Shubham Padiafef1953099c8
diff --git a/.claude/skills/commit-message/evals/README.md b/.claude/skills/commit-message/evals/README.mdnew file mode 100644index 00000000000..d6b034aa88c--- /dev/null+++ b/.claude/skills/commit-message/evals/README.md@@ -0,0 +1,175 @@+# Evals for the commit-message skill++These evals check whether a change to `../SKILL.md` makes agents+write better commit messages. Run them before changing the skill's+wording, and when a new model changes how agents write.++The cases are in `cases.txt`: 100 commits by Tim Abbott and Lauryn+Menard, written before they used AI tools. Half are from each+author. They cover bug fixes, refactors, docs, and migrations, and+range from no body to several paragraphs. The file says how they+were chosen.++There is no runner script here. This file gives the procedure and+both prompts, and an agent can write a short script from it in a few+minutes.++## Procedure++For each case, a writer model gets the skill and the commit's diff,+and writes a commit message. A judge model then scores that message+from 1 to 10 against the message the maintainer wrote.++1. Get the maintainer's message with `git log -1 --format=%B <hash>`.+2. Build the case's context, as four parts separated by blank lines:+ - If the maintainer's message has a `Fixes #123.` line: "This+ change resolves GitHub issue #123: {the issue's title}."+ - "The author's previous commits, newest first:" and the summary+ lines of the same author's three commits before this one.+ - "The author's next commits, oldest first:" and the summary lines+ of the same author's next three commits.+ - "The diff:" and the output of+ `git show --format= --stat --patch <hash>`.+3. Call the writer with the writer prompt below.+4. Call the judge with the judge prompt below as the system prompt.+ The user message is the context in `<change>` tags, the+ maintainer's message in `<human_message>` tags, and the writer's+ message in `<candidate_message>` tags. The reply ends with the+ score.++Each call must be a fresh model with no tools and none of this+repository's instructions or skills loaded. Otherwise the writer+follows the checked-in skill, not the one being tested.++Save every reply under `var/`, which Git ignores, so that a rerun+only does missing work. Read the saved messages, not only the+scores.++The writer sees only the diff, so it can't know why a change was+made unless the code shows it. The judge is told not to mark a+message down for that.++## Writer prompt++Replace `{skill}` with the full text of the skill file being tested,+and `{context}` with the case's context.++The system prompt:++```+You write git commit messages for the Zulip project.+Follow these instructions:++{skill}+```++The user message:++```+You just made the change below in the Zulip repository.+Write its commit message. Reply with only the commit message.++{context}+```++## Judge prompt++```+You are a Zulip maintainer with high standards for commit messages.+You get a change (diff and context), the message its human author+actually wrote, and a candidate message for the same change.++The human message shows the house style: how much to say, in what+order, and what to leave out. Don't punish the candidate for missing+a fact that only the author could know. Do punish it for:++- missing the change's main point, or its reason when that is known;+- inventing reasons or effects;+- opening with a function name instead of the idea in plain words;+- repeating the summary line, or re-explaining it in the body;+- details the reader can see in the code, such as parameters or+ tests updated to pass. Naming a function, file, or setting is fine+ when the change is about it (a rename, a move, a new helper, an+ API field) or when it explains the bug;+- terms that add nothing, like "attribute", or internal names;+- any word the message doesn't need. Wordiness is a real cost.++Score the candidate from 1 to 10:++- 10: as good as the human message; merge as is.+- 8: one or two small wording changes.+- 6: rewrite a sentence or two.+- 4: rewrite most of it.+- 2: start over.++Use odd numbers for in-between cases. Name the main problem in one+sentence, then on the last line give a JSON object like {"score": 7}.+```++## What to report++- The mean score.+- The median length: the words in the writer's message divided by+ the words in the maintainer's. Count the summary line in both.+- How often the writer added a body where the maintainer wrote none,+ and how often it left one out.+- Format problems: a summary over 72 characters or without the+ `area: Summary.` form, a body line over 70 characters, a dropped+ `Fixes #123.` line, or a trailer line.++## Reading the results++- Always run the old and new skill with the same models. Scores from+ different writer or judge models can't be compared.+- With one message per case, a gap under 0.3 on all the cases is+ noise. Write three messages per case on a small set to see how+ much one case varies.+- Start with 15 to 20 cases that your change should affect, plus a+ few it shouldn't. Run all 100 only for a change that looks good.+ A full run is 200 model calls.+- Watch the length. A higher score with much longer messages is+ usually worse.+- The judge is a rough guide. It can miss problems a person would+ catch, such as an invented term. See "Checking by hand".++## Checking by hand++The judge's score alone is not enough to accept a change. Have a+person who knows Zulip's commit style check a sample:++1. Pick 10 cases, spread over both authors and all body lengths.+2. Show the person one case at a time: the maintainer's message+ and the writer's message.+3. Ask for a score from 1 (would rewrite it) to 5 (would merge it+ as is), and what is wrong with the message.+4. Save the comments word for word. They show what to change in+ the skill better than any score does.+5. Compare the scores with the judge's. If the judge ranks the+ messages differently, fix the judge prompt before trusting it.++Use new cases for each round.++## Results so far++These used `claude-opus-5-5` as the writer, `claude-fable-5-1` as+the judge, and one message per case, on all 100 cases.++| Skill | Score | Length |+| ------------------ | ----- | ------ |+| Before these evals | 5.7 | 3.1x |+| Current | 7.5 | 1.3x |++Things the evals showed:++- Without "Prefer a sentence or two", short messages grew to two or+ three times the maintainer's length.+- A skill tuned with one model may not suit another. Run the evals+ again before using the skill with a different model.++## Known limits++- The skill writes a short body for most small commits where the+ maintainer wrote only a summary. Rewording the skill did not+ reduce this.+- About 1 message in 25 has a body that is not wrapped at all.+ gitlint catches these.diff --git a/.claude/skills/commit-message/evals/cases.txt b/.claude/skills/commit-message/evals/cases.txtnew file mode 100644index 00000000000..e61ceba396e--- /dev/null+++ b/.claude/skills/commit-message/evals/cases.txt@@ -0,0 +1,155 @@+# The commits used as eval cases. See README.md.+#+# How they were chosen:+#+# - The target style is how Tim Abbott and Lauryn Menard write, with+# no AI help. Claude Code launched in February 2025, and the first+# commit here with a Claude co-author line is from February 2026.+# So the cases come from commits that landed in 2023 and 2024:+# 1,558 non-merge commits, 654 by Tim and 904 by Lauryn.+# - All cases were picked at random within fixed groups, so that the+# set covers the kinds of commit a skill must handle.+# - 80 cases are grouped by author and body length. They come from+# commits that change 3 to 700 lines and have no upkeep word in+# the summary. Each author has 8 with no body, 12 with 1-3 lines,+# 14 with 4-9 lines, and 6 with 10 or more. About 30% of their+# commits have no body, so a good skill must also know when to+# write none.+# - 10 cases have a word in the summary that often marks routine+# upkeep, such as "typo", "version", "upgrade", or "changelog".+# - 5 cases change fewer than 3 lines, and 5 change more than 700.+#+# The cases were not picked by hand or by type of change. The mix of+# bug fixes, refactors, docs, and migrations is what random sampling+# gave.+#+# The no-body and small cases are the hardest. With only the diff to+# go on, the current skill still writes a short body for most of+# them. See "Known limits" in README.md.+#+# Format: <hash> # <commit summary>+#+# Summaries are copied exactly, typos included, so+# tools/run-codespell skips this file.++# Tim Abbott, no body+798391824b4f0e6d0c160f27bcaf536727d03f32 # compose: Refactor dropdown-menu CSS rules.+9e5c1eff4b93c66fcc67eb053c398aa205fd4b95 # scheduled messages: Fix incorrect mapping of 4PM to 16:00 hours.+2ce3f520304292425bb832b17d663ad7b330ed73 # gear_menu: Remove logout button, moved to new personal menu.+807de37561349cc60304a3bd6700057c3a40271f # analytics: Fix misplaced zilencer import statement.+95f4b31ce73a8c3c015407b020cbafef0b050d25 # billing: Don't require data to add to legacy plan.+4395c54c23dfb517e3ce904c54ee8d67d15fdfce # css: Remove unused Bootstrap CSS for nav elements.+563485a37eebdb3ff92734f912a0b744ae7f427d # zulip_updates: Fix line removal logic for bulleted lists.+1cd11a4c61e52452bc0081a84961cd06c2ec17e5 # zulip_updates: Adjust formatting for initial updates.++# Tim Abbott, body of 1-3 lines+0232d97ae91c4058df3fe273eee9bc27bcba08ae # actions: Tag resolve topic messages with a message type.+2b6ec56238f459341a8ed975205a52fe1d856f83 # popovers: Use default our font size for Tippy popovers.+1c077561504b0a20c6168f0c085634255ae852ba # scheduled messages: Use data-scheduled-message-id.+859a96e0d7fc8dfc25d0b2246364b273c085308e # message_store: Keep track of wildcard_mentioned.+59daf54ec95d9c424a68d289e3a3837f1685789c # scheduled_messages: Fix broken node test.+a379fbe4424c51ef2587d29c94b1b6f91793de45 # portico: Inline common.autofocus.+03f3e17a40330b0413a1f6b873e6fa0b273dcf32 # narrow: Clarify first parameter to deactivate.+e32468e97fc5ecc88f5ff3d419e22c342f19692a # compose: Simplify convert_mentions_to_silent_in_direct_messages.+d0c276d8632b979be0eda213d67b6e086073a2dc # corporate: Fix billing_session variable reuse confusion.+f37082c4caeba1441711231e6abcac2f9079f90c # models: Switch font size default to non-compact.+858b93f588d96495a218d2826e1730082e9e822d # settings_config: Allow using new user list style in production.+9a83d211127b900299a8e180407cf6167eeae03d # settings_users: Recalculate user ID lists after presence fetch.++# Tim Abbott, body of 4-9 lines+56c5c8902b5bbb9d1abcf144c2ad8d69a22fd7bc # scheduled_messages: Disallow dates in the past in the UI.+18e2cee572eff958956eb17880ff3946deda57e2 # typeahead: Do not prioritize exact name matches for users.+804f4732140f3a3c2e4d318dce1e337d6f035a8f # message_list: Extract visibly_empty helper method.+396cedd0e83f8802ac1f16475a975f4b84dc84ad # ci: Reorder tests to run unique tests first.+6cdf3d638b5c73f79f927468c04ef4f425dcf1a0 # timerender: Reuse Intl.DateTimeFormat objects.+e560519dba6d954b3286ea2957b052375d34e2ef # portico: Fix new organization form focus scrolling.+7db15176f3f013b7927cd86b940bc3d6d1066e45 # push bouncer: Submit basic metadata unconditionally.+bafa476fd3f26f093ba4b27c61d4864ac6df5329 # presence: Avoid having incomplete missing test coverage.+d8ec141de28b36eae046b69c48c7dbb8d788570b # navigate: Fix buggy detection of start/end.+652fea9bdfc32a69aed0a79b6a520648b6970f40 # narrow: Clarify some confusing details.+5f4e1de661fa8295d4687898c2029639c9e9502f # util: Fix call_function_periodically.+4ae16b7568e684e29731b0ccfa6101cab067dc1c # stream_list: Fix handling of unsubscribing from last channel.+eaa02a10a47d60ffeea7347deb980a74dd99f046 # migrations: Squash most confirmation migrations.+30c54d023b57926e9311da79a51f32f54a549809 # settings: Write fewer values into template settings files.++# Tim Abbott, body of 10 or more lines+7425079814d674672b5e1208f0742f800ba470c7 # topic_list: Show muted unread counts in muted streams.+1ce17f74036fc5468c1e4188c85881bad759a018 # message_list: Fix incorrect usage of visible empty check.+d24e81e1b24aca979055b23178a4233611d5ac20 # unread: Fix exception handling mentions in private messages.+22d59f113260fa923fa694008956ea0057d9944d # typing notifications: Increase various time constants.+b9af5ce86e0c3e2fe5b1e04d7adff37182903f3a # unread: Fix process_visible race with fetching.+525c3d4321c5496c32a667087c3bd35450e32ae7 # migrations: Squash corporate migrations.++# Lauryn Menard, no body+dd59e83d54947865d9efbfb7f3a19abc412ba615 # welcome-emails: Make some code comments and docstrings more evergreen.+3d8090a116fc124cf20d9a359fb2c53f601843a2 # sentry-webhook: Revise documentation page to be clearer.+03ba4df8ee96d5280509d6bf4d2c9a6c50c8b96c # api-docs: Add 'code' field to unauthorized error in delete attachments.+1b14745b1772f668302d9c0fb3076a0aa3d0da24 # api-docs: Update InvalidMessageError and use for read receipts doc.+b6d50c158d0721206ad1e6e7ab04bc62b1756658 # support: Add CSS grid for confirmation objects in query results.+4dbda25e2e5e9d8aea1460c200f35680b0a770de # help: Rename and redirect streams-and-topics for channel.+265f85fa9583d2465284248de36c714b4790b260 # python-examples: Remove unused result variable assigments.+1e7f2fe6dd9d13663699528005d206e278b9b2c7 # docs: Add missing links to web-public channels in dev community.++# Lauryn Menard, body of 1-3 lines+d98ba32588598412f127c28ab05c220aa9bdd637 # rendered-markdown: Copy anchor tag CSS rules from bootstrap.+87e948267276f85906df6cb9fd4581bb9ad5d412 # api-docs: Update "Zulip display email" to "Zulip API email".+4b2b278a53dbd108d52f9a737e3326b68b53a526 # dm-conversations: Update direct message data structure name.+83d6481f6f491c74899e56647f5cf0ea08220a9b # support: Add the remote realm's plan type to the support view.+4e5c62527cfbc446835983ae9e9633f03da57466 # activity: Update installation total row for moving referrer column.+3c11872ebd77736054ec29998e66956a048311da # api-docs: Update descriptions in POST /drafts for channel.+016ffd8078320121a13694fbebda1dfcdc90c104 # api-docs: Update python tests to not use "stream" in channel name.+03b0a8785af2dd1554cd01e24cd2d0306bd644ae # api-docs: Update GET /messages/{message_id}/history for channel.+c3577d9cdda0d01a53b79a8c724f4bfe881175b3 # api-docs: Update python example user-facing comments for channel.+bd9a10cfbb4ad2adf152bda9bc61fbbd60fc00d6 # integration-docs: Update Jira for new doc format.+95b74a7bfbc85b0a5b29c6f574b0016d6a8ddf40 # integration-docs: Update Intercom for new doc format.+53e7fc38d68565f5a4aeef79676e36733d87724e # help: Use screenshots for global times examples.++# Lauryn Menard, body of 4-9 lines+665deac7c0c88583164ed44dd349049381e7aa31 # demo-orgs: Disable show/change API key if email not configured.+d8b8b34ecd209dbda03d4f15c63caa885c1e016b # profile-settings: Change `name-input` class to a more specific id.+42d9560413c50a116ee5567240a6ae927dacae44 # message: Use `recipient_type_name` for API message type references.+e68a69e91f99e09d91757ea71de30448111c5426 # api-docs: Update docs to reference max lengths in register response.+957382253afa76a297e9da638d58ba19a3a8fb39 # scheduled-messages: Split out edit scheduled message endpoint.+a4d1211ec6928b0a126e905e218c2c56f336df7b # integrations: Reformat Github pull request assigned message body.+39bfab0ec4bfd5aa2de0defac669f792518f2761 # corporate: Move plan tier change to separate helper function.+b167eeff08592109425c819574a7d5dc167f26f8 # corporate: Create change plan tier function for BillingSession.+1249b5929e4cc18a97ed2c7066308e9a146b68a7 # remote-activity: Update column data for remote realms.+9c7d0c6e5a7e0f855834807b7489eb05d9681485 # remote-support: Note if emails were sent for remote server sponsorship.+5892e48ba4a92bfb014c5791de675ee43fbbd7a0 # analytics: Update "Messages sent by client" chart for Flutter app.+206531f703bb4acc5af38c3630cb8f0f90c53aa2 # help: Fix details in "Change your call provider" section.+7861c1ba637615efe81c03b7d377962028ebf492 # billing: Enforce manual billing renewal licenses for new users.+3ebc507ddba49029f2ee476301b7bfe99677c3a8 # signups: Move logic for realm admin notification to corporate app.++# Lauryn Menard, body of 10 or more lines+c2bcfb52aa7987cfc91c6f7f90a1baeca27203b7 # api-tests: Reduce error output for `/register` openapi validation.+d3a513f4bd4637ab6e27bdb435f1c2b0c6adb826 # image-upload-widget: Render `image_disabled` div if text to display.+7abf476443f864772685021c7390ab4c027b4692 # settings_account: Update avatar widget if user/org permissions changed.+213d0f4990ed79401d23f235158ead93bbd48320 # create-user: Remove notifications sent to admin realm.+c8a4b401c7585ddb85bbf9900bf9434e1d4a1da5 # unsubscribe-success: Update template for org name and welcome emails.+d41f12495dafc3f259ab220939c9845f9cd42afe # api-docs: Document create/edit scheduled message endpoint and events.++# Upkeep word in the summary+5e5d151c0c15ba633e4622fdc6298ba002a26796 # docs: Tweak a few details on the release checklist.+2a59edda53214d5789ca6f2cc170e88ba78dd461 # version: Update version and changelog following 8.2 release.+bcff5580d1584c194e14e03f6c89c307e363d8d2 # makemessages: Fix handling of handlebars whitespace control.+e64b67525f5069da4ca60478e967c86eb17c07b6 # docs: Update changelog with recently merged changes.+43c0c776103fbc02cf0f0bdf5c74604e33bf1afd # puppet: Update rules for number of uwsgi processes.+227645ab65d47556eba4638b5c5fdce6683264f7 # corporate: Update setup upgrade checkout to only get price per license.+2c606ed99ede4096da63036679a8a5b37b63fcec # api-changelog: Revise feature level 164 entry.+71263ac2abfc294798f0b2baa7d14c3e89e67c05 # support: Include legacy plan without upgrade in remote support tests.+b86654d10f8c74d8bcb5267613234bec5e4f3bf3 # analytics: Add column for Zulip version in remote installations page.+0019ce639ca75a506b282ca8399527b40f55fffc # api-docs: Fix small typos and copy/paste errors in API changelog.++# Fewer than 3 changed lines+3dfc24a908ffa30ead93a8698a4ced8c13111bd5 # test_push_notifications: Fix sort order assumption.+6c981d938590c0029ce4975fc352967fdd9e749c # custom_profile_fields: Remove unused generic title CSS class.+f0fe86d5e325ddd9bc34ce66c11c0a7c9bf29803 # api-docs: Move simple success example up to JsonSuccess schema.+3d5c72f7627ccd6622a571db9626d87993c965bd # invite-modal: Remove "expires_in" for attribute for custom time input.+d8fd0d85991b6797dd0302f19519f0f235857039 # scheduled-messages-ui: Set channel ID in narrow term to a string.++# More than 700 changed lines+72942c6bb9be6f4cd765a95744ce3c8faaff549c # portico: Delete now-unused tour,carousel,testimonials CSS.+c0708a892a5a618383f4e27d7d8c7609708f2744 # bootstrap: Remove tooltip/popovers JS modules.+0f7341dd48139339f39b15683f090326b0300974 # narrow: Add frontend support for `dm:` narrow and `/dm/...` URL.+df2f4b6469fa103430939ffa67b8bd18e19619ce # corporate: Move support and activity views to /corporate.+f4d98b03d5ef3372aa295bc8eb928e7f17ac2344 # migrations: Squash zerver migrations.diff --git a/tools/run-codespell b/tools/run-codespellindex 2e21396ced6..cf0b9aa61eb 100755--- a/tools/run-codespell+++ b/tools/run-codespell@@ -11,6 +11,7 @@ if [ "$#" -eq 0 ]; thenecho "No argument specified. Checking on the entire repo..."FILES_TO_BE_CHECKED="$(git ls-files -- \+ ':!.claude/skills/commit-message/evals/cases.txt' \':!*.asc' \':!*.svg' \':!*/fixtures/*' \