Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
Install ratchet-files skill and hook
David Zhangee4fb1de3343
diff --git a/.agents/skills/ratchet-files/SKILL.md b/.agents/skills/ratchet-files/SKILL.mdnew file mode 100644index 0000000..3d73da2--- /dev/null+++ b/.agents/skills/ratchet-files/SKILL.md@@ -0,0 +1,106 @@+---+name: ratchet-files+description: Ratchet source file size so every file stays small enough for an agent to read whole: find every file over a token budget, split oversized files by concept, and set up the pre-commit ratchet that stops them from growing. Use when a commit is blocked because a file is over the size budget, when asked to find or split oversized/long/god files, when a file you need to change is too large to read in one pass, or when asked to add a file-size limit or hook to a repository.+---++# Ratchet Files++A source file is over budget when it is past what an agent can read in one+pass (8,000 tokens, counted as bytes / 4, unless the repository sets+another). [`scripts/check-file-size.mjs`](scripts/check-file-size.mjs) owns+what counts as source, the budget, the ratchet, and the split rules; run it+with `--help` before anything else.++## Splitting++1. Run the script with `--all` from the repository root, passing the+ repository's `--max-tokens` if its hook sets one. Its list is the backlog.+2. Pick the files to split: the one blocking the work in front of you, or+ with no blocker, the largest ones a change is about to touch.+3. Split each one by the rules the script prints. A seam is a concept the+ file already owns, not a line count; when+ [refactor-clean](../refactor-clean/SKILL.md) is installed, use it to find+ them.+4. Prove the split changed no behaviour with the repository's own checks,+ and commit it alone, before the change that needed the room.++A file whose every split would cost more than it saves (one table or state+machine that only reads whole) may opt out instead. Put+`@ratchet-files-ignore` in a comment within its first 10 lines, in the+file's own comment syntax, followed by why splitting fails:++```js+// @ratchet-files-ignore: one opcode table; split, each half reads without the other.+```++The check then skips the file entirely, even as it grows, so opt out only+after looking for a seam and finding none worth its cost, never to get past a+refusal. Tell the user each file you opt out and why. `--all` counts+opted-out files so they stay visible.++Done when every file you picked is under budget in its own commit, the+behaviour checks pass, and `--all` no longer lists those files.++## Setting up the ratchet++Never install the hook without the user's explicit yes, even when they asked+to set up the repository. Installing it freezes every file already over+budget, so audit first and leave the repository untouched while you do:++1. Copy the script to a scratch directory outside the repository, run+ `npm install --prefix <scratch> linguist-languages` there, and run+ `node <scratch>/check-file-size.mjs --all` from the repository root (Node+ 18.3 or later). The script finds the package beside itself when the+ repository lacks it.+2. Ask once, with everything the user is agreeing to: how many files the hook+ would freeze and which are largest, leaving out generated or vendored ones+ you will mark below (name those); the budget, 8,000 tokens unless they want+ another; and which pre-commit runner you will wire it into, including any+ you would add. In a repository without Node tooling, say the check adds+ it.++Once they agree, set up everything the check needs:++- **Script.** The hook runs a copy committed in the repository, so every+ clone runs the same check. If this skill folder is committed in the+ repository, point the hook at its `scripts/check-file-size.mjs`; otherwise+ copy that file to where the repository keeps its tool scripts (`scripts/`+ if it has none). The entries below use `scripts/check-file-size.mjs`;+ substitute the real path.+- **Dependency.** Add `linguist-languages` at an exact version as a dev+ dependency of the repository's root `package.json`, with the package+ manager its lockfile names, or npm if there is none. The check needs Node+ even in a repository written in another language: with no `package.json`,+ create one at the root holding only `"private": true` and what this setup+ adds, and make sure `.gitignore` covers `node_modules/`.+- **Hook.** Run the script on every staged file with `node` (or `bun` where+ the repository already runs its tools with it), appending `--max-tokens N`+ for a budget other than the default. Add it beside the existing entries of+ the pre-commit runner the repository already has:++ | Runner | Where | Entry |+ |---|---|---|+ | lint-staged | its config | `"*": "node scripts/check-file-size.mjs"` |+ | lefthook | `pre-commit.commands` | `run: node scripts/check-file-size.mjs {staged_files}` |+ | pre-commit | `.pre-commit-config.yaml`, `repo: local` | `id: ratchet-files`, `name: source file size`, `entry: node scripts/check-file-size.mjs`, `language: system` |+ | husky or a plain hook | the `pre-commit` script | `git diff --cached --name-only -z \| xargs -0 node scripts/check-file-size.mjs` |++ With no runner, add husky with lint-staged as dev dependencies, run+ `npx husky init`, replace the whole of `.husky/pre-commit` (it starts as+ `npm test`) with `npx lint-staged` or the package manager's equivalent, and+ add the lint-staged entry above. Husky installs the hook on each clone's+ package install.+- **Exclusions.** Mark generated and vendored code `linguist-generated` or+ `linguist-vendored` in `.gitattributes` instead of adding skip lists. The+ script reads the staged `.gitattributes`, so stage it before rerunning+ `--all`.+- **Docs.** Document the budget, the opt-out pragma, and that a clone needs a+ package install for the hook, where the repository documents its other+ hooks; failing that, in a section of its README, creating one if needed.++Commit the setup through the new hook, without `--no-verify`.++Done when the setup commit passed the hook, and the installed hook refuses+`git commit` both of a staged new source file over the budget (a `.js` file+of more than four bytes per token of budget) and of a line added to a file the audit+listed, printing the split rules. Unstage and discard both probes afterwards.diff --git a/.gitattributes b/.gitattributesindex f516526..0f7ae55 100644--- a/.gitattributes+++ b/.gitattributes@@ -1,2 +1,7 @@# Verbatim packet evidence keeps spaces after numbered blank source lines.specs/done/provider-support/assets/provider-stdout.txt whitespace=-blank-at-eol++# Ratchet exclusions: generated and vendored source is maintained by its producer.+**/generated/** linguist-generated+**/*.generated.* linguist-generated+**/vendor/** linguist-vendoreddiff --git a/.husky/pre-commit b/.husky/pre-commitnew file mode 100755index 0000000..a926c36--- /dev/null+++ b/.husky/pre-commit@@ -0,0 +1,2 @@+#!/bin/sh+npx lint-stageddiff --git a/README.md b/README.mdindex 8cf9968..0b7fd4d 100644--- a/README.md+++ b/README.md@@ -171,3 +171,7 @@ See [Contributing](CONTRIBUTING.md) for the pull-request review and maintenancepolicy.[MIT](LICENSE). [Artwork and generation prompts](assets/README.md).++## Ratchet Files++Source files are limited to 8,000 estimated tokens (bytes divided by four). The pre-commit hook runs `scripts/check-file-size.mjs` through lint-staged, so a clone must install the repository dependencies before commits. Existing oversized authored files are frozen at their current size; generated and vendored paths are excluded through `.gitattributes`. A file may use `@ratchet-files-ignore` in its first 10 lines only when splitting it would cost more than it saves, with the reason beside the pragma.diff --git a/bun.lock b/bun.lockindex 9f06f90..542c8a7 100644--- a/bun.lock+++ b/bun.lock@@ -5,6 +5,9 @@"": {"name": "jevgrep","devDependencies": {+ "husky": "9.1.7",+ "linguist-languages": "9.5.0",+ "lint-staged": "17.6.0","oxfmt": "0.57.0","oxlint": "1.72.0","turbo": "2.10.3",@@ -13,7 +16,7 @@},"apps/cli": {"name": "@dzhng/jevgrep",- "version": "0.8.2",+ "version": "0.8.5","bin": {"jg": "./dist/bin/index.js",},@@ -185,10 +188,16 @@"fast-wrap-ansi": ["[email protected]", "", { "dependencies": { "fast-string-width": "^3.0.2" } }, "sha512-7F2Fl+TjRSenLqlU3UjSH0iyqopqoZIu7eZVpEirP2g1GtWa2G/ecEmBdgz31+Mxr+ELclgg6sokpSFIQiZ02Q=="],+ "husky": ["[email protected]", "", { "bin": { "husky": "bin.js" } }, "sha512-5gs5ytaNjBrh5Ow3zrvdUUY+0VxIuWVL4i9irt6friV+BqdCfmV11CQTWMiBYWHbXhco+J1kHfTOUkePhCDvMA=="],+"ignore": ["[email protected]", "", {}, "sha512-Hs59xBNfUIunMFgWAbGX5cq6893IbWg4KnrjbYwX3tx0ztorVgTDA6B2sxf8ejHJ4wz8BqGUMYlnzNBer5NvGg=="],"json-schema": ["[email protected]", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="],+ "linguist-languages": ["[email protected]", "", {}, "sha512-Wtg9O6iHNBYFJQVtPNwWCaWkNEqZqiCGI4gSkeH8TqvgZUyb1IUWmFSa9Z9UhT392/aX6Peoa91bH9sDah1ZVw=="],++ "lint-staged": ["[email protected]", "", { "dependencies": { "picomatch": "^4.0.7", "string-argv": "^0.3.2", "tinyexec": "^1.3.1" }, "optionalDependencies": { "yaml": "^2.9.1" }, "bin": { "lint-staged": "bin/lint-staged.js" } }, "sha512-nOhBfYkZEwkFtNzD5pK9s9LkcWoxJ+whLzEnhqi9nl7zTB5n7tg8+9iYgMGx2Wm3VlLu8cbNj0RACzY+nWi5iQ=="],+"node-addon-api": ["[email protected]", "", {}, "sha512-VijLXbi3UACN69I0JVXJsX4tjACjNoQDgv2gTF6sx2wWEi8tkSg2eX8p5gSIFi8z2+DL3oHmY6OyKce38SDolg=="],"node-gyp-build": ["[email protected]", "", { "bin": { "node-gyp-build": "bin.js", "node-gyp-build-optional": "optional.js", "node-gyp-build-test": "build-test.js" } }, "sha512-LA4ZjwlnUblHVgq0oBF3Jl/6h/Nvs5fzBLwdEF4nuxnFdsfajde4WfxtJr3CaiH+F6ewcIB/q4jQ4UzPyid+CQ=="],@@ -197,8 +206,14 @@"oxlint": ["[email protected]", "", { "optionalDependencies": { "@oxlint/binding-android-arm-eabi": "1.72.0", "@oxlint/binding-android-arm64": "1.72.0", "@oxlint/binding-darwin-arm64": "1.72.0", "@oxlint/binding-darwin-x64": "1.72.0", "@oxlint/binding-freebsd-x64": "1.72.0", "@oxlint/binding-linux-arm-gnueabihf": "1.72.0", "@oxlint/binding-linux-arm-musleabihf": "1.72.0", "@oxlint/binding-linux-arm64-gnu": "1.72.0", "@oxlint/binding-linux-arm64-musl": "1.72.0", "@oxlint/binding-linux-ppc64-gnu": "1.72.0", "@oxlint/binding-linux-riscv64-gnu": "1.72.0", "@oxlint/binding-linux-riscv64-musl": "1.72.0", "@oxlint/binding-linux-s390x-gnu": "1.72.0", "@oxlint/binding-linux-x64-gnu": "1.72.0", "@oxlint/binding-linux-x64-musl": "1.72.0", "@oxlint/binding-openharmony-arm64": "1.72.0", "@oxlint/binding-win32-arm64-msvc": "1.72.0", "@oxlint/binding-win32-ia32-msvc": "1.72.0", "@oxlint/binding-win32-x64-msvc": "1.72.0" }, "peerDependencies": { "oxlint-tsgolint": ">=0.22.1", "vite-plus": "*" }, "optionalPeers": ["oxlint-tsgolint", "vite-plus"], "bin": { "oxlint": "bin/oxlint" } }, "sha512-1rhdZIP/EvoI91ABIwNU5Q8+bWf8mjrS5UzIOZld4d4bXxJvtlUhlQvaoTogIGin/qdErMOrwaIJvCSIAKTLhA=="],+ "picomatch": ["[email protected]", "", {}, "sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA=="],+"sisteransi": ["[email protected]", "", {}, "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg=="],+ "string-argv": ["[email protected]", "", {}, "sha512-aqD2Q0144Z+/RqG52NeHEkZauTAUWJO8c6yTftGJKO3Tja5tUgIfmIl6kExvhtxSDP7fXB6DvzkfMpCd/F3G+Q=="],++ "tinyexec": ["[email protected]", "", {}, "sha512-GCvB3aoys96IuDFBMcTB46JOR6mdMtAToqwiW8JlWhsoh1mhHi/xn9ss/Dg7N555GiJyEt2qzoG/NHCwM6h1EA=="],+"tinypool": ["[email protected]", "", {}, "sha512-Pugqs6M0m7Lv1I7FtxN4aoyToKg1C4tu+/381vH35y8oENM/Ai7f7C4StcoK4/+BSw9ebcS8jRiVrORFKCALLw=="],"tree-sitter-go": ["[email protected]", "", { "dependencies": { "node-addon-api": "^8.3.1", "node-gyp-build": "^4.8.4" }, "peerDependencies": { "tree-sitter": "^0.25.0" }, "optionalPeers": ["tree-sitter"] }, "sha512-APBc/Dq3xz/e35Xpkhb1blu5UgW+2E3RyGWawZSCNcbGwa7jhSQPS8KsUupuzBla8PCo8+lz9W/JDJjmfRa2tw=="],@@ -217,6 +232,8 @@"web-tree-sitter": ["[email protected]", "", {}, "sha512-XK08gj6RwTMQatAG7uVRP8MunqotL/XC19vHgkSPKmELgbGPBj4ECvB8haHOUnyj6ls2B8t42UTro14zxGgAHg=="],+ "yaml": ["[email protected]", "", { "bin": { "yaml": "bin.mjs" } }, "sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw=="],+"zod": ["[email protected]", "", {}, "sha512-v5l/aFXZQeai4awLbOpSoHecE9UiMrnfx75tEXLjNonXVARxQ5mOeipTjROUchszUNCqnE+hqAMujRsRHsut2Q=="],"@ai-sdk/gateway/@ai-sdk/provider": ["@ai-sdk/[email protected]", "", { "dependencies": { "json-schema": "^0.4.0" } }, "sha512-VYMBxIQdcHqbIf1j+YZlI9Ati6LZ4wJe0GGd4z4a5H/KxTggjeOiyaVYTnfF7LHZK5jMQ+rofmzz4QPqf++NUw=="],diff --git a/package.json b/package.jsonindex a2d7ce0..ab1ca7f 100644--- a/package.json+++ b/package.json@@ -28,16 +28,23 @@"test:e2e": "bun run test:installed","verify": "bun run check-types && bun run lint && bun run test && bun run test:installed","eval:swebench": "python3 evals/implementation/swebench/installed.py",- "test:native": "node scripts/test-native.mjs"+ "test:native": "node scripts/test-native.mjs",+ "prepare": "husky"},"devDependencies": {"oxfmt": "0.57.0","oxlint": "1.72.0","turbo": "2.10.3",- "typescript": "catalog:"+ "typescript": "catalog:",+ "linguist-languages": "9.5.0",+ "husky": "9.1.7",+ "lint-staged": "17.6.0"},"engines": {"node": ">=22"},- "packageManager": "[email protected]"+ "packageManager": "[email protected]",+ "lint-staged": {+ "*": "node scripts/check-file-size.mjs"+ }}diff --git a/scripts/check-file-size.mjs b/scripts/check-file-size.mjsnew file mode 100755index 0000000..5bf829d--- /dev/null+++ b/scripts/check-file-size.mjs@@ -0,0 +1,195 @@+#!/usr/bin/env node+// Size budget for source files, so an agent can read any file whole. Tokens+// are estimated as bytes / 4. Run with --help for modes and hook setup.+import { execFileSync } from "node:child_process";+import { realpathSync } from "node:fs";+import { createRequire } from "node:module";+import { extname, join, relative } from "node:path";+import { pathToFileURL } from "node:url";+import { parseArgs } from "node:util";++const pragma = "@ratchet-files-ignore";+const pragmaLines = 10;++const usage = `Usage:+ check-file-size.mjs --all [--max-tokens N]+ List every tracked source file over the budget, largest first.+ check-file-size.mjs [--max-tokens N] <path>...+ Ratchet the staged versions of <path>: a file over the budget fails+ unless HEAD already had it over the budget and it did not grow. A+ staged rename is measured against the file it came from.++Source is any extension GitHub Linguist assigns to a programming language+and no prose language (.md is not source). Files marked linguist-generated+or linguist-vendored in the staged .gitattributes are skipped, and so is a+file with ${pragma} in a comment in its first ${pragmaLines} lines,+followed by why splitting it would cost more than it saves. Language data+comes from the linguist-languages package, resolved from the repository,+else from beside this script. Bun fetches it in a repository without+node_modules.++The default budget is 8,000 tokens. Exit 1 means a file broke the budget.++Pre-commit install: run it on every staged file with bun or node (18.3+), e.g.+ lint-staged: "*": "node <path-to>/check-file-size.mjs"+ plain hook: git diff --cached --name-only -z | xargs -0 node <path-to>/check-file-size.mjs+Commit this script into the repository and point the hook at that copy, so+every clone runs the same check.`;++const splitRules = `Split a file before adding to it:+ - Split by concept, never by line count: each new file owns one idea and is+ named for it. No part1/part2, utils, helpers, or misc files.+ - Move whole items (types with their impls, functions with the contract+ their callers rely on) and keep the public interface where callers+ already import it.+ - Move tests beside the code they test, grouped the way the code is.+ - Commit the split on its own, before the change that needed the room. A+ split must not change behaviour; prove it with the checks the repository+ already uses for that.`;++const optOut = `Opt a file out only when every split would cost more than it saves, such+as one table or state machine that reads only as a whole. Put+${pragma} in a comment within its first ${pragmaLines} lines, followed by why:+ // ${pragma}: one opcode table; split, each half reads without the other.+The check then skips the file entirely, even as it grows, so it is a last+resort, never a way past a refusal. Tell the user each file you opt out.`;++const { values, positionals } = parseArgs({+ allowPositionals: true,+ options: {+ all: { type: "boolean" },+ help: { type: "boolean" },+ "max-tokens": { type: "string", default: "8000" },+ },+});+if (values.help) {+ console.log(`${usage}\n\n${splitRules}\n\n${optOut}`);+ process.exit(0);+}+const maxTokens = Number(values["max-tokens"]);+const maxBytes = maxTokens * 4;++const git = (args, input) =>+ execFileSync("git", args, {+ encoding: "utf8",+ input,+ maxBuffer: 1 << 30, // A large repository lists tens of megabytes of paths.+ stdio: ["pipe", "pipe", "ignore"],+ });+const root = git(["rev-parse", "--show-toplevel"]).trim();++// Resolved from the repository, so the hook runs its pinned version; failing+// that, from beside this script, where an audit before setup can install it.+let languages;+for (const base of [join(root, "package.json"), import.meta.url]) {+ try {+ const require = createRequire(base);+ languages = await import(pathToFileURL(require.resolve("linguist-languages")).href);+ break;+ } catch {}+}+if (!languages) {+ console.error(+ "check-file-size: install linguist-languages as a dev dependency of this repository.",+ );+ process.exit(1);+}+const claims = new Map();+for (const { type, extensions = [] } of Object.values(languages)) {+ for (const extension of extensions) {+ claims.set(extension, [...(claims.get(extension) ?? []), type]);+ }+}+const isSource = (path) => {+ const types = claims.get(extname(path).toLowerCase()) ?? [];+ return types.includes("programming") && !types.includes("prose");+};++// Paths .gitattributes marks as generated or vendored, read from the index.+const unauthored = (paths) => {+ if (paths.length === 0) return new Set();+ const output = git(+ ["check-attr", "--cached", "-z", "--stdin", "linguist-generated", "linguist-vendored"],+ paths.join("\0"),+ ).split("\0");+ const marked = new Set();+ for (let i = 0; i + 2 < output.length; i += 3) {+ if (output[i + 2] === "true" || output[i + 2] === "set") marked.add(output[i]);+ }+ return marked;+};++const blobBytes = (spec) => {+ try {+ return Number(git(["cat-file", "-s", spec]).trim());+ } catch {+ return null; // Not in that tree: a new file.+ }+};+// A file opts out with the pragma near its top, in whatever comment syntax+// its language uses.+const ignored = (spec) =>+ git(["cat-file", "blob", spec])+ .split("\n", pragmaLines)+ .some((line) => line.includes(pragma));+const tokens = (bytes) => `~${Math.round(bytes / 4).toLocaleString("en")} tokens`;+const sources = (paths) => {+ const candidates = paths.filter(isSource);+ const skipped = unauthored(candidates);+ return candidates.filter((path) => !skipped.has(path));+};++if (values.all) {+ const over = sources(git(["ls-files", "-z"]).split("\0").filter(Boolean))+ .map((path) => ({ path, bytes: blobBytes(`:${path}`) }))+ .filter(({ bytes }) => bytes > maxBytes)+ .sort((a, b) => b.bytes - a.bytes);+ const opted = over.filter(({ path }) => ignored(`:${path}`));+ const listed = over.filter((file) => !opted.includes(file));+ for (const { path, bytes } of listed) console.log(`${tokens(bytes).padStart(15)} ${path}`);+ console.log(`Source files over ${maxTokens.toLocaleString("en")} tokens: ${listed.length}.`);+ if (opted.length > 0)+ console.log(`Over the budget but opted out with ${pragma}: ${opted.length}.`);+ process.exit(listed.length > 0 ? 1 : 0);+}++const origin = new Map();+for (const line of git(["diff", "--cached", "--find-renames", "--name-status"]).split("\n")) {+ const [status, from, to] = line.split("\t");+ if (status?.startsWith("R")) origin.set(to, from);+}+const failures = [];+// Git reports the root with symlinks resolved (macOS /tmp is /private/tmp).+const inRepo = (path) => {+ try {+ return relative(root, realpathSync(path));+ } catch {+ return relative(root, path); // Deleted, so nothing to measure.+ }+};+for (const path of sources(positionals.map(inRepo))) {+ const bytes = blobBytes(`:${path}`);+ if (bytes === null || bytes <= maxBytes || ignored(`:${path}`)) continue;+ const before = blobBytes(`HEAD:${origin.get(path) ?? path}`);+ if (before !== null && bytes <= before) continue;+ failures.push({ path, bytes, before });+}+if (failures.length > 0) {+ const budget = tokens(maxBytes);+ console.error(`Commit refused: source files over the budget of ${budget} (bytes / 4).`);+ for (const { path, bytes, before } of failures) {+ // A file HEAD already had over the budget is held to its HEAD size, not+ // to the budget.+ const frozen = before !== null && before > maxBytes;+ const limit = frozen ? before : maxBytes;+ const why = frozen+ ? `already over the budget at ${tokens(before)}, so it may not grow; it does not have to get under the budget`+ : `${before === null ? "a new file" : `was ${tokens(before)}`}, so it must fit the budget`;+ console.error(` ${path}: ${tokens(bytes)}, ${why}.`);+ console.error(` Cut at least ${(bytes - limit).toLocaleString("en")} bytes.`);+ }+ console.error(+ `\nMake room by moving a concept into its own file, not by squeezing code to fit.\n${splitRules}\n\n${optOut}`,+ );+ process.exit(1);+}