Beautiful, AI-native markdown IDE and LLM wiki
TypeScript 2nd-brainagent-skillsclaudecodexcompany-braindocsknowledge-knowledge-baseknowledge-managementllmllm-wikillm-wiki-karpathymarkdownmarkdown-editormdmdxnotespkmsecond-brainskills
Exclude gitignored outputs with shared check-ignore discovery (#4378)
Nick Gomez437a2632cf2e
diff --git a/lint-plugins/no-comments/README.md b/lint-plugins/no-comments/README.mdindex 982931504..732d4b599 100644--- a/lint-plugins/no-comments/README.md+++ b/lint-plugins/no-comments/README.md@@ -4,8 +4,9 @@ Open Knowledge writes **no code comments by default**. A comment is legal only wparses it, or when it carries a contract that code cannot express. Everything else belongs inthe commit message, the PR body, `AGENTS.md`, a spec, or a `README`.-This directory is the single source of truth for that rule. It is a zero-dependency plain-ESM-module so that every consumer runs the *same* predicate:+This directory is the single source of truth for that rule. Its plain-ESM matcher and comment+predicate require no installed packages, so every consumer runs the *same* predicate. Whole-tree+discovery additionally requires the `git` executable on `PATH` to apply ignore rules:| Consumer | Lane | Why it exists ||---|---|---|@@ -478,14 +479,20 @@ Scope is declared, not compiled in: `no-comments.config.jsonc` at the repositorygrammar families (an extension list plus an extractor id), the named units that cross a familywith a list of roots, and the exclusion set. Membership is DERIVED from `units[].roots` crossedwith the family's extensions, plus any explicit `units[].files`; the walk roots and the-directories discovery prunes derive from the same declaration. `scope.mjs` compiles that config-and nothing else, so an adopter points the predicate at their own root and gets their own scope.+directories discovery prunes derive from the same declaration. The matcher in `scope.mjs`+compiles that config. Whole-tree discovery also removes candidates that Git reports as ignored,+using the ignore rules Git resolves from the containing worktree, including parent-directory+rules when the scope root is nested in a repository. Git filtering retains tracked candidates+even beneath an ignored directory, and untracked nonignored source before staging; the declared+static scope still applies to both. A non-Git content directory retains filesystem discovery+when Git is installed. Missing Git or unexpected+Git failures stop discovery rather than silently disabling ignore handling.Read the config rather than a restatement here: this paragraph has drifted from the real scopetwice, because an enumeration in prose has nothing checking it.`scope.test.mjs` also pins discovery against `git ls-files`, so an in-scope file git does nottrack yet fails it by name: track it if it is source, move it outside the declared roots if it is-scratch, or add it to `exclude` if it is generated output.+scratch, or declare generated output in `.gitignore` (or in `exclude` for a non-Git adopter).A family that no unit names is declared but ungated: it gives the extractor a grammar to readthat extension under, while contributing nothing to membership. That is how a family reaches the@@ -529,13 +536,27 @@ This directory ships to the public mirror, which means its own tests may not *spshipping file. `scope.test.mjs` composes those paths from parts for that reason, and a testadded here has to do the same.+The Git environment scrub mirrors `scripts/git-clean-env.mjs`. The hook import-closure constraint+above prevents importing that helper into this module, so a parity test keeps the local scrub+aligned with the canonical helper; the local copy additionally fixes `LC_ALL=C` for diagnostic+classification. Repository redirection and upward-discovery controls are removed before Git runs.+The baseline scrub removes GIT_DIR, GIT_WORK_TREE, GIT_COMMON_DIR, GIT_INDEX_FILE, GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES, GIT_NAMESPACE, and GIT_PREFIX. The discovery controls add GIT_CEILING_DIRECTORIES and GIT_DISCOVERY_ACROSS_FILESYSTEM for this scope; they do not redefine every Git helper's contract. Git configuration, including configured ignore files,+remains an input: this scrub is not isolation from all Git configuration.++Static `exclude` rules and Git ignore rules serve different boundaries. Static exclusions also+work for non-Git adopters and can prune directories before filesystem traversal. Git filtering+follows the walk so force-added tracked files under an ignored directory are retained. This+change removes ignored candidates from parsing, not all traversal cost; the with/without-output+gate timings are separate observations, not a controlled performance benchmark.+### Symlinks are refused, never followedDiscovery walks real entries only. Any symlink it meets that would otherwise have contributed —one whose own path is in scope, and one that resolves to a directory the walk would have descended-— is recorded as a skip, and `discoverInScopeFiles` throws rather than return a list that quietly-lost a subtree. `discoverInScopeFilesWithSkips` is the exported escape hatch for a caller that-wants to handle skips itself. Both refuse to run without `lstatSync` and `statSync`: `lstatSync`+— is recorded as a skip. Paths Git reports as ignored are removed from both the candidate and+skip sets. For every remaining skip, `discoverInScopeFiles` throws rather than return a list+that quietly lost a subtree. `discoverInScopeFilesWithSkips` is the exported escape hatch for+a caller that wants to handle skips itself. Both refuse to run without `lstatSync` and `statSync`: `lstatSync`is what tells a link from a real entry, `statSync` is what tells a link-to-a-directory from alink-to-a-file, and passing one where the other belongs makes the whole branch unreachable whilethe corpus silently widens to whatever the links point at.diff --git a/lint-plugins/no-comments/scope.mjs b/lint-plugins/no-comments/scope.mjsindex 3801a7e10..ca450f9ca 100644--- a/lint-plugins/no-comments/scope.mjs+++ b/lint-plugins/no-comments/scope.mjs@@ -1,12 +1,13 @@+import { spawnSync } from 'node:child_process';import { realpathSync } from 'node:fs';import { dirname, join, resolve } from 'node:path';import { fileURLToPath } from 'node:url';import {- UNSUPPORTED_GLOB_SYNTAX,loadScopeConfig,SCOPE_CONFIG_FILENAME,ScopeConfigError,ScopeConfigMissingError,+ UNSUPPORTED_GLOB_SYNTAX,} from './config.mjs';import { hashDialectsFor, hashDialectsWithoutReference } from './extract-hash.mjs';import { GRAMMAR_EXTRACTORS } from './extractors.mjs';@@ -39,6 +40,50 @@ export function normalizeRelativePath(relPath) {return relPath.replace(/\\/g, '/').replace(/^\.\//, '');}+export function gitEnvironment(base = process.env) {+ const {+ GIT_DIR: _gitDir,+ GIT_WORK_TREE: _gitWorkTree,+ GIT_COMMON_DIR: _gitCommonDir,+ GIT_INDEX_FILE: _gitIndexFile,+ GIT_OBJECT_DIRECTORY: _gitObjectDirectory,+ GIT_ALTERNATE_OBJECT_DIRECTORIES: _gitAlternateObjectDirectories,+ GIT_NAMESPACE: _gitNamespace,+ GIT_PREFIX: _gitPrefix,+ GIT_CEILING_DIRECTORIES: _gitCeilingDirectories,+ GIT_DISCOVERY_ACROSS_FILESYSTEM: _gitDiscoveryAcrossFilesystem,+ ...env+ } = base;+ return { ...env, LC_ALL: 'C' };+}++function runGit(repoRoot, args, input) {+ return spawnSync('git', args, {+ cwd: repoRoot,+ encoding: 'utf8',+ env: gitEnvironment(),+ input,+ maxBuffer: 16 * 1024 * 1024,+ });+}++function gitIgnoredPaths(repoRoot, relPaths) {+ if (relPaths.length === 0) return new Set();+ const result = runGit(repoRoot, ['check-ignore', '--stdin', '-z'], `${relPaths.join('\0')}\0`);+ if (result.status === 1) return new Set();+ if (result.status === 128 && /not a git repository/i.test(result.stderr)) return new Set();+ if (result.error || result.status !== 0) {+ const outcome = result.signal+ ? `killed by ${result.signal}`+ : result.status === null+ ? 'no exit status'+ : `exit ${result.status}`;+ const detail = result.stderr?.trim() || result.error?.message || outcome;+ throw new Error(`git check-ignore failed in ${repoRoot}: ${detail}`, { cause: result.error });+ }+ return new Set(result.stdout.split('\0').filter(Boolean).map(normalizeRelativePath));+}+const compile = (globs) => globs.map(globToRegExp);function extensionOf(relPath) {@@ -384,8 +429,15 @@ export function discoverInScopeFilesWithSkips(repoRoot, options) {else if (stat.isFile()) found.add(file);}}- skips.sort((a, b) => (a.path < b.path ? -1 : 1));- return { files: [...found].sort(), skips, declaredAbsent: scope.config.declaredAbsent };+ const ignored = gitIgnoredPaths(repoRoot, [...found, ...skips.map((skip) => skip.path)]);+ for (const path of ignored) found.delete(path);+ const visibleSkips = skips.filter((skip) => !ignored.has(skip.path));+ visibleSkips.sort((a, b) => (a.path < b.path ? -1 : 1));+ return {+ files: [...found].sort(),+ skips: visibleSkips,+ declaredAbsent: scope.config.declaredAbsent,+ };}export function discoverInScopeFiles(repoRoot, options) {diff --git a/lint-plugins/no-comments/scope.test.mjs b/lint-plugins/no-comments/scope.test.mjsindex 33a728d85..c2235344c 100644--- a/lint-plugins/no-comments/scope.test.mjs+++ b/lint-plugins/no-comments/scope.test.mjs@@ -7,12 +7,14 @@ import {readFileSync,rmSync,statSync,+ symlinkSync,utimesSync,writeFileSync,} from 'node:fs';import { tmpdir } from 'node:os';import { basename, dirname, join } from 'node:path';-import { afterAll, describe, expect, test } from 'vitest';+import { afterAll, beforeAll, describe, expect, test } from 'vitest';+import { gitCleanEnv } from '../../scripts/git-clean-env.mjs';import {loadScopeConfig,SCOPE_CONFIG_FILENAME,@@ -24,6 +26,7 @@ import {diagnoseScope,discoverInScopeFiles,discoverInScopeFilesWithSkips,+ gitEnvironment,globToRegExp,isExcluded,isInScope,@@ -32,11 +35,7 @@ import {subjectScope,unitFor,} from './scope.mjs';-import {- configWith,- createScratchRoot,- removeScratchRoots,-} from './scratch-root.test-helper.mjs';+import { configWith, createScratchRoot, removeScratchRoots } from './scratch-root.test-helper.mjs';const CONFIG = subjectScope().config;@@ -227,6 +226,201 @@ describe('discovery over the real tree', () => {});});+describe('gitignored output is outside discovery', () => {+ afterAll(removeScratchRoots);++ let root;+ const expected = [+ 'packages/desktop/out/tracked.ts',+ 'packages/desktop/src/authoring.ts',+ 'packages/desktop/src/tracked.ts',+ ];+ beforeAll(() => {+ root = createScratchRoot({+ prefix: 'no-comments-gitignore-',+ config: configWith({+ units: [{ id: 'desktop', family: 'typescript', roots: ['packages/desktop'] }],+ }),+ files: {+ '.gitignore': 'packages/desktop/out/\n',+ 'packages/desktop/src/tracked.ts': 'export const tracked = true;\n',+ 'packages/desktop/src/deleted.ts': 'export const deleted = true;\n',+ 'packages/desktop/src/authoring.ts': 'export const authoring = true;\n',+ 'packages/desktop/out/generated.ts': 'export const generated = true;\n',+ 'packages/desktop/out/tracked.ts': 'export const trackedOutput = true;\n',+ },+ });+ symlinkSync('../src/tracked.ts', join(root, 'packages/desktop/out/linked.ts'));+ execFileSync('git', ['init', '-q'], { cwd: root });+ execFileSync(+ 'git',+ [+ 'add',+ '.gitignore',+ SCOPE_CONFIG_FILENAME,+ 'packages/desktop/src/deleted.ts',+ 'packages/desktop/src/tracked.ts',+ ],+ { cwd: root },+ );+ execFileSync('git', ['add', '-f', 'packages/desktop/out/tracked.ts'], { cwd: root });+ rmSync(join(root, 'packages/desktop/src/deleted.ts'));+ });++ test('retains tracked and untracked source while excluding ignored desktop output', () => {+ expect(discoverInScopeFiles(root, { readdirSync, statSync, lstatSync })).toStrictEqual(+ expected,+ );+ });++ test('ignored skips disappear while a visible unreadable entry remains a refusal', () => {+ const unreadable = {+ readdirSync,+ statSync,+ lstatSync: (path) => {+ if (+ String(path).endsWith('/packages/desktop/out/generated.ts') ||+ String(path).endsWith('/packages/desktop/src/tracked.ts')+ ) {+ const error = new Error(`EACCES: simulated failure, lstat '${path}'`);+ error.code = 'EACCES';+ throw error;+ }+ return lstatSync(path);+ },+ };+ const result = discoverInScopeFilesWithSkips(root, unreadable);+ expect(result.skips).toStrictEqual([+ { path: 'packages/desktop/src/tracked.ts', reason: 'EACCES' },+ ]);+ expect(result.files).toStrictEqual([+ 'packages/desktop/out/tracked.ts',+ 'packages/desktop/src/authoring.ts',+ ]);+ expect(() => discoverInScopeFiles(root, unreadable)).toThrow(+ /packages\/desktop\/src\/tracked\.ts \(EACCES\)/,+ );+ });++ test('inherited git redirection cannot retarget the ignore lookup', () => {+ const decoy = createScratchRoot({ prefix: 'no-comments-gitignore-decoy-' });+ execFileSync('git', ['init', '-q'], { cwd: decoy });+ const gitDirKey = 'GIT_DIR';+ const previous = Reflect.get(process.env, gitDirKey);+ Reflect.set(process.env, gitDirKey, join(decoy, '.git'));+ try {+ expect(discoverInScopeFiles(root, { readdirSync, statSync, lstatSync })).toStrictEqual(+ expected,+ );+ } finally {+ if (previous === undefined) Reflect.deleteProperty(process.env, gitDirKey);+ else Reflect.set(process.env, gitDirKey, previous);+ }+ });++ test('missing Git fails with the spawn error instead of admitting ignored output', () => {+ const pathKey = 'PATH';+ const previous = Reflect.get(process.env, pathKey);+ const emptyPath = join(root, 'empty-path');+ mkdirSync(emptyPath);+ Reflect.set(process.env, pathKey, emptyPath);+ try {+ expect(() => discoverInScopeFiles(root, { readdirSync, statSync, lstatSync })).toThrow(+ /git check-ignore failed.*ENOENT/,+ );+ } finally {+ if (previous === undefined) Reflect.deleteProperty(process.env, pathKey);+ else Reflect.set(process.env, pathKey, previous);+ }+ });++ test('a signal-killed Git names the termination instead of echoing a bare token', () => {+ const pathKey = 'PATH';+ const previous = Reflect.get(process.env, pathKey);+ const shimPath = join(root, 'signal-path');+ mkdirSync(shimPath);+ writeFileSync(join(shimPath, 'git'), '#!/bin/sh\nkill -KILL $$\n', { mode: 0o755 });+ Reflect.set(process.env, pathKey, shimPath);+ try {+ expect(() => discoverInScopeFiles(root, { readdirSync, statSync, lstatSync })).toThrow(+ /git check-ignore failed.*killed by SIGKILL/,+ );+ } finally {+ if (previous === undefined) Reflect.deleteProperty(process.env, pathKey);+ else Reflect.set(process.env, pathKey, previous);+ }+ });++ test('the local scrub stays aligned with the canonical Git environment helper', () => {+ const keysReadBy = (helper) => {+ const reads = new Set();+ const base = new Proxy(+ { KEEP: 'yes', LC_ALL: 'caller' },+ {+ get(target, key, receiver) {+ if (typeof key === 'string') reads.add(key);+ return Reflect.get(target, key, receiver);+ },+ },+ );+ helper(base);+ return reads;+ };+ const canonicalReads = keysReadBy(gitCleanEnv);+ const localReads = keysReadBy(gitEnvironment);+ expect(localReads).toStrictEqual(canonicalReads);+ const redirectKeys = [...canonicalReads].filter((key) => key.startsWith('GIT_'));+ const base = {+ KEEP: 'yes',+ LC_ALL: 'caller',+ ...Object.fromEntries(redirectKeys.map((key) => [key, 'redirect'])),+ };+ const local = gitEnvironment(base);+ expect(local).toStrictEqual({ ...gitCleanEnv(base), LC_ALL: 'C' });+ for (const key of redirectKeys) expect(local).not.toHaveProperty(key);+ });+});++describe('discovery from a nested Git worktree root', () => {+ afterAll(removeScratchRoots);++ let root;+ beforeAll(() => {+ root = createScratchRoot({+ prefix: 'no-comments-nested-git-',+ nest: 'public/open-knowledge',+ config: configWith({+ units: [{ id: 'desktop', family: 'typescript', roots: ['packages/desktop'] }],+ }),+ files: {+ 'packages/desktop/src/authoring.ts': 'export const authoring = true;\n',+ 'packages/desktop/out/generated.ts': 'export const generated = true;\n',+ },+ });+ const outer = dirname(dirname(root));+ writeFileSync(join(outer, '.gitignore'), 'public/open-knowledge/packages/desktop/out/\n');+ execFileSync('git', ['init', '-q'], { cwd: outer });+ });++ const discover = () => discoverInScopeFiles(root, { readdirSync, statSync, lstatSync });++ test('inherits ignore rules from the enclosing repository', () => {+ expect(discover()).toStrictEqual(['packages/desktop/src/authoring.ts']);+ });++ test('a strict-ancestor discovery ceiling cannot hide the enclosing repository', () => {+ const ceilingKey = 'GIT_CEILING_DIRECTORIES';+ const previousCeiling = Reflect.get(process.env, ceilingKey);+ Reflect.set(process.env, ceilingKey, dirname(root));+ try {+ expect(discover()).toStrictEqual(['packages/desktop/src/authoring.ts']);+ } finally {+ if (previousCeiling === undefined) Reflect.deleteProperty(process.env, ceilingKey);+ else Reflect.set(process.env, ceilingKey, previousCeiling);+ }+ });+});+describe('the scope definition is derived, not restated', () => {test('include is the flatten of the per-unit globs the matcher itself compiles', () => {expect(CONFIG.include).toEqual(CONFIG.units.flatMap((unit) => unit.include));diff --git a/scripts/git-clean-env.mjs b/scripts/git-clean-env.mjsindex 346fbced6..521478273 100644--- a/scripts/git-clean-env.mjs+++ b/scripts/git-clean-env.mjs@@ -8,6 +8,8 @@ export function gitCleanEnv(base = process.env) {GIT_ALTERNATE_OBJECT_DIRECTORIES: _a,GIT_NAMESPACE: _n,GIT_PREFIX: _p,+ GIT_CEILING_DIRECTORIES: _cdirs,+ GIT_DISCOVERY_ACROSS_FILESYSTEM: _dafs,...env} = base;return env;