Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

Exclude gitignored outputs with shared check-ignore discovery (#4378)

Nick Gomez437a2632cf2e
diff --git a/lint-plugins/no-comments/README.md b/lint-plugins/no-comments/README.md
index 982931504..732d4b599 100644
--- a/lint-plugins/no-comments/README.md
+++ b/lint-plugins/no-comments/README.md
@@ -4,8 +4,9 @@ Open Knowledge writes **no code comments by default**. A comment is legal only w
parses it, or when it carries a contract that code cannot express. Everything else belongs in
the commit message, the PR body, `AGENTS.md`, a spec, or a `README`.
-This directory is the single source of truth for that rule. It is a zero-dependency plain-ESM
-module so that every consumer runs the *same* predicate:
+This directory is the single source of truth for that rule. Its plain-ESM matcher and comment
+predicate require no installed packages, so every consumer runs the *same* predicate. Whole-tree
+discovery additionally requires the `git` executable on `PATH` to apply ignore rules:
| Consumer | Lane | Why it exists |
|---|---|---|
@@ -478,14 +479,20 @@ Scope is declared, not compiled in: `no-comments.config.jsonc` at the repository
grammar families (an extension list plus an extractor id), the named units that cross a family
with a list of roots, and the exclusion set. Membership is DERIVED from `units[].roots` crossed
with the family's extensions, plus any explicit `units[].files`; the walk roots and the
-directories discovery prunes derive from the same declaration. `scope.mjs` compiles that config
-and nothing else, so an adopter points the predicate at their own root and gets their own scope.
+directories discovery prunes derive from the same declaration. The matcher in `scope.mjs`
+compiles that config. Whole-tree discovery also removes candidates that Git reports as ignored,
+using the ignore rules Git resolves from the containing worktree, including parent-directory
+rules when the scope root is nested in a repository. Git filtering retains tracked candidates
+even beneath an ignored directory, and untracked nonignored source before staging; the declared
+static scope still applies to both. A non-Git content directory retains filesystem discovery
+when Git is installed. Missing Git or unexpected
+Git failures stop discovery rather than silently disabling ignore handling.
Read the config rather than a restatement here: this paragraph has drifted from the real scope
twice, because an enumeration in prose has nothing checking it.
`scope.test.mjs` also pins discovery against `git ls-files`, so an in-scope file git does not
track yet fails it by name: track it if it is source, move it outside the declared roots if it is
-scratch, or add it to `exclude` if it is generated output.
+scratch, or declare generated output in `.gitignore` (or in `exclude` for a non-Git adopter).
A family that no unit names is declared but ungated: it gives the extractor a grammar to read
that extension under, while contributing nothing to membership. That is how a family reaches the
@@ -529,13 +536,27 @@ This directory ships to the public mirror, which means its own tests may not *sp
shipping file. `scope.test.mjs` composes those paths from parts for that reason, and a test
added here has to do the same.
+The Git environment scrub mirrors `scripts/git-clean-env.mjs`. The hook import-closure constraint
+above prevents importing that helper into this module, so a parity test keeps the local scrub
+aligned with the canonical helper; the local copy additionally fixes `LC_ALL=C` for diagnostic
+classification. Repository redirection and upward-discovery controls are removed before Git runs.
+The baseline scrub removes GIT_DIR, GIT_WORK_TREE, GIT_COMMON_DIR, GIT_INDEX_FILE, GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES, GIT_NAMESPACE, and GIT_PREFIX. The discovery controls add GIT_CEILING_DIRECTORIES and GIT_DISCOVERY_ACROSS_FILESYSTEM for this scope; they do not redefine every Git helper's contract. Git configuration, including configured ignore files,
+remains an input: this scrub is not isolation from all Git configuration.
+
+Static `exclude` rules and Git ignore rules serve different boundaries. Static exclusions also
+work for non-Git adopters and can prune directories before filesystem traversal. Git filtering
+follows the walk so force-added tracked files under an ignored directory are retained. This
+change removes ignored candidates from parsing, not all traversal cost; the with/without-output
+gate timings are separate observations, not a controlled performance benchmark.
+
### Symlinks are refused, never followed
Discovery walks real entries only. Any symlink it meets that would otherwise have contributed —
one whose own path is in scope, and one that resolves to a directory the walk would have descended
-— is recorded as a skip, and `discoverInScopeFiles` throws rather than return a list that quietly
-lost a subtree. `discoverInScopeFilesWithSkips` is the exported escape hatch for a caller that
-wants to handle skips itself. Both refuse to run without `lstatSync` and `statSync`: `lstatSync`
+— is recorded as a skip. Paths Git reports as ignored are removed from both the candidate and
+skip sets. For every remaining skip, `discoverInScopeFiles` throws rather than return a list
+that quietly lost a subtree. `discoverInScopeFilesWithSkips` is the exported escape hatch for
+a caller that wants to handle skips itself. Both refuse to run without `lstatSync` and `statSync`: `lstatSync`
is what tells a link from a real entry, `statSync` is what tells a link-to-a-directory from a
link-to-a-file, and passing one where the other belongs makes the whole branch unreachable while
the corpus silently widens to whatever the links point at.
diff --git a/lint-plugins/no-comments/scope.mjs b/lint-plugins/no-comments/scope.mjs
index 3801a7e10..ca450f9ca 100644
--- a/lint-plugins/no-comments/scope.mjs
+++ b/lint-plugins/no-comments/scope.mjs
@@ -1,12 +1,13 @@
+import { spawnSync } from 'node:child_process';
import { realpathSync } from 'node:fs';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import {
- UNSUPPORTED_GLOB_SYNTAX,
loadScopeConfig,
SCOPE_CONFIG_FILENAME,
ScopeConfigError,
ScopeConfigMissingError,
+ UNSUPPORTED_GLOB_SYNTAX,
} from './config.mjs';
import { hashDialectsFor, hashDialectsWithoutReference } from './extract-hash.mjs';
import { GRAMMAR_EXTRACTORS } from './extractors.mjs';
@@ -39,6 +40,50 @@ export function normalizeRelativePath(relPath) {
return relPath.replace(/\\/g, '/').replace(/^\.\//, '');
}
+export function gitEnvironment(base = process.env) {
+ const {
+ GIT_DIR: _gitDir,
+ GIT_WORK_TREE: _gitWorkTree,
+ GIT_COMMON_DIR: _gitCommonDir,
+ GIT_INDEX_FILE: _gitIndexFile,
+ GIT_OBJECT_DIRECTORY: _gitObjectDirectory,
+ GIT_ALTERNATE_OBJECT_DIRECTORIES: _gitAlternateObjectDirectories,
+ GIT_NAMESPACE: _gitNamespace,
+ GIT_PREFIX: _gitPrefix,
+ GIT_CEILING_DIRECTORIES: _gitCeilingDirectories,
+ GIT_DISCOVERY_ACROSS_FILESYSTEM: _gitDiscoveryAcrossFilesystem,
+ ...env
+ } = base;
+ return { ...env, LC_ALL: 'C' };
+}
+
+function runGit(repoRoot, args, input) {
+ return spawnSync('git', args, {
+ cwd: repoRoot,
+ encoding: 'utf8',
+ env: gitEnvironment(),
+ input,
+ maxBuffer: 16 * 1024 * 1024,
+ });
+}
+
+function gitIgnoredPaths(repoRoot, relPaths) {
+ if (relPaths.length === 0) return new Set();
+ const result = runGit(repoRoot, ['check-ignore', '--stdin', '-z'], `${relPaths.join('\0')}\0`);
+ if (result.status === 1) return new Set();
+ if (result.status === 128 && /not a git repository/i.test(result.stderr)) return new Set();
+ if (result.error || result.status !== 0) {
+ const outcome = result.signal
+ ? `killed by ${result.signal}`
+ : result.status === null
+ ? 'no exit status'
+ : `exit ${result.status}`;
+ const detail = result.stderr?.trim() || result.error?.message || outcome;
+ throw new Error(`git check-ignore failed in ${repoRoot}: ${detail}`, { cause: result.error });
+ }
+ return new Set(result.stdout.split('\0').filter(Boolean).map(normalizeRelativePath));
+}
+
const compile = (globs) => globs.map(globToRegExp);
function extensionOf(relPath) {
@@ -384,8 +429,15 @@ export function discoverInScopeFilesWithSkips(repoRoot, options) {
else if (stat.isFile()) found.add(file);
}
}
- skips.sort((a, b) => (a.path < b.path ? -1 : 1));
- return { files: [...found].sort(), skips, declaredAbsent: scope.config.declaredAbsent };
+ const ignored = gitIgnoredPaths(repoRoot, [...found, ...skips.map((skip) => skip.path)]);
+ for (const path of ignored) found.delete(path);
+ const visibleSkips = skips.filter((skip) => !ignored.has(skip.path));
+ visibleSkips.sort((a, b) => (a.path < b.path ? -1 : 1));
+ return {
+ files: [...found].sort(),
+ skips: visibleSkips,
+ declaredAbsent: scope.config.declaredAbsent,
+ };
}
export function discoverInScopeFiles(repoRoot, options) {
diff --git a/lint-plugins/no-comments/scope.test.mjs b/lint-plugins/no-comments/scope.test.mjs
index 33a728d85..c2235344c 100644
--- a/lint-plugins/no-comments/scope.test.mjs
+++ b/lint-plugins/no-comments/scope.test.mjs
@@ -7,12 +7,14 @@ import {
readFileSync,
rmSync,
statSync,
+ symlinkSync,
utimesSync,
writeFileSync,
} from 'node:fs';
import { tmpdir } from 'node:os';
import { basename, dirname, join } from 'node:path';
-import { afterAll, describe, expect, test } from 'vitest';
+import { afterAll, beforeAll, describe, expect, test } from 'vitest';
+import { gitCleanEnv } from '../../scripts/git-clean-env.mjs';
import {
loadScopeConfig,
SCOPE_CONFIG_FILENAME,
@@ -24,6 +26,7 @@ import {
diagnoseScope,
discoverInScopeFiles,
discoverInScopeFilesWithSkips,
+ gitEnvironment,
globToRegExp,
isExcluded,
isInScope,
@@ -32,11 +35,7 @@ import {
subjectScope,
unitFor,
} from './scope.mjs';
-import {
- configWith,
- createScratchRoot,
- removeScratchRoots,
-} from './scratch-root.test-helper.mjs';
+import { configWith, createScratchRoot, removeScratchRoots } from './scratch-root.test-helper.mjs';
const CONFIG = subjectScope().config;
@@ -227,6 +226,201 @@ describe('discovery over the real tree', () => {
});
});
+describe('gitignored output is outside discovery', () => {
+ afterAll(removeScratchRoots);
+
+ let root;
+ const expected = [
+ 'packages/desktop/out/tracked.ts',
+ 'packages/desktop/src/authoring.ts',
+ 'packages/desktop/src/tracked.ts',
+ ];
+ beforeAll(() => {
+ root = createScratchRoot({
+ prefix: 'no-comments-gitignore-',
+ config: configWith({
+ units: [{ id: 'desktop', family: 'typescript', roots: ['packages/desktop'] }],
+ }),
+ files: {
+ '.gitignore': 'packages/desktop/out/\n',
+ 'packages/desktop/src/tracked.ts': 'export const tracked = true;\n',
+ 'packages/desktop/src/deleted.ts': 'export const deleted = true;\n',
+ 'packages/desktop/src/authoring.ts': 'export const authoring = true;\n',
+ 'packages/desktop/out/generated.ts': 'export const generated = true;\n',
+ 'packages/desktop/out/tracked.ts': 'export const trackedOutput = true;\n',
+ },
+ });
+ symlinkSync('../src/tracked.ts', join(root, 'packages/desktop/out/linked.ts'));
+ execFileSync('git', ['init', '-q'], { cwd: root });
+ execFileSync(
+ 'git',
+ [
+ 'add',
+ '.gitignore',
+ SCOPE_CONFIG_FILENAME,
+ 'packages/desktop/src/deleted.ts',
+ 'packages/desktop/src/tracked.ts',
+ ],
+ { cwd: root },
+ );
+ execFileSync('git', ['add', '-f', 'packages/desktop/out/tracked.ts'], { cwd: root });
+ rmSync(join(root, 'packages/desktop/src/deleted.ts'));
+ });
+
+ test('retains tracked and untracked source while excluding ignored desktop output', () => {
+ expect(discoverInScopeFiles(root, { readdirSync, statSync, lstatSync })).toStrictEqual(
+ expected,
+ );
+ });
+
+ test('ignored skips disappear while a visible unreadable entry remains a refusal', () => {
+ const unreadable = {
+ readdirSync,
+ statSync,
+ lstatSync: (path) => {
+ if (
+ String(path).endsWith('/packages/desktop/out/generated.ts') ||
+ String(path).endsWith('/packages/desktop/src/tracked.ts')
+ ) {
+ const error = new Error(`EACCES: simulated failure, lstat '${path}'`);
+ error.code = 'EACCES';
+ throw error;
+ }
+ return lstatSync(path);
+ },
+ };
+ const result = discoverInScopeFilesWithSkips(root, unreadable);
+ expect(result.skips).toStrictEqual([
+ { path: 'packages/desktop/src/tracked.ts', reason: 'EACCES' },
+ ]);
+ expect(result.files).toStrictEqual([
+ 'packages/desktop/out/tracked.ts',
+ 'packages/desktop/src/authoring.ts',
+ ]);
+ expect(() => discoverInScopeFiles(root, unreadable)).toThrow(
+ /packages\/desktop\/src\/tracked\.ts \(EACCES\)/,
+ );
+ });
+
+ test('inherited git redirection cannot retarget the ignore lookup', () => {
+ const decoy = createScratchRoot({ prefix: 'no-comments-gitignore-decoy-' });
+ execFileSync('git', ['init', '-q'], { cwd: decoy });
+ const gitDirKey = 'GIT_DIR';
+ const previous = Reflect.get(process.env, gitDirKey);
+ Reflect.set(process.env, gitDirKey, join(decoy, '.git'));
+ try {
+ expect(discoverInScopeFiles(root, { readdirSync, statSync, lstatSync })).toStrictEqual(
+ expected,
+ );
+ } finally {
+ if (previous === undefined) Reflect.deleteProperty(process.env, gitDirKey);
+ else Reflect.set(process.env, gitDirKey, previous);
+ }
+ });
+
+ test('missing Git fails with the spawn error instead of admitting ignored output', () => {
+ const pathKey = 'PATH';
+ const previous = Reflect.get(process.env, pathKey);
+ const emptyPath = join(root, 'empty-path');
+ mkdirSync(emptyPath);
+ Reflect.set(process.env, pathKey, emptyPath);
+ try {
+ expect(() => discoverInScopeFiles(root, { readdirSync, statSync, lstatSync })).toThrow(
+ /git check-ignore failed.*ENOENT/,
+ );
+ } finally {
+ if (previous === undefined) Reflect.deleteProperty(process.env, pathKey);
+ else Reflect.set(process.env, pathKey, previous);
+ }
+ });
+
+ test('a signal-killed Git names the termination instead of echoing a bare token', () => {
+ const pathKey = 'PATH';
+ const previous = Reflect.get(process.env, pathKey);
+ const shimPath = join(root, 'signal-path');
+ mkdirSync(shimPath);
+ writeFileSync(join(shimPath, 'git'), '#!/bin/sh\nkill -KILL $$\n', { mode: 0o755 });
+ Reflect.set(process.env, pathKey, shimPath);
+ try {
+ expect(() => discoverInScopeFiles(root, { readdirSync, statSync, lstatSync })).toThrow(
+ /git check-ignore failed.*killed by SIGKILL/,
+ );
+ } finally {
+ if (previous === undefined) Reflect.deleteProperty(process.env, pathKey);
+ else Reflect.set(process.env, pathKey, previous);
+ }
+ });
+
+ test('the local scrub stays aligned with the canonical Git environment helper', () => {
+ const keysReadBy = (helper) => {
+ const reads = new Set();
+ const base = new Proxy(
+ { KEEP: 'yes', LC_ALL: 'caller' },
+ {
+ get(target, key, receiver) {
+ if (typeof key === 'string') reads.add(key);
+ return Reflect.get(target, key, receiver);
+ },
+ },
+ );
+ helper(base);
+ return reads;
+ };
+ const canonicalReads = keysReadBy(gitCleanEnv);
+ const localReads = keysReadBy(gitEnvironment);
+ expect(localReads).toStrictEqual(canonicalReads);
+ const redirectKeys = [...canonicalReads].filter((key) => key.startsWith('GIT_'));
+ const base = {
+ KEEP: 'yes',
+ LC_ALL: 'caller',
+ ...Object.fromEntries(redirectKeys.map((key) => [key, 'redirect'])),
+ };
+ const local = gitEnvironment(base);
+ expect(local).toStrictEqual({ ...gitCleanEnv(base), LC_ALL: 'C' });
+ for (const key of redirectKeys) expect(local).not.toHaveProperty(key);
+ });
+});
+
+describe('discovery from a nested Git worktree root', () => {
+ afterAll(removeScratchRoots);
+
+ let root;
+ beforeAll(() => {
+ root = createScratchRoot({
+ prefix: 'no-comments-nested-git-',
+ nest: 'public/open-knowledge',
+ config: configWith({
+ units: [{ id: 'desktop', family: 'typescript', roots: ['packages/desktop'] }],
+ }),
+ files: {
+ 'packages/desktop/src/authoring.ts': 'export const authoring = true;\n',
+ 'packages/desktop/out/generated.ts': 'export const generated = true;\n',
+ },
+ });
+ const outer = dirname(dirname(root));
+ writeFileSync(join(outer, '.gitignore'), 'public/open-knowledge/packages/desktop/out/\n');
+ execFileSync('git', ['init', '-q'], { cwd: outer });
+ });
+
+ const discover = () => discoverInScopeFiles(root, { readdirSync, statSync, lstatSync });
+
+ test('inherits ignore rules from the enclosing repository', () => {
+ expect(discover()).toStrictEqual(['packages/desktop/src/authoring.ts']);
+ });
+
+ test('a strict-ancestor discovery ceiling cannot hide the enclosing repository', () => {
+ const ceilingKey = 'GIT_CEILING_DIRECTORIES';
+ const previousCeiling = Reflect.get(process.env, ceilingKey);
+ Reflect.set(process.env, ceilingKey, dirname(root));
+ try {
+ expect(discover()).toStrictEqual(['packages/desktop/src/authoring.ts']);
+ } finally {
+ if (previousCeiling === undefined) Reflect.deleteProperty(process.env, ceilingKey);
+ else Reflect.set(process.env, ceilingKey, previousCeiling);
+ }
+ });
+});
+
describe('the scope definition is derived, not restated', () => {
test('include is the flatten of the per-unit globs the matcher itself compiles', () => {
expect(CONFIG.include).toEqual(CONFIG.units.flatMap((unit) => unit.include));
diff --git a/scripts/git-clean-env.mjs b/scripts/git-clean-env.mjs
index 346fbced6..521478273 100644
--- a/scripts/git-clean-env.mjs
+++ b/scripts/git-clean-env.mjs
@@ -8,6 +8,8 @@ export function gitCleanEnv(base = process.env) {
GIT_ALTERNATE_OBJECT_DIRECTORIES: _a,
GIT_NAMESPACE: _n,
GIT_PREFIX: _p,
+ GIT_CEILING_DIRECTORIES: _cdirs,
+ GIT_DISCOVERY_ACROSS_FILESYSTEM: _dafs,
...env
} = base;
return env;