Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

fix(ok): ok start crashes natively on non-ASCII project paths (PRD-8034) (#3519)

mike-inkeep53d31329caaf
diff --git a/.changeset/watcher-native-glob-crash.md b/.changeset/watcher-native-glob-crash.md
new file mode 100644
index 000000000..0209e6e80
--- /dev/null
+++ b/.changeset/watcher-native-glob-crash.md
@@ -0,0 +1,12 @@
+---
+"@inkeep/open-knowledge": patch
+---
+
+`ok start` no longer dies without an error message when a project's path, or a folder inside it, is named in a non-Latin script. The server handed its ignore list to the file watcher's native matcher as glob patterns, and that matcher compiles each one to a C++ regular expression it evaluates against every path an event carries. The evaluation recurses once per byte of the path, so a long enough path exhausted the watcher thread's stack and killed the whole process — reported on Windows as exit code `0xC0000409`. Nothing surfaced in JavaScript, so there was no exception, no stack trace, and an empty error log; from the outside the server had simply vanished mid-startup, and every MCP write tool that needed it began failing with `server did not start within 5000ms`.
+
+Because the recursion tracks bytes rather than characters, a Japanese, Chinese, Korean, Cyrillic, or Greek folder name reached the limit at roughly a third of the depth an ASCII one needed, which is why a knowledge base whose folders are named in the author's own language could not be served at all while an otherwise identical ASCII copy started fine. Renaming the folder to ASCII was the only workaround, and pointing an ASCII junction or symlink at it did not help.
+
+The watcher now pre-filters on plain directory prefixes instead of globs, so no path ever reaches the native regular-expression engine. `.git`, `node_modules`, and OK's own `.ok/local` and `.ok/worktrees` are still skipped before OK sees them, at every depth: the startup scan records each one it finds and hands the watcher that list, so a monorepo's per-package `node_modules` stay outside the watch. That list is taken once, at startup, so a dependency tree installed into an open project while the server is running is no longer skipped before OK sees it — the events are still discarded, just a step later, until the next restart.
+
+This also fixes a quieter bug in the same place. An exclusion with no wildcard in it — `dist/`, `drafts/`, `coverage` — was being handed to the watcher as a whole-subtree skip, which is broader than the rule you wrote: it also hid the templates at `drafts/.ok/templates/*.md`, which OK otherwise treats as ordinary editable content. External edits to such a template now reach the editor, and so do edits under a folder you un-ignore while the server is running, which previously stayed invisible until restart. Both apply to the CLI's watcher; the desktop app uses the fallback watcher, which prunes excluded folders itself and is unaffected either way.
+
diff --git a/packages/server/src/content-filter.ts b/packages/server/src/content-filter.ts
index 43ca93d07..8e9119f16 100644
--- a/packages/server/src/content-filter.ts
+++ b/packages/server/src/content-filter.ts
@@ -291,11 +291,11 @@ function isSkillContentAncestorDir(relativePath: string): boolean {
* artifacts (`.ok/config.yml`, `.ok/.gitignore`, `.ok/schemas/*.json`, folder
* `frontmatter.yml`). The watcher-ignore list is glob-derived from
* `.gitignore` / `.okignore` / `.git/info/exclude` (e.g. clone appends
- * `.ok/`), and the @parcel/watcher backend consults THESE globs — not the
- * function predicates (`isDirExcluded` / `isExcluded`) that carry the
- * carve-outs and that the chokidar backend uses. So a blanket `.ok` ignore glob
- * makes parcel (the default backend on Linux) never deliver external edits to
- * project skills, templates, or the shareable artifacts. Dropping the
+ * `.ok/`), and it is consulted INSTEAD of the function predicates
+ * (`isDirExcluded` / `isExcluded`) that carry the carve-outs and that the
+ * chokidar backend uses. So a blanket `.ok` ignore glob would make a watcher
+ * reading this list never deliver external edits to project skills, templates,
+ * or the shareable artifacts. Dropping the
* blanket-`.ok` globs lets the watcher reach them; `.ok` children still never
* reach the file index — `handleRawEvents` consults the unscoped predicates,
* which admit only the skill/template leaves and keep pruning `.ok/local`,
@@ -308,9 +308,15 @@ function isSkillContentAncestorDir(relativePath: string): boolean {
* list — a surviving children-exclude would prune the re-included skills tree
* (and the admitted template/config leaves) with nothing left to re-admit
* them. Sanitization probes the managed roots directly. Ordinary user
- * exclusions remain watcher bounds; otherwise a large ignored tree can
- * exhaust the recursive backend's watch budget before downstream predicates
- * run.
+ * exclusions are retained on the list so a watcher backend that can safely
+ * apply them has them available.
+ *
+ * Today's only consumer, `toParcelIgnorePaths` in `file-watcher.ts`, applies
+ * just the `WATCHER_STRUCTURAL_IGNORE_DIRS` subset: @parcel/watcher matches a
+ * glob with a recursive `std::regex` that overruns its thread stack on a long
+ * path, so nothing pattern-shaped may be handed to it. The retention and
+ * sanitization here therefore constrain a list that is narrowed again before
+ * use — kept because it is what makes the list safe to widen from.
*/
const WATCHER_CARVE_OUT_CAPABLE_DIRS = new Set<string>([OK_DIR, ...EDITOR_HOST_DIRS]);
@@ -318,25 +324,32 @@ const WATCHER_SAFE_BUILTIN_SKIP_DIRS = new Set(
[...BUILTIN_SKIP_DIRS].filter((dir) => !WATCHER_CARVE_OUT_CAPABLE_DIRS.has(dir)),
);
-const WATCHER_STRUCTURAL_IGNORE_GLOBS = [
+/**
+ * The directory roots whose ENTIRE subtree the content predicates reject with
+ * no carve-out reachable inside — `.ok`'s two always-skip children rather than
+ * `.ok` itself, which does admit skills, templates, and the shareable sync
+ * artifacts.
+ *
+ * That unconditional-rejection property is what lets a consumer treat one of
+ * these as a PREFIX (everything at or below it is ignorable), which is
+ * stronger than the per-path glob match below and is not true of the ordinary
+ * user exclusions the list is otherwise built from. `toParcelIgnorePaths` in
+ * `file-watcher.ts` depends on it.
+ */
+export const WATCHER_STRUCTURAL_IGNORE_DIRS = [
'.git',
- '.git/**',
- '**/.git',
- '**/.git/**',
'node_modules',
- 'node_modules/**',
- '**/node_modules',
- '**/node_modules/**',
'.ok/local',
- '.ok/local/**',
- '**/.ok/local',
- '**/.ok/local/**',
'.ok/worktrees',
- '.ok/worktrees/**',
- '**/.ok/worktrees',
- '**/.ok/worktrees/**',
] as const;
+const WATCHER_STRUCTURAL_IGNORE_GLOBS = WATCHER_STRUCTURAL_IGNORE_DIRS.flatMap((dir) => [
+ dir,
+ `${dir}/**`,
+ `**/${dir}`,
+ `**/${dir}/**`,
+]);
+
function watcherPatternIsConfinedToAlwaysSkippedTree(pattern: string): boolean {
const segments = pattern.replace(/^\/+/, '').replace(/\/+$/, '').split('/');
if (segments.includes('..')) return false;
@@ -1068,7 +1081,13 @@ export interface ContentFilter {
* directory without a sibling `.md` file).
*/
isPathIgnored(relativePath: string, opts?: ContentFilterPathReadOpts): boolean;
- /** Relative glob patterns for @parcel/watcher ignore option (best-effort). */
+ /**
+ * Relative patterns a file-watcher backend MAY pre-filter on (best-effort;
+ * the predicates above stay authoritative). Not passed to a backend as-is —
+ * `toParcelIgnorePaths` narrows this to the prefix-shaped subset, because
+ * @parcel/watcher matches a pattern with a recursive `std::regex` that
+ * overruns its thread stack on a long path.
+ */
getWatcherIgnoreGlobs(): string[];
/** Increment refcount for a directory containing an included .md file. */
incrementMdDir(dir: string): void;
diff --git a/packages/server/src/file-watcher-parcel-ignore.test.ts b/packages/server/src/file-watcher-parcel-ignore.test.ts
new file mode 100644
index 000000000..d69ec0329
--- /dev/null
+++ b/packages/server/src/file-watcher-parcel-ignore.test.ts
@@ -0,0 +1,91 @@
+/**
+ * Pins what `startWatcher` actually hands `@parcel/watcher`.
+ *
+ * The sibling `toParcelIgnorePaths` unit tests cover the narrowing itself, but
+ * they cannot catch the wiring regressing to pass the raw ignore list — a
+ * one-token edit at the `subscribe` call that would restore the native
+ * `std::regex` path and its process-killing stack overrun while leaving every
+ * other test green. So this file intercepts the module and asserts on the
+ * options object itself.
+ *
+ * Lives apart from `file-watcher.test.ts` because the mock is file-scoped and
+ * those tests exercise the real backend.
+ */
+import { mkdirSync, writeFileSync } from 'node:fs';
+import { mkdtemp, rm } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { resolve } from 'node:path';
+import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest';
+import { createContentFilter } from './content-filter.ts';
+import { startWatcher } from './file-watcher.ts';
+
+const subscribeCalls: Array<{ dir: string; ignore: string[] | undefined }> = [];
+
+vi.mock('@parcel/watcher', () => {
+ const subscribe = async (
+ dir: string,
+ _onEvent: unknown,
+ opts?: { ignore?: string[] },
+ ): Promise<{ unsubscribe: () => Promise<void> }> => {
+ subscribeCalls.push({ dir, ignore: opts?.ignore });
+ return { unsubscribe: async () => {} };
+ };
+ return { default: { subscribe }, subscribe };
+});
+
+describe('what startWatcher hands @parcel/watcher as `ignore`', () => {
+ let contentDir: string;
+
+ beforeEach(async () => {
+ subscribeCalls.length = 0;
+ contentDir = await mkdtemp(resolve(tmpdir(), 'ok-parcel-wiring-'));
+ });
+
+ afterEach(async () => {
+ await rm(contentDir, { recursive: true, force: true });
+ });
+
+ test('prefix paths only, roots plus discovered nested dirs, no pattern', async () => {
+ writeFileSync(
+ resolve(contentDir, '.gitignore'),
+ ['dist/', '*.log', 'build/**', '**/*.tmp', 'a[0-9]/'].join('\n'),
+ );
+ mkdirSync(resolve(contentDir, 'packages/app/node_modules'), { recursive: true });
+ mkdirSync(resolve(contentDir, 'notes'), { recursive: true });
+ writeFileSync(resolve(contentDir, 'notes/keep.md'), '# keep\n');
+
+ const filter = createContentFilter({ projectDir: contentDir, contentDir });
+ const watcher = await startWatcher(contentDir, async () => {}, filter, {
+ forceBackend: 'parcel',
+ });
+
+ try {
+ expect(subscribeCalls).toHaveLength(1);
+ const ignore = subscribeCalls[0].ignore ?? [];
+
+ // The invariant the crash fix rests on: nothing `is-glob` would route to
+ // the native regex matcher. Non-empty guards against a vacuous pass.
+ expect(ignore.length).toBeGreaterThan(0);
+ for (const entry of ignore) {
+ expect(entry).not.toMatch(/[*?[\]{}()!+@|\\]/);
+ }
+
+ expect(ignore).toEqual(
+ expect.arrayContaining([
+ '.git',
+ 'node_modules',
+ '.ok/local',
+ '.ok/worktrees',
+ 'packages/app/node_modules',
+ ]),
+ );
+
+ // The user's own patterns stay on the JS side.
+ expect(ignore).not.toContain('dist/');
+ expect(ignore).not.toContain('*.log');
+ expect(ignore).not.toContain('build/**');
+ } finally {
+ await watcher.unsubscribe();
+ }
+ });
+});
diff --git a/packages/server/src/file-watcher.test.ts b/packages/server/src/file-watcher.test.ts
index 57f571517..3b5f3b722 100644
--- a/packages/server/src/file-watcher.test.ts
+++ b/packages/server/src/file-watcher.test.ts
@@ -16,6 +16,7 @@ import {
reconcileFileIndexAfterFilterRebuild,
registerWrite,
startWatcher,
+ toParcelIgnorePaths,
updateFileIndex,
updateLastKnownHash,
writeTracker,
@@ -1872,3 +1873,194 @@ describe('startWatcher symlink handling', () => {
expect(aliasMap.get('alias')).toBe('fresh-target');
});
});
+
+describe('toParcelIgnorePaths', () => {
+ let projectDir: string;
+
+ beforeEach(async () => {
+ projectDir = await mkdtemp(resolve(tmpdir(), 'ok-parcel-ignore-'));
+ });
+
+ afterEach(async () => {
+ await rm(projectDir, { recursive: true, force: true });
+ });
+
+ test('keeps the structural roots a prefix match can stand in for', () => {
+ const filter = createContentFilter({ projectDir, contentDir: projectDir });
+
+ expect(toParcelIgnorePaths(filter.getWatcherIgnoreGlobs()).sort()).toEqual([
+ '.git',
+ '.ok/local',
+ '.ok/worktrees',
+ 'node_modules',
+ ]);
+ });
+
+ test('drops the recursive and children forms of those same roots', () => {
+ // The bare root covers the `<dir>/**` form by prefix. The `**/` forms are
+ // dropped outright, since a prefix cannot express "at any depth" — the
+ // seed walk's discovered occurrences stand in for them (below).
+ expect(
+ toParcelIgnorePaths([
+ '.git',
+ '.git/**',
+ '**/.git',
+ '**/.git/**',
+ 'node_modules',
+ 'node_modules/**',
+ '**/node_modules',
+ '**/node_modules/**',
+ ]),
+ ).toEqual(['.git', 'node_modules']);
+ });
+
+ test('drops ordinary user exclusions, whose subtrees can still admit content', () => {
+ writeFileSync(resolve(projectDir, '.gitignore'), 'dist/\ntmp/\n');
+ writeFileSync(resolve(projectDir, '.okignore'), 'drafts/\n');
+
+ const filter = createContentFilter({ projectDir, contentDir: projectDir });
+ const ignorePaths = toParcelIgnorePaths(filter.getWatcherIgnoreGlobs());
+
+ // Pin that they reach the filter's list at all, so the assertions below
+ // cannot pass by the patterns having been stripped further upstream.
+ expect(filter.getWatcherIgnoreGlobs()).toEqual(
+ expect.arrayContaining(['dist/', 'tmp/', 'drafts/']),
+ );
+ expect(ignorePaths).not.toContain('dist/');
+ expect(ignorePaths).not.toContain('tmp/');
+ expect(ignorePaths).not.toContain('drafts/');
+ });
+
+ test('never hands @parcel/watcher an entry its native matcher reads as a glob', () => {
+ writeFileSync(
+ resolve(projectDir, '.gitignore'),
+ ['dist/', '*.log', 'build/**', 'coverage', '**/*.tmp', 'a[0-9]/', 'x{1,2}/'].join('\n'),
+ );
+
+ const filter = createContentFilter({ projectDir, contentDir: projectDir });
+ const ignorePaths = toParcelIgnorePaths(filter.getWatcherIgnoreGlobs());
+
+ // A glob-shaped entry compiles to a std::regex the backend evaluates per
+ // event path, and that match recurses deep enough on a long path to kill
+ // the process outright — so the list must stay prefix-only. Non-empty
+ // guards against a vacuous pass.
+ expect(ignorePaths.length).toBeGreaterThan(0);
+ for (const entry of ignorePaths) {
+ expect(entry).not.toMatch(/[*?[\]{}()!+@|\\]/);
+ }
+ });
+});
+
+describe('structural-ignore discovery feeds the parcel prefix list', () => {
+ let contentDir: string;
+
+ beforeEach(async () => {
+ contentDir = await mkdtemp(resolve(tmpdir(), 'ok-structural-discovery-'));
+ });
+
+ afterEach(async () => {
+ await rm(contentDir, { recursive: true, force: true });
+ });
+
+ test('merges discovered nested occurrences with the structural roots', () => {
+ const merged = toParcelIgnorePaths(
+ ['.git', 'node_modules', '.ok/local', '.ok/worktrees', '**/node_modules/**', 'dist/'],
+ ['packages/app/node_modules', 'vendor/lib/.git', 'sub/.ok/local'],
+ );
+
+ expect(merged).toEqual(
+ expect.arrayContaining([
+ '.git',
+ 'node_modules',
+ 'packages/app/node_modules',
+ 'vendor/lib/.git',
+ 'sub/.ok/local',
+ ]),
+ );
+ expect(merged).not.toContain('**/node_modules/**');
+ expect(merged).not.toContain('dist/');
+ });
+
+ test('drops a discovered directory whose real name contains a glob metacharacter', () => {
+ // Unlike the structural roots, these paths come off disk, so this guard is
+ // reachable: `is-glob` would route `a[0-9]/node_modules` to the crashing
+ // regex branch. Losing its prune is the safe trade.
+ const merged = toParcelIgnorePaths(
+ ['.git', 'node_modules'],
+ ['a[0-9]/node_modules', 'ok-dir/node_modules'],
+ );
+
+ expect(merged).toContain('ok-dir/node_modules');
+ expect(merged).not.toContain('a[0-9]/node_modules');
+ });
+
+ test('the seed walk records nested structural dirs it prunes', async () => {
+ mkdirSync(resolve(contentDir, 'packages/app/node_modules/pkg'), { recursive: true });
+ mkdirSync(resolve(contentDir, 'docs/lib/.git/objects'), { recursive: true });
+ mkdirSync(resolve(contentDir, 'notes'), { recursive: true });
+ writeFileSync(resolve(contentDir, 'notes/keep.md'), '# keep\n');
+ writeFileSync(resolve(contentDir, 'packages/app/node_modules/pkg/index.md'), 'dep\n');
+
+ const filter = createContentFilter({ projectDir: contentDir, contentDir });
+ const watcher = await startWatcher(contentDir, async () => {}, filter, {
+ forceBackend: 'chokidar',
+ });
+ try {
+ // The walk admits ordinary content and prunes the structural trees.
+ expect([...watcher.getFileIndex().keys()]).toContain('notes/keep');
+ expect([...watcher.getFileIndex().keys()]).not.toContain(
+ 'packages/app/node_modules/pkg/index',
+ );
+ // …and records both pruned structural roots so the parcel backend can
+ // skip the same subtrees by prefix.
+ expect([...watcher.getStructuralIgnoreDirs()].sort()).toEqual([
+ 'docs/lib/.git',
+ 'packages/app/node_modules',
+ ]);
+ } finally {
+ await watcher.unsubscribe();
+ }
+ });
+
+ test('records a nested two-segment root, whose match is a suffix not a segment', async () => {
+ // `.ok/local` and `.ok/worktrees` are matched by whole-entry suffix rather
+ // than by last segment, so a nested one exercises a different branch than
+ // `node_modules` does. A bare nested `.ok` must NOT be recorded — it holds
+ // admitted skills and templates.
+ mkdirSync(resolve(contentDir, 'packages/sub/.ok/local/logs'), { recursive: true });
+ mkdirSync(resolve(contentDir, 'packages/sub/.ok/templates'), { recursive: true });
+ writeFileSync(resolve(contentDir, 'packages/sub/.ok/templates/note.md'), '# tmpl\n');
+
+ const filter = createContentFilter({ projectDir: contentDir, contentDir });
+ const watcher = await startWatcher(contentDir, async () => {}, filter, {
+ forceBackend: 'chokidar',
+ });
+ try {
+ const discovered = [...watcher.getStructuralIgnoreDirs()];
+ expect(discovered).toContain('packages/sub/.ok/local');
+ expect(discovered).not.toContain('packages/sub/.ok');
+ } finally {
+ await watcher.unsubscribe();
+ }
+ });
+
+ test('discovery stops at a non-structural excluded directory', async () => {
+ // `vendor` is a builtin skip dir, so the walk prunes there and never
+ // reaches the `node_modules` beneath it. That subtree is not handed to
+ // parcel as a prefix — `vendor` itself cannot be, since an in-place skill
+ // root may sit under it and `isDirExcluded` admits skill ancestors ahead
+ // of the builtin skip list.
+ mkdirSync(resolve(contentDir, 'vendor/lib/node_modules/pkg'), { recursive: true });
+ mkdirSync(resolve(contentDir, 'app/node_modules'), { recursive: true });
+
+ const filter = createContentFilter({ projectDir: contentDir, contentDir });
+ const watcher = await startWatcher(contentDir, async () => {}, filter, {
+ forceBackend: 'chokidar',
+ });
+ try {
+ expect([...watcher.getStructuralIgnoreDirs()]).toEqual(['app/node_modules']);
+ } finally {
+ await watcher.unsubscribe();
+ }
+ });
+});
diff --git a/packages/server/src/file-watcher.ts b/packages/server/src/file-watcher.ts
index f46a5189e..638fd8565 100644
--- a/packages/server/src/file-watcher.ts
+++ b/packages/server/src/file-watcher.ts
@@ -27,7 +27,7 @@ import { lstat, readdir, readFile, realpath, stat } from 'node:fs/promises';
import { dirname, extname, join, relative } from 'node:path';
import { LINKABLE_ASSET_EXTENSIONS } from '@inkeep/open-knowledge-core';
import { isConfigDoc, isReservedForUserTree, isSystemDoc } from './cc1-broadcast.ts';
-import type { ContentFilter } from './content-filter.ts';
+import { type ContentFilter, WATCHER_STRUCTURAL_IGNORE_DIRS } from './content-filter.ts';
import { isWithinContentDir } from './content-path.ts';
import {
forgetDocExtension,
@@ -275,6 +275,15 @@ export interface WatcherHandle {
getAliasMap: () => ReadonlyMap<string, string>;
/** Map from alias folder docName → canonical folder docName (directory symlinks). */
getFolderAliasIndex: () => ReadonlyMap<string, string>;
+ /**
+ * The structural-ignore roots the startup seed walk found below the content
+ * root, contentDir-relative. These are what the parcel backend prunes by
+ * prefix in place of the `**\/node_modules/**`-style globs it cannot be
+ * given (`toParcelIgnorePaths`), so reading it is how you tell whether a
+ * nested `node_modules` is actually being kept out of the watch.
+ * Snapshot-at-startup: a directory created later is not added.
+ */
+ getStructuralIgnoreDirs: () => ReadonlySet<string>;
/**
* Apply a `DiskEvent` to the live file index synchronously. The typed
* mutator API for handlers that need to keep `/api/documents` consistent
@@ -890,6 +899,9 @@ async function seedLastKnownHashes(
aliasMap: Map<string, string>,
folderAliasIndex: Map<string, string>,
visitedInodes?: Set<number>,
+ /** Collects every structural-ignore-root occurrence the walk prunes, so the
+ * parcel backend can prune the same subtrees without a glob. */
+ structuralIgnoreDirs?: Set<string>,
): Promise<void> {
const visited = visitedInodes ?? new Set<number>();
try {
@@ -979,6 +991,7 @@ async function seedLastKnownHashes(
aliasMap,
folderAliasIndex,
visited,
+ structuralIgnoreDirs,
);
} else if (canonStat.isFile() && isSupportedDocFile(entry.name)) {
if (contentFilter) {
@@ -1051,7 +1064,13 @@ async function seedLastKnownHashes(
} else if (lst.isDirectory()) {
const relPath = contentRelativePath(contentDir, fullPath);
if (contentFilter) {
- if (!relPath || contentFilter.isDirExcluded(relPath)) continue;
+ if (!relPath || contentFilter.isDirExcluded(relPath)) {
+ // This walk stops here, and so should the watcher's — record the
+ // ones it can prune by prefix before dropping the subtree.
+ const structural = relPath && structuralIgnoreOccurrence(relPath);
+ if (structural) structuralIgnoreDirs?.add(structural);
+ continue;
+ }
}
upsertFolderIndexEntry(folderIndex, contentDir, fullPath, lst);
await seedLastKnownHashes(
@@ -1063,6 +1082,7 @@ async function seedLastKnownHashes(
aliasMap,
folderAliasIndex,
visited,
+ structuralIgnoreDirs,
);
} else if (lst.isFile() && isSupportedDocFile(entry.name)) {
if (visited.has(lst.ino)) continue;
@@ -1803,6 +1823,99 @@ function _fileWatcherDropsCounter() {
// ─── Backend: @parcel/watcher ───────────────────────────────────────────────
+/** Any character `is-glob` could read as a pattern, which is what routes an
+ * `ignore` entry to the native regex matcher instead of the prefix compare. */
+const GLOB_METACHARACTER_RE = /[*?[\]{}()!+@|\\]/;
+
+/**
+ * The contentDir-relative path of a directory that IS an occurrence of a
+ * structural ignore root, at any depth — `packages/app/node_modules`, a nested
+ * `.git`, `sub/.ok/local`. `null` for anything else.
+ *
+ * The seed walk uses this to enumerate what a plain prefix cannot express.
+ * `WATCHER_STRUCTURAL_IGNORE_DIRS` covers one- and two-segment roots, so the
+ * suffix test is against the whole entry rather than the last segment.
+ */
+function structuralIgnoreOccurrence(relativePath: string): string | null {
+ for (const dir of WATCHER_STRUCTURAL_IGNORE_DIRS) {
+ if (relativePath === dir || relativePath.endsWith(`/${dir}`)) return relativePath;
+ }
+ return null;
+}
+
+/**
+ * Build the `ignore` list for `@parcel/watcher` out of entries its native
+ * matcher can handle WITHOUT a regex.
+ *
+ * The native backend splits `ignore` in two: a glob-shaped entry is compiled to
+ * a `std::regex` and evaluated with `std::regex_match` against every path it
+ * walks or reports, while a plain path is resolved to one absolute prefix and
+ * compared with `std::string::compare`. Only the first form is dangerous —
+ * `std::regex_match` recurses proportionally to the subject's UTF-8 BYTE
+ * length and runs on the backend's own thread, so a long enough path overruns
+ * that thread's stack and takes the process down where no JS `catch` can
+ * see it (`STATUS_STACK_BUFFER_OVERRUN` / exit `0xC0000409` on Windows,
+ * `SIGSEGV` elsewhere). Bytes are what count, not characters, so a directory
+ * named in a non-Latin script trips the limit at roughly a third of the path
+ * length an ASCII one needs — which is why this surfaced as "non-ASCII project
+ * paths crash on startup" (parcel-bundler/watcher#250).
+ *
+ * The prefix form is therefore the only form handed over, and only for
+ * `WATCHER_STRUCTURAL_IGNORE_DIRS`: a prefix ignores an entire subtree, a
+ * strictly stronger claim than the per-path glob it replaces, and one that
+ * holds only because those roots have no carve-out inside them. An ordinary
+ * user exclusion does not qualify — a prefix-ignored `drafts/` hides the
+ * admitted `drafts/.ok/templates/*.md` — so those are left to the
+ * ContentFilter, which `handleRawEvents` consults on every surviving event
+ * anyway.
+ *
+ * Leaving them there is a pre-filter decision, not a correctness one, with one
+ * asymmetry worth knowing: `onRawBatch` runs ahead of `handleRawEvents` and is
+ * gated on its own skills-path pattern rather than on the ContentFilter. An
+ * event matching that pattern but no structural prefix therefore reaches the
+ * debounced in-place-skill re-scan even where the ContentFilter would drop it.
+ * The re-scan broadcasts nothing unless the canonical skill set changed, so
+ * the cost is a wasted scan, not a wrong result.
+ *
+ * A prefix cannot express "at any depth", so `discoveredStructuralDirs`
+ * carries the nested occurrences the seed walk found, keeping the pruning the
+ * dropped `**\/node_modules/**` globs used to do. That pruning is load-bearing
+ * rather than cosmetic: parcel's Unix walk skips an ignored directory outright
+ * (`FTS_SKIP`) and records every surviving entry in an in-memory tree, one
+ * inotify watch each on Linux.
+ *
+ * Those paths come off disk, so unlike the structural roots they can genuinely
+ * contain a glob metacharacter — a directory really named `a[0-9]`. Such an
+ * entry is dropped rather than handed over, since `is-glob` would route it
+ * straight back to the crashing branch.
+ *
+ * Two bounds on the discovered set, both from it riding the seed walk. It is a
+ * startup snapshot, so a `node_modules` created later is not pruned until the
+ * next start. And the walk stops at every excluded directory, not just the
+ * structural ones, so a structural root below (say) `vendor/` is never
+ * reached; `vendor` itself cannot stand in for it, because an in-place skill
+ * root can sit anywhere and `isDirExcluded` admits its ancestors ahead of the
+ * builtin skip list.
+ *
+ * That same ordering is what keeps the set free of admitted content: an entry
+ * is recorded only where the walk itself refused to descend, so a directory
+ * holding anything the ContentFilter admits is never a candidate.
+ *
+ * Nothing is recorded for a structural root reachable only through a symlink,
+ * and nothing needs to be: parcel walks with `FTS_PHYSICAL` and never descends
+ * one. A canonical target inside `contentDir` is reached by its own route and
+ * recorded there; one outside is refused as a symlink escape.
+ */
+export function toParcelIgnorePaths(
+ watcherIgnoreGlobs: readonly string[],
+ discoveredStructuralDirs: Iterable<string> = [],
+): string[] {
+ const prefixIgnorable = new Set<string>(WATCHER_STRUCTURAL_IGNORE_DIRS);
+ const entries = new Set(watcherIgnoreGlobs.filter((entry) => prefixIgnorable.has(entry)));
+ for (const dir of discoveredStructuralDirs) entries.add(dir);
+ return [...entries].filter((entry) => !GLOB_METACHARACTER_RE.test(entry));
+}
+
async function startParcelWatcher(
contentDir: string,
contentFilter: ContentFilter | undefined,
@@ -1812,6 +1925,7 @@ async function startParcelWatcher(
aliasMap: Map<string, string>,
onAfterMutation: () => void,
onRawBatch?: (absPaths: readonly string[]) => void,
+ discoveredStructuralDirs?: ReadonlySet<string>,
): Promise<AsyncSubscription | null> {
let parcel: typeof import('@parcel/watcher');
try {
@@ -1830,7 +1944,12 @@ async function startParcelWatcher(
try {
const subscribeOpts = contentFilter
- ? { ignore: contentFilter.getWatcherIgnoreGlobs() }
+ ? {
+ ignore: toParcelIgnorePaths(
+ contentFilter.getWatcherIgnoreGlobs(),
+ discoveredStructuralDirs ?? [],
+ ),
+ }
: undefined;
const subscription = await parcel.subscribe(
@@ -2043,7 +2162,8 @@ async function startChokidarWatcher(
* When a ContentFilter is provided:
* - Excluded files are skipped during the initial scan
* - Excluded events are dropped in classifyEvents
- * - Best-effort ignore globs are passed to @parcel/watcher
+ * - The prefix-ignorable subset of the ignore list is passed to
+ * @parcel/watcher as a best-effort pre-filter (see `toParcelIgnorePaths`)
*
* Returns a WatcherHandle with unsubscribe() and getFileIndex().
*/
@@ -2088,6 +2208,11 @@ export async function startWatcher(
fileIndexGeneration++;
};
+ // Filled by the seed walk below, read by `startParcelWatcher` after it — the
+ // ordering is what lets the parcel backend prune nested structural dirs by
+ // prefix instead of by the glob that crashes its matcher.
+ const structuralIgnoreDirs = new Set<string>();
+
await seedLastKnownHashes(
contentDir,
contentDir,
@@ -2096,6 +2221,8 @@ export async function startWatcher(
folderIndex,
aliasMap,
folderAliasIndex,
+ undefined,
+ structuralIgnoreDirs,
);
bumpFileIndexGeneration();
@@ -2126,6 +2253,7 @@ export async function startWatcher(
aliasMap,
bumpFileIndexGeneration,
onRawBatch,
+ structuralIgnoreDirs,
);
if (parcelSub) {
subscription = parcelSub;
@@ -2202,6 +2330,9 @@ export async function startWatcher(
getFolderAliasIndex() {
return folderAliasIndex;
},
+ getStructuralIgnoreDirs() {
+ return structuralIgnoreDirs;
+ },
mutateFileIndex(event) {
updateFileIndex(event, fileIndex);
bumpFileIndexGeneration();