Understand any codebase instantly. System intelligence for codebases, built for humans and AI.
TypeScript aiai-memoryclaude-codeclicode-analysiscode-mappingcode-memorycodebase-understandingcodexdeveloper-toolsllmopenclawpersistent-memoryprogram-analysissoftware-architecturesystem-intelligencesystem-mapping
docs(ingest): correct the segfault's blast radius, measured end to end (#650)
Joseph Mikhail7171c3d68cc4
diff --git a/core-ingestion/src/index.ts b/core-ingestion/src/index.tsindex 5cfd7cc2..bff87d7e 100644--- a/core-ingestion/src/index.ts+++ b/core-ingestion/src/index.ts@@ -13,6 +13,7 @@ import Parser from 'tree-sitter';// fresh process on Node 26://// tree-sitter-c-sharp required type=module -> 1+// tree-sitter-powershell required type=module -> 1// @tree-sitter-grammars/tree-sitter-lua optional type=module -> 1// tree-sitter-css optional type=module -> 1// the other 11 required grammars required type=commonjs -> 0@@ -20,14 +21,20 @@ import Parser from 'tree-sitter';// tree-sitter-sas optional type=module -> 0// (full-filename main + "exports")//-// The two 11s are a coincidence, not a copy-paste: 12 required grammars minus-// c-sharp, and 14 optional minus lua, css and sas.+// The two 11s are a coincidence, not a copy-paste: 13 required grammars minus+// c-sharp and powershell, and 14 optional minus lua, css and sas.//// So the rule is `"type": "module"` + a directory `"main"` + no `"exports"`,-// which today means exactly c-sharp (static, below), plus css and lua (dynamic,-// through tryImportGrammar further down). The parse pool loads this module once-// per worker and every worker shares the parent's stderr, so each warning-// arrives once per worker and shreds the `ix map` progress bar.+// which today means c-sharp (static, below) plus powershell, css and lua+// (dynamic, through tryImportGrammar further down). The parse pool loads this+// module once per worker and every worker shares the parent's stderr, so each+// warning arrives once per worker and shreds the `ix map` progress bar.+//+// powershell was added in Ix#595 and this table was not updated: it said "12+// required grammars" and named the shape as "exactly c-sharp, plus css and+// lua", while the comment on the powershell import ~90 lines below has always+// said it is "the same shape". The table was wrong on both counts, and Ix#650+// -- a change about these files disagreeing on counts -- is where it showed.//// Do NOT pre-emptively convert the CJS grammars "for consistency". A subpath// import is a dependency on a package's internal layout, and it fails *hard* —diff --git a/core-ingestion/src/parse-worker.ts b/core-ingestion/src/parse-worker.tsindex e99fa73e..56800dfd 100644--- a/core-ingestion/src/parse-worker.ts+++ b/core-ingestion/src/parse-worker.ts@@ -1,6 +1,7 @@/*** Worker thread entry point for parallel file parsing.- * Each worker maintains its own Parser singleton (safe — module state is per-thread).+ * Each worker maintains its own Parser singleton (safe — module state is+ * per-thread).* Receives: { filePath: string, source: string } | { __shutdown: true }* Posts: { ok: true, result: FileParseResult } | { ok: false }*/@@ -15,13 +16,47 @@ if (!parentPort) throw new Error('parse-worker must run inside a worker thread')* `Worker.terminate()` tears a thread down from the outside, and doing that to* a thread that has loaded the tree-sitter native bindings SEGFAULTS the whole* process -- the parses need not even be in flight, an idle worker that has- * parsed once is enough. Measured on Windows/Node 26 at ~8% per teardown, and- * the pool terminates every worker at the end of every run, so roughly one- * `ix map` in twelve exited 139 after a completely successful ingest: patches- * committed, summary printed, non-zero status for anything reading `$?`.+ * parsed once is enough.+ *+ * How it presents: exit 139 after a completely successful ingest -- patches+ * committed, summary printed, and a non-zero `$?` for anything that reads it.+ * If you are here because a clean `ix map` exited 139, this is it.+ *+ * NEVER-DISPATCHED WORKERS ARE NOT EXEMPT. The addon is held from module+ * EVALUATION, not from the first parse: `index.ts` resolves its grammars at+ * module scope and this file imports it statically, so a worker reaches that+ * state on its own without being dispatched. `tree-sitter` and the twelve+ * grammars are `index.ts`'s own static imports, near the top of its dependency+ * graph, so the isolate holds them within moments of `new Worker()` -- long+ * before that file's body or its top-level `await`s run. A worker that has+ * parsed nothing, or that is still suspended at one of those awaits, holds+ * them all the same. That is the precondition the crash needs, so an+ * undispatched worker is not known to be safe to terminate.+ *+ * Whether it is as dangerous as a parsed one has never been measured: the one+ * arm that looked -- spawn-then-destroy, 0 of 120 -- destroyed its pool with+ * no wait for an ack or an `'online'` event, so it tore threads down before+ * they had finished loading: a different population.+ *+ * `parse-pool.test.ts` used to say the opposite -- "an untouched worker has+ * not loaded the addon" -- which is what made a `terminate()` fast path for+ * undispatched workers look safe. That is the claim being retracted; it was+ * never in this file.+ *+ * Rates, populations and the reasoning are in `docs/parse-pool-teardown.md`,+ * which is versioned and travels with a clone. Do not reconstruct them from+ * this file's history: `main` squash-merges, so the log here also carries+ * #598, whose figures this work retracts, and the squashed body concatenates+ * every superseded value beside its correction.+ *+ * They are deliberately not restated here. Keeping a statistical write-up+ * consistent across three source files produced more defects over successive+ * reviews than it prevented, and none of it changes what this code must do.** Closing the port from INSIDE lets the thread unwind its own event loop and- * dispose its isolate in order. Measured 0 crashes in the same experiment.+ * dispose its isolate in order, and did not crash once where terminating+ * crashed most runs -- the comparison is on `ParsePool.shutdown`, in+ * `ix-cli/src/cli/commands/parse-pool.ts`.** The teardown contract, precisely, because the pool's half of it changed and* this is the only place the worker states it: `ParsePool` never terminates adiff --git a/docs/README.md b/docs/README.mdindex 707e30bd..73d62c3b 100644--- a/docs/README.md+++ b/docs/README.md@@ -3,4 +3,5 @@- [HTTP API reference](api/) — the backend JSON API exposed on `http://localhost:8090`- [`--format llm` output convention](llm-format.md) — the machine-readable record stream- [Consolidating the per-host plugins onto `ix mcp`](mcp-plugin-consolidation.md)-- [System prerequisites](prerequisites.md) — what the installer checks for and installs\ No newline at end of file+- [System prerequisites](prerequisites.md) — what the installer checks for and installs+- [Parse-pool teardown](parse-pool-teardown.md) — why nothing calls `terminate()`, and what was measureddiff --git a/docs/parse-pool-teardown.md b/docs/parse-pool-teardown.mdnew file mode 100644index 00000000..d22659f1--- /dev/null+++ b/docs/parse-pool-teardown.md@@ -0,0 +1,241 @@+# Parse-pool teardown: why nothing calls `terminate()`++`ParsePool` never terminates a worker. It asks (`__shutdown`), waits, and+`unref()`s one that will not answer. This file is the durable record of why,+and of what was and was not measured. The code carries the rule; the numbers+live here so they do not have to be kept consistent across three source files.++Origin: Ix#598 (the fix) and Ix#650 (this correction). Windows / Node 26,+against a live backend.++The harness and real-ingest arms are from the pre-fix build `084f472` --+necessarily, since they need the `terminate()` teardown that #598 removed. The+vitest consistency check below is NOT: it needs the current tree, and its+"36 of 38" is today's file count. At `084f472` the suite was smaller, so+anyone reproducing that arm by checking out `084f472` will count differently+and should redo the arithmetic with what they find.++## The bug++`Worker.terminate()` on a thread that has loaded the tree-sitter native+bindings segfaults the **process**. It presents as **exit 139 after a+completely successful ingest** — patches committed, summary printed, and a+non-zero `$?` for anything that reads it.++The comparison that settles the verb, twenty pool teardowns per process, six+runs of each variant. Those six `terminate()` runs are **not a separate+experiment**: they are one of the three arms pooled into the 19-of-28 figure+below (7 of 12, 7 of 10, and these 5 of 6). Read on their own they imply 8.6%+per POOL teardown (every rate in this file is per pool unless it says+per-isolate; for this 21-worker pool the two differ by ~20×, and the+conversion is worked through below), which is close to the retracted "one in+twelve" — that is the+hazard of quoting a single arm, and the reason the fit below uses all of+them.++| teardown | result |+|---|---|+| `terminate()` | 5 of 6 runs died with SIGSEGV (139) |+| worker closes its own port | 0 of 6 |+| worker calls `process.exit(0)` | 0 of 6 |++## How often++Two populations, and they differ by more than an order of magnitude. All of+this is for a **21-worker pool** (`os.cpus().length - 1` on the measuring+machine); exposure depends on how many addon-loaded isolates are disposed, so+none of it carries to a pool of another size. CI runners are 3–4 vCPU, giving+pools of 2–3.++**Minimal harness** — nothing else in the process:++| configuration | result |+|---|---|+| twenty teardowns per process | 19 of 28 runs crashed |+| one teardown then exit | 5 of 40 |++Both arms fit one rate: **6.3% per POOL teardown, 95% CI 4.1–9.3%** — that+is one teardown disposing all 21 of the pool's isolates, not one isolate. The+per-isolate hazard is 0.31%; see the vitest section for why the difference+matters. Pooling is+licensed by asking whether the one-teardown arm contradicts the+twenty-teardown arm's own rate (5.5%, again per pool):+`P(≥5 of 40 | p=0.055) = 0.067` — a failure to reject, borderline, not a+demonstration of agreement.++**Real ingests** — `ingest-files.test.ts` under vitest, ~9.5 ingests per+process at the time of measuring (it drives 14 today):++| configuration | result | per POOL teardown |+|---|---|---|+| idle machine | 2 of 75 processes | 0.28% |+| loaded machine | 9 of 50 processes | 2.1% |++(Per pool, as everywhere else here — not per isolate. 0.28% sits next to the+0.31% per-ISOLATE hazard derived below, and they are not the same quantity.)++Fisher exact on 2/75 vs 9/50 is **p = 0.007** two-sided (0.004 one-sided), so+load matters. A real `ix ingest` of 300 files was **0 of 60** — that is *one*+teardown per process, and `P(zero in 60)` is 0.84 at the idle rate, 0.29 at the+loaded one. Unremarkable under either.++## The grammar tally++The source comments defer here for this, so it has to actually be here.+`core-ingestion/src/index.ts` resolves **27 grammars** plus the tree-sitter+core:++| how it loads | count | can it be absent? |+|---|---|---|+| static `import` | 12 | no -- evaluation fails if the package is missing |+| `tryLoadGrammar` (sync helper) | 11 | yes, returns null |+| `tryImportGrammar` (async helper) | 4 | yes, returns null |++Cross-cutting that, `package.json` has **13 required** grammars and 14+optional. The two splits do not line up: `tree-sitter-powershell` is a+REQUIRED dependency that still loads through the async helper, which is why+the guaranteed floor is "the core plus the twelve static ones" and not "the 13+required". That one package is the whole reason the distinction is worth+writing down.++## What this does and does not establish++The minimal harness overstates real exposure by **22×** on the only+load-matched comparison available (idle vs idle). Against the loaded real rate+it is 3.1×, but that mixes conditions — the harness was never run loaded, and+load raises the rate — so treat 3.1× as a lower bound on what is unexplained,+not the residue after subtracting load.++**Parses per worker is not controlled.** `ingestFiles` parses a `.ts` file+twice (index prescan, then streaming loop, both on the same pool), so file+counts double; and `ingest-files.test.ts` is not uniformly 30 files — it calls+`fixture(30)` eight times, `fixture(12)` once and `fixture(4)` twice, giving+0.38 to 2.9 parses per worker across its runs and straddling the harness's+~1.4. An earlier claim that "the harness parses the least per worker and+crashes the most" is therefore false for several of the teardowns behind the+data. This remains an open confound.++**The addon is held from module evaluation, not from the first parse.**+`core-ingestion/src/index.ts` resolves grammars at module scope and+`parse-worker.ts` imports it statically, so a worker reaches that state on its+own, without being dispatched. Undispatched workers at teardown are ordinary,+not exotic: `init()` spawns `concurrency` of them up front, so any batch+smaller than the pool leaves some that never received a task. (The addon-free+window is much narrower than "before the first parse": `tree-sitter` and the+twelve grammars are `index.ts`'s own static imports, near the top of its+dependency graph, so the isolate holds them within moments of `new Worker()`+and long before `index.ts`'s body — let alone its top-level `await`s —+runs. A worker suspended at one of those awaits already holds all twelve.) Once+evaluated, a worker holds the core+plus the **twelve statically imported** grammars — not "13 required", because+`tree-sitter-powershell` is a required dependency that nonetheless loads+through a null-returning helper. That is the precondition the crash needs, so+an undispatched worker is **not known to be safe to terminate**. Workers that+had parsed were the ones observed to crash, and a spawn-then-destroy arm went+0 of 120 teardowns. **Do not use that arm.** Its harness calls `pool.init()`+and then `await pool.destroy()` with nothing in between — no wait for an ack+or an `'online'` event — so it tore the workers down inside the addon-free+window+described above. It measured threads that had not finished loading, which is a+different population from "evaluated but never dispatched", and it therefore+says nothing about whether parsing matters. Earlier revisions of this document+read a `p = 4e-4` rejection and a 2.5% upper bound out of it; both are+withdrawn.++So there is **no measurement** distinguishing an evaluated-but-undispatched+worker from a parsed one. The rule rests on the mechanism instead: such a+worker holds the addon, which is the precondition the crash needs, and nothing+has shown it is safe to terminate.++**A consistency check on another consumer that terminates addon-loaded+threads.** `core-ingestion`'s own suite runs `vitest run --pool threads`, and+36 of its 38 test files import `./index.js`. The mechanism differs from the+parse worker's: vitest's worker entry does not import `core-ingestion`, so a+thread picks the addon up when it *evaluates* such a test file, not at spawn —+a thread can be spawned and torn down having run none. Either way the threads+are terminated, which is the exposed shape: vitest 4's `ThreadsPoolWorker`+does `await this.thread.terminate()` in its `stop()`, called per worker.++Not tinypool, which earlier revisions of this file named. Vitest 4 dropped it:+it is absent from `[email protected]`'s dependencies, from both lockfiles and from+`node_modules` entirely. The conclusion was right and the attribution was not,+which is worse than useless in the one document whose job is to be checkable.++Measured: **0 segfault signatures in 10 local runs.** The `core-ingestion+tests` CI step also passes on all five matrix legs, but do not weigh that+equally: CI runs Node 22 and 24, where everything here was measured on Node+26. That is the whole of the objection — the runners' 3–4 vCPUs are NOT a+second reason, because under per-file isolation vCPU count sets how many+threads run at once, not how many are terminated, and it is the latter the+arithmetic below indexes. The 10 local runs are the measurement; CI is a+weaker corroboration.++Either way the result is consistent with everything else here, and it is worth+showing the conversion, because getting it wrong once made this section claim a+falsification it does not support.++The 6.3% is per POOL teardown, and a pool disposes 21 isolates. Treating+those disposals as independent gives `1-(1-h)^21 = 0.063`, i.e. **h = 0.31%**+— a factor of 20.5, and for a pool of 21 that is the only shape the answer can+take: at small h the pool rate is about 21h, so the ratio can approach 21 and+never exceed it. Any conversion factor larger than the pool size is arithmetic+that went wrong, which is how the ~28× an earlier revision quoted here was+caught (it was 8.6% over 0.31% — a single arm's per-pool rate against the+pooled per-isolate one).++**Independence is an assumption, and this document has direct evidence+against it.** The idle-vs-loaded real-ingest rates differ 7× at identical pool+size, and the harness overstates idle real ingests by 22×, so something not in+the model moves the rate. Carrying `h` from the harness onto vitest's+one-at-a-time terminations is also a cross-population transfer — the same move+this file retracts for the "one in twelve" figure. The conclusion survives+either way, which is why it is stated rather than hedged away: redo it with+the real-ingest IDLE hazard instead and `P(0 in 10 runs)` is about 0.95+against 0.33, both unremarkable. Nothing here rests on the exact h.++How many addon-loaded isolates a run terminates is not a guess: `isolate`+defaults to `true`, `core-ingestion` ships no vitest config to change it, and+vitest reuses a runner only when the finished task AND the next queued one are+both non-isolated. Otherwise it stops the worker after the file. So the run+spawns and terminates about one worker per test file — **36 addon-loaded**, of+38 — giving `P(0 crashes in 10 runs) = 0.33`. Unremarkable, and nothing to+explain.++Note this is the count of TERMINATED threads, which is what the hazard applies+to. Machine parallelism (21 here) bounds how many run concurrently and is not+this quantity; an earlier revision offered 10 / 21 / 36 as a bracket over an+unknown, when the config already determined it.++Mixing the two units — applying a per-pool rate to individual isolates — is+the easiest error in this document to make, and the reason every rate here+says which it is.++## The unref-at-exit arm++`shutdown()`'s give-up path leaves an unresponsive worker alive and+`unref()`'d, and the source comments defer here for what that costs.++Measured: **four addon-loaded threads left live and unref'd across process+exit, 0 failures in 10 runs.** On its own that carries very little, and the+comparison to make is not against the `terminate()` row of the table above:+that row is twenty teardowns of a 21-worker pool per run, while this is four+isolates in one teardown per run. At the per-isolate h = 0.31%, four isolates+is ~1.2% per run and `P(0 in 10)` is 0.88 -- so this arm would look exactly+like this whether the path is safe or not. It has no power, and it is recorded+so nobody mistakes it for reassurance.++What carries the argument is the mechanism, and it was probed separately:+spawn a worker, `unref()` it, let the process end, and the parent never+receives an `'exit'` event while the worker's own `process.on('exit')` never+runs -- the process leaves at code 0 with the thread still live. Process exit+does not run the orderly per-worker shutdown that `terminate()` drives. That+is why `unref()` is not simply a deferred `terminate()`.++## Superseded figures++Earlier revisions of this work stated, and then retracted, that roughly one+`ix map` in twelve exited 139. That was the harness rate applied to a consumer+it was never measured on. Do not reconstruct numbers from the commit history:+`main` squash-merges, so the log for these files still carries #598's retracted+figures, and this PR's squashed body concatenates every superseded value beside+its correction. This file is the current record.diff --git a/ix-cli/src/cli/__tests__/parse-pool.test.ts b/ix-cli/src/cli/__tests__/parse-pool.test.tsindex 82494974..376bbed7 100644--- a/ix-cli/src/cli/__tests__/parse-pool.test.ts+++ b/ix-cli/src/cli/__tests__/parse-pool.test.ts@@ -169,15 +169,50 @@ describe("ParsePool", () => {// hang. `Worker.terminate()` tears a thread down from outside, and doing// that to one that has loaded the tree-sitter native bindings segfaults the// PROCESS -- an idle worker that parsed a single file is enough, because- // the crash is in disposing an isolate that still holds the addon. Twenty- // teardowns per process, six runs each, on Windows/Node 26:+ // the crash is in disposing an isolate that still holds the addon. The+ // comparison that settles it is on `ParsePool.shutdown` in+ // `../commands/parse-pool.ts`; rates and populations are in+ // `docs/parse-pool-teardown.md`, not restated here. What matters+ // for this file: a pool built on one of the inline `.mjs` fixtures never+ // imports `core-ingestion`, so its teardown cannot crash, and only a pool+ // pointed at the real built worker can. Stated as a property of the+ // fixture rather than as "only the test at the bottom", which is true+ // today and goes stale silently the first time a second real-worker test+ // is added -- and goes stale in the dangerous direction, telling whoever+ // added it that their teardown cannot crash. Deliberately no tally+ // either, for the same reason: a count goes stale with the suite green.//- // terminate() 5 of 6 runs died with SIGSEGV (139)- // worker closes its own port 0 of 6+ // And on the shipped build nothing here calls `terminate()` at all --+ // Ix#598 removed it. A worker that will not+ // answer `__shutdown` is left alive and `unref()`'d, and process exit+ // does NOT run the shutdown path that `terminate()` drives. Probed three+ // times, in a standalone script rather than through this pool: the parent+ // never receives an 'exit' event for such a worker, and the worker's own+ // `process.on('exit')` never runs -- the process leaves at code 0 with the+ // thread still live.//- // `destroy()` runs in `ingestFiles`'s outermost `finally`, so this was one- // `ix map` in roughly twelve exiting 139 with every patch committed and the- // summary printed -- which is why nobody reported it.+ // Standalone matters. The give-up path calls `removeAllListeners('exit')`+ // before it unrefs, so measured through `ParsePool` the missing event+ // would be guaranteed by construction and would show nothing. It is the+ // worker-side handler that carries this.+ //+ // The give-up branch of `shutdown` in `../commands/parse-pool.ts` says the+ // same thing -- "nobody disposes an isolate that still holds the addon" --+ // though that is in the branch body, not in `shutdown`'s doc header, which+ // only covers how long a wedged worker survives. An earlier revision of+ // THIS comment said the opposite ("its isolate is still disposed when the+ // process exits"), which would have told a reader the give-up path is+ // exposed to the crash. It was the one claim in this PR that two files+ // answered differently.+ //+ // The arm behind it is weak on its own -- four addon-loaded threads+ // unref'd across exit, 0 failures in 10 runs, which at the per-isolate+ // hazard is the expected result either way -- so the mechanism is what+ // carries it, not the count.+ //+ // Nor is the repo clear: `core-ingestion`'s own suite runs on vitest's+ // threads pool, which does terminate threads that have loaded the addon.+ // The doc covers both.//// Asserted through a marker the worker writes when ASKED to go, because the// crash itself is probabilistic: a test that just tore pools down would@@ -201,8 +236,12 @@ describe("ParsePool", () => {const pool = new ParsePool(path, 2);pool.init();- // Parse first: an untouched worker has not loaded the addon, and it is the- // loaded-then-idle thread that crashes.+ // Parse first so the worker is idle-after-work, which is the state+ // `destroy()` meets in a real run. Note this fixture is an inline .mjs that+ // never imports `core-ingestion`, so no addon is loaded here and the crash+ // itself cannot occur -- what is pinned is the MECHANISM, that the pool+ // asks rather than terminates. The real-worker test lower down is the one+ // that runs against the addon.await Promise.all([pool.parse("a.ts", "x"), pool.parse("b.ts", "x")]);await pool.destroy();@@ -263,7 +302,8 @@ describe("ParsePool", () => {const pool = new ParsePool(path, 1, 100);pool.init();const parsing = pool.parse("slow.ts", "x");- // Destroy while it is mid-parse, so the first expiry lands on a BUSY worker.+ // Destroy while it is mid-parse, so the first expiry lands on a BUSY+ // worker.await new Promise(resolve => setTimeout(resolve, 50));const started = Date.now();@@ -281,9 +321,9 @@ describe("ParsePool", () => {// and only then does the worker reach the queued `__shutdown` and start// unwinding its isolate. An expiry landing in that window sees a worker// that looks idle and unresponsive and abandons it -- one that was about to- // answer. Under the old `terminate()` this was the ~8% per-teardown- // segfault itself; the pool unrefs now, so the cost is a lost parse result- // rather than the process, and it is still wrong.+ // answer. Under the old `terminate()` this was the segfault itself; the+ // pool unrefs now, so the cost is a lost parse result rather than the+ // process, and it is still wrong.//// The timings are chosen so the two rules give different answers, which is// the only way to catch this. Grace 300ms, so ticks land at 300/600/900. A@@ -538,9 +578,23 @@ describe("ParsePool", () => {// bound by a factor of three, on any runner.const pool = new ParsePool(real, 2, 10000);pool.init();- // Parse for real, so the threads have the tree-sitter addon loaded -- an- // untouched worker does not, and it is the loaded-then-idle thread that- // crashes under `terminate()`.+ // Parse for real, because having PARSED is what was observed to arm the+ // crash. Not because parsing loads the addon: `index.ts` resolves+ // 27 grammars plus the core at module scope -- twelve by static import, the+ // rest eagerly through helpers -- so these threads hold the core and the+ // twelve within moments of spawn, well before that module finishes+ // evaluating. The other 15 resolve through null-returning helpers and are+ // absent wherever a platform has no prebuild, so the guaranteed floor is+ // the core plus the statically imported twelve -- which is all this test+ // needs. (One of the 15, `tree-sitter-powershell`, is a required+ // dependency that loads through a helper anyway.) The exact tally is in+ // `docs/parse-pool-teardown.md` rather than here.+ //+ // Workers that had parsed were the ones observed to crash under+ // `terminate()`, and spawn-then-destroy was not -- but the mechanism was+ // never isolated, and the addon is held either way, so that does not+ // establish an undispatched worker is safe to terminate. Parse here+ // because the real run does.const results = await Promise.all([pool.parse("a.ts", "export function a(): number { return 1; }"),pool.parse("b.ts", "export function b(): number { return 2; }"),diff --git a/ix-cli/src/cli/commands/parse-pool.ts b/ix-cli/src/cli/commands/parse-pool.tsindex f7714cfb..3b8b8ca4 100644--- a/ix-cli/src/cli/commands/parse-pool.ts+++ b/ix-cli/src/cli/commands/parse-pool.ts@@ -185,16 +185,42 @@ export class ParsePool {* that has loaded the tree-sitter native bindings segfaults the process. The* thread does not have to be busy: an idle worker that has parsed a single* file is enough, because the crash is in disposing an isolate that still- * holds the addon. Measured on Windows/Node 26, twenty pool teardowns per- * process, six runs each:+ * holds the addon.+ *+ * DO NOT add a `terminate()` fast path for workers that have never been+ * dispatched. They hold the addon too: `core-ingestion/src/index.ts`+ * resolves grammars at module scope, and `tree-sitter` plus the twelve+ * static grammars sit near the top of its dependency graph -- so a worker+ * holds them within moments of `new Worker()`, without ever being+ * dispatched, and long before `index.ts`'s own body or its top-level+ * `await`s run. Note which end that pins: the addon-free window CLOSES when+ * those static imports evaluate, not when evaluation finishes. A worker+ * suspended at one of those awaits already holds all twelve, and is not+ * safe to terminate.+ *+ * Not every grammar -- 15 of the 27 go through null-returning helpers and+ * can be absent -- but the core and the twelve static ones are always+ * there, which is the PRECONDITION the crash needs. Whether it is also+ * sufficient has never been measured: the one experiment that looked --+ * spawn-then-destroy, 0 of 120 -- destroyed its pool with no wait for an+ * ack or an `'online'` event, so it tore threads down inside that window,+ * before they had finished loading. Different population; it says nothing+ * about undispatched workers. This is the file where a fast path would be+ * written, which is why the warning is here and not only in the tests.+ *+ * The comparison that settles the verb, on Windows/Node 26, twenty pool+ * teardowns per process, six runs each:** terminate() 5 of 6 runs died with SIGSEGV (139)* worker closes its own port 0 of 6* worker calls process.exit(0) 0 of 6*- * `destroy()` runs in `ingestFiles`'s outermost `finally`, so this was one- * `ix map` in roughly twelve exiting 139 with every patch committed and the- * summary already printed -- invisible unless something reads the status.+ * Per-consumer rates, the populations behind them and the statistics are in+ * `docs/parse-pool-teardown.md` -- versioned, and not this file's history,+ * which still carries #598's retracted figures and, in the squashed body,+ * every superseded value next to its correction. They do not change the+ * rule, and keeping them consistent across three files proved to be its own+ * source of errors.** What the grace period bounds, precisely: the wait for a reply from a* worker that is IDLE and does not answer -- one whose JS event loop is@@ -207,15 +233,20 @@ export class ParsePool {* rule had a counterexample, because `terminate()` is simply the wrong verb* for teardown. It is what segfaults. Against a worker inside a native call* it does not even preempt: V8's termination interrupt is only checked at JS- * boundaries, so it resolves when the call returns on its own (3981ms- * against ~4s of CPU-bound native work, measured), making it slower AND- * crash-prone there. Only a JS-wedged worker dies promptly, in about 2ms.+ * boundaries, so it resolves only when the call returns on its own --+ * measured twice against ~4-4.5s of CPU-bound native work, at 3981ms and+ * 4457ms (two runs of the same experiment, not a discrepancy) -- making it+ * slower AND crash-prone there. Only a JS-wedged worker dies promptly, in+ * about 2ms.** Teardown does not need the thread to die. It needs the pool to stop* waiting on it and the thread to stop holding the process open, which is* exactly `unref()`. Measured on the real parse worker, four addon-loaded- * threads left live and unref'd across process exit: 0 failures in 10 runs,- * against 5 of 6 for `terminate()`.+ * threads left live and unref'd across process exit: 0 failures in 10 runs+ * -- though that arm is four isolates in one teardown, against a table of+ * twenty teardowns of a 21-worker pool, so it carries little on its own.+ * `docs/parse-pool-teardown.md` has the section, including why 0 of 10 has+ * no power here.** So the clocks below decide WHEN to give up, never whether it is safe to* kill -- and the `onError` path needs no special case either, which is what@@ -308,13 +339,14 @@ export class ParsePool {// `terminate()` was the wrong verb for this whole branch. It is what// segfaults -- that is the bug this file exists to fix -- and against a// worker inside a native call it does not even preempt: it resolves- // when the call returns (4457ms against ~4.5s of work, measured), so it- // was strictly slower AND crash-prone there. `unref()` gives the only- // thing teardown actually needs: the thread stops keeping the event- // loop alive, so the CLI exits, and nobody disposes an isolate that- // still holds the addon. Measured on the real parse worker, four+ // only when the call returns, so it was strictly slower AND+ // crash-prone there (the timings are on `shutdown` above). `unref()`+ // gives the only thing teardown needs: the thread stops keeping the+ // event loop alive, so the CLI exits, and nobody disposes an isolate+ // that still holds the addon. Measured on the real parse worker, four// addon-loaded threads left live and unref'd across process exit: 0- // failures in 10 runs, against 5 of 6 for `terminate()`.+ // failures in 10 runs. Note that is one teardown per run, so it is+ // not comparable to `shutdown`'s twenty-teardown table.//// The cost is written up on `shutdown` above: a wedged worker survives// until the process exits. Nothing for the CLI, and `ix watch` runs