Understand any codebase instantly. System intelligence for codebases, built for humans and AI.
TypeScript aiai-memoryclaude-codeclicode-analysiscode-mappingcode-memorycodebase-understandingcodexdeveloper-toolsllmopenclawpersistent-memoryprogram-analysissoftware-architecturesystem-intelligencesystem-mapping
fix(parse-pool): count deaths outside the respawn cap, and correct the teardown doc's divisor (#654)
Joseph Mikhaila82578057dec
diff --git a/core-ingestion/src/index.ts b/core-ingestion/src/index.tsindex bff87d7e..742973b8 100644--- a/core-ingestion/src/index.ts+++ b/core-ingestion/src/index.ts@@ -13,7 +13,7 @@ import Parser from 'tree-sitter';// fresh process on Node 26://// tree-sitter-c-sharp required type=module -> 1-// tree-sitter-powershell required type=module -> 1+// tree-sitter-powershell required type=module -> 1 (*)// @tree-sitter-grammars/tree-sitter-lua optional type=module -> 1// tree-sitter-css optional type=module -> 1// the other 11 required grammars required type=commonjs -> 0@@ -21,6 +21,14 @@ import Parser from 'tree-sitter';// tree-sitter-sas optional type=module -> 0// (full-filename main + "exports")//+// (*) INFERRED, not measured. `tree-sitter-powershell` is not installed in+// every checkout -- it is absent from this one -- so its row comes from the+// import comment below rather than from running Node against the package. CI+// proves the subpath `bindings/node/index.js` resolves; it does not prove the+// bare specifier would warn. Re-measure before relying on it. Marked because+// an unverifiable row in this table is what went stale in #595 and cost+// Ix#650 sixteen review rounds.+//// The two 11s are a coincidence, not a copy-paste: 13 required grammars minus// c-sharp and powershell, and 14 optional minus lua, css and sas.//diff --git a/core-ingestion/src/parse-worker.ts b/core-ingestion/src/parse-worker.tsindex 56800dfd..e9dd3026 100644--- a/core-ingestion/src/parse-worker.ts+++ b/core-ingestion/src/parse-worker.ts@@ -25,9 +25,10 @@ if (!parentPort) throw new Error('parse-worker must run inside a worker thread')* NEVER-DISPATCHED WORKERS ARE NOT EXEMPT. The addon is held from module* EVALUATION, not from the first parse: `index.ts` resolves its grammars at* module scope and this file imports it statically, so a worker reaches that- * state on its own without being dispatched. `tree-sitter` and the twelve- * grammars are `index.ts`'s own static imports, near the top of its dependency- * graph, so the isolate holds them within moments of `new Worker()` -- long+ * state on its own without being dispatched. `tree-sitter` and the statically+ * imported grammars are `index.ts`'s own imports, near the top of its+ * dependency graph, so the isolate holds them within moments of+ * `new Worker()` -- long* before that file's body or its top-level `await`s run. A worker that has* parsed nothing, or that is still suspended at one of those awaits, holds* them all the same. That is the precondition the crash needs, so an@@ -35,8 +36,9 @@ if (!parentPort) throw new Error('parse-worker must run inside a worker thread')** Whether it is as dangerous as a parsed one has never been measured: the one* arm that looked -- spawn-then-destroy, 0 of 120 -- destroyed its pool with- * no wait for an ack or an `'online'` event, so it tore threads down before- * they had finished loading: a different population.+ * no wait for an ack or an `'online'` event, so it very probably tore threads+ * down before they had finished loading: a different population. Inferred+ * from that teardown code, not measured.** `parse-pool.test.ts` used to say the opposite -- "an untouched worker has* not loaded the addon" -- which is what made a `terminate()` fast path fordiff --git a/docs/parse-pool-teardown.md b/docs/parse-pool-teardown.mdindex d22659f1..f8666ff9 100644--- a/docs/parse-pool-teardown.md+++ b/docs/parse-pool-teardown.md@@ -8,12 +8,26 @@ live here so they do not have to be kept consistent across three source files.Origin: Ix#598 (the fix) and Ix#650 (this correction). Windows / Node 26,against a live backend.-The harness and real-ingest arms are from the pre-fix build `084f472` ---necessarily, since they need the `terminate()` teardown that #598 removed. The-vitest consistency check below is NOT: it needs the current tree, and its-"36 of 38" is today's file count. At `084f472` the suite was smaller, so-anyone reproducing that arm by checking out `084f472` will count differently-and should redo the arithmetic with what they find.+**Reproducing these arms, precisely, because no single commit runs them+all.** They need the `terminate()` teardown that #598 removed, and the fix+landed BEFORE the test file the real-ingest arm drives:++| commit | | |+|---|---|---|+| `084f472` | #570 | last build with the `terminate()` teardown |+| `20e1dae` | #598 | the fix -- teardown becomes `__shutdown` + `unref()` |+| `b9b84ef` | #597 | adds `ingest-files.test.ts` |++So:++- The **minimal harness** arm runs at `084f472`.+- The **real-ingest** arm needs both, and no commit has both -- checked every+ commit on `main`. It was measured on a hand-assembled tree: `b9b84ef`'s+ `ingest-files.test.ts` over `084f472`'s `parse-pool.ts`. An earlier revision+ of this file said "check out `084f472`", which gives "no such file". Say the+ recipe or the arm is not reproducible.+- The **vitest** check needs the CURRENT tree, not either of those: its+ "36 of 38" is today's count.## The bug@@ -63,21 +77,30 @@ twenty-teardown arm's own rate (5.5%, again per pool):`P(≥5 of 40 | p=0.055) = 0.067` — a failure to reject, borderline, not ademonstration of agreement.-**Real ingests** — `ingest-files.test.ts` under vitest, ~9.5 ingests per-process at the time of measuring (it drives 14 today):+**Real ingests** — `ingest-files.test.ts` under vitest. That file drives a+FIXED number of ingests per process, not an average: **12** at `b9b84ef`, the+version measured (14 today). The per-teardown rate is `1-(1-p)^12` solved+against the process counts:| configuration | result | per POOL teardown ||---|---|---|-| idle machine | 2 of 75 processes | 0.28% |-| loaded machine | 9 of 50 processes | 2.1% |+| idle machine | 2 of 75 processes | 0.225% |+| loaded machine | 9 of 50 processes | 1.64% |-(Per pool, as everywhere else here — not per isolate. 0.28% sits next to the+(Per pool, as everywhere else here — not per isolate. 0.225% sits near the0.31% per-ISOLATE hazard derived below, and they are not the same quantity.)-Fisher exact on 2/75 vs 9/50 is **p = 0.007** two-sided (0.004 one-sided), so-load matters. A real `ix ingest` of 300 files was **0 of 60** — that is *one*-teardown per process, and `P(zero in 60)` is 0.84 at the idle rate, 0.29 at the-loaded one. Unremarkable under either.+Earlier revisions divided by **9.5**, giving 0.28% and 2.1%. That number is+not derivable from any version of the file and no derivation was ever+recorded, so it is withdrawn. It was not harmless: it sets the headline ratio+below, and 9.5 is the flattering end — the correction moves the harness+overstatement from 22× to 28×, i.e. further from the harness, not nearer.++Fisher exact on 2/75 vs 9/50 is **p = 0.007** two-sided (0.004 one-sided), and+is unaffected by the divisor since it compares process counts. A real+`ix ingest` of 300 files was **0 of 60** — that is *one* teardown per process,+and `P(zero in 60)` is 0.87 at the idle rate, 0.37 at the loaded one.+Unremarkable under either.## The grammar tally@@ -100,20 +123,21 @@ writing down.## What this does and does not establish-The minimal harness overstates real exposure by **22×** on the only+The minimal harness overstates real exposure by **28×** on the onlyload-matched comparison available (idle vs idle). Against the loaded real rate-it is 3.1×, but that mixes conditions — the harness was never run loaded, and-load raises the rate — so treat 3.1× as a lower bound on what is unexplained,+it is 3.9×, but that mixes conditions — the harness was never run loaded, and+load raises the rate — so treat 3.9× as a lower bound on what is unexplained,not the residue after subtracting load.**Parses per worker is not controlled.** `ingestFiles` parses a `.ts` filetwice (index prescan, then streaming loop, both on the same pool), so filecounts double; and `ingest-files.test.ts` is not uniformly 30 files — it calls-`fixture(30)` eight times, `fixture(12)` once and `fixture(4)` twice, giving-0.38 to 2.9 parses per worker across its runs and straddling the harness's-~1.4. An earlier claim that "the harness parses the least per worker and-crashes the most" is therefore false for several of the teardowns behind the-data. This remains an open confound.+`fixture(30)` **seven** times, `fixture(12)` once and `fixture(4)` twice at+`b9b84ef` — the measured version; it is eight/one/two today — giving 0.38 to+2.9 parses per worker across its runs and straddling the harness's ~1.4.+An earlier claim that "the harness parses the least per worker and crashes the+most" is therefore false for several of the teardowns behind the data. This+remains an open confound.**The addon is held from module evaluation, not from the first parse.**`core-ingestion/src/index.ts` resolves grammars at module scope and@@ -134,8 +158,8 @@ an undispatched worker is **not known to be safe to terminate**. Workers thathad parsed were the ones observed to crash, and a spawn-then-destroy arm went0 of 120 teardowns. **Do not use that arm.** Its harness calls `pool.init()`and then `await pool.destroy()` with nothing in between — no wait for an ack-or an `'online'` event — so it tore the workers down inside the addon-free-window+or an `'online'` event. The inference is that it therefore tore the workers+down inside the addon-free windowdescribed above. It measured threads that had not finished loading, which is adifferent population from "evaluated but never dispatched", and it thereforesays nothing about whether parsing matters. Earlier revisions of this document@@ -179,19 +203,26 @@ those disposals as independent gives `1-(1-h)^21 = 0.063`, i.e. **h = 0.31%**— a factor of 20.5, and for a pool of 21 that is the only shape the answer cantake: at small h the pool rate is about 21h, so the ratio can approach 21 andnever exceed it. Any conversion factor larger than the pool size is arithmetic-that went wrong, which is how the ~28× an earlier revision quoted here was-caught (it was 8.6% over 0.31% — a single arm's per-pool rate against the-pooled per-isolate one).+that went wrong, which is how a bad conversion factor quoted here in an+earlier revision was caught: it divided 8.6% by 0.31%, a single arm's per-pool+rate against the pooled per-isolate one, and got 27.7. Written out because it+rounds to 28 and so does the harness-overstatement figure above, which is+correct and unrelated (6.3% over 0.225%). Two different quantities landing on+the same rounded number, one of them retracted, is exactly the confusion this+file exists to prevent.**Independence is an assumption, and this document has direct evidenceagainst it.** The idle-vs-loaded real-ingest rates differ 7× at identical pool-size, and the harness overstates idle real ingests by 22×, so something not in+size, and the harness overstates idle real ingests by 28×, so something not inthe model moves the rate. Carrying `h` from the harness onto vitest'sone-at-a-time terminations is also a cross-population transfer — the same movethis file retracts for the "one in twelve" figure. The conclusion surviveseither way, which is why it is stated rather than hedged away: redo it with-the real-ingest IDLE hazard instead and `P(0 in 10 runs)` is about 0.95-against 0.33, both unremarkable. Nothing here rests on the exact h.+the real-ingest IDLE hazard instead — 0.225% per pool over 21 isolates is+h = 1.07e-4, and 36 terminated isolates over 10 runs gives **0.96** — against+0.33, both unremarkable. Nothing here rests on the exact h. (An earlier+revision said 0.95; the stated inputs give 0.962, and every other figure in+this file reproduces to the digit.)How many addon-loaded isolates a run terminates is not a guess: `isolate`defaults to `true`, `core-ingestion` ships no vitest config to change it, anddiff --git a/ix-cli/src/cli/__tests__/parse-pool.test.ts b/ix-cli/src/cli/__tests__/parse-pool.test.tsindex 09810a7d..53ed48fa 100644--- a/ix-cli/src/cli/__tests__/parse-pool.test.ts+++ b/ix-cli/src/cli/__tests__/parse-pool.test.ts@@ -228,17 +228,17 @@ describe("ParsePool", () => {// delivered before the two parses below went out. The test then failed on// `b2.ts` coming back null, which looks exactly like the bug it guards.//- // `respawnCount()` is the pool's own signal that `onError` ran to+ // `workerDeaths()` is the pool's own signal that `onError` ran to// completion, so this waits on the state the test actually depends on and// is done as soon as it holds.// `> 0` is only correct because this is the FIRST death of the run. The// counter is monotonic and never resets, so a second wait written this// way returns immediately -- a silent no-op, which is the same class of// bug this test was fixed for. A later fault must capture a baseline- // first: `const before = pool.respawnCount()` then `() => pool- // .respawnCount() > before`.+ // first: `const before = pool.workerDeaths()` then+ // `() => pool.workerDeaths() > before`.await waitUntil(- () => pool.respawnCount() > 0,+ () => pool.workerDeaths() > 0,// Ask the fixture first. If its claim failed there is no fault to wait// for, and reporting that the pool never reacted would blame the pool// for a harness problem -- the misdiagnosis this whole test exists to@@ -271,7 +271,7 @@ describe("ParsePool", () => {// that cannot happen, so adding a redundant per-thread guard leaves this// green, which is correct and not a gap.//- // What it replaced was an assertion on `respawnCount()`, which pinned+ // What it replaced was an assertion on `workerDeaths()`, which pinned// nothing: the replacement's fault is a 250ms timer and the parses above// take a few ms, so the count still reads 1 either way. Reverting the// fixture passed that one 5 runs out of 5.@@ -280,9 +280,11 @@ describe("ParsePool", () => {// only proves SOME respawn happened -- a fixture that died during module// evaluation would satisfy it and never arm -- and the ENOENT would then// point at this line instead of at the contract.- // No thread failed to claim for an unrelated reason. Without this, a- // first-thread EPERM is invisible: the second thread claims successfully,- // arms, and every other assertion here stays green.+ // No thread failed to claim for an unrelated reason. This catches a+ // REPLACEMENT's claim failing -- it does not catch the first thread's,+ // because at concurrency 1 that is the only worker, nothing respawns, and+ // the `waitUntil` thunk above reports it before execution ever reaches+ // here. The two checks cover different threads; neither covers both.expect(existsSync(`${marker}.failed`) ? readFileSync(`${marker}.failed`, "utf8") : "","a worker failed to claim the fault for a reason other than EEXIST",@@ -300,7 +302,7 @@ describe("ParsePool", () => {// Kept, but for what it is: a cheap check that no EXTRA fault landed// during the two parses. It is not the guard for the line above.- expect(pool.respawnCount(), "no further worker died during the parses").toBe(1);+ expect(pool.workerDeaths(), "no further worker died during the parses").toBe(1);await pool.destroy();});@@ -746,14 +748,14 @@ describe("ParsePool", () => {pool.init();// Parse for real, because having PARSED is what was observed to arm the// crash. Not because parsing loads the addon: `index.ts` resolves- // 27 grammars plus the core at module scope -- twelve by static import, the- // rest eagerly through helpers -- so these threads hold the core and the- // twelve within moments of spawn, well before that module finishes- // evaluating. The other 15 resolve through null-returning helpers and are- // absent wherever a platform has no prebuild, so the guaranteed floor is- // the core plus the statically imported twelve -- which is all this test- // needs. (One of the 15, `tree-sitter-powershell`, is a required- // dependency that loads through a helper anyway.) The exact tally is in+ // its grammars at module scope -- some by static import, the rest eagerly+ // through helpers -- so these threads hold the core and the static ones+ // within moments of spawn, well before that module finishes evaluating.+ // The helper-loaded ones return null wherever a platform has no prebuild,+ // so the guaranteed floor is the core plus the statically imported+ // grammars, which is all this test needs. (Being a REQUIRED dependency+ // does not put a grammar in that floor: `tree-sitter-powershell` is+ // required and still loads through a helper.) The counts are in// `docs/parse-pool-teardown.md` rather than here.//// Workers that had parsed were the ones observed to crash underdiff --git a/ix-cli/src/cli/commands/parse-pool.ts b/ix-cli/src/cli/commands/parse-pool.tsindex 39ce04a6..8676be5b 100644--- a/ix-cli/src/cli/commands/parse-pool.ts+++ b/ix-cli/src/cli/commands/parse-pool.ts@@ -189,8 +189,8 @@ export class ParsePool {** DO NOT add a `terminate()` fast path for workers that have never been* dispatched. They hold the addon too: `core-ingestion/src/index.ts`- * resolves grammars at module scope, and `tree-sitter` plus the twelve- * static grammars sit near the top of its dependency graph -- so a worker+ * resolves grammars at module scope, and `tree-sitter` plus the statically+ * imported grammars sit near the top of its dependency graph -- so a worker* holds them within moments of `new Worker()`, without ever being* dispatched, and long before `index.ts`'s own body or its top-level* `await`s run. Note which end that pins: the addon-free window CLOSES when@@ -198,18 +198,31 @@ export class ParsePool {* suspended at one of those awaits already holds all twelve, and is not* safe to terminate.*- * Not every grammar -- 15 of the 27 go through null-returning helpers and- * can be absent -- but the core and the twelve static ones are always- * there, which is the PRECONDITION the crash needs. Whether it is also+ * Not every grammar: most go through null-returning helpers and can be+ * absent, but the core and the statically imported ones are always there,+ * which is the PRECONDITION the crash needs. Counts in+ * `docs/parse-pool-teardown.md` rather than here -- the same reason the+ * rates live there, and #595 is the worked example of what restating them+ * costs: it added a required grammar and left a tally in+ * `core-ingestion/src/index.ts` stale, with nothing red. Whether it is also* sufficient has never been measured: the one experiment that looked --* spawn-then-destroy, 0 of 120 -- destroyed its pool with no wait for an- * ack or an `'online'` event, so it tore threads down inside that window,- * before they had finished loading. Different population; it says nothing- * about undispatched workers. This is the file where a fast path would be- * written, which is why the warning is here and not only in the tests.+ * ack or an `'online'` event. So it very probably tore threads down inside+ * that window, before they had finished loading -- a different population,+ * saying nothing about undispatched workers. Probably, not certainly: that+ * is an inference from the teardown code, not a measurement, and it is the+ * sole reason for discarding the only evidence AGAINST this rule.+ *+ * This is the file where a fast path would be written, which is why the+ * warning is here and not only in the tests.** The comparison that settles the verb, on Windows/Node 26, twenty pool- * teardowns per process, six runs each:+ * teardowns per process, six runs each. It settles WHICH VERB, and that is+ * all: do not derive a rate from it. Read alone the first row implies 8.6%+ * per pool teardown, which is close to the "one in twelve" this whole+ * change exists to retract -- it is one of three arms, and the pooled fit+ * over all of them is in `docs/parse-pool-teardown.md`. Quoting a single+ * arm is how the retracted figure got published the first time.** terminate() 5 of 6 runs died with SIGSEGV (139)* worker closes its own port 0 of 6@@ -431,7 +444,14 @@ export class ParsePool {}/**- * Replacement workers spawned over the whole run. Monotonic.+ * Worker deaths this run has reacted to. Monotonic.+ *+ * Deaths, not replacements: past `MAX_RESPAWNS` the pool stops replacing but+ * still reacts -- splicing `idle`, latching `dead`, draining the queue -- and+ * a caller watching for "did the pool notice a worker die" needs those too.+ * An earlier revision counted replacements and sat inside the cap branch,+ * which silently stopped advancing at exactly the point a caller most wants+ * to know.** Deliberately NOT `respawns`, which is a budget rather than a tally:* `onResult` clears it on every successful round trip, so a run that lost a@@ -440,12 +460,12 @@ export class ParsePool {* is the right behaviour for the cap and the wrong number for a reader, and* it makes `respawns` useless to wait on: any completed parse resets it.*- * This one is a LEVEL. `onError` increments it and pushes the replacement in- * the same synchronous handler, so once it is non-zero the pool has finished- * reacting to a worker death -- including one that faulted with no task in- * flight, which `crashedTasks()` deliberately does not count because no file- * was lost. That combination is otherwise unobservable from outside, which- * is what this exists for.+ * This one is a LEVEL. `onError` increments it in the same synchronous+ * handler that does the splicing, so once it has advanced the pool has+ * finished reacting to that death -- including one that faulted with no task+ * in flight, which `crashedTasks()` deliberately does not count because no+ * file was lost. That combination is otherwise unobservable from outside,+ * which is what this exists for.** Test observability, and only that today -- nothing in `ingest.ts` reads* it, so a run that respawned a dozen workers still prints the same summary@@ -456,11 +476,11 @@ export class ParsePool {* latches for the life of the run, so `> 0` answers "has any worker ever* died", which is true forever after the first one -- correct only for a* test observing that first death. Anything later wants- * `const before = pool.respawnCount()` and then `> before`, or it is a+ * `const before = pool.workerDeaths()` and then `> before`, or it is a* poll that returns immediately and waits for nothing.*/- respawnCount(): number {- return this.respawnsTotal;+ workerDeaths(): number {+ return this.deaths;}private onResult(w: Worker, msg: { ok: boolean; result: unknown }): void {@@ -488,12 +508,12 @@ export class ParsePool {* `MAX_RESPAWNS` -- so 0 means the full budget is available and 16 means it* is exhausted, which is the opposite of how "budget" usually reads.* `onResult` clears it on any success, so it is not a tally either: read- * `respawnsTotal` for "how many did this run spawn?".+ * `deaths` for "how many workers died this run?".*/private respawns = 0;- /** Every replacement this run has spawned. Never reset. */- private respawnsTotal = 0;+ /** Worker deaths this run has reacted to. Never reset. */+ private deaths = 0;/*** True once the pool is out of workers AND out of replacements.@@ -551,9 +571,16 @@ export class ParsePool {const idleIdx = this.idle.indexOf(w);if (idleIdx !== -1) this.idle.splice(idleIdx, 1);+ // Before the cap branch, because a capped death still reacts fully --+ // it splices `idle`, latches `dead` and drains the queue -- and a signal+ // that means "the pool has finished reacting" has to advance for those+ // too. Inside the branch it counted replacements instead, so past the cap+ // it stopped moving and the `> before` wait below would hang on exactly+ // the deaths a caller most wants to observe.+ this.deaths++;+if (this.respawns < ParsePool.MAX_RESPAWNS) {this.respawns++;- this.respawnsTotal++;this.spawnWorker();} else if (this.workers.length === 0) {// Out of workers and out of replacements. Nothing queued can ever be