Skip to content

armgit / Ix

Public Imported from GitHub
0

Understand any codebase instantly. System intelligence for codebases, built for humans and AI.

fix(parse-pool): count deaths outside the respawn cap, and correct the teardown doc's divisor (#654)

Joseph Mikhaila82578057dec
diff --git a/core-ingestion/src/index.ts b/core-ingestion/src/index.ts
index bff87d7e..742973b8 100644
--- a/core-ingestion/src/index.ts
+++ b/core-ingestion/src/index.ts
@@ -13,7 +13,7 @@ import Parser from 'tree-sitter';
// fresh process on Node 26:
//
// tree-sitter-c-sharp required type=module -> 1
-// tree-sitter-powershell required type=module -> 1
+// tree-sitter-powershell required type=module -> 1 (*)
// @tree-sitter-grammars/tree-sitter-lua optional type=module -> 1
// tree-sitter-css optional type=module -> 1
// the other 11 required grammars required type=commonjs -> 0
@@ -21,6 +21,14 @@ import Parser from 'tree-sitter';
// tree-sitter-sas optional type=module -> 0
// (full-filename main + "exports")
//
+// (*) INFERRED, not measured. `tree-sitter-powershell` is not installed in
+// every checkout -- it is absent from this one -- so its row comes from the
+// import comment below rather than from running Node against the package. CI
+// proves the subpath `bindings/node/index.js` resolves; it does not prove the
+// bare specifier would warn. Re-measure before relying on it. Marked because
+// an unverifiable row in this table is what went stale in #595 and cost
+// Ix#650 sixteen review rounds.
+//
// The two 11s are a coincidence, not a copy-paste: 13 required grammars minus
// c-sharp and powershell, and 14 optional minus lua, css and sas.
//
diff --git a/core-ingestion/src/parse-worker.ts b/core-ingestion/src/parse-worker.ts
index 56800dfd..e9dd3026 100644
--- a/core-ingestion/src/parse-worker.ts
+++ b/core-ingestion/src/parse-worker.ts
@@ -25,9 +25,10 @@ if (!parentPort) throw new Error('parse-worker must run inside a worker thread')
* NEVER-DISPATCHED WORKERS ARE NOT EXEMPT. The addon is held from module
* EVALUATION, not from the first parse: `index.ts` resolves its grammars at
* module scope and this file imports it statically, so a worker reaches that
- * state on its own without being dispatched. `tree-sitter` and the twelve
- * grammars are `index.ts`'s own static imports, near the top of its dependency
- * graph, so the isolate holds them within moments of `new Worker()` -- long
+ * state on its own without being dispatched. `tree-sitter` and the statically
+ * imported grammars are `index.ts`'s own imports, near the top of its
+ * dependency graph, so the isolate holds them within moments of
+ * `new Worker()` -- long
* before that file's body or its top-level `await`s run. A worker that has
* parsed nothing, or that is still suspended at one of those awaits, holds
* them all the same. That is the precondition the crash needs, so an
@@ -35,8 +36,9 @@ if (!parentPort) throw new Error('parse-worker must run inside a worker thread')
*
* Whether it is as dangerous as a parsed one has never been measured: the one
* arm that looked -- spawn-then-destroy, 0 of 120 -- destroyed its pool with
- * no wait for an ack or an `'online'` event, so it tore threads down before
- * they had finished loading: a different population.
+ * no wait for an ack or an `'online'` event, so it very probably tore threads
+ * down before they had finished loading: a different population. Inferred
+ * from that teardown code, not measured.
*
* `parse-pool.test.ts` used to say the opposite -- "an untouched worker has
* not loaded the addon" -- which is what made a `terminate()` fast path for
diff --git a/docs/parse-pool-teardown.md b/docs/parse-pool-teardown.md
index d22659f1..f8666ff9 100644
--- a/docs/parse-pool-teardown.md
+++ b/docs/parse-pool-teardown.md
@@ -8,12 +8,26 @@ live here so they do not have to be kept consistent across three source files.
Origin: Ix#598 (the fix) and Ix#650 (this correction). Windows / Node 26,
against a live backend.
-The harness and real-ingest arms are from the pre-fix build `084f472` --
-necessarily, since they need the `terminate()` teardown that #598 removed. The
-vitest consistency check below is NOT: it needs the current tree, and its
-"36 of 38" is today's file count. At `084f472` the suite was smaller, so
-anyone reproducing that arm by checking out `084f472` will count differently
-and should redo the arithmetic with what they find.
+**Reproducing these arms, precisely, because no single commit runs them
+all.** They need the `terminate()` teardown that #598 removed, and the fix
+landed BEFORE the test file the real-ingest arm drives:
+
+| commit | | |
+|---|---|---|
+| `084f472` | #570 | last build with the `terminate()` teardown |
+| `20e1dae` | #598 | the fix -- teardown becomes `__shutdown` + `unref()` |
+| `b9b84ef` | #597 | adds `ingest-files.test.ts` |
+
+So:
+
+- The **minimal harness** arm runs at `084f472`.
+- The **real-ingest** arm needs both, and no commit has both -- checked every
+ commit on `main`. It was measured on a hand-assembled tree: `b9b84ef`'s
+ `ingest-files.test.ts` over `084f472`'s `parse-pool.ts`. An earlier revision
+ of this file said "check out `084f472`", which gives "no such file". Say the
+ recipe or the arm is not reproducible.
+- The **vitest** check needs the CURRENT tree, not either of those: its
+ "36 of 38" is today's count.
## The bug
@@ -63,21 +77,30 @@ twenty-teardown arm's own rate (5.5%, again per pool):
`P(≥5 of 40 | p=0.055) = 0.067` — a failure to reject, borderline, not a
demonstration of agreement.
-**Real ingests** — `ingest-files.test.ts` under vitest, ~9.5 ingests per
-process at the time of measuring (it drives 14 today):
+**Real ingests** — `ingest-files.test.ts` under vitest. That file drives a
+FIXED number of ingests per process, not an average: **12** at `b9b84ef`, the
+version measured (14 today). The per-teardown rate is `1-(1-p)^12` solved
+against the process counts:
| configuration | result | per POOL teardown |
|---|---|---|
-| idle machine | 2 of 75 processes | 0.28% |
-| loaded machine | 9 of 50 processes | 2.1% |
+| idle machine | 2 of 75 processes | 0.225% |
+| loaded machine | 9 of 50 processes | 1.64% |
-(Per pool, as everywhere else here — not per isolate. 0.28% sits next to the
+(Per pool, as everywhere else here — not per isolate. 0.225% sits near the
0.31% per-ISOLATE hazard derived below, and they are not the same quantity.)
-Fisher exact on 2/75 vs 9/50 is **p = 0.007** two-sided (0.004 one-sided), so
-load matters. A real `ix ingest` of 300 files was **0 of 60** — that is *one*
-teardown per process, and `P(zero in 60)` is 0.84 at the idle rate, 0.29 at the
-loaded one. Unremarkable under either.
+Earlier revisions divided by **9.5**, giving 0.28% and 2.1%. That number is
+not derivable from any version of the file and no derivation was ever
+recorded, so it is withdrawn. It was not harmless: it sets the headline ratio
+below, and 9.5 is the flattering end — the correction moves the harness
+overstatement from 22× to 28×, i.e. further from the harness, not nearer.
+
+Fisher exact on 2/75 vs 9/50 is **p = 0.007** two-sided (0.004 one-sided), and
+is unaffected by the divisor since it compares process counts. A real
+`ix ingest` of 300 files was **0 of 60** — that is *one* teardown per process,
+and `P(zero in 60)` is 0.87 at the idle rate, 0.37 at the loaded one.
+Unremarkable under either.
## The grammar tally
@@ -100,20 +123,21 @@ writing down.
## What this does and does not establish
-The minimal harness overstates real exposure by **22×** on the only
+The minimal harness overstates real exposure by **28×** on the only
load-matched comparison available (idle vs idle). Against the loaded real rate
-it is 3.1×, but that mixes conditions — the harness was never run loaded, and
-load raises the rate — so treat 3.1× as a lower bound on what is unexplained,
+it is 3.9×, but that mixes conditions — the harness was never run loaded, and
+load raises the rate — so treat 3.9× as a lower bound on what is unexplained,
not the residue after subtracting load.
**Parses per worker is not controlled.** `ingestFiles` parses a `.ts` file
twice (index prescan, then streaming loop, both on the same pool), so file
counts double; and `ingest-files.test.ts` is not uniformly 30 files — it calls
-`fixture(30)` eight times, `fixture(12)` once and `fixture(4)` twice, giving
-0.38 to 2.9 parses per worker across its runs and straddling the harness's
-~1.4. An earlier claim that "the harness parses the least per worker and
-crashes the most" is therefore false for several of the teardowns behind the
-data. This remains an open confound.
+`fixture(30)` **seven** times, `fixture(12)` once and `fixture(4)` twice at
+`b9b84ef` — the measured version; it is eight/one/two today — giving 0.38 to
+2.9 parses per worker across its runs and straddling the harness's ~1.4.
+An earlier claim that "the harness parses the least per worker and crashes the
+most" is therefore false for several of the teardowns behind the data. This
+remains an open confound.
**The addon is held from module evaluation, not from the first parse.**
`core-ingestion/src/index.ts` resolves grammars at module scope and
@@ -134,8 +158,8 @@ an undispatched worker is **not known to be safe to terminate**. Workers that
had parsed were the ones observed to crash, and a spawn-then-destroy arm went
0 of 120 teardowns. **Do not use that arm.** Its harness calls `pool.init()`
and then `await pool.destroy()` with nothing in between — no wait for an ack
-or an `'online'` event — so it tore the workers down inside the addon-free
-window
+or an `'online'` event. The inference is that it therefore tore the workers
+down inside the addon-free window
described above. It measured threads that had not finished loading, which is a
different population from "evaluated but never dispatched", and it therefore
says nothing about whether parsing matters. Earlier revisions of this document
@@ -179,19 +203,26 @@ those disposals as independent gives `1-(1-h)^21 = 0.063`, i.e. **h = 0.31%**
— a factor of 20.5, and for a pool of 21 that is the only shape the answer can
take: at small h the pool rate is about 21h, so the ratio can approach 21 and
never exceed it. Any conversion factor larger than the pool size is arithmetic
-that went wrong, which is how the ~28× an earlier revision quoted here was
-caught (it was 8.6% over 0.31% — a single arm's per-pool rate against the
-pooled per-isolate one).
+that went wrong, which is how a bad conversion factor quoted here in an
+earlier revision was caught: it divided 8.6% by 0.31%, a single arm's per-pool
+rate against the pooled per-isolate one, and got 27.7. Written out because it
+rounds to 28 and so does the harness-overstatement figure above, which is
+correct and unrelated (6.3% over 0.225%). Two different quantities landing on
+the same rounded number, one of them retracted, is exactly the confusion this
+file exists to prevent.
**Independence is an assumption, and this document has direct evidence
against it.** The idle-vs-loaded real-ingest rates differ 7× at identical pool
-size, and the harness overstates idle real ingests by 22×, so something not in
+size, and the harness overstates idle real ingests by 28×, so something not in
the model moves the rate. Carrying `h` from the harness onto vitest's
one-at-a-time terminations is also a cross-population transfer — the same move
this file retracts for the "one in twelve" figure. The conclusion survives
either way, which is why it is stated rather than hedged away: redo it with
-the real-ingest IDLE hazard instead and `P(0 in 10 runs)` is about 0.95
-against 0.33, both unremarkable. Nothing here rests on the exact h.
+the real-ingest IDLE hazard instead — 0.225% per pool over 21 isolates is
+h = 1.07e-4, and 36 terminated isolates over 10 runs gives **0.96** — against
+0.33, both unremarkable. Nothing here rests on the exact h. (An earlier
+revision said 0.95; the stated inputs give 0.962, and every other figure in
+this file reproduces to the digit.)
How many addon-loaded isolates a run terminates is not a guess: `isolate`
defaults to `true`, `core-ingestion` ships no vitest config to change it, and
diff --git a/ix-cli/src/cli/__tests__/parse-pool.test.ts b/ix-cli/src/cli/__tests__/parse-pool.test.ts
index 09810a7d..53ed48fa 100644
--- a/ix-cli/src/cli/__tests__/parse-pool.test.ts
+++ b/ix-cli/src/cli/__tests__/parse-pool.test.ts
@@ -228,17 +228,17 @@ describe("ParsePool", () => {
// delivered before the two parses below went out. The test then failed on
// `b2.ts` coming back null, which looks exactly like the bug it guards.
//
- // `respawnCount()` is the pool's own signal that `onError` ran to
+ // `workerDeaths()` is the pool's own signal that `onError` ran to
// completion, so this waits on the state the test actually depends on and
// is done as soon as it holds.
// `> 0` is only correct because this is the FIRST death of the run. The
// counter is monotonic and never resets, so a second wait written this
// way returns immediately -- a silent no-op, which is the same class of
// bug this test was fixed for. A later fault must capture a baseline
- // first: `const before = pool.respawnCount()` then `() => pool
- // .respawnCount() > before`.
+ // first: `const before = pool.workerDeaths()` then
+ // `() => pool.workerDeaths() > before`.
await waitUntil(
- () => pool.respawnCount() > 0,
+ () => pool.workerDeaths() > 0,
// Ask the fixture first. If its claim failed there is no fault to wait
// for, and reporting that the pool never reacted would blame the pool
// for a harness problem -- the misdiagnosis this whole test exists to
@@ -271,7 +271,7 @@ describe("ParsePool", () => {
// that cannot happen, so adding a redundant per-thread guard leaves this
// green, which is correct and not a gap.
//
- // What it replaced was an assertion on `respawnCount()`, which pinned
+ // What it replaced was an assertion on `workerDeaths()`, which pinned
// nothing: the replacement's fault is a 250ms timer and the parses above
// take a few ms, so the count still reads 1 either way. Reverting the
// fixture passed that one 5 runs out of 5.
@@ -280,9 +280,11 @@ describe("ParsePool", () => {
// only proves SOME respawn happened -- a fixture that died during module
// evaluation would satisfy it and never arm -- and the ENOENT would then
// point at this line instead of at the contract.
- // No thread failed to claim for an unrelated reason. Without this, a
- // first-thread EPERM is invisible: the second thread claims successfully,
- // arms, and every other assertion here stays green.
+ // No thread failed to claim for an unrelated reason. This catches a
+ // REPLACEMENT's claim failing -- it does not catch the first thread's,
+ // because at concurrency 1 that is the only worker, nothing respawns, and
+ // the `waitUntil` thunk above reports it before execution ever reaches
+ // here. The two checks cover different threads; neither covers both.
expect(
existsSync(`${marker}.failed`) ? readFileSync(`${marker}.failed`, "utf8") : "",
"a worker failed to claim the fault for a reason other than EEXIST",
@@ -300,7 +302,7 @@ describe("ParsePool", () => {
// Kept, but for what it is: a cheap check that no EXTRA fault landed
// during the two parses. It is not the guard for the line above.
- expect(pool.respawnCount(), "no further worker died during the parses").toBe(1);
+ expect(pool.workerDeaths(), "no further worker died during the parses").toBe(1);
await pool.destroy();
});
@@ -746,14 +748,14 @@ describe("ParsePool", () => {
pool.init();
// Parse for real, because having PARSED is what was observed to arm the
// crash. Not because parsing loads the addon: `index.ts` resolves
- // 27 grammars plus the core at module scope -- twelve by static import, the
- // rest eagerly through helpers -- so these threads hold the core and the
- // twelve within moments of spawn, well before that module finishes
- // evaluating. The other 15 resolve through null-returning helpers and are
- // absent wherever a platform has no prebuild, so the guaranteed floor is
- // the core plus the statically imported twelve -- which is all this test
- // needs. (One of the 15, `tree-sitter-powershell`, is a required
- // dependency that loads through a helper anyway.) The exact tally is in
+ // its grammars at module scope -- some by static import, the rest eagerly
+ // through helpers -- so these threads hold the core and the static ones
+ // within moments of spawn, well before that module finishes evaluating.
+ // The helper-loaded ones return null wherever a platform has no prebuild,
+ // so the guaranteed floor is the core plus the statically imported
+ // grammars, which is all this test needs. (Being a REQUIRED dependency
+ // does not put a grammar in that floor: `tree-sitter-powershell` is
+ // required and still loads through a helper.) The counts are in
// `docs/parse-pool-teardown.md` rather than here.
//
// Workers that had parsed were the ones observed to crash under
diff --git a/ix-cli/src/cli/commands/parse-pool.ts b/ix-cli/src/cli/commands/parse-pool.ts
index 39ce04a6..8676be5b 100644
--- a/ix-cli/src/cli/commands/parse-pool.ts
+++ b/ix-cli/src/cli/commands/parse-pool.ts
@@ -189,8 +189,8 @@ export class ParsePool {
*
* DO NOT add a `terminate()` fast path for workers that have never been
* dispatched. They hold the addon too: `core-ingestion/src/index.ts`
- * resolves grammars at module scope, and `tree-sitter` plus the twelve
- * static grammars sit near the top of its dependency graph -- so a worker
+ * resolves grammars at module scope, and `tree-sitter` plus the statically
+ * imported grammars sit near the top of its dependency graph -- so a worker
* holds them within moments of `new Worker()`, without ever being
* dispatched, and long before `index.ts`'s own body or its top-level
* `await`s run. Note which end that pins: the addon-free window CLOSES when
@@ -198,18 +198,31 @@ export class ParsePool {
* suspended at one of those awaits already holds all twelve, and is not
* safe to terminate.
*
- * Not every grammar -- 15 of the 27 go through null-returning helpers and
- * can be absent -- but the core and the twelve static ones are always
- * there, which is the PRECONDITION the crash needs. Whether it is also
+ * Not every grammar: most go through null-returning helpers and can be
+ * absent, but the core and the statically imported ones are always there,
+ * which is the PRECONDITION the crash needs. Counts in
+ * `docs/parse-pool-teardown.md` rather than here -- the same reason the
+ * rates live there, and #595 is the worked example of what restating them
+ * costs: it added a required grammar and left a tally in
+ * `core-ingestion/src/index.ts` stale, with nothing red. Whether it is also
* sufficient has never been measured: the one experiment that looked --
* spawn-then-destroy, 0 of 120 -- destroyed its pool with no wait for an
- * ack or an `'online'` event, so it tore threads down inside that window,
- * before they had finished loading. Different population; it says nothing
- * about undispatched workers. This is the file where a fast path would be
- * written, which is why the warning is here and not only in the tests.
+ * ack or an `'online'` event. So it very probably tore threads down inside
+ * that window, before they had finished loading -- a different population,
+ * saying nothing about undispatched workers. Probably, not certainly: that
+ * is an inference from the teardown code, not a measurement, and it is the
+ * sole reason for discarding the only evidence AGAINST this rule.
+ *
+ * This is the file where a fast path would be written, which is why the
+ * warning is here and not only in the tests.
*
* The comparison that settles the verb, on Windows/Node 26, twenty pool
- * teardowns per process, six runs each:
+ * teardowns per process, six runs each. It settles WHICH VERB, and that is
+ * all: do not derive a rate from it. Read alone the first row implies 8.6%
+ * per pool teardown, which is close to the "one in twelve" this whole
+ * change exists to retract -- it is one of three arms, and the pooled fit
+ * over all of them is in `docs/parse-pool-teardown.md`. Quoting a single
+ * arm is how the retracted figure got published the first time.
*
* terminate() 5 of 6 runs died with SIGSEGV (139)
* worker closes its own port 0 of 6
@@ -431,7 +444,14 @@ export class ParsePool {
}
/**
- * Replacement workers spawned over the whole run. Monotonic.
+ * Worker deaths this run has reacted to. Monotonic.
+ *
+ * Deaths, not replacements: past `MAX_RESPAWNS` the pool stops replacing but
+ * still reacts -- splicing `idle`, latching `dead`, draining the queue -- and
+ * a caller watching for "did the pool notice a worker die" needs those too.
+ * An earlier revision counted replacements and sat inside the cap branch,
+ * which silently stopped advancing at exactly the point a caller most wants
+ * to know.
*
* Deliberately NOT `respawns`, which is a budget rather than a tally:
* `onResult` clears it on every successful round trip, so a run that lost a
@@ -440,12 +460,12 @@ export class ParsePool {
* is the right behaviour for the cap and the wrong number for a reader, and
* it makes `respawns` useless to wait on: any completed parse resets it.
*
- * This one is a LEVEL. `onError` increments it and pushes the replacement in
- * the same synchronous handler, so once it is non-zero the pool has finished
- * reacting to a worker death -- including one that faulted with no task in
- * flight, which `crashedTasks()` deliberately does not count because no file
- * was lost. That combination is otherwise unobservable from outside, which
- * is what this exists for.
+ * This one is a LEVEL. `onError` increments it in the same synchronous
+ * handler that does the splicing, so once it has advanced the pool has
+ * finished reacting to that death -- including one that faulted with no task
+ * in flight, which `crashedTasks()` deliberately does not count because no
+ * file was lost. That combination is otherwise unobservable from outside,
+ * which is what this exists for.
*
* Test observability, and only that today -- nothing in `ingest.ts` reads
* it, so a run that respawned a dozen workers still prints the same summary
@@ -456,11 +476,11 @@ export class ParsePool {
* latches for the life of the run, so `> 0` answers "has any worker ever
* died", which is true forever after the first one -- correct only for a
* test observing that first death. Anything later wants
- * `const before = pool.respawnCount()` and then `> before`, or it is a
+ * `const before = pool.workerDeaths()` and then `> before`, or it is a
* poll that returns immediately and waits for nothing.
*/
- respawnCount(): number {
- return this.respawnsTotal;
+ workerDeaths(): number {
+ return this.deaths;
}
private onResult(w: Worker, msg: { ok: boolean; result: unknown }): void {
@@ -488,12 +508,12 @@ export class ParsePool {
* `MAX_RESPAWNS` -- so 0 means the full budget is available and 16 means it
* is exhausted, which is the opposite of how "budget" usually reads.
* `onResult` clears it on any success, so it is not a tally either: read
- * `respawnsTotal` for "how many did this run spawn?".
+ * `deaths` for "how many workers died this run?".
*/
private respawns = 0;
- /** Every replacement this run has spawned. Never reset. */
- private respawnsTotal = 0;
+ /** Worker deaths this run has reacted to. Never reset. */
+ private deaths = 0;
/**
* True once the pool is out of workers AND out of replacements.
@@ -551,9 +571,16 @@ export class ParsePool {
const idleIdx = this.idle.indexOf(w);
if (idleIdx !== -1) this.idle.splice(idleIdx, 1);
+ // Before the cap branch, because a capped death still reacts fully --
+ // it splices `idle`, latches `dead` and drains the queue -- and a signal
+ // that means "the pool has finished reacting" has to advance for those
+ // too. Inside the branch it counted replacements instead, so past the cap
+ // it stopped moving and the `> before` wait below would hang on exactly
+ // the deaths a caller most wants to observe.
+ this.deaths++;
+
if (this.respawns < ParsePool.MAX_RESPAWNS) {
this.respawns++;
- this.respawnsTotal++;
this.spawnWorker();
} else if (this.workers.length === 0) {
// Out of workers and out of replacements. Nothing queued can ever be