Understand any codebase instantly. System intelligence for codebases, built for humans and AI.
TypeScript aiai-memoryclaude-codeclicode-analysiscode-mappingcode-memorycodebase-understandingcodexdeveloper-toolsllmopenclawpersistent-memoryprogram-analysissoftware-architecturesystem-intelligencesystem-mapping
fix(ingest): stop ix map piling stitch queries onto a backend that is still running the last one (#571)
Joseph Mikhailfb46941e6a0b
diff --git a/docs/api/README.md b/docs/api/README.mdindex 37528ffb..db775749 100644--- a/docs/api/README.md+++ b/docs/api/README.md@@ -156,6 +156,14 @@ Ingest a path into the graph. Long-running: the client allows **30 minutes**.}```+> The **CLI's** `ix ingest --format json` emits a different, narrower breakdown+> under the same key: `unchanged` there counts only files skipped as *mtime- or+> hash-unchanged*, `emptyFile` is a real count rather than a hardcoded `0`, and+> there is an extra `unparsed` bucket for files the parse pool returned nothing+> for. That is the client's own summary of its own run and is not this response;+> see the stitch section below for why the narrower `unchanged` is load-bearing+> there.+#### POST `/v1/map`Return the full code map (systems → subsystems → members). The visualizer and@@ -223,6 +231,152 @@ cross-repo `IMPORTS` edges.**Response** — `{ stitched: number, systemId: string | null, edges: [{ src, dst, name }] }`.+**Client-side admission control (Ix#568).** The join behind this endpoint runs+server-side for as long as it needs to, and outlives the HTTP call that started+it — a proxy answering 500 at ~60s does not stop the query. The CLI therefore+does not issue this call unconditionally:++| Rule | Behaviour |+|---|---|+| One at a time **per backend endpoint** | A second `ix map` — including one for a *different* workspace — waits up to `IX_STITCH_WAIT_MS` (default 30s) for the in-flight stitch, then skips. `ix map`'s own lock is per workspace and does not bound a cross-workspace join. |+| A cooldown written when a stitch **starts** | It is removed only on proof that nothing is running. Until then, no further stitch is sent to that endpoint for `IX_STITCH_COOLDOWN_MS` (default 15 min). |++The second rule is the one that stops the pile-up, and it is written the+opposite way round from the obvious design. Rather than inspecting the failure+and deciding whether it looked like a timeout, the marker goes down before the+request and comes back up only on **proof** that the backend did not run the+join:++* the stitch succeeded;+* the backend answered **4xx**, which is it refusing the request rather than+ executing it — with **408** excluded, since a proxy reporting that *it* gave+ up waiting says nothing about whether the backend did;+* the backend answered **501**, which is how this codebase already spells "no+ `/v1/stitch` here" (`isStitchUnsupported` accepts 404 or 501);+* the connection was never established, so no bytes reached the backend.+ Decided by the **syscall** Node stamps on the underlying error — `connect` or+ `getaddrinfo` — with a small errno set (`ECONNREFUSED`, `ENOTFOUND`,+ `ENETUNREACH`, `EHOSTUNREACH`, `EAI_AGAIN`) and undici's+ `UND_ERR_CONNECT_TIMEOUT` alongside it, and a walk into `AggregateError.errors`+ because a multi-address host like `localhost` — the default endpoint — reports+ a refusal that way. Deliberately narrower than "a transport error": a socket+ dropped *after* the request went out (`UND_ERR_SOCKET`) is the ambiguous case+ — an upstream that restarted killed its join, a proxy that hung up did not —+ and keeps the marker. A read/write `ETIMEDOUT` is excluded for the same+ reason; a connect-phase one is caught by the syscall.++Everything else — a 5xx, a timeout, an abort, a socket dropped mid-flight, or+the process being killed before it could report anything — leaves the marker in+place. That+last case is why the marker is written up front: a hook whose timeout is shorter+than the stitch takes the CLI down mid-request, and nothing it *would* have done+on the way out can be relied on.++The cost is that a stitch failing for an unclassified reason cools down when it+need not have. That errs toward skipping one stitch rather than stacking joins+on a database that is already the reason.++Both the lock and the cooldown are keyed on a normalised endpoint, so+`http://localhost:8090`, `http://localhost:8090/` and `http://127.0.0.1:8090`+are one backend rather than three. Without that, an `ix mcp` server started with+`IX_ENDPOINT` set to an IP and a shell `ix map` reading the config file would+each hold their own "single-flight" lock and stitch simultaneously.++`IX_STITCH_COOLDOWN_MS` is re-read on every attempt and applied to cooldowns+already on disk, so setting it to `0` releases an active one rather than only+affecting the next.++The cooldown is stamped at the stitch's **start** and re-stamped to its **end**+when the attempt reports back without proving anything stopped. The re-stamp is+what makes short values mean anything: `IxClient` caps a request at two minutes,+so a cooldown measured only from the start would already have expired by the+time a timing-out stitch returned, and the next map would be admitted straight+into a second join. One residue remains — a process that is *killed* never+re-stamps, so a cooldown shorter than the attempt it is protecting is expired+when the next map looks at it. Values below the two-minute request cap are+therefore only reliable on the paths that report back.++A stitch that does not happen is reported as `stitchSkipped` in+`ix ingest --format json`, as+`stitch_skipped` in `ix map --format json` and `--format llm`, and as a+`stitch_skipped_rule=<rule>` token on `ix map --silent`, so an automated consumer can tell it+apart from a clean run. It is not an error: it does not set a non-zero exit code+and does not count towards `stitchErrors`, and the previous registration stands —+the same position a stitch that *failed* already left the graph in.++The rule is machine-readable, and carried alongside the prose as+`stitchSkippedRule` / `stitch_skipped_rule`: `in-flight`, `cooling` and+`deadline` are the guard's, and **`incomplete`**, **`lost-parses`** and+**`run-errors`** are `ix ingest`'s own gates — an incremental map that did not re-parse every file has+no complete registration to send, and a run with parse or commit errors would+build one from an incomplete picture of the repo. Those two are by far the+commonest, and they are reported for the same reason as the others: a consumer+asking "are the cross-repo edges current?" gets the wrong answer if the field is+absent whenever the stitch was never attempted. Branch on the rule, never on the+sentence.++`incomplete` and `run-errors` print no human Note and emit no `--silent` token:+the first would appear on nearly every incremental map, and the second restates+lines the run has already printed. `--format json` and `--format llm` carry them+either way. **`lost-parses`** — a parse worker crashed, so some files went+unparsed — *is* printed, because on a `--force` run it is the only one of the+three that can fire, and staying quiet there would make the recovery command+this CLI advertises exit 0 having done nothing.++The wait happens **inside** `ix map`'s per-workspace lock, which the run holds+until it exits. So while one map is waiting out another repo's stitch — up to+`IX_STITCH_WAIT_MS`, 30s by default — any further `ix map` fired for that same+workspace (an auto-map hook, for instance) finds the lock held, coalesces, and+exits 0. Edits made in that window get no graph refresh and nothing says so.+That is a new source of staleness, bounded by `IX_STITCH_WAIT_MS`; set it to `0`+to shed on contention immediately instead, at the cost of losing that map's+cross-repo registration.++Both the lock and the cooldown are keyed on the **endpoint**, not on the+workspace, because the join they bound is cross-workspace. That is the point of+the guard, and it is also its cost: one transient failure while mapping repo A+refuses the stitch for repos B..E on the same backend for the whole cooldown,+and when it expires none of them re-attempt on their own for the reason below.+In a multi-repo setup that means every repo's cross-repo edges stay as they were+until somebody runs a full re-ingest in each. The alternative -- keying per+workspace -- does not bound the query at all, since the query is not per+workspace.++Note that re-registration is not automatic on the next map, and was not before+this change: the stitch is gated on every file having been parsed this run, so an+incremental map that skips an mtime- or hash-unchanged file neither reaches it nor+has the registration data to send, having only parsed what changed. A run that+re-ingests every file (`ix ingest <root> --force`, a post-reset re-map) is what+picks it back up.++The gate counts files skipped **as unchanged**, plus parses lost to a **crashed+worker**. It used to be the whole `filesSkipped` total, which also counts+zero-byte files, ones that look minified, and ones the parser simply returned+nothing for — none of which have anything to contribute to the registration, so+their absence does not make the collected set partial. That last class matters:+fourteen tree-sitter grammars are optional dependencies, and a file whose grammar+did not build (`tree-sitter-sas` has no win32 prebuild) comes back unparsed on+*every* run, `--force` included, so counting it blocked stitching permanently for+any repo containing one. A crashed parse worker is different — it lost a file we+would have indexed — and `ParsePool.crashedTasks()` is what the gate reads for+it. Counting them meant a single empty `__init__.py` disqualified a repo from+stitching permanently, `--force` included: the run printed nothing, exited 0, and+left the cross-repo edges stale, while the message above advertised `--force` as+the way to fix it. `skipReasons.unchanged` in `--format json` is now that narrower+count rather than every skip, and `skipReasons.emptyFile`, previously hardcoded to+`0`, is the real number.++| Variable | Default | Effect |+|---|---|---|+| `IX_STITCH_COOLDOWN_MS` | `900000` | How long to hold off after a stitch that did not prove it stopped, measured from when the attempt ended. `0` disables the cooldown; single-flight stays. Values under ~2 min are not honoured after a killed process — see above. |+| `IX_STITCH_WAIT_MS` | `30000` | How long to wait for an in-flight stitch before skipping. `0` sheds immediately. |+| `IX_LOCK_DIR` | `~/.ix/locks` | Where the stitch lock and cooldown record live (shared with the map lock). `ix reset` clears the cooldown, so the full re-ingest that follows one is not refused by it. |+| `IX_MAP_LOCK_MAX_MS` | `1200000` | Shared with the map lock: how old a held lock must be before it is presumed abandoned and stolen. Lowering it to a few seconds so a wedged `ix map` self-heals faster also lets a second process steal the stitch lock from an in-flight stitch. The cooldown normally catches that on the next read, so it only matters together with `IX_STITCH_COOLDOWN_MS=0` — which is the one case where "single-flight stays" stops being true. |++This bounds the client. Cancelling the server-side query when the client hangs+up, and making the join indexed rather than a full scan, are backend concerns+and are not addressed here.+#### GET `/v1/stitch/system/{workspaceId}`Return the `system_id` a workspace currently belongs to (null for a singleton).diff --git a/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts b/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.tsindex 9ab2f4c7..ae64a53b 100644--- a/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts+++ b/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts@@ -1,6 +1,6 @@import { describe, expect, it } from "vitest";-import { describeCommitOutcome, describeStitchFailure, ingestCompletedCleanly, isStitchUnsupported } from "../commands/ingest.js";+import { describeCommitOutcome, describeStitchFailure, describeStitchSkipped, ingestCompletedCleanly, isStitchUnsupported } from "../commands/ingest.js";describe("describeCommitOutcome", () => {it("says nothing when every patch committed", () => {@@ -235,6 +235,66 @@ describe("describeStitchFailure", () => {});});+describe("describeStitchSkipped", () => {+ const cooling = "the last stitch was cut off after 62s and may still be running";+ const contended = "another ix run is already stitching http://localhost:8090";++ it("is a Note, not a failure, and leads with the reason", () => {+ const msg = describeStitchSkipped(cooling, "cooling");+ expect(msg.startsWith("Note:")).toBe(true);+ expect(msg).not.toContain("Warning");+ expect(msg).toContain(cooling);+ });++ it("stays short by default, but still carries a remedy that works", () => {+ // 15 minutes of auto-map hooks, one Note each, so the long form's three+ // sentences of unchanging advice were too much. But the reason alone is not+ // enough either: waiting the cooldown out and re-running does NOT recover,+ // because the mtime baseline is already written and the next map never+ // enters the stitch block. The one thing that does has to be in the short+ // form.+ const msg = describeStitchSkipped(cooling, "cooling");+ expect(msg.length).toBeLessThan(260);+ expect(msg).toContain("ix ingest <root> --force");+ expect(msg).not.toContain("Source patches were committed");+ });++ it("offers the plain re-run FIRST on contention, but still names --force", () => {+ // --verbose is off by default, so this is the advice almost everyone sees,+ // and it has to be both true and proportionate. The other run may be+ // registering this same workspace and finishing a second later, so a forced+ // monorepo re-ingest is the wrong thing to lead with -- but a plain re-run+ // ALONE was a promise this code cannot keep: a skipped stitch does not stop+ // the run persisting its mtime baseline, so the re-run is incremental and+ // never reaches the stitch block. It prints nothing and exits 0.+ const msg = describeStitchSkipped(contended, "in-flight");+ expect(msg).toContain("Re-run once that finishes");+ expect(msg).toContain("--force");+ expect(msg.indexOf("Re-run")).toBeLessThan(msg.indexOf("--force"));+ });++ it("names --force on the deadline rule too, since a longer budget alone does not register", () => {+ const msg = describeStitchSkipped("the map ran out of time", "deadline");+ expect(msg).toContain("IX_MAP_DEADLINE_MS");+ expect(msg).toContain("--force");++ const verbose = describeStitchSkipped("the map ran out of time", "deadline", true);+ expect(verbose).toContain("nothing was registered either");+ expect(verbose).toContain("--force");+ });++ it("gives the remedy under --verbose, and picks it by RULE", () => {+ const cool = describeStitchSkipped(cooling, "cooling", true);+ expect(cool).toContain("Once the cooldown expires");+ expect(cool).toContain("ix ingest <root> --force");+ expect(cool).toContain("Source patches were committed");++ const busy = describeStitchSkipped(contended, "in-flight", true);+ expect(busy).toContain("may be registering a different workspace");+ expect(busy).not.toContain("Once the cooldown expires");+ });+});+describe("isStitchUnsupported", () => {it.each([404, 501])("treats %d as a backend that does not implement stitch", (status) => {expect(isStitchUnsupported(new Error(`${status}: Not Found`))).toBe(true);diff --git a/ix-cli/src/cli/__tests__/parse-pool.test.ts b/ix-cli/src/cli/__tests__/parse-pool.test.tsnew file mode 100644index 00000000..07f08201--- /dev/null+++ b/ix-cli/src/cli/__tests__/parse-pool.test.ts@@ -0,0 +1,153 @@+import { afterEach, beforeEach, describe, expect, it } from "vitest";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { ParsePool } from "../commands/parse-pool.js";++/**+ * These exist because this file has produced a HANG three separate ways, and+ * every one of them was found by a reviewer or by timing a real `ix ingest`+ * rather than by the suite:+ *+ * - a worker that emitted 'error' while idle was spliced out of `workers` but+ * left in `idle`, so a later `drain()` posted to a dead thread;+ * - an 'exit' listener added to close that hole treated `destroy()`'s own+ * `terminate()` as a crash and respawned the pool it was closing, and a+ * worker thread refs the event loop, so the CLI hung after printing its+ * summary;+ * - the respawn cap that bounded THAT could empty the pool, and only the+ * queue existing at that instant was resolved -- every later `parse()`+ * waited forever.+ *+ * Each test below is one of those. They use real worker threads, because every+ * one of these bugs is in the interaction with the thread lifecycle and a fake+ * would have reproduced none of them.+ */+describe("ParsePool", () => {+ let dir: string;++ beforeEach(() => {+ dir = mkdtempSync(join(tmpdir(), "ix-parse-pool-"));+ });++ afterEach(() => {+ rmSync(dir, { recursive: true, force: true });+ });++ /** Write a worker and return its path. */+ function worker(name: string, body: string): string {+ const path = join(dir, `${name}.mjs`);+ writeFileSync(path, body, "utf8");+ return path;+ }++ const ECHO = `+ import { parentPort } from 'node:worker_threads';+ parentPort.on('message', ({ filePath }) => {+ parentPort.postMessage({ ok: true, result: { filePath } });+ });+ `;++ /** Exits on the first message, without ever emitting 'error'. */+ const QUIET_DEATH = `+ import { parentPort } from 'node:worker_threads';+ parentPort.on('message', () => { process.exit(1); });+ `;++ /** Dies on construction, deterministically — the spawn-loop shape. */+ const BORN_DEAD = `+ process.exit(1);+ `;++ /** Answers its first message, then faults while IDLE. */+ const IDLE_FAULT = `+ import { parentPort } from 'node:worker_threads';+ let served = 0;+ parentPort.on('message', ({ filePath }) => {+ parentPort.postMessage({ ok: true, result: { filePath } });+ if (++served === 1) setTimeout(() => { throw new Error('idle fault'); }, 20);+ });+ `;++ it("parses through the pool and shuts down without hanging", async () => {+ const pool = new ParsePool(worker("echo", ECHO), 2);+ pool.init();++ const results = await Promise.all([pool.parse("a.ts", "x"), pool.parse("b.ts", "y")]);++ expect(results).toEqual([{ filePath: "a.ts" }, { filePath: "b.ts" }]);+ expect(pool.crashedTasks()).toBe(0);++ // The hang: `destroy()` terminates every worker, each emits 'exit', and an+ // exit handler that respawns hands back a pool of live threads that nothing+ // owns. If that happens this test still passes -- the process is what hangs+ // -- so `crashedTasks()` is asserted as the observable proxy: a deliberate+ // teardown must not be recorded as a crash.+ await pool.destroy();+ expect(pool.crashedTasks(), "destroy() must not read as a crash").toBe(0);+ });++ it("resolves the in-flight task of a worker that dies without an error", async () => {+ // `parseFile` catches its own exceptions, so a worker that goes quiet --+ // `process.exit()`, or the thread killed under the runtime -- emits no+ // 'error' at all. Before the 'exit' listener its task never settled and the+ // `Promise.all` over the parse batch waited forever.+ const pool = new ParsePool(worker("quiet", QUIET_DEATH), 1);+ pool.init();++ await expect(pool.parse("a.ts", "x")).resolves.toBeNull();+ expect(pool.crashedTasks()).toBeGreaterThan(0);++ await pool.destroy();+ });++ it("takes a worker that faults while IDLE out of the free list", async () => {+ // The first of the three hangs. A worker can emit 'error' with no task in+ // flight -- an uncaught async throw, ERR_WORKER_OUT_OF_MEMORY between+ // tasks -- and it was spliced out of `workers` but left in `idle`, so the+ // next `drain()` popped the terminated thread and posted to nothing: that+ // task's promise never settled.+ const pool = new ParsePool(worker("idlefault", IDLE_FAULT), 1);+ pool.init();++ expect(await pool.parse("first.ts", "x")).toEqual({ filePath: "first.ts" });+ // Let the fault land while the pool is idle.+ await new Promise(resolve => setTimeout(resolve, 120));++ // TWO at once, deliberately. `drain()` pops the free list, so a single+ // parse takes the replacement worker that `onError` just pushed and never+ // touches the dead entry underneath it -- the bug hides completely at+ // depth 1. The second parse is the one that reaches the terminated thread,+ // posts to nothing, and never settles.+ const both = await Promise.all([pool.parse("a2.ts", "x"), pool.parse("b2.ts", "x")]);+ expect(both).toEqual([{ filePath: "a2.ts" }, { filePath: "b2.ts" }]);++ await pool.destroy();+ });++ it("does not queue forever once the respawn cap has emptied the pool", async () => {+ // A worker that dies deterministically on construction burns through the+ // cap. Draining only the queue that existed at that instant left every+ // LATER parse waiting on a `drain()` that is a no-op with no idle workers.+ const pool = new ParsePool(worker("dead", BORN_DEAD), 1);+ pool.init();++ // Enough calls to outlast the cap, then more after it.+ const first = await Promise.all(+ Array.from({ length: 20 }, (_, i) => pool.parse(`f${i}.ts`, "x")),+ );+ expect(first.every(r => r === null)).toBe(true);++ // The regression: these arrive after the pool is dead.+ await expect(pool.parse("later.ts", "x")).resolves.toBeNull();+ await expect(+ Promise.all([pool.parse("l1.ts", "x"), pool.parse("l2.ts", "x")]),+ ).resolves.toEqual([null, null]);++ // And every one of them is counted, so the stitch gate sees the loss.+ expect(pool.crashedTasks()).toBeGreaterThanOrEqual(23);++ await pool.destroy();+ });+});diff --git a/ix-cli/src/cli/__tests__/stitch-guard.test.ts b/ix-cli/src/cli/__tests__/stitch-guard.test.tsnew file mode 100644index 00000000..56a6df3e--- /dev/null+++ b/ix-cli/src/cli/__tests__/stitch-guard.test.ts@@ -0,0 +1,668 @@+import { afterEach, beforeEach, describe, expect, it } from "vitest";+import { existsSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ admitStitch,+ admitStitchWaiting,+ clearStitchCooldown,+ connectionNeverEstablished,+ cooldownPathForTest,+ outcomeProvesNothingRunning,+ stitchKey,+} from "../stitch-guard.js";+import * as net from "node:net";+import { createServer } from "node:http";+import { IxClient } from "../../client/api.js";+import { namedLockPath } from "../single-flight.js";++const ENDPOINT = "http://localhost:8090";+const OTHER = "http://localhost:9999";++let lockDir: string;+let savedLockDir: string | undefined;+let savedCooldown: string | undefined;++beforeEach(() => {+ // IX_LOCK_DIR, not HOME: single-flight reads it from process.env on every+ // call, so the redirect actually takes effect. (Overriding HOME would not --+ // Windows os.homedir() reads USERPROFILE.)+ lockDir = mkdtempSync(join(tmpdir(), "ix-stitch-guard-"));+ savedLockDir = process.env.IX_LOCK_DIR;+ savedCooldown = process.env.IX_STITCH_COOLDOWN_MS;+ process.env.IX_LOCK_DIR = lockDir;+ delete process.env.IX_STITCH_COOLDOWN_MS;+});++afterEach(() => {+ const restore = (k: string, v: string | undefined): void => {+ if (v === undefined) delete process.env[k];+ else process.env[k] = v;+ };+ restore("IX_LOCK_DIR", savedLockDir);+ restore("IX_STITCH_COOLDOWN_MS", savedCooldown);+ rmSync(lockDir, { recursive: true, force: true });+});++/**+ * Was the admission granted? Releases it either way.+ *+ * Every GRANTED admission holds a real lock handle, and single-flight arms+ * `exit`, `SIGINT` and `SIGTERM` listeners on each one. An assertion that reads+ * `.admitted` inline and drops the handle leaks all three, and enough of them+ * made this file print MaxListenersExceededWarning three times -- noise that+ * would hide a real listener leak the next time one appears.+ *+ * Settling as a clean success is the right release for these: every call below+ * is a terminal assertion, so clearing the marker changes nothing later read.+ */+function admitted(endpoint: string, now?: number): boolean {+ const a = now === undefined ? admitStitch(endpoint) : admitStitch(endpoint, now);+ if (a.admitted) a.settle({ ok: true, elapsedMs: 1 });+ return a.admitted;+}++/** Take an admission and settle it, asserting it was granted. */+function stitchOnce(outcome: { ok: boolean; elapsedMs: number; status?: number | null }): void {+ const a = admitStitch(ENDPOINT);+ expect(a.admitted, "admitStitch refused when the test needed it granted").toBe(true);+ if (a.admitted) a.settle(outcome);+}++describe("outcomeProvesNothingRunning", () => {+ // The guard writes the cooldown when the stitch STARTS, so this function does+ // not decide whether to cool down -- it decides whether we have PROOF that we+ // need not. Only two facts qualify, and neither is an inference from the+ // shape of a failure.++ it("is true for a success", () => {+ expect(outcomeProvesNothingRunning({ ok: true, elapsedMs: 90_000 })).toBe(true);+ });++ it("is true for 501, which is how this codebase spells 'no /v1/stitch'", () => {+ // isStitchUnsupported treats 404 AND 501 as "this backend has no stitch",+ // and the run swallows that silently -- so a marker left behind would refuse+ // every map for 15 minutes claiming a join may still be running, against a+ // backend that has never had one.+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 30, status: 501 })).toBe(true);+ });++ it("is true for a 4xx: the backend refused it rather than ran it", () => {+ // Including a SLOW one. Elapsed covers the request upload, and the stitch+ // payload is megabytes on a large monorepo, so a 413 can arrive tens of+ // seconds in -- the old elapsed rule read that as "cut off mid-join".+ for (const status of [400, 404, 413, 422]) {+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 25_000, status })).toBe(true);+ }+ });++ it("is false for 408, which is a proxy saying IT gave up waiting", () => {+ // Says nothing about whether the backend did, and that is the whole bug.+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 60_000, status: 408 })).toBe(false);+ });++ it("is false for a 5xx, a transport error, and an abort alike", () => {+ // The reported case is a 500 at 60s. The others reach here with no status+ // at all, and none of them proves the join stopped.+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 60_000, status: 500 })).toBe(false);+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 60_000, status: 504 })).toBe(false);+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 40, status: null })).toBe(false);+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 40 })).toBe(false);+ });++ it("does not consult the clock at all", () => {+ // Four review rounds of an elapsed-based rule each broke the previous+ // round's version of it. Same status, opposite ends of the clock, one answer.+ for (const elapsedMs of [0, 40, 25_000, 600_000]) {+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs, status: 413 })).toBe(true);+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs, status: 500 })).toBe(false);+ }+ });+});++describe("the cooldown is written when the stitch starts", () => {+ it("is on disk before the caller has sent anything", () => {+ // This is the whole design. Every way an attempt can end without a definite+ // refusal -- timeout, abort, deadline, SIGTERM, power cut -- leaves it in+ // place, because leaving it is the default rather than something a dying+ // process must still manage to do.+ const a = admitStitch(ENDPOINT);+ expect(a.admitted).toBe(true);+ expect(existsSync(cooldownPathForTest(ENDPOINT)), "marker must exist before the request").toBe(true);+ if (a.admitted) a.settle({ ok: true, elapsedMs: 30 });+ });++ it("survives a process that is killed mid-stitch and never settles", () => {+ // A hook whose timeout is shorter than the stitch. single-flight's+ // SIGINT/SIGTERM handlers release the lock and exit, so the NEXT map finds+ // the lock free -- and before this design there was no cooldown either,+ // because writing one was something the dying process still had to do. It+ // was admitted, and pushed a second join onto the one still running.+ const killed = admitStitch(ENDPOINT);+ expect(killed.admitted).toBe(true);+ // The signal handler's release(), and then nothing. No settle() call.+ rmSync(namedLockPath("stitch", stitchKey(ENDPOINT)), { force: true });++ const next = admitStitch(ENDPOINT);+ expect(next.admitted).toBe(false);+ if (!next.admitted) {+ expect(next.rule).toBe("cooling");+ expect(next.reason).toContain("never reported back");+ }+ });++ it("is cleared by a success", () => {+ stitchOnce({ ok: true, elapsedMs: 300 });+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(false);+ expect(admitted(ENDPOINT)).toBe(true);+ });++ it("is cleared by a 4xx, so an older backend with no /v1/stitch keeps working", () => {+ stitchOnce({ ok: false, elapsedMs: 40, status: 404 });+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(false);+ expect(admitted(ENDPOINT)).toBe(true);+ });++ it("is kept by a 500, and says how long the attempt ran", () => {+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });++ const next = admitStitch(ENDPOINT);+ expect(next.admitted).toBe(false);+ if (!next.admitted) {+ expect(next.reason).toContain("62s");+ expect(next.reason).toContain("may still be running");+ expect(next.reason).toContain("IX_STITCH_COOLDOWN_MS=0");+ }+ });++ it("is kept when the run deadline aborts before the request even left", () => {+ // Conservative by construction: no status, so no proof, so the marker+ // stays. The old rules had to distinguish this from a mid-flight abort and+ // got it wrong in both directions across rounds 2, 3 and 4.+ stitchOnce({ ok: false, elapsedMs: 0, status: null });+ expect(admitted(ENDPOINT)).toBe(false);+ });+});++describe("cooldown expiry", () => {+ it("admits again once the cooldown has expired", () => {+ process.env.IX_STITCH_COOLDOWN_MS = "1000";+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });++ expect(admitted(ENDPOINT, Date.now() + 500)).toBe(false);+ expect(admitted(ENDPOINT, Date.now() + 5_000)).toBe(true);+ });++ it("applies a changed IX_STITCH_COOLDOWN_MS to a cooldown already on disk", () => {+ // The refusal message tells the user 0 disables this, so it has to be true+ // of the cooldown they are looking at -- otherwise the only escape is+ // deleting a state file whose name they cannot compute.+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });+ expect(admitted(ENDPOINT)).toBe(false);++ process.env.IX_STITCH_COOLDOWN_MS = "0";+ expect(admitted(ENDPOINT)).toBe(true);+ });++ it("a lower value shortens an active cooldown", () => {+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });+ process.env.IX_STITCH_COOLDOWN_MS = "1000";+ expect(admitted(ENDPOINT, Date.now() + 5_000)).toBe(true);+ });++ it("scopes the cooldown to one backend", () => {+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });+ expect(admitted(OTHER)).toBe(true);+ });++ it("fails open on an unreadable cooldown record rather than refusing forever", () => {+ writeFileSync(cooldownPathForTest(ENDPOINT), "{not json");+ expect(admitted(ENDPOINT)).toBe(true);+ });+});++describe("admitStitch single-flight", () => {+ it("refuses a second stitch against the same backend while one is unsettled", () => {+ const first = admitStitch(ENDPOINT);+ expect(first.admitted).toBe(true);++ const second = admitStitch(ENDPOINT);+ expect(second.admitted).toBe(false);+ if (!second.admitted) expect(second.rule).toBe("in-flight");++ if (first.admitted) first.settle({ ok: true, elapsedMs: 10 });+ expect(admitted(ENDPOINT)).toBe(true);+ });++ it("does not let one backend's in-flight stitch block another backend", () => {+ const first = admitStitch(ENDPOINT);+ expect(first.admitted).toBe(true);+ const other = admitStitch(OTHER);+ expect(other.admitted).toBe(true);++ if (first.admitted) first.settle({ ok: true, elapsedMs: 1 });+ if (other.admitted) other.settle({ ok: true, elapsedMs: 1 });+ });++ it("keys the lock on the endpoint text, not on the directory the command runs in", () => {+ // The map lock canonicalises its key with realpath because it IS a path.+ // An endpoint is not: realpath fails and the fallback resolve() would join+ // it to process.cwd(), so `ix map` run from two directories would take two+ // different "single-flight" locks and stitch twice at once.+ const cwd = process.cwd();+ const elsewhere = mkdtempSync(join(tmpdir(), "ix-stitch-cwd-"));+ try {+ const here = namedLockPath("stitch", ENDPOINT);+ process.chdir(elsewhere);+ expect(namedLockPath("stitch", ENDPOINT)).toBe(here);+ } finally {+ process.chdir(cwd);+ rmSync(elsewhere, { recursive: true, force: true });+ }+ });++ it("releases the lock even when the outcome is a failure", () => {+ stitchOnce({ ok: false, elapsedMs: 5, status: 500 });+ expect(readdirSync(lockDir).filter(f => f.endsWith(".lock"))).toEqual([]);+ });+});++describe("stitchKey", () => {+ it("folds the spellings of one local backend together", () => {+ const canonical = stitchKey("http://localhost:8090");+ expect(stitchKey("http://localhost:8090/")).toBe(canonical);+ expect(stitchKey("http://127.0.0.1:8090")).toBe(canonical);+ expect(stitchKey("HTTP://LOCALHOST:8090")).toBe(canonical);+ });++ it("treats a default port and an explicit one as one backend", () => {+ // The WHATWG URL parser strips a default port, so both spellings give+ // port === "" and this needs no defaults table of its own. Pinned because+ // the absence of one reads like an omission -- and because if the parser+ // ever stopped doing it, one backend would get two locks and two cooldowns,+ // which is exactly the concurrency stitchKey exists to prevent.+ expect(stitchKey("https://ix.example")).toBe(stitchKey("https://ix.example:443"));+ expect(stitchKey("http://ix.example")).toBe(stitchKey("http://ix.example:80"));+ expect(stitchKey("http://ix.example")).not.toBe(stitchKey("https://ix.example"));+ });++ it("keeps genuinely different backends apart", () => {+ expect(stitchKey("http://localhost:8090")).not.toBe(stitchKey("http://localhost:9090"));+ expect(stitchKey("http://localhost:8090")).not.toBe(stitchKey("http://ix.example:8090"));+ expect(stitchKey("http://localhost:8090/a")).not.toBe(stitchKey("http://localhost:8090/b"));+ });++ it("passes through something that is not a URL rather than refusing to guard", () => {+ expect(stitchKey("not a url")).toBe("not a url");+ });++ it("gives two spellings of one backend one lock", () => {+ const first = admitStitch("http://localhost:8090");+ expect(first.admitted).toBe(true);+ expect(admitted("http://127.0.0.1:8090/")).toBe(false);++ if (first.admitted) first.settle({ ok: true, elapsedMs: 1 });+ });+});++describe("the cooldown is read while holding the lock, not before it", () => {+ it("has no window in which a stale read can admit a second stitch", () => {+ // A read-then-lock order leaves a gap: a waiter reads "no cooldown", the+ // holder writes one and releases, the waiter takes the freed lock and+ // sends the second concurrent stitch. Reading under the lock closes it by+ // construction -- there is no interleaving to test, which is the point.+ const holder = admitStitch(ENDPOINT);+ expect(holder.admitted).toBe(true);+ if (holder.admitted) holder.settle({ ok: false, elapsedMs: 62_000, status: 500 });++ const waiter = admitStitch(ENDPOINT);+ expect(waiter.admitted).toBe(false);+ if (!waiter.admitted) expect(waiter.rule).toBe("cooling");+ expect(readdirSync(lockDir).filter(f => f.endsWith(".lock")), "refusal must not leak a lock").toEqual([]);+ });++ it("tells a LIVE holder apart from a finished one, which is what the lock is for", () => {+ // Both states have a marker on disk. Only the lock distinguishes them, and+ // they want opposite answers: wait for the first, refuse the second.+ const live = admitStitch(ENDPOINT);+ expect(live.admitted).toBe(true);++ const duringFlight = admitStitch(ENDPOINT);+ expect(duringFlight.admitted).toBe(false);+ if (!duringFlight.admitted) expect(duringFlight.rule).toBe("in-flight");++ if (live.admitted) live.settle({ ok: false, elapsedMs: 62_000, status: 500 });++ const afterFlight = admitStitch(ENDPOINT);+ expect(afterFlight.admitted).toBe(false);+ if (!afterFlight.admitted) expect(afterFlight.rule).toBe("cooling");+ });+});++describe("connectionNeverEstablished", () => {+ // Driven with errors Node actually produced, never hand-built ones: a+ // classifier keyed on a shape nobody verified is the failure mode this+ // codebase keeps hitting, and it looks identical to a working one.++ async function failureOf(url: string): Promise<unknown> {+ try {+ await fetch(url, { method: "POST", body: "{}" });+ return new Error("expected the request to fail");+ } catch (err) {+ return err;+ }+ }++ it("is true for a port nothing is listening on — nothing was sent", async () => {+ // Bound and closed, so the port is real and free. A hardcoded low port is+ // not an option: fetch refuses the WHATWG "bad port" list outright, and+ // that failure has no code at all — which this correctly reads as no proof.+ const probe = net.createServer();+ await new Promise<void>(resolve => probe.listen(0, "127.0.0.1", resolve));+ const port = (probe.address() as net.AddressInfo).port;+ await new Promise<void>(resolve => probe.close(() => resolve()));++ expect(connectionNeverEstablished(await failureOf(`http://127.0.0.1:${port}/v1/stitch`))).toBe(true);+ });++ it("is true for a closed port on a MULTI-ADDRESS host, which localhost is", async () => {+ // The default endpoint is `http://localhost:8090`, and localhost resolves to+ // both ::1 and 127.0.0.1 -- so happy-eyeballs wraps the refusal in an+ // AggregateError with the real errors under `.errors`, no `.cause`, and no+ // `syscall` of its own. Every other test here uses a literal IP, which takes+ // the plain-Error shape and cannot see the difference; a syscall-only rule+ // walking only `.cause` answered false for the single commonest way this+ // fires, and a backend that was merely down took the full 15-minute+ // cooldown for a request that never left the machine.+ // Bound and closed, like the sibling test: a hardcoded port turns this red+ // with a message about the classifier if anything happens to be listening.+ const probe = net.createServer();+ await new Promise<void>(resolve => probe.listen(0, "127.0.0.1", resolve));+ const port = (probe.address() as net.AddressInfo).port;+ await new Promise<void>(resolve => probe.close(() => resolve()));++ const err = await failureOf(`http://localhost:${port}/v1/stitch`);+ // The predicate is the claim on every platform.+ expect(connectionNeverEstablished(err)).toBe(true);+ // The aggregate shape only appears where localhost is dual-stack, which is+ // not everywhere; where it does, it is the trap -- no syscall of its own.+ // The platform-independent proof that the descent works is the+ // mixed-code test below.+ const cause = (err as { cause?: { name?: string; syscall?: unknown } }).cause;+ if (cause?.name === "AggregateError") expect(cause.syscall).toBeUndefined();+ });++ it("reads the sub-errors when the AggregateError carries no code of its own", () => {+ // The live localhost case above happens to set `code` on the aggregate too,+ // because both addresses failed the same way -- so the errno set alone+ // catches it and the descent into `.errors` is not what saves it there.+ // This is the case where the descent IS decisive: addresses that failed+ // DIFFERENTLY, leaving nothing at the top to key on. Both sub-errors are+ // still connect-phase, so nothing was sent either way.+ //+ // Built rather than driven: producing it needs a host resolving to one+ // unroutable and one refused address, which I could not arrange. The+ // sub-error shape is the one observed live against localhost.+ const aggregate = Object.assign(new AggregateError([+ Object.assign(new Error("connect ECONNREFUSED ::1:8090"), { code: "ECONNREFUSED", syscall: "connect" }),+ Object.assign(new Error("connect ENETUNREACH 127.0.0.1:8090"), { code: "ENETUNREACH", syscall: "connect" }),+ ], "")); // no `code`, no `syscall`+ const err = Object.assign(new TypeError("fetch failed"), { cause: aggregate });++ expect((aggregate as { code?: unknown }).code).toBeUndefined();+ expect(connectionNeverEstablished(err)).toBe(true);+ });++ it("is true for an unroutable address — a neighbouring errno, same phase", () => {+ // A VPN drop produces this, and an enumerated code list missed it: fifteen+ // minutes of endpoint-wide cooldown for a request that never left.+ //+ // Pinned, not driven. Observed live against 240.0.0.1 (reserved, so the+ // stack refuses to route) on Windows: ENETUNREACH with syscall "connect".+ // It cannot be driven in CI -- Windows rejects immediately, while Linux and+ // macOS silently hang until their own connect timeout, which failed this+ // suite on four of its five platforms.+ const err = Object.assign(new TypeError("fetch failed"), {+ cause: Object.assign(new Error("connect ENETUNREACH 240.0.0.1:8090"), {+ code: "ENETUNREACH",+ syscall: "connect",+ }),+ });+ expect(connectionNeverEstablished(err)).toBe(true);+ });++ it("is true for undici's connect timeout, which carries no syscall", () => {+ // Shape observed through IxClient.stitch against a blackholed address+ // (192.0.2.1, TEST-NET-1): ConnectTimeoutError / UND_ERR_CONNECT_TIMEOUT,+ // syscall undefined. Pinned from the observation rather than driven live,+ // because reproducing it costs undici's full 10s connect timeout.+ const observed = new TypeError("fetch failed");+ (observed as { cause?: unknown }).cause = Object.assign(+ new Error("Connect Timeout Error (attempted address: 192.0.2.1:8090)"),+ { name: "ConnectTimeoutError", code: "UND_ERR_CONNECT_TIMEOUT" },+ );+ expect(connectionNeverEstablished(observed)).toBe(true);+ });++ it("is false for a blocked port, whose failure carries no code", async () => {+ // fetch rejects port 1 before it opens anything, with a bare+ // `Error: bad port`. No code means no proof, which is the safe answer.+ expect(connectionNeverEstablished(await failureOf("http://127.0.0.1:1/v1/stitch"))).toBe(false);+ });++ it("is true for a host that does not resolve", async () => {+ expect(+ connectionNeverEstablished(await failureOf("http://ix-568-no-such-host.invalid/v1/stitch")),+ ).toBe(true);+ });++ it("is FALSE for a socket dropped after the request went out", async () => {+ // The ambiguous case, and the reason this is narrower than "a transport+ // error". Once the bytes are gone, a reset means either an upstream that+ // restarted (its join died with it) or a proxy that hung up (the join is+ // still running). Both spell UND_ERR_SOCKET, so the marker stays.+ const server = net.createServer(sock => {+ sock.once("data", () => setTimeout(() => sock.destroy(), 10));+ });+ await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));+ const port = (server.address() as net.AddressInfo).port;+ try {+ const err = await failureOf(`http://127.0.0.1:${port}/v1/stitch`);+ expect(String(err)).toContain("fetch failed");+ expect(connectionNeverEstablished(err)).toBe(false);+ } finally {+ await new Promise<void>(resolve => server.close(() => resolve()));+ }+ });++ it("clears the marker, so a backend that went down mid-run is not blamed for 15 minutes", () => {+ const a = admitStitch(ENDPOINT);+ expect(a.admitted).toBe(true);+ if (a.admitted) a.settle({ ok: false, elapsedMs: 3, status: null, neverConnected: true });+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(false);+ expect(admitted(ENDPOINT)).toBe(true);+ });+});++describe("a 2xx that still failed", () => {+ it("KEEPS the marker, because that is what a proxy error page looks like", async () => {+ // Two earlier revisions cleared it here, reasoning that the request+ // completed so the join must have. A proxy answering 200 with its own HTML+ // error page when the upstream stalls produces exactly this shape -- the+ // #528 pattern -- and clearing the marker then admits the next map onto a+ // join that is still running, which is the #568 failure itself. This is the+ // one arm that could cause it, so it is the one arm that does not fire.+ const server = createServer((_req, res) => {+ res.writeHead(200, { "Content-Type": "text/html" });+ res.end("<html>proxy says hi</html>");+ });+ await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));+ const port = (server.address() as net.AddressInfo).port;+ try {+ const client = new IxClient(`http://127.0.0.1:${port}`, undefined);+ let caught: unknown;+ try {+ await client.stitch({ workspaceId: "w", provides: [], consumes: [], exports: [], symbolConsumes: [] } as never);+ } catch (err) {+ caught = err;+ }+ // The status is in the message, which is what makes this decidable at all.+ expect(String(caught)).toContain("200: response body is not JSON");+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 5, status: 200 })).toBe(false);+ } finally {+ await new Promise<void>(resolve => server.close(() => resolve()));+ }+ });++ it("cools the endpoint down, so the next map does not stack a join", () => {+ const a = admitStitch(ENDPOINT);+ expect(a.admitted).toBe(true);+ if (a.admitted) a.settle({ ok: false, elapsedMs: 40, status: 200 });+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(true);+ expect(admitStitch(ENDPOINT).admitted).toBe(false);+ });++ it("still keeps the marker for the failures that mean a join may be running", () => {+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 5, status: 500 })).toBe(false);+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 5, status: 408 })).toBe(false);+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 5, status: null })).toBe(false);+ });++ it("a stitch that actually returned still clears it", () => {+ expect(outcomeProvesNothingRunning({ ok: true, elapsedMs: 5 })).toBe(true);+ });+});++describe("the cooldown runs from the end of the attempt, not its start", () => {+ it("does not expire mid-attempt when IX_STITCH_COOLDOWN_MS is shorter than the request", () => {+ // IxClient.post caps a request at two minutes, so before the restamp EVERY+ // setting under that bought nothing on the timeout path: the marker was+ // stamped at the start, the attempt outlived it, and the next map was+ // admitted immediately and stacked a second join.+ process.env.IX_STITCH_COOLDOWN_MS = "1000";+ const startedLongAgo = Date.now() - 60_000;++ const a = admitStitch(ENDPOINT, startedLongAgo);+ expect(a.admitted).toBe(true);+ if (a.admitted) a.settle({ ok: false, elapsedMs: 60_000, status: 500 });++ const next = admitStitch(ENDPOINT);+ expect(next.admitted, "a 1s cooldown must still be live the instant a 60s attempt ends").toBe(false);+ if (!next.admitted) expect(next.reason).toContain("60s");+ });++ it("still expires on schedule once the attempt has ended", () => {+ process.env.IX_STITCH_COOLDOWN_MS = "1000";+ const a = admitStitch(ENDPOINT, Date.now() - 60_000);+ if (a.admitted) a.settle({ ok: false, elapsedMs: 60_000, status: 500 });++ expect(admitted(ENDPOINT, Date.now() + 5_000)).toBe(true);+ });+});++describe("clearStitchCooldown, which is what `ix reset` calls", () => {+ it("releases an active cooldown so the re-ingest after a reset is admitted", () => {+ // A reset wipes the registration on the backend, and the full re-ingest+ // that follows is the ONE run that can put it back -- an incremental map+ // never reaches the stitch block. A live cooldown refusing exactly that run+ // leaves the workspace unregistered with nothing to retry, while this+ // file's own comments promise a post-reset re-map re-registers.+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });+ expect(admitStitch(ENDPOINT).admitted).toBe(false);++ clearStitchCooldown(ENDPOINT);++ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(false);+ expect(admitted(ENDPOINT)).toBe(true);+ });++ it("leaves another backend's cooldown alone", () => {+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });+ clearStitchCooldown(OTHER);+ expect(admitStitch(ENDPOINT).admitted).toBe(false);+ });++ it("is a no-op when there is no cooldown, rather than throwing", () => {+ expect(() => clearStitchCooldown(ENDPOINT)).not.toThrow();+ expect(admitted(ENDPOINT)).toBe(true);+ });+});++describe("admitStitchWaiting", () => {+ it("waits out a stitch that is merely in flight rather than shedding it", async () => {+ // On a healthy backend a stitch is over in tens of milliseconds, and two+ // maps overlapping by that much is ordinary. Shedding there loses that+ // workspace's registration until somebody re-ingests every file.+ const holder = admitStitch(ENDPOINT);+ expect(holder.admitted).toBe(true);++ let slept = 0;+ const sleep = async (ms: number): Promise<void> => {+ slept += ms;+ if (slept >= 500 && holder.admitted) holder.settle({ ok: true, elapsedMs: 30 });+ };++ const second = await admitStitchWaiting(ENDPOINT, 30_000, sleep);+ expect(second.admitted, "should have waited for the holder to finish").toBe(true);+ if (second.admitted) second.settle({ ok: true, elapsedMs: 30 });+ });++ it("gives up once the budget is spent, so an unhealthy backend still sheds", async () => {+ const holder = admitStitch(ENDPOINT);+ expect(holder.admitted).toBe(true);++ const second = await admitStitchWaiting(ENDPOINT, 1_000, async () => {});+ expect(second.admitted).toBe(false);+ if (!second.admitted) expect(second.reason).toContain("already stitching");+ });++ it("never waits on a cooldown — outlasting the last stitch is the point", async () => {+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });++ let slept = 0;+ const result = await admitStitchWaiting(ENDPOINT, 30_000, async (ms) => { slept += ms; });++ expect(result.admitted).toBe(false);+ expect(slept, "waited on a cooldown").toBe(0);+ });++ it("refuses outright when the run deadline has ALREADY fired, without marking", async () => {+ // Admission writes the cooldown marker and fetch then rejects instantly on+ // the aborted signal -- so admitting here blocks the backend for 15 minutes+ // over a request that never left the process. And it repeats: the stitch is+ // the last thing an ingest does, so a run that routinely overruns would+ // block its own next attempt every time and never stitch again.+ const result = await admitStitchWaiting(ENDPOINT, 30_000, async () => {}, { aborted: true });++ expect(result.admitted).toBe(false);+ if (!result.admitted) expect(result.rule).toBe("deadline");+ expect(existsSync(cooldownPathForTest(ENDPOINT)), "must not mark the backend").toBe(false);+ expect(readdirSync(lockDir).filter(f => f.endsWith(".lock"))).toEqual([]);+ });++ it("stops waiting when the run deadline fires WHILE it is waiting", async () => {+ // The signal flips mid-wait, which is the case the in-loop check is for --+ // and the reason it has to be read through a call rather than a property+ // access TypeScript will narrow to a constant.+ const holder = admitStitch(ENDPOINT);+ expect(holder.admitted).toBe(true);+ const budget = { aborted: false };++ let slept = 0;+ const result = await admitStitchWaiting(ENDPOINT, 30_000, async (ms) => {+ slept += ms;+ budget.aborted = true; // the run runs out mid-wait+ if (holder.admitted) holder.settle({ ok: true, elapsedMs: 5 }); // ...and the holder frees the lock+ }, budget);++ // Admitting here would mark the backend for 15 minutes over a request that+ // cannot leave: fetch rejects instantly on the aborted deadline, with no+ // status, so the marker stays. The refusal must also name the real cause --+ // reporting "in-flight" sends the user to the wrong remedy entirely.+ expect(result.admitted).toBe(false);+ if (!result.admitted) expect(result.rule).toBe("deadline");+ expect(existsSync(cooldownPathForTest(ENDPOINT)), "must not mark the backend").toBe(false);+ expect(slept, "should have stopped after the first poll").toBeLessThan(1_000);+ });+});diff --git a/ix-cli/src/cli/__tests__/stitch-scope-cache.test.ts b/ix-cli/src/cli/__tests__/stitch-scope-cache.test.tsindex 03a7dff8..f13be61c 100644--- a/ix-cli/src/cli/__tests__/stitch-scope-cache.test.ts+++ b/ix-cli/src/cli/__tests__/stitch-scope-cache.test.ts@@ -5,18 +5,33 @@ import { join } from "node:path";let home: string;let savedHome: string | undefined;+let savedUserProfile: string | undefined;beforeEach(() => {vi.resetModules();home = mkdtempSync(join(tmpdir(), "ix-stitch-scope-"));savedHome = process.env.HOME;+ // USERPROFILE as well as HOME. `config.ts` reaches the cache through+ // `os.homedir()`, which on Windows reads USERPROFILE and ignores HOME -- so+ // this redirect was inert there, and these tests wrote into the developer's+ // REAL ~/.ix. That made the file self-poisoning rather than merely untidy:+ // the "asks the backend on a miss" case recorded ws-fresh in the real home+ // on its first run and then found it cached on every run after, so the file+ // passed once per machine and failed forever. CI never saw it because each+ // job starts on a clean home.+ savedUserProfile = process.env.USERPROFILE;process.env.HOME = home;+ process.env.USERPROFILE = home;mkdirSync(join(home, ".ix"), { recursive: true });});afterEach(() => {- if (savedHome === undefined) delete process.env.HOME;- else process.env.HOME = savedHome;+ const restore = (k: string, v: string | undefined): void => {+ if (v === undefined) delete process.env[k];+ else process.env[k] = v;+ };+ restore("HOME", savedHome);+ restore("USERPROFILE", savedUserProfile);rmSync(home, { recursive: true, force: true });vi.restoreAllMocks();});diff --git a/ix-cli/src/cli/commands/ingest.ts b/ix-cli/src/cli/commands/ingest.tsindex 910eecd3..cc7c3e7b 100644--- a/ix-cli/src/cli/commands/ingest.ts+++ b/ix-cli/src/cli/commands/ingest.ts@@ -18,6 +18,7 @@ import { loadIngestionModules } from './ingestion-loader.js';import { ensureWorkspaceIdState } from '../bootstrap.js';import { detectSystem, repoWorkspaceIdFor, lookupPackage, readPackageNames, readPackageDeps } from '../system.js';import { CLIENT_EXPECTED_SCHEMA_VERSION } from '../backend-status.js';+import { admitStitchWaiting, connectionNeverEstablished, type StitchRefusal } from '../stitch-guard.js';import {createCommitBreaker,createDrainGate,@@ -400,7 +401,7 @@ async function loadStoredPatchEntities(if (entities) return entities;throw new Error(`Patch ${patchId} has no entity manifest`);} catch (err) {- if (!String(err).includes('404:')) throw err;+ if (!isNotFoundError(err)) throw err;}}throw new Error('Previous source patch was not found');@@ -469,7 +470,7 @@ export async function reconcileRemovedEntities(}}} catch (err) {- if (!String(err).includes('404:')) throw err;+ if (!isNotFoundError(err)) throw err;// Already absent. Emitting the delete anyway keeps the patch a complete// statement of intent and costs nothing.removedNodeIds.push(nodeId);@@ -918,6 +919,25 @@ export interface IngestFilesSummary {parseErrors: number;commitErrors: number;stitchErrors: number;+ /**+ * Why the cross-workspace stitch was not attempted, if it was not (Ix#568).+ *+ * A skip is not an error and deliberately does not move the exit code -- but+ * CLAUDE.md RULE 5 has hooks running `ix map --silent` and reading the exit+ * code, so without a field here an automated consumer sees a wholly+ * successful map for up to 15 minutes while cross-repo edges are stale, and+ * the only signal is an English sentence on stderr.+ */+ stitchSkipped?: string;+ /**+ * Which rule kept the stitch from happening, for a consumer that must branch.+ *+ * The prose in `stitchSkipped` is for people. `incomplete` -- an incremental+ * map with nothing complete to send -- fires on almost every run, so without+ * this a hook watching for the #568 cooldown could only tell the two apart by+ * substring-matching English, which `StitchRefusal`'s own doc forbids.+ */+ stitchSkippedRule?: StitchRefusal;}/**@@ -943,6 +963,22 @@ export function isStitchUnsupported(error: unknown): boolean {return status === 404 || status === 501;}+/**+ * Is this "the backend does not have it", as opposed to any other failure?+ *+ * Anchored, the way `stitchFailureStatus` is. A bare+ * `String(err).includes('404:')` was safe only while an unparseable body+ * surfaced as V8's `SyntaxError`, which quotes about ten characters of input.+ * `IxClient` now reports it as `"${status}: response body is not JSON: ..."`+ * with a 200-character preview -- so a gateway answering 200 with an HTML error+ * page whose title is `404: Not Found` matched, the error was swallowed as+ * "absent", and `reconcileRemovedEntities` emitted a DeleteNode for a node that+ * exists. That is the #528 proxy shape this file already cites twice.+ */+function isNotFoundError(error: unknown): boolean {+ return /^(?:Error:\s*)?404:/.test(String(error));+}+/*** Ix#528: describe the failure without echoing the response body. The observed* case is a proxy 504 whose body is a full HTML error page, and pasting that@@ -950,10 +986,111 @@ export function isStitchUnsupported(error: unknown): boolean {*/export function describeStitchFailure(error: unknown): string {const status = stitchFailureStatus(error);- const detail = status === null ? "backend request failed" : `HTTP ${status}`;+ // A 2xx here means the request succeeded and something downstream made the+ // body unreadable -- the #528 proxy shape -- so "HTTP 200" as the failure+ // status is accurate and useless. Say what actually happened.+ const detail =+ status === null+ ? "backend request failed"+ : status >= 200 && status < 300+ ? `HTTP ${status} with an unreadable body`+ : `HTTP ${status}`;return `Warning: Cross-workspace stitch failed (${detail}). Source patches were committed, but cross-repository relationships may be incomplete.`;}+/**+ * Ix#568: the stitch was never sent, and why.+ *+ * Deliberately not phrased as a failure. Nothing broke and the previous+ * registration still stands -- the same position a stitch that FAILED already+ * left the graph in. Saying only that cross-repo relationships may be+ * incomplete, with no reason, is what would make this look like a regression.+ *+ * It does not promise that the next map re-registers, because that is not+ * reliably true and predates this guard: the stitch block is gated on+ * `filesSkippedAsUnchanged === 0`, so an incremental map that skips any+ * mtime- or hash-unchanged file never reaches it -- and by then it has no+ * registration data to send+ * either, since it only parsed what changed. Its own comment already names+ * the ways back in: a fresh map, `--force`, or a post-reset re-map. What the+ * cooldown changes is only how often this path is taken, not where it leads.+ */+export function describeStitchSkipped(+ reason: string,+ rule?: StitchRefusal,+ /** Whether the caller is showing per-file detail (`--verbose` / `--debug`). */+ verbose = false,+): string {+ // Short by default. A cooldown lasts 15 minutes, and under CLAUDE.md RULE 5+ // an auto-map hook fires after every edit -- so the long form printed three+ // sentences of unchanging advice on every map in that window. The reason is+ // the part that changes and the part worth reading; the rest is on --verbose,+ // and `stitchSkipped` / `stitch_skipped` carry it to machine consumers either+ // way.+ // The short form has to carry a real remedy, not just the reason. Waiting the+ // cooldown out and re-running does NOT recover: `persistIngestBaselineIfClean`+ // has already written the mtime baseline, so the next map has+ // `filesSkipped > 0` and never enters the stitch block at all -- it prints+ // nothing and exits 0, and the user reads that as fixed while cross-repo edges+ // stay frozen. Only a run that re-ingests every file gets back in.+ // Every branch names `--force`, because nothing else re-registers this+ // workspace. A skipped stitch does NOT stop the run persisting its mtime+ // baseline -- `persistIngestBaselineIfClean` gates on parse and commit errors+ // only -- so an ordinary re-run is incremental, skips unchanged files, and+ // never enters the stitch block. "Re-run once that finishes" and "raise+ // IX_MAP_DEADLINE_MS" were both promises this code cannot keep: the re-run+ // prints nothing, exits 0, and the cross-repo edges stay exactly as stale.+ //+ // The hedge that used to justify withholding `--force` from the in-flight+ // branch is still right, and now lives in the wording rather than in the+ // omission: the other run may well be registering this same workspace, in+ // which case the force is unnecessary -- but it is the only thing that+ // recovers the case where it was not.+ const recover =+ rule === "deadline"+ ? " Raise IX_MAP_DEADLINE_MS or map a smaller path, then re-register with `ix ingest <root> --force`."+ : rule === "in-flight"+ ? " Re-run once that finishes; if the cross-repo edges still look stale, `ix ingest <root> --force`."+ : rule === "lost-parses"+ // Nothing "clears" here, and this rule fires ON a --force run, so the+ // cooling remedy told the user to wait for a condition that does not+ // exist and then re-run the command they had just run.+ ? " Re-run `ix ingest <root> --force`; if it keeps happening, the crash is reproducible and IX_PARSE_DEBUG=1 will show it."+ : rule === "run-errors"+ ? " Fix the errors above and re-run `ix ingest <root> --force`."+ : " Recover with `ix ingest <root> --force` once it clears.";+ const head = `Note: Cross-workspace stitch not started — ${reason}.${recover}`;+ if (!verbose) return head;++ // The remedy follows the RULE, never the prose. Note the other run may be+ // registering this SAME workspace: only `ix map` takes the per-workspace+ // lock, so two `ix ingest` runs on one repo, or an `ix ingest` racing an+ // `ix map`, both reach here. Hence "may be" rather than a confident+ // instruction to force a full re-ingest nobody needs.+ const remedy =+ rule === "in-flight"+ ? "That run may be registering a different workspace, in which case this one was not " ++ "registered; re-run once it has finished, or force a full re-ingest " ++ "(`ix ingest <root> --force`) if the cross-repo edges still look stale."+ : rule === "lost-parses"+ ? "A parse worker crashed, so this run never saw some files and its registration would be " ++ "missing them. Nothing is running on the backend and nothing needs to clear: re-run " ++ "`ix ingest <root> --force`, and if it recurs the crash is reproducible -- IX_PARSE_DEBUG=1 " ++ "will show what the parser choked on."+ : rule === "run-errors"+ ? "The parse or commit errors above mean this run has an incomplete picture of the repo, so " ++ "registering from it would overwrite a good registration with a worse one. Fix those and " ++ "re-run `ix ingest <root> --force`."+ : rule === "deadline"+ ? "Nothing was sent, so nothing is running -- but nothing was registered either, and a longer " ++ "budget does not fix that on its own: the next map is incremental and never reaches the stitch " ++ "block. Raise IX_MAP_DEADLINE_MS or map a smaller path, then `ix ingest <root> --force`."+ : "Once the cooldown expires and the backend is healthy, re-register with a run that re-ingests " ++ "every file (`ix ingest <root> --force`) — an incremental map that skips unchanged files does not " ++ "re-attempt the stitch.";+ return `${head} Source patches were committed; cross-repository edges are unchanged since the last successful stitch. ${remedy}`;+}+/*** Decide how loudly a run should complain about patches that failed to commit.*@@ -1348,6 +1485,12 @@ export async function ingestFiles('../../../../core-ingestion/dist/parse-worker.js',);let pool: ParsePool | null = null;+ /**+ * In-flight parses lost to a crashed worker, read where the pool is still in+ * scope. A function rather than a read at the use site because control-flow+ * narrowing makes `pool` `never` down there.+ */+ const crashedParses = (): number => (pool === null ? 0 : pool.crashedTasks());const ensureParsePool = (): ParsePool => {if (pool) return pool;pool = new ParsePool(workerPath, Math.max(1, os.cpus().length - 1));@@ -1383,7 +1526,12 @@ export async function ingestFiles(for (let i = 0; i < targets.length; i += PRESCAN_PARSE_CHUNK) {const chunk = targets.slice(i, i + PRESCAN_PARSE_CHUNK);const parsed = await Promise.all(- chunk.map(fp => ensureParsePool().parse(fp, relSources.get(fp)!).catch(() => null)),+ // `false`: this prescan is best-effort by construction -- the caller+ // drops nulls, and every file here is read and parsed again by the+ // streaming loop below. A loss here is not a file the RUN lost, and+ // counting it as one refused the stitch and withheld the baseline over+ // files that were all present.+ chunk.map(fp => ensureParsePool().parse(fp, relSources.get(fp)!, false).catch(() => null)),);for (let j = 0; j < chunk.length; j++) {if (parsed[j]) preParsed.set(chunk[j], parsed[j]);@@ -1421,10 +1569,79 @@ export async function ingestFiles(let filesChanged = 0;let patchesApplied = 0;let filesSkipped = 0;+ /**+ * Files skipped because we ASSUMED THEY WERE UNCHANGED. (Ix#568)+ *+ * A subset of `filesSkipped`, and the one the cross-workspace stitch cares+ * about. The stitch registers this repo's provides/consumes, collected over+ * the files actually parsed this run, so it must only run when that set+ * covers the whole repo -- otherwise it overwrites a full registration with+ * a partial one.+ *+ * `filesSkipped` is the wrong test for that, because it also counts files+ * that contribute nothing under ANY run: a zero-byte file and one that+ * looks minified. Both are decided by the file's own content under a rule+ * that does not vary between runs, so the collected set is complete and+ * correct without them -- and their presence in the count was silently+ * disqualifying repos from ever stitching. A single empty `__init__.py` was+ * enough, `--force` or not, which made `ix ingest <root> --force` -- the+ * remedy this command prints when it refuses a stitch -- a promise it could+ * not keep: the run would print nothing, exit 0, and leave the cross-repo+ * edges stale.+ *+ * An mtime- or hash-clean skip is different in kind: the file has content+ * we would have indexed and we did not look at it this run, so the set+ * really is missing it. So is a file the parser returned nothing for --+ * see `filesSkippedUnparsed`, which the gate reads alongside this.+ */+ let filesSkippedAsUnchanged = 0;+ /** Zero-byte files. Reported as `skipReasons.emptyFile`, which was hardcoded 0. */+ let filesSkippedAsEmpty = 0;+ /**+ * Files the parse pool returned nothing for.+ *+ * Reported, but NOT part of the stitch gate -- `ParsePool.crashedTasks()` is.+ * `parse()` resolves `null` for two different things and only one of them+ * makes the run incomplete:+ *+ * - `parseFile` returned null, which is deterministic and means the file+ * has nothing to give. The dominant cause is an unavailable optional+ * grammar -- fourteen are optionalDependencies loaded through a try/catch,+ * and `tree-sitter-sas` has no win32 prebuild at all -- and those+ * extensions are still in SUPPORTED_EXTENSIONS, so the files are+ * discovered, read, dispatched and come back null on EVERY run. Gating on+ * this count blocked stitching permanently for any repo containing one,+ * `--force` included, since forcing does not change a parse result. That+ * is the empty-`__init__.py` bug again with a different file class.+ * - a worker CRASHED with the task in flight, which is transient and did+ * lose a file we would have indexed. That one belongs in the gate.+ *+ * A parse that THREW is in the first category, not the second, even though it+ * sounds like the second: `parseFile` catches every exception itself and+ * returns null (logging only under IX_PARSE_DEBUG=1), so the worker cannot+ * tell it from a missing grammar -- and neither can this. It is also+ * deterministic for the same bytes in every case we can observe, so gating on+ * it would block stitching permanently for that repo, which is exactly the+ * bug this counter was taken OUT of the gate to fix. The residue is the+ * transient sub-case, an OOM caught by that same handler; it is narrower than+ * the bug the alternative creates, and it is a deliberate choice rather than+ * an oversight.+ *+ * Reported as `skipReasons.unparsed`, its own bucket: `parseError` counts+ * every parse-stage failure including stat, read and patch-build errors, most+ * of which are not skips, so adding to it made that number mean neither thing.+ */+ let filesSkippedUnparsed = 0;let parseErrors = 0;let commitErrors = 0;let stitchErrors = 0;let stitchError: unknown;+ // Set when the stitch was refused before it was ever sent (Ix#568).+ // Distinct from stitchError: nothing failed, so it must not count as a+ // stitch error or move the exit code -- but the user is about to be told+ // cross-repo edges may be incomplete and is owed the reason.+ let stitchSkipped: string | undefined;+ let stitchSkippedRule: StitchRefusal | undefined;// Ix#560: a backend refusing every commit gets one failed request per// patch out of the per-file fallback. Stop after a streak instead.const commitBreaker = createCommitBreaker();@@ -1582,15 +1799,29 @@ export async function ingestFiles(// Stat all files and partition into mtime-clean (skip) and mtime-changed (need hash check).const mtimeChangedPaths: string[] = [];+ /**+ * Paths this loop has already accounted for, and under which counter.+ *+ * Recorded explicitly rather than inferred from `currentMtimes`. The loop+ * leaves a path out of that map in three cases -- empty, oversized, and+ * `statSync` THREW -- and only the first two are counted here. So a file+ * that errored on the first stat and then read as size 0 on the second (a+ * transient lock, an antivirus scan, a file recreated underneath us) was+ * treated as already counted and landed in no bucket at all: absent from+ * the graph, from `filesSkipped` and from every `skipReasons` entry, which+ * is precisely what the check that reads this exists to prevent.+ */+ const accountedByStatLoop = new Map<string, 'empty' | 'tooLarge'>();for (const filePath of filePaths) {try {const st = fs.statSync(filePath);- if (st.size === 0) { filesSkipped++; progressCurrent++; continue; }- if (st.size > MAX_FILE_BYTES) { tooLarge++; progressCurrent++; continue; }+ if (st.size === 0) { filesSkipped++; filesSkippedAsEmpty++; accountedByStatLoop.set(filePath, 'empty'); progressCurrent++; continue; }+ if (st.size > MAX_FILE_BYTES) { tooLarge++; accountedByStatLoop.set(filePath, 'tooLarge'); progressCurrent++; continue; }const mtime = st.mtimeMs;currentMtimes.set(filePath, mtime);if (!opts.force && !forceReingestPaths.has(filePath) && mtimeCache.get(filePath) === mtime) {filesSkipped++;+ filesSkippedAsUnchanged++;progressCurrent++; // mtime clean — assume unchanged} else {mtimeChangedPaths.push(filePath);@@ -2489,6 +2720,7 @@ export async function ingestFiles(const hash = sha256(bytes);if (!forceReingestPaths.has(filePath) && knownHashes.get(filePath) === hash) {filesSkipped++;+ filesSkippedAsUnchanged++;progressCurrent++;continue;}@@ -2579,7 +2811,7 @@ export async function ingestFiles(const batch: ParsedFile[] = [];for (let j = 0; j < chunk.length; j++) {const parsed = parseResults[j] as any;- if (!parsed) { filesSkipped++; continue; }+ if (!parsed) { filesSkipped++; filesSkippedUnparsed++; continue; }entitiesParsed += parsed.entities.length;batch.push({ filePath: chunk[j].filePath, parsed, hash: chunk[j].hash, previousHash: chunk[j].previousHash });}@@ -2590,6 +2822,19 @@ export async function ingestFiles(}} else {// Path B: no baseline (first ingest) or --force → load modules, then stream parse + commit.+ //+ // Ix#568: this branch walks ALL of `filePaths`, re-reading, re-hashing and+ // re-parsing every one, and its hash-clean short-circuit cannot fire here+ // -- `knownHashes` is empty by construction on the non-force entry, and+ // the check is `!opts.force` on the other. So the mtime skips the stat+ // loop recorded did not actually happen, and leaving them counted refused+ // the stitch on the commonest incremental shape there is: add ONE new+ // file to a mapped repo, and `loadExistingHashes` returns nothing for it,+ // so `knownHashes.size === 0` sends the whole repo down here -- every file+ // parsed, registration complete -- while the gate still saw N-1 files+ // "skipped as unchanged" and silently declined to stitch.+ filesSkipped -= filesSkippedAsUnchanged;+ filesSkippedAsUnchanged = 0;const moduleStart = performance.now();const [ingestion, patchBuilder] = await loadIngestionModules();timings.moduleLoadMs = Math.round(performance.now() - moduleStart);@@ -2648,8 +2893,57 @@ export async function ingestFiles(let bytes!: Buffer;try {const st = await fh.stat();- if (st.size === 0) { filesSkipped++; return; }- if (st.size > MAX_FILE_BYTES) { tooLarge++; return; }+ // The stat loop above walks the same `filePaths` unconditionally,+ // so a file it already rejected is counted there and must not be+ // counted again here -- incrementing unconditionally counted every+ // empty or oversized file TWICE, invisible until+ // `skipReasons.emptyFile` stopped being hardcoded to 0 and+ // reported 2 for a repo with one empty `__init__.py`.+ //+ // `accountedByStatLoop` is exactly that record. It is a set the+ // loop writes deliberately, not `currentMtimes` read backwards:+ // that map also omits files whose `statSync` THREW, which the loop+ // counts as a parse error and not as a skip, so inferring from it+ // treated those as already counted. Reaching here for a file the+ // loop passed means it changed between the two stats -- truncated,+ // or grown past the cap -- and it must be counted, or it vanishes+ // from the graph, from `filesSkipped` and from every `skipReasons`+ // bucket at once.+ //+ // The check itself is not optional either way: it is the fstat on+ // the handle we are about to read, which is what makes the size cap+ // TOCTOU-free (CodeQL js/file-system-race).+ const countedByStatLoop = accountedByStatLoop.get(absFilePath);+ if (st.size === 0) {+ // Re-bucket rather than just skip: the stat loop may have+ // recorded this as `tooLarge`, and the file has since been+ // truncated. Leaving it there reports an empty file as+ // oversized and keeps it out of `filesSkipped` entirely.+ if (countedByStatLoop === undefined) { filesSkipped++; filesSkippedAsEmpty++; }+ else if (countedByStatLoop === 'tooLarge') { tooLarge--; filesSkipped++; filesSkippedAsEmpty++; }+ accountedByStatLoop.set(absFilePath, 'empty');+ return;+ }+ if (st.size > MAX_FILE_BYTES) {+ // ...and the mirror image, for a file that has since grown.+ if (countedByStatLoop === undefined) tooLarge++;+ else if (countedByStatLoop === 'empty') { filesSkipped--; filesSkippedAsEmpty--; tooLarge++; }+ accountedByStatLoop.set(absFilePath, 'tooLarge');+ return;+ }+ // The other direction of the same race, and it needs undoing+ // rather than skipping: the stat loop saw this file as empty or+ // oversized and counted it, and the fstat -- the one we are about+ // to READ from, so the authoritative one -- says it is neither.+ // The file is ingested, so leaving the earlier count in place+ // reports it as skipped in the very number the docs now call the+ // real one.+ if (countedByStatLoop !== undefined) {+ accountedByStatLoop.delete(absFilePath);+ // Which counter, from the record rather than a guess.+ if (countedByStatLoop === 'empty') { filesSkipped--; filesSkippedAsEmpty--; }+ else tooLarge--;+ }bytes = await fh.readFile();} finally {await fh.close();@@ -2657,6 +2951,7 @@ export async function ingestFiles(const hash = sha256(bytes);if (!opts.force && !forceReingestPaths.has(absFilePath) && knownHashes.get(absFilePath) === hash) {filesSkipped++;+ filesSkippedAsUnchanged++;return;}const previousHash = knownHashes.get(absFilePath);@@ -2684,7 +2979,7 @@ export async function ingestFiles(const f = fileData[j];if (!f) continue;const parsed = parseResults[j] as any;- if (!parsed) { filesSkipped++; continue; }+ if (!parsed) { filesSkipped++; filesSkippedUnparsed++; continue; }entitiesParsed += parsed.entities.length;batch.push({ filePath: f.filePath, parsed, hash: f.hash, previousHash: f.previousHash });}@@ -2786,7 +3081,15 @@ export async function ingestFiles(projectRoot,currentMtimes,latestRev,- parseErrors,+ // Lost parses count as parse errors HERE, whatever they are called+ // elsewhere. A run whose worker pool died resolves every later file as+ // null, and those increment `filesSkippedUnparsed` rather than+ // `parseErrors` -- so without this the run wrote an mtime baseline for+ // tens of thousands of files it never parsed or committed, exited 0, and+ // every later incremental map skipped them as unchanged. Half the graph+ // missing, recoverable only by `--force`, and worse than `main`, which+ // respawns without a cap and loses one file per crash.+ parseErrors + crashedParses(),commitErrors,undefined,nextDeletedFiles,@@ -2795,7 +3098,19 @@ export async function ingestFiles(// Migration cleanup (Ix#225 gap 2): the re-ingest under the new path-based id has// committed, so delete the OLD id's now-orphaned nodes/edges/patches. Best-effort —// a failure here is non-fatal (orphans are harmless dead storage, cleanable later).- if (workspaceMigrated && previousWorkspaceId && ingestCompletedCleanly(parseErrors, commitErrors)) {+ // `+ crashedParses()`, exactly as the baseline guard below. This one+ // DELETES the old workspace's graph, so it must be at least as strict: a+ // run whose parse pool died resolves every later file as null without+ // raising `parseErrors`, so both guards saw a clean run while the new+ // workspace held a half-populated graph -- and this one would drop the+ // complete old one permanently. The two adjacent guards must not disagree+ // about whether the run was clean, and the destructive one certainly must+ // not be the laxer of the pair.+ if (+ workspaceMigrated &&+ previousWorkspaceId &&+ ingestCompletedCleanly(parseErrors + crashedParses(), commitErrors)+ ) {try {await client.deleteWorkspace(previousWorkspaceId);if (debug) process.stderr.write(` Cleaned up pre-migration nodes under ${previousWorkspaceId}.\n`);@@ -2808,12 +3123,63 @@ export async function ingestFiles(// write the cross-repo IMPORTS edges to other separately-ingested repos.// Best-effort: a failure (or an older backend with no /v1/stitch) never fails// the ingest. Collection is over the files actually parsed this run, so it is- // only COMPLETE on a full ingest. Gating on filesSkipped === 0 means a partial- // incremental re-map (some files mtime-skipped) leaves the prior full- // registration intact rather than overwriting it with a partial set; a fresh- // map, `--force`, or a post-reset re-map re-registers. (Incremental registry- // updates that touch only changed files are a future refinement.)- if (stitchEnabled && filesSkipped === 0 && stitchFiles.length > 0 && ingestCompletedCleanly(parseErrors, commitErrors)) {+ // only COMPLETE on a full ingest. Gating on filesSkippedAsUnchanged === 0+ // means a partial incremental re-map (some files mtime- or hash-skipped)+ // leaves the prior full registration intact rather than overwriting it with a+ // partial set; a fresh map, `--force`, or a post-reset re-map re-registers.+ // (Incremental registry updates that touch only changed files are a future+ // refinement.)+ //+ // Ix#568: the two narrow counts, not `filesSkipped` -- see their+ // declarations. The broad total also includes empty and minified-looking+ // files, which contribute nothing under any run, so one empty `__init__.py`+ // used to disqualify a repo from ever stitching, with or without `--force`.+ // A CRASHED parse worker is in that category and is counted here: it+ // resolves its in-flight task as null without raising `parseErrors`, and+ // stitching then would overwrite a complete registration with one missing+ // that file's exports. A file `parseFile` simply returned null for is not --+ // that is deterministic, usually an unavailable optional grammar, and+ // gating on it blocked stitching forever for any repo containing one.+ const registrationIsComplete = filesSkippedAsUnchanged === 0 && crashedParses() === 0;+ // Every way the stitch does not happen is REPORTED, not just the guard's.+ // `stitch_skipped: null` means "the cross-repo edges are current", so every+ // path that leaves it unset while skipping the stitch says something false+ // -- and the two below are far commoner than any refusal the guard makes.+ // Neither prints a human Note or a `--silent` token: `incomplete` would+ // appear on nearly every incremental map, and a run with errors has already+ // said so, loudly, in the lines above.+ // `lost-parses` FIRST. It is the only one of the three that prints, and a+ // dead pool also raises `parseErrors`, so testing `run-errors` first meant a+ // `--force` run with one patch-build failure and four thousand lost files+ // reported the one and said nothing about the rest -- suppressing the very+ // Note this rule was added to show.+ if (stitchEnabled && crashedParses() > 0) {+ // Its OWN rule, and one that IS printed. `incomplete` is silent because+ // it fires on nearly every incremental map -- but on a `--force` run+ // nothing is skipped as unchanged, so `incomplete` there could only ever+ // mean a crashed worker, and staying silent made the recovery command+ // this CLI advertises exit 0 having quietly done nothing.+ stitchSkippedRule = "lost-parses";+ stitchSkipped =+ `a parse worker crashed and ${crashedParses()} file(s) went unparsed, so the ` ++ "cross-workspace registration would be missing them";+ } else if (stitchEnabled && !ingestCompletedCleanly(parseErrors, commitErrors)) {+ stitchSkippedRule = "run-errors";+ stitchSkipped =+ "this run had parse or commit errors, so its registration would be built from " ++ "an incomplete picture of the repo";+ } else if (stitchEnabled && !registrationIsComplete) {+ // `stitchFiles.length > 0` is deliberately NOT required on any of these+ // three. A map where nothing changed parses nothing, so `stitchFiles` is+ // empty and the field stayed unset -- reporting "the cross-repo edges are+ // current" for the commonest map there is, while a cooldown was refusing+ // stitches the whole time. The staleness is identical either way.+ stitchSkippedRule = "incomplete";+ stitchSkipped =+ "this map did not re-parse every file, so it has no complete cross-workspace " ++ "registration to send";+ }+ if (stitchEnabled && registrationIsComplete && stitchFiles.length > 0 && ingestCompletedCleanly(parseErrors, commitErrors)) {try {const entry = pickEntryFile(stitchFiles);const provides = entry@@ -2860,13 +3226,50 @@ export async function ingestFiles(symbolConsumes.push({ symbol, callerNodeId, pkg });}if (provides.length > 0 || stitchConsumes.length > 0 || stitchExports.length > 0 || symbolConsumes.length > 0) {- const res = await client.stitch({ workspaceId, provides, consumes: stitchConsumes, exports: stitchExports, symbolConsumes });- if (debug) process.stderr.write(` [stitch] provides=${provides.length} consumes=${stitchConsumes.length} exports=${stitchExports.length} symConsumes=${symbolConsumes.length} -> ${res.stitched} edges\n`);- // This call is the one thing that can change whether the workspace- // belongs to a system, and reads cache that answer on disk. Drop it- // here so the next read asks again rather than scoping to the- // pre-stitch workspace for as long as the file survives.- clearStitchScopeCache(workspaceId);+ // Ix#568: ask before stitching. The join is cross-workspace and+ // outlives the HTTP call that starts it, so `ix map`'s per-workspace+ // lock does not bound it -- see stitch-guard.ts. A refusal is not a+ // failure: the previous registration stands, exactly as it does after+ // a stitch that FAILED. It is not picked up by the next map either --+ // see describeStitchSkipped for what actually re-registers.+ const admission = await admitStitchWaiting(+ client.endpoint,+ undefined,+ undefined,+ opts.deadlineSignal,+ );+ if (!admission.admitted) {+ stitchSkipped = admission.reason;+ stitchSkippedRule = admission.rule;+ if (debug) process.stderr.write(` [stitch not started] ${admission.reason}\n`);+ } else {+ const stitchStart = performance.now();+ let res;+ try {+ res = await client.stitch({ workspaceId, provides, consumes: stitchConsumes, exports: stitchExports, symbolConsumes });+ } catch (err) {+ // Settle before rethrowing, so the shared catch below still sees+ // the original error and the lock is never held past the request.+ admission.settle({+ ok: false,+ elapsedMs: performance.now() - stitchStart,+ // Two proofs that the backend is not running a join, and only+ // two: it answered and refused (a status), or we never reached+ // it at all. Everything else -- a 5xx, a timeout, an abort, a+ // socket dropped after the request went out -- leaves it.+ status: stitchFailureStatus(err),+ neverConnected: connectionNeverEstablished(err),+ });+ throw err;+ }+ admission.settle({ ok: true, elapsedMs: performance.now() - stitchStart });+ if (debug) process.stderr.write(` [stitch] provides=${provides.length} consumes=${stitchConsumes.length} exports=${stitchExports.length} symConsumes=${symbolConsumes.length} -> ${res.stitched} edges\n`);+ // This call is the one thing that can change whether the workspace+ // belongs to a system, and reads cache that answer on disk. Drop it+ // here so the next read asks again rather than scoping to the+ // pre-stitch workspace for as long as the file survives.+ clearStitchScopeCache(workspaceId);+ }}} catch (err) {// Unsupported is not failed: an older backend answers 404 here, and@@ -2925,9 +3328,19 @@ export async function ingestFiles(const summary: IngestFilesSummary = {filesDiscovered,patchesApplied,- parseErrors,+ // `+ crashedParses()`, as the baseline and delete guards already do. Files+ // lost to a dead parse pool raise `filesSkippedUnparsed`, never+ // `parseErrors`, so without this everything downstream read the run as+ // clean: `describeDroppedFiles` stayed silent, and the completed-map checks+ // blamed the backend "after a clean local ingest" and cleared the map+ // baseline. On a co-ingest workspace the `lost-parses` channel is off too+ // (`stitchEnabled` is false there), so the run exited 0 with no signal+ // anywhere at all.+ parseErrors: parseErrors + crashedParses(),commitErrors,stitchErrors,+ stitchSkipped,+ stitchSkippedRule,};// `everTripped()` alone, and it is not a weakening: `recordFailure` has one// call site, in the per-file catch immediately after `commitErrors++`, so a@@ -2954,6 +3367,25 @@ export async function ingestFiles(if (stitchErrors > 0) {process.stderr.write(` ${describeStitchFailure(stitchError)}\n`);process.exitCode = 1;+ } else if (+ stitchSkipped !== undefined &&+ // Not on `--silent`. `ix map` always passes `suppressOutput`, and that+ // surface is deliberately one terse line per run -- which is the stated+ // reason `incomplete` is filtered out of its token. A ~250-character Note+ // on every map for the whole 15-minute cooldown is the repetition the short+ // form exists to avoid; the token still carries the rule.+ opts.suppressOutput !== true &&+ // The guard's refusals only. `incomplete` would print on nearly every+ // incremental map, and `run-errors` restates lines the run has already+ // printed; both are still on the machine surfaces.+ stitchSkippedRule !== "incomplete" &&+ stitchSkippedRule !== "run-errors"+ ) {+ // Not an error and not an exit code: nothing failed, and the graph is+ // exactly where a failed stitch would have left it. But the sentence a+ // user gets otherwise -- cross-repo relationships may be incomplete --+ // reads as an unexplained regression without the reason.+ process.stderr.write(` ${describeStitchSkipped(stitchSkipped, stitchSkippedRule, debug)}\n`);}if (commitReport.kind === "warn") {process.stderr.write(` ${commitReport.message}\n`);@@ -2979,9 +3411,32 @@ export async function ingestFiles(filesSkipped,entitiesParsed,latestRev,- skipReasons: { unchanged: filesSkipped, emptyFile: 0, parseError: parseErrors, tooLarge, minifiedLikely, outsideRoot },+ // `unchanged` is the files we ASSUMED unchanged, not every skip. It used+ // to be `filesSkipped`, which also counts empty, minified-looking and+ // unparseable files -- so a consumer reading this could not tell an+ // incremental no-op from a repo full of empty __init__.py. `emptyFile` was+ // hardcoded 0 for the same reason and is now the real count.+ //+ // `unparsed` is its own bucket. Narrowing `unchanged` left the pool's+ // nulls in no bucket at all, but folding them into `parseError` was+ // worse: that counts every parse-stage failure, including stat, read and+ // patch-build errors that were never skips, so one unstat-able file gave+ // `filesSkipped: 0` with a bucket claiming 1. `unchanged`, `emptyFile`,+ // `minifiedLikely` and `unparsed` are the buckets that are subsets of+ // `filesSkipped`; `parseError` and `tooLarge` are counted separately and+ // always were.+ skipReasons: { unchanged: filesSkippedAsUnchanged, emptyFile: filesSkippedAsEmpty, parseError: parseErrors + crashedParses(), unparsed: filesSkippedUnparsed, tooLarge, minifiedLikely, outsideRoot },commitErrors,stitchErrors,+ // Ix#568. Present only when the stitch was refused before it was sent, so+ // a consumer reading this body can tell "cross-repo edges are current"+ // from "cross-repo edges are as stale as the last successful stitch" --+ // a distinction the exit code deliberately does not make.+ // `?? null`, not left undefined: JSON.stringify drops an undefined+ // value, so a consumer could not tell the field apart from an older CLI+ // that never emitted it. `ix map --format json` does the same.+ stitchSkipped: stitchSkipped ?? null,+ stitchSkippedRule: stitchSkippedRule ?? null,elapsedSeconds: parseFloat(elapsed),timings: {...timings,@@ -2999,8 +3454,18 @@ export async function ingestFiles(console.log(` processed: ${patchesApplied} files (${elapsed}s)`);console.log(` discovered: ${filesDiscovered} files`);console.log(` changed: ${filesChanged} files`);- if (filesSkipped > 0) console.log(` ${chalk.dim('skipped unchanged:')} ${filesSkipped}`);- if (parseErrors > 0) console.log(` ${chalk.red('parse errors:')} ${parseErrors}`);+ // "skipped" without "unchanged": the count includes empty, minified-looking+ // and unparseable files, none of which were assumed unchanged. Splitting the+ // counter for the stitch gate (Ix#568) made the old label demonstrably+ // wrong on a repo with an empty __init__.py and --force.+ if (filesSkipped > 0) console.log(` ${chalk.dim('skipped:')} ${filesSkipped}`);+ // `+ crashedParses()`, matching the summary field. On a co-ingest workspace+ // `stitchEnabled` is false, so the `lost-parses` Note never fires and+ // `describeDroppedFiles` is only wired into `ix map` -- which left+ // `ix ingest` on such a workspace printing "skipped: 4000", no parse-error+ // line at all, and exiting 0 after losing its whole parse pool.+ const parseErrorsShown = parseErrors + crashedParses();+ if (parseErrorsShown > 0) console.log(` ${chalk.red('parse errors:')} ${parseErrorsShown}`);if (commitErrors > 0) console.log(` ${chalk.red('commit errors:')} ${commitErrors}`);if (tooLarge > 0) console.log(` ${chalk.dim('skipped too large:')} ${tooLarge}`);if (minifiedLikely > 0) console.log(` ${chalk.dim('skipped minified:')} ${minifiedLikely}`);diff --git a/ix-cli/src/cli/commands/map.ts b/ix-cli/src/cli/commands/map.tsindex 57f0141b..735d972c 100644--- a/ix-cli/src/cli/commands/map.ts+++ b/ix-cli/src/cli/commands/map.ts@@ -535,8 +535,33 @@ Examples:const systems = result.regions.filter(r => r.label_kind === "system").length;const subsystems = result.regions.filter(r => r.label_kind === "subsystem").length;const modules = result.regions.filter(r => r.label_kind === "module").length;+ // Ix#568: `--silent` returns before both format branches AND wins over+ // `--format llm`, so it is the one output the hooks this field exists+ // for actually see. A skipped stitch deliberately does not move the exit+ // code -- without a token here an automated consumer cannot tell a clean+ // map from one whose cross-repo edges are up to 15 minutes stale.+ // Ix#568. The RULE, not just the fact -- and not for `incomplete`.+ //+ // `--silent` is the hook surface: one terse line per run. `incomplete`+ // fires on nearly every incremental map, so emitting it here would put+ // a token on almost every line and make `stitch_skipped` useless as a+ // signal, while a consumer that actually needs to know an incremental+ // map registered nothing has `--format json` and `--format llm`, which+ // both carry it. What stays here is the guard refusing -- the case that+ // means a backend is being protected from stacked joins.+ const rule = localIngest?.stitchSkippedRule;+ const stitch =+ rule === undefined || rule === "incomplete" || rule === "run-errors"+ ? ""+ // `stitch_skipped_rule`, not `stitch_skipped`. The json and llm+ // formats put the English prose under `stitch_skipped` and the rule+ // under `stitch_skipped_rule`, and emitting the RULE under the+ // prose key here gave one name two value spaces: a hook matching+ // `stitch_skipped=cooling` on this line silently stopped matching+ // the moment it was pointed at `--format json`.+ : ` · stitch_skipped_rule=${rule}`;process.stderr.write(- `map: ${result.file_count} files · ${systems}s/${subsystems}ss/${modules}m regions · ${mapMs}ms\n`+ `map: ${result.file_count} files · ${systems}s/${subsystems}ss/${modules}m regions · ${mapMs}ms${stitch}\n`);return;}@@ -569,6 +594,14 @@ Examples:// from one missing every file that failed to build a patch (#554).parse_errors: localIngest?.parseErrors ?? 0,commit_errors: localIngest?.commitErrors ?? 0,+ // Ix#568. The whole reason this is reported at all is hooks that run+ // `ix map` and read the machine output; leaving it only in+ // `ix ingest --format json` puts it where those hooks never look.+ // `?? null`, not left undefined: JSON.stringify drops an undefined+ // value, so a consumer could not tell the field apart from an older+ // CLI that never emitted it. Its siblings are always present too.+ stitch_skipped: localIngest?.stitchSkipped ?? null,+ stitch_skipped_rule: localIngest?.stitchSkippedRule ?? null,regions: regions.map((r: any) => ({label: r.label,level: r.level,@@ -593,7 +626,7 @@ Examples:export function renderMapLlm(result: MapResult,regions: MapRegion[],- ingest?: Pick<IngestFilesSummary, "parseErrors" | "commitErrors">,+ ingest?: Pick<IngestFilesSummary, "parseErrors" | "commitErrors" | "stitchSkipped" | "stitchSkippedRule">,): void {console.log(llmLine("map", [["files", result.file_count],@@ -608,6 +641,11 @@ export function renderMapLlm(// signal are dropped (docs/llm-format.md); a clean ingest says nothing.["parse_errors", ingest?.parseErrors ? ingest.parseErrors : undefined],["commit_errors", ingest?.commitErrors ? ingest.commitErrors : undefined],+ // Ix#568. `--format llm` and `--silent` are what the hooks this field was+ // added for actually read; shipping it only in `--format json` put it+ // where they never look.+ ["stitch_skipped", ingest?.stitchSkipped],+ ["stitch_skipped_rule", ingest?.stitchSkippedRule],]));for (const r of regions) {console.log(llmLine("region", [diff --git a/ix-cli/src/cli/commands/parse-pool.ts b/ix-cli/src/cli/commands/parse-pool.tsindex 324e2aba..e25470c7 100644--- a/ix-cli/src/cli/commands/parse-pool.ts+++ b/ix-cli/src/cli/commands/parse-pool.ts@@ -13,6 +13,8 @@ type Task = {filePath: string;source: string;resolve: (result: unknown) => void;+ /** Does losing this one mean the RUN lost a file? See `parse`. */+ counts: boolean;};export class ParsePool {@@ -29,14 +31,49 @@ export class ParsePool {}}- parse(filePath: string, source: string): Promise<unknown> {+ /**+ * @param countLoss Whether losing this task means the run lost a file.+ *+ * False for the index prescan, which dispatches through this same pool and+ * whose caller drops nulls on the floor: those files are read and parsed+ * again in the streaming loop and land in the graph regardless. Counting them+ * made one transient prescan crash report `lost-parses`, refuse the stitch,+ * withhold the mtime baseline (so every later map re-read the whole repo) and+ * skip the post-migration delete -- for files that were all present. And+ * because most extensions are prescanned AND parsed, a dead pool could count+ * the same file twice and report more losses than the run discovered files.+ */+ parse(filePath: string, source: string, countLoss = true): Promise<unknown> {+ // A pool with no workers left cannot ever run this, so resolve it here+ // rather than enqueue it. Draining only the queue that existed at the+ // moment the last worker died left every LATER `parse()` waiting on a+ // `drain()` that is a no-op with an empty idle list -- the promise never+ // settled and the `Promise.all` over the next chunk hung the ingest, which+ // is the same failure this file has now produced three different ways.+ if (this.dead) {+ if (countLoss) this.crashed++;+ return Promise.resolve(null);+ }return new Promise((resolve) => {- this.queue.push({ filePath, source, resolve });+ this.queue.push({ filePath, source, resolve, counts: countLoss });this.drain();});}async destroy(): Promise<void> {+ // Set FIRST. `terminate()` makes every worker emit 'exit', and the handler+ // for that treats an exit as a crash and spawns a replacement -- so+ // shutting the pool down span up a fresh set of threads, `this.workers = []`+ // orphaned them, and because a worker thread refs the event loop and the+ // CLI has no `process.exit(0)` on its success path, `ix map` printed its+ // summary and then hung forever.+ this.destroyed = true;+ // ...and `dead`, so a `parse()` after teardown resolves instead of queueing+ // onto a pool with no workers left to run it. Not reachable from+ // `ingestFiles` today -- `destroy()` is in the outermost `finally` -- but it+ // is the identical hang this file has already produced three times, and one+ // line closes it.+ this.dead = true;await Promise.all(this.workers.map(w => w.terminate()));this.workers = [];this.idle = [];@@ -46,6 +83,14 @@ export class ParsePool {const w = new Worker(this.workerPath);w.on('message', (msg: { ok: boolean; result: unknown }) => this.onResult(w, msg));w.on('error', (err) => this.onError(w, err));+ // 'exit' as well as 'error'. A worker can go without ever emitting 'error'+ // -- `process.exit()` inside it, or the thread killed out from under the+ // runtime -- and then its in-flight task never resolves, the `Promise.all`+ // over the parse batch never settles, and the ingest hangs before it can+ // even reach `destroy()`. `onError` already covers the crash path; this+ // covers the quiet one, and both feed `crashedTasks()` so the stitch gate+ // sees a run that lost a file either way.+ w.on('exit', () => this.onError(w, new Error('parse worker exited')));this.workers.push(w);this.idle.push(w);return w;@@ -60,28 +105,106 @@ export class ParsePool {}}+ /**+ * In-flight tasks lost to a crashed worker.+ *+ * `parse()` resolves `null` for two very different things, and collapsing+ * them hid a real difference (Ix#568): a worker that CRASHED lost a file it+ * would otherwise have parsed, which is transient and makes the run's results+ * incomplete -- while `parseFile` returning null is deterministic and says+ * this file has nothing to give, most often because its grammar is an+ * unavailable optional dependency (`tree-sitter-sas` has no win32 prebuild,+ * for one). Callers that need "did we miss anything we would have indexed?"+ * must ask this and not count nulls.+ */+ crashedTasks(): number {+ return this.crashed;+ }+private onResult(w: Worker, msg: { ok: boolean; result: unknown }): void {const task = this.active.get(w);if (!task) return;this.active.delete(w);+ // A successful round trip clears the respawn budget. The cap exists to+ // stop a worker that dies deterministically from spinning spawn -> die ->+ // spawn; it is not meant to be a lifetime quota. As a per-run total it made+ // ~23 transient deaths on a long run kill the pool for good, and on `main`+ // those cost 23 files rather than the remainder of the ingest.+ this.respawns = 0;task.resolve(msg.ok ? msg.result : null);this.idle.push(w);this.drain();}+ private crashed = 0;++ /** True once `destroy()` has begun, so a deliberate exit is not read as a crash. */+ private destroyed = false;++ /** Replacements spawned for crashed workers. Capped -- see `onError`. */+ private respawns = 0;++ /**+ * True once the pool is out of workers AND out of replacements.+ *+ * Latched, because the condition is permanent: nothing spawns a worker after+ * the cap, so every later `parse()` would queue forever.+ */+ private dead = false;++ /** Generous enough for real flakiness, small enough to stop a spawn loop. */+ private static readonly MAX_RESPAWNS = 16;+private onError(w: Worker, _err: Error): void {+ // Not during shutdown. `destroy()` terminates every worker on purpose, and+ // treating that as a crash respawns the pool it is trying to close: the+ // replacements outlive `this.workers = []`, a worker thread refs the event+ // loop, and the CLI prints its summary and then hangs forever. The guard+ // lives HERE rather than on the 'exit' listener because 'error' reaches the+ // same code: a worker that faults while `destroy()` is awaiting+ // `terminate()` took the identical path.+ if (this.destroyed) return;const task = this.active.get(w);if (task) {this.active.delete(w);+ if (task.counts) this.crashed++;task.resolve(null); // isolate: failed file = null parse result}- // Replace the crashed worker+ // Replace the crashed worker -- but not forever. A worker that dies+ // deterministically and without an 'error' (a native module that calls+ // `process.exit(1)` on load, say) would otherwise spin spawn -> exit ->+ // spawn for the rest of the run. That loop is newly reachable, because+ // before the 'exit' listener a silent death was simply ignored. Past the+ // cap the pool runs smaller, which the queue drains through fine, and every+ // lost task is still counted for the stitch gate.const idx = this.workers.indexOf(w);- if (idx !== -1) {- w.terminate().catch(() => {});- this.workers.splice(idx, 1);+ if (idx === -1) return;+ w.terminate().catch(() => {});+ this.workers.splice(idx, 1);+ // ...and out of `idle` too, ALWAYS, cap or no cap. A worker can emit 'error'+ // while it is IDLE -- an uncaught async throw, or ERR_WORKER_OUT_OF_MEMORY+ // between tasks -- and leaving the terminated thread in the free list means+ // a later `drain()` pops it and posts to nothing: the task's promise never+ // settles and the `Promise.all` over the parse batch hangs the ingest.+ const idleIdx = this.idle.indexOf(w);+ if (idleIdx !== -1) this.idle.splice(idleIdx, 1);++ if (this.respawns < ParsePool.MAX_RESPAWNS) {+ this.respawns++;this.spawnWorker();- this.drain();+ } else if (this.workers.length === 0) {+ // Out of workers and out of replacements. Nothing queued can ever be+ // parsed, so resolve it rather than leave `Promise.all` waiting on a pool+ // that no longer exists -- and LATCH it, because `parse()` must answer the+ // same way for every call after this, not just for what happened to be+ // queued at this instant. Counted as crashed, which is what it is, so the+ // stitch gate knows this run lost files.+ this.dead = true;+ const stranded = this.queue.splice(0, this.queue.length);+ this.crashed += stranded.filter(t => t.counts).length;+ for (const t of stranded) t.resolve(null);+ return;}+ this.drain();}}diff --git a/ix-cli/src/cli/commands/reset.ts b/ix-cli/src/cli/commands/reset.tsindex ac11969e..9e9c2ed7 100644--- a/ix-cli/src/cli/commands/reset.ts+++ b/ix-cli/src/cli/commands/reset.ts@@ -5,6 +5,7 @@ import { IxClient } from "../../client/api.js";import { getEndpoint, clearIngestMtimeCache, clearStitchScopeCache } from "../config.js";import { canRenderProgress } from "../stderr.js";import { resolveWorkspaceId } from "../bootstrap.js";+import { clearStitchCooldown } from "../stitch-guard.js";/*** Drop this workspace's cached "am I stitched into a system?" answer.@@ -70,6 +71,19 @@ export function registerResetCommand(program: Command): void {// Clear the mtime cache so the next ix map re-ingests all filesclearIngestMtimeCache(process.cwd());clearStitchScopeCacheForCwd();+ // Ix#568: and the stitch cooldown. The re-ingest below is the one+ // run that can re-register this workspace, and a live cooldown would+ // refuse exactly it -- leaving the workspace unregistered with+ // nothing to retry, since an incremental map never reaches the stitch+ // block.+ //+ // This IS a trade, not a free clear, and the trade is worth stating:+ // wiping the graph does not stop a runaway AQL join -- that is the+ // whole premise of #568, the query outlives everything -- so clearing+ // the marker here can put a second cross-workspace join onto one that+ // is still running. One extra join against a workspace that would+ // otherwise stay unregistered with no way back.+ clearStitchCooldown(getEndpoint());console.log(chalk.green("✓") + " Code graph wiped. Planning artifacts preserved.");console.log(chalk.dim(" Run `ix map` to rebuild the code graph."));} else {@@ -77,6 +91,19 @@ export function registerResetCommand(program: Command): void {stopSpinner();clearIngestMtimeCache(process.cwd());clearStitchScopeCacheForCwd();+ // Ix#568: and the stitch cooldown. The re-ingest below is the one+ // run that can re-register this workspace, and a live cooldown would+ // refuse exactly it -- leaving the workspace unregistered with+ // nothing to retry, since an incremental map never reaches the stitch+ // block.+ //+ // This IS a trade, not a free clear, and the trade is worth stating:+ // wiping the graph does not stop a runaway AQL join -- that is the+ // whole premise of #568, the query outlives everything -- so clearing+ // the marker here can put a second cross-workspace join onto one that+ // is still running. One extra join against a workspace that would+ // otherwise stay unregistered with no way back.+ clearStitchCooldown(getEndpoint());console.log(chalk.green("✓") + " Graph wiped.");}} catch (err: any) {diff --git a/ix-cli/src/cli/single-flight.ts b/ix-cli/src/cli/single-flight.tsindex d78c3378..ade62fa7 100644--- a/ix-cli/src/cli/single-flight.ts+++ b/ix-cli/src/cli/single-flight.ts@@ -49,6 +49,12 @@ export interface LockHandle {function lockMaxMs(): number {const raw = process.env.IX_MAP_LOCK_MAX_MS;if (!raw) return DEFAULT_LOCK_MAX_MS;+ // Matched, not parsed. `Number.parseInt` reads any prefix, so an operator+ // following the env table and writing `20m` got TWENTY MILLISECONDS: every+ // held lock reads as abandoned on the next process's first look, and+ // single-flight goes inert for the map lock and the stitch lock alike. The+ // stitch cooldown backstops one of those; the map lock has nothing.+ if (!/^\d+$/.test(raw)) return DEFAULT_LOCK_MAX_MS;const n = Number.parseInt(raw, 10);return Number.isFinite(n) && n > 0 ? n : DEFAULT_LOCK_MAX_MS;}@@ -57,8 +63,22 @@ function lockPathFor(key: string): string {let canonicalKey: string;try { canonicalKey = realpathSync.native(key); }catch { canonicalKey = resolve(key); }- const h = createHash("sha256").update(canonicalKey).digest("hex").slice(0, 16);- return join(lockDir(), `map-${h}.lock`);+ return namedLockPath("map", canonicalKey);+}++/**+ * Lock file for an arbitrary key, hashed verbatim.+ *+ * The map lock canonicalises its key first because it is a filesystem path and+ * two spellings of one directory must take the same lock. A key that is not a+ * path -- a backend endpoint, say -- must NOT go through realpath: it does not+ * name a file, so realpath fails and the fallback `resolve()` would join it to+ * the current working directory, giving one endpoint a different lock per+ * directory the command happens to run from.+ */+export function namedLockPath(namespace: string, key: string): string {+ const h = createHash("sha256").update(key).digest("hex").slice(0, 16);+ return join(lockDir(), `${namespace}-${h}.lock`);}/** True when a PID is alive on this host. signal 0 = existence check, no-op. */@@ -100,8 +120,20 @@ function isStale(meta: LockMeta | null): boolean {* acquisition retried once.*/export function acquireMapLock(workspaceRoot: string, label: string): LockHandle | null {+ return acquireLockAt(lockPathFor(workspaceRoot), label);+}++/**+ * Acquire the lock stored at `path`, with the same semantics as+ * [[acquireMapLock]]: a LockHandle on success, null when a live holder owns it.+ *+ * Split out so a caller that is not keyed by a workspace root -- the stitch+ * guard, which is keyed by backend endpoint because the stitch join is+ * cross-workspace -- gets the identical, already-proven acquisition, staleness+ * and release behaviour instead of a second implementation of it.+ */+export function acquireLockAt(path: string, label: string): LockHandle | null {try { mkdirSync(lockDir(), { recursive: true }); } catch { /* best effort */ }- const path = lockPathFor(workspaceRoot);const meta: LockMeta = { pid: process.pid, host: hostname(), startedAt: Date.now(), label };const tryCreate = (): boolean => {diff --git a/ix-cli/src/cli/stitch-guard.ts b/ix-cli/src/cli/stitch-guard.tsnew file mode 100644index 00000000..136ce0af--- /dev/null+++ b/ix-cli/src/cli/stitch-guard.ts@@ -0,0 +1,569 @@+import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { dirname } from "node:path";+import { acquireLockAt, namedLockPath, type LockHandle } from "./single-flight.js";++// ---------------------------------------------------------------------------+// Admission control for POST /v1/stitch.+//+// Background (Ix#568): the cross-workspace stitch is a single unbounded AQL+// join over `symbol_consumes x provides x consumes x exports`. On a large+// multi-workspace graph it runs for minutes. The HTTP call in front of it gives+// up long before the query does — the reported case is a proxy answering 500 at+// ~60s — and ArangoDB keeps executing the query regardless. The client learning+// "that failed" is therefore NOT the same as the work having stopped.+//+// `ix map` already single-flights per workspace, and that does not help here+// for two independent reasons:+//+// * The stitch is cross-workspace, so two maps of two DIFFERENT workspaces+// hold two different locks and issue two stitches at once. Reproduced: two+// concurrent `ix map` runs put two stitch queries in flight together.+// * Copies mostly do not need concurrency at all. An auto-map hook fires one+// `ix map` per change; each completes, releases its workspace lock, and the+// next one starts a fresh stitch. Reproduced: five sequential maps, five+// stitch queries, every one started after the previous had already answered+// 500 — i.e. while its server-side join was still running.+//+// ── The rule ───────────────────────────────────────────────────────────────+//+// The cooldown marker is written when a stitch STARTS, and removed only on+// proof that nothing is running: a clean answer, or a refusal the backend+// issued without doing the work.+//+// This is deliberately the opposite way round from the obvious design, which+// inspects the FAILURE and writes a cooldown when it looks like a timeout.+// Four review rounds of that classifier each broke the previous round's rule,+// because it has to answer "did the backend start the join?" from an error, and+// the evidence for that keeps not being there:+//+// * the message is whatever proxy sits in front (a 500 here, a 504 with an+// HTML body in #528), so text matching is inert in the deployment it was+// written for;+// * elapsed time cannot tell a slow join from a slow UPLOAD, and the stitch+// payload is megabytes on a large monorepo;+// * "the client aborted" cannot tell a mid-flight hang-up from a run deadline+// that fired before the request was even serialized;+// * and none of it survives the process being killed, which is the ordinary+// end of a hook that timed out.+//+// Writing the marker up front needs none of those answers. Every way a stitch+// can end without a definite refusal — a timeout, an abort, a deadline, SIGTERM,+// a power cut — leaves the marker exactly where it was written, because leaving+// it is the default rather than something the dying process must still do.+//+// The cost is that a stitch which failed for an unclassified reason cools down+// when it need not have. That errs toward skipping one stitch rather than+// stacking joins on a database that is already the reason, which is the safe+// direction here: a skipped stitch leaves the previous registration in place,+// and #568 is a report of 10–16 concurrent copies at 800–1200% CPU.+// ---------------------------------------------------------------------------++const DEFAULT_COOLDOWN_MS = 15 * 60 * 1000;+/** Budget for waiting out a stitch that is merely in flight. */+const DEFAULT_WAIT_MS = 30 * 1000;+/** Poll interval while waiting. Short: a healthy stitch is over in ~30ms. */+const POLL_MS = 250;++function positiveEnvMs(name: string, fallback: number): number {+ const raw = process.env[name];+ if (raw === undefined || raw === "") return fallback;+ // Matched, not parsed. `Number.parseInt` reads any prefix it can, so an+ // operator following the env table and writing `15m` for the cooldown got+ // FIFTEEN MILLISECONDS -- past the `n >= 0` check, no fallback, nothing+ // logged, and `ix map` back to stacking joins with no sign anything is wrong.+ // `1e6` gave 1 the same way. `commitFailureLimit` learned this already.+ if (!/^\d+$/.test(raw)) return fallback;+ const n = Number.parseInt(raw, 10);+ // 0 is meaningful (disable), so only a negative or unparseable value falls+ // back. `Number.isFinite` alone would let "" and "abc" through as NaN.+ return Number.isFinite(n) && n >= 0 ? n : fallback;+}++/** How long to refuse a new stitch after one that may still be running. 0 disables. */+export function stitchCooldownMs(): number {+ return positiveEnvMs("IX_STITCH_COOLDOWN_MS", DEFAULT_COOLDOWN_MS);+}++/** How long to wait for the in-flight stitch to finish before shedding. */+export function stitchWaitMs(): number {+ return positiveEnvMs("IX_STITCH_WAIT_MS", DEFAULT_WAIT_MS);+}++/**+ * One key per backend, however the endpoint was spelled.+ *+ * `getEndpoint()` returns `IX_ENDPOINT` or the config file verbatim, so the+ * same backend reaches this function as `http://localhost:8090`,+ * `http://localhost:8090/` and `http://127.0.0.1:8090` depending on which of+ * those a given process was started with. Hashing the raw string gives each+ * spelling its own lock, and the guard then permits exactly the concurrency it+ * exists to prevent — an `ix mcp` server launched with an IP and a shell+ * `ix map` reading the config both stitching at once.+ *+ * Loopback spellings are folded together deliberately. `localhost`,+ * `127.0.0.1` and `::1` are not the same host in general — and the endpoint+ * is not always local either; `IxClient.isLocalEndpoint` exists precisely+ * because a remote backend is supported. The folding is justified by its cost,+ * not by a claim about deployments: two loopback spellings that really did+ * name different backends cost one deferred stitch, while not folding them+ * costs the concurrency this guard exists to prevent.+ */+export function stitchKey(endpoint: string): string {+ try {+ const u = new URL(endpoint);+ const host = ["localhost", "127.0.0.1", "[::1]", "::1"].includes(u.hostname.toLowerCase())+ ? "localhost"+ : u.hostname.toLowerCase();+ const path = u.pathname.replace(/\/+$/, "");+ // `u.port` is deliberately used raw. The WHATWG parser already strips a+ // DEFAULT port, so `https://h` and `https://h:443` both give "" and key+ // identically -- adding a defaults table here would be dead code. Verified,+ // and pinned by a test, because it looks like an omission.+ return `${u.protocol.toLowerCase()}//${host}:${u.port}${path}`;+ } catch {+ // Not a URL. Nothing to normalise, and refusing to guard is worse than+ // guarding on the raw text.+ return endpoint;+ }+}++/** Cooldown record. Sits beside the lock so IX_LOCK_DIR redirects both. */+function cooldownPath(endpoint: string): string {+ return namedLockPath("stitch", stitchKey(endpoint)).replace(/\.lock$/, ".cooldown");+}++interface Cooldown {+ /**+ * Epoch ms the cooldown runs from.+ *+ * Written as the attempt's START time, and rewritten to its END time when it+ * reports back without proving anything stopped. Both matter:+ *+ * - the start-time write is what survives a kill, and is the whole reason+ * the marker goes down before the request rather than after it;+ * - but leaving it AT the start time would make any IX_STITCH_COOLDOWN_MS+ * shorter than the attempt itself already expired by the time the attempt+ * ends. `IxClient.post` caps a request at two minutes, so every setting+ * under that silently bought nothing on the timeout path -- the next map+ * was admitted immediately and stacked a second join, which is the bug.+ *+ * A process that is killed keeps the start stamp, so a cooldown shorter than+ * the attempt is still expired there; that residue is noted in docs/api.+ */+ at: number;+ /** Set when the attempt reported back without clearing. Message only. */+ elapsedMs?: number;+}++function readCooldown(endpoint: string): Cooldown | null {+ try {+ const parsed = JSON.parse(readFileSync(cooldownPath(endpoint), "utf-8")) as Cooldown;+ return typeof parsed?.at === "number" ? parsed : null;+ } catch {+ // Missing, unreadable, or corrupt: no cooldown. Failing open here is right+ // — a lost cooldown costs one extra stitch, and refusing to stitch because+ // a state file cannot be parsed would break the feature outright.+ return null;+ }+}++/**+ * Did the request fail before the connection was ever established?+ *+ * This is the one transport fact that is a positive PROOF rather than an+ * inference about a failure: if we never connected, no bytes reached the+ * backend, so there is no join to wait out. Without it a backend that goes down+ * mid-run takes a full 15-minute endpoint-wide cooldown for a request it never+ * received.+ *+ * Keyed on the SYSCALL, not on a list of codes. `IxClient` flattens HTTP errors+ * into `"NNN: ..."` strings but rethrows transport failures intact, so what+ * arrives is Node's `TypeError: fetch failed` with the real error on `.cause` --+ * and Node stamps `syscall` on every libuv error, which says which phase failed+ * far more reliably than the errno does. Observed through `IxClient.stitch`:+ *+ * closed port code ECONNREFUSED syscall connect+ * unresolvable DNS code ENOTFOUND syscall getaddrinfo+ * unroutable address code ENETUNREACH syscall connect+ * blackholed address code UND_ERR_CONNECT_TIMEOUT syscall undefined+ * reset AFTER send code UND_ERR_SOCKET syscall undefined+ *+ * An enumerated code list got the first two and missed the next two, which is+ * the same harm for a neighbouring errno: a VPN drop or an unreachable host+ * cooling the endpoint down for fifteen minutes over a request that never left.+ * `connect` and `getaddrinfo` cover every one of those without naming them, and+ * they extend to `EHOSTUNREACH` and a connect-phase `ETIMEDOUT` for free --+ * while a `read`/`write` `ETIMEDOUT`, which happens after the bytes are gone,+ * correctly does not qualify.+ *+ * `UND_ERR_CONNECT_TIMEOUT` is undici's own and carries no syscall, so it is+ * named. `UND_ERR_SOCKET` is the deliberate exclusion: "other side closed"+ * happens after the request went out, so it is exactly the ambiguous case this+ * guard exists for -- an upstream that restarted killed its join, a proxy that+ * dropped the connection did not, and both spell the same code.+ *+ * The walk descends into `AggregateError.errors` as well as `.cause`, and the+ * errno set is kept alongside the syscall rule rather than replaced by it. That+ * is not belt and braces: a multi-address host -- which `localhost` is, and it+ * is this CLI's DEFAULT endpoint -- fails happy-eyeballs with+ *+ * AggregateError { code: "ECONNREFUSED", syscall: undefined, cause: undefined,+ * errors: [ {ECONNREFUSED, connect}, {ECONNREFUSED, connect} ] }+ *+ * so a syscall-only rule that walks only `.cause` answers false for the single+ * most common way this fires. Verified by execution: `http://localhost:8099`+ * takes that shape, `http://127.0.0.1:8099` and `http://[::1]:8099` do not.+ * Tests that used a literal IP could not see the difference.+ */+export function connectionNeverEstablished(error: unknown): boolean {+ // Codes that only ever arise before any byte is sent. `ETIMEDOUT` is+ // deliberately absent -- it is a connect timeout OR a retransmission timeout+ // long after the request left -- and is picked up by the syscall rule when it+ // is the former.+ const PRE_CONNECT = new Set([+ "ECONNREFUSED",+ "ENOTFOUND",+ "ENETUNREACH",+ "EHOSTUNREACH",+ "EAI_AGAIN",+ "UND_ERR_CONNECT_TIMEOUT",+ ]);++ // Depth-capped rather than cycle-tracked: undici nests two or three levels and+ // an AggregateError adds one, so the cap is what stops a pathological chain.+ const seen = (e: unknown, depth: number): boolean => {+ if (e === null || e === undefined || depth > 5) return false;+ const { syscall, code, cause, errors } = e as {+ syscall?: unknown; code?: unknown; cause?: unknown; errors?: unknown;+ };+ if (syscall === "connect" || syscall === "getaddrinfo") return true;+ if (typeof code === "string" && PRE_CONNECT.has(code)) return true;+ if (Array.isArray(errors) && errors.some(inner => seen(inner, depth + 1))) return true;+ return seen(cause, depth + 1);+ };+ return seen(error, 0);+}++/** How the stitch attempt ended, as the guard needs to see it. */+export interface StitchOutcome {+ ok: boolean;+ /** Wall-clock the client spent. Message only — never part of the decision. */+ elapsedMs: number;+ /** HTTP status the backend answered with, when it answered at all. */+ status?: number | null;+ /** Set when the connection was never established — see the function above. */+ neverConnected?: boolean;+}++/**+ * Did this outcome PROVE that no join is running?+ *+ * Every proof here is a positive fact rather than an inference drawn from the+ * shape of a failure:+ *+ * - the stitch succeeded, so the backend is done with it;+ * - the backend answered 4xx, which is it refusing the request rather than+ * executing it. 408 is excluded: a proxy reporting that IT gave up waiting+ * says nothing about whether the backend did, and that is this whole bug.+ * - the backend answered 501. `isStitchUnsupported` already treats 404 AND+ * 501 as "this backend has no /v1/stitch", and the run swallows it+ * silently for exactly that reason -- so leaving a marker behind would+ * refuse every map for 15 minutes, claiming a join may still be running,+ * against a backend that has never had one.+ *+ * - the connection was never established, so nothing was sent. See+ * `connectionNeverEstablished` for why that is narrower than "a transport+ * error" and why the narrowing matters;+ * A 2xx whose body would not parse is deliberately NOT on that list, though+ * two earlier revisions put it there. The argument for it was that the request+ * completed, so the join did -- and the argument against is stronger: a proxy+ * answering 200 with its own HTML error page when the upstream stalls produces+ * exactly that shape, and it is the #528 pattern. Clearing the marker there is+ * the one way any proof arm can cause the #568 failure itself, by admitting the+ * next map onto a join that is still running.+ *+ * The symmetry argument does not hold either: a proxy answering 200 with+ * PARSEABLE JSON is `ok: true` and has always cleared the marker, but a proxy+ * error page is HTML, so the unparseable case is the LIKELY one rather than the+ * equivalent one. The cost of excluding it is a cooldown after a stitch that+ * really did succeed and got garbled in transit, which is the direction this+ * whole file errs in by design.+ *+ * Anything else — another 5xx, a timeout, an abort, a socket dropped after the+ * request went out, or the process being killed before it could say anything —+ * leaves the marker.+ */+export function outcomeProvesNothingRunning(outcome: StitchOutcome): boolean {+ if (outcome.ok) return true;+ if (outcome.neverConnected === true) return true;+ const status = outcome.status;+ if (typeof status !== "number") return false;+ // Deliberately no 2xx arm -- see the note above. `ok` covers a stitch that+ // actually returned; a 2xx reaching the catch means the body was unreadable,+ // which is what a proxy error page looks like.+ if (status === 501) return true;+ return status >= 400 && status < 500 && status !== 408;+}++/**+ * Which rule refused. Callers branch on this, never on the prose.+ *+ * `incomplete`, `lost-parses` and `run-errors` are not this guard's rules --+ * they are `ix ingest`'s own gates, reported through the same field because a machine+ * consumer asking "are the cross-repo edges current?" needs the same answer for+ * them. Between them they are by far the commonest reason a stitch does not+ * happen (every incremental map, and every run with a failed patch), and+ * leaving them unreported meant the field said "current" for exactly the cases+ * it was added to describe.+ */+export type StitchRefusal =+ | "in-flight"+ | "cooling"+ | "deadline"+ | "incomplete"+ | "lost-parses"+ | "run-errors";++export type StitchAdmission =+ | { admitted: true; settle: (outcome: StitchOutcome) => void }+ | { admitted: false; rule: StitchRefusal; reason: string };++function formatMs(ms: number): string {+ // Seconds up to two minutes: the numbers that matter here are request+ // durations near the proxy timeout, and rounding 62s or 90s to whole minutes+ // loses exactly the detail the reader needs.+ return ms >= 120_000 ? `${Math.round(ms / 60_000)}m` : `${Math.round(ms / 1000)}s`;+}++/**+ * Expiry of a cooldown under the CURRENT setting.+ *+ * Re-derived rather than stamped into the record, so that lowering+ * IX_STITCH_COOLDOWN_MS — or setting it to 0, which the refusal message tells+ * the user disables this — takes effect on the cooldown they are looking at.+ * Otherwise the only escape from a 15-minute block is deleting a state file+ * whose name they cannot compute.+ */+function coolingUntil(cooldown: Cooldown): number {+ return cooldown.at + stitchCooldownMs();+}++function refuseForCooldown(cooldown: Cooldown, now: number): StitchAdmission {+ // No elapsed means the attempt never reported back at all -- killed, crashed,+ // or still running in another process right now. Worth saying differently:+ // the reader's next question is different in each case.+ const how =+ cooldown.elapsedMs === undefined+ ? "a stitch was started and never reported back"+ : `the last stitch was cut off after ${formatMs(cooldown.elapsedMs)}`;+ return {+ admitted: false,+ rule: "cooling",+ // "stitches allowed again in", not "next attempt in". Nothing attempts one+ // automatically: the stitch block is gated on nothing having been skipped+ // as unchanged, this run has already persisted its mtime baseline, and the+ // next map is incremental and never reaches it. Rounds 8, 10 and 12 each+ // removed that same promise from a remedy string -- and this one matters+ // more than those did, because it is what lands verbatim in+ // `stitch_skipped` on the json and llm surfaces, where no remedy follows it.+ reason:+ `${how} and may still be running on the backend; ` ++ `stitches allowed again in ${formatMs(coolingUntil(cooldown) - now)}, ` ++ `and \`ix ingest <root> --force\` is what re-registers ` ++ `(IX_STITCH_COOLDOWN_MS=0 disables)`,+ };+}++function writeCooldown(endpoint: string, record: Cooldown): void {+ try {+ mkdirSync(dirname(cooldownPath(endpoint)), { recursive: true });+ writeFileSync(cooldownPath(endpoint), JSON.stringify(record), { mode: 0o600 });+ } catch (err) {+ // An unwritable lock dir is the one way this guard goes silently inert, so+ // say so rather than leaving the user to wonder why it never engages. Never+ // rethrow: settle() runs inside a catch that must unwind with the ORIGINAL+ // stitch error, not with this one.+ process.stderr.write(+ ` Warning: could not record the cross-workspace stitch cooldown (${err}). ` ++ `The guard against stacking stitch queries (Ix#568) is not active.\n`,+ );+ }+}++/**+ * Ask permission to POST /v1/stitch against `endpoint`.+ *+ * On `{ admitted: true }` the cooldown marker is ALREADY on disk — see the note+ * at the top of this file. The caller should call `settle` when the attempt+ * ends; NOT calling it (a crash, a kill, a hook timeout) is a supported outcome+ * and leaves the marker in place, which is the conservative answer.+ */+export function admitStitch(endpoint: string, now = Date.now()): StitchAdmission {+ // The lock FIRST, and the cooldown only while holding it.+ //+ // Order matters, and not for the reason it first appears. Because the marker+ // is written at the START of an attempt, a marker on disk means either "an+ // attempt is running right now" or "an attempt ended without proving it+ // stopped" -- and those want opposite answers: the first should be WAITED+ // for (a healthy stitch is over in milliseconds, and shedding it loses that+ // workspace's registration), the second refused outright.+ //+ // The lock is exactly that distinction, so it is the thing to ask first. It+ // also means the cooldown is only ever read while holding the lock, which+ // removes the stale-read race a read-then-lock order has: no window exists in+ // which a holder can write a marker between our read and our acquisition.+ const lock: LockHandle | null = acquireLockAt(namedLockPath("stitch", stitchKey(endpoint)), `ix stitch ${endpoint}`);+ if (!lock) {+ return {+ admitted: false,+ rule: "in-flight",+ reason: `another ix run is already stitching ${endpoint}`,+ };+ }++ const cooldown = readCooldown(endpoint);+ if (cooldown !== null && coolingUntil(cooldown) > now) {+ lock.release();+ return refuseForCooldown(cooldown, now);+ }++ // The marker goes down BEFORE the caller sends. Everything after this point+ // is about REMOVING it, never about deciding whether to write it.+ writeCooldown(endpoint, { at: now });++ let alreadySettled = false;+ return {+ admitted: true,+ settle: (outcome) => {+ if (alreadySettled) return;+ alreadySettled = true;+ try {+ if (outcomeProvesNothingRunning(outcome)) {+ try {+ rmSync(cooldownPath(endpoint), { force: true });+ } catch (err) {+ // NOT silent, unlike the usual best-effort. `force: true` already+ // swallows ENOENT, so reaching here means the file exists and could+ // not be removed -- an EPERM or EBUSY, which Windows produces+ // readily. This is the one failure that fails CLOSED: the+ // start-time marker survives a stitch that provably finished, and+ // every map against this endpoint, for every workspace, is refused+ // for the full cooldown with "a stitch was started and never+ // reported back". `writeCooldown` warns on the mirror-image+ // failure, which merely fails open.+ // The wording follows the PROOF, because this branch runs for all+ // of them: a success, a 4xx, a 501, and a connection that never+ // opened. Saying "the stitch succeeded" after an ECONNREFUSED would+ // be plainly false on the one line the reader has to act on.+ const what = outcome.ok+ ? "the cross-workspace stitch succeeded"+ : "the cross-workspace stitch did not start a query on the backend";+ process.stderr.write(+ ` Warning: ${what}, but its cooldown marker could not be removed ` ++ `(${err}). Further stitches to this backend will be refused until it ` ++ `ages out; delete ${cooldownPath(endpoint)} to clear it.+`,+ );+ }+ } else {+ // Keep the marker, restamped to NOW so the cooldown runs from the+ // end of the attempt rather than its start -- see `Cooldown.at`. The+ // elapsed figure is message data only; it decides nothing.+ writeCooldown(endpoint, { at: Date.now(), elapsedMs: Math.round(outcome.elapsedMs) });+ }+ } finally {+ // Always, even if the state write threw: holding the lock past the+ // request would block every later stitch until it aged out.+ lock.release();+ }+ },+ };+}++/**+ * [[admitStitch]], but waits out a stitch that is merely in flight.+ *+ * Shedding on contention is right when the backend is struggling and wrong+ * when it is not. On a healthy backend the stitch takes tens of milliseconds,+ * and two `ix map` runs for two workspaces overlapping by that much is+ * ordinary in a multi-repo setup with a per-repo hook. Dropping one there+ * loses that workspace's registration until somebody re-ingests every file,+ * because the stitch block is gated on nothing having been skipped as+ * mtime- or hash-unchanged, and an incremental map never reaches it.+ *+ * So wait, briefly. A healthy holder is gone long before the budget; an+ * unhealthy one is still holding when it runs out, which is the case worth+ * shedding. The cooldown — which is what actually stops the Ix#568 pile-up —+ * is checked first and never waited on.+ */+export async function admitStitchWaiting(+ endpoint: string,+ waitMs = stitchWaitMs(),+ sleep: (ms: number) => Promise<void> = ms => new Promise(r => setTimeout(r, ms)),+ /**+ * The run's own wall-clock budget. Waiting past it is pure delay: every+ * request after it is aborted before it leaves, so the stitch this is+ * queueing for cannot be sent even if the lock frees.+ */+ runDeadline?: { readonly aborted: boolean },+): Promise<StitchAdmission> {+ // Refuse outright if the run's budget is ALREADY gone. Admission writes+ // the cooldown marker, and `fetch` rejects instantly on an aborted signal --+ // so admitting here marks the backend as maybe-busy for 15 minutes over a+ // request that never left the process. Worse, it repeats: the stitch is the+ // last thing an ingest does, so a run that routinely overruns would block+ // its own next attempt every time and never stitch again.+ // Read through a call, not a property access: after the early return below,+ // TypeScript narrows `runDeadline.aborted` to false for the rest of the+ // function and flags the in-loop check as unreachable -- but the whole point+ // of that check is that the signal flips WHILE we wait.+ const deadlineFired = (): boolean => runDeadline?.aborted === true;+ const refuseForDeadline = (): StitchAdmission => ({+ admitted: false,+ rule: "deadline",+ reason: `the map ran out of time before the stitch could start`,+ });+++ const deadline = Date.now() + waitMs;+ for (;;) {+ // Re-checked at the TOP of every iteration, before admitStitch. Checking it+ // only after admission is granted is too late: the holder can release while+ // we wait, and then we take the lock, write the marker, and `fetch` rejects+ // instantly on the already-aborted deadline -- marking the backend for 15+ // minutes over a request that never left. The pre-loop check alone does not+ // cover it, because the deadline can fire DURING the wait.+ if (deadlineFired()) return refuseForDeadline();++ const admission = admitStitch(endpoint);+ // Only contention is worth waiting out. A cooldown means the backend may+ // still be running the last one, and outlasting THAT is the whole point.+ if (admission.admitted || admission.rule !== "in-flight") return admission;+ if (Date.now() >= deadline) return admission;+ await sleep(Math.min(POLL_MS, Math.max(0, deadline - Date.now())));+ }+}++/**+ * Drop the cooldown for an endpoint, so the next stitch is admitted.+ *+ * For `ix reset`, and only for it. A reset wipes the registration on the+ * backend, and the full re-ingest that follows is the one run that can put it+ * back -- so a live cooldown refusing exactly that run leaves the workspace+ * unregistered with no automatic retry, while this file's own comments promise+ * that "a post-reset re-map re-registers". The reset also removes whatever+ * graph the outstanding join was building, so there is nothing left to protect.+ *+ * Best-effort: a cooldown that cannot be removed costs one skipped stitch, and+ * failing `ix reset` over it would be worse.+ */+export function clearStitchCooldown(endpoint: string): void {+ try {+ rmSync(cooldownPath(endpoint), { force: true });+ } catch { /* best effort */ }+}++// ── Test-only surface ──────────────────────────────────────────────────────+export function cooldownPathForTest(endpoint: string): string {+ return cooldownPath(endpoint);+}diff --git a/ix-cli/src/client/api.ts b/ix-cli/src/client/api.tsindex 8261e24c..3a0b7742 100644--- a/ix-cli/src/client/api.ts+++ b/ix-cli/src/client/api.ts@@ -558,7 +558,7 @@ export class IxClient {const text = await resp.text();throw new Error(`${resp.status}: ${text}`);}- return resp.json() as Promise<T>;+ return parseOrThrowWithStatus<T>(resp);}private async get<T>(path: string): Promise<T> {@@ -569,6 +569,28 @@ export class IxClient {const text = await resp.text();throw new Error(`${resp.status}: ${text}`);}- return resp.json() as Promise<T>;+ return parseOrThrowWithStatus<T>(resp);+ }+}++/**+ * Read a 2xx body as JSON, keeping the status in the error if it will not parse.+ *+ * A bare `resp.json()` rejects with a `SyntaxError` that says nothing about+ * what the server answered, so a caller could only infer "it must have been a+ * 2xx, because a non-2xx would have thrown above" -- an inference this codebase+ * has no business making, since the whole premise of Ix#528 and Ix#568 is that+ * whatever proxy sits in front spells things the way it likes, up to and+ * including a 200 with an HTML error page. Stating the status turns that+ * inference into a fact the error carries, and it matches the `${status}: ...`+ * shape every other failure here already uses.+ */+async function parseOrThrowWithStatus<T>(resp: Response): Promise<T> {+ const text = await resp.text();+ try {+ return JSON.parse(text) as T;+ } catch {+ const preview = text.length > 200 ? `${text.slice(0, 200)}…` : text;+ throw new Error(`${resp.status}: response body is not JSON: ${preview}`);}}