Skip to content

armgit / Ix

Public Imported from GitHub
0

Understand any codebase instantly. System intelligence for codebases, built for humans and AI.

fix(ingest): stop ix map piling stitch queries onto a backend that is still running the last one (#571)

Joseph Mikhailfb46941e6a0b
diff --git a/docs/api/README.md b/docs/api/README.md
index 37528ffb..db775749 100644
--- a/docs/api/README.md
+++ b/docs/api/README.md
@@ -156,6 +156,14 @@ Ingest a path into the graph. Long-running: the client allows **30 minutes**.
}
```
+> The **CLI's** `ix ingest --format json` emits a different, narrower breakdown
+> under the same key: `unchanged` there counts only files skipped as *mtime- or
+> hash-unchanged*, `emptyFile` is a real count rather than a hardcoded `0`, and
+> there is an extra `unparsed` bucket for files the parse pool returned nothing
+> for. That is the client's own summary of its own run and is not this response;
+> see the stitch section below for why the narrower `unchanged` is load-bearing
+> there.
+
#### POST `/v1/map`
Return the full code map (systems → subsystems → members). The visualizer and
@@ -223,6 +231,152 @@ cross-repo `IMPORTS` edges.
**Response** — `{ stitched: number, systemId: string | null, edges: [{ src, dst, name }] }`.
+**Client-side admission control (Ix#568).** The join behind this endpoint runs
+server-side for as long as it needs to, and outlives the HTTP call that started
+it — a proxy answering 500 at ~60s does not stop the query. The CLI therefore
+does not issue this call unconditionally:
+
+| Rule | Behaviour |
+|---|---|
+| One at a time **per backend endpoint** | A second `ix map` — including one for a *different* workspace — waits up to `IX_STITCH_WAIT_MS` (default 30s) for the in-flight stitch, then skips. `ix map`'s own lock is per workspace and does not bound a cross-workspace join. |
+| A cooldown written when a stitch **starts** | It is removed only on proof that nothing is running. Until then, no further stitch is sent to that endpoint for `IX_STITCH_COOLDOWN_MS` (default 15 min). |
+
+The second rule is the one that stops the pile-up, and it is written the
+opposite way round from the obvious design. Rather than inspecting the failure
+and deciding whether it looked like a timeout, the marker goes down before the
+request and comes back up only on **proof** that the backend did not run the
+join:
+
+* the stitch succeeded;
+* the backend answered **4xx**, which is it refusing the request rather than
+ executing it — with **408** excluded, since a proxy reporting that *it* gave
+ up waiting says nothing about whether the backend did;
+* the backend answered **501**, which is how this codebase already spells "no
+ `/v1/stitch` here" (`isStitchUnsupported` accepts 404 or 501);
+* the connection was never established, so no bytes reached the backend.
+ Decided by the **syscall** Node stamps on the underlying error — `connect` or
+ `getaddrinfo` — with a small errno set (`ECONNREFUSED`, `ENOTFOUND`,
+ `ENETUNREACH`, `EHOSTUNREACH`, `EAI_AGAIN`) and undici's
+ `UND_ERR_CONNECT_TIMEOUT` alongside it, and a walk into `AggregateError.errors`
+ because a multi-address host like `localhost` — the default endpoint — reports
+ a refusal that way. Deliberately narrower than "a transport error": a socket
+ dropped *after* the request went out (`UND_ERR_SOCKET`) is the ambiguous case
+ — an upstream that restarted killed its join, a proxy that hung up did not —
+ and keeps the marker. A read/write `ETIMEDOUT` is excluded for the same
+ reason; a connect-phase one is caught by the syscall.
+
+Everything else — a 5xx, a timeout, an abort, a socket dropped mid-flight, or
+the process being killed before it could report anything — leaves the marker in
+place. That
+last case is why the marker is written up front: a hook whose timeout is shorter
+than the stitch takes the CLI down mid-request, and nothing it *would* have done
+on the way out can be relied on.
+
+The cost is that a stitch failing for an unclassified reason cools down when it
+need not have. That errs toward skipping one stitch rather than stacking joins
+on a database that is already the reason.
+
+Both the lock and the cooldown are keyed on a normalised endpoint, so
+`http://localhost:8090`, `http://localhost:8090/` and `http://127.0.0.1:8090`
+are one backend rather than three. Without that, an `ix mcp` server started with
+`IX_ENDPOINT` set to an IP and a shell `ix map` reading the config file would
+each hold their own "single-flight" lock and stitch simultaneously.
+
+`IX_STITCH_COOLDOWN_MS` is re-read on every attempt and applied to cooldowns
+already on disk, so setting it to `0` releases an active one rather than only
+affecting the next.
+
+The cooldown is stamped at the stitch's **start** and re-stamped to its **end**
+when the attempt reports back without proving anything stopped. The re-stamp is
+what makes short values mean anything: `IxClient` caps a request at two minutes,
+so a cooldown measured only from the start would already have expired by the
+time a timing-out stitch returned, and the next map would be admitted straight
+into a second join. One residue remains — a process that is *killed* never
+re-stamps, so a cooldown shorter than the attempt it is protecting is expired
+when the next map looks at it. Values below the two-minute request cap are
+therefore only reliable on the paths that report back.
+
+A stitch that does not happen is reported as `stitchSkipped` in
+`ix ingest --format json`, as
+`stitch_skipped` in `ix map --format json` and `--format llm`, and as a
+`stitch_skipped_rule=<rule>` token on `ix map --silent`, so an automated consumer can tell it
+apart from a clean run. It is not an error: it does not set a non-zero exit code
+and does not count towards `stitchErrors`, and the previous registration stands —
+the same position a stitch that *failed* already left the graph in.
+
+The rule is machine-readable, and carried alongside the prose as
+`stitchSkippedRule` / `stitch_skipped_rule`: `in-flight`, `cooling` and
+`deadline` are the guard's, and **`incomplete`**, **`lost-parses`** and
+**`run-errors`** are `ix ingest`'s own gates — an incremental map that did not re-parse every file has
+no complete registration to send, and a run with parse or commit errors would
+build one from an incomplete picture of the repo. Those two are by far the
+commonest, and they are reported for the same reason as the others: a consumer
+asking "are the cross-repo edges current?" gets the wrong answer if the field is
+absent whenever the stitch was never attempted. Branch on the rule, never on the
+sentence.
+
+`incomplete` and `run-errors` print no human Note and emit no `--silent` token:
+the first would appear on nearly every incremental map, and the second restates
+lines the run has already printed. `--format json` and `--format llm` carry them
+either way. **`lost-parses`** — a parse worker crashed, so some files went
+unparsed — *is* printed, because on a `--force` run it is the only one of the
+three that can fire, and staying quiet there would make the recovery command
+this CLI advertises exit 0 having done nothing.
+
+The wait happens **inside** `ix map`'s per-workspace lock, which the run holds
+until it exits. So while one map is waiting out another repo's stitch — up to
+`IX_STITCH_WAIT_MS`, 30s by default — any further `ix map` fired for that same
+workspace (an auto-map hook, for instance) finds the lock held, coalesces, and
+exits 0. Edits made in that window get no graph refresh and nothing says so.
+That is a new source of staleness, bounded by `IX_STITCH_WAIT_MS`; set it to `0`
+to shed on contention immediately instead, at the cost of losing that map's
+cross-repo registration.
+
+Both the lock and the cooldown are keyed on the **endpoint**, not on the
+workspace, because the join they bound is cross-workspace. That is the point of
+the guard, and it is also its cost: one transient failure while mapping repo A
+refuses the stitch for repos B..E on the same backend for the whole cooldown,
+and when it expires none of them re-attempt on their own for the reason below.
+In a multi-repo setup that means every repo's cross-repo edges stay as they were
+until somebody runs a full re-ingest in each. The alternative -- keying per
+workspace -- does not bound the query at all, since the query is not per
+workspace.
+
+Note that re-registration is not automatic on the next map, and was not before
+this change: the stitch is gated on every file having been parsed this run, so an
+incremental map that skips an mtime- or hash-unchanged file neither reaches it nor
+has the registration data to send, having only parsed what changed. A run that
+re-ingests every file (`ix ingest <root> --force`, a post-reset re-map) is what
+picks it back up.
+
+The gate counts files skipped **as unchanged**, plus parses lost to a **crashed
+worker**. It used to be the whole `filesSkipped` total, which also counts
+zero-byte files, ones that look minified, and ones the parser simply returned
+nothing for — none of which have anything to contribute to the registration, so
+their absence does not make the collected set partial. That last class matters:
+fourteen tree-sitter grammars are optional dependencies, and a file whose grammar
+did not build (`tree-sitter-sas` has no win32 prebuild) comes back unparsed on
+*every* run, `--force` included, so counting it blocked stitching permanently for
+any repo containing one. A crashed parse worker is different — it lost a file we
+would have indexed — and `ParsePool.crashedTasks()` is what the gate reads for
+it. Counting them meant a single empty `__init__.py` disqualified a repo from
+stitching permanently, `--force` included: the run printed nothing, exited 0, and
+left the cross-repo edges stale, while the message above advertised `--force` as
+the way to fix it. `skipReasons.unchanged` in `--format json` is now that narrower
+count rather than every skip, and `skipReasons.emptyFile`, previously hardcoded to
+`0`, is the real number.
+
+| Variable | Default | Effect |
+|---|---|---|
+| `IX_STITCH_COOLDOWN_MS` | `900000` | How long to hold off after a stitch that did not prove it stopped, measured from when the attempt ended. `0` disables the cooldown; single-flight stays. Values under ~2 min are not honoured after a killed process — see above. |
+| `IX_STITCH_WAIT_MS` | `30000` | How long to wait for an in-flight stitch before skipping. `0` sheds immediately. |
+| `IX_LOCK_DIR` | `~/.ix/locks` | Where the stitch lock and cooldown record live (shared with the map lock). `ix reset` clears the cooldown, so the full re-ingest that follows one is not refused by it. |
+| `IX_MAP_LOCK_MAX_MS` | `1200000` | Shared with the map lock: how old a held lock must be before it is presumed abandoned and stolen. Lowering it to a few seconds so a wedged `ix map` self-heals faster also lets a second process steal the stitch lock from an in-flight stitch. The cooldown normally catches that on the next read, so it only matters together with `IX_STITCH_COOLDOWN_MS=0` — which is the one case where "single-flight stays" stops being true. |
+
+This bounds the client. Cancelling the server-side query when the client hangs
+up, and making the join indexed rather than a full scan, are backend concerns
+and are not addressed here.
+
#### GET `/v1/stitch/system/{workspaceId}`
Return the `system_id` a workspace currently belongs to (null for a singleton).
diff --git a/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts b/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts
index 9ab2f4c7..ae64a53b 100644
--- a/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts
+++ b/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts
@@ -1,6 +1,6 @@
import { describe, expect, it } from "vitest";
-import { describeCommitOutcome, describeStitchFailure, ingestCompletedCleanly, isStitchUnsupported } from "../commands/ingest.js";
+import { describeCommitOutcome, describeStitchFailure, describeStitchSkipped, ingestCompletedCleanly, isStitchUnsupported } from "../commands/ingest.js";
describe("describeCommitOutcome", () => {
it("says nothing when every patch committed", () => {
@@ -235,6 +235,66 @@ describe("describeStitchFailure", () => {
});
});
+describe("describeStitchSkipped", () => {
+ const cooling = "the last stitch was cut off after 62s and may still be running";
+ const contended = "another ix run is already stitching http://localhost:8090";
+
+ it("is a Note, not a failure, and leads with the reason", () => {
+ const msg = describeStitchSkipped(cooling, "cooling");
+ expect(msg.startsWith("Note:")).toBe(true);
+ expect(msg).not.toContain("Warning");
+ expect(msg).toContain(cooling);
+ });
+
+ it("stays short by default, but still carries a remedy that works", () => {
+ // 15 minutes of auto-map hooks, one Note each, so the long form's three
+ // sentences of unchanging advice were too much. But the reason alone is not
+ // enough either: waiting the cooldown out and re-running does NOT recover,
+ // because the mtime baseline is already written and the next map never
+ // enters the stitch block. The one thing that does has to be in the short
+ // form.
+ const msg = describeStitchSkipped(cooling, "cooling");
+ expect(msg.length).toBeLessThan(260);
+ expect(msg).toContain("ix ingest <root> --force");
+ expect(msg).not.toContain("Source patches were committed");
+ });
+
+ it("offers the plain re-run FIRST on contention, but still names --force", () => {
+ // --verbose is off by default, so this is the advice almost everyone sees,
+ // and it has to be both true and proportionate. The other run may be
+ // registering this same workspace and finishing a second later, so a forced
+ // monorepo re-ingest is the wrong thing to lead with -- but a plain re-run
+ // ALONE was a promise this code cannot keep: a skipped stitch does not stop
+ // the run persisting its mtime baseline, so the re-run is incremental and
+ // never reaches the stitch block. It prints nothing and exits 0.
+ const msg = describeStitchSkipped(contended, "in-flight");
+ expect(msg).toContain("Re-run once that finishes");
+ expect(msg).toContain("--force");
+ expect(msg.indexOf("Re-run")).toBeLessThan(msg.indexOf("--force"));
+ });
+
+ it("names --force on the deadline rule too, since a longer budget alone does not register", () => {
+ const msg = describeStitchSkipped("the map ran out of time", "deadline");
+ expect(msg).toContain("IX_MAP_DEADLINE_MS");
+ expect(msg).toContain("--force");
+
+ const verbose = describeStitchSkipped("the map ran out of time", "deadline", true);
+ expect(verbose).toContain("nothing was registered either");
+ expect(verbose).toContain("--force");
+ });
+
+ it("gives the remedy under --verbose, and picks it by RULE", () => {
+ const cool = describeStitchSkipped(cooling, "cooling", true);
+ expect(cool).toContain("Once the cooldown expires");
+ expect(cool).toContain("ix ingest <root> --force");
+ expect(cool).toContain("Source patches were committed");
+
+ const busy = describeStitchSkipped(contended, "in-flight", true);
+ expect(busy).toContain("may be registering a different workspace");
+ expect(busy).not.toContain("Once the cooldown expires");
+ });
+});
+
describe("isStitchUnsupported", () => {
it.each([404, 501])("treats %d as a backend that does not implement stitch", (status) => {
expect(isStitchUnsupported(new Error(`${status}: Not Found`))).toBe(true);
diff --git a/ix-cli/src/cli/__tests__/parse-pool.test.ts b/ix-cli/src/cli/__tests__/parse-pool.test.ts
new file mode 100644
index 00000000..07f08201
--- /dev/null
+++ b/ix-cli/src/cli/__tests__/parse-pool.test.ts
@@ -0,0 +1,153 @@
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { ParsePool } from "../commands/parse-pool.js";
+
+/**
+ * These exist because this file has produced a HANG three separate ways, and
+ * every one of them was found by a reviewer or by timing a real `ix ingest`
+ * rather than by the suite:
+ *
+ * - a worker that emitted 'error' while idle was spliced out of `workers` but
+ * left in `idle`, so a later `drain()` posted to a dead thread;
+ * - an 'exit' listener added to close that hole treated `destroy()`'s own
+ * `terminate()` as a crash and respawned the pool it was closing, and a
+ * worker thread refs the event loop, so the CLI hung after printing its
+ * summary;
+ * - the respawn cap that bounded THAT could empty the pool, and only the
+ * queue existing at that instant was resolved -- every later `parse()`
+ * waited forever.
+ *
+ * Each test below is one of those. They use real worker threads, because every
+ * one of these bugs is in the interaction with the thread lifecycle and a fake
+ * would have reproduced none of them.
+ */
+describe("ParsePool", () => {
+ let dir: string;
+
+ beforeEach(() => {
+ dir = mkdtempSync(join(tmpdir(), "ix-parse-pool-"));
+ });
+
+ afterEach(() => {
+ rmSync(dir, { recursive: true, force: true });
+ });
+
+ /** Write a worker and return its path. */
+ function worker(name: string, body: string): string {
+ const path = join(dir, `${name}.mjs`);
+ writeFileSync(path, body, "utf8");
+ return path;
+ }
+
+ const ECHO = `
+ import { parentPort } from 'node:worker_threads';
+ parentPort.on('message', ({ filePath }) => {
+ parentPort.postMessage({ ok: true, result: { filePath } });
+ });
+ `;
+
+ /** Exits on the first message, without ever emitting 'error'. */
+ const QUIET_DEATH = `
+ import { parentPort } from 'node:worker_threads';
+ parentPort.on('message', () => { process.exit(1); });
+ `;
+
+ /** Dies on construction, deterministically — the spawn-loop shape. */
+ const BORN_DEAD = `
+ process.exit(1);
+ `;
+
+ /** Answers its first message, then faults while IDLE. */
+ const IDLE_FAULT = `
+ import { parentPort } from 'node:worker_threads';
+ let served = 0;
+ parentPort.on('message', ({ filePath }) => {
+ parentPort.postMessage({ ok: true, result: { filePath } });
+ if (++served === 1) setTimeout(() => { throw new Error('idle fault'); }, 20);
+ });
+ `;
+
+ it("parses through the pool and shuts down without hanging", async () => {
+ const pool = new ParsePool(worker("echo", ECHO), 2);
+ pool.init();
+
+ const results = await Promise.all([pool.parse("a.ts", "x"), pool.parse("b.ts", "y")]);
+
+ expect(results).toEqual([{ filePath: "a.ts" }, { filePath: "b.ts" }]);
+ expect(pool.crashedTasks()).toBe(0);
+
+ // The hang: `destroy()` terminates every worker, each emits 'exit', and an
+ // exit handler that respawns hands back a pool of live threads that nothing
+ // owns. If that happens this test still passes -- the process is what hangs
+ // -- so `crashedTasks()` is asserted as the observable proxy: a deliberate
+ // teardown must not be recorded as a crash.
+ await pool.destroy();
+ expect(pool.crashedTasks(), "destroy() must not read as a crash").toBe(0);
+ });
+
+ it("resolves the in-flight task of a worker that dies without an error", async () => {
+ // `parseFile` catches its own exceptions, so a worker that goes quiet --
+ // `process.exit()`, or the thread killed under the runtime -- emits no
+ // 'error' at all. Before the 'exit' listener its task never settled and the
+ // `Promise.all` over the parse batch waited forever.
+ const pool = new ParsePool(worker("quiet", QUIET_DEATH), 1);
+ pool.init();
+
+ await expect(pool.parse("a.ts", "x")).resolves.toBeNull();
+ expect(pool.crashedTasks()).toBeGreaterThan(0);
+
+ await pool.destroy();
+ });
+
+ it("takes a worker that faults while IDLE out of the free list", async () => {
+ // The first of the three hangs. A worker can emit 'error' with no task in
+ // flight -- an uncaught async throw, ERR_WORKER_OUT_OF_MEMORY between
+ // tasks -- and it was spliced out of `workers` but left in `idle`, so the
+ // next `drain()` popped the terminated thread and posted to nothing: that
+ // task's promise never settled.
+ const pool = new ParsePool(worker("idlefault", IDLE_FAULT), 1);
+ pool.init();
+
+ expect(await pool.parse("first.ts", "x")).toEqual({ filePath: "first.ts" });
+ // Let the fault land while the pool is idle.
+ await new Promise(resolve => setTimeout(resolve, 120));
+
+ // TWO at once, deliberately. `drain()` pops the free list, so a single
+ // parse takes the replacement worker that `onError` just pushed and never
+ // touches the dead entry underneath it -- the bug hides completely at
+ // depth 1. The second parse is the one that reaches the terminated thread,
+ // posts to nothing, and never settles.
+ const both = await Promise.all([pool.parse("a2.ts", "x"), pool.parse("b2.ts", "x")]);
+ expect(both).toEqual([{ filePath: "a2.ts" }, { filePath: "b2.ts" }]);
+
+ await pool.destroy();
+ });
+
+ it("does not queue forever once the respawn cap has emptied the pool", async () => {
+ // A worker that dies deterministically on construction burns through the
+ // cap. Draining only the queue that existed at that instant left every
+ // LATER parse waiting on a `drain()` that is a no-op with no idle workers.
+ const pool = new ParsePool(worker("dead", BORN_DEAD), 1);
+ pool.init();
+
+ // Enough calls to outlast the cap, then more after it.
+ const first = await Promise.all(
+ Array.from({ length: 20 }, (_, i) => pool.parse(`f${i}.ts`, "x")),
+ );
+ expect(first.every(r => r === null)).toBe(true);
+
+ // The regression: these arrive after the pool is dead.
+ await expect(pool.parse("later.ts", "x")).resolves.toBeNull();
+ await expect(
+ Promise.all([pool.parse("l1.ts", "x"), pool.parse("l2.ts", "x")]),
+ ).resolves.toEqual([null, null]);
+
+ // And every one of them is counted, so the stitch gate sees the loss.
+ expect(pool.crashedTasks()).toBeGreaterThanOrEqual(23);
+
+ await pool.destroy();
+ });
+});
diff --git a/ix-cli/src/cli/__tests__/stitch-guard.test.ts b/ix-cli/src/cli/__tests__/stitch-guard.test.ts
new file mode 100644
index 00000000..56a6df3e
--- /dev/null
+++ b/ix-cli/src/cli/__tests__/stitch-guard.test.ts
@@ -0,0 +1,668 @@
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import { existsSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ admitStitch,
+ admitStitchWaiting,
+ clearStitchCooldown,
+ connectionNeverEstablished,
+ cooldownPathForTest,
+ outcomeProvesNothingRunning,
+ stitchKey,
+} from "../stitch-guard.js";
+import * as net from "node:net";
+import { createServer } from "node:http";
+import { IxClient } from "../../client/api.js";
+import { namedLockPath } from "../single-flight.js";
+
+const ENDPOINT = "http://localhost:8090";
+const OTHER = "http://localhost:9999";
+
+let lockDir: string;
+let savedLockDir: string | undefined;
+let savedCooldown: string | undefined;
+
+beforeEach(() => {
+ // IX_LOCK_DIR, not HOME: single-flight reads it from process.env on every
+ // call, so the redirect actually takes effect. (Overriding HOME would not --
+ // Windows os.homedir() reads USERPROFILE.)
+ lockDir = mkdtempSync(join(tmpdir(), "ix-stitch-guard-"));
+ savedLockDir = process.env.IX_LOCK_DIR;
+ savedCooldown = process.env.IX_STITCH_COOLDOWN_MS;
+ process.env.IX_LOCK_DIR = lockDir;
+ delete process.env.IX_STITCH_COOLDOWN_MS;
+});
+
+afterEach(() => {
+ const restore = (k: string, v: string | undefined): void => {
+ if (v === undefined) delete process.env[k];
+ else process.env[k] = v;
+ };
+ restore("IX_LOCK_DIR", savedLockDir);
+ restore("IX_STITCH_COOLDOWN_MS", savedCooldown);
+ rmSync(lockDir, { recursive: true, force: true });
+});
+
+/**
+ * Was the admission granted? Releases it either way.
+ *
+ * Every GRANTED admission holds a real lock handle, and single-flight arms
+ * `exit`, `SIGINT` and `SIGTERM` listeners on each one. An assertion that reads
+ * `.admitted` inline and drops the handle leaks all three, and enough of them
+ * made this file print MaxListenersExceededWarning three times -- noise that
+ * would hide a real listener leak the next time one appears.
+ *
+ * Settling as a clean success is the right release for these: every call below
+ * is a terminal assertion, so clearing the marker changes nothing later read.
+ */
+function admitted(endpoint: string, now?: number): boolean {
+ const a = now === undefined ? admitStitch(endpoint) : admitStitch(endpoint, now);
+ if (a.admitted) a.settle({ ok: true, elapsedMs: 1 });
+ return a.admitted;
+}
+
+/** Take an admission and settle it, asserting it was granted. */
+function stitchOnce(outcome: { ok: boolean; elapsedMs: number; status?: number | null }): void {
+ const a = admitStitch(ENDPOINT);
+ expect(a.admitted, "admitStitch refused when the test needed it granted").toBe(true);
+ if (a.admitted) a.settle(outcome);
+}
+
+describe("outcomeProvesNothingRunning", () => {
+ // The guard writes the cooldown when the stitch STARTS, so this function does
+ // not decide whether to cool down -- it decides whether we have PROOF that we
+ // need not. Only two facts qualify, and neither is an inference from the
+ // shape of a failure.
+
+ it("is true for a success", () => {
+ expect(outcomeProvesNothingRunning({ ok: true, elapsedMs: 90_000 })).toBe(true);
+ });
+
+ it("is true for 501, which is how this codebase spells 'no /v1/stitch'", () => {
+ // isStitchUnsupported treats 404 AND 501 as "this backend has no stitch",
+ // and the run swallows that silently -- so a marker left behind would refuse
+ // every map for 15 minutes claiming a join may still be running, against a
+ // backend that has never had one.
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 30, status: 501 })).toBe(true);
+ });
+
+ it("is true for a 4xx: the backend refused it rather than ran it", () => {
+ // Including a SLOW one. Elapsed covers the request upload, and the stitch
+ // payload is megabytes on a large monorepo, so a 413 can arrive tens of
+ // seconds in -- the old elapsed rule read that as "cut off mid-join".
+ for (const status of [400, 404, 413, 422]) {
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 25_000, status })).toBe(true);
+ }
+ });
+
+ it("is false for 408, which is a proxy saying IT gave up waiting", () => {
+ // Says nothing about whether the backend did, and that is the whole bug.
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 60_000, status: 408 })).toBe(false);
+ });
+
+ it("is false for a 5xx, a transport error, and an abort alike", () => {
+ // The reported case is a 500 at 60s. The others reach here with no status
+ // at all, and none of them proves the join stopped.
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 60_000, status: 500 })).toBe(false);
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 60_000, status: 504 })).toBe(false);
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 40, status: null })).toBe(false);
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 40 })).toBe(false);
+ });
+
+ it("does not consult the clock at all", () => {
+ // Four review rounds of an elapsed-based rule each broke the previous
+ // round's version of it. Same status, opposite ends of the clock, one answer.
+ for (const elapsedMs of [0, 40, 25_000, 600_000]) {
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs, status: 413 })).toBe(true);
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs, status: 500 })).toBe(false);
+ }
+ });
+});
+
+describe("the cooldown is written when the stitch starts", () => {
+ it("is on disk before the caller has sent anything", () => {
+ // This is the whole design. Every way an attempt can end without a definite
+ // refusal -- timeout, abort, deadline, SIGTERM, power cut -- leaves it in
+ // place, because leaving it is the default rather than something a dying
+ // process must still manage to do.
+ const a = admitStitch(ENDPOINT);
+ expect(a.admitted).toBe(true);
+ expect(existsSync(cooldownPathForTest(ENDPOINT)), "marker must exist before the request").toBe(true);
+ if (a.admitted) a.settle({ ok: true, elapsedMs: 30 });
+ });
+
+ it("survives a process that is killed mid-stitch and never settles", () => {
+ // A hook whose timeout is shorter than the stitch. single-flight's
+ // SIGINT/SIGTERM handlers release the lock and exit, so the NEXT map finds
+ // the lock free -- and before this design there was no cooldown either,
+ // because writing one was something the dying process still had to do. It
+ // was admitted, and pushed a second join onto the one still running.
+ const killed = admitStitch(ENDPOINT);
+ expect(killed.admitted).toBe(true);
+ // The signal handler's release(), and then nothing. No settle() call.
+ rmSync(namedLockPath("stitch", stitchKey(ENDPOINT)), { force: true });
+
+ const next = admitStitch(ENDPOINT);
+ expect(next.admitted).toBe(false);
+ if (!next.admitted) {
+ expect(next.rule).toBe("cooling");
+ expect(next.reason).toContain("never reported back");
+ }
+ });
+
+ it("is cleared by a success", () => {
+ stitchOnce({ ok: true, elapsedMs: 300 });
+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(false);
+ expect(admitted(ENDPOINT)).toBe(true);
+ });
+
+ it("is cleared by a 4xx, so an older backend with no /v1/stitch keeps working", () => {
+ stitchOnce({ ok: false, elapsedMs: 40, status: 404 });
+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(false);
+ expect(admitted(ENDPOINT)).toBe(true);
+ });
+
+ it("is kept by a 500, and says how long the attempt ran", () => {
+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });
+
+ const next = admitStitch(ENDPOINT);
+ expect(next.admitted).toBe(false);
+ if (!next.admitted) {
+ expect(next.reason).toContain("62s");
+ expect(next.reason).toContain("may still be running");
+ expect(next.reason).toContain("IX_STITCH_COOLDOWN_MS=0");
+ }
+ });
+
+ it("is kept when the run deadline aborts before the request even left", () => {
+ // Conservative by construction: no status, so no proof, so the marker
+ // stays. The old rules had to distinguish this from a mid-flight abort and
+ // got it wrong in both directions across rounds 2, 3 and 4.
+ stitchOnce({ ok: false, elapsedMs: 0, status: null });
+ expect(admitted(ENDPOINT)).toBe(false);
+ });
+});
+
+describe("cooldown expiry", () => {
+ it("admits again once the cooldown has expired", () => {
+ process.env.IX_STITCH_COOLDOWN_MS = "1000";
+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });
+
+ expect(admitted(ENDPOINT, Date.now() + 500)).toBe(false);
+ expect(admitted(ENDPOINT, Date.now() + 5_000)).toBe(true);
+ });
+
+ it("applies a changed IX_STITCH_COOLDOWN_MS to a cooldown already on disk", () => {
+ // The refusal message tells the user 0 disables this, so it has to be true
+ // of the cooldown they are looking at -- otherwise the only escape is
+ // deleting a state file whose name they cannot compute.
+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });
+ expect(admitted(ENDPOINT)).toBe(false);
+
+ process.env.IX_STITCH_COOLDOWN_MS = "0";
+ expect(admitted(ENDPOINT)).toBe(true);
+ });
+
+ it("a lower value shortens an active cooldown", () => {
+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });
+ process.env.IX_STITCH_COOLDOWN_MS = "1000";
+ expect(admitted(ENDPOINT, Date.now() + 5_000)).toBe(true);
+ });
+
+ it("scopes the cooldown to one backend", () => {
+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });
+ expect(admitted(OTHER)).toBe(true);
+ });
+
+ it("fails open on an unreadable cooldown record rather than refusing forever", () => {
+ writeFileSync(cooldownPathForTest(ENDPOINT), "{not json");
+ expect(admitted(ENDPOINT)).toBe(true);
+ });
+});
+
+describe("admitStitch single-flight", () => {
+ it("refuses a second stitch against the same backend while one is unsettled", () => {
+ const first = admitStitch(ENDPOINT);
+ expect(first.admitted).toBe(true);
+
+ const second = admitStitch(ENDPOINT);
+ expect(second.admitted).toBe(false);
+ if (!second.admitted) expect(second.rule).toBe("in-flight");
+
+ if (first.admitted) first.settle({ ok: true, elapsedMs: 10 });
+ expect(admitted(ENDPOINT)).toBe(true);
+ });
+
+ it("does not let one backend's in-flight stitch block another backend", () => {
+ const first = admitStitch(ENDPOINT);
+ expect(first.admitted).toBe(true);
+ const other = admitStitch(OTHER);
+ expect(other.admitted).toBe(true);
+
+ if (first.admitted) first.settle({ ok: true, elapsedMs: 1 });
+ if (other.admitted) other.settle({ ok: true, elapsedMs: 1 });
+ });
+
+ it("keys the lock on the endpoint text, not on the directory the command runs in", () => {
+ // The map lock canonicalises its key with realpath because it IS a path.
+ // An endpoint is not: realpath fails and the fallback resolve() would join
+ // it to process.cwd(), so `ix map` run from two directories would take two
+ // different "single-flight" locks and stitch twice at once.
+ const cwd = process.cwd();
+ const elsewhere = mkdtempSync(join(tmpdir(), "ix-stitch-cwd-"));
+ try {
+ const here = namedLockPath("stitch", ENDPOINT);
+ process.chdir(elsewhere);
+ expect(namedLockPath("stitch", ENDPOINT)).toBe(here);
+ } finally {
+ process.chdir(cwd);
+ rmSync(elsewhere, { recursive: true, force: true });
+ }
+ });
+
+ it("releases the lock even when the outcome is a failure", () => {
+ stitchOnce({ ok: false, elapsedMs: 5, status: 500 });
+ expect(readdirSync(lockDir).filter(f => f.endsWith(".lock"))).toEqual([]);
+ });
+});
+
+describe("stitchKey", () => {
+ it("folds the spellings of one local backend together", () => {
+ const canonical = stitchKey("http://localhost:8090");
+ expect(stitchKey("http://localhost:8090/")).toBe(canonical);
+ expect(stitchKey("http://127.0.0.1:8090")).toBe(canonical);
+ expect(stitchKey("HTTP://LOCALHOST:8090")).toBe(canonical);
+ });
+
+ it("treats a default port and an explicit one as one backend", () => {
+ // The WHATWG URL parser strips a default port, so both spellings give
+ // port === "" and this needs no defaults table of its own. Pinned because
+ // the absence of one reads like an omission -- and because if the parser
+ // ever stopped doing it, one backend would get two locks and two cooldowns,
+ // which is exactly the concurrency stitchKey exists to prevent.
+ expect(stitchKey("https://ix.example")).toBe(stitchKey("https://ix.example:443"));
+ expect(stitchKey("http://ix.example")).toBe(stitchKey("http://ix.example:80"));
+ expect(stitchKey("http://ix.example")).not.toBe(stitchKey("https://ix.example"));
+ });
+
+ it("keeps genuinely different backends apart", () => {
+ expect(stitchKey("http://localhost:8090")).not.toBe(stitchKey("http://localhost:9090"));
+ expect(stitchKey("http://localhost:8090")).not.toBe(stitchKey("http://ix.example:8090"));
+ expect(stitchKey("http://localhost:8090/a")).not.toBe(stitchKey("http://localhost:8090/b"));
+ });
+
+ it("passes through something that is not a URL rather than refusing to guard", () => {
+ expect(stitchKey("not a url")).toBe("not a url");
+ });
+
+ it("gives two spellings of one backend one lock", () => {
+ const first = admitStitch("http://localhost:8090");
+ expect(first.admitted).toBe(true);
+ expect(admitted("http://127.0.0.1:8090/")).toBe(false);
+
+ if (first.admitted) first.settle({ ok: true, elapsedMs: 1 });
+ });
+});
+
+describe("the cooldown is read while holding the lock, not before it", () => {
+ it("has no window in which a stale read can admit a second stitch", () => {
+ // A read-then-lock order leaves a gap: a waiter reads "no cooldown", the
+ // holder writes one and releases, the waiter takes the freed lock and
+ // sends the second concurrent stitch. Reading under the lock closes it by
+ // construction -- there is no interleaving to test, which is the point.
+ const holder = admitStitch(ENDPOINT);
+ expect(holder.admitted).toBe(true);
+ if (holder.admitted) holder.settle({ ok: false, elapsedMs: 62_000, status: 500 });
+
+ const waiter = admitStitch(ENDPOINT);
+ expect(waiter.admitted).toBe(false);
+ if (!waiter.admitted) expect(waiter.rule).toBe("cooling");
+ expect(readdirSync(lockDir).filter(f => f.endsWith(".lock")), "refusal must not leak a lock").toEqual([]);
+ });
+
+ it("tells a LIVE holder apart from a finished one, which is what the lock is for", () => {
+ // Both states have a marker on disk. Only the lock distinguishes them, and
+ // they want opposite answers: wait for the first, refuse the second.
+ const live = admitStitch(ENDPOINT);
+ expect(live.admitted).toBe(true);
+
+ const duringFlight = admitStitch(ENDPOINT);
+ expect(duringFlight.admitted).toBe(false);
+ if (!duringFlight.admitted) expect(duringFlight.rule).toBe("in-flight");
+
+ if (live.admitted) live.settle({ ok: false, elapsedMs: 62_000, status: 500 });
+
+ const afterFlight = admitStitch(ENDPOINT);
+ expect(afterFlight.admitted).toBe(false);
+ if (!afterFlight.admitted) expect(afterFlight.rule).toBe("cooling");
+ });
+});
+
+describe("connectionNeverEstablished", () => {
+ // Driven with errors Node actually produced, never hand-built ones: a
+ // classifier keyed on a shape nobody verified is the failure mode this
+ // codebase keeps hitting, and it looks identical to a working one.
+
+ async function failureOf(url: string): Promise<unknown> {
+ try {
+ await fetch(url, { method: "POST", body: "{}" });
+ return new Error("expected the request to fail");
+ } catch (err) {
+ return err;
+ }
+ }
+
+ it("is true for a port nothing is listening on — nothing was sent", async () => {
+ // Bound and closed, so the port is real and free. A hardcoded low port is
+ // not an option: fetch refuses the WHATWG "bad port" list outright, and
+ // that failure has no code at all — which this correctly reads as no proof.
+ const probe = net.createServer();
+ await new Promise<void>(resolve => probe.listen(0, "127.0.0.1", resolve));
+ const port = (probe.address() as net.AddressInfo).port;
+ await new Promise<void>(resolve => probe.close(() => resolve()));
+
+ expect(connectionNeverEstablished(await failureOf(`http://127.0.0.1:${port}/v1/stitch`))).toBe(true);
+ });
+
+ it("is true for a closed port on a MULTI-ADDRESS host, which localhost is", async () => {
+ // The default endpoint is `http://localhost:8090`, and localhost resolves to
+ // both ::1 and 127.0.0.1 -- so happy-eyeballs wraps the refusal in an
+ // AggregateError with the real errors under `.errors`, no `.cause`, and no
+ // `syscall` of its own. Every other test here uses a literal IP, which takes
+ // the plain-Error shape and cannot see the difference; a syscall-only rule
+ // walking only `.cause` answered false for the single commonest way this
+ // fires, and a backend that was merely down took the full 15-minute
+ // cooldown for a request that never left the machine.
+ // Bound and closed, like the sibling test: a hardcoded port turns this red
+ // with a message about the classifier if anything happens to be listening.
+ const probe = net.createServer();
+ await new Promise<void>(resolve => probe.listen(0, "127.0.0.1", resolve));
+ const port = (probe.address() as net.AddressInfo).port;
+ await new Promise<void>(resolve => probe.close(() => resolve()));
+
+ const err = await failureOf(`http://localhost:${port}/v1/stitch`);
+ // The predicate is the claim on every platform.
+ expect(connectionNeverEstablished(err)).toBe(true);
+ // The aggregate shape only appears where localhost is dual-stack, which is
+ // not everywhere; where it does, it is the trap -- no syscall of its own.
+ // The platform-independent proof that the descent works is the
+ // mixed-code test below.
+ const cause = (err as { cause?: { name?: string; syscall?: unknown } }).cause;
+ if (cause?.name === "AggregateError") expect(cause.syscall).toBeUndefined();
+ });
+
+ it("reads the sub-errors when the AggregateError carries no code of its own", () => {
+ // The live localhost case above happens to set `code` on the aggregate too,
+ // because both addresses failed the same way -- so the errno set alone
+ // catches it and the descent into `.errors` is not what saves it there.
+ // This is the case where the descent IS decisive: addresses that failed
+ // DIFFERENTLY, leaving nothing at the top to key on. Both sub-errors are
+ // still connect-phase, so nothing was sent either way.
+ //
+ // Built rather than driven: producing it needs a host resolving to one
+ // unroutable and one refused address, which I could not arrange. The
+ // sub-error shape is the one observed live against localhost.
+ const aggregate = Object.assign(new AggregateError([
+ Object.assign(new Error("connect ECONNREFUSED ::1:8090"), { code: "ECONNREFUSED", syscall: "connect" }),
+ Object.assign(new Error("connect ENETUNREACH 127.0.0.1:8090"), { code: "ENETUNREACH", syscall: "connect" }),
+ ], "")); // no `code`, no `syscall`
+ const err = Object.assign(new TypeError("fetch failed"), { cause: aggregate });
+
+ expect((aggregate as { code?: unknown }).code).toBeUndefined();
+ expect(connectionNeverEstablished(err)).toBe(true);
+ });
+
+ it("is true for an unroutable address — a neighbouring errno, same phase", () => {
+ // A VPN drop produces this, and an enumerated code list missed it: fifteen
+ // minutes of endpoint-wide cooldown for a request that never left.
+ //
+ // Pinned, not driven. Observed live against 240.0.0.1 (reserved, so the
+ // stack refuses to route) on Windows: ENETUNREACH with syscall "connect".
+ // It cannot be driven in CI -- Windows rejects immediately, while Linux and
+ // macOS silently hang until their own connect timeout, which failed this
+ // suite on four of its five platforms.
+ const err = Object.assign(new TypeError("fetch failed"), {
+ cause: Object.assign(new Error("connect ENETUNREACH 240.0.0.1:8090"), {
+ code: "ENETUNREACH",
+ syscall: "connect",
+ }),
+ });
+ expect(connectionNeverEstablished(err)).toBe(true);
+ });
+
+ it("is true for undici's connect timeout, which carries no syscall", () => {
+ // Shape observed through IxClient.stitch against a blackholed address
+ // (192.0.2.1, TEST-NET-1): ConnectTimeoutError / UND_ERR_CONNECT_TIMEOUT,
+ // syscall undefined. Pinned from the observation rather than driven live,
+ // because reproducing it costs undici's full 10s connect timeout.
+ const observed = new TypeError("fetch failed");
+ (observed as { cause?: unknown }).cause = Object.assign(
+ new Error("Connect Timeout Error (attempted address: 192.0.2.1:8090)"),
+ { name: "ConnectTimeoutError", code: "UND_ERR_CONNECT_TIMEOUT" },
+ );
+ expect(connectionNeverEstablished(observed)).toBe(true);
+ });
+
+ it("is false for a blocked port, whose failure carries no code", async () => {
+ // fetch rejects port 1 before it opens anything, with a bare
+ // `Error: bad port`. No code means no proof, which is the safe answer.
+ expect(connectionNeverEstablished(await failureOf("http://127.0.0.1:1/v1/stitch"))).toBe(false);
+ });
+
+ it("is true for a host that does not resolve", async () => {
+ expect(
+ connectionNeverEstablished(await failureOf("http://ix-568-no-such-host.invalid/v1/stitch")),
+ ).toBe(true);
+ });
+
+ it("is FALSE for a socket dropped after the request went out", async () => {
+ // The ambiguous case, and the reason this is narrower than "a transport
+ // error". Once the bytes are gone, a reset means either an upstream that
+ // restarted (its join died with it) or a proxy that hung up (the join is
+ // still running). Both spell UND_ERR_SOCKET, so the marker stays.
+ const server = net.createServer(sock => {
+ sock.once("data", () => setTimeout(() => sock.destroy(), 10));
+ });
+ await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
+ const port = (server.address() as net.AddressInfo).port;
+ try {
+ const err = await failureOf(`http://127.0.0.1:${port}/v1/stitch`);
+ expect(String(err)).toContain("fetch failed");
+ expect(connectionNeverEstablished(err)).toBe(false);
+ } finally {
+ await new Promise<void>(resolve => server.close(() => resolve()));
+ }
+ });
+
+ it("clears the marker, so a backend that went down mid-run is not blamed for 15 minutes", () => {
+ const a = admitStitch(ENDPOINT);
+ expect(a.admitted).toBe(true);
+ if (a.admitted) a.settle({ ok: false, elapsedMs: 3, status: null, neverConnected: true });
+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(false);
+ expect(admitted(ENDPOINT)).toBe(true);
+ });
+});
+
+describe("a 2xx that still failed", () => {
+ it("KEEPS the marker, because that is what a proxy error page looks like", async () => {
+ // Two earlier revisions cleared it here, reasoning that the request
+ // completed so the join must have. A proxy answering 200 with its own HTML
+ // error page when the upstream stalls produces exactly this shape -- the
+ // #528 pattern -- and clearing the marker then admits the next map onto a
+ // join that is still running, which is the #568 failure itself. This is the
+ // one arm that could cause it, so it is the one arm that does not fire.
+ const server = createServer((_req, res) => {
+ res.writeHead(200, { "Content-Type": "text/html" });
+ res.end("<html>proxy says hi</html>");
+ });
+ await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
+ const port = (server.address() as net.AddressInfo).port;
+ try {
+ const client = new IxClient(`http://127.0.0.1:${port}`, undefined);
+ let caught: unknown;
+ try {
+ await client.stitch({ workspaceId: "w", provides: [], consumes: [], exports: [], symbolConsumes: [] } as never);
+ } catch (err) {
+ caught = err;
+ }
+ // The status is in the message, which is what makes this decidable at all.
+ expect(String(caught)).toContain("200: response body is not JSON");
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 5, status: 200 })).toBe(false);
+ } finally {
+ await new Promise<void>(resolve => server.close(() => resolve()));
+ }
+ });
+
+ it("cools the endpoint down, so the next map does not stack a join", () => {
+ const a = admitStitch(ENDPOINT);
+ expect(a.admitted).toBe(true);
+ if (a.admitted) a.settle({ ok: false, elapsedMs: 40, status: 200 });
+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(true);
+ expect(admitStitch(ENDPOINT).admitted).toBe(false);
+ });
+
+ it("still keeps the marker for the failures that mean a join may be running", () => {
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 5, status: 500 })).toBe(false);
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 5, status: 408 })).toBe(false);
+ expect(outcomeProvesNothingRunning({ ok: false, elapsedMs: 5, status: null })).toBe(false);
+ });
+
+ it("a stitch that actually returned still clears it", () => {
+ expect(outcomeProvesNothingRunning({ ok: true, elapsedMs: 5 })).toBe(true);
+ });
+});
+
+describe("the cooldown runs from the end of the attempt, not its start", () => {
+ it("does not expire mid-attempt when IX_STITCH_COOLDOWN_MS is shorter than the request", () => {
+ // IxClient.post caps a request at two minutes, so before the restamp EVERY
+ // setting under that bought nothing on the timeout path: the marker was
+ // stamped at the start, the attempt outlived it, and the next map was
+ // admitted immediately and stacked a second join.
+ process.env.IX_STITCH_COOLDOWN_MS = "1000";
+ const startedLongAgo = Date.now() - 60_000;
+
+ const a = admitStitch(ENDPOINT, startedLongAgo);
+ expect(a.admitted).toBe(true);
+ if (a.admitted) a.settle({ ok: false, elapsedMs: 60_000, status: 500 });
+
+ const next = admitStitch(ENDPOINT);
+ expect(next.admitted, "a 1s cooldown must still be live the instant a 60s attempt ends").toBe(false);
+ if (!next.admitted) expect(next.reason).toContain("60s");
+ });
+
+ it("still expires on schedule once the attempt has ended", () => {
+ process.env.IX_STITCH_COOLDOWN_MS = "1000";
+ const a = admitStitch(ENDPOINT, Date.now() - 60_000);
+ if (a.admitted) a.settle({ ok: false, elapsedMs: 60_000, status: 500 });
+
+ expect(admitted(ENDPOINT, Date.now() + 5_000)).toBe(true);
+ });
+});
+
+describe("clearStitchCooldown, which is what `ix reset` calls", () => {
+ it("releases an active cooldown so the re-ingest after a reset is admitted", () => {
+ // A reset wipes the registration on the backend, and the full re-ingest
+ // that follows is the ONE run that can put it back -- an incremental map
+ // never reaches the stitch block. A live cooldown refusing exactly that run
+ // leaves the workspace unregistered with nothing to retry, while this
+ // file's own comments promise a post-reset re-map re-registers.
+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });
+ expect(admitStitch(ENDPOINT).admitted).toBe(false);
+
+ clearStitchCooldown(ENDPOINT);
+
+ expect(existsSync(cooldownPathForTest(ENDPOINT))).toBe(false);
+ expect(admitted(ENDPOINT)).toBe(true);
+ });
+
+ it("leaves another backend's cooldown alone", () => {
+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });
+ clearStitchCooldown(OTHER);
+ expect(admitStitch(ENDPOINT).admitted).toBe(false);
+ });
+
+ it("is a no-op when there is no cooldown, rather than throwing", () => {
+ expect(() => clearStitchCooldown(ENDPOINT)).not.toThrow();
+ expect(admitted(ENDPOINT)).toBe(true);
+ });
+});
+
+describe("admitStitchWaiting", () => {
+ it("waits out a stitch that is merely in flight rather than shedding it", async () => {
+ // On a healthy backend a stitch is over in tens of milliseconds, and two
+ // maps overlapping by that much is ordinary. Shedding there loses that
+ // workspace's registration until somebody re-ingests every file.
+ const holder = admitStitch(ENDPOINT);
+ expect(holder.admitted).toBe(true);
+
+ let slept = 0;
+ const sleep = async (ms: number): Promise<void> => {
+ slept += ms;
+ if (slept >= 500 && holder.admitted) holder.settle({ ok: true, elapsedMs: 30 });
+ };
+
+ const second = await admitStitchWaiting(ENDPOINT, 30_000, sleep);
+ expect(second.admitted, "should have waited for the holder to finish").toBe(true);
+ if (second.admitted) second.settle({ ok: true, elapsedMs: 30 });
+ });
+
+ it("gives up once the budget is spent, so an unhealthy backend still sheds", async () => {
+ const holder = admitStitch(ENDPOINT);
+ expect(holder.admitted).toBe(true);
+
+ const second = await admitStitchWaiting(ENDPOINT, 1_000, async () => {});
+ expect(second.admitted).toBe(false);
+ if (!second.admitted) expect(second.reason).toContain("already stitching");
+ });
+
+ it("never waits on a cooldown — outlasting the last stitch is the point", async () => {
+ stitchOnce({ ok: false, elapsedMs: 62_000, status: 500 });
+
+ let slept = 0;
+ const result = await admitStitchWaiting(ENDPOINT, 30_000, async (ms) => { slept += ms; });
+
+ expect(result.admitted).toBe(false);
+ expect(slept, "waited on a cooldown").toBe(0);
+ });
+
+ it("refuses outright when the run deadline has ALREADY fired, without marking", async () => {
+ // Admission writes the cooldown marker and fetch then rejects instantly on
+ // the aborted signal -- so admitting here blocks the backend for 15 minutes
+ // over a request that never left the process. And it repeats: the stitch is
+ // the last thing an ingest does, so a run that routinely overruns would
+ // block its own next attempt every time and never stitch again.
+ const result = await admitStitchWaiting(ENDPOINT, 30_000, async () => {}, { aborted: true });
+
+ expect(result.admitted).toBe(false);
+ if (!result.admitted) expect(result.rule).toBe("deadline");
+ expect(existsSync(cooldownPathForTest(ENDPOINT)), "must not mark the backend").toBe(false);
+ expect(readdirSync(lockDir).filter(f => f.endsWith(".lock"))).toEqual([]);
+ });
+
+ it("stops waiting when the run deadline fires WHILE it is waiting", async () => {
+ // The signal flips mid-wait, which is the case the in-loop check is for --
+ // and the reason it has to be read through a call rather than a property
+ // access TypeScript will narrow to a constant.
+ const holder = admitStitch(ENDPOINT);
+ expect(holder.admitted).toBe(true);
+ const budget = { aborted: false };
+
+ let slept = 0;
+ const result = await admitStitchWaiting(ENDPOINT, 30_000, async (ms) => {
+ slept += ms;
+ budget.aborted = true; // the run runs out mid-wait
+ if (holder.admitted) holder.settle({ ok: true, elapsedMs: 5 }); // ...and the holder frees the lock
+ }, budget);
+
+ // Admitting here would mark the backend for 15 minutes over a request that
+ // cannot leave: fetch rejects instantly on the aborted deadline, with no
+ // status, so the marker stays. The refusal must also name the real cause --
+ // reporting "in-flight" sends the user to the wrong remedy entirely.
+ expect(result.admitted).toBe(false);
+ if (!result.admitted) expect(result.rule).toBe("deadline");
+ expect(existsSync(cooldownPathForTest(ENDPOINT)), "must not mark the backend").toBe(false);
+ expect(slept, "should have stopped after the first poll").toBeLessThan(1_000);
+ });
+});
diff --git a/ix-cli/src/cli/__tests__/stitch-scope-cache.test.ts b/ix-cli/src/cli/__tests__/stitch-scope-cache.test.ts
index 03a7dff8..f13be61c 100644
--- a/ix-cli/src/cli/__tests__/stitch-scope-cache.test.ts
+++ b/ix-cli/src/cli/__tests__/stitch-scope-cache.test.ts
@@ -5,18 +5,33 @@ import { join } from "node:path";
let home: string;
let savedHome: string | undefined;
+let savedUserProfile: string | undefined;
beforeEach(() => {
vi.resetModules();
home = mkdtempSync(join(tmpdir(), "ix-stitch-scope-"));
savedHome = process.env.HOME;
+ // USERPROFILE as well as HOME. `config.ts` reaches the cache through
+ // `os.homedir()`, which on Windows reads USERPROFILE and ignores HOME -- so
+ // this redirect was inert there, and these tests wrote into the developer's
+ // REAL ~/.ix. That made the file self-poisoning rather than merely untidy:
+ // the "asks the backend on a miss" case recorded ws-fresh in the real home
+ // on its first run and then found it cached on every run after, so the file
+ // passed once per machine and failed forever. CI never saw it because each
+ // job starts on a clean home.
+ savedUserProfile = process.env.USERPROFILE;
process.env.HOME = home;
+ process.env.USERPROFILE = home;
mkdirSync(join(home, ".ix"), { recursive: true });
});
afterEach(() => {
- if (savedHome === undefined) delete process.env.HOME;
- else process.env.HOME = savedHome;
+ const restore = (k: string, v: string | undefined): void => {
+ if (v === undefined) delete process.env[k];
+ else process.env[k] = v;
+ };
+ restore("HOME", savedHome);
+ restore("USERPROFILE", savedUserProfile);
rmSync(home, { recursive: true, force: true });
vi.restoreAllMocks();
});
diff --git a/ix-cli/src/cli/commands/ingest.ts b/ix-cli/src/cli/commands/ingest.ts
index 910eecd3..cc7c3e7b 100644
--- a/ix-cli/src/cli/commands/ingest.ts
+++ b/ix-cli/src/cli/commands/ingest.ts
@@ -18,6 +18,7 @@ import { loadIngestionModules } from './ingestion-loader.js';
import { ensureWorkspaceIdState } from '../bootstrap.js';
import { detectSystem, repoWorkspaceIdFor, lookupPackage, readPackageNames, readPackageDeps } from '../system.js';
import { CLIENT_EXPECTED_SCHEMA_VERSION } from '../backend-status.js';
+import { admitStitchWaiting, connectionNeverEstablished, type StitchRefusal } from '../stitch-guard.js';
import {
createCommitBreaker,
createDrainGate,
@@ -400,7 +401,7 @@ async function loadStoredPatchEntities(
if (entities) return entities;
throw new Error(`Patch ${patchId} has no entity manifest`);
} catch (err) {
- if (!String(err).includes('404:')) throw err;
+ if (!isNotFoundError(err)) throw err;
}
}
throw new Error('Previous source patch was not found');
@@ -469,7 +470,7 @@ export async function reconcileRemovedEntities(
}
}
} catch (err) {
- if (!String(err).includes('404:')) throw err;
+ if (!isNotFoundError(err)) throw err;
// Already absent. Emitting the delete anyway keeps the patch a complete
// statement of intent and costs nothing.
removedNodeIds.push(nodeId);
@@ -918,6 +919,25 @@ export interface IngestFilesSummary {
parseErrors: number;
commitErrors: number;
stitchErrors: number;
+ /**
+ * Why the cross-workspace stitch was not attempted, if it was not (Ix#568).
+ *
+ * A skip is not an error and deliberately does not move the exit code -- but
+ * CLAUDE.md RULE 5 has hooks running `ix map --silent` and reading the exit
+ * code, so without a field here an automated consumer sees a wholly
+ * successful map for up to 15 minutes while cross-repo edges are stale, and
+ * the only signal is an English sentence on stderr.
+ */
+ stitchSkipped?: string;
+ /**
+ * Which rule kept the stitch from happening, for a consumer that must branch.
+ *
+ * The prose in `stitchSkipped` is for people. `incomplete` -- an incremental
+ * map with nothing complete to send -- fires on almost every run, so without
+ * this a hook watching for the #568 cooldown could only tell the two apart by
+ * substring-matching English, which `StitchRefusal`'s own doc forbids.
+ */
+ stitchSkippedRule?: StitchRefusal;
}
/**
@@ -943,6 +963,22 @@ export function isStitchUnsupported(error: unknown): boolean {
return status === 404 || status === 501;
}
+/**
+ * Is this "the backend does not have it", as opposed to any other failure?
+ *
+ * Anchored, the way `stitchFailureStatus` is. A bare
+ * `String(err).includes('404:')` was safe only while an unparseable body
+ * surfaced as V8's `SyntaxError`, which quotes about ten characters of input.
+ * `IxClient` now reports it as `"${status}: response body is not JSON: ..."`
+ * with a 200-character preview -- so a gateway answering 200 with an HTML error
+ * page whose title is `404: Not Found` matched, the error was swallowed as
+ * "absent", and `reconcileRemovedEntities` emitted a DeleteNode for a node that
+ * exists. That is the #528 proxy shape this file already cites twice.
+ */
+function isNotFoundError(error: unknown): boolean {
+ return /^(?:Error:\s*)?404:/.test(String(error));
+}
+
/**
* Ix#528: describe the failure without echoing the response body. The observed
* case is a proxy 504 whose body is a full HTML error page, and pasting that
@@ -950,10 +986,111 @@ export function isStitchUnsupported(error: unknown): boolean {
*/
export function describeStitchFailure(error: unknown): string {
const status = stitchFailureStatus(error);
- const detail = status === null ? "backend request failed" : `HTTP ${status}`;
+ // A 2xx here means the request succeeded and something downstream made the
+ // body unreadable -- the #528 proxy shape -- so "HTTP 200" as the failure
+ // status is accurate and useless. Say what actually happened.
+ const detail =
+ status === null
+ ? "backend request failed"
+ : status >= 200 && status < 300
+ ? `HTTP ${status} with an unreadable body`
+ : `HTTP ${status}`;
return `Warning: Cross-workspace stitch failed (${detail}). Source patches were committed, but cross-repository relationships may be incomplete.`;
}
+/**
+ * Ix#568: the stitch was never sent, and why.
+ *
+ * Deliberately not phrased as a failure. Nothing broke and the previous
+ * registration still stands -- the same position a stitch that FAILED already
+ * left the graph in. Saying only that cross-repo relationships may be
+ * incomplete, with no reason, is what would make this look like a regression.
+ *
+ * It does not promise that the next map re-registers, because that is not
+ * reliably true and predates this guard: the stitch block is gated on
+ * `filesSkippedAsUnchanged === 0`, so an incremental map that skips any
+ * mtime- or hash-unchanged file never reaches it -- and by then it has no
+ * registration data to send
+ * either, since it only parsed what changed. Its own comment already names
+ * the ways back in: a fresh map, `--force`, or a post-reset re-map. What the
+ * cooldown changes is only how often this path is taken, not where it leads.
+ */
+export function describeStitchSkipped(
+ reason: string,
+ rule?: StitchRefusal,
+ /** Whether the caller is showing per-file detail (`--verbose` / `--debug`). */
+ verbose = false,
+): string {
+ // Short by default. A cooldown lasts 15 minutes, and under CLAUDE.md RULE 5
+ // an auto-map hook fires after every edit -- so the long form printed three
+ // sentences of unchanging advice on every map in that window. The reason is
+ // the part that changes and the part worth reading; the rest is on --verbose,
+ // and `stitchSkipped` / `stitch_skipped` carry it to machine consumers either
+ // way.
+ // The short form has to carry a real remedy, not just the reason. Waiting the
+ // cooldown out and re-running does NOT recover: `persistIngestBaselineIfClean`
+ // has already written the mtime baseline, so the next map has
+ // `filesSkipped > 0` and never enters the stitch block at all -- it prints
+ // nothing and exits 0, and the user reads that as fixed while cross-repo edges
+ // stay frozen. Only a run that re-ingests every file gets back in.
+ // Every branch names `--force`, because nothing else re-registers this
+ // workspace. A skipped stitch does NOT stop the run persisting its mtime
+ // baseline -- `persistIngestBaselineIfClean` gates on parse and commit errors
+ // only -- so an ordinary re-run is incremental, skips unchanged files, and
+ // never enters the stitch block. "Re-run once that finishes" and "raise
+ // IX_MAP_DEADLINE_MS" were both promises this code cannot keep: the re-run
+ // prints nothing, exits 0, and the cross-repo edges stay exactly as stale.
+ //
+ // The hedge that used to justify withholding `--force` from the in-flight
+ // branch is still right, and now lives in the wording rather than in the
+ // omission: the other run may well be registering this same workspace, in
+ // which case the force is unnecessary -- but it is the only thing that
+ // recovers the case where it was not.
+ const recover =
+ rule === "deadline"
+ ? " Raise IX_MAP_DEADLINE_MS or map a smaller path, then re-register with `ix ingest <root> --force`."
+ : rule === "in-flight"
+ ? " Re-run once that finishes; if the cross-repo edges still look stale, `ix ingest <root> --force`."
+ : rule === "lost-parses"
+ // Nothing "clears" here, and this rule fires ON a --force run, so the
+ // cooling remedy told the user to wait for a condition that does not
+ // exist and then re-run the command they had just run.
+ ? " Re-run `ix ingest <root> --force`; if it keeps happening, the crash is reproducible and IX_PARSE_DEBUG=1 will show it."
+ : rule === "run-errors"
+ ? " Fix the errors above and re-run `ix ingest <root> --force`."
+ : " Recover with `ix ingest <root> --force` once it clears.";
+ const head = `Note: Cross-workspace stitch not started — ${reason}.${recover}`;
+ if (!verbose) return head;
+
+ // The remedy follows the RULE, never the prose. Note the other run may be
+ // registering this SAME workspace: only `ix map` takes the per-workspace
+ // lock, so two `ix ingest` runs on one repo, or an `ix ingest` racing an
+ // `ix map`, both reach here. Hence "may be" rather than a confident
+ // instruction to force a full re-ingest nobody needs.
+ const remedy =
+ rule === "in-flight"
+ ? "That run may be registering a different workspace, in which case this one was not " +
+ "registered; re-run once it has finished, or force a full re-ingest " +
+ "(`ix ingest <root> --force`) if the cross-repo edges still look stale."
+ : rule === "lost-parses"
+ ? "A parse worker crashed, so this run never saw some files and its registration would be " +
+ "missing them. Nothing is running on the backend and nothing needs to clear: re-run " +
+ "`ix ingest <root> --force`, and if it recurs the crash is reproducible -- IX_PARSE_DEBUG=1 " +
+ "will show what the parser choked on."
+ : rule === "run-errors"
+ ? "The parse or commit errors above mean this run has an incomplete picture of the repo, so " +
+ "registering from it would overwrite a good registration with a worse one. Fix those and " +
+ "re-run `ix ingest <root> --force`."
+ : rule === "deadline"
+ ? "Nothing was sent, so nothing is running -- but nothing was registered either, and a longer " +
+ "budget does not fix that on its own: the next map is incremental and never reaches the stitch " +
+ "block. Raise IX_MAP_DEADLINE_MS or map a smaller path, then `ix ingest <root> --force`."
+ : "Once the cooldown expires and the backend is healthy, re-register with a run that re-ingests " +
+ "every file (`ix ingest <root> --force`) — an incremental map that skips unchanged files does not " +
+ "re-attempt the stitch.";
+ return `${head} Source patches were committed; cross-repository edges are unchanged since the last successful stitch. ${remedy}`;
+}
+
/**
* Decide how loudly a run should complain about patches that failed to commit.
*
@@ -1348,6 +1485,12 @@ export async function ingestFiles(
'../../../../core-ingestion/dist/parse-worker.js',
);
let pool: ParsePool | null = null;
+ /**
+ * In-flight parses lost to a crashed worker, read where the pool is still in
+ * scope. A function rather than a read at the use site because control-flow
+ * narrowing makes `pool` `never` down there.
+ */
+ const crashedParses = (): number => (pool === null ? 0 : pool.crashedTasks());
const ensureParsePool = (): ParsePool => {
if (pool) return pool;
pool = new ParsePool(workerPath, Math.max(1, os.cpus().length - 1));
@@ -1383,7 +1526,12 @@ export async function ingestFiles(
for (let i = 0; i < targets.length; i += PRESCAN_PARSE_CHUNK) {
const chunk = targets.slice(i, i + PRESCAN_PARSE_CHUNK);
const parsed = await Promise.all(
- chunk.map(fp => ensureParsePool().parse(fp, relSources.get(fp)!).catch(() => null)),
+ // `false`: this prescan is best-effort by construction -- the caller
+ // drops nulls, and every file here is read and parsed again by the
+ // streaming loop below. A loss here is not a file the RUN lost, and
+ // counting it as one refused the stitch and withheld the baseline over
+ // files that were all present.
+ chunk.map(fp => ensureParsePool().parse(fp, relSources.get(fp)!, false).catch(() => null)),
);
for (let j = 0; j < chunk.length; j++) {
if (parsed[j]) preParsed.set(chunk[j], parsed[j]);
@@ -1421,10 +1569,79 @@ export async function ingestFiles(
let filesChanged = 0;
let patchesApplied = 0;
let filesSkipped = 0;
+ /**
+ * Files skipped because we ASSUMED THEY WERE UNCHANGED. (Ix#568)
+ *
+ * A subset of `filesSkipped`, and the one the cross-workspace stitch cares
+ * about. The stitch registers this repo's provides/consumes, collected over
+ * the files actually parsed this run, so it must only run when that set
+ * covers the whole repo -- otherwise it overwrites a full registration with
+ * a partial one.
+ *
+ * `filesSkipped` is the wrong test for that, because it also counts files
+ * that contribute nothing under ANY run: a zero-byte file and one that
+ * looks minified. Both are decided by the file's own content under a rule
+ * that does not vary between runs, so the collected set is complete and
+ * correct without them -- and their presence in the count was silently
+ * disqualifying repos from ever stitching. A single empty `__init__.py` was
+ * enough, `--force` or not, which made `ix ingest <root> --force` -- the
+ * remedy this command prints when it refuses a stitch -- a promise it could
+ * not keep: the run would print nothing, exit 0, and leave the cross-repo
+ * edges stale.
+ *
+ * An mtime- or hash-clean skip is different in kind: the file has content
+ * we would have indexed and we did not look at it this run, so the set
+ * really is missing it. So is a file the parser returned nothing for --
+ * see `filesSkippedUnparsed`, which the gate reads alongside this.
+ */
+ let filesSkippedAsUnchanged = 0;
+ /** Zero-byte files. Reported as `skipReasons.emptyFile`, which was hardcoded 0. */
+ let filesSkippedAsEmpty = 0;
+ /**
+ * Files the parse pool returned nothing for.
+ *
+ * Reported, but NOT part of the stitch gate -- `ParsePool.crashedTasks()` is.
+ * `parse()` resolves `null` for two different things and only one of them
+ * makes the run incomplete:
+ *
+ * - `parseFile` returned null, which is deterministic and means the file
+ * has nothing to give. The dominant cause is an unavailable optional
+ * grammar -- fourteen are optionalDependencies loaded through a try/catch,
+ * and `tree-sitter-sas` has no win32 prebuild at all -- and those
+ * extensions are still in SUPPORTED_EXTENSIONS, so the files are
+ * discovered, read, dispatched and come back null on EVERY run. Gating on
+ * this count blocked stitching permanently for any repo containing one,
+ * `--force` included, since forcing does not change a parse result. That
+ * is the empty-`__init__.py` bug again with a different file class.
+ * - a worker CRASHED with the task in flight, which is transient and did
+ * lose a file we would have indexed. That one belongs in the gate.
+ *
+ * A parse that THREW is in the first category, not the second, even though it
+ * sounds like the second: `parseFile` catches every exception itself and
+ * returns null (logging only under IX_PARSE_DEBUG=1), so the worker cannot
+ * tell it from a missing grammar -- and neither can this. It is also
+ * deterministic for the same bytes in every case we can observe, so gating on
+ * it would block stitching permanently for that repo, which is exactly the
+ * bug this counter was taken OUT of the gate to fix. The residue is the
+ * transient sub-case, an OOM caught by that same handler; it is narrower than
+ * the bug the alternative creates, and it is a deliberate choice rather than
+ * an oversight.
+ *
+ * Reported as `skipReasons.unparsed`, its own bucket: `parseError` counts
+ * every parse-stage failure including stat, read and patch-build errors, most
+ * of which are not skips, so adding to it made that number mean neither thing.
+ */
+ let filesSkippedUnparsed = 0;
let parseErrors = 0;
let commitErrors = 0;
let stitchErrors = 0;
let stitchError: unknown;
+ // Set when the stitch was refused before it was ever sent (Ix#568).
+ // Distinct from stitchError: nothing failed, so it must not count as a
+ // stitch error or move the exit code -- but the user is about to be told
+ // cross-repo edges may be incomplete and is owed the reason.
+ let stitchSkipped: string | undefined;
+ let stitchSkippedRule: StitchRefusal | undefined;
// Ix#560: a backend refusing every commit gets one failed request per
// patch out of the per-file fallback. Stop after a streak instead.
const commitBreaker = createCommitBreaker();
@@ -1582,15 +1799,29 @@ export async function ingestFiles(
// Stat all files and partition into mtime-clean (skip) and mtime-changed (need hash check).
const mtimeChangedPaths: string[] = [];
+ /**
+ * Paths this loop has already accounted for, and under which counter.
+ *
+ * Recorded explicitly rather than inferred from `currentMtimes`. The loop
+ * leaves a path out of that map in three cases -- empty, oversized, and
+ * `statSync` THREW -- and only the first two are counted here. So a file
+ * that errored on the first stat and then read as size 0 on the second (a
+ * transient lock, an antivirus scan, a file recreated underneath us) was
+ * treated as already counted and landed in no bucket at all: absent from
+ * the graph, from `filesSkipped` and from every `skipReasons` entry, which
+ * is precisely what the check that reads this exists to prevent.
+ */
+ const accountedByStatLoop = new Map<string, 'empty' | 'tooLarge'>();
for (const filePath of filePaths) {
try {
const st = fs.statSync(filePath);
- if (st.size === 0) { filesSkipped++; progressCurrent++; continue; }
- if (st.size > MAX_FILE_BYTES) { tooLarge++; progressCurrent++; continue; }
+ if (st.size === 0) { filesSkipped++; filesSkippedAsEmpty++; accountedByStatLoop.set(filePath, 'empty'); progressCurrent++; continue; }
+ if (st.size > MAX_FILE_BYTES) { tooLarge++; accountedByStatLoop.set(filePath, 'tooLarge'); progressCurrent++; continue; }
const mtime = st.mtimeMs;
currentMtimes.set(filePath, mtime);
if (!opts.force && !forceReingestPaths.has(filePath) && mtimeCache.get(filePath) === mtime) {
filesSkipped++;
+ filesSkippedAsUnchanged++;
progressCurrent++; // mtime clean — assume unchanged
} else {
mtimeChangedPaths.push(filePath);
@@ -2489,6 +2720,7 @@ export async function ingestFiles(
const hash = sha256(bytes);
if (!forceReingestPaths.has(filePath) && knownHashes.get(filePath) === hash) {
filesSkipped++;
+ filesSkippedAsUnchanged++;
progressCurrent++;
continue;
}
@@ -2579,7 +2811,7 @@ export async function ingestFiles(
const batch: ParsedFile[] = [];
for (let j = 0; j < chunk.length; j++) {
const parsed = parseResults[j] as any;
- if (!parsed) { filesSkipped++; continue; }
+ if (!parsed) { filesSkipped++; filesSkippedUnparsed++; continue; }
entitiesParsed += parsed.entities.length;
batch.push({ filePath: chunk[j].filePath, parsed, hash: chunk[j].hash, previousHash: chunk[j].previousHash });
}
@@ -2590,6 +2822,19 @@ export async function ingestFiles(
}
} else {
// Path B: no baseline (first ingest) or --force → load modules, then stream parse + commit.
+ //
+ // Ix#568: this branch walks ALL of `filePaths`, re-reading, re-hashing and
+ // re-parsing every one, and its hash-clean short-circuit cannot fire here
+ // -- `knownHashes` is empty by construction on the non-force entry, and
+ // the check is `!opts.force` on the other. So the mtime skips the stat
+ // loop recorded did not actually happen, and leaving them counted refused
+ // the stitch on the commonest incremental shape there is: add ONE new
+ // file to a mapped repo, and `loadExistingHashes` returns nothing for it,
+ // so `knownHashes.size === 0` sends the whole repo down here -- every file
+ // parsed, registration complete -- while the gate still saw N-1 files
+ // "skipped as unchanged" and silently declined to stitch.
+ filesSkipped -= filesSkippedAsUnchanged;
+ filesSkippedAsUnchanged = 0;
const moduleStart = performance.now();
const [ingestion, patchBuilder] = await loadIngestionModules();
timings.moduleLoadMs = Math.round(performance.now() - moduleStart);
@@ -2648,8 +2893,57 @@ export async function ingestFiles(
let bytes!: Buffer;
try {
const st = await fh.stat();
- if (st.size === 0) { filesSkipped++; return; }
- if (st.size > MAX_FILE_BYTES) { tooLarge++; return; }
+ // The stat loop above walks the same `filePaths` unconditionally,
+ // so a file it already rejected is counted there and must not be
+ // counted again here -- incrementing unconditionally counted every
+ // empty or oversized file TWICE, invisible until
+ // `skipReasons.emptyFile` stopped being hardcoded to 0 and
+ // reported 2 for a repo with one empty `__init__.py`.
+ //
+ // `accountedByStatLoop` is exactly that record. It is a set the
+ // loop writes deliberately, not `currentMtimes` read backwards:
+ // that map also omits files whose `statSync` THREW, which the loop
+ // counts as a parse error and not as a skip, so inferring from it
+ // treated those as already counted. Reaching here for a file the
+ // loop passed means it changed between the two stats -- truncated,
+ // or grown past the cap -- and it must be counted, or it vanishes
+ // from the graph, from `filesSkipped` and from every `skipReasons`
+ // bucket at once.
+ //
+ // The check itself is not optional either way: it is the fstat on
+ // the handle we are about to read, which is what makes the size cap
+ // TOCTOU-free (CodeQL js/file-system-race).
+ const countedByStatLoop = accountedByStatLoop.get(absFilePath);
+ if (st.size === 0) {
+ // Re-bucket rather than just skip: the stat loop may have
+ // recorded this as `tooLarge`, and the file has since been
+ // truncated. Leaving it there reports an empty file as
+ // oversized and keeps it out of `filesSkipped` entirely.
+ if (countedByStatLoop === undefined) { filesSkipped++; filesSkippedAsEmpty++; }
+ else if (countedByStatLoop === 'tooLarge') { tooLarge--; filesSkipped++; filesSkippedAsEmpty++; }
+ accountedByStatLoop.set(absFilePath, 'empty');
+ return;
+ }
+ if (st.size > MAX_FILE_BYTES) {
+ // ...and the mirror image, for a file that has since grown.
+ if (countedByStatLoop === undefined) tooLarge++;
+ else if (countedByStatLoop === 'empty') { filesSkipped--; filesSkippedAsEmpty--; tooLarge++; }
+ accountedByStatLoop.set(absFilePath, 'tooLarge');
+ return;
+ }
+ // The other direction of the same race, and it needs undoing
+ // rather than skipping: the stat loop saw this file as empty or
+ // oversized and counted it, and the fstat -- the one we are about
+ // to READ from, so the authoritative one -- says it is neither.
+ // The file is ingested, so leaving the earlier count in place
+ // reports it as skipped in the very number the docs now call the
+ // real one.
+ if (countedByStatLoop !== undefined) {
+ accountedByStatLoop.delete(absFilePath);
+ // Which counter, from the record rather than a guess.
+ if (countedByStatLoop === 'empty') { filesSkipped--; filesSkippedAsEmpty--; }
+ else tooLarge--;
+ }
bytes = await fh.readFile();
} finally {
await fh.close();
@@ -2657,6 +2951,7 @@ export async function ingestFiles(
const hash = sha256(bytes);
if (!opts.force && !forceReingestPaths.has(absFilePath) && knownHashes.get(absFilePath) === hash) {
filesSkipped++;
+ filesSkippedAsUnchanged++;
return;
}
const previousHash = knownHashes.get(absFilePath);
@@ -2684,7 +2979,7 @@ export async function ingestFiles(
const f = fileData[j];
if (!f) continue;
const parsed = parseResults[j] as any;
- if (!parsed) { filesSkipped++; continue; }
+ if (!parsed) { filesSkipped++; filesSkippedUnparsed++; continue; }
entitiesParsed += parsed.entities.length;
batch.push({ filePath: f.filePath, parsed, hash: f.hash, previousHash: f.previousHash });
}
@@ -2786,7 +3081,15 @@ export async function ingestFiles(
projectRoot,
currentMtimes,
latestRev,
- parseErrors,
+ // Lost parses count as parse errors HERE, whatever they are called
+ // elsewhere. A run whose worker pool died resolves every later file as
+ // null, and those increment `filesSkippedUnparsed` rather than
+ // `parseErrors` -- so without this the run wrote an mtime baseline for
+ // tens of thousands of files it never parsed or committed, exited 0, and
+ // every later incremental map skipped them as unchanged. Half the graph
+ // missing, recoverable only by `--force`, and worse than `main`, which
+ // respawns without a cap and loses one file per crash.
+ parseErrors + crashedParses(),
commitErrors,
undefined,
nextDeletedFiles,
@@ -2795,7 +3098,19 @@ export async function ingestFiles(
// Migration cleanup (Ix#225 gap 2): the re-ingest under the new path-based id has
// committed, so delete the OLD id's now-orphaned nodes/edges/patches. Best-effort —
// a failure here is non-fatal (orphans are harmless dead storage, cleanable later).
- if (workspaceMigrated && previousWorkspaceId && ingestCompletedCleanly(parseErrors, commitErrors)) {
+ // `+ crashedParses()`, exactly as the baseline guard below. This one
+ // DELETES the old workspace's graph, so it must be at least as strict: a
+ // run whose parse pool died resolves every later file as null without
+ // raising `parseErrors`, so both guards saw a clean run while the new
+ // workspace held a half-populated graph -- and this one would drop the
+ // complete old one permanently. The two adjacent guards must not disagree
+ // about whether the run was clean, and the destructive one certainly must
+ // not be the laxer of the pair.
+ if (
+ workspaceMigrated &&
+ previousWorkspaceId &&
+ ingestCompletedCleanly(parseErrors + crashedParses(), commitErrors)
+ ) {
try {
await client.deleteWorkspace(previousWorkspaceId);
if (debug) process.stderr.write(` Cleaned up pre-migration nodes under ${previousWorkspaceId}.\n`);
@@ -2808,12 +3123,63 @@ export async function ingestFiles(
// write the cross-repo IMPORTS edges to other separately-ingested repos.
// Best-effort: a failure (or an older backend with no /v1/stitch) never fails
// the ingest. Collection is over the files actually parsed this run, so it is
- // only COMPLETE on a full ingest. Gating on filesSkipped === 0 means a partial
- // incremental re-map (some files mtime-skipped) leaves the prior full
- // registration intact rather than overwriting it with a partial set; a fresh
- // map, `--force`, or a post-reset re-map re-registers. (Incremental registry
- // updates that touch only changed files are a future refinement.)
- if (stitchEnabled && filesSkipped === 0 && stitchFiles.length > 0 && ingestCompletedCleanly(parseErrors, commitErrors)) {
+ // only COMPLETE on a full ingest. Gating on filesSkippedAsUnchanged === 0
+ // means a partial incremental re-map (some files mtime- or hash-skipped)
+ // leaves the prior full registration intact rather than overwriting it with a
+ // partial set; a fresh map, `--force`, or a post-reset re-map re-registers.
+ // (Incremental registry updates that touch only changed files are a future
+ // refinement.)
+ //
+ // Ix#568: the two narrow counts, not `filesSkipped` -- see their
+ // declarations. The broad total also includes empty and minified-looking
+ // files, which contribute nothing under any run, so one empty `__init__.py`
+ // used to disqualify a repo from ever stitching, with or without `--force`.
+ // A CRASHED parse worker is in that category and is counted here: it
+ // resolves its in-flight task as null without raising `parseErrors`, and
+ // stitching then would overwrite a complete registration with one missing
+ // that file's exports. A file `parseFile` simply returned null for is not --
+ // that is deterministic, usually an unavailable optional grammar, and
+ // gating on it blocked stitching forever for any repo containing one.
+ const registrationIsComplete = filesSkippedAsUnchanged === 0 && crashedParses() === 0;
+ // Every way the stitch does not happen is REPORTED, not just the guard's.
+ // `stitch_skipped: null` means "the cross-repo edges are current", so every
+ // path that leaves it unset while skipping the stitch says something false
+ // -- and the two below are far commoner than any refusal the guard makes.
+ // Neither prints a human Note or a `--silent` token: `incomplete` would
+ // appear on nearly every incremental map, and a run with errors has already
+ // said so, loudly, in the lines above.
+ // `lost-parses` FIRST. It is the only one of the three that prints, and a
+ // dead pool also raises `parseErrors`, so testing `run-errors` first meant a
+ // `--force` run with one patch-build failure and four thousand lost files
+ // reported the one and said nothing about the rest -- suppressing the very
+ // Note this rule was added to show.
+ if (stitchEnabled && crashedParses() > 0) {
+ // Its OWN rule, and one that IS printed. `incomplete` is silent because
+ // it fires on nearly every incremental map -- but on a `--force` run
+ // nothing is skipped as unchanged, so `incomplete` there could only ever
+ // mean a crashed worker, and staying silent made the recovery command
+ // this CLI advertises exit 0 having quietly done nothing.
+ stitchSkippedRule = "lost-parses";
+ stitchSkipped =
+ `a parse worker crashed and ${crashedParses()} file(s) went unparsed, so the ` +
+ "cross-workspace registration would be missing them";
+ } else if (stitchEnabled && !ingestCompletedCleanly(parseErrors, commitErrors)) {
+ stitchSkippedRule = "run-errors";
+ stitchSkipped =
+ "this run had parse or commit errors, so its registration would be built from " +
+ "an incomplete picture of the repo";
+ } else if (stitchEnabled && !registrationIsComplete) {
+ // `stitchFiles.length > 0` is deliberately NOT required on any of these
+ // three. A map where nothing changed parses nothing, so `stitchFiles` is
+ // empty and the field stayed unset -- reporting "the cross-repo edges are
+ // current" for the commonest map there is, while a cooldown was refusing
+ // stitches the whole time. The staleness is identical either way.
+ stitchSkippedRule = "incomplete";
+ stitchSkipped =
+ "this map did not re-parse every file, so it has no complete cross-workspace " +
+ "registration to send";
+ }
+ if (stitchEnabled && registrationIsComplete && stitchFiles.length > 0 && ingestCompletedCleanly(parseErrors, commitErrors)) {
try {
const entry = pickEntryFile(stitchFiles);
const provides = entry
@@ -2860,13 +3226,50 @@ export async function ingestFiles(
symbolConsumes.push({ symbol, callerNodeId, pkg });
}
if (provides.length > 0 || stitchConsumes.length > 0 || stitchExports.length > 0 || symbolConsumes.length > 0) {
- const res = await client.stitch({ workspaceId, provides, consumes: stitchConsumes, exports: stitchExports, symbolConsumes });
- if (debug) process.stderr.write(` [stitch] provides=${provides.length} consumes=${stitchConsumes.length} exports=${stitchExports.length} symConsumes=${symbolConsumes.length} -> ${res.stitched} edges\n`);
- // This call is the one thing that can change whether the workspace
- // belongs to a system, and reads cache that answer on disk. Drop it
- // here so the next read asks again rather than scoping to the
- // pre-stitch workspace for as long as the file survives.
- clearStitchScopeCache(workspaceId);
+ // Ix#568: ask before stitching. The join is cross-workspace and
+ // outlives the HTTP call that starts it, so `ix map`'s per-workspace
+ // lock does not bound it -- see stitch-guard.ts. A refusal is not a
+ // failure: the previous registration stands, exactly as it does after
+ // a stitch that FAILED. It is not picked up by the next map either --
+ // see describeStitchSkipped for what actually re-registers.
+ const admission = await admitStitchWaiting(
+ client.endpoint,
+ undefined,
+ undefined,
+ opts.deadlineSignal,
+ );
+ if (!admission.admitted) {
+ stitchSkipped = admission.reason;
+ stitchSkippedRule = admission.rule;
+ if (debug) process.stderr.write(` [stitch not started] ${admission.reason}\n`);
+ } else {
+ const stitchStart = performance.now();
+ let res;
+ try {
+ res = await client.stitch({ workspaceId, provides, consumes: stitchConsumes, exports: stitchExports, symbolConsumes });
+ } catch (err) {
+ // Settle before rethrowing, so the shared catch below still sees
+ // the original error and the lock is never held past the request.
+ admission.settle({
+ ok: false,
+ elapsedMs: performance.now() - stitchStart,
+ // Two proofs that the backend is not running a join, and only
+ // two: it answered and refused (a status), or we never reached
+ // it at all. Everything else -- a 5xx, a timeout, an abort, a
+ // socket dropped after the request went out -- leaves it.
+ status: stitchFailureStatus(err),
+ neverConnected: connectionNeverEstablished(err),
+ });
+ throw err;
+ }
+ admission.settle({ ok: true, elapsedMs: performance.now() - stitchStart });
+ if (debug) process.stderr.write(` [stitch] provides=${provides.length} consumes=${stitchConsumes.length} exports=${stitchExports.length} symConsumes=${symbolConsumes.length} -> ${res.stitched} edges\n`);
+ // This call is the one thing that can change whether the workspace
+ // belongs to a system, and reads cache that answer on disk. Drop it
+ // here so the next read asks again rather than scoping to the
+ // pre-stitch workspace for as long as the file survives.
+ clearStitchScopeCache(workspaceId);
+ }
}
} catch (err) {
// Unsupported is not failed: an older backend answers 404 here, and
@@ -2925,9 +3328,19 @@ export async function ingestFiles(
const summary: IngestFilesSummary = {
filesDiscovered,
patchesApplied,
- parseErrors,
+ // `+ crashedParses()`, as the baseline and delete guards already do. Files
+ // lost to a dead parse pool raise `filesSkippedUnparsed`, never
+ // `parseErrors`, so without this everything downstream read the run as
+ // clean: `describeDroppedFiles` stayed silent, and the completed-map checks
+ // blamed the backend "after a clean local ingest" and cleared the map
+ // baseline. On a co-ingest workspace the `lost-parses` channel is off too
+ // (`stitchEnabled` is false there), so the run exited 0 with no signal
+ // anywhere at all.
+ parseErrors: parseErrors + crashedParses(),
commitErrors,
stitchErrors,
+ stitchSkipped,
+ stitchSkippedRule,
};
// `everTripped()` alone, and it is not a weakening: `recordFailure` has one
// call site, in the per-file catch immediately after `commitErrors++`, so a
@@ -2954,6 +3367,25 @@ export async function ingestFiles(
if (stitchErrors > 0) {
process.stderr.write(` ${describeStitchFailure(stitchError)}\n`);
process.exitCode = 1;
+ } else if (
+ stitchSkipped !== undefined &&
+ // Not on `--silent`. `ix map` always passes `suppressOutput`, and that
+ // surface is deliberately one terse line per run -- which is the stated
+ // reason `incomplete` is filtered out of its token. A ~250-character Note
+ // on every map for the whole 15-minute cooldown is the repetition the short
+ // form exists to avoid; the token still carries the rule.
+ opts.suppressOutput !== true &&
+ // The guard's refusals only. `incomplete` would print on nearly every
+ // incremental map, and `run-errors` restates lines the run has already
+ // printed; both are still on the machine surfaces.
+ stitchSkippedRule !== "incomplete" &&
+ stitchSkippedRule !== "run-errors"
+ ) {
+ // Not an error and not an exit code: nothing failed, and the graph is
+ // exactly where a failed stitch would have left it. But the sentence a
+ // user gets otherwise -- cross-repo relationships may be incomplete --
+ // reads as an unexplained regression without the reason.
+ process.stderr.write(` ${describeStitchSkipped(stitchSkipped, stitchSkippedRule, debug)}\n`);
}
if (commitReport.kind === "warn") {
process.stderr.write(` ${commitReport.message}\n`);
@@ -2979,9 +3411,32 @@ export async function ingestFiles(
filesSkipped,
entitiesParsed,
latestRev,
- skipReasons: { unchanged: filesSkipped, emptyFile: 0, parseError: parseErrors, tooLarge, minifiedLikely, outsideRoot },
+ // `unchanged` is the files we ASSUMED unchanged, not every skip. It used
+ // to be `filesSkipped`, which also counts empty, minified-looking and
+ // unparseable files -- so a consumer reading this could not tell an
+ // incremental no-op from a repo full of empty __init__.py. `emptyFile` was
+ // hardcoded 0 for the same reason and is now the real count.
+ //
+ // `unparsed` is its own bucket. Narrowing `unchanged` left the pool's
+ // nulls in no bucket at all, but folding them into `parseError` was
+ // worse: that counts every parse-stage failure, including stat, read and
+ // patch-build errors that were never skips, so one unstat-able file gave
+ // `filesSkipped: 0` with a bucket claiming 1. `unchanged`, `emptyFile`,
+ // `minifiedLikely` and `unparsed` are the buckets that are subsets of
+ // `filesSkipped`; `parseError` and `tooLarge` are counted separately and
+ // always were.
+ skipReasons: { unchanged: filesSkippedAsUnchanged, emptyFile: filesSkippedAsEmpty, parseError: parseErrors + crashedParses(), unparsed: filesSkippedUnparsed, tooLarge, minifiedLikely, outsideRoot },
commitErrors,
stitchErrors,
+ // Ix#568. Present only when the stitch was refused before it was sent, so
+ // a consumer reading this body can tell "cross-repo edges are current"
+ // from "cross-repo edges are as stale as the last successful stitch" --
+ // a distinction the exit code deliberately does not make.
+ // `?? null`, not left undefined: JSON.stringify drops an undefined
+ // value, so a consumer could not tell the field apart from an older CLI
+ // that never emitted it. `ix map --format json` does the same.
+ stitchSkipped: stitchSkipped ?? null,
+ stitchSkippedRule: stitchSkippedRule ?? null,
elapsedSeconds: parseFloat(elapsed),
timings: {
...timings,
@@ -2999,8 +3454,18 @@ export async function ingestFiles(
console.log(` processed: ${patchesApplied} files (${elapsed}s)`);
console.log(` discovered: ${filesDiscovered} files`);
console.log(` changed: ${filesChanged} files`);
- if (filesSkipped > 0) console.log(` ${chalk.dim('skipped unchanged:')} ${filesSkipped}`);
- if (parseErrors > 0) console.log(` ${chalk.red('parse errors:')} ${parseErrors}`);
+ // "skipped" without "unchanged": the count includes empty, minified-looking
+ // and unparseable files, none of which were assumed unchanged. Splitting the
+ // counter for the stitch gate (Ix#568) made the old label demonstrably
+ // wrong on a repo with an empty __init__.py and --force.
+ if (filesSkipped > 0) console.log(` ${chalk.dim('skipped:')} ${filesSkipped}`);
+ // `+ crashedParses()`, matching the summary field. On a co-ingest workspace
+ // `stitchEnabled` is false, so the `lost-parses` Note never fires and
+ // `describeDroppedFiles` is only wired into `ix map` -- which left
+ // `ix ingest` on such a workspace printing "skipped: 4000", no parse-error
+ // line at all, and exiting 0 after losing its whole parse pool.
+ const parseErrorsShown = parseErrors + crashedParses();
+ if (parseErrorsShown > 0) console.log(` ${chalk.red('parse errors:')} ${parseErrorsShown}`);
if (commitErrors > 0) console.log(` ${chalk.red('commit errors:')} ${commitErrors}`);
if (tooLarge > 0) console.log(` ${chalk.dim('skipped too large:')} ${tooLarge}`);
if (minifiedLikely > 0) console.log(` ${chalk.dim('skipped minified:')} ${minifiedLikely}`);
diff --git a/ix-cli/src/cli/commands/map.ts b/ix-cli/src/cli/commands/map.ts
index 57f0141b..735d972c 100644
--- a/ix-cli/src/cli/commands/map.ts
+++ b/ix-cli/src/cli/commands/map.ts
@@ -535,8 +535,33 @@ Examples:
const systems = result.regions.filter(r => r.label_kind === "system").length;
const subsystems = result.regions.filter(r => r.label_kind === "subsystem").length;
const modules = result.regions.filter(r => r.label_kind === "module").length;
+ // Ix#568: `--silent` returns before both format branches AND wins over
+ // `--format llm`, so it is the one output the hooks this field exists
+ // for actually see. A skipped stitch deliberately does not move the exit
+ // code -- without a token here an automated consumer cannot tell a clean
+ // map from one whose cross-repo edges are up to 15 minutes stale.
+ // Ix#568. The RULE, not just the fact -- and not for `incomplete`.
+ //
+ // `--silent` is the hook surface: one terse line per run. `incomplete`
+ // fires on nearly every incremental map, so emitting it here would put
+ // a token on almost every line and make `stitch_skipped` useless as a
+ // signal, while a consumer that actually needs to know an incremental
+ // map registered nothing has `--format json` and `--format llm`, which
+ // both carry it. What stays here is the guard refusing -- the case that
+ // means a backend is being protected from stacked joins.
+ const rule = localIngest?.stitchSkippedRule;
+ const stitch =
+ rule === undefined || rule === "incomplete" || rule === "run-errors"
+ ? ""
+ // `stitch_skipped_rule`, not `stitch_skipped`. The json and llm
+ // formats put the English prose under `stitch_skipped` and the rule
+ // under `stitch_skipped_rule`, and emitting the RULE under the
+ // prose key here gave one name two value spaces: a hook matching
+ // `stitch_skipped=cooling` on this line silently stopped matching
+ // the moment it was pointed at `--format json`.
+ : ` · stitch_skipped_rule=${rule}`;
process.stderr.write(
- `map: ${result.file_count} files · ${systems}s/${subsystems}ss/${modules}m regions · ${mapMs}ms\n`
+ `map: ${result.file_count} files · ${systems}s/${subsystems}ss/${modules}m regions · ${mapMs}ms${stitch}\n`
);
return;
}
@@ -569,6 +594,14 @@ Examples:
// from one missing every file that failed to build a patch (#554).
parse_errors: localIngest?.parseErrors ?? 0,
commit_errors: localIngest?.commitErrors ?? 0,
+ // Ix#568. The whole reason this is reported at all is hooks that run
+ // `ix map` and read the machine output; leaving it only in
+ // `ix ingest --format json` puts it where those hooks never look.
+ // `?? null`, not left undefined: JSON.stringify drops an undefined
+ // value, so a consumer could not tell the field apart from an older
+ // CLI that never emitted it. Its siblings are always present too.
+ stitch_skipped: localIngest?.stitchSkipped ?? null,
+ stitch_skipped_rule: localIngest?.stitchSkippedRule ?? null,
regions: regions.map((r: any) => ({
label: r.label,
level: r.level,
@@ -593,7 +626,7 @@ Examples:
export function renderMapLlm(
result: MapResult,
regions: MapRegion[],
- ingest?: Pick<IngestFilesSummary, "parseErrors" | "commitErrors">,
+ ingest?: Pick<IngestFilesSummary, "parseErrors" | "commitErrors" | "stitchSkipped" | "stitchSkippedRule">,
): void {
console.log(llmLine("map", [
["files", result.file_count],
@@ -608,6 +641,11 @@ export function renderMapLlm(
// signal are dropped (docs/llm-format.md); a clean ingest says nothing.
["parse_errors", ingest?.parseErrors ? ingest.parseErrors : undefined],
["commit_errors", ingest?.commitErrors ? ingest.commitErrors : undefined],
+ // Ix#568. `--format llm` and `--silent` are what the hooks this field was
+ // added for actually read; shipping it only in `--format json` put it
+ // where they never look.
+ ["stitch_skipped", ingest?.stitchSkipped],
+ ["stitch_skipped_rule", ingest?.stitchSkippedRule],
]));
for (const r of regions) {
console.log(llmLine("region", [
diff --git a/ix-cli/src/cli/commands/parse-pool.ts b/ix-cli/src/cli/commands/parse-pool.ts
index 324e2aba..e25470c7 100644
--- a/ix-cli/src/cli/commands/parse-pool.ts
+++ b/ix-cli/src/cli/commands/parse-pool.ts
@@ -13,6 +13,8 @@ type Task = {
filePath: string;
source: string;
resolve: (result: unknown) => void;
+ /** Does losing this one mean the RUN lost a file? See `parse`. */
+ counts: boolean;
};
export class ParsePool {
@@ -29,14 +31,49 @@ export class ParsePool {
}
}
- parse(filePath: string, source: string): Promise<unknown> {
+ /**
+ * @param countLoss Whether losing this task means the run lost a file.
+ *
+ * False for the index prescan, which dispatches through this same pool and
+ * whose caller drops nulls on the floor: those files are read and parsed
+ * again in the streaming loop and land in the graph regardless. Counting them
+ * made one transient prescan crash report `lost-parses`, refuse the stitch,
+ * withhold the mtime baseline (so every later map re-read the whole repo) and
+ * skip the post-migration delete -- for files that were all present. And
+ * because most extensions are prescanned AND parsed, a dead pool could count
+ * the same file twice and report more losses than the run discovered files.
+ */
+ parse(filePath: string, source: string, countLoss = true): Promise<unknown> {
+ // A pool with no workers left cannot ever run this, so resolve it here
+ // rather than enqueue it. Draining only the queue that existed at the
+ // moment the last worker died left every LATER `parse()` waiting on a
+ // `drain()` that is a no-op with an empty idle list -- the promise never
+ // settled and the `Promise.all` over the next chunk hung the ingest, which
+ // is the same failure this file has now produced three different ways.
+ if (this.dead) {
+ if (countLoss) this.crashed++;
+ return Promise.resolve(null);
+ }
return new Promise((resolve) => {
- this.queue.push({ filePath, source, resolve });
+ this.queue.push({ filePath, source, resolve, counts: countLoss });
this.drain();
});
}
async destroy(): Promise<void> {
+ // Set FIRST. `terminate()` makes every worker emit 'exit', and the handler
+ // for that treats an exit as a crash and spawns a replacement -- so
+ // shutting the pool down span up a fresh set of threads, `this.workers = []`
+ // orphaned them, and because a worker thread refs the event loop and the
+ // CLI has no `process.exit(0)` on its success path, `ix map` printed its
+ // summary and then hung forever.
+ this.destroyed = true;
+ // ...and `dead`, so a `parse()` after teardown resolves instead of queueing
+ // onto a pool with no workers left to run it. Not reachable from
+ // `ingestFiles` today -- `destroy()` is in the outermost `finally` -- but it
+ // is the identical hang this file has already produced three times, and one
+ // line closes it.
+ this.dead = true;
await Promise.all(this.workers.map(w => w.terminate()));
this.workers = [];
this.idle = [];
@@ -46,6 +83,14 @@ export class ParsePool {
const w = new Worker(this.workerPath);
w.on('message', (msg: { ok: boolean; result: unknown }) => this.onResult(w, msg));
w.on('error', (err) => this.onError(w, err));
+ // 'exit' as well as 'error'. A worker can go without ever emitting 'error'
+ // -- `process.exit()` inside it, or the thread killed out from under the
+ // runtime -- and then its in-flight task never resolves, the `Promise.all`
+ // over the parse batch never settles, and the ingest hangs before it can
+ // even reach `destroy()`. `onError` already covers the crash path; this
+ // covers the quiet one, and both feed `crashedTasks()` so the stitch gate
+ // sees a run that lost a file either way.
+ w.on('exit', () => this.onError(w, new Error('parse worker exited')));
this.workers.push(w);
this.idle.push(w);
return w;
@@ -60,28 +105,106 @@ export class ParsePool {
}
}
+ /**
+ * In-flight tasks lost to a crashed worker.
+ *
+ * `parse()` resolves `null` for two very different things, and collapsing
+ * them hid a real difference (Ix#568): a worker that CRASHED lost a file it
+ * would otherwise have parsed, which is transient and makes the run's results
+ * incomplete -- while `parseFile` returning null is deterministic and says
+ * this file has nothing to give, most often because its grammar is an
+ * unavailable optional dependency (`tree-sitter-sas` has no win32 prebuild,
+ * for one). Callers that need "did we miss anything we would have indexed?"
+ * must ask this and not count nulls.
+ */
+ crashedTasks(): number {
+ return this.crashed;
+ }
+
private onResult(w: Worker, msg: { ok: boolean; result: unknown }): void {
const task = this.active.get(w);
if (!task) return;
this.active.delete(w);
+ // A successful round trip clears the respawn budget. The cap exists to
+ // stop a worker that dies deterministically from spinning spawn -> die ->
+ // spawn; it is not meant to be a lifetime quota. As a per-run total it made
+ // ~23 transient deaths on a long run kill the pool for good, and on `main`
+ // those cost 23 files rather than the remainder of the ingest.
+ this.respawns = 0;
task.resolve(msg.ok ? msg.result : null);
this.idle.push(w);
this.drain();
}
+ private crashed = 0;
+
+ /** True once `destroy()` has begun, so a deliberate exit is not read as a crash. */
+ private destroyed = false;
+
+ /** Replacements spawned for crashed workers. Capped -- see `onError`. */
+ private respawns = 0;
+
+ /**
+ * True once the pool is out of workers AND out of replacements.
+ *
+ * Latched, because the condition is permanent: nothing spawns a worker after
+ * the cap, so every later `parse()` would queue forever.
+ */
+ private dead = false;
+
+ /** Generous enough for real flakiness, small enough to stop a spawn loop. */
+ private static readonly MAX_RESPAWNS = 16;
+
private onError(w: Worker, _err: Error): void {
+ // Not during shutdown. `destroy()` terminates every worker on purpose, and
+ // treating that as a crash respawns the pool it is trying to close: the
+ // replacements outlive `this.workers = []`, a worker thread refs the event
+ // loop, and the CLI prints its summary and then hangs forever. The guard
+ // lives HERE rather than on the 'exit' listener because 'error' reaches the
+ // same code: a worker that faults while `destroy()` is awaiting
+ // `terminate()` took the identical path.
+ if (this.destroyed) return;
const task = this.active.get(w);
if (task) {
this.active.delete(w);
+ if (task.counts) this.crashed++;
task.resolve(null); // isolate: failed file = null parse result
}
- // Replace the crashed worker
+ // Replace the crashed worker -- but not forever. A worker that dies
+ // deterministically and without an 'error' (a native module that calls
+ // `process.exit(1)` on load, say) would otherwise spin spawn -> exit ->
+ // spawn for the rest of the run. That loop is newly reachable, because
+ // before the 'exit' listener a silent death was simply ignored. Past the
+ // cap the pool runs smaller, which the queue drains through fine, and every
+ // lost task is still counted for the stitch gate.
const idx = this.workers.indexOf(w);
- if (idx !== -1) {
- w.terminate().catch(() => {});
- this.workers.splice(idx, 1);
+ if (idx === -1) return;
+ w.terminate().catch(() => {});
+ this.workers.splice(idx, 1);
+ // ...and out of `idle` too, ALWAYS, cap or no cap. A worker can emit 'error'
+ // while it is IDLE -- an uncaught async throw, or ERR_WORKER_OUT_OF_MEMORY
+ // between tasks -- and leaving the terminated thread in the free list means
+ // a later `drain()` pops it and posts to nothing: the task's promise never
+ // settles and the `Promise.all` over the parse batch hangs the ingest.
+ const idleIdx = this.idle.indexOf(w);
+ if (idleIdx !== -1) this.idle.splice(idleIdx, 1);
+
+ if (this.respawns < ParsePool.MAX_RESPAWNS) {
+ this.respawns++;
this.spawnWorker();
- this.drain();
+ } else if (this.workers.length === 0) {
+ // Out of workers and out of replacements. Nothing queued can ever be
+ // parsed, so resolve it rather than leave `Promise.all` waiting on a pool
+ // that no longer exists -- and LATCH it, because `parse()` must answer the
+ // same way for every call after this, not just for what happened to be
+ // queued at this instant. Counted as crashed, which is what it is, so the
+ // stitch gate knows this run lost files.
+ this.dead = true;
+ const stranded = this.queue.splice(0, this.queue.length);
+ this.crashed += stranded.filter(t => t.counts).length;
+ for (const t of stranded) t.resolve(null);
+ return;
}
+ this.drain();
}
}
diff --git a/ix-cli/src/cli/commands/reset.ts b/ix-cli/src/cli/commands/reset.ts
index ac11969e..9e9c2ed7 100644
--- a/ix-cli/src/cli/commands/reset.ts
+++ b/ix-cli/src/cli/commands/reset.ts
@@ -5,6 +5,7 @@ import { IxClient } from "../../client/api.js";
import { getEndpoint, clearIngestMtimeCache, clearStitchScopeCache } from "../config.js";
import { canRenderProgress } from "../stderr.js";
import { resolveWorkspaceId } from "../bootstrap.js";
+import { clearStitchCooldown } from "../stitch-guard.js";
/**
* Drop this workspace's cached "am I stitched into a system?" answer.
@@ -70,6 +71,19 @@ export function registerResetCommand(program: Command): void {
// Clear the mtime cache so the next ix map re-ingests all files
clearIngestMtimeCache(process.cwd());
clearStitchScopeCacheForCwd();
+ // Ix#568: and the stitch cooldown. The re-ingest below is the one
+ // run that can re-register this workspace, and a live cooldown would
+ // refuse exactly it -- leaving the workspace unregistered with
+ // nothing to retry, since an incremental map never reaches the stitch
+ // block.
+ //
+ // This IS a trade, not a free clear, and the trade is worth stating:
+ // wiping the graph does not stop a runaway AQL join -- that is the
+ // whole premise of #568, the query outlives everything -- so clearing
+ // the marker here can put a second cross-workspace join onto one that
+ // is still running. One extra join against a workspace that would
+ // otherwise stay unregistered with no way back.
+ clearStitchCooldown(getEndpoint());
console.log(chalk.green("✓") + " Code graph wiped. Planning artifacts preserved.");
console.log(chalk.dim(" Run `ix map` to rebuild the code graph."));
} else {
@@ -77,6 +91,19 @@ export function registerResetCommand(program: Command): void {
stopSpinner();
clearIngestMtimeCache(process.cwd());
clearStitchScopeCacheForCwd();
+ // Ix#568: and the stitch cooldown. The re-ingest below is the one
+ // run that can re-register this workspace, and a live cooldown would
+ // refuse exactly it -- leaving the workspace unregistered with
+ // nothing to retry, since an incremental map never reaches the stitch
+ // block.
+ //
+ // This IS a trade, not a free clear, and the trade is worth stating:
+ // wiping the graph does not stop a runaway AQL join -- that is the
+ // whole premise of #568, the query outlives everything -- so clearing
+ // the marker here can put a second cross-workspace join onto one that
+ // is still running. One extra join against a workspace that would
+ // otherwise stay unregistered with no way back.
+ clearStitchCooldown(getEndpoint());
console.log(chalk.green("✓") + " Graph wiped.");
}
} catch (err: any) {
diff --git a/ix-cli/src/cli/single-flight.ts b/ix-cli/src/cli/single-flight.ts
index d78c3378..ade62fa7 100644
--- a/ix-cli/src/cli/single-flight.ts
+++ b/ix-cli/src/cli/single-flight.ts
@@ -49,6 +49,12 @@ export interface LockHandle {
function lockMaxMs(): number {
const raw = process.env.IX_MAP_LOCK_MAX_MS;
if (!raw) return DEFAULT_LOCK_MAX_MS;
+ // Matched, not parsed. `Number.parseInt` reads any prefix, so an operator
+ // following the env table and writing `20m` got TWENTY MILLISECONDS: every
+ // held lock reads as abandoned on the next process's first look, and
+ // single-flight goes inert for the map lock and the stitch lock alike. The
+ // stitch cooldown backstops one of those; the map lock has nothing.
+ if (!/^\d+$/.test(raw)) return DEFAULT_LOCK_MAX_MS;
const n = Number.parseInt(raw, 10);
return Number.isFinite(n) && n > 0 ? n : DEFAULT_LOCK_MAX_MS;
}
@@ -57,8 +63,22 @@ function lockPathFor(key: string): string {
let canonicalKey: string;
try { canonicalKey = realpathSync.native(key); }
catch { canonicalKey = resolve(key); }
- const h = createHash("sha256").update(canonicalKey).digest("hex").slice(0, 16);
- return join(lockDir(), `map-${h}.lock`);
+ return namedLockPath("map", canonicalKey);
+}
+
+/**
+ * Lock file for an arbitrary key, hashed verbatim.
+ *
+ * The map lock canonicalises its key first because it is a filesystem path and
+ * two spellings of one directory must take the same lock. A key that is not a
+ * path -- a backend endpoint, say -- must NOT go through realpath: it does not
+ * name a file, so realpath fails and the fallback `resolve()` would join it to
+ * the current working directory, giving one endpoint a different lock per
+ * directory the command happens to run from.
+ */
+export function namedLockPath(namespace: string, key: string): string {
+ const h = createHash("sha256").update(key).digest("hex").slice(0, 16);
+ return join(lockDir(), `${namespace}-${h}.lock`);
}
/** True when a PID is alive on this host. signal 0 = existence check, no-op. */
@@ -100,8 +120,20 @@ function isStale(meta: LockMeta | null): boolean {
* acquisition retried once.
*/
export function acquireMapLock(workspaceRoot: string, label: string): LockHandle | null {
+ return acquireLockAt(lockPathFor(workspaceRoot), label);
+}
+
+/**
+ * Acquire the lock stored at `path`, with the same semantics as
+ * [[acquireMapLock]]: a LockHandle on success, null when a live holder owns it.
+ *
+ * Split out so a caller that is not keyed by a workspace root -- the stitch
+ * guard, which is keyed by backend endpoint because the stitch join is
+ * cross-workspace -- gets the identical, already-proven acquisition, staleness
+ * and release behaviour instead of a second implementation of it.
+ */
+export function acquireLockAt(path: string, label: string): LockHandle | null {
try { mkdirSync(lockDir(), { recursive: true }); } catch { /* best effort */ }
- const path = lockPathFor(workspaceRoot);
const meta: LockMeta = { pid: process.pid, host: hostname(), startedAt: Date.now(), label };
const tryCreate = (): boolean => {
diff --git a/ix-cli/src/cli/stitch-guard.ts b/ix-cli/src/cli/stitch-guard.ts
new file mode 100644
index 00000000..136ce0af
--- /dev/null
+++ b/ix-cli/src/cli/stitch-guard.ts
@@ -0,0 +1,569 @@
+import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { dirname } from "node:path";
+import { acquireLockAt, namedLockPath, type LockHandle } from "./single-flight.js";
+
+// ---------------------------------------------------------------------------
+// Admission control for POST /v1/stitch.
+//
+// Background (Ix#568): the cross-workspace stitch is a single unbounded AQL
+// join over `symbol_consumes x provides x consumes x exports`. On a large
+// multi-workspace graph it runs for minutes. The HTTP call in front of it gives
+// up long before the query does — the reported case is a proxy answering 500 at
+// ~60s — and ArangoDB keeps executing the query regardless. The client learning
+// "that failed" is therefore NOT the same as the work having stopped.
+//
+// `ix map` already single-flights per workspace, and that does not help here
+// for two independent reasons:
+//
+// * The stitch is cross-workspace, so two maps of two DIFFERENT workspaces
+// hold two different locks and issue two stitches at once. Reproduced: two
+// concurrent `ix map` runs put two stitch queries in flight together.
+// * Copies mostly do not need concurrency at all. An auto-map hook fires one
+// `ix map` per change; each completes, releases its workspace lock, and the
+// next one starts a fresh stitch. Reproduced: five sequential maps, five
+// stitch queries, every one started after the previous had already answered
+// 500 — i.e. while its server-side join was still running.
+//
+// ── The rule ───────────────────────────────────────────────────────────────
+//
+// The cooldown marker is written when a stitch STARTS, and removed only on
+// proof that nothing is running: a clean answer, or a refusal the backend
+// issued without doing the work.
+//
+// This is deliberately the opposite way round from the obvious design, which
+// inspects the FAILURE and writes a cooldown when it looks like a timeout.
+// Four review rounds of that classifier each broke the previous round's rule,
+// because it has to answer "did the backend start the join?" from an error, and
+// the evidence for that keeps not being there:
+//
+// * the message is whatever proxy sits in front (a 500 here, a 504 with an
+// HTML body in #528), so text matching is inert in the deployment it was
+// written for;
+// * elapsed time cannot tell a slow join from a slow UPLOAD, and the stitch
+// payload is megabytes on a large monorepo;
+// * "the client aborted" cannot tell a mid-flight hang-up from a run deadline
+// that fired before the request was even serialized;
+// * and none of it survives the process being killed, which is the ordinary
+// end of a hook that timed out.
+//
+// Writing the marker up front needs none of those answers. Every way a stitch
+// can end without a definite refusal — a timeout, an abort, a deadline, SIGTERM,
+// a power cut — leaves the marker exactly where it was written, because leaving
+// it is the default rather than something the dying process must still do.
+//
+// The cost is that a stitch which failed for an unclassified reason cools down
+// when it need not have. That errs toward skipping one stitch rather than
+// stacking joins on a database that is already the reason, which is the safe
+// direction here: a skipped stitch leaves the previous registration in place,
+// and #568 is a report of 10–16 concurrent copies at 800–1200% CPU.
+// ---------------------------------------------------------------------------
+
+const DEFAULT_COOLDOWN_MS = 15 * 60 * 1000;
+/** Budget for waiting out a stitch that is merely in flight. */
+const DEFAULT_WAIT_MS = 30 * 1000;
+/** Poll interval while waiting. Short: a healthy stitch is over in ~30ms. */
+const POLL_MS = 250;
+
+function positiveEnvMs(name: string, fallback: number): number {
+ const raw = process.env[name];
+ if (raw === undefined || raw === "") return fallback;
+ // Matched, not parsed. `Number.parseInt` reads any prefix it can, so an
+ // operator following the env table and writing `15m` for the cooldown got
+ // FIFTEEN MILLISECONDS -- past the `n >= 0` check, no fallback, nothing
+ // logged, and `ix map` back to stacking joins with no sign anything is wrong.
+ // `1e6` gave 1 the same way. `commitFailureLimit` learned this already.
+ if (!/^\d+$/.test(raw)) return fallback;
+ const n = Number.parseInt(raw, 10);
+ // 0 is meaningful (disable), so only a negative or unparseable value falls
+ // back. `Number.isFinite` alone would let "" and "abc" through as NaN.
+ return Number.isFinite(n) && n >= 0 ? n : fallback;
+}
+
+/** How long to refuse a new stitch after one that may still be running. 0 disables. */
+export function stitchCooldownMs(): number {
+ return positiveEnvMs("IX_STITCH_COOLDOWN_MS", DEFAULT_COOLDOWN_MS);
+}
+
+/** How long to wait for the in-flight stitch to finish before shedding. */
+export function stitchWaitMs(): number {
+ return positiveEnvMs("IX_STITCH_WAIT_MS", DEFAULT_WAIT_MS);
+}
+
+/**
+ * One key per backend, however the endpoint was spelled.
+ *
+ * `getEndpoint()` returns `IX_ENDPOINT` or the config file verbatim, so the
+ * same backend reaches this function as `http://localhost:8090`,
+ * `http://localhost:8090/` and `http://127.0.0.1:8090` depending on which of
+ * those a given process was started with. Hashing the raw string gives each
+ * spelling its own lock, and the guard then permits exactly the concurrency it
+ * exists to prevent — an `ix mcp` server launched with an IP and a shell
+ * `ix map` reading the config both stitching at once.
+ *
+ * Loopback spellings are folded together deliberately. `localhost`,
+ * `127.0.0.1` and `::1` are not the same host in general — and the endpoint
+ * is not always local either; `IxClient.isLocalEndpoint` exists precisely
+ * because a remote backend is supported. The folding is justified by its cost,
+ * not by a claim about deployments: two loopback spellings that really did
+ * name different backends cost one deferred stitch, while not folding them
+ * costs the concurrency this guard exists to prevent.
+ */
+export function stitchKey(endpoint: string): string {
+ try {
+ const u = new URL(endpoint);
+ const host = ["localhost", "127.0.0.1", "[::1]", "::1"].includes(u.hostname.toLowerCase())
+ ? "localhost"
+ : u.hostname.toLowerCase();
+ const path = u.pathname.replace(/\/+$/, "");
+ // `u.port` is deliberately used raw. The WHATWG parser already strips a
+ // DEFAULT port, so `https://h` and `https://h:443` both give "" and key
+ // identically -- adding a defaults table here would be dead code. Verified,
+ // and pinned by a test, because it looks like an omission.
+ return `${u.protocol.toLowerCase()}//${host}:${u.port}${path}`;
+ } catch {
+ // Not a URL. Nothing to normalise, and refusing to guard is worse than
+ // guarding on the raw text.
+ return endpoint;
+ }
+}
+
+/** Cooldown record. Sits beside the lock so IX_LOCK_DIR redirects both. */
+function cooldownPath(endpoint: string): string {
+ return namedLockPath("stitch", stitchKey(endpoint)).replace(/\.lock$/, ".cooldown");
+}
+
+interface Cooldown {
+ /**
+ * Epoch ms the cooldown runs from.
+ *
+ * Written as the attempt's START time, and rewritten to its END time when it
+ * reports back without proving anything stopped. Both matter:
+ *
+ * - the start-time write is what survives a kill, and is the whole reason
+ * the marker goes down before the request rather than after it;
+ * - but leaving it AT the start time would make any IX_STITCH_COOLDOWN_MS
+ * shorter than the attempt itself already expired by the time the attempt
+ * ends. `IxClient.post` caps a request at two minutes, so every setting
+ * under that silently bought nothing on the timeout path -- the next map
+ * was admitted immediately and stacked a second join, which is the bug.
+ *
+ * A process that is killed keeps the start stamp, so a cooldown shorter than
+ * the attempt is still expired there; that residue is noted in docs/api.
+ */
+ at: number;
+ /** Set when the attempt reported back without clearing. Message only. */
+ elapsedMs?: number;
+}
+
+function readCooldown(endpoint: string): Cooldown | null {
+ try {
+ const parsed = JSON.parse(readFileSync(cooldownPath(endpoint), "utf-8")) as Cooldown;
+ return typeof parsed?.at === "number" ? parsed : null;
+ } catch {
+ // Missing, unreadable, or corrupt: no cooldown. Failing open here is right
+ // — a lost cooldown costs one extra stitch, and refusing to stitch because
+ // a state file cannot be parsed would break the feature outright.
+ return null;
+ }
+}
+
+/**
+ * Did the request fail before the connection was ever established?
+ *
+ * This is the one transport fact that is a positive PROOF rather than an
+ * inference about a failure: if we never connected, no bytes reached the
+ * backend, so there is no join to wait out. Without it a backend that goes down
+ * mid-run takes a full 15-minute endpoint-wide cooldown for a request it never
+ * received.
+ *
+ * Keyed on the SYSCALL, not on a list of codes. `IxClient` flattens HTTP errors
+ * into `"NNN: ..."` strings but rethrows transport failures intact, so what
+ * arrives is Node's `TypeError: fetch failed` with the real error on `.cause` --
+ * and Node stamps `syscall` on every libuv error, which says which phase failed
+ * far more reliably than the errno does. Observed through `IxClient.stitch`:
+ *
+ * closed port code ECONNREFUSED syscall connect
+ * unresolvable DNS code ENOTFOUND syscall getaddrinfo
+ * unroutable address code ENETUNREACH syscall connect
+ * blackholed address code UND_ERR_CONNECT_TIMEOUT syscall undefined
+ * reset AFTER send code UND_ERR_SOCKET syscall undefined
+ *
+ * An enumerated code list got the first two and missed the next two, which is
+ * the same harm for a neighbouring errno: a VPN drop or an unreachable host
+ * cooling the endpoint down for fifteen minutes over a request that never left.
+ * `connect` and `getaddrinfo` cover every one of those without naming them, and
+ * they extend to `EHOSTUNREACH` and a connect-phase `ETIMEDOUT` for free --
+ * while a `read`/`write` `ETIMEDOUT`, which happens after the bytes are gone,
+ * correctly does not qualify.
+ *
+ * `UND_ERR_CONNECT_TIMEOUT` is undici's own and carries no syscall, so it is
+ * named. `UND_ERR_SOCKET` is the deliberate exclusion: "other side closed"
+ * happens after the request went out, so it is exactly the ambiguous case this
+ * guard exists for -- an upstream that restarted killed its join, a proxy that
+ * dropped the connection did not, and both spell the same code.
+ *
+ * The walk descends into `AggregateError.errors` as well as `.cause`, and the
+ * errno set is kept alongside the syscall rule rather than replaced by it. That
+ * is not belt and braces: a multi-address host -- which `localhost` is, and it
+ * is this CLI's DEFAULT endpoint -- fails happy-eyeballs with
+ *
+ * AggregateError { code: "ECONNREFUSED", syscall: undefined, cause: undefined,
+ * errors: [ {ECONNREFUSED, connect}, {ECONNREFUSED, connect} ] }
+ *
+ * so a syscall-only rule that walks only `.cause` answers false for the single
+ * most common way this fires. Verified by execution: `http://localhost:8099`
+ * takes that shape, `http://127.0.0.1:8099` and `http://[::1]:8099` do not.
+ * Tests that used a literal IP could not see the difference.
+ */
+export function connectionNeverEstablished(error: unknown): boolean {
+ // Codes that only ever arise before any byte is sent. `ETIMEDOUT` is
+ // deliberately absent -- it is a connect timeout OR a retransmission timeout
+ // long after the request left -- and is picked up by the syscall rule when it
+ // is the former.
+ const PRE_CONNECT = new Set([
+ "ECONNREFUSED",
+ "ENOTFOUND",
+ "ENETUNREACH",
+ "EHOSTUNREACH",
+ "EAI_AGAIN",
+ "UND_ERR_CONNECT_TIMEOUT",
+ ]);
+
+ // Depth-capped rather than cycle-tracked: undici nests two or three levels and
+ // an AggregateError adds one, so the cap is what stops a pathological chain.
+ const seen = (e: unknown, depth: number): boolean => {
+ if (e === null || e === undefined || depth > 5) return false;
+ const { syscall, code, cause, errors } = e as {
+ syscall?: unknown; code?: unknown; cause?: unknown; errors?: unknown;
+ };
+ if (syscall === "connect" || syscall === "getaddrinfo") return true;
+ if (typeof code === "string" && PRE_CONNECT.has(code)) return true;
+ if (Array.isArray(errors) && errors.some(inner => seen(inner, depth + 1))) return true;
+ return seen(cause, depth + 1);
+ };
+ return seen(error, 0);
+}
+
+/** How the stitch attempt ended, as the guard needs to see it. */
+export interface StitchOutcome {
+ ok: boolean;
+ /** Wall-clock the client spent. Message only — never part of the decision. */
+ elapsedMs: number;
+ /** HTTP status the backend answered with, when it answered at all. */
+ status?: number | null;
+ /** Set when the connection was never established — see the function above. */
+ neverConnected?: boolean;
+}
+
+/**
+ * Did this outcome PROVE that no join is running?
+ *
+ * Every proof here is a positive fact rather than an inference drawn from the
+ * shape of a failure:
+ *
+ * - the stitch succeeded, so the backend is done with it;
+ * - the backend answered 4xx, which is it refusing the request rather than
+ * executing it. 408 is excluded: a proxy reporting that IT gave up waiting
+ * says nothing about whether the backend did, and that is this whole bug.
+ * - the backend answered 501. `isStitchUnsupported` already treats 404 AND
+ * 501 as "this backend has no /v1/stitch", and the run swallows it
+ * silently for exactly that reason -- so leaving a marker behind would
+ * refuse every map for 15 minutes, claiming a join may still be running,
+ * against a backend that has never had one.
+ *
+ * - the connection was never established, so nothing was sent. See
+ * `connectionNeverEstablished` for why that is narrower than "a transport
+ * error" and why the narrowing matters;
+ * A 2xx whose body would not parse is deliberately NOT on that list, though
+ * two earlier revisions put it there. The argument for it was that the request
+ * completed, so the join did -- and the argument against is stronger: a proxy
+ * answering 200 with its own HTML error page when the upstream stalls produces
+ * exactly that shape, and it is the #528 pattern. Clearing the marker there is
+ * the one way any proof arm can cause the #568 failure itself, by admitting the
+ * next map onto a join that is still running.
+ *
+ * The symmetry argument does not hold either: a proxy answering 200 with
+ * PARSEABLE JSON is `ok: true` and has always cleared the marker, but a proxy
+ * error page is HTML, so the unparseable case is the LIKELY one rather than the
+ * equivalent one. The cost of excluding it is a cooldown after a stitch that
+ * really did succeed and got garbled in transit, which is the direction this
+ * whole file errs in by design.
+ *
+ * Anything else — another 5xx, a timeout, an abort, a socket dropped after the
+ * request went out, or the process being killed before it could say anything —
+ * leaves the marker.
+ */
+export function outcomeProvesNothingRunning(outcome: StitchOutcome): boolean {
+ if (outcome.ok) return true;
+ if (outcome.neverConnected === true) return true;
+ const status = outcome.status;
+ if (typeof status !== "number") return false;
+ // Deliberately no 2xx arm -- see the note above. `ok` covers a stitch that
+ // actually returned; a 2xx reaching the catch means the body was unreadable,
+ // which is what a proxy error page looks like.
+ if (status === 501) return true;
+ return status >= 400 && status < 500 && status !== 408;
+}
+
+/**
+ * Which rule refused. Callers branch on this, never on the prose.
+ *
+ * `incomplete`, `lost-parses` and `run-errors` are not this guard's rules --
+ * they are `ix ingest`'s own gates, reported through the same field because a machine
+ * consumer asking "are the cross-repo edges current?" needs the same answer for
+ * them. Between them they are by far the commonest reason a stitch does not
+ * happen (every incremental map, and every run with a failed patch), and
+ * leaving them unreported meant the field said "current" for exactly the cases
+ * it was added to describe.
+ */
+export type StitchRefusal =
+ | "in-flight"
+ | "cooling"
+ | "deadline"
+ | "incomplete"
+ | "lost-parses"
+ | "run-errors";
+
+export type StitchAdmission =
+ | { admitted: true; settle: (outcome: StitchOutcome) => void }
+ | { admitted: false; rule: StitchRefusal; reason: string };
+
+function formatMs(ms: number): string {
+ // Seconds up to two minutes: the numbers that matter here are request
+ // durations near the proxy timeout, and rounding 62s or 90s to whole minutes
+ // loses exactly the detail the reader needs.
+ return ms >= 120_000 ? `${Math.round(ms / 60_000)}m` : `${Math.round(ms / 1000)}s`;
+}
+
+/**
+ * Expiry of a cooldown under the CURRENT setting.
+ *
+ * Re-derived rather than stamped into the record, so that lowering
+ * IX_STITCH_COOLDOWN_MS — or setting it to 0, which the refusal message tells
+ * the user disables this — takes effect on the cooldown they are looking at.
+ * Otherwise the only escape from a 15-minute block is deleting a state file
+ * whose name they cannot compute.
+ */
+function coolingUntil(cooldown: Cooldown): number {
+ return cooldown.at + stitchCooldownMs();
+}
+
+function refuseForCooldown(cooldown: Cooldown, now: number): StitchAdmission {
+ // No elapsed means the attempt never reported back at all -- killed, crashed,
+ // or still running in another process right now. Worth saying differently:
+ // the reader's next question is different in each case.
+ const how =
+ cooldown.elapsedMs === undefined
+ ? "a stitch was started and never reported back"
+ : `the last stitch was cut off after ${formatMs(cooldown.elapsedMs)}`;
+ return {
+ admitted: false,
+ rule: "cooling",
+ // "stitches allowed again in", not "next attempt in". Nothing attempts one
+ // automatically: the stitch block is gated on nothing having been skipped
+ // as unchanged, this run has already persisted its mtime baseline, and the
+ // next map is incremental and never reaches it. Rounds 8, 10 and 12 each
+ // removed that same promise from a remedy string -- and this one matters
+ // more than those did, because it is what lands verbatim in
+ // `stitch_skipped` on the json and llm surfaces, where no remedy follows it.
+ reason:
+ `${how} and may still be running on the backend; ` +
+ `stitches allowed again in ${formatMs(coolingUntil(cooldown) - now)}, ` +
+ `and \`ix ingest <root> --force\` is what re-registers ` +
+ `(IX_STITCH_COOLDOWN_MS=0 disables)`,
+ };
+}
+
+function writeCooldown(endpoint: string, record: Cooldown): void {
+ try {
+ mkdirSync(dirname(cooldownPath(endpoint)), { recursive: true });
+ writeFileSync(cooldownPath(endpoint), JSON.stringify(record), { mode: 0o600 });
+ } catch (err) {
+ // An unwritable lock dir is the one way this guard goes silently inert, so
+ // say so rather than leaving the user to wonder why it never engages. Never
+ // rethrow: settle() runs inside a catch that must unwind with the ORIGINAL
+ // stitch error, not with this one.
+ process.stderr.write(
+ ` Warning: could not record the cross-workspace stitch cooldown (${err}). ` +
+ `The guard against stacking stitch queries (Ix#568) is not active.\n`,
+ );
+ }
+}
+
+/**
+ * Ask permission to POST /v1/stitch against `endpoint`.
+ *
+ * On `{ admitted: true }` the cooldown marker is ALREADY on disk — see the note
+ * at the top of this file. The caller should call `settle` when the attempt
+ * ends; NOT calling it (a crash, a kill, a hook timeout) is a supported outcome
+ * and leaves the marker in place, which is the conservative answer.
+ */
+export function admitStitch(endpoint: string, now = Date.now()): StitchAdmission {
+ // The lock FIRST, and the cooldown only while holding it.
+ //
+ // Order matters, and not for the reason it first appears. Because the marker
+ // is written at the START of an attempt, a marker on disk means either "an
+ // attempt is running right now" or "an attempt ended without proving it
+ // stopped" -- and those want opposite answers: the first should be WAITED
+ // for (a healthy stitch is over in milliseconds, and shedding it loses that
+ // workspace's registration), the second refused outright.
+ //
+ // The lock is exactly that distinction, so it is the thing to ask first. It
+ // also means the cooldown is only ever read while holding the lock, which
+ // removes the stale-read race a read-then-lock order has: no window exists in
+ // which a holder can write a marker between our read and our acquisition.
+ const lock: LockHandle | null = acquireLockAt(namedLockPath("stitch", stitchKey(endpoint)), `ix stitch ${endpoint}`);
+ if (!lock) {
+ return {
+ admitted: false,
+ rule: "in-flight",
+ reason: `another ix run is already stitching ${endpoint}`,
+ };
+ }
+
+ const cooldown = readCooldown(endpoint);
+ if (cooldown !== null && coolingUntil(cooldown) > now) {
+ lock.release();
+ return refuseForCooldown(cooldown, now);
+ }
+
+ // The marker goes down BEFORE the caller sends. Everything after this point
+ // is about REMOVING it, never about deciding whether to write it.
+ writeCooldown(endpoint, { at: now });
+
+ let alreadySettled = false;
+ return {
+ admitted: true,
+ settle: (outcome) => {
+ if (alreadySettled) return;
+ alreadySettled = true;
+ try {
+ if (outcomeProvesNothingRunning(outcome)) {
+ try {
+ rmSync(cooldownPath(endpoint), { force: true });
+ } catch (err) {
+ // NOT silent, unlike the usual best-effort. `force: true` already
+ // swallows ENOENT, so reaching here means the file exists and could
+ // not be removed -- an EPERM or EBUSY, which Windows produces
+ // readily. This is the one failure that fails CLOSED: the
+ // start-time marker survives a stitch that provably finished, and
+ // every map against this endpoint, for every workspace, is refused
+ // for the full cooldown with "a stitch was started and never
+ // reported back". `writeCooldown` warns on the mirror-image
+ // failure, which merely fails open.
+ // The wording follows the PROOF, because this branch runs for all
+ // of them: a success, a 4xx, a 501, and a connection that never
+ // opened. Saying "the stitch succeeded" after an ECONNREFUSED would
+ // be plainly false on the one line the reader has to act on.
+ const what = outcome.ok
+ ? "the cross-workspace stitch succeeded"
+ : "the cross-workspace stitch did not start a query on the backend";
+ process.stderr.write(
+ ` Warning: ${what}, but its cooldown marker could not be removed ` +
+ `(${err}). Further stitches to this backend will be refused until it ` +
+ `ages out; delete ${cooldownPath(endpoint)} to clear it.
+`,
+ );
+ }
+ } else {
+ // Keep the marker, restamped to NOW so the cooldown runs from the
+ // end of the attempt rather than its start -- see `Cooldown.at`. The
+ // elapsed figure is message data only; it decides nothing.
+ writeCooldown(endpoint, { at: Date.now(), elapsedMs: Math.round(outcome.elapsedMs) });
+ }
+ } finally {
+ // Always, even if the state write threw: holding the lock past the
+ // request would block every later stitch until it aged out.
+ lock.release();
+ }
+ },
+ };
+}
+
+/**
+ * [[admitStitch]], but waits out a stitch that is merely in flight.
+ *
+ * Shedding on contention is right when the backend is struggling and wrong
+ * when it is not. On a healthy backend the stitch takes tens of milliseconds,
+ * and two `ix map` runs for two workspaces overlapping by that much is
+ * ordinary in a multi-repo setup with a per-repo hook. Dropping one there
+ * loses that workspace's registration until somebody re-ingests every file,
+ * because the stitch block is gated on nothing having been skipped as
+ * mtime- or hash-unchanged, and an incremental map never reaches it.
+ *
+ * So wait, briefly. A healthy holder is gone long before the budget; an
+ * unhealthy one is still holding when it runs out, which is the case worth
+ * shedding. The cooldown — which is what actually stops the Ix#568 pile-up —
+ * is checked first and never waited on.
+ */
+export async function admitStitchWaiting(
+ endpoint: string,
+ waitMs = stitchWaitMs(),
+ sleep: (ms: number) => Promise<void> = ms => new Promise(r => setTimeout(r, ms)),
+ /**
+ * The run's own wall-clock budget. Waiting past it is pure delay: every
+ * request after it is aborted before it leaves, so the stitch this is
+ * queueing for cannot be sent even if the lock frees.
+ */
+ runDeadline?: { readonly aborted: boolean },
+): Promise<StitchAdmission> {
+ // Refuse outright if the run's budget is ALREADY gone. Admission writes
+ // the cooldown marker, and `fetch` rejects instantly on an aborted signal --
+ // so admitting here marks the backend as maybe-busy for 15 minutes over a
+ // request that never left the process. Worse, it repeats: the stitch is the
+ // last thing an ingest does, so a run that routinely overruns would block
+ // its own next attempt every time and never stitch again.
+ // Read through a call, not a property access: after the early return below,
+ // TypeScript narrows `runDeadline.aborted` to false for the rest of the
+ // function and flags the in-loop check as unreachable -- but the whole point
+ // of that check is that the signal flips WHILE we wait.
+ const deadlineFired = (): boolean => runDeadline?.aborted === true;
+ const refuseForDeadline = (): StitchAdmission => ({
+ admitted: false,
+ rule: "deadline",
+ reason: `the map ran out of time before the stitch could start`,
+ });
+
+
+ const deadline = Date.now() + waitMs;
+ for (;;) {
+ // Re-checked at the TOP of every iteration, before admitStitch. Checking it
+ // only after admission is granted is too late: the holder can release while
+ // we wait, and then we take the lock, write the marker, and `fetch` rejects
+ // instantly on the already-aborted deadline -- marking the backend for 15
+ // minutes over a request that never left. The pre-loop check alone does not
+ // cover it, because the deadline can fire DURING the wait.
+ if (deadlineFired()) return refuseForDeadline();
+
+ const admission = admitStitch(endpoint);
+ // Only contention is worth waiting out. A cooldown means the backend may
+ // still be running the last one, and outlasting THAT is the whole point.
+ if (admission.admitted || admission.rule !== "in-flight") return admission;
+ if (Date.now() >= deadline) return admission;
+ await sleep(Math.min(POLL_MS, Math.max(0, deadline - Date.now())));
+ }
+}
+
+/**
+ * Drop the cooldown for an endpoint, so the next stitch is admitted.
+ *
+ * For `ix reset`, and only for it. A reset wipes the registration on the
+ * backend, and the full re-ingest that follows is the one run that can put it
+ * back -- so a live cooldown refusing exactly that run leaves the workspace
+ * unregistered with no automatic retry, while this file's own comments promise
+ * that "a post-reset re-map re-registers". The reset also removes whatever
+ * graph the outstanding join was building, so there is nothing left to protect.
+ *
+ * Best-effort: a cooldown that cannot be removed costs one skipped stitch, and
+ * failing `ix reset` over it would be worse.
+ */
+export function clearStitchCooldown(endpoint: string): void {
+ try {
+ rmSync(cooldownPath(endpoint), { force: true });
+ } catch { /* best effort */ }
+}
+
+// ── Test-only surface ──────────────────────────────────────────────────────
+export function cooldownPathForTest(endpoint: string): string {
+ return cooldownPath(endpoint);
+}
diff --git a/ix-cli/src/client/api.ts b/ix-cli/src/client/api.ts
index 8261e24c..3a0b7742 100644
--- a/ix-cli/src/client/api.ts
+++ b/ix-cli/src/client/api.ts
@@ -558,7 +558,7 @@ export class IxClient {
const text = await resp.text();
throw new Error(`${resp.status}: ${text}`);
}
- return resp.json() as Promise<T>;
+ return parseOrThrowWithStatus<T>(resp);
}
private async get<T>(path: string): Promise<T> {
@@ -569,6 +569,28 @@ export class IxClient {
const text = await resp.text();
throw new Error(`${resp.status}: ${text}`);
}
- return resp.json() as Promise<T>;
+ return parseOrThrowWithStatus<T>(resp);
+ }
+}
+
+/**
+ * Read a 2xx body as JSON, keeping the status in the error if it will not parse.
+ *
+ * A bare `resp.json()` rejects with a `SyntaxError` that says nothing about
+ * what the server answered, so a caller could only infer "it must have been a
+ * 2xx, because a non-2xx would have thrown above" -- an inference this codebase
+ * has no business making, since the whole premise of Ix#528 and Ix#568 is that
+ * whatever proxy sits in front spells things the way it likes, up to and
+ * including a 200 with an HTML error page. Stating the status turns that
+ * inference into a fact the error carries, and it matches the `${status}: ...`
+ * shape every other failure here already uses.
+ */
+async function parseOrThrowWithStatus<T>(resp: Response): Promise<T> {
+ const text = await resp.text();
+ try {
+ return JSON.parse(text) as T;
+ } catch {
+ const preview = text.length > 200 ? `${text.slice(0, 200)}…` : text;
+ throw new Error(`${resp.status}: response body is not JSON: ${preview}`);
}
}