Understand any codebase instantly. System intelligence for codebases, built for humans and AI.
TypeScript aiai-memoryclaude-codeclicode-analysiscode-mappingcode-memorycodebase-understandingcodexdeveloper-toolsllmopenclawpersistent-memoryprogram-analysissoftware-architecturesystem-intelligencesystem-mapping
// Copyright 2026 Ix Infrastructure Inc.
import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import { acquireLockAt, namedLockPath, type LockHandle } from "./single-flight.js";
// ---------------------------------------------------------------------------
// Admission control for POST /v1/stitch.
//
// Background (Ix#568): the cross-workspace stitch is a single unbounded AQL
// join over `symbol_consumes x provides x consumes x exports`. On a large
// multi-workspace graph it runs for minutes. The HTTP call in front of it gives
// up long before the query does — the reported case is a proxy answering 500 at
// ~60s — and ArangoDB keeps executing the query regardless. The client learning
// "that failed" is therefore NOT the same as the work having stopped.
//
// `ix map` already single-flights per workspace, and that does not help here
// for two independent reasons:
//
// * The stitch is cross-workspace, so two maps of two DIFFERENT workspaces
// hold two different locks and issue two stitches at once. Reproduced: two
// concurrent `ix map` runs put two stitch queries in flight together.
// * Copies mostly do not need concurrency at all. An auto-map hook fires one
// `ix map` per change; each completes, releases its workspace lock, and the
// next one starts a fresh stitch. Reproduced: five sequential maps, five
// stitch queries, every one started after the previous had already answered
// 500 — i.e. while its server-side join was still running.
//
// ── The rule ───────────────────────────────────────────────────────────────
//
// The cooldown marker is written when a stitch STARTS, and removed only on
// proof that nothing is running: a clean answer, or a refusal the backend
// issued without doing the work.
//
// This is deliberately the opposite way round from the obvious design, which
// inspects the FAILURE and writes a cooldown when it looks like a timeout.
// Four review rounds of that classifier each broke the previous round's rule,
// because it has to answer "did the backend start the join?" from an error, and
// the evidence for that keeps not being there:
//
// * the message is whatever proxy sits in front (a 500 here, a 504 with an
// HTML body in #528), so text matching is inert in the deployment it was
// written for;
// * elapsed time cannot tell a slow join from a slow UPLOAD, and the stitch
// payload is megabytes on a large monorepo;
// * "the client aborted" cannot tell a mid-flight hang-up from a run deadline
// that fired before the request was even serialized;
// * and none of it survives the process being killed, which is the ordinary
// end of a hook that timed out.
//
// Writing the marker up front needs none of those answers. Every way a stitch
// can end without a definite refusal — a timeout, an abort, a deadline, SIGTERM,
// a power cut — leaves the marker exactly where it was written, because leaving
// it is the default rather than something the dying process must still do.
//
// The cost is that a stitch which failed for an unclassified reason cools down
// when it need not have. That errs toward skipping one stitch rather than
// stacking joins on a database that is already the reason, which is the safe
// direction here: a skipped stitch leaves the previous registration in place,
// and #568 is a report of 10–16 concurrent copies at 800–1200% CPU.
// ---------------------------------------------------------------------------
const DEFAULT_COOLDOWN_MS = 15 * 60 * 1000;
/** Budget for waiting out a stitch that is merely in flight. */
const DEFAULT_WAIT_MS = 30 * 1000;
/** Poll interval while waiting. Short: a healthy stitch is over in ~30ms. */
const POLL_MS = 250;
function positiveEnvMs(name: string, fallback: number): number {
const raw = process.env[name];
if (raw === undefined || raw === "") return fallback;
// Matched, not parsed. `Number.parseInt` reads any prefix it can, so an
// operator following the env table and writing `15m` for the cooldown got
// FIFTEEN MILLISECONDS -- past the `n >= 0` check, no fallback, nothing
// logged, and `ix map` back to stacking joins with no sign anything is wrong.
// `1e6` gave 1 the same way. `commitFailureLimit` learned this already.
if (!/^\d+$/.test(raw)) return fallback;
const n = Number.parseInt(raw, 10);
// 0 is meaningful (disable), so only a negative or unparseable value falls
// back. `Number.isFinite` alone would let "" and "abc" through as NaN.
return Number.isFinite(n) && n >= 0 ? n : fallback;
}
/** How long to refuse a new stitch after one that may still be running. 0 disables. */
export function stitchCooldownMs(): number {
return positiveEnvMs("IX_STITCH_COOLDOWN_MS", DEFAULT_COOLDOWN_MS);
}
/** How long to wait for the in-flight stitch to finish before shedding. */
export function stitchWaitMs(): number {
return positiveEnvMs("IX_STITCH_WAIT_MS", DEFAULT_WAIT_MS);
}
/**
* One key per backend, however the endpoint was spelled.
*
* `getEndpoint()` returns `IX_ENDPOINT` or the config file verbatim, so the
* same backend reaches this function as `http://localhost:8090`,
* `http://localhost:8090/` and `http://127.0.0.1:8090` depending on which of
* those a given process was started with. Hashing the raw string gives each
* spelling its own lock, and the guard then permits exactly the concurrency it
* exists to prevent — an `ix mcp` server launched with an IP and a shell
* `ix map` reading the config both stitching at once.
*
* Loopback spellings are folded together deliberately. `localhost`,
* `127.0.0.1` and `::1` are not the same host in general — and the endpoint
* is not always local either; `IxClient.isLocalEndpoint` exists precisely
* because a remote backend is supported. The folding is justified by its cost,
* not by a claim about deployments: two loopback spellings that really did
* name different backends cost one deferred stitch, while not folding them
* costs the concurrency this guard exists to prevent.
*/
export function stitchKey(endpoint: string): string {
try {
const u = new URL(endpoint);
const host = ["localhost", "127.0.0.1", "[::1]", "::1"].includes(u.hostname.toLowerCase())
? "localhost"
: u.hostname.toLowerCase();
const path = u.pathname.replace(/\/+$/, "");
// `u.port` is deliberately used raw. The WHATWG parser already strips a
// DEFAULT port, so `https://h` and `https://h:443` both give "" and key
// identically -- adding a defaults table here would be dead code. Verified,
// and pinned by a test, because it looks like an omission.
return `${u.protocol.toLowerCase()}//${host}:${u.port}${path}`;
} catch {
// Not a URL. Nothing to normalise, and refusing to guard is worse than
// guarding on the raw text.
return endpoint;
}
}
/** Cooldown record. Sits beside the lock so IX_LOCK_DIR redirects both. */
function cooldownPath(endpoint: string): string {
return namedLockPath("stitch", stitchKey(endpoint)).replace(/\.lock$/, ".cooldown");
}
interface Cooldown {
/**
* Epoch ms the cooldown runs from.
*
* Written as the attempt's START time, and rewritten to its END time when it
* reports back without proving anything stopped. Both matter:
*
* - the start-time write is what survives a kill, and is the whole reason
* the marker goes down before the request rather than after it;
* - but leaving it AT the start time would make any IX_STITCH_COOLDOWN_MS
* shorter than the attempt itself already expired by the time the attempt
* ends. `IxClient.post` caps a request at two minutes, so every setting
* under that silently bought nothing on the timeout path -- the next map
* was admitted immediately and stacked a second join, which is the bug.
*
* A process that is killed keeps the start stamp, so a cooldown shorter than
* the attempt is still expired there; that residue is noted in docs/api.
*/
at: number;
/** Set when the attempt reported back without clearing. Message only. */
elapsedMs?: number;
}
function readCooldown(endpoint: string): Cooldown | null {
try {
const parsed = JSON.parse(readFileSync(cooldownPath(endpoint), "utf-8")) as Cooldown;
return typeof parsed?.at === "number" ? parsed : null;
} catch {
// Missing, unreadable, or corrupt: no cooldown. Failing open here is right
// — a lost cooldown costs one extra stitch, and refusing to stitch because
// a state file cannot be parsed would break the feature outright.
return null;
}
}
/**
* Did the request fail before the connection was ever established?
*
* This is the one transport fact that is a positive PROOF rather than an
* inference about a failure: if we never connected, no bytes reached the
* backend, so there is no join to wait out. Without it a backend that goes down
* mid-run takes a full 15-minute endpoint-wide cooldown for a request it never
* received.
*
* Keyed on the SYSCALL, not on a list of codes. `IxClient` flattens HTTP errors
* into `"NNN: ..."` strings but rethrows transport failures intact, so what
* arrives is Node's `TypeError: fetch failed` with the real error on `.cause` --
* and Node stamps `syscall` on every libuv error, which says which phase failed
* far more reliably than the errno does. Observed through `IxClient.stitch`:
*
* closed port code ECONNREFUSED syscall connect
* unresolvable DNS code ENOTFOUND syscall getaddrinfo
* unroutable address code ENETUNREACH syscall connect
* blackholed address code UND_ERR_CONNECT_TIMEOUT syscall undefined
* reset AFTER send code UND_ERR_SOCKET syscall undefined
*
* An enumerated code list got the first two and missed the next two, which is
* the same harm for a neighbouring errno: a VPN drop or an unreachable host
* cooling the endpoint down for fifteen minutes over a request that never left.
* `connect` and `getaddrinfo` cover every one of those without naming them, and
* they extend to `EHOSTUNREACH` and a connect-phase `ETIMEDOUT` for free --
* while a `read`/`write` `ETIMEDOUT`, which happens after the bytes are gone,
* correctly does not qualify.
*
* `UND_ERR_CONNECT_TIMEOUT` is undici's own and carries no syscall, so it is
* named. `UND_ERR_SOCKET` is the deliberate exclusion: "other side closed"
* happens after the request went out, so it is exactly the ambiguous case this
* guard exists for -- an upstream that restarted killed its join, a proxy that
* dropped the connection did not, and both spell the same code.
*
* The walk descends into `AggregateError.errors` as well as `.cause`, and the
* errno set is kept alongside the syscall rule rather than replaced by it. That
* is not belt and braces: a multi-address host -- which `localhost` is, and it
* is this CLI's DEFAULT endpoint -- fails happy-eyeballs with
*
* AggregateError { code: "ECONNREFUSED", syscall: undefined, cause: undefined,
* errors: [ {ECONNREFUSED, connect}, {ECONNREFUSED, connect} ] }
*
* so a syscall-only rule that walks only `.cause` answers false for the single
* most common way this fires. Verified by execution: `http://localhost:8099`
* takes that shape, `http://127.0.0.1:8099` and `http://[::1]:8099` do not.
* Tests that used a literal IP could not see the difference.
*/
export function connectionNeverEstablished(error: unknown): boolean {
// Codes that only ever arise before any byte is sent. `ETIMEDOUT` is
// deliberately absent -- it is a connect timeout OR a retransmission timeout
// long after the request left -- and is picked up by the syscall rule when it
// is the former.
const PRE_CONNECT = new Set([
"ECONNREFUSED",
"ENOTFOUND",
"ENETUNREACH",
"EHOSTUNREACH",
"EAI_AGAIN",
"UND_ERR_CONNECT_TIMEOUT",
]);
// Depth-capped rather than cycle-tracked: undici nests two or three levels and
// an AggregateError adds one, so the cap is what stops a pathological chain.
const seen = (e: unknown, depth: number): boolean => {
if (e === null || e === undefined || depth > 5) return false;
const { syscall, code, cause, errors } = e as {
syscall?: unknown; code?: unknown; cause?: unknown; errors?: unknown;
};
if (syscall === "connect" || syscall === "getaddrinfo") return true;
if (typeof code === "string" && PRE_CONNECT.has(code)) return true;
if (Array.isArray(errors) && errors.some(inner => seen(inner, depth + 1))) return true;
return seen(cause, depth + 1);
};
return seen(error, 0);
}
/** How the stitch attempt ended, as the guard needs to see it. */
export interface StitchOutcome {
ok: boolean;
/** Wall-clock the client spent. Message only — never part of the decision. */
elapsedMs: number;
/** HTTP status the backend answered with, when it answered at all. */
status?: number | null;
/** Set when the connection was never established — see the function above. */
neverConnected?: boolean;
}
/**
* Did this outcome PROVE that no join is running?
*
* Every proof here is a positive fact rather than an inference drawn from the
* shape of a failure:
*
* - the stitch succeeded, so the backend is done with it;
* - the backend answered 4xx, which is it refusing the request rather than
* executing it. 408 is excluded: a proxy reporting that IT gave up waiting
* says nothing about whether the backend did, and that is this whole bug.
* - the backend answered 501. `isStitchUnsupported` already treats 404 AND
* 501 as "this backend has no /v1/stitch", and the run swallows it
* silently for exactly that reason -- so leaving a marker behind would
* refuse every map for 15 minutes, claiming a join may still be running,
* against a backend that has never had one.
*
* - the connection was never established, so nothing was sent. See
* `connectionNeverEstablished` for why that is narrower than "a transport
* error" and why the narrowing matters;
* A 2xx whose body would not parse is deliberately NOT on that list, though
* two earlier revisions put it there. The argument for it was that the request
* completed, so the join did -- and the argument against is stronger: a proxy
* answering 200 with its own HTML error page when the upstream stalls produces
* exactly that shape, and it is the #528 pattern. Clearing the marker there is
* the one way any proof arm can cause the #568 failure itself, by admitting the
* next map onto a join that is still running.
*
* The symmetry argument does not hold either: a proxy answering 200 with
* PARSEABLE JSON is `ok: true` and has always cleared the marker, but a proxy
* error page is HTML, so the unparseable case is the LIKELY one rather than the
* equivalent one. The cost of excluding it is a cooldown after a stitch that
* really did succeed and got garbled in transit, which is the direction this
* whole file errs in by design.
*
* Anything else — another 5xx, a timeout, an abort, a socket dropped after the
* request went out, or the process being killed before it could say anything —
* leaves the marker.
*/
export function outcomeProvesNothingRunning(outcome: StitchOutcome): boolean {
if (outcome.ok) return true;
if (outcome.neverConnected === true) return true;
const status = outcome.status;
if (typeof status !== "number") return false;
// Deliberately no 2xx arm -- see the note above. `ok` covers a stitch that
// actually returned; a 2xx reaching the catch means the body was unreadable,
// which is what a proxy error page looks like.
if (status === 501) return true;
return status >= 400 && status < 500 && status !== 408;
}
/**
* Which rule refused. Callers branch on this, never on the prose.
*
* `incomplete`, `lost-parses` and `run-errors` are not this guard's rules --
* they are `ix ingest`'s own gates, reported through the same field because a machine
* consumer asking "are the cross-repo edges current?" needs the same answer for
* them. Between them they are by far the commonest reason a stitch does not
* happen (every incremental map, and every run with a failed patch), and
* leaving them unreported meant the field said "current" for exactly the cases
* it was added to describe.
*/
export type StitchRefusal =
| "in-flight"
| "cooling"
| "deadline"
| "incomplete"
| "lost-parses"
| "run-errors";
export type StitchAdmission =
| { admitted: true; settle: (outcome: StitchOutcome) => void }
| { admitted: false; rule: StitchRefusal; reason: string };
function formatMs(ms: number): string {
// Seconds up to two minutes: the numbers that matter here are request
// durations near the proxy timeout, and rounding 62s or 90s to whole minutes
// loses exactly the detail the reader needs.
return ms >= 120_000 ? `${Math.round(ms / 60_000)}m` : `${Math.round(ms / 1000)}s`;
}
/**
* Expiry of a cooldown under the CURRENT setting.
*
* Re-derived rather than stamped into the record, so that lowering
* IX_STITCH_COOLDOWN_MS — or setting it to 0, which the refusal message tells
* the user disables this — takes effect on the cooldown they are looking at.
* Otherwise the only escape from a 15-minute block is deleting a state file
* whose name they cannot compute.
*/
function coolingUntil(cooldown: Cooldown): number {
return cooldown.at + stitchCooldownMs();
}
function refuseForCooldown(cooldown: Cooldown, now: number): StitchAdmission {
// No elapsed means the attempt never reported back at all -- killed, crashed,
// or still running in another process right now. Worth saying differently:
// the reader's next question is different in each case.
const how =
cooldown.elapsedMs === undefined
? "a stitch was started and never reported back"
: `the last stitch was cut off after ${formatMs(cooldown.elapsedMs)}`;
return {
admitted: false,
rule: "cooling",
// "stitches allowed again in", not "next attempt in". Nothing attempts one
// automatically: the stitch block is gated on nothing having been skipped
// as unchanged, this run has already persisted its mtime baseline, and the
// next map is incremental and never reaches it. Rounds 8, 10 and 12 each
// removed that same promise from a remedy string -- and this one matters
// more than those did, because it is what lands verbatim in
// `stitch_skipped` on the json and llm surfaces, where no remedy follows it.
reason:
`${how} and may still be running on the backend; ` +
`stitches allowed again in ${formatMs(coolingUntil(cooldown) - now)}, ` +
`and \`ix ingest <root> --force\` is what re-registers ` +
`(IX_STITCH_COOLDOWN_MS=0 disables)`,
};
}
function writeCooldown(endpoint: string, record: Cooldown): void {
try {
mkdirSync(dirname(cooldownPath(endpoint)), { recursive: true });
writeFileSync(cooldownPath(endpoint), JSON.stringify(record), { mode: 0o600 });
} catch (err) {
// An unwritable lock dir is the one way this guard goes silently inert, so
// say so rather than leaving the user to wonder why it never engages. Never
// rethrow: settle() runs inside a catch that must unwind with the ORIGINAL
// stitch error, not with this one.
process.stderr.write(
` Warning: could not record the cross-workspace stitch cooldown (${err}). ` +
`The guard against stacking stitch queries (Ix#568) is not active.\n`,
);
}
}
/**
* Ask permission to POST /v1/stitch against `endpoint`.
*
* On `{ admitted: true }` the cooldown marker is ALREADY on disk — see the note
* at the top of this file. The caller should call `settle` when the attempt
* ends; NOT calling it (a crash, a kill, a hook timeout) is a supported outcome
* and leaves the marker in place, which is the conservative answer.
*/
export function admitStitch(endpoint: string, now = Date.now()): StitchAdmission {
// The lock FIRST, and the cooldown only while holding it.
//
// Order matters, and not for the reason it first appears. Because the marker
// is written at the START of an attempt, a marker on disk means either "an
// attempt is running right now" or "an attempt ended without proving it
// stopped" -- and those want opposite answers: the first should be WAITED
// for (a healthy stitch is over in milliseconds, and shedding it loses that
// workspace's registration), the second refused outright.
//
// The lock is exactly that distinction, so it is the thing to ask first. It
// also means the cooldown is only ever read while holding the lock, which
// removes the stale-read race a read-then-lock order has: no window exists in
// which a holder can write a marker between our read and our acquisition.
const lock: LockHandle | null = acquireLockAt(namedLockPath("stitch", stitchKey(endpoint)), `ix stitch ${endpoint}`);
if (!lock) {
return {
admitted: false,
rule: "in-flight",
reason: `another ix run is already stitching ${endpoint}`,
};
}
const cooldown = readCooldown(endpoint);
if (cooldown !== null && coolingUntil(cooldown) > now) {
lock.release();
return refuseForCooldown(cooldown, now);
}
// The marker goes down BEFORE the caller sends. Everything after this point
// is about REMOVING it, never about deciding whether to write it.
writeCooldown(endpoint, { at: now });
let alreadySettled = false;
return {
admitted: true,
settle: (outcome) => {
if (alreadySettled) return;
alreadySettled = true;
try {
if (outcomeProvesNothingRunning(outcome)) {
try {
rmSync(cooldownPath(endpoint), { force: true });
} catch (err) {
// NOT silent, unlike the usual best-effort. `force: true` already
// swallows ENOENT, so reaching here means the file exists and could
// not be removed -- an EPERM or EBUSY, which Windows produces
// readily. This is the one failure that fails CLOSED: the
// start-time marker survives a stitch that provably finished, and
// every map against this endpoint, for every workspace, is refused
// for the full cooldown with "a stitch was started and never
// reported back". `writeCooldown` warns on the mirror-image
// failure, which merely fails open.
// The wording follows the PROOF, because this branch runs for all
// of them: a success, a 4xx, a 501, and a connection that never
// opened. Saying "the stitch succeeded" after an ECONNREFUSED would
// be plainly false on the one line the reader has to act on.
const what = outcome.ok
? "the cross-workspace stitch succeeded"
: "the cross-workspace stitch did not start a query on the backend";
process.stderr.write(
` Warning: ${what}, but its cooldown marker could not be removed ` +
`(${err}). Further stitches to this backend will be refused until it ` +
`ages out; delete ${cooldownPath(endpoint)} to clear it.
`,
);
}
} else {
// Keep the marker, restamped to NOW so the cooldown runs from the
// end of the attempt rather than its start -- see `Cooldown.at`. The
// elapsed figure is message data only; it decides nothing.
writeCooldown(endpoint, { at: Date.now(), elapsedMs: Math.round(outcome.elapsedMs) });
}
} finally {
// Always, even if the state write threw: holding the lock past the
// request would block every later stitch until it aged out.
lock.release();
}
},
};
}
/**
* [[admitStitch]], but waits out a stitch that is merely in flight.
*
* Shedding on contention is right when the backend is struggling and wrong
* when it is not. On a healthy backend the stitch takes tens of milliseconds,
* and two `ix map` runs for two workspaces overlapping by that much is
* ordinary in a multi-repo setup with a per-repo hook. Dropping one there
* loses that workspace's registration until somebody re-ingests every file,
* because the stitch block is gated on nothing having been skipped as
* mtime- or hash-unchanged, and an incremental map never reaches it.
*
* So wait, briefly. A healthy holder is gone long before the budget; an
* unhealthy one is still holding when it runs out, which is the case worth
* shedding. The cooldown — which is what actually stops the Ix#568 pile-up —
* is checked first and never waited on.
*/
export async function admitStitchWaiting(
endpoint: string,
waitMs = stitchWaitMs(),
sleep: (ms: number) => Promise<void> = ms => new Promise(r => setTimeout(r, ms)),
/**
* The run's own wall-clock budget. Waiting past it is pure delay: every
* request after it is aborted before it leaves, so the stitch this is
* queueing for cannot be sent even if the lock frees.
*/
runDeadline?: { readonly aborted: boolean },
): Promise<StitchAdmission> {
// Refuse outright if the run's budget is ALREADY gone. Admission writes
// the cooldown marker, and `fetch` rejects instantly on an aborted signal --
// so admitting here marks the backend as maybe-busy for 15 minutes over a
// request that never left the process. Worse, it repeats: the stitch is the
// last thing an ingest does, so a run that routinely overruns would block
// its own next attempt every time and never stitch again.
// Read through a call, not a property access: after the early return below,
// TypeScript narrows `runDeadline.aborted` to false for the rest of the
// function and flags the in-loop check as unreachable -- but the whole point
// of that check is that the signal flips WHILE we wait.
const deadlineFired = (): boolean => runDeadline?.aborted === true;
const refuseForDeadline = (): StitchAdmission => ({
admitted: false,
rule: "deadline",
reason: `the map ran out of time before the stitch could start`,
});
const deadline = Date.now() + waitMs;
for (;;) {
// Re-checked at the TOP of every iteration, before admitStitch. Checking it
// only after admission is granted is too late: the holder can release while
// we wait, and then we take the lock, write the marker, and `fetch` rejects
// instantly on the already-aborted deadline -- marking the backend for 15
// minutes over a request that never left. The pre-loop check alone does not
// cover it, because the deadline can fire DURING the wait.
if (deadlineFired()) return refuseForDeadline();
const admission = admitStitch(endpoint);
// Only contention is worth waiting out. A cooldown means the backend may
// still be running the last one, and outlasting THAT is the whole point.
if (admission.admitted || admission.rule !== "in-flight") return admission;
if (Date.now() >= deadline) return admission;
await sleep(Math.min(POLL_MS, Math.max(0, deadline - Date.now())));
}
}
/**
* Drop the cooldown for an endpoint, so the next stitch is admitted.
*
* For `ix reset`, and only for it. A reset wipes the registration on the
* backend, and the full re-ingest that follows is the one run that can put it
* back -- so a live cooldown refusing exactly that run leaves the workspace
* unregistered with no automatic retry, while this file's own comments promise
* that "a post-reset re-map re-registers". The reset also removes whatever
* graph the outstanding join was building, so there is nothing left to protect.
*
* Best-effort: a cooldown that cannot be removed costs one skipped stitch, and
* failing `ix reset` over it would be worse.
*/
export function clearStitchCooldown(endpoint: string): void {
try {
rmSync(cooldownPath(endpoint), { force: true });
} catch { /* best effort */ }
}
// ── Test-only surface ──────────────────────────────────────────────────────
export function cooldownPathForTest(endpoint: string): string {
return cooldownPath(endpoint);
}