Skip to content

armgit / Ix

Public Imported from GitHub
0

Understand any codebase instantly. System intelligence for codebases, built for humans and AI.

fix(ingest): retry transient 5xx/429 bulk commits with Retry-After, and the CL-17 breaker fixes (IN-19C) (#845)

Ian Hockd4f9e4fd9fdd
diff --git a/docs-site/src/content/docs/guides/troubleshooting.md b/docs-site/src/content/docs/guides/troubleshooting.md
index 0bdae529..8556f616 100644
--- a/docs-site/src/content/docs/guides/troubleshooting.md
+++ b/docs-site/src/content/docs/guides/troubleshooting.md
@@ -31,8 +31,11 @@ Set `IX_DEBUG=1` to see full stack traces on any error.
## `ix map` stops with "Commits against … kept failing"
-After several consecutive failed commits, `ix map` stops instead of sending one doomed request per file. Read the
-`Last error` line it prints. The backend returns the same error for two different causes:
+After several consecutive failed commits, `ix map` stops instead of sending one doomed request per file. It stops
+after the first failure if the backend's health check reports that it cannot reach its database (ArangoDB); the message
+then says so, and `ix doctor` shows the same. Start the database, or wait for it to come back, and re-map.
+
+Otherwise, read the `Last error` line it prints. The backend returns the same error for two different causes:
- **The database is saturated.** `ix doctor` can still pass. Check `docker stats`, wait for ArangoDB to go idle, and
re-map.
diff --git a/docs-site/src/content/docs/reference/configuration.md b/docs-site/src/content/docs/reference/configuration.md
index 71247fba..a28691ac 100644
--- a/docs-site/src/content/docs/reference/configuration.md
+++ b/docs-site/src/content/docs/reference/configuration.md
@@ -21,9 +21,11 @@ ix config set format llm # set a value
| `IX_FORMAT` | Default output format: `text`, `json` or `llm`. Overrides the config file. `--format` overrides it |
| `IX_TOKEN=<token>` | Bearer token sent to a local backend (localhost or another loopback address) on every request. Overrides the stored local token (see [Local token](#local-token)). Never sent to any other endpoint |
| `IX_DEBUG=1` | Prints full stack traces on errors, and one stderr line per backend request with the `X-Correlation-Id` it sent (the backend tags its request and slow-query logs with it). Every request of one `ix` run shares the id's prefix |
-| `IX_READ_DEADLINE_MS=<ms>` | How long one command's reads may take in all before it fails (default 60000), so a hung backend fails a query in a minute rather than after each request's 2-minute timeout. The clock restarts after each write the command makes. `ix map`, `ix ingest` and `ix stats` have no read deadline. `0` turns it off. A read that meets a refused or dropped connection, or a 502/503/504, is retried once; writes never are |
+| `IX_READ_DEADLINE_MS=<ms>` | How long one command's reads may take in all before it fails (default 60000), so a hung backend fails a query in a minute rather than after each request's 2-minute timeout. The clock restarts after each write the command makes. `ix map`, `ix ingest` and `ix stats` have no read deadline. `0` turns it off. A read that meets a refused or dropped connection, or a 502/503/504, is retried once; writes are not, except as `IX_COMMIT_CONFLICT_RETRIES` and `IX_COMMIT_TRANSIENT_RETRY_MS` describe |
| `IX_MCP_SUBPROCESS=1` | Runs each MCP tool call in its own `ix` process |
-| `IX_COMMIT_FAILURE_LIMIT=<n>` | Consecutive failed commits before `ix map` stops (default 5). `0` never stops |
+| `IX_COMMIT_FAILURE_LIMIT=<n>` | Consecutive failed commits before `ix map` stops (default 5). It stops after one when the backend reports its database unreachable. `0` never stops |
+| `IX_COMMIT_CONFLICT_RETRIES=<n>` | Retries of one commit after a lock conflict or a dropped connection (default 6). `0` turns these retries off. It does not affect the 502/503/504/429 retries; see `IX_COMMIT_TRANSIENT_RETRY_MS` |
+| `IX_COMMIT_TRANSIENT_RETRY_MS=<ms>` | The total time one bulk commit may spend retrying after a 502, 503, 504 or 429, counting both the requests and the waits between them (default 5000, at most three retries, and at most one after a 504). A bulk is sent again only if another request as long as the one that failed still fits, so a slow bulk goes straight to the per-file fallback. `Retry-After` is honoured up to that limit; a longer value is cut to it. `0` turns these retries off |
| `IX_COMMIT_BASE_REV_RETRIES=<n>` | Re-sends of a commit after another writer on the same backend moved the graph revision under it (default 8) |
| `IX_PARSE_BUDGET_MS=<ms>` | How long `ix map` / `ix ingest` may spend parsing one file before skipping it and naming it in the summary (default 10000). A skipped file is tried again on every run, and `ix status` warns about it until it is in the graph. `0` turns the budget off |
| `IX_PARSE_WORKERS=<n>` | Most parse workers `ix map` / `ix ingest` starts (default 8, never more than the cores less one). Workers start as files need them, one per 50 files |
diff --git a/docs/api/openapi.yaml b/docs/api/openapi.yaml
index dafa21e7..7eb22513 100644
--- a/docs/api/openapi.yaml
+++ b/docs/api/openapi.yaml
@@ -47,6 +47,12 @@ paths:
application/json:
schema:
$ref: "#/components/schemas/HealthResponse"
+ "503":
+ description: The backend is up but cannot reach its database (status degraded, database unreachable)
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/HealthResponse"
/v1/capabilities:
get:
@@ -1057,6 +1063,10 @@ components:
status: { type: string }
schema_version: { type: integer, description: On-disk graph format version }
release_version: { type: string, description: Backend release semver }
+ database:
+ type: string
+ enum: [reachable, unreachable]
+ description: Whether the backend can reach its database (1.0.33+). Absent on older backends
CapabilitiesResponse:
type: object
properties:
diff --git a/ix-cli/src/cli/__tests__/backend-release-report.test.ts b/ix-cli/src/cli/__tests__/backend-release-report.test.ts
index 07b2ee4e..405b6149 100644
--- a/ix-cli/src/cli/__tests__/backend-release-report.test.ts
+++ b/ix-cli/src/cli/__tests__/backend-release-report.test.ts
@@ -280,6 +280,21 @@ describe("fetchBackendHealth records what it read", () => {
expect(tracked()).toBe("1.0.16");
});
+ it("passes a probe's timeout through readBackendHealth, and still records", async () => {
+ // The ingest's database probe needs a short timeout. It reaches health
+ // through here rather than calling client.health(ms) itself.
+ writeFileSync(stamp(), "1.0.13");
+ writeFileSync(
+ join(home, ".version-check.json"),
+ JSON.stringify({ latest: "0.9.3", backendLatest: "1.0.16", checkedAt: Date.now() }),
+ );
+ const health = vi.fn(async (_timeoutMs?: number) => ({ status: "ok", release_version: "1.0.16" }));
+ const { readBackendHealth } = await import("../commands/upgrade.js");
+ await readBackendHealth({ endpoint: "http://localhost:8090", health } as unknown as IxClient, 1234);
+ expect(health).toHaveBeenCalledWith(1234);
+ expect(tracked()).toBe("1.0.16");
+ });
+
it("applies the ceiling through readBackendHealth, the path commands take", async () => {
// The JOIN, not the pieces. backendCeiling() is pinned on its own and
// fetchBackendHealth's use of knownLatest is pinned on its own — but
@@ -492,7 +507,9 @@ describe("health is fetched in exactly one place", () => {
// comments are left alone for the same reason: a `**/*.ts` glob literal ate
// from the string to the next `*/` and dropped the chokepoint itself.
.map((f) => ({ ...f, src: f.src.replace(/^\s*(\/\/|\*|\/\*).*$/gm, "") }))
- .filter((f) => /\.health\s*\(\s*\)/.test(f.src))
+ // Any arguments: `client.health(timeoutMs)` is as much a health read as
+ // `client.health()`, and an empty-parens-only pattern let one slip past.
+ .filter((f) => /\.health\s*\(/.test(f.src))
.map((f) => f.file.slice(SRC.length + 1).replace(/\\/g, "/"));
it("finds the health callers at all", () => {
diff --git a/ix-cli/src/cli/__tests__/commit-breaker.test.ts b/ix-cli/src/cli/__tests__/commit-breaker.test.ts
index b441104e..9cb10163 100644
--- a/ix-cli/src/cli/__tests__/commit-breaker.test.ts
+++ b/ix-cli/src/cli/__tests__/commit-breaker.test.ts
@@ -124,6 +124,26 @@ describe("createCommitBreaker", () => {
});
});
+describe("trip, which the database probe uses (IN-19C)", () => {
+ it("latches tripped and everTripped at once, and records the error", () => {
+ const b = createCommitBreaker(5);
+ const err = new Error("500: Connection refused: arangodb:8529");
+ b.trip(err);
+ expect(b.tripped()).toBe(true);
+ expect(b.everTripped()).toBe(true);
+ expect(b.lastError()).toBe(err);
+ expect(b.consecutiveFailures(), "no streak was spent").toBe(0);
+ });
+
+ it("is cleared by a success like any trip, and everTripped stays", () => {
+ const b = createCommitBreaker(5);
+ b.trip(new Error("x"));
+ b.recordSuccess();
+ expect(b.tripped()).toBe(false);
+ expect(b.everTripped()).toBe(true);
+ });
+});
+
describe("everTripped", () => {
it("is false until the limit is reached", () => {
const b = createCommitBreaker(3);
@@ -596,6 +616,35 @@ describe("describeCommitCutoff", () => {
expect(describeCommitCutoff(b, "e", 0, false)).toContain("400 patches were not sent");
});
+ it("says the graph is still empty, not unchanged, when it held none of these files (IN-19C)", () => {
+ // After `ix reset` or `reset --ingest` there is no graph to be unchanged,
+ // and "The graph is unchanged" reads as "your old graph is still there".
+ const b = createCommitBreaker(1);
+ b.recordFailure(new Error("500: transaction begin timeout"));
+ b.recordSkipped(3);
+
+ const empty = describeCommitCutoff(b, "e", 0, false, { graphStartedEmpty: true });
+ expect(empty).not.toContain("unchanged");
+ expect(empty).toContain("this workspace's graph is still empty");
+ // Only with nothing landed: a partial run is described as before.
+ const partial = describeCommitCutoff(b, "e", 4, false, { graphStartedEmpty: true });
+ expect(partial).toContain("4 other patches did land");
+ // And the defaults keep the old text.
+ expect(describeCommitCutoff(b, "e", 0)).toContain("The graph is unchanged.");
+ expect(describeCommitCutoff(b, "e", 0, false, {})).toContain("The graph is unchanged.");
+ });
+
+ it("names the database, and drops the `ix doctor` still passes line, when the backend reported it down", () => {
+ const b = createCommitBreaker(1);
+ b.trip(new Error("500: Connection refused: arangodb:8529"));
+
+ const msg = describeCommitCutoff(b, "e", 0, false, { databaseUnreachable: true });
+ expect(msg).toContain("cannot reach its database (ArangoDB)");
+ expect(msg).not.toContain("still passes");
+ expect(msg).not.toContain("Either the database cannot keep up");
+ expect(describeCommitCutoff(b, "e", 0)).toContain("`ix doctor` still passes");
+ });
+
it("says 'patch was' for one and 'patches were' for many", () => {
const one = createCommitBreaker(1);
one.recordFailure(new Error("x"));
@@ -614,3 +663,12 @@ describe("describeCommitCutoff", () => {
expect(msg).toContain("…");
});
});
+
+describe("trip() with the breaker disabled", () => {
+ it("does not latch when the limit is 0", () => {
+ const b = createCommitBreaker(0);
+ b.trip(new Error("db down"));
+ expect(b.tripped()).toBe(false);
+ expect(b.everTripped()).toBe(false);
+ });
+});
diff --git a/ix-cli/src/cli/__tests__/commit-retry.test.ts b/ix-cli/src/cli/__tests__/commit-retry.test.ts
new file mode 100644
index 00000000..7971f748
--- /dev/null
+++ b/ix-cli/src/cli/__tests__/commit-retry.test.ts
@@ -0,0 +1,372 @@
+// Copyright 2026 Ix Infrastructure Inc.
+
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { GATEWAY_TIMEOUT_MAX_RETRIES, isTransientCommitStatus, retryAfterMs, retryCommit, TRANSIENT_RETRY_BUDGET_MS } from "../commit-retry.js";
+import { httpError } from "../../client/api.js";
+
+/** An error shaped the way `httpError` builds one for a commit route. */
+function statusError(status: number, retryAfter?: string): Error {
+ const headers = new Headers(retryAfter !== undefined ? { "retry-after": retryAfter } : {});
+ return httpError(status, JSON.stringify({ error: `${status}: nope` }), headers);
+}
+
+/** `fn` that fails with each of `failures` in turn, then answers "ok". */
+function failing(...failures: unknown[]): { fn: () => Promise<string>; calls: () => number } {
+ let n = 0;
+ return {
+ fn: async () => {
+ const failure = failures[n++];
+ if (failure !== undefined) throw failure;
+ return "ok";
+ },
+ calls: () => n,
+ };
+}
+
+describe("isTransientCommitStatus", () => {
+ it("is true for 502, 503, 504 and 429, from the status or the message", () => {
+ for (const status of [502, 503, 504, 429]) {
+ expect(isTransientCommitStatus(statusError(status)), String(status)).toBe(true);
+ expect(isTransientCommitStatus(new Error(`${status}: body`)), `${status} message`).toBe(true);
+ }
+ });
+
+ it("is false for 500, 413, 409, 401 and anything without a status", () => {
+ for (const status of [500, 413, 409, 401, 400]) {
+ expect(isTransientCommitStatus(statusError(status)), String(status)).toBe(false);
+ }
+ expect(isTransientCommitStatus(new Error("fetch failed"))).toBe(false);
+ expect(isTransientCommitStatus("503")).toBe(false);
+ });
+
+ it("is never true for an abort", () => {
+ const abort = new Error("503: upstream aborted");
+ expect(isTransientCommitStatus(abort)).toBe(false);
+ expect(isTransientCommitStatus(new DOMException("timed out", "TimeoutError"))).toBe(false);
+ });
+});
+
+describe("retryAfterMs", () => {
+ const now = Date.parse("Sun, 11 Oct 2026 12:00:00 GMT");
+
+ it("reads delta-seconds", () => {
+ expect(retryAfterMs("2", now)).toBe(2000);
+ expect(retryAfterMs(" 0 ", now)).toBe(0);
+ });
+
+ it("reads an IMF-fixdate, and clamps a past one to 0", () => {
+ expect(retryAfterMs("Sun, 11 Oct 2026 12:00:03 GMT", now)).toBe(3000);
+ expect(retryAfterMs("Sun, 11 Oct 2026 11:59:00 GMT", now)).toBe(0);
+ });
+
+ it("is undefined for absent or unreadable values", () => {
+ expect(retryAfterMs(undefined, now)).toBeUndefined();
+ expect(retryAfterMs("", now)).toBeUndefined();
+ expect(retryAfterMs("soon", now)).toBeUndefined();
+ expect(retryAfterMs("-1", now)).toBeUndefined();
+ });
+});
+
+describe("retryCommit", () => {
+ beforeEach(() => { vi.useFakeTimers(); });
+ afterEach(() => { vi.useRealTimers(); });
+
+ const transient = { maxRetries: 6, transient: true, random: () => 0 } as const;
+
+ it("a 503 then a 200 succeeds after exactly one retry, once the backoff has passed", async () => {
+ const { fn, calls } = failing(statusError(503));
+ let settled = false;
+ const result = retryCommit(fn, transient).then(v => { settled = true; return v; });
+
+ await vi.advanceTimersByTimeAsync(499);
+ expect(calls()).toBe(1);
+ expect(settled).toBe(false);
+ await vi.advanceTimersByTimeAsync(1);
+ await expect(result).resolves.toBe("ok");
+ expect(calls()).toBe(2);
+ });
+
+ it("a 429 with Retry-After: 2 waits 2 s, not the backoff", async () => {
+ const { fn, calls } = failing(statusError(429, "2"));
+ const result = retryCommit(fn, transient);
+
+ await vi.advanceTimersByTimeAsync(1999);
+ expect(calls(), "no second call before 2 s").toBe(1);
+ await vi.advanceTimersByTimeAsync(1);
+ await expect(result).resolves.toBe("ok");
+ expect(calls()).toBe(2);
+ });
+
+ it("a Retry-After HTTP-date waits until that date, and a past one retries at once", async () => {
+ vi.setSystemTime(Date.parse("Sun, 11 Oct 2026 12:00:00 GMT"));
+ const ahead = failing(statusError(503, "Sun, 11 Oct 2026 12:00:03 GMT"));
+ const first = retryCommit(ahead.fn, transient);
+ await vi.advanceTimersByTimeAsync(2999);
+ expect(ahead.calls()).toBe(1);
+ await vi.advanceTimersByTimeAsync(1);
+ await expect(first).resolves.toBe("ok");
+
+ const past = failing(statusError(503, "Sun, 11 Oct 2026 11:00:00 GMT"));
+ const second = retryCommit(past.fn, transient);
+ await vi.advanceTimersByTimeAsync(0);
+ await expect(second).resolves.toBe("ok");
+ expect(past.calls()).toBe(2);
+ });
+
+ it("caps transient retries at three, inside about 5 s by default", async () => {
+ const waits: number[] = [];
+ const always503 = vi.fn(async () => { throw statusError(503); });
+ const result = retryCommit(always503, {
+ maxRetries: 100,
+ transient: true,
+ random: () => 0,
+ sleep: async ms => { waits.push(ms); },
+ });
+
+ await expect(result).rejects.toThrow(/^503:/);
+ // 500 + 1000 + 2000 = 3.5 s, under the 5 s default; the cap of three stops it.
+ expect(waits).toEqual([500, 1000, 2000]);
+ expect(always503).toHaveBeenCalledTimes(4);
+ });
+
+ it("cuts the last backoff to what is left of a smaller budget", async () => {
+ const waits: number[] = [];
+ const always503 = vi.fn(async () => { throw statusError(503); });
+ await expect(retryCommit(always503, {
+ maxRetries: 6,
+ transient: true,
+ transientBudgetMs: 2_000,
+ random: () => 0,
+ sleep: async ms => { waits.push(ms); },
+ })).rejects.toThrow(/^503:/);
+ expect(waits).toEqual([500, 1000, 500]);
+ expect(waits.reduce((a, b) => a + b, 0)).toBe(2_000);
+ });
+
+ it("honours a Retry-After only up to the budget, and never waits past it", async () => {
+ const waits: number[] = [];
+ const { fn, calls } = failing(statusError(503, "60"), statusError(503, "60"));
+ await expect(retryCommit(fn, { ...transient, sleep: async ms => { waits.push(ms); } })).rejects.toThrow(/^503:/);
+ expect(waits, "60 s cut to the 5 s budget, then nothing left").toEqual([5_000]);
+ expect(calls()).toBe(2);
+ });
+
+ it("a Retry-After of 0 still stops after three retries", async () => {
+ const always = vi.fn(async () => { throw statusError(429, "0"); });
+ await expect(retryCommit(always, { ...transient, sleep: async () => {} })).rejects.toThrow(/^429:/);
+ expect(always).toHaveBeenCalledTimes(4);
+ });
+
+ it("transientBudgetMs 0 makes exactly one attempt for a 503", async () => {
+ const sleep = vi.fn(async () => {});
+ const { fn, calls } = failing(statusError(503));
+ await expect(retryCommit(fn, { ...transient, transientBudgetMs: 0, sleep })).rejects.toThrow(/^503:/);
+ expect(sleep).not.toHaveBeenCalled();
+ expect(calls()).toBe(1);
+ });
+
+ it("maxRetries 0 stops conflict retries only; a 503 is still retried on the transient budget", async () => {
+ const conflict = failing(new Error("500: write-write conflict"));
+ await expect(retryCommit(conflict.fn, { maxRetries: 0, transient: true })).rejects.toThrow(/write-write/);
+ expect(conflict.calls()).toBe(1);
+
+ const unavailable = failing(statusError(503, "0"));
+ await expect(retryCommit(unavailable.fn, { maxRetries: 0, transient: true, sleep: async () => {} })).resolves.toBe("ok");
+ expect(unavailable.calls()).toBe(2);
+ });
+
+ it("still retries a lock conflict on its old short backoff", async () => {
+ const { fn, calls } = failing(new Error("500: write-write conflict"));
+ const result = retryCommit(fn, { maxRetries: 6, transient: false, random: () => 0 });
+ await vi.advanceTimersByTimeAsync(10);
+ await expect(result).resolves.toBe("ok");
+ expect(calls()).toBe(2);
+ });
+
+ it("never retries 500, 413 or 409", async () => {
+ for (const status of [500, 413, 409]) {
+ const { fn, calls } = failing(statusError(status));
+ await expect(retryCommit(fn, transient), String(status)).rejects.toThrow(new RegExp(`^${status}:`));
+ expect(calls(), String(status)).toBe(1);
+ }
+ });
+
+ it("does not retry a transient status unless asked to", async () => {
+ const { fn, calls } = failing(statusError(503, "0"));
+ await expect(retryCommit(fn, { maxRetries: 6, transient: false })).rejects.toThrow(/^503:/);
+ expect(calls()).toBe(1);
+ });
+
+ it("does not retry an abort", async () => {
+ const { fn, calls } = failing(new DOMException("This operation was aborted", "AbortError"));
+ await expect(retryCommit(fn, transient)).rejects.toThrow(/aborted/);
+ expect(calls()).toBe(1);
+ });
+
+ it("an abort during the wait rejects promptly, with the commit's own error", async () => {
+ const deadline = new AbortController();
+ const { fn, calls } = failing(statusError(503, "20"));
+ const result = retryCommit(fn, { ...transient, signal: deadline.signal });
+ const assertion = expect(result).rejects.toThrow(/^503:/);
+ await vi.advanceTimersByTimeAsync(100);
+ deadline.abort();
+ await assertion;
+ expect(calls()).toBe(1);
+ });
+
+ it("stops retrying a transient status once the caller has given up", async () => {
+ const { fn, calls } = failing(statusError(503, "0"));
+ await expect(retryCommit(fn, { ...transient, shouldGiveUp: () => true })).rejects.toThrow(/^503:/);
+ expect(calls()).toBe(1);
+ });
+
+ it("waits for an async give-up check, and hands it the failure", async () => {
+ const { fn, calls } = failing(statusError(503, "0"));
+ const shouldGiveUp = vi.fn(async (_err: unknown) => true);
+ await expect(retryCommit(fn, { ...transient, shouldGiveUp })).rejects.toThrow(/^503:/);
+ expect(calls()).toBe(1);
+ expect(String(shouldGiveUp.mock.calls[0][0])).toMatch(/^Error: 503:/);
+ });
+
+ it("keeps retrying when the give-up check itself fails", async () => {
+ const { fn, calls } = failing(statusError(503, "0"));
+ const result = retryCommit(fn, {
+ ...transient,
+ shouldGiveUp: async () => { throw new Error("probe broke"); },
+ });
+ await vi.advanceTimersByTimeAsync(0);
+ await expect(result).resolves.toBe("ok");
+ expect(calls()).toBe(2);
+ });
+
+ it("reports each retry", async () => {
+ const onRetry = vi.fn();
+ const { fn } = failing(statusError(503, "1"));
+ const result = retryCommit(fn, { ...transient, onRetry });
+ await vi.advanceTimersByTimeAsync(1000);
+ await result;
+ expect(onRetry).toHaveBeenCalledWith(expect.objectContaining({ attempt: 1, delayMs: 1000, kind: "transient" }));
+ });
+});
+
+// F1 of the #845 review: the budget used to bound only the sleeps, so a 504 on
+// a bulk that took minutes could be sent three more times. It now bounds the
+// requests too, and a 504 gets at most one re-send.
+describe("retryCommit: the transient budget bounds total time", () => {
+ beforeEach(() => { vi.useFakeTimers(); });
+ afterEach(() => { vi.useRealTimers(); });
+
+ /** `fn` whose every attempt takes `ms` on the (fake) clock, then fails with `failures` in turn, then answers `answer`. */
+ function slow<T>(ms: number, failures: unknown[], answer: T): { fn: () => Promise<T>; calls: () => number } {
+ let n = 0;
+ return {
+ fn: () => new Promise<T>((resolve, reject) => {
+ const failure = failures[n++];
+ setTimeout(() => (failure !== undefined ? reject(failure) : resolve(answer)), ms);
+ }),
+ calls: () => n,
+ };
+ }
+
+ /** Run `retryCommit` to completion on the fake clock; the outcome and the time it took. */
+ async function run<T>(fn: () => Promise<T>, budget = TRANSIENT_RETRY_BUDGET_MS, shouldGiveUp?: () => Promise<boolean>) {
+ const start = Date.now();
+ const settled = retryCommit(fn, { maxRetries: 6, transient: true, transientBudgetMs: budget, random: () => 0, shouldGiveUp })
+ .then(value => ({ value, error: undefined as unknown }), (error: unknown) => ({ value: undefined, error }))
+ .then(outcome => ({ ...outcome, elapsed: Date.now() - start }));
+ await vi.advanceTimersByTimeAsync(10 * 60_000);
+ return settled;
+ }
+
+ it("never re-sends a slow 504 bulk: one request, then straight to the caller's fallback", async () => {
+ // A proxy timing out a five-minute bulk. The backend is likely still applying it.
+ const { fn, calls } = slow(300_000, [statusError(504), statusError(504), statusError(504), statusError(504)], "ok");
+ const outcome = await run(fn);
+ expect(String(outcome.error)).toMatch(/^Error: 504:/);
+ expect(calls(), "no full re-send of a slow bulk").toBe(1);
+ expect(outcome.elapsed).toBe(300_000);
+ });
+
+ it("re-sends a fast 504 at most once, with budget left for more", async () => {
+ expect(GATEWAY_TIMEOUT_MAX_RETRIES).toBe(1);
+ const { fn, calls } = slow(10, [statusError(504), statusError(504), statusError(504)], "ok");
+ const outcome = await run(fn);
+ expect(String(outcome.error)).toMatch(/^Error: 504:/);
+ expect(calls()).toBe(2);
+ });
+
+ it("a 503 after a 504 is still bounded by the one-504 rule only for 504s", async () => {
+ const { fn, calls } = slow(10, [statusError(504), statusError(503)], "ok");
+ const outcome = await run(fn);
+ expect(outcome.value).toBe("ok");
+ expect(calls()).toBe(3);
+ });
+
+ it("a 504 the backend applied comes back Idempotent on the re-send, and that is the answer", async () => {
+ const { fn, calls } = slow(10, [statusError(504)], { rev: 7, status: "Idempotent" });
+ const outcome = await run(fn);
+ expect(outcome.value).toEqual({ rev: 7, status: "Idempotent" });
+ expect(calls()).toBe(2);
+ });
+
+ it("a 504 the backend partly applied comes back 409 on the re-send, which is thrown to the partial-commit handling", async () => {
+ const partial = httpError(409, JSON.stringify({
+ error: "conflict",
+ message: "bulk request is partially committed (1/2 patch IDs)",
+ committed_patch_ids: ["a"],
+ }), new Headers());
+ const { fn, calls } = slow(10, [statusError(504), partial], "ok");
+ const outcome = await run(fn);
+ expect(outcome.error).toBe(partial);
+ expect(calls()).toBe(2);
+ });
+
+ it("charges each request to the budget: two 1.5 s attempts and a wait, not four", async () => {
+ const { fn, calls } = slow(1_500, Array(10).fill(statusError(503)), "ok");
+ const outcome = await run(fn);
+ expect(String(outcome.error)).toMatch(/^Error: 503:/);
+ // 1.5 s + 0.5 s wait + 1.5 s = 3.5 s; a third attempt would end at 6.5 s.
+ expect(calls()).toBe(2);
+ expect(outcome.elapsed).toBe(3_500);
+ });
+
+ it.each([0, 50, 400, 900, 1_500, 2_400, 3_000, 4_999, 5_000, 20_000, 300_000])(
+ "never runs past the budget once a retry is involved (attempts of %i ms)",
+ async attemptMs => {
+ const { fn, calls } = slow(attemptMs, Array(10).fill(statusError(503)), "ok");
+ const outcome = await run(fn);
+ expect(outcome.error).toBeDefined();
+ // A single attempt is whatever the request takes; anything retried fits the budget.
+ if (calls() > 1) expect(outcome.elapsed).toBeLessThanOrEqual(TRANSIENT_RETRY_BUDGET_MS);
+ else expect(outcome.elapsed).toBe(attemptMs);
+ expect(calls()).toBeLessThanOrEqual(4);
+ },
+ );
+
+ /** A give-up check that takes `ms` on the (fake) clock and then says "keep going". */
+ const slowProbe = (ms: number) => () => new Promise<boolean>(resolve => setTimeout(() => resolve(false), ms));
+
+ it("charges the give-up check to the budget: 100 ms attempts and a 2 s probe stay inside 5 s", async () => {
+ const { fn, calls } = slow(100, Array(10).fill(statusError(503)), "ok");
+ const outcome = await run(fn, TRANSIENT_RETRY_BUDGET_MS, slowProbe(2_000));
+ expect(String(outcome.error)).toMatch(/^Error: 503:/);
+ // 0.1 + 2 probe + 0.5 wait + 0.1 + 2 probe + 0.2 wait (all that is left) + 0.1 = 5 s.
+ expect(calls()).toBe(3);
+ expect(outcome.elapsed).toBeLessThanOrEqual(TRANSIENT_RETRY_BUDGET_MS);
+ });
+
+ it("a give-up check slower than what is left of the budget is cut off there, with no re-send", async () => {
+ const { fn, calls } = slow(100, Array(10).fill(statusError(503)), "ok");
+ const outcome = await run(fn, TRANSIENT_RETRY_BUDGET_MS, slowProbe(60_000));
+ expect(String(outcome.error)).toMatch(/^Error: 503:/);
+ expect(calls()).toBe(1);
+ expect(outcome.elapsed).toBe(TRANSIENT_RETRY_BUDGET_MS - 100);
+ });
+
+ it("a request longer than the whole budget is never re-sent, even for a 503", async () => {
+ const { fn, calls } = slow(TRANSIENT_RETRY_BUDGET_MS, [statusError(503, "0")], "ok");
+ const outcome = await run(fn);
+ expect(outcome.error).toBeDefined();
+ expect(calls()).toBe(1);
+ });
+});
diff --git a/ix-cli/src/cli/__tests__/database-probe.test.ts b/ix-cli/src/cli/__tests__/database-probe.test.ts
new file mode 100644
index 00000000..42b1cb44
--- /dev/null
+++ b/ix-cli/src/cli/__tests__/database-probe.test.ts
@@ -0,0 +1,105 @@
+// Copyright 2026 Ix Infrastructure Inc.
+
+import { describe, expect, it, vi } from "vitest";
+import { createDatabaseProbe, probeDatabase, type DatabaseVerdict } from "../database-probe.js";
+import { httpError, LocalTokenRequiredError } from "../../client/api.js";
+import type { HealthResponse } from "../../client/types.js";
+
+/** A client whose health and revision reads answer as told. */
+function client(health: () => Promise<HealthResponse>, revision: () => Promise<unknown> = async () => 1) {
+ return { readHealth: vi.fn(health), currentRevision: vi.fn(revision) };
+}
+
+describe("probeDatabase", () => {
+ it("is unreachable on the 503 degraded answer of a 1.0.33+ backend", async () => {
+ const c = client(async () => { throw httpError(503, '{"status":"degraded","database":"unreachable"}'); });
+ expect(await probeDatabase(c)).toBe("unreachable");
+ expect(c.currentRevision).not.toHaveBeenCalled();
+ });
+
+ it("is unreachable on a 200 that reports the database unreachable", async () => {
+ expect(await probeDatabase(client(async () => ({ status: "ok", database: "unreachable" })))).toBe("unreachable");
+ });
+
+ it("is reachable when health says so, without a second request", async () => {
+ const c = client(async () => ({ status: "ok", database: "reachable" }));
+ expect(await probeDatabase(c)).toBe("reachable");
+ expect(c.currentRevision).not.toHaveBeenCalled();
+ });
+
+ it("passes its short timeout to both requests", async () => {
+ const c = client(async () => ({ status: "ok" }));
+ await probeDatabase(c, { timeoutMs: 1234 });
+ expect(c.readHealth).toHaveBeenCalledWith(1234);
+ expect(c.currentRevision).toHaveBeenCalledWith(1234);
+ });
+
+ describe("on a backend whose health does not report the database", () => {
+ const old = (revision: () => Promise<unknown>) => client(async () => ({ status: "ok" }), revision);
+
+ it("is reachable when the current revision reads", async () => {
+ expect(await probeDatabase(old(async () => ({ rev: 4 })))).toBe("reachable");
+ });
+
+ it("is unknown when that read fails with a 5xx (not proof of a database outage)", async () => {
+ expect(await probeDatabase(old(async () => { throw httpError(500, '{"error":"arango down"}'); }))).toBe("unknown");
+ });
+
+ it("is unknown when the read is refused for the token, times out, is a 4xx, or never connects", async () => {
+ const failures: unknown[] = [
+ new LocalTokenRequiredError('{"code":"local_token_required"}'),
+ new DOMException("The operation timed out", "TimeoutError"),
+ httpError(404, '{"error":"not found"}'),
+ new TypeError("fetch failed"),
+ ];
+ for (const failure of failures) {
+ expect(await probeDatabase(old(async () => { throw failure; })), String(failure)).toBe("unknown");
+ }
+ });
+ });
+
+ it("is unknown when health itself fails, times out or is refused", async () => {
+ const failures: unknown[] = [
+ new TypeError("fetch failed"),
+ new DOMException("The operation timed out", "TimeoutError"),
+ new LocalTokenRequiredError('{"code":"local_token_required"}'),
+ httpError(503, "<html>bad gateway</html>"),
+ ];
+ for (const failure of failures) {
+ const c = client(async () => { throw failure; });
+ expect(await probeDatabase(c), String(failure)).toBe("unknown");
+ expect(c.currentRevision).not.toHaveBeenCalled();
+ }
+ });
+});
+
+describe("createDatabaseProbe", () => {
+ it("asks once for everyone in flight, and reuses the answer for the interval", async () => {
+ let clock = 0;
+ let release: (v: DatabaseVerdict) => void = () => {};
+ const ask = vi.fn(() => new Promise<DatabaseVerdict>(resolve => { release = resolve; }));
+ const probe = createDatabaseProbe(ask, 3_000, () => clock);
+
+ const a = probe();
+ const b = probe();
+ release("unreachable");
+ expect(await a).toBe("unreachable");
+ expect(await b).toBe("unreachable");
+ expect(ask).toHaveBeenCalledTimes(1);
+
+ clock = 2_999;
+ expect(await probe()).toBe("unreachable");
+ expect(ask).toHaveBeenCalledTimes(1);
+
+ clock = 3_000;
+ const c = probe();
+ release("reachable");
+ expect(await c).toBe("reachable");
+ expect(ask).toHaveBeenCalledTimes(2);
+ });
+
+ it("turns a probe that throws into unknown", async () => {
+ const probe = createDatabaseProbe(async () => { throw new Error("boom"); });
+ expect(await probe()).toBe("unknown");
+ });
+});
diff --git a/ix-cli/src/cli/__tests__/env-durations.test.ts b/ix-cli/src/cli/__tests__/env-durations.test.ts
index bb510516..052c26c0 100644
--- a/ix-cli/src/cli/__tests__/env-durations.test.ts
+++ b/ix-cli/src/cli/__tests__/env-durations.test.ts
@@ -1,7 +1,7 @@
// Copyright 2026 Ix Infrastructure Inc.
import { afterEach, describe, expect, it } from "vitest";
-import { parsePositiveIntEnv } from "../commands/ingest.js";
+import { parseNonNegativeIntEnv, parsePositiveIntEnv } from "../commands/ingest.js";
import { mapDeadlineMs } from "../commands/map.js";
// `Number.parseInt` reads any numeric prefix, so a unit suffix or an exponent
@@ -43,3 +43,28 @@ describe("parsePositiveIntEnv", () => {
}
});
});
+
+// A count of retries, where 0 is an instruction and not "unset":
+// IX_COMMIT_CONFLICT_RETRIES=0 used to retry six times (IN-19C).
+describe("parseNonNegativeIntEnv", () => {
+ const name = "IX_TEST_NON_NEGATIVE_INT_ENV";
+ afterEach(() => { delete process.env[name]; });
+
+ it("reads 0 as 0", () => {
+ process.env[name] = "0";
+ expect(parseNonNegativeIntEnv(name, 6)).toBe(0);
+ });
+
+ it("reads a plain positive integer", () => {
+ process.env[name] = "3";
+ expect(parseNonNegativeIntEnv(name, 6)).toBe(3);
+ });
+
+ it("falls back for unset, empty, a suffix, an exponent or a sign", () => {
+ expect(parseNonNegativeIntEnv(name, 6)).toBe(6);
+ for (const raw of ["", "1e3", "-1", "2k"]) {
+ process.env[name] = raw;
+ expect(parseNonNegativeIntEnv(name, 6), raw).toBe(6);
+ }
+ });
+});
diff --git a/ix-cli/src/cli/__tests__/helpers/fake-backend.ts b/ix-cli/src/cli/__tests__/helpers/fake-backend.ts
index eb749ee1..11aa3e63 100644
--- a/ix-cli/src/cli/__tests__/helpers/fake-backend.ts
+++ b/ix-cli/src/cli/__tests__/helpers/fake-backend.ts
@@ -121,6 +121,39 @@ export class FakeBackend {
rememberHashes = false;
/** Answer `/v1/source-hashes` with a 500, as a backend under load does. */
failSourceHashes = false;
+ /**
+ * Answer the next commits (bulk or single) with these statuses, in order,
+ * before any other decision: a proxy or a restarting backend saying "not
+ * now" (IN-19C). `retryAfter` is sent as the `Retry-After` header.
+ */
+ transientCommitFailures: Array<{ status: number; retryAfter?: string }> = [];
+ /**
+ * ArangoDB is down: `/v1/health` answers 503 `degraded` with
+ * `database: "unreachable"` (Ix-memory 1.0.33+), every commit answers 500,
+ * and `/v1/revisions/current` answers 500.
+ */
+ databaseDown = false;
+ /**
+ * The status commits answer while `databaseDown`. 500 is what the backend
+ * sends; 503 is a backend (or a proxy in front of it) that reads a lost
+ * database as "not now", which the bulk sites would otherwise retry.
+ */
+ databaseDownCommitStatus = 500;
+ /**
+ * Whether `/v1/health` reports `database` at all. Off is a backend older
+ * than 1.0.33: health answers 200 `ok` whatever the database is doing, and
+ * only a read that needs it (`/v1/revisions/current`) can tell.
+ */
+ healthReportsDatabase = true;
+ /**
+ * `/v1/health` reports the database unreachable while it is in fact up and
+ * commits land: a wrong or stale health answer.
+ */
+ healthSaysDatabaseDown = false;
+ /** `/v1/health` requests, in order. */
+ healthRequests = 0;
+ /** `/v1/revisions/current` requests, in order. */
+ revisionRequests = 0;
/**
* Answer `GET /v1/patches/{id}` with a 500, as a backend under load does:
* every read when true, or the reads the predicate picks (given the id and
@@ -406,7 +439,7 @@ export class FakeBackend {
private route(url: string, body: string, res: ServerResponse): void {
const path = new URL(url, "http://x").pathname;
- const send = (code: number, payload: unknown): void => {
+ const send = (code: number, payload: unknown, headers: Record<string, string> = {}): void => {
// Stamp the answer onto the request that produced it. Whether a bulk was
// ACCEPTED or refused is the difference between two completely different
// code paths in `ingestFiles`, and a test that assumes the wrong one is
@@ -414,7 +447,7 @@ export class FakeBackend {
// went wrong.
const last = this.requests[this.requests.length - 1];
if (last !== undefined && last.code === undefined) last.code = code;
- res.writeHead(code, { "Content-Type": "application/json" });
+ res.writeHead(code, { "Content-Type": "application/json", ...headers });
res.end(JSON.stringify(payload));
};
@@ -438,6 +471,19 @@ export class FakeBackend {
this.aborter.abort();
}
+ const transientFailure = this.transientCommitFailures.shift();
+ if (transientFailure !== undefined) {
+ return send(
+ transientFailure.status,
+ { error: `${transientFailure.status}: service unavailable` },
+ transientFailure.retryAfter !== undefined ? { "Retry-After": transientFailure.retryAfter } : {},
+ );
+ }
+ if (this.databaseDown) {
+ const status = this.databaseDownCommitStatus;
+ return send(status, { error: `${status}: Connection refused: arangodb:8529` });
+ }
+
if (path === "/v1/patches/bulk") this.bulkBodyBytes.push(Buffer.byteLength(body));
if (path === "/v1/patches/bulk" && this.maxBulkBytes !== undefined && patches.length > 1 &&
Buffer.byteLength(body) > this.maxBulkBytes) {
@@ -530,8 +576,17 @@ export class FakeBackend {
}
if (path === "/v1/health") {
+ this.healthRequests++;
const release = this.releaseVersion !== undefined ? this.releaseVersion : this.semantics === "head" ? "1.0.34" : null;
- return send(200, { status: "ok", version: "1.0.28", ...(release ? { release_version: release } : {}) });
+ const base = { version: "1.0.28", ...(release ? { release_version: release } : {}) };
+ if (!this.healthReportsDatabase) return send(200, { status: "ok", ...base });
+ if (this.databaseDown || this.healthSaysDatabaseDown) return send(503, { status: "degraded", database: "unreachable", ...base });
+ return send(200, { status: "ok", database: "reachable", ...base });
+ }
+ if (path === "/v1/revisions/current") {
+ this.revisionRequests++;
+ if (this.databaseDown) return send(500, { error: "500: Connection refused: arangodb:8529" });
+ return send(200, { rev: this.rev });
}
if (path === "/v1/source-hashes") {
if (this.failSourceHashes) return send(500, { error: "500: transaction begin timeout" });
diff --git a/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts b/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts
index 572cbe6e..78b81315 100644
--- a/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts
+++ b/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts
@@ -89,6 +89,23 @@ describe("describeCommitOutcome", () => {
});
});
+describe("describeCommitOutcome after `ix reset` (IN-19C)", () => {
+ it("is still fatal, and says the graph is still empty rather than unchanged", () => {
+ const outcome = describeCommitOutcome(12, 0, "--verbose", false, 0, false, true);
+ expect(outcome.kind).toBe("fatal");
+ const message = outcome.kind === "fatal" ? outcome.message : "";
+ expect(message).not.toContain("unchanged");
+ expect(message).toContain("this workspace's graph is still empty");
+ expect(message).toContain("--verbose");
+ });
+
+ it("keeps the old text without the flag, and warns the same on a partial run", () => {
+ const old = describeCommitOutcome(12, 0, "--verbose", false, 0, false);
+ expect(old.kind === "fatal" && old.message).toContain("The graph is unchanged");
+ expect(describeCommitOutcome(3, 9, "--verbose", false, 0, false, true).kind).toBe("warn");
+ });
+});
+
describe("ingestCompletedCleanly", () => {
it("is false when patches failed to commit", () => {
// The guard the PR calls load-bearing, and the one nothing pinned: it
diff --git a/ix-cli/src/cli/__tests__/ingest-files.test.ts b/ix-cli/src/cli/__tests__/ingest-files.test.ts
index 5aedf344..071d81d1 100644
--- a/ix-cli/src/cli/__tests__/ingest-files.test.ts
+++ b/ix-cli/src/cli/__tests__/ingest-files.test.ts
@@ -193,6 +193,8 @@ describe("ingestFiles against a fake backend", () => {
"IX_COMMIT_FAILURE_LIMIT",
"IX_COMMIT_HTTP_MAX_FILES",
"IX_COMMIT_CONCURRENCY",
+ "IX_COMMIT_CONFLICT_RETRIES",
+ "IX_COMMIT_TRANSIENT_RETRY_MS",
"IX_STITCH_COOLDOWN_MS",
"IX_STITCH_WAIT_MS",
"IX_MAP_DEADLINE_MS",
@@ -1267,6 +1269,196 @@ describe("ingestFiles against a fake backend", () => {
expect(backend.singleCount, "the fan-out is for failures only").toBe(0);
});
+ describe("commit retries and the database probe (IN-19C)", () => {
+ let stderr: ReturnType<typeof vi.spyOn>;
+ const written = (): string => stderr.mock.calls.map((call: unknown[]) => String(call[0])).join("");
+ const sequence = (): string[] =>
+ backend.requests.filter(r => r.path !== "/v1/stitch").map(r => `${r.path} ${r.code}`);
+ beforeEach(() => {
+ stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true);
+ });
+ afterEach(() => {
+ stderr.mockRestore();
+ process.exitCode = undefined;
+ });
+
+ it("re-sends a bulk the backend answered 503, and lands everything with no fan-out", async () => {
+ fixture(12);
+ backend.transientCommitFailures = [{ status: 503, retryAfter: "0" }];
+
+ const summary = await run();
+
+ expect(summary.commitErrors).toBe(0);
+ expect(summary.patchesApplied).toBe(12);
+ expect(sequence()).toEqual(["/v1/patches/bulk 503", "/v1/patches/bulk 200"]);
+ expect(backend.singleCount, "no per-file fallback").toBe(0);
+ });
+
+ it("still re-sends a 503 bulk with IX_COMMIT_CONFLICT_RETRIES=0: that knob is for conflicts only", async () => {
+ fixture(4);
+ process.env.IX_COMMIT_CONFLICT_RETRIES = "0";
+ backend.transientCommitFailures = [{ status: 503, retryAfter: "0" }];
+
+ const summary = await run();
+
+ expect(summary.commitErrors).toBe(0);
+ expect(summary.patchesApplied).toBe(4);
+ expect(sequence()).toEqual(["/v1/patches/bulk 503", "/v1/patches/bulk 200"]);
+ });
+
+ it("sends the bulk once with IX_COMMIT_TRANSIENT_RETRY_MS=0, then falls back per file as before", async () => {
+ fixture(4);
+ process.env.IX_COMMIT_TRANSIENT_RETRY_MS = "0";
+ backend.transientCommitFailures = [{ status: 503, retryAfter: "0" }];
+
+ const summary = await run();
+
+ expect(summary.commitErrors).toBe(0);
+ expect(summary.patchesApplied).toBe(4);
+ expect(sequence()).toEqual([
+ "/v1/patches/bulk 503",
+ "/v1/patch 200",
+ "/v1/patch 200",
+ "/v1/patch 200",
+ "/v1/patch 200",
+ ]);
+ });
+
+ it("trips at once when the backend reports its database down and says so, while the commits still decide the exit", async () => {
+ fixture(30);
+ backend.databaseDown = true;
+
+ const message = await runFatal();
+
+ expect(message).toContain("30 of 30");
+ expect(backend.healthRequests, "the probe asked").toBeGreaterThan(0);
+ // Tripped on the first per-file failure, so the fan-out holds the rest.
+ // The held patches are still sent afterwards, as on main, so the exit
+ // code comes from their failures, not from the health answer -- and the
+ // drain's failure budget still bounds them below one request per file.
+ expect(backend.bulkCount, "the held set was re-sent").toBeGreaterThan(1);
+ expect(backend.singleCount).toBeLessThan(30);
+ const banner = written();
+ expect(banner).toContain("cannot reach its database (ArangoDB)");
+ expect(banner).not.toContain("still passes");
+ });
+
+ it("asks about the database before waiting out a 503 on the first bulk", async () => {
+ fixture(30);
+ backend.databaseDown = true;
+ backend.databaseDownCommitStatus = 503;
+ const started = Date.now();
+
+ await runFatal();
+
+ // Asked after the first 503, not after the backoff: the database is
+ // down, so the bulk is not retried and the chunk goes to the cutoff
+ // rather than the per-file fallback. The held set is then still sent,
+ // so the commits, not the probe, decide the exit.
+ expect(Date.now() - started, "no transient backoff").toBeLessThan(5_000);
+ expect(backend.healthRequests, "the probe asked").toBeGreaterThan(0);
+ expect(backend.singleCount).toBeLessThan(30);
+ const banner = written();
+ expect(banner).toContain("cannot reach its database (ArangoDB)");
+ });
+
+ it("a health answer that wrongly says the database is down does not fail a run whose commits land", async () => {
+ fixture(30);
+ backend.healthSaysDatabaseDown = true;
+ backend.transientCommitFailures = [{ status: 503, retryAfter: "0" }];
+
+ const summary = await run();
+
+ expect(backend.healthRequests, "the probe asked, and was told 'unreachable'").toBeGreaterThan(0);
+ expect(summary.commitErrors, "the held patches were sent and landed").toBe(0);
+ expect(summary.patchesApplied).toBe(30);
+ expect(process.exitCode ?? 0).toBe(0);
+ // Every patch landed, so no cutoff banner: it would tell the user the
+ // graph is "partly updated" and to re-run, which is false.
+ expect(written()).not.toContain("kept failing");
+ expect(written()).not.toContain("partly updated");
+ });
+
+ it("does not trip on a 5xx from the current-revision fallback (an older backend proves nothing)", async () => {
+ fixture(30);
+ backend.databaseDown = true;
+ backend.healthReportsDatabase = false;
+
+ await runFatal();
+
+ expect(backend.revisionRequests).toBeGreaterThan(0);
+ expect(written()).not.toContain("cannot reach its database (ArangoDB)");
+ });
+
+ it("changes nothing on a backend that refuses every commit while its database is reachable", async () => {
+ fixture(30);
+ backend.refuseEverything = true;
+
+ const message = await runFatal();
+
+ expect(backend.healthRequests, "the probe asked").toBeGreaterThan(0);
+ expect(message).toContain("30 of 30");
+ // The streak, not the probe, tripped it: five per-file failures, as before.
+ expect(backend.singleCount).toBeGreaterThanOrEqual(5);
+ expect(backend.commitCount, "well under main's 31").toBeLessThan(25);
+ expect(written()).toContain("`ix doctor` still passes");
+ });
+
+ it("does not call a graph wiped under it 'unchanged' when nothing lands", async () => {
+ // A baseline from an earlier map, then a backend that holds none of it:
+ // the DB-reset guard fires, as it does after `ix reset` from another dir.
+ fixture(8);
+ const incremental = () =>
+ ingestFiles(repo, { format: "text", suppressOutput: true, printSummary: false });
+ await incremental();
+ backend.refuseEverything = true;
+
+ let message = "";
+ try {
+ await incremental();
+ } catch (err) {
+ message = String(err);
+ }
+
+ expect(message, "still fatal").toContain("8 of 8");
+ const out = `${written()}\n${message}`;
+ expect(out).toContain("kept failing");
+ expect(out).not.toContain("unchanged");
+ expect(out).toContain("this workspace's graph is still empty");
+ });
+
+ it("does not call it 'unchanged' after a first map that committed nothing, below the cutoff", async () => {
+ fixture(2);
+ backend.refuseEverything = true;
+ let message = "";
+ try {
+ await ingestFiles(repo, { format: "text", suppressOutput: true, printSummary: false });
+ } catch (err) {
+ message = String(err);
+ }
+ expect(message).toContain("committed nothing");
+ expect(message).not.toContain("unchanged");
+ expect(message).toContain("still empty");
+ });
+
+ it("does not call the workspace's graph empty when a scoped run with no local baseline commits nothing", async () => {
+ // `ix ingest <new file>` in a repo mapped from another HOME or machine:
+ // no local baseline, and the lookup covers that one file only, so an
+ // empty answer says nothing about the rest of the graph.
+ fixture(2);
+ backend.refuseEverything = true;
+ let message = "";
+ try {
+ await ingestFiles(join(repo, "src", "m000.ts"), { format: "text", suppressOutput: true, printSummary: false });
+ } catch (err) {
+ message = String(err);
+ }
+ expect(message).toContain("committed nothing");
+ expect(message).not.toContain("still empty");
+ expect(message).toContain("may now be stale");
+ });
+ });
+
describe("a patch that deletes (IN-04)", () => {
const incremental = () =>
ingestFiles(repo, { format: "text", suppressOutput: true, printSummary: false });
diff --git a/ix-cli/src/cli/__tests__/ingest-retry.test.ts b/ix-cli/src/cli/__tests__/ingest-retry.test.ts
index 496d3af7..f7015000 100644
--- a/ix-cli/src/cli/__tests__/ingest-retry.test.ts
+++ b/ix-cli/src/cli/__tests__/ingest-retry.test.ts
@@ -1,6 +1,6 @@
// Copyright 2026 Ix Infrastructure Inc.
-import { describe, expect, it, vi } from 'vitest';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
commitBulkWithPayloadSplit,
commitFailureIndictsBackend,
@@ -14,6 +14,8 @@ import {
parseBulkCommittedPatchIds,
parseSuggestedMaxPatches,
} from '../commands/ingest.js';
+import { retryCommit } from '../commit-retry.js';
+import { httpError } from '../../client/api.js';
// Verbatim from a backend that refused a bulk save of a >1,000-file repo (Ix#516).
// A bulk commit runs as one exclusive Arango transaction, and Arango caps that at
@@ -637,3 +639,74 @@ describe('bulk request bounds (IN-05)', () => {
expect(cut.map(c => c.map(item => item.i))).toEqual([[0, 1], [2], [3, 4], [5]]);
});
});
+
+// IN-19C wraps the bulk commit in `retryCommit` with transient retries on. A
+// 413 must reach the payload split exactly as before: no retry of the same
+// oversized body, and no wait.
+describe('commitBulkWithPayloadSplit under retryCommit', () => {
+ it('splits a 413 without retrying it, with the same requests as before', async () => {
+ const bulkCalls: number[][] = [];
+ const committed: number[] = [];
+ const sleep = vi.fn(async () => {});
+ const commitIndividually = vi.fn(async () => {});
+
+ await commitBulkWithPayloadSplit([1, 2, 3, 4, 5], {
+ commitBulk: batch => retryCommit(async () => {
+ bulkCalls.push([...batch]);
+ if (batch.length > 2) {
+ throw httpError(413, '{"error":"payload_too_large"}', new Headers({ 'Retry-After': '1' }));
+ }
+ return batch.length;
+ }, { maxRetries: 6, transient: true, sleep }),
+ onBulkCommitted: batch => committed.push(...batch),
+ commitIndividually,
+ });
+
+ expect(bulkCalls).toEqual([
+ [1, 2, 3, 4, 5],
+ [1, 2, 3],
+ [1, 2],
+ [3],
+ [4, 5],
+ ]);
+ expect(committed).toEqual([1, 2, 3, 4, 5]);
+ expect(sleep, 'a 413 is never waited on').not.toHaveBeenCalled();
+ expect(commitIndividually).not.toHaveBeenCalled();
+ });
+});
+
+// F1 of the #845 review: a 504 on a long bulk used to buy up to three more
+// full re-sends before the per-file fallback. Now the bulk is sent once, and
+// the fallback gets the group at once.
+describe('a slow 504 bulk under retryCommit', () => {
+ beforeEach(() => { vi.useFakeTimers(); });
+ afterEach(() => { vi.useRealTimers(); });
+
+ it('is sent once and falls back to the per-file path, without three full re-sends', async () => {
+ const bulkCalls: number[][] = [];
+ const commitIndividually = vi.fn(async (_items: number[], _err?: unknown) => {});
+ const start = Date.now();
+
+ const done = commitBulkWithPayloadSplit([1, 2, 3], {
+ commitBulk: batch => retryCommit(
+ () => new Promise<number>((_resolve, reject) => {
+ bulkCalls.push([...batch]);
+ // The gateway gives up after two minutes; the backend may still be at it.
+ setTimeout(() => reject(httpError(504, '<html>Gateway Timeout</html>', new Headers())), 120_000);
+ }),
+ { maxRetries: 6, transient: true },
+ ),
+ onBulkCommitted: vi.fn(),
+ commitIndividually,
+ }).then(() => Date.now() - start);
+ await vi.advanceTimersByTimeAsync(30 * 60_000);
+ const elapsed = await done;
+
+ expect(bulkCalls).toEqual([[1, 2, 3]]);
+ expect(elapsed, 'one request, no waits').toBe(120_000);
+ expect(commitIndividually).toHaveBeenCalledOnce();
+ const [items, err] = commitIndividually.mock.calls[0];
+ expect(items).toEqual([1, 2, 3]);
+ expect(String(err)).toMatch(/^Error: 504:/);
+ });
+});
diff --git a/ix-cli/src/cli/__tests__/map-cloud-no-retry.test.ts b/ix-cli/src/cli/__tests__/map-cloud-no-retry.test.ts
new file mode 100644
index 00000000..2ac537a6
--- /dev/null
+++ b/ix-cli/src/cli/__tests__/map-cloud-no-retry.test.ts
@@ -0,0 +1,80 @@
+// Copyright 2026 Ix Infrastructure Inc.
+
+// The cloud ingest path is not the bulk commit path, and the transient-status
+// retry added for bulk commits (IN-19C, commit-retry.ts) must never reach it.
+// `ix map` hands a cloud-ready workspace to the registered RemoteRunner in one
+// call. A long cloud upload that fails, even with a 502/503/504/429 and a
+// `Retry-After`, is reported once and never sent again from here, and nothing
+// on that path goes through a commit route.
+
+import * as fs from "node:fs";
+import * as os from "node:os";
+import * as path from "node:path";
+import { Command } from "commander";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { httpError, IxClient } from "../../client/api.js";
+import { registerMapCommand } from "../commands/map.js";
+import { registerRemoteRunner, type RemoteIngestOptions } from "../remote.js";
+
+describe("ix map on the cloud path", () => {
+ let dir: string;
+ const saved: Record<string, string | undefined> = {};
+ const envKeys = ["IX_LOCK_DIR", "IX_ENDPOINT", "IX_HOME", "IX_AUTO_MAP", "IX_MAP_DEADLINE_MS"];
+
+ beforeEach(() => {
+ dir = fs.mkdtempSync(path.join(os.tmpdir(), "ix-map-cloud-"));
+ for (const key of envKeys) saved[key] = process.env[key];
+ process.env.IX_LOCK_DIR = path.join(dir, "locks");
+ fs.mkdirSync(process.env.IX_LOCK_DIR);
+ process.env.IX_HOME = path.join(dir, "home");
+ // Nothing here may reach a real backend: a stray request fails fast.
+ process.env.IX_ENDPOINT = "http://127.0.0.1:9";
+ delete process.env.IX_AUTO_MAP;
+ delete process.env.IX_MAP_DEADLINE_MS;
+ process.exitCode = undefined;
+ });
+
+ afterEach(() => {
+ vi.restoreAllMocks();
+ // No unregister: a runner that is never ready routes everything local again.
+ registerRemoteRunner({ runIngestion: async () => {}, isReady: async () => false });
+ for (const key of envKeys) {
+ if (saved[key] === undefined) delete process.env[key];
+ else process.env[key] = saved[key];
+ }
+ process.exitCode = undefined;
+ fs.rmSync(dir, { recursive: true, force: true });
+ });
+
+ it.each([
+ ["a gateway timeout", () => httpError(504, "<html>Gateway Timeout</html>", new Headers())],
+ ["a 503 asking to retry at once", () => httpError(503, '{"error":"unavailable"}', new Headers({ "Retry-After": "0" }))],
+ ["a 429 asking to retry", () => httpError(429, '{"error":"slow down"}', new Headers({ "Retry-After": "1" }))],
+ ])("sends a failed cloud upload once, after %s, and never through a commit route", async (_label, failure) => {
+ const root = path.join(dir, "workspace");
+ fs.mkdirSync(root);
+ const calls: RemoteIngestOptions[] = [];
+ registerRemoteRunner({
+ isReady: async () => true,
+ runIngestion: async opts => {
+ calls.push(opts);
+ throw failure();
+ },
+ });
+ const bulk = vi.spyOn(IxClient.prototype, "commitPatchBulk");
+ const single = vi.spyOn(IxClient.prototype, "commitPatch");
+ vi.spyOn(console, "error").mockImplementation(() => {});
+
+ const program = new Command();
+ registerMapCommand(program);
+ await program.parseAsync(["node", "ix", "map", root, "--silent"]);
+
+ expect(calls, "one upload, no re-send").toHaveLength(1);
+ // `ix map` uploads the canonical root (realpath): /var -> /private/var on
+ // macOS, 8.3 short names expanded on Windows.
+ expect(calls[0]).toMatchObject({ cwd: fs.realpathSync.native(root) });
+ expect(bulk).not.toHaveBeenCalled();
+ expect(single).not.toHaveBeenCalled();
+ expect(process.exitCode).toBe(1);
+ });
+});
diff --git a/ix-cli/src/cli/backend-version.ts b/ix-cli/src/cli/backend-version.ts
index c802014b..e9dc006f 100644
--- a/ix-cli/src/cli/backend-version.ts
+++ b/ix-cli/src/cli/backend-version.ts
@@ -176,8 +176,10 @@ export async function fetchBackendHealth(
client: IxClient,
knownLatest: string | null | undefined,
isNewer: (a: string, b: string) => boolean,
+ /** Bounds a probe that must answer in seconds; the client's default otherwise. */
+ timeoutMs?: number,
): Promise<HealthResponse> {
- const health = await client.health();
+ const health = await client.health(timeoutMs);
// From the client the request actually went to. Taking it as a second
// argument reintroduced exactly the "a call site can get it wrong" mistake
// this module exists to make impossible.
diff --git a/ix-cli/src/cli/commands/doctor.ts b/ix-cli/src/cli/commands/doctor.ts
index c178b3ff..fcaef0c0 100644
--- a/ix-cli/src/cli/commands/doctor.ts
+++ b/ix-cli/src/cli/commands/doctor.ts
@@ -41,6 +41,7 @@ import { loadIngestBaseline } from "../ingest-baseline.js";
import { isCloudReady } from "../remote.js";
import { hasCompletedMapBaseline } from "../stale.js";
import { printJson } from "../format.js";
+import { healthFromError } from "../database-probe.js";
import type { CapabilitiesResponse, HealthResponse } from "../../client/types.js";
interface CheckResult {
@@ -250,21 +251,8 @@ export function assessLocalAuth(
};
}
-/**
- * A health answer, including the 503 a backend sends while it cannot reach its
- * database: that is a server that answered, and the body says what is wrong.
- * Thrown for anything else.
- */
-export function healthFromError(err: unknown): HealthResponse | null {
- const m = /^503:\s*(\{.*\})\s*$/s.exec((err as Error)?.message ?? "");
- if (!m) return null;
- try {
- const body = JSON.parse(m[1]!) as HealthResponse;
- return typeof body?.status === "string" ? body : null;
- } catch {
- return null;
- }
-}
+// Shared with the ingest's database probe; re-exported for existing importers.
+export { healthFromError };
/**
* "Database reachable" from the health answer. A backend that reports
diff --git a/ix-cli/src/cli/commands/ingest.ts b/ix-cli/src/cli/commands/ingest.ts
index 9ce32893..405b7d8c 100644
--- a/ix-cli/src/cli/commands/ingest.ts
+++ b/ix-cli/src/cli/commands/ingest.ts
@@ -44,6 +44,11 @@ import {
drainInPasses,
perFileAction,
} from '../commit-breaker.js';
+import { retryCommit, TRANSIENT_RETRY_BUDGET_MS } from '../commit-retry.js';
+import { createDatabaseProbe, probeDatabase } from '../database-probe.js';
+// Moved to commit-retry.ts with the retry loop that uses them; re-exported so
+// existing importers keep working.
+export { isAbortError, isRetryableCommitConflict } from '../commit-retry.js';
import { readBackendHealth } from './upgrade.js';
import {
createIgnoreMatcher, parseIgnoreFile, parseIgnorePattern, type IgnorePattern,
@@ -135,6 +140,19 @@ export function parsePositiveIntEnv(name: string, fallback: number): number {
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
}
+/**
+ * `parsePositiveIntEnv`, but 0 is a value rather than "unset". For a count of
+ * RETRIES, where 0 means "do not retry" and reading it as the default made the
+ * setting impossible to turn off (IX_COMMIT_CONFLICT_RETRIES=0 retried 6 times).
+ */
+export function parseNonNegativeIntEnv(name: string, fallback: number): number {
+ const raw = process.env[name]?.trim();
+ if (!raw) return fallback;
+ if (!/^\d+$/.test(raw)) return fallback;
+ const parsed = Number.parseInt(raw, 10);
+ return Number.isSafeInteger(parsed) ? parsed : fallback;
+}
+
/**
* The patterns in force for this ingest: `--exclude` flags first, then the
@@ -950,32 +968,6 @@ export function planDeletedFileRecovery(
return { previousDeletedFiles, nextDeletedFiles, recreatedPaths, forceReingestPaths };
}
-/** Lock contention, as opposed to the transport failures below. Kept separate
- * only so each group's reason for being retryable stays readable. */
-const COMMIT_CONFLICT_RETRY_PATTERNS = [
- // ArangoDB lock conflicts. These were briefly split into their own constant
- // so `commitFailureIndictsBackend` could exclude them; that exclusion was
- // reverted in round 3 -- it made the cutoff inert in a real saturation mode --
- // and the split had no consumer left, so it is inlined.
- 'write-write conflict',
- 'timeout waiting to lock key',
- 'error: 1200',
- // Transport-level failures (k8s ingress reset, socket drop under load).
- //
- // These are retried because the request usually never reached the server --
- // but "usually" is the whole of Ix#495. A bulk commit is chunked and writes
- // its idempotency keys as each chunk lands, so a socket that drops after the
- // server started work leaves part of the group committed. Retrying then
- // re-sends a set the server has partly seen, and because the bulk group id is
- // a hash of the patch ids, the retry reproduces the same group and is refused
- // with a 409 that no further retry can clear.
- //
- // The retry stays (it is right far more often than not); what changed is that
- // the resulting 409 is now recoverable — see isBulkPartiallyCommittedError.
- 'fetch failed',
- 'econnreset',
- 'econnrefused',
-];
/**
* A bulk commit the server accepted only in part, or whose patches all landed
@@ -1027,23 +1019,6 @@ export function parseBulkCommittedPatchIds(err: unknown): Set<string> | undefine
}
}
-// An abort from the shared wall-clock deadline (or a per-request timeout) must
-// NOT be retried — the whole point of the deadline is to stop work. Retrying
-// here would defeat it and add 6 more backed-off requests to an already-late
-// run. AbortSignal.timeout raises a TimeoutError; an aborted deadline raises an
-// AbortError; both also stringify with "aborted".
-export function isAbortError(err: unknown): boolean {
- const name = (err as { name?: string } | null)?.name;
- if (name === "AbortError" || name === "TimeoutError") return true;
- return String(err).toLowerCase().includes("aborted");
-}
-
-export function isRetryableCommitConflict(err: unknown): boolean {
- if (isAbortError(err)) return false;
- const message = String(err).toLowerCase();
- return COMMIT_CONFLICT_RETRY_PATTERNS.some(pattern => message.includes(pattern));
-}
-
/**
* Does this commit failure say anything about the BACKEND? (Ix#560)
*
@@ -1080,18 +1055,14 @@ export function commitFailureIndictsBackend(err: unknown, runDeadlineExpired: bo
return true;
}
+/**
+ * The per-file commit's retry: lock conflicts and transport drops only.
+ * Transient statuses (502/503/504/429) are retried at the BULK sites, which
+ * call `retryCommit` themselves; retrying them per patch would multiply a
+ * per-bulk wait budget by the number of patches.
+ */
async function retryOnConflict<T>(fn: () => Promise<T>, maxRetries: number): Promise<T> {
- for (let attempt = 0; ; attempt++) {
- try {
- return await fn();
- } catch (err) {
- if (attempt >= maxRetries || !isRetryableCommitConflict(err)) throw err;
- // ArangoDB lock conflicts usually clear as soon as the competing transaction
- // commits. Transport errors may need a bit more breathing room under k8s load.
- const delay = 10 * (1 << attempt) + Math.random() * 10;
- await new Promise<void>(r => setTimeout(r, delay));
- }
- }
+ return retryCommit(fn, { maxRetries, transient: false });
}
/**
@@ -2007,7 +1978,13 @@ export function describeCommitOutcome(
* patches? When it did, the banner above has already given the reason, and
* this line must not send the reader back to a flag that adds nothing.
*/
- cutoffFired: boolean = cutoffSkipped > 0
+ cutoffFired: boolean = cutoffSkipped > 0,
+ /**
+ * The backend held none of this run's files before it started (a first map,
+ * or one after `ix reset`). With nothing landed, "the graph is unchanged"
+ * would read as "the old graph is still there", and there is none.
+ */
+ graphStartedEmpty: boolean = false
): CommitOutcome {
if (commitErrors <= 0) return { kind: "ok" };
const attempted = commitErrors + patchesApplied;
@@ -2112,7 +2089,10 @@ export function describeCommitOutcome(
kind: "fatal",
message:
`Ingest committed nothing: all ${commitErrors} ${patches} failed to commit. ` +
- `The graph is unchanged and may now be stale. Re-run with ${debugFlag} to see why.`,
+ (graphStartedEmpty
+ ? `The backend held none of these files before this run, so this workspace's graph is still empty. `
+ : `The graph is unchanged and may now be stale. `) +
+ `Re-run with ${debugFlag} to see why.`,
};
}
return {
@@ -2507,6 +2487,14 @@ export async function ingestFiles(
// a way this run can have written to the graph, or have found it not where
// the last run left it, without a patch being counted.
let dbResetDetected = false;
+ /**
+ * The backend held none of this run's files when it started: the DB-reset
+ * guard fired, or a lookup over the whole file set found no hashes (a first
+ * map, or one after `ix reset` / `reset --ingest`). Only the failure messages
+ * read it: "the graph is unchanged" is misleading when there was no graph.
+ * Unset on any path that ends before the lookup.
+ */
+ let graphStartedEmpty: boolean | undefined;
let extractorRebuild = false;
let deletionsFound = false;
let stitchSent = false;
@@ -2614,6 +2602,60 @@ export async function ingestFiles(
* raises TimeoutError too, and that is a saturation signal, not our clock.
*/
const runDeadlineExpired = (): boolean => opts.deadlineSignal?.aborted === true;
+ /** The same deadline, for the commit retries' waits to abort on. */
+ const runDeadlineSignal = opts.deadlineSignal;
+ /**
+ * The backend's own word on its database, asked only once commits are
+ * failing (IN-19C). The cutoff judges by a streak, which with ArangoDB down
+ * costs a streak of doomed requests here and a failure budget per drain
+ * pass; the backend can simply say so. One probe answers every caller for a
+ * few seconds, under a 2 s timeout, and only `unreachable` changes anything
+ * -- see `database-probe.ts`.
+ */
+ const newDatabaseProbe = () => createDatabaseProbe(() => probeDatabase({
+ // Through the one place health is fetched and its release recorded.
+ readHealth: timeoutMs => readBackendHealth(client, timeoutMs),
+ currentRevision: timeoutMs => client.currentRevision(timeoutMs),
+ }), 3_000);
+ let askDatabase = newDatabaseProbe();
+ /** `patchesTheBackendTook` when `askDatabase`'s cache was started; a landed patch is newer evidence than any cached verdict. */
+ let askedAtPatches = 0;
+ /** `patchesTheBackendTook` when the backend last reported its database down; undefined since it reported it up. */
+ let databaseDownAsOf: number | undefined;
+ /** The last answer was "unreachable", and nothing has landed since to say otherwise. */
+ const databaseKnownDown = (): boolean =>
+ databaseDownAsOf !== undefined && databaseDownAsOf === patchesTheBackendTook;
+ /** Ask (or reuse a recent answer) and record it. True when the database is down. */
+ const checkDatabase = async (): Promise<boolean> => {
+ // IX_COMMIT_FAILURE_LIMIT=0 opts out of giving up; the probe never stops a run then.
+ if (commitBreaker.limit <= 0) return false;
+ if (runDeadlineExpired()) return databaseKnownDown();
+ if (askedAtPatches !== patchesTheBackendTook) {
+ askDatabase = newDatabaseProbe();
+ askedAtPatches = patchesTheBackendTook;
+ }
+ const verdict = await askDatabase();
+ if (verdict === 'unreachable') databaseDownAsOf = patchesTheBackendTook;
+ else if (verdict === 'reachable') databaseDownAsOf = undefined;
+ return verdict === 'unreachable';
+ };
+ /**
+ * After a commit failure that indicts the backend: if its database is down,
+ * give up now rather than after a streak. A later commit landing clears the
+ * trip, as it clears one a streak set.
+ */
+ const tripIfDatabaseDown = async (err: unknown): Promise<void> => {
+ if (commitBreaker.tripped()) return;
+ if ((await checkDatabase()) && !commitBreaker.tripped()) commitBreaker.trip(err);
+ };
+ const noteCommitRetry = (where: string) =>
+ (info: { attempt: number; delayMs: number; error: unknown; kind: string }): void => {
+ if (!debug) return;
+ const msg = String(info.error);
+ process.stderr.write(
+ `\n [${where} retry ${info.attempt}, ${info.kind}] waiting ${Math.round(info.delayMs)} ms: ${msg.length > 200 ? `${msg.slice(0, 200)}…` : msg}\n`,
+ );
+ };
let tooLarge = 0;
/** The stat loop's record of empty and oversized files, once it exists; `tooLarge` counts its oversized ones. */
let skippedForSize: ReadonlyMap<string, 'empty' | 'tooLarge'> | undefined;
@@ -2974,6 +3016,10 @@ export async function ingestFiles(
// file, so the next run's baseline did not name them, they were never
// found deleted again, and their nodes stayed in the graph for good.
const deferredDeletions: string[] = [];
+ // Whether the lookup ANSWERED. Thrown every time now, rather than only with
+ // deletions pending, so an empty answer can be told from a failed one; the
+ // catch below does what `loadExistingHashes` did with a swallowed failure.
+ let hashLookupAnswered = false;
if (hashLookupPaths.length > 0) {
try {
knownHashes = await loadExistingHashes(
@@ -2982,10 +3028,11 @@ export async function ingestFiles(
toWorkspaceRelative,
sourceWorkspaceIdOf,
debug,
- deletedPaths.length > 0,
+ true,
);
+ hashLookupAnswered = true;
} catch (err) {
- if (debug) process.stderr.write(`\n [deletion cleanup skipped] hash lookup failed: ${err}\n`);
+ if (debug && deletedPaths.length > 0) process.stderr.write(`\n [deletion cleanup skipped] hash lookup failed: ${err}\n`);
deferredDeletions.push(...deletedPaths);
deletedPaths.length = 0;
knownHashes = new Map();
@@ -2996,6 +3043,19 @@ export async function ingestFiles(
knownHashes = new Map();
if (debug) process.stderr.write(`\n All ${filePaths.length} files mtime-clean — skipping hash lookup\n`);
}
+ // With no local baseline every file was looked up, so an answer naming
+ // none of them means the backend holds none. Not under --force, which
+ // says nothing about the baseline, and not on a failed lookup, which
+ // says nothing about the backend. Not on a workspace-id migration either:
+ // the old id's graph is still there until the cleanup after this run.
+ // And not on a partial run (a file or directory scope, or --lang): its
+ // lookup covered only its own files, so an empty answer says nothing about
+ // the rest of the workspace -- `ix ingest <new file>` in a repo mapped from
+ // another machine has no local baseline but a full graph.
+ graphStartedEmpty =
+ dbResetDetected ||
+ (!opts.force && !workspaceMigrated && !scopedRun && !opts.lang &&
+ mtimeCache.size === 0 && hashLookupAnswered && knownHashes.size === 0);
const hashed = performance.now();
timings.hashMs = Math.round(hashed - discovered);
@@ -3229,7 +3289,12 @@ export async function ingestFiles(
return cutToLimits(items);
};
const COMMIT_CONCURRENCY = parsePositiveIntEnv('IX_COMMIT_CONCURRENCY', 8); // parallel HTTP save requests
- const COMMIT_CONFLICT_RETRIES = parsePositiveIntEnv('IX_COMMIT_CONFLICT_RETRIES', 6); // retry transient Arango lock conflicts
+ // Retries of one commit after a lock conflict or a transport drop. 0: none.
+ const COMMIT_CONFLICT_RETRIES = parseNonNegativeIntEnv('IX_COMMIT_CONFLICT_RETRIES', 6);
+ // Total wait for re-sending a BULK the backend answered 502/503/504/429
+ // (at most three retries, `Retry-After` cut to it). 0: no such retries.
+ // Its own knob, so turning conflict retries off leaves these alone.
+ const COMMIT_TRANSIENT_RETRY_MS = parseNonNegativeIntEnv('IX_COMMIT_TRANSIENT_RETRY_MS', TRANSIENT_RETRY_BUDGET_MS);
// Re-sends of a commit that lost the base-rev race to another writer; see retryOnBaseRevRace.
const COMMIT_BASE_REV_RETRIES = parsePositiveIntEnv('IX_COMMIT_BASE_REV_RETRIES', 8);
let baseRevRetries = 0;
@@ -3542,7 +3607,13 @@ export async function ingestFiles(
}
} else {
commitErrors++;
- if (indicts) commitBreaker.recordFailure(commitErr);
+ if (indicts) {
+ commitBreaker.recordFailure(commitErr);
+ // Not in the drain (the only caller with a failure
+ // budget): it ignores `tripped`, so a probe there would
+ // only add health requests and latency.
+ if (fallbackOpts?.failureBudget === undefined) await tripIfDatabaseDown(commitErr);
+ }
}
if (debug) {
const errMsg = String(commitErr);
@@ -3586,6 +3657,10 @@ export async function ingestFiles(
/** Re-send the held set. What it cannot place is left on `deferredByCutoff`. */
const commitIndividuallyAfterCutoff = async (itemsIn: PreparedPatch[]): Promise<void> => {
let items = itemsIn;
+ // Re-sent even when the backend has reported its database down. The
+ // probe may hold patches back during the fan-out, but whether the run
+ // fails is decided by commits, as on main: a wrong or stale health
+ // answer must not turn a run whose patches would land into an exit 1.
if (debug) {
process.stderr.write(
`\n [cutoff] re-sending ${items.length} held-back patches with a fresh streak\n`,
@@ -3629,9 +3704,19 @@ export async function ingestFiles(
const bulkStart = performance.now();
try {
const result = await retryOnBaseRevRace(
- () => retryOnConflict(
+ () => retryCommit(
() => client.commitPatchBulk(chunk.map(item => item.patch)),
- COMMIT_CONFLICT_RETRIES,
+ {
+ maxRetries: COMMIT_CONFLICT_RETRIES,
+ transient: COMMIT_TRANSIENT_RETRY_MS > 0,
+ transientBudgetMs: COMMIT_TRANSIENT_RETRY_MS,
+ signal: runDeadlineSignal,
+ // Not the breaker: this bulk exists because the breaker
+ // tripped, so it would always say give up. Asked, not just
+ // read: a 503 here may be the database going down.
+ shouldGiveUp: () => checkDatabase(),
+ onRetry: noteCommitRetry('cutoff bulk'),
+ },
),
COMMIT_BASE_REV_RETRIES,
noteBaseRevRetry,
@@ -3776,6 +3861,9 @@ export async function ingestFiles(
// tests, which previously asserted the stranding guarantee at a budget
// the CLI never runs at and so could not see it fail.
const leftover = await drainInPasses(items, async pending => {
+ // No database probe here (the per-file loop skips it when given a
+ // failure budget): the drain sends, as on main, so the exit code
+ // follows what the commits did, never a health answer.
commitBreaker.reset();
const tookBefore = patchesTheBackendTook;
await commitItemsSerially(pending, recordDrainMs, undefined, {
@@ -3823,9 +3911,31 @@ export async function ingestFiles(
);
const commitStart = performance.now();
const result = await retryOnBaseRevRace(
- () => retryOnConflict(
+ () => retryCommit(
() => client.commitPatchBulk(items.map(item => item.patch)),
- COMMIT_CONFLICT_RETRIES,
+ {
+ maxRetries: COMMIT_CONFLICT_RETRIES,
+ // 502/503/504/429 too, honouring Retry-After up to the
+ // budget (5 s per bulk by default, requests included; a
+ // slow bulk is not sent again). A 413 and a 409 are
+ // thrown at once, to the payload split and the
+ // partial-commit handling below.
+ transient: COMMIT_TRANSIENT_RETRY_MS > 0,
+ transientBudgetMs: COMMIT_TRANSIENT_RETRY_MS,
+ signal: runDeadlineSignal,
+ // Asked, not just read. On the first bulk of a run nothing
+ // has probed yet, and a 503 from a backend whose database
+ // is down would otherwise spend the whole budget
+ // before anything asked it. A database reported down trips
+ // the breaker, so `shouldStop` hands the chunk to the
+ // cutoff instead of the per-file fallback.
+ shouldGiveUp: async err => {
+ if (commitBreaker.tripped()) return true;
+ await tripIfDatabaseDown(err);
+ return commitBreaker.tripped() || databaseKnownDown();
+ },
+ onRetry: noteCommitRetry('bulk'),
+ },
),
COMMIT_BASE_REV_RETRIES,
noteBaseRevRetry,
@@ -5049,6 +5159,14 @@ export async function ingestFiles(
// left the agent believing the graph was current while `ix search` and
// `ix impact` answered from a stale one. A wrong answer nobody can see is
// worse than a failed command.
+ // A trip alone is not enough: the database probe can trip the breaker on a
+ // bulk failure with no `commitErrors++` (a health answer of "unreachable"
+ // that turns out wrong), and the cutoff and drain may then land every patch.
+ // Printing the cutoff banner then would say the graph is "partly updated"
+ // and ask for a re-run when nothing is missing. A skipped patch is counted
+ // in `commitErrors`, so this still fires whenever the cutoff held anything
+ // back that did not land.
+ const cutoffFired = commitBreaker.everTripped() && commitErrors > 0;
const commitReport = describeCommitOutcome(
commitErrors,
patchesApplied,
@@ -5058,7 +5176,8 @@ export async function ingestFiles(
commitBreaker.skipped(),
// The same expression the banner is gated on, so this can never say "See
// the cutoff above" with no cutoff above it.
- commitBreaker.everTripped()
+ cutoffFired,
+ graphStartedEmpty === true
);
const summary: IngestFilesSummary = {
filesDiscovered,
@@ -5096,15 +5215,7 @@ export async function ingestFiles(
commitErrors === 0 &&
stitchErrors === 0,
};
- // `everTripped()` alone, and it is not a weakening: `recordFailure` has one
- // call site, in the per-file catch immediately after `commitErrors++`, so a
- // trip already implies a real commit failure. A `commitLoopErrors` counter
- // guarded this for two rounds against a scenario that no longer exists --
- // replay failures were moved off `recordFailure` onto a local counter, and a
- // deletion patch that fails to BUILD never reaches the breaker at all. A dead
- // conjunct defended by a comment describing an impossible case is the exact
- // shape this PR has spent rounds removing.
- if (commitBreaker.everTripped()) {
+ if (cutoffFired) {
// Gated on the cutoff having FIRED, not on what was left unsent. A patch
// that was never sent produces no `[commit error]` line even under
// --verbose, so this has to print for those -- but gating on them alone
@@ -5116,7 +5227,10 @@ export async function ingestFiles(
//
// Before the stitch line and before the commit report: this is the cause,
// and everything else printed about this run is its consequence.
- process.stderr.write(`${describeCommitCutoff(commitBreaker, client.endpoint, patchesApplied, opts.deadlineSignal?.aborted === true)}\n`);
+ process.stderr.write(`${describeCommitCutoff(commitBreaker, client.endpoint, patchesApplied, opts.deadlineSignal?.aborted === true, {
+ graphStartedEmpty: graphStartedEmpty === true,
+ databaseUnreachable: databaseKnownDown(),
+ })}\n`);
}
if (stitchErrors > 0) {
process.stderr.write(` ${describeStitchFailure(stitchError)}\n`);
diff --git a/ix-cli/src/cli/commands/upgrade.ts b/ix-cli/src/cli/commands/upgrade.ts
index a19ddbe2..0ed5e9d2 100644
--- a/ix-cli/src/cli/commands/upgrade.ts
+++ b/ix-cli/src/cli/commands/upgrade.ts
@@ -39,11 +39,11 @@ export function backendCeiling(): string | null {
return readCache()?.backendLatest ?? null;
}
-export async function readBackendHealth(client: IxClient) {
+export async function readBackendHealth(client: IxClient, timeoutMs?: number) {
// backendLatest, NOT latest: `latest` is the CLI's own release series, so a
// ceiling taken from it would refuse every legitimate container report for
// ever — silently reinstating the bug this exists to fix.
- return fetchBackendHealth(client, backendCeiling(), isNewer);
+ return fetchBackendHealth(client, backendCeiling(), isNewer, timeoutMs);
}
const __filename = fileURLToPath(import.meta.url);
diff --git a/ix-cli/src/cli/commit-breaker.ts b/ix-cli/src/cli/commit-breaker.ts
index bcbbe3af..35d4de8c 100644
--- a/ix-cli/src/cli/commit-breaker.ts
+++ b/ix-cli/src/cli/commit-breaker.ts
@@ -98,6 +98,15 @@ export interface CommitBreaker {
recordSuccess(): void;
/** A commit failed in a way that sending it again cannot fix. */
recordFailure(error: unknown): void;
+ /**
+ * Give up now, without waiting for a streak: the backend has said itself
+ * that it cannot reach its database (see `database-probe.ts`).
+ *
+ * The same latch a streak sets, and cleared the same way: a commit that lands
+ * is direct evidence the probe was stale, and `recordSuccess` un-trips.
+ * `everTripped()` stays set, so the banner still explains the run.
+ */
+ trip(error: unknown): void;
/**
* A failure worth QUOTING, but not worth counting.
*
@@ -166,6 +175,13 @@ export function createCommitBreaker(limit = commitFailureLimit()): CommitBreaker
everTripped: () => everLatched,
recordSuccess: () => { consecutive = 0; latched = false; },
noteError: (error) => { last = error; },
+ trip: (error) => {
+ last = error;
+ // limit 0 is the documented opt-out: this breaker never trips.
+ if (limit <= 0) return;
+ latched = true;
+ everLatched = true;
+ },
recordFailure: (error) => {
consecutive++;
last = error;
@@ -474,6 +490,17 @@ export function describeCommitCutoff(
* attribution of the count is not.
*/
deadlineHit = false,
+ context: {
+ /**
+ * The backend held none of this run's files before it started: a first
+ * map, or one after `ix reset` (or `reset --ingest`). "The graph is
+ * unchanged" is then technically true and practically misleading -- it
+ * reads as "your old graph is still there", and there is none.
+ */
+ graphStartedEmpty?: boolean;
+ /** The backend reported during this run that it cannot reach its database. */
+ databaseUnreachable?: boolean;
+ } = {},
): string {
const detail = String(breaker.lastError() ?? "unknown error");
const trimmed = detail.length > 300 ? `${detail.slice(0, 300)}…` : detail;
@@ -488,8 +515,20 @@ export function describeCommitCutoff(
// be true, which is also one less piece of run-scoped state to keep correct.
const state =
patchesApplied === 0
- ? "The graph is unchanged."
+ ? context.graphStartedEmpty
+ ? "Nothing landed, and the backend held none of these files before this run (a first map, or a map after `ix reset`), so this workspace's graph is still empty."
+ : "The graph is unchanged."
: `${patchesApplied} other ${patchesApplied === 1 ? "patch" : "patches"} did land, so the graph is partly updated.`;
+ // When the backend has SAID its database is unreachable there is nothing
+ // left to tell apart, and "`ix doctor` still passes" would be false: doctor
+ // reads the same health answer and fails its database check on it.
+ const cause = context.databaseUnreachable
+ ? [` The backend reports that it cannot reach its database (ArangoDB); \`ix doctor\` shows it.`]
+ : [
+ ` Either the database cannot keep up — \`ix doctor\` still passes, since an ArangoDB that`,
+ ` cannot begin a transaction is reachable and consistent — or the backend is rejecting these`,
+ ` particular patches. The error above is the backend's own answer and says which.`,
+ ];
// No failure count in this message at all, deliberately, and neither
// `breaker.limit` nor the live streak is read here. The fan-out stops on a
// streak, the drain on a per-pass budget, and the batch gate on two empty
@@ -529,9 +568,7 @@ export function describeCommitCutoff(
// to `docker stats` and `/_api/query/current`, which is the wrong
// subsystem entirely. The backend's own words are quoted above; let them
// say which.
- ` Either the database cannot keep up — \`ix doctor\` still passes, since an ArangoDB that`,
- ` cannot begin a transaction is reachable and consistent — or the backend is rejecting these`,
- ` particular patches. The error above is the backend's own answer and says which.`,
+ ...cause,
` Set IX_COMMIT_FAILURE_LIMIT=0 to send every patch regardless.`,
].join("\n");
}
diff --git a/ix-cli/src/cli/commit-retry.ts b/ix-cli/src/cli/commit-retry.ts
new file mode 100644
index 00000000..aef32cc5
--- /dev/null
+++ b/ix-cli/src/cli/commit-retry.ts
@@ -0,0 +1,297 @@
+// Copyright 2026 Ix Infrastructure Inc.
+
+// ---------------------------------------------------------------------------
+// Retrying one commit request (IN-19C).
+//
+// Two classes of failure are worth sending the same commit again for, and they
+// want different waits:
+//
+// - lock conflicts and transport drops, which clear in milliseconds. These
+// have always been retried, everywhere a commit is sent, on a 10 ms * 2^n
+// backoff.
+// - a backend or proxy saying "not now": 502, 503, 504 and 429. These clear
+// in seconds, and a 429 or 503 may say how many in `Retry-After`. They are
+// retried only where the caller asks (`transient: true`), which is the
+// ingest's BULK commits: one request per chunk, at most three retries in
+// a 5-second budget per bulk (`IX_COMMIT_TRANSIENT_RETRY_MS`). A
+// `Retry-After` longer than what is left of the budget is cut to it. Hook
+// runners give `ix map` about 30 s in all, so a long wait here would cost
+// more than the retry can win back.
+//
+// The budget bounds TOTAL time, not just the sleeps: every attempt's own
+// duration and every give-up check (a health probe) is charged to it, and a re-send is made only when the time spent so
+// far, the wait, and one more attempt as long as the one that just failed all
+// fit. A bulk that took longer than the budget to fail is therefore never sent
+// again here; it goes straight to the caller's per-file fallback. A 504 gets at
+// most ONE re-send on top of that: a gateway timeout means the request was
+// slow, and the backend may still be applying it. Re-sending is safe either
+// way -- the backend keys every patch by its id, so a bulk it already applied
+// answers `Idempotent`, and one it partly applied answers a 409 naming the ids
+// that landed -- but it is not free, so it is kept rare.
+//
+// Everything else is thrown at once. In particular 413 (the payload split owns
+// that), 409 (the partial-commit and plan-mismatch handling owns that) and 500
+// (a backend that answered and refused; the cutoff owns that). A 504 that the
+// backend did apply comes back on the retry as a 409 naming the ids or as
+// `Idempotent`, and both are already handled by the caller.
+//
+// Each class has its own bound: `IX_COMMIT_CONFLICT_RETRIES` counts conflict
+// and transport retries only (0: none), and `IX_COMMIT_TRANSIENT_RETRY_MS`
+// bounds the transient waits (0: no transient retries).
+// ---------------------------------------------------------------------------
+
+/** Lock contention, as opposed to the transport failures below. Kept separate
+ * only so each group's reason for being retryable stays readable. */
+const COMMIT_CONFLICT_RETRY_PATTERNS = [
+ // ArangoDB lock conflicts. These were briefly split into their own constant
+ // so `commitFailureIndictsBackend` could exclude them; that exclusion was
+ // reverted in round 3 -- it made the cutoff inert in a real saturation mode --
+ // and the split had no consumer left, so it is inlined.
+ 'write-write conflict',
+ 'timeout waiting to lock key',
+ 'error: 1200',
+ // Transport-level failures (k8s ingress reset, socket drop under load).
+ //
+ // These are retried because the request usually never reached the server --
+ // but "usually" is the whole of Ix#495. A bulk commit is chunked and writes
+ // its idempotency keys as each chunk lands, so a socket that drops after the
+ // server started work leaves part of the group committed. Retrying then
+ // re-sends a set the server has partly seen, and because the bulk group id is
+ // a hash of the patch ids, the retry reproduces the same group and is refused
+ // with a 409 that no further retry can clear.
+ //
+ // The retry stays (it is right far more often than not); what changed is that
+ // the resulting 409 is now recoverable — see isBulkPartiallyCommittedError.
+ 'fetch failed',
+ 'econnreset',
+ 'econnrefused',
+];
+
+// An abort from the shared wall-clock deadline (or a per-request timeout) must
+// NOT be retried — the whole point of the deadline is to stop work. Retrying
+// here would defeat it and add 6 more backed-off requests to an already-late
+// run. AbortSignal.timeout raises a TimeoutError; an aborted deadline raises an
+// AbortError; both also stringify with "aborted".
+export function isAbortError(err: unknown): boolean {
+ const name = (err as { name?: string } | null)?.name;
+ if (name === "AbortError" || name === "TimeoutError") return true;
+ return String(err).toLowerCase().includes("aborted");
+}
+
+export function isRetryableCommitConflict(err: unknown): boolean {
+ if (isAbortError(err)) return false;
+ const message = String(err).toLowerCase();
+ return COMMIT_CONFLICT_RETRY_PATTERNS.some(pattern => message.includes(pattern));
+}
+
+/** Statuses that mean "not now" rather than "no". */
+const TRANSIENT_COMMIT_STATUSES = new Set([429, 502, 503, 504]);
+
+/**
+ * The HTTP status of a failed request: the `status` `httpError` puts on the
+ * error when it has the headers, else the `<status>:` its message starts with.
+ */
+function statusOf(err: unknown): number | undefined {
+ const status = (err as { status?: unknown } | null)?.status;
+ if (typeof status === "number") return status;
+ const m = /^(\d{3}):/.exec((err as Error | null)?.message ?? "");
+ return m ? Number(m[1]) : undefined;
+}
+
+/** A 502, 503, 504 or 429: worth one more try after a wait. Never an abort. */
+export function isTransientCommitStatus(err: unknown): boolean {
+ if (isAbortError(err)) return false;
+ const status = statusOf(err);
+ return status !== undefined && TRANSIENT_COMMIT_STATUSES.has(status);
+}
+
+/**
+ * Milliseconds a `Retry-After` value asks for: delta-seconds or an HTTP-date
+ * (IMF-fixdate), never below 0. Undefined when absent or unreadable, in which
+ * case the caller backs off on its own schedule.
+ */
+export function retryAfterMs(value: string | undefined | null, now: number = Date.now()): number | undefined {
+ const raw = value?.trim();
+ if (!raw) return undefined;
+ if (/^\d+$/.test(raw)) return Number(raw) * 1000;
+ // Every HTTP-date form starts with the day name. Without this, `Date.parse`
+ // reads "-1" as the year -1 and a malformed delay became "retry now".
+ if (!/^[A-Za-z]/.test(raw)) return undefined;
+ const at = Date.parse(raw);
+ if (Number.isNaN(at)) return undefined;
+ return Math.max(0, at - now);
+}
+
+function retryAfterOf(err: unknown): string | undefined {
+ const value = (err as { retryAfter?: unknown } | null)?.retryAfter;
+ return typeof value === "string" ? value : undefined;
+}
+
+/** A wait that ends early, rejecting, when `signal` aborts. */
+function abortableSleep(ms: number, signal?: AbortSignal): Promise<void> {
+ return new Promise<void>((resolve, reject) => {
+ if (signal?.aborted) {
+ reject(signal.reason);
+ return;
+ }
+ const onAbort = (): void => {
+ clearTimeout(timer);
+ reject(signal!.reason);
+ };
+ const timer = setTimeout(() => {
+ signal?.removeEventListener("abort", onAbort);
+ resolve();
+ }, ms);
+ signal?.addEventListener("abort", onAbort, { once: true });
+ });
+}
+
+/** The caller's answer, with a broken check read as "do not give up". */
+async function giveUpSafely(
+ shouldGiveUp: (error: unknown) => boolean | Promise<boolean>,
+ error: unknown,
+): Promise<boolean> {
+ try {
+ return (await shouldGiveUp(error)) === true;
+ } catch {
+ return false;
+ }
+}
+
+/**
+ * `promise`'s answer, or undefined if it has not answered within `ms`. The
+ * promise is left to settle on its own; `giveUpSafely` never rejects.
+ */
+function withinBudget<T>(promise: Promise<T>, ms: number): Promise<T | undefined> {
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const timeout = new Promise<undefined>(resolve => { timer = setTimeout(() => resolve(undefined), ms); });
+ return Promise.race([promise, timeout]).finally(() => clearTimeout(timer));
+}
+
+/** Total time one call may spend on transient statuses, attempts and waits together (the default of `IX_COMMIT_TRANSIENT_RETRY_MS`). */
+export const TRANSIENT_RETRY_BUDGET_MS = 5_000;
+/** Most transient retries one call makes, whatever the budget: a `Retry-After: 0` must not loop. */
+export const TRANSIENT_MAX_RETRIES = 3;
+/** Most re-sends after a 504: the backend may still be working on the first one. */
+export const GATEWAY_TIMEOUT_MAX_RETRIES = 1;
+/** First backoff for a transient status without `Retry-After`; doubles each time. */
+const TRANSIENT_BASE_DELAY_MS = 500;
+
+export interface RetryCommitOptions {
+ /** Conflict and transport retries after the first attempt. 0 means none. Transient statuses are bounded by the budget instead. */
+ maxRetries: number;
+ /** Retry 502/503/504/429 too. Only the bulk commit sites set this. */
+ transient: boolean;
+ /**
+ * Total time allowed for transient retries: every attempt's duration, every
+ * give-up check and the waits. A re-send is made only if one more attempt as long as the last one
+ * still fits; a longer wait is cut to what is left. 0 means no transient retries.
+ */
+ transientBudgetMs?: number;
+ /** The run's deadline. A wait it interrupts ends the loop with the last error. */
+ signal?: AbortSignal;
+ /**
+ * Has the caller given up on the backend? Consulted before a TRANSIENT retry
+ * only: a run whose cutoff has tripped, or whose backend reports its database
+ * down, should not spend its transient budget waiting on it. Conflict and transport
+ * retries keep their existing behaviour.
+ *
+ * It may be async, so a caller can ASK the backend (a short health probe)
+ * rather than only read what it already knows: on the first bulk of a run
+ * nothing has asked yet. It gets the failure that prompted it. A rejection
+ * counts as "keep going", so a probe that breaks never ends the loop.
+ */
+ shouldGiveUp?: (error: unknown) => boolean | Promise<boolean>;
+ sleep?: (ms: number, signal?: AbortSignal) => Promise<void>;
+ random?: () => number;
+ now?: () => number;
+ onRetry?: (info: { attempt: number; delayMs: number; error: unknown; kind: "conflict" | "transient" }) => void;
+}
+
+/**
+ * Send a commit, and send it again while the failure is one a wait can fix.
+ * The last error is rethrown when the retries, the budget or the caller's
+ * patience run out, and at once for anything else.
+ */
+export async function retryCommit<T>(fn: () => Promise<T>, options: RetryCommitOptions): Promise<T> {
+ const {
+ maxRetries,
+ transient,
+ transientBudgetMs = TRANSIENT_RETRY_BUDGET_MS,
+ signal,
+ shouldGiveUp,
+ sleep = abortableSleep,
+ random = Math.random,
+ now = Date.now,
+ onRetry,
+ } = options;
+ // Time charged to the transient budget: every attempt's own duration, every give-up check and
+ // every transient wait.
+ let transientSpentMs = 0;
+ let transientRetries = 0;
+ let gatewayTimeoutRetries = 0;
+ let conflictRetries = 0;
+ for (;;) {
+ const attemptStart = now();
+ try {
+ return await fn();
+ } catch (err) {
+ const attemptMs = Math.max(0, now() - attemptStart);
+ transientSpentMs += attemptMs;
+ if (signal?.aborted) throw err;
+ let delay: number;
+ let kind: "conflict" | "transient";
+ if (isRetryableCommitConflict(err)) {
+ if (conflictRetries >= maxRetries) throw err;
+ // ArangoDB lock conflicts usually clear as soon as the competing transaction
+ // commits. Transport errors may need a bit more breathing room under k8s load.
+ delay = 10 * (1 << conflictRetries) + random() * 10;
+ conflictRetries++;
+ kind = "conflict";
+ } else if (transient && isTransientCommitStatus(err)) {
+ // What is left once the re-send itself is paid for, estimated as
+ // long as the attempt that just failed. Nothing left: no re-send.
+ let remaining = transientBudgetMs - transientSpentMs - attemptMs;
+ if (transientRetries >= TRANSIENT_MAX_RETRIES || remaining <= 0) throw err;
+ const gatewayTimeout = statusOf(err) === 504;
+ if (gatewayTimeout && gatewayTimeoutRetries >= GATEWAY_TIMEOUT_MAX_RETRIES) throw err;
+ if (shouldGiveUp) {
+ // The give-up check may be a health probe that takes seconds. Its
+ // time is charged to the budget like an attempt's, it may take no
+ // longer than what is left (a check that has not answered by then
+ // leaves no room for a re-send anyway), and what is left is worked
+ // out again after it.
+ const probeStart = now();
+ const giveUp = await withinBudget(giveUpSafely(shouldGiveUp, err), remaining);
+ transientSpentMs += Math.max(0, now() - probeStart);
+ if (giveUp !== false) throw err;
+ if (signal?.aborted) throw err;
+ remaining = transientBudgetMs - transientSpentMs - attemptMs;
+ if (remaining <= 0) throw err;
+ }
+ const asked = retryAfterMs(retryAfterOf(err), now());
+ // Jittered by up to a quarter, so the chunks of one run that all met
+ // the same restart do not come back in lockstep. Either wait is cut to
+ // what is left of the budget, so `Retry-After` is honoured only up to it.
+ delay = Math.min(
+ remaining,
+ asked ?? TRANSIENT_BASE_DELAY_MS * 2 ** transientRetries * (1 + random() * 0.25),
+ );
+ transientSpentMs += delay;
+ transientRetries++;
+ if (gatewayTimeout) gatewayTimeoutRetries++;
+ kind = "transient";
+ } else {
+ throw err;
+ }
+ onRetry?.({ attempt: conflictRetries + transientRetries, delayMs: delay, error: err, kind });
+ try {
+ await sleep(delay, signal);
+ } catch {
+ // The deadline fired during the wait. The commit's own failure is the
+ // one worth reporting, and the caller reads the deadline separately.
+ throw err;
+ }
+ }
+ }
+}
diff --git a/ix-cli/src/cli/database-probe.ts b/ix-cli/src/cli/database-probe.ts
new file mode 100644
index 00000000..a57322fb
--- /dev/null
+++ b/ix-cli/src/cli/database-probe.ts
@@ -0,0 +1,121 @@
+// Copyright 2026 Ix Infrastructure Inc.
+
+// ---------------------------------------------------------------------------
+// "Can the backend reach its database?", asked quickly, for a run whose
+// commits are failing (IN-19C, the CL-17 breaker follow-up).
+//
+// The commit cutoff (commit-breaker.ts) judges the backend by behaviour: a
+// streak of refused commits. That is right when the cause is unknown, and slow
+// when it is not: with ArangoDB down, every commit waits out its own failure
+// and the drain spends its whole failure budget, three passes over, before the
+// run stops. The backend knows the answer already. Since Ix-memory 1.0.33,
+// `/v1/health` reports `database: "reachable" | "unreachable"`, and answers 503
+// `degraded` while the database is down. An older backend never touched the
+// database for health, so a read that does (`/v1/revisions/current`) stands in.
+//
+// Only an explicit answer counts. A probe that fails, times out, or is refused
+// for want of the local token is `unknown`, and the caller then does exactly
+// what it would have done without one.
+//
+// The answer only shortens the fan-out (the breaker trips, the rest is held)
+// and the transient waits. It never decides the exit code: the held patches
+// are still sent by the cutoff and the drain, as on main, so a wrong or stale
+// "unreachable" cannot fail a run whose commits would land.
+// ---------------------------------------------------------------------------
+
+import type { HealthResponse } from "../client/types.js";
+
+/**
+ * The two reads a probe makes. `readHealth` is the caller's way to fetch
+ * health, which in the CLI is `readBackendHealth`: health is fetched, and the
+ * release it reports recorded, in exactly one place (backend-version.ts), and
+ * this probe goes through it like every other reader.
+ */
+export interface DatabaseProbeReads {
+ readHealth(timeoutMs: number): Promise<HealthResponse>;
+ currentRevision(timeoutMs: number): Promise<unknown>;
+}
+
+/**
+ * A health answer, including the 503 a backend sends while it cannot reach its
+ * database: that is a server that answered, and the body says what is wrong.
+ * Thrown for anything else.
+ */
+export function healthFromError(err: unknown): HealthResponse | null {
+ const m = /^503:\s*(\{.*\})\s*$/s.exec((err as Error)?.message ?? "");
+ if (!m) return null;
+ try {
+ const body = JSON.parse(m[1]!) as HealthResponse;
+ return typeof body?.status === "string" ? body : null;
+ } catch {
+ return null;
+ }
+}
+
+export type DatabaseVerdict = "reachable" | "unreachable" | "unknown";
+
+/** How long a probe may take. It runs on a failing run's critical path. */
+export const DATABASE_PROBE_TIMEOUT_MS = 2_000;
+
+/**
+ * Ask the backend whether it can reach its database.
+ *
+ * `unreachable` only on the backend's own word: health reporting
+ * `database: "unreachable"` (with a 200, or the 503 `degraded` answer);
+ * anything else, including a failed fallback read on an older backend, is
+ * `unknown`.
+ */
+export async function probeDatabase(
+ reads: DatabaseProbeReads,
+ opts: { timeoutMs?: number } = {},
+): Promise<DatabaseVerdict> {
+ const timeoutMs = opts.timeoutMs ?? DATABASE_PROBE_TIMEOUT_MS;
+ let health: HealthResponse;
+ try {
+ health = await reads.readHealth(timeoutMs);
+ } catch (err) {
+ const degraded = healthFromError(err);
+ if (!degraded) return "unknown";
+ health = degraded;
+ }
+ if (health.database === "unreachable") return "unreachable";
+ if (health.database === "reachable") return "reachable";
+ // A backend older than 1.0.33: health says nothing about the database.
+ try {
+ await reads.currentRevision(timeoutMs);
+ return "reachable";
+ } catch {
+ // A failed read proves nothing about the database: a 5xx may be a saturated
+ // Arango, a proxy or a restarting pod. Without health's own word, stay quiet.
+ return "unknown";
+ }
+}
+
+/**
+ * One probe shared by every caller that asks within `minIntervalMs` of the
+ * last answer, and by everyone who asks while one is in flight. A failing run
+ * reaches the probe once per failed commit, and concurrency 8 means eight at
+ * a time; the backend should be asked once.
+ */
+export function createDatabaseProbe(
+ probe: () => Promise<DatabaseVerdict>,
+ minIntervalMs = 5_000,
+ now: () => number = Date.now,
+): () => Promise<DatabaseVerdict> {
+ let pending: Promise<DatabaseVerdict> | undefined;
+ let last: DatabaseVerdict | undefined;
+ let answeredAt = 0;
+ return () => {
+ if (pending) return pending;
+ if (last !== undefined && now() - answeredAt < minIntervalMs) return Promise.resolve(last);
+ pending = probe()
+ .catch((): DatabaseVerdict => "unknown")
+ .then(verdict => {
+ last = verdict;
+ answeredAt = now();
+ pending = undefined;
+ return verdict;
+ });
+ return pending;
+ };
+}
diff --git a/ix-cli/src/client/__tests__/http-error.test.ts b/ix-cli/src/client/__tests__/http-error.test.ts
new file mode 100644
index 00000000..2a95cee0
--- /dev/null
+++ b/ix-cli/src/client/__tests__/http-error.test.ts
@@ -0,0 +1,42 @@
+// Copyright 2026 Ix Infrastructure Inc.
+
+import { describe, expect, it } from "vitest";
+import { httpError, LocalTokenRequiredError } from "../api.js";
+
+// `httpError`'s message is read back by many callers (status prefixes, error
+// codes in the JSON body), so the headers it now takes for the commit retry
+// (IN-19C) may add properties and must not touch the message.
+describe("httpError", () => {
+ const body = '{"error":"service unavailable"}';
+
+ it("keeps the message byte-identical and carries status and Retry-After when given headers", () => {
+ const err = httpError(503, body, new Headers({ "Retry-After": "2" })) as Error & { status?: number; retryAfter?: string };
+ expect(err.message).toBe(`503: ${body}`);
+ expect(err.message).toBe(httpError(503, body).message);
+ expect(err.status).toBe(503);
+ expect(err.retryAfter).toBe("2");
+ // Non-enumerable: nothing that serialises or spreads an error picks them up.
+ expect(Object.keys(err)).toEqual([]);
+ });
+
+ it("has no Retry-After when the response had none", () => {
+ const err = httpError(502, "bad gateway", new Headers()) as Error & { status?: number; retryAfter?: string };
+ expect(err.status).toBe(502);
+ expect("retryAfter" in err).toBe(false);
+ });
+
+ it("adds nothing without headers", () => {
+ const err = httpError(503, body);
+ expect(err.message).toBe(`503: ${body}`);
+ expect("status" in err).toBe(false);
+ expect("retryAfter" in err).toBe(false);
+ });
+
+ it("still raises LocalTokenRequiredError for the loopback guard's 401", () => {
+ const guard = '{"code":"local_token_required","error":"token required"}';
+ const err = httpError(401, guard, new Headers({ "Retry-After": "1" }));
+ expect(err).toBeInstanceOf(LocalTokenRequiredError);
+ expect(err.message).toBe(`401: ${guard}`);
+ expect("retryAfter" in err).toBe(false);
+ });
+});
diff --git a/ix-cli/src/client/api.ts b/ix-cli/src/client/api.ts
index a9104f7f..f2a50d09 100644
--- a/ix-cli/src/client/api.ts
+++ b/ix-cli/src/client/api.ts
@@ -367,8 +367,8 @@ export class IxClient {
* The body is the head's revision record when the backend has one, and the
* bare revision number when it does not; both are returned as-is.
*/
- async currentRevision(): Promise<unknown> {
- return this.get("/v1/revisions/current");
+ async currentRevision(timeoutMs?: number): Promise<unknown> {
+ return this.get("/v1/revisions/current", timeoutMs);
}
async diff(
@@ -405,7 +405,7 @@ export class IxClient {
});
if (!resp.ok) {
const text = await resp.text();
- throw httpError(resp.status, text);
+ throw httpError(resp.status, text, resp.headers);
}
return resp.json() as Promise<PatchCommitResult>;
}
@@ -487,7 +487,7 @@ export class IxClient {
});
if (!resp.ok) {
const text = await resp.text();
- throw httpError(resp.status, text);
+ throw httpError(resp.status, text, resp.headers);
}
return resp.json() as Promise<PatchCommitResult>;
}
@@ -779,8 +779,9 @@ export class IxClient {
return this.get(`/v1/stats${qs ? `?${qs}` : ""}`, opts?.timeoutMs);
}
- async health(): Promise<HealthResponse> {
- return this.get("/v1/health");
+ /** `timeoutMs` bounds a probe that must answer in seconds; the default is unchanged. */
+ async health(timeoutMs?: number): Promise<HealthResponse> {
+ return this.get("/v1/health", timeoutMs);
}
async capabilities(): Promise<CapabilitiesResponse> {
@@ -898,15 +899,37 @@ export class LocalTokenRequiredError extends Error {
}
}
-/** The error for a non-2xx response: `<status>: <body>`, or a typed one we can explain. */
-export function httpError(status: number, body: string): Error {
+/** What `httpError` adds to its Error when it is handed the response headers. */
+export interface HttpErrorDetail {
+ /** The HTTP status, the same number the message starts with. */
+ readonly status?: number;
+ /** The raw `Retry-After` header, when the response carried one. */
+ readonly retryAfter?: string;
+}
+
+/**
+ * The error for a non-2xx response: `<status>: <body>`, or a typed one we can explain.
+ *
+ * With `headers`, the Error also carries `status` and, when the response had
+ * one, `retryAfter` (see `HttpErrorDetail`), both non-enumerable. The message
+ * is the same either way: many callers read the status and the error code back
+ * out of it, so it must not change shape. Only the commit routes pass headers,
+ * for the ingest's transient-status retry (`cli/commit-retry.ts`).
+ */
+export function httpError(status: number, body: string, headers?: Pick<Headers, "get">): Error {
if (status === 401) {
try {
const parsed = JSON.parse(body) as { code?: unknown } | null;
if (parsed?.code === "local_token_required") return new LocalTokenRequiredError(body);
} catch { /* not the guard's JSON: a plain 401 */ }
}
- return new Error(`${status}: ${errorBodyForMessage(body)}`);
+ const err = new Error(`${status}: ${errorBodyForMessage(body)}`);
+ if (headers !== undefined) {
+ Object.defineProperty(err, "status", { value: status, enumerable: false });
+ const retryAfter = headers.get("retry-after");
+ if (retryAfter !== null) Object.defineProperty(err, "retryAfter", { value: retryAfter, enumerable: false });
+ }
+ return err;
}
/**