Understand any codebase instantly. System intelligence for codebases, built for humans and AI.
TypeScript aiai-memoryclaude-codeclicode-analysiscode-mappingcode-memorycodebase-understandingcodexdeveloper-toolsllmopenclawpersistent-memoryprogram-analysissoftware-architecturesystem-intelligencesystem-mapping
fix(ingest): retry transient 5xx/429 bulk commits with Retry-After, and the CL-17 breaker fixes (IN-19C) (#845)
Ian Hockd4f9e4fd9fdd
diff --git a/docs-site/src/content/docs/guides/troubleshooting.md b/docs-site/src/content/docs/guides/troubleshooting.mdindex 0bdae529..8556f616 100644--- a/docs-site/src/content/docs/guides/troubleshooting.md+++ b/docs-site/src/content/docs/guides/troubleshooting.md@@ -31,8 +31,11 @@ Set `IX_DEBUG=1` to see full stack traces on any error.## `ix map` stops with "Commits against … kept failing"-After several consecutive failed commits, `ix map` stops instead of sending one doomed request per file. Read the-`Last error` line it prints. The backend returns the same error for two different causes:+After several consecutive failed commits, `ix map` stops instead of sending one doomed request per file. It stops+after the first failure if the backend's health check reports that it cannot reach its database (ArangoDB); the message+then says so, and `ix doctor` shows the same. Start the database, or wait for it to come back, and re-map.++Otherwise, read the `Last error` line it prints. The backend returns the same error for two different causes:- **The database is saturated.** `ix doctor` can still pass. Check `docker stats`, wait for ArangoDB to go idle, andre-map.diff --git a/docs-site/src/content/docs/reference/configuration.md b/docs-site/src/content/docs/reference/configuration.mdindex 71247fba..a28691ac 100644--- a/docs-site/src/content/docs/reference/configuration.md+++ b/docs-site/src/content/docs/reference/configuration.md@@ -21,9 +21,11 @@ ix config set format llm # set a value| `IX_FORMAT` | Default output format: `text`, `json` or `llm`. Overrides the config file. `--format` overrides it || `IX_TOKEN=<token>` | Bearer token sent to a local backend (localhost or another loopback address) on every request. Overrides the stored local token (see [Local token](#local-token)). Never sent to any other endpoint || `IX_DEBUG=1` | Prints full stack traces on errors, and one stderr line per backend request with the `X-Correlation-Id` it sent (the backend tags its request and slow-query logs with it). Every request of one `ix` run shares the id's prefix |-| `IX_READ_DEADLINE_MS=<ms>` | How long one command's reads may take in all before it fails (default 60000), so a hung backend fails a query in a minute rather than after each request's 2-minute timeout. The clock restarts after each write the command makes. `ix map`, `ix ingest` and `ix stats` have no read deadline. `0` turns it off. A read that meets a refused or dropped connection, or a 502/503/504, is retried once; writes never are |+| `IX_READ_DEADLINE_MS=<ms>` | How long one command's reads may take in all before it fails (default 60000), so a hung backend fails a query in a minute rather than after each request's 2-minute timeout. The clock restarts after each write the command makes. `ix map`, `ix ingest` and `ix stats` have no read deadline. `0` turns it off. A read that meets a refused or dropped connection, or a 502/503/504, is retried once; writes are not, except as `IX_COMMIT_CONFLICT_RETRIES` and `IX_COMMIT_TRANSIENT_RETRY_MS` describe || `IX_MCP_SUBPROCESS=1` | Runs each MCP tool call in its own `ix` process |-| `IX_COMMIT_FAILURE_LIMIT=<n>` | Consecutive failed commits before `ix map` stops (default 5). `0` never stops |+| `IX_COMMIT_FAILURE_LIMIT=<n>` | Consecutive failed commits before `ix map` stops (default 5). It stops after one when the backend reports its database unreachable. `0` never stops |+| `IX_COMMIT_CONFLICT_RETRIES=<n>` | Retries of one commit after a lock conflict or a dropped connection (default 6). `0` turns these retries off. It does not affect the 502/503/504/429 retries; see `IX_COMMIT_TRANSIENT_RETRY_MS` |+| `IX_COMMIT_TRANSIENT_RETRY_MS=<ms>` | The total time one bulk commit may spend retrying after a 502, 503, 504 or 429, counting both the requests and the waits between them (default 5000, at most three retries, and at most one after a 504). A bulk is sent again only if another request as long as the one that failed still fits, so a slow bulk goes straight to the per-file fallback. `Retry-After` is honoured up to that limit; a longer value is cut to it. `0` turns these retries off || `IX_COMMIT_BASE_REV_RETRIES=<n>` | Re-sends of a commit after another writer on the same backend moved the graph revision under it (default 8) || `IX_PARSE_BUDGET_MS=<ms>` | How long `ix map` / `ix ingest` may spend parsing one file before skipping it and naming it in the summary (default 10000). A skipped file is tried again on every run, and `ix status` warns about it until it is in the graph. `0` turns the budget off || `IX_PARSE_WORKERS=<n>` | Most parse workers `ix map` / `ix ingest` starts (default 8, never more than the cores less one). Workers start as files need them, one per 50 files |diff --git a/docs/api/openapi.yaml b/docs/api/openapi.yamlindex dafa21e7..7eb22513 100644--- a/docs/api/openapi.yaml+++ b/docs/api/openapi.yaml@@ -47,6 +47,12 @@ paths:application/json:schema:$ref: "#/components/schemas/HealthResponse"+ "503":+ description: The backend is up but cannot reach its database (status degraded, database unreachable)+ content:+ application/json:+ schema:+ $ref: "#/components/schemas/HealthResponse"/v1/capabilities:get:@@ -1057,6 +1063,10 @@ components:status: { type: string }schema_version: { type: integer, description: On-disk graph format version }release_version: { type: string, description: Backend release semver }+ database:+ type: string+ enum: [reachable, unreachable]+ description: Whether the backend can reach its database (1.0.33+). Absent on older backendsCapabilitiesResponse:type: objectproperties:diff --git a/ix-cli/src/cli/__tests__/backend-release-report.test.ts b/ix-cli/src/cli/__tests__/backend-release-report.test.tsindex 07b2ee4e..405b6149 100644--- a/ix-cli/src/cli/__tests__/backend-release-report.test.ts+++ b/ix-cli/src/cli/__tests__/backend-release-report.test.ts@@ -280,6 +280,21 @@ describe("fetchBackendHealth records what it read", () => {expect(tracked()).toBe("1.0.16");});+ it("passes a probe's timeout through readBackendHealth, and still records", async () => {+ // The ingest's database probe needs a short timeout. It reaches health+ // through here rather than calling client.health(ms) itself.+ writeFileSync(stamp(), "1.0.13");+ writeFileSync(+ join(home, ".version-check.json"),+ JSON.stringify({ latest: "0.9.3", backendLatest: "1.0.16", checkedAt: Date.now() }),+ );+ const health = vi.fn(async (_timeoutMs?: number) => ({ status: "ok", release_version: "1.0.16" }));+ const { readBackendHealth } = await import("../commands/upgrade.js");+ await readBackendHealth({ endpoint: "http://localhost:8090", health } as unknown as IxClient, 1234);+ expect(health).toHaveBeenCalledWith(1234);+ expect(tracked()).toBe("1.0.16");+ });+it("applies the ceiling through readBackendHealth, the path commands take", async () => {// The JOIN, not the pieces. backendCeiling() is pinned on its own and// fetchBackendHealth's use of knownLatest is pinned on its own — but@@ -492,7 +507,9 @@ describe("health is fetched in exactly one place", () => {// comments are left alone for the same reason: a `**/*.ts` glob literal ate// from the string to the next `*/` and dropped the chokepoint itself..map((f) => ({ ...f, src: f.src.replace(/^\s*(\/\/|\*|\/\*).*$/gm, "") }))- .filter((f) => /\.health\s*\(\s*\)/.test(f.src))+ // Any arguments: `client.health(timeoutMs)` is as much a health read as+ // `client.health()`, and an empty-parens-only pattern let one slip past.+ .filter((f) => /\.health\s*\(/.test(f.src)).map((f) => f.file.slice(SRC.length + 1).replace(/\\/g, "/"));it("finds the health callers at all", () => {diff --git a/ix-cli/src/cli/__tests__/commit-breaker.test.ts b/ix-cli/src/cli/__tests__/commit-breaker.test.tsindex b441104e..9cb10163 100644--- a/ix-cli/src/cli/__tests__/commit-breaker.test.ts+++ b/ix-cli/src/cli/__tests__/commit-breaker.test.ts@@ -124,6 +124,26 @@ describe("createCommitBreaker", () => {});});+describe("trip, which the database probe uses (IN-19C)", () => {+ it("latches tripped and everTripped at once, and records the error", () => {+ const b = createCommitBreaker(5);+ const err = new Error("500: Connection refused: arangodb:8529");+ b.trip(err);+ expect(b.tripped()).toBe(true);+ expect(b.everTripped()).toBe(true);+ expect(b.lastError()).toBe(err);+ expect(b.consecutiveFailures(), "no streak was spent").toBe(0);+ });++ it("is cleared by a success like any trip, and everTripped stays", () => {+ const b = createCommitBreaker(5);+ b.trip(new Error("x"));+ b.recordSuccess();+ expect(b.tripped()).toBe(false);+ expect(b.everTripped()).toBe(true);+ });+});+describe("everTripped", () => {it("is false until the limit is reached", () => {const b = createCommitBreaker(3);@@ -596,6 +616,35 @@ describe("describeCommitCutoff", () => {expect(describeCommitCutoff(b, "e", 0, false)).toContain("400 patches were not sent");});+ it("says the graph is still empty, not unchanged, when it held none of these files (IN-19C)", () => {+ // After `ix reset` or `reset --ingest` there is no graph to be unchanged,+ // and "The graph is unchanged" reads as "your old graph is still there".+ const b = createCommitBreaker(1);+ b.recordFailure(new Error("500: transaction begin timeout"));+ b.recordSkipped(3);++ const empty = describeCommitCutoff(b, "e", 0, false, { graphStartedEmpty: true });+ expect(empty).not.toContain("unchanged");+ expect(empty).toContain("this workspace's graph is still empty");+ // Only with nothing landed: a partial run is described as before.+ const partial = describeCommitCutoff(b, "e", 4, false, { graphStartedEmpty: true });+ expect(partial).toContain("4 other patches did land");+ // And the defaults keep the old text.+ expect(describeCommitCutoff(b, "e", 0)).toContain("The graph is unchanged.");+ expect(describeCommitCutoff(b, "e", 0, false, {})).toContain("The graph is unchanged.");+ });++ it("names the database, and drops the `ix doctor` still passes line, when the backend reported it down", () => {+ const b = createCommitBreaker(1);+ b.trip(new Error("500: Connection refused: arangodb:8529"));++ const msg = describeCommitCutoff(b, "e", 0, false, { databaseUnreachable: true });+ expect(msg).toContain("cannot reach its database (ArangoDB)");+ expect(msg).not.toContain("still passes");+ expect(msg).not.toContain("Either the database cannot keep up");+ expect(describeCommitCutoff(b, "e", 0)).toContain("`ix doctor` still passes");+ });+it("says 'patch was' for one and 'patches were' for many", () => {const one = createCommitBreaker(1);one.recordFailure(new Error("x"));@@ -614,3 +663,12 @@ describe("describeCommitCutoff", () => {expect(msg).toContain("…");});});++describe("trip() with the breaker disabled", () => {+ it("does not latch when the limit is 0", () => {+ const b = createCommitBreaker(0);+ b.trip(new Error("db down"));+ expect(b.tripped()).toBe(false);+ expect(b.everTripped()).toBe(false);+ });+});diff --git a/ix-cli/src/cli/__tests__/commit-retry.test.ts b/ix-cli/src/cli/__tests__/commit-retry.test.tsnew file mode 100644index 00000000..7971f748--- /dev/null+++ b/ix-cli/src/cli/__tests__/commit-retry.test.ts@@ -0,0 +1,372 @@+// Copyright 2026 Ix Infrastructure Inc.++import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";+import { GATEWAY_TIMEOUT_MAX_RETRIES, isTransientCommitStatus, retryAfterMs, retryCommit, TRANSIENT_RETRY_BUDGET_MS } from "../commit-retry.js";+import { httpError } from "../../client/api.js";++/** An error shaped the way `httpError` builds one for a commit route. */+function statusError(status: number, retryAfter?: string): Error {+ const headers = new Headers(retryAfter !== undefined ? { "retry-after": retryAfter } : {});+ return httpError(status, JSON.stringify({ error: `${status}: nope` }), headers);+}++/** `fn` that fails with each of `failures` in turn, then answers "ok". */+function failing(...failures: unknown[]): { fn: () => Promise<string>; calls: () => number } {+ let n = 0;+ return {+ fn: async () => {+ const failure = failures[n++];+ if (failure !== undefined) throw failure;+ return "ok";+ },+ calls: () => n,+ };+}++describe("isTransientCommitStatus", () => {+ it("is true for 502, 503, 504 and 429, from the status or the message", () => {+ for (const status of [502, 503, 504, 429]) {+ expect(isTransientCommitStatus(statusError(status)), String(status)).toBe(true);+ expect(isTransientCommitStatus(new Error(`${status}: body`)), `${status} message`).toBe(true);+ }+ });++ it("is false for 500, 413, 409, 401 and anything without a status", () => {+ for (const status of [500, 413, 409, 401, 400]) {+ expect(isTransientCommitStatus(statusError(status)), String(status)).toBe(false);+ }+ expect(isTransientCommitStatus(new Error("fetch failed"))).toBe(false);+ expect(isTransientCommitStatus("503")).toBe(false);+ });++ it("is never true for an abort", () => {+ const abort = new Error("503: upstream aborted");+ expect(isTransientCommitStatus(abort)).toBe(false);+ expect(isTransientCommitStatus(new DOMException("timed out", "TimeoutError"))).toBe(false);+ });+});++describe("retryAfterMs", () => {+ const now = Date.parse("Sun, 11 Oct 2026 12:00:00 GMT");++ it("reads delta-seconds", () => {+ expect(retryAfterMs("2", now)).toBe(2000);+ expect(retryAfterMs(" 0 ", now)).toBe(0);+ });++ it("reads an IMF-fixdate, and clamps a past one to 0", () => {+ expect(retryAfterMs("Sun, 11 Oct 2026 12:00:03 GMT", now)).toBe(3000);+ expect(retryAfterMs("Sun, 11 Oct 2026 11:59:00 GMT", now)).toBe(0);+ });++ it("is undefined for absent or unreadable values", () => {+ expect(retryAfterMs(undefined, now)).toBeUndefined();+ expect(retryAfterMs("", now)).toBeUndefined();+ expect(retryAfterMs("soon", now)).toBeUndefined();+ expect(retryAfterMs("-1", now)).toBeUndefined();+ });+});++describe("retryCommit", () => {+ beforeEach(() => { vi.useFakeTimers(); });+ afterEach(() => { vi.useRealTimers(); });++ const transient = { maxRetries: 6, transient: true, random: () => 0 } as const;++ it("a 503 then a 200 succeeds after exactly one retry, once the backoff has passed", async () => {+ const { fn, calls } = failing(statusError(503));+ let settled = false;+ const result = retryCommit(fn, transient).then(v => { settled = true; return v; });++ await vi.advanceTimersByTimeAsync(499);+ expect(calls()).toBe(1);+ expect(settled).toBe(false);+ await vi.advanceTimersByTimeAsync(1);+ await expect(result).resolves.toBe("ok");+ expect(calls()).toBe(2);+ });++ it("a 429 with Retry-After: 2 waits 2 s, not the backoff", async () => {+ const { fn, calls } = failing(statusError(429, "2"));+ const result = retryCommit(fn, transient);++ await vi.advanceTimersByTimeAsync(1999);+ expect(calls(), "no second call before 2 s").toBe(1);+ await vi.advanceTimersByTimeAsync(1);+ await expect(result).resolves.toBe("ok");+ expect(calls()).toBe(2);+ });++ it("a Retry-After HTTP-date waits until that date, and a past one retries at once", async () => {+ vi.setSystemTime(Date.parse("Sun, 11 Oct 2026 12:00:00 GMT"));+ const ahead = failing(statusError(503, "Sun, 11 Oct 2026 12:00:03 GMT"));+ const first = retryCommit(ahead.fn, transient);+ await vi.advanceTimersByTimeAsync(2999);+ expect(ahead.calls()).toBe(1);+ await vi.advanceTimersByTimeAsync(1);+ await expect(first).resolves.toBe("ok");++ const past = failing(statusError(503, "Sun, 11 Oct 2026 11:00:00 GMT"));+ const second = retryCommit(past.fn, transient);+ await vi.advanceTimersByTimeAsync(0);+ await expect(second).resolves.toBe("ok");+ expect(past.calls()).toBe(2);+ });++ it("caps transient retries at three, inside about 5 s by default", async () => {+ const waits: number[] = [];+ const always503 = vi.fn(async () => { throw statusError(503); });+ const result = retryCommit(always503, {+ maxRetries: 100,+ transient: true,+ random: () => 0,+ sleep: async ms => { waits.push(ms); },+ });++ await expect(result).rejects.toThrow(/^503:/);+ // 500 + 1000 + 2000 = 3.5 s, under the 5 s default; the cap of three stops it.+ expect(waits).toEqual([500, 1000, 2000]);+ expect(always503).toHaveBeenCalledTimes(4);+ });++ it("cuts the last backoff to what is left of a smaller budget", async () => {+ const waits: number[] = [];+ const always503 = vi.fn(async () => { throw statusError(503); });+ await expect(retryCommit(always503, {+ maxRetries: 6,+ transient: true,+ transientBudgetMs: 2_000,+ random: () => 0,+ sleep: async ms => { waits.push(ms); },+ })).rejects.toThrow(/^503:/);+ expect(waits).toEqual([500, 1000, 500]);+ expect(waits.reduce((a, b) => a + b, 0)).toBe(2_000);+ });++ it("honours a Retry-After only up to the budget, and never waits past it", async () => {+ const waits: number[] = [];+ const { fn, calls } = failing(statusError(503, "60"), statusError(503, "60"));+ await expect(retryCommit(fn, { ...transient, sleep: async ms => { waits.push(ms); } })).rejects.toThrow(/^503:/);+ expect(waits, "60 s cut to the 5 s budget, then nothing left").toEqual([5_000]);+ expect(calls()).toBe(2);+ });++ it("a Retry-After of 0 still stops after three retries", async () => {+ const always = vi.fn(async () => { throw statusError(429, "0"); });+ await expect(retryCommit(always, { ...transient, sleep: async () => {} })).rejects.toThrow(/^429:/);+ expect(always).toHaveBeenCalledTimes(4);+ });++ it("transientBudgetMs 0 makes exactly one attempt for a 503", async () => {+ const sleep = vi.fn(async () => {});+ const { fn, calls } = failing(statusError(503));+ await expect(retryCommit(fn, { ...transient, transientBudgetMs: 0, sleep })).rejects.toThrow(/^503:/);+ expect(sleep).not.toHaveBeenCalled();+ expect(calls()).toBe(1);+ });++ it("maxRetries 0 stops conflict retries only; a 503 is still retried on the transient budget", async () => {+ const conflict = failing(new Error("500: write-write conflict"));+ await expect(retryCommit(conflict.fn, { maxRetries: 0, transient: true })).rejects.toThrow(/write-write/);+ expect(conflict.calls()).toBe(1);++ const unavailable = failing(statusError(503, "0"));+ await expect(retryCommit(unavailable.fn, { maxRetries: 0, transient: true, sleep: async () => {} })).resolves.toBe("ok");+ expect(unavailable.calls()).toBe(2);+ });++ it("still retries a lock conflict on its old short backoff", async () => {+ const { fn, calls } = failing(new Error("500: write-write conflict"));+ const result = retryCommit(fn, { maxRetries: 6, transient: false, random: () => 0 });+ await vi.advanceTimersByTimeAsync(10);+ await expect(result).resolves.toBe("ok");+ expect(calls()).toBe(2);+ });++ it("never retries 500, 413 or 409", async () => {+ for (const status of [500, 413, 409]) {+ const { fn, calls } = failing(statusError(status));+ await expect(retryCommit(fn, transient), String(status)).rejects.toThrow(new RegExp(`^${status}:`));+ expect(calls(), String(status)).toBe(1);+ }+ });++ it("does not retry a transient status unless asked to", async () => {+ const { fn, calls } = failing(statusError(503, "0"));+ await expect(retryCommit(fn, { maxRetries: 6, transient: false })).rejects.toThrow(/^503:/);+ expect(calls()).toBe(1);+ });++ it("does not retry an abort", async () => {+ const { fn, calls } = failing(new DOMException("This operation was aborted", "AbortError"));+ await expect(retryCommit(fn, transient)).rejects.toThrow(/aborted/);+ expect(calls()).toBe(1);+ });++ it("an abort during the wait rejects promptly, with the commit's own error", async () => {+ const deadline = new AbortController();+ const { fn, calls } = failing(statusError(503, "20"));+ const result = retryCommit(fn, { ...transient, signal: deadline.signal });+ const assertion = expect(result).rejects.toThrow(/^503:/);+ await vi.advanceTimersByTimeAsync(100);+ deadline.abort();+ await assertion;+ expect(calls()).toBe(1);+ });++ it("stops retrying a transient status once the caller has given up", async () => {+ const { fn, calls } = failing(statusError(503, "0"));+ await expect(retryCommit(fn, { ...transient, shouldGiveUp: () => true })).rejects.toThrow(/^503:/);+ expect(calls()).toBe(1);+ });++ it("waits for an async give-up check, and hands it the failure", async () => {+ const { fn, calls } = failing(statusError(503, "0"));+ const shouldGiveUp = vi.fn(async (_err: unknown) => true);+ await expect(retryCommit(fn, { ...transient, shouldGiveUp })).rejects.toThrow(/^503:/);+ expect(calls()).toBe(1);+ expect(String(shouldGiveUp.mock.calls[0][0])).toMatch(/^Error: 503:/);+ });++ it("keeps retrying when the give-up check itself fails", async () => {+ const { fn, calls } = failing(statusError(503, "0"));+ const result = retryCommit(fn, {+ ...transient,+ shouldGiveUp: async () => { throw new Error("probe broke"); },+ });+ await vi.advanceTimersByTimeAsync(0);+ await expect(result).resolves.toBe("ok");+ expect(calls()).toBe(2);+ });++ it("reports each retry", async () => {+ const onRetry = vi.fn();+ const { fn } = failing(statusError(503, "1"));+ const result = retryCommit(fn, { ...transient, onRetry });+ await vi.advanceTimersByTimeAsync(1000);+ await result;+ expect(onRetry).toHaveBeenCalledWith(expect.objectContaining({ attempt: 1, delayMs: 1000, kind: "transient" }));+ });+});++// F1 of the #845 review: the budget used to bound only the sleeps, so a 504 on+// a bulk that took minutes could be sent three more times. It now bounds the+// requests too, and a 504 gets at most one re-send.+describe("retryCommit: the transient budget bounds total time", () => {+ beforeEach(() => { vi.useFakeTimers(); });+ afterEach(() => { vi.useRealTimers(); });++ /** `fn` whose every attempt takes `ms` on the (fake) clock, then fails with `failures` in turn, then answers `answer`. */+ function slow<T>(ms: number, failures: unknown[], answer: T): { fn: () => Promise<T>; calls: () => number } {+ let n = 0;+ return {+ fn: () => new Promise<T>((resolve, reject) => {+ const failure = failures[n++];+ setTimeout(() => (failure !== undefined ? reject(failure) : resolve(answer)), ms);+ }),+ calls: () => n,+ };+ }++ /** Run `retryCommit` to completion on the fake clock; the outcome and the time it took. */+ async function run<T>(fn: () => Promise<T>, budget = TRANSIENT_RETRY_BUDGET_MS, shouldGiveUp?: () => Promise<boolean>) {+ const start = Date.now();+ const settled = retryCommit(fn, { maxRetries: 6, transient: true, transientBudgetMs: budget, random: () => 0, shouldGiveUp })+ .then(value => ({ value, error: undefined as unknown }), (error: unknown) => ({ value: undefined, error }))+ .then(outcome => ({ ...outcome, elapsed: Date.now() - start }));+ await vi.advanceTimersByTimeAsync(10 * 60_000);+ return settled;+ }++ it("never re-sends a slow 504 bulk: one request, then straight to the caller's fallback", async () => {+ // A proxy timing out a five-minute bulk. The backend is likely still applying it.+ const { fn, calls } = slow(300_000, [statusError(504), statusError(504), statusError(504), statusError(504)], "ok");+ const outcome = await run(fn);+ expect(String(outcome.error)).toMatch(/^Error: 504:/);+ expect(calls(), "no full re-send of a slow bulk").toBe(1);+ expect(outcome.elapsed).toBe(300_000);+ });++ it("re-sends a fast 504 at most once, with budget left for more", async () => {+ expect(GATEWAY_TIMEOUT_MAX_RETRIES).toBe(1);+ const { fn, calls } = slow(10, [statusError(504), statusError(504), statusError(504)], "ok");+ const outcome = await run(fn);+ expect(String(outcome.error)).toMatch(/^Error: 504:/);+ expect(calls()).toBe(2);+ });++ it("a 503 after a 504 is still bounded by the one-504 rule only for 504s", async () => {+ const { fn, calls } = slow(10, [statusError(504), statusError(503)], "ok");+ const outcome = await run(fn);+ expect(outcome.value).toBe("ok");+ expect(calls()).toBe(3);+ });++ it("a 504 the backend applied comes back Idempotent on the re-send, and that is the answer", async () => {+ const { fn, calls } = slow(10, [statusError(504)], { rev: 7, status: "Idempotent" });+ const outcome = await run(fn);+ expect(outcome.value).toEqual({ rev: 7, status: "Idempotent" });+ expect(calls()).toBe(2);+ });++ it("a 504 the backend partly applied comes back 409 on the re-send, which is thrown to the partial-commit handling", async () => {+ const partial = httpError(409, JSON.stringify({+ error: "conflict",+ message: "bulk request is partially committed (1/2 patch IDs)",+ committed_patch_ids: ["a"],+ }), new Headers());+ const { fn, calls } = slow(10, [statusError(504), partial], "ok");+ const outcome = await run(fn);+ expect(outcome.error).toBe(partial);+ expect(calls()).toBe(2);+ });++ it("charges each request to the budget: two 1.5 s attempts and a wait, not four", async () => {+ const { fn, calls } = slow(1_500, Array(10).fill(statusError(503)), "ok");+ const outcome = await run(fn);+ expect(String(outcome.error)).toMatch(/^Error: 503:/);+ // 1.5 s + 0.5 s wait + 1.5 s = 3.5 s; a third attempt would end at 6.5 s.+ expect(calls()).toBe(2);+ expect(outcome.elapsed).toBe(3_500);+ });++ it.each([0, 50, 400, 900, 1_500, 2_400, 3_000, 4_999, 5_000, 20_000, 300_000])(+ "never runs past the budget once a retry is involved (attempts of %i ms)",+ async attemptMs => {+ const { fn, calls } = slow(attemptMs, Array(10).fill(statusError(503)), "ok");+ const outcome = await run(fn);+ expect(outcome.error).toBeDefined();+ // A single attempt is whatever the request takes; anything retried fits the budget.+ if (calls() > 1) expect(outcome.elapsed).toBeLessThanOrEqual(TRANSIENT_RETRY_BUDGET_MS);+ else expect(outcome.elapsed).toBe(attemptMs);+ expect(calls()).toBeLessThanOrEqual(4);+ },+ );++ /** A give-up check that takes `ms` on the (fake) clock and then says "keep going". */+ const slowProbe = (ms: number) => () => new Promise<boolean>(resolve => setTimeout(() => resolve(false), ms));++ it("charges the give-up check to the budget: 100 ms attempts and a 2 s probe stay inside 5 s", async () => {+ const { fn, calls } = slow(100, Array(10).fill(statusError(503)), "ok");+ const outcome = await run(fn, TRANSIENT_RETRY_BUDGET_MS, slowProbe(2_000));+ expect(String(outcome.error)).toMatch(/^Error: 503:/);+ // 0.1 + 2 probe + 0.5 wait + 0.1 + 2 probe + 0.2 wait (all that is left) + 0.1 = 5 s.+ expect(calls()).toBe(3);+ expect(outcome.elapsed).toBeLessThanOrEqual(TRANSIENT_RETRY_BUDGET_MS);+ });++ it("a give-up check slower than what is left of the budget is cut off there, with no re-send", async () => {+ const { fn, calls } = slow(100, Array(10).fill(statusError(503)), "ok");+ const outcome = await run(fn, TRANSIENT_RETRY_BUDGET_MS, slowProbe(60_000));+ expect(String(outcome.error)).toMatch(/^Error: 503:/);+ expect(calls()).toBe(1);+ expect(outcome.elapsed).toBe(TRANSIENT_RETRY_BUDGET_MS - 100);+ });++ it("a request longer than the whole budget is never re-sent, even for a 503", async () => {+ const { fn, calls } = slow(TRANSIENT_RETRY_BUDGET_MS, [statusError(503, "0")], "ok");+ const outcome = await run(fn);+ expect(outcome.error).toBeDefined();+ expect(calls()).toBe(1);+ });+});diff --git a/ix-cli/src/cli/__tests__/database-probe.test.ts b/ix-cli/src/cli/__tests__/database-probe.test.tsnew file mode 100644index 00000000..42b1cb44--- /dev/null+++ b/ix-cli/src/cli/__tests__/database-probe.test.ts@@ -0,0 +1,105 @@+// Copyright 2026 Ix Infrastructure Inc.++import { describe, expect, it, vi } from "vitest";+import { createDatabaseProbe, probeDatabase, type DatabaseVerdict } from "../database-probe.js";+import { httpError, LocalTokenRequiredError } from "../../client/api.js";+import type { HealthResponse } from "../../client/types.js";++/** A client whose health and revision reads answer as told. */+function client(health: () => Promise<HealthResponse>, revision: () => Promise<unknown> = async () => 1) {+ return { readHealth: vi.fn(health), currentRevision: vi.fn(revision) };+}++describe("probeDatabase", () => {+ it("is unreachable on the 503 degraded answer of a 1.0.33+ backend", async () => {+ const c = client(async () => { throw httpError(503, '{"status":"degraded","database":"unreachable"}'); });+ expect(await probeDatabase(c)).toBe("unreachable");+ expect(c.currentRevision).not.toHaveBeenCalled();+ });++ it("is unreachable on a 200 that reports the database unreachable", async () => {+ expect(await probeDatabase(client(async () => ({ status: "ok", database: "unreachable" })))).toBe("unreachable");+ });++ it("is reachable when health says so, without a second request", async () => {+ const c = client(async () => ({ status: "ok", database: "reachable" }));+ expect(await probeDatabase(c)).toBe("reachable");+ expect(c.currentRevision).not.toHaveBeenCalled();+ });++ it("passes its short timeout to both requests", async () => {+ const c = client(async () => ({ status: "ok" }));+ await probeDatabase(c, { timeoutMs: 1234 });+ expect(c.readHealth).toHaveBeenCalledWith(1234);+ expect(c.currentRevision).toHaveBeenCalledWith(1234);+ });++ describe("on a backend whose health does not report the database", () => {+ const old = (revision: () => Promise<unknown>) => client(async () => ({ status: "ok" }), revision);++ it("is reachable when the current revision reads", async () => {+ expect(await probeDatabase(old(async () => ({ rev: 4 })))).toBe("reachable");+ });++ it("is unknown when that read fails with a 5xx (not proof of a database outage)", async () => {+ expect(await probeDatabase(old(async () => { throw httpError(500, '{"error":"arango down"}'); }))).toBe("unknown");+ });++ it("is unknown when the read is refused for the token, times out, is a 4xx, or never connects", async () => {+ const failures: unknown[] = [+ new LocalTokenRequiredError('{"code":"local_token_required"}'),+ new DOMException("The operation timed out", "TimeoutError"),+ httpError(404, '{"error":"not found"}'),+ new TypeError("fetch failed"),+ ];+ for (const failure of failures) {+ expect(await probeDatabase(old(async () => { throw failure; })), String(failure)).toBe("unknown");+ }+ });+ });++ it("is unknown when health itself fails, times out or is refused", async () => {+ const failures: unknown[] = [+ new TypeError("fetch failed"),+ new DOMException("The operation timed out", "TimeoutError"),+ new LocalTokenRequiredError('{"code":"local_token_required"}'),+ httpError(503, "<html>bad gateway</html>"),+ ];+ for (const failure of failures) {+ const c = client(async () => { throw failure; });+ expect(await probeDatabase(c), String(failure)).toBe("unknown");+ expect(c.currentRevision).not.toHaveBeenCalled();+ }+ });+});++describe("createDatabaseProbe", () => {+ it("asks once for everyone in flight, and reuses the answer for the interval", async () => {+ let clock = 0;+ let release: (v: DatabaseVerdict) => void = () => {};+ const ask = vi.fn(() => new Promise<DatabaseVerdict>(resolve => { release = resolve; }));+ const probe = createDatabaseProbe(ask, 3_000, () => clock);++ const a = probe();+ const b = probe();+ release("unreachable");+ expect(await a).toBe("unreachable");+ expect(await b).toBe("unreachable");+ expect(ask).toHaveBeenCalledTimes(1);++ clock = 2_999;+ expect(await probe()).toBe("unreachable");+ expect(ask).toHaveBeenCalledTimes(1);++ clock = 3_000;+ const c = probe();+ release("reachable");+ expect(await c).toBe("reachable");+ expect(ask).toHaveBeenCalledTimes(2);+ });++ it("turns a probe that throws into unknown", async () => {+ const probe = createDatabaseProbe(async () => { throw new Error("boom"); });+ expect(await probe()).toBe("unknown");+ });+});diff --git a/ix-cli/src/cli/__tests__/env-durations.test.ts b/ix-cli/src/cli/__tests__/env-durations.test.tsindex bb510516..052c26c0 100644--- a/ix-cli/src/cli/__tests__/env-durations.test.ts+++ b/ix-cli/src/cli/__tests__/env-durations.test.ts@@ -1,7 +1,7 @@// Copyright 2026 Ix Infrastructure Inc.import { afterEach, describe, expect, it } from "vitest";-import { parsePositiveIntEnv } from "../commands/ingest.js";+import { parseNonNegativeIntEnv, parsePositiveIntEnv } from "../commands/ingest.js";import { mapDeadlineMs } from "../commands/map.js";// `Number.parseInt` reads any numeric prefix, so a unit suffix or an exponent@@ -43,3 +43,28 @@ describe("parsePositiveIntEnv", () => {}});});++// A count of retries, where 0 is an instruction and not "unset":+// IX_COMMIT_CONFLICT_RETRIES=0 used to retry six times (IN-19C).+describe("parseNonNegativeIntEnv", () => {+ const name = "IX_TEST_NON_NEGATIVE_INT_ENV";+ afterEach(() => { delete process.env[name]; });++ it("reads 0 as 0", () => {+ process.env[name] = "0";+ expect(parseNonNegativeIntEnv(name, 6)).toBe(0);+ });++ it("reads a plain positive integer", () => {+ process.env[name] = "3";+ expect(parseNonNegativeIntEnv(name, 6)).toBe(3);+ });++ it("falls back for unset, empty, a suffix, an exponent or a sign", () => {+ expect(parseNonNegativeIntEnv(name, 6)).toBe(6);+ for (const raw of ["", "1e3", "-1", "2k"]) {+ process.env[name] = raw;+ expect(parseNonNegativeIntEnv(name, 6), raw).toBe(6);+ }+ });+});diff --git a/ix-cli/src/cli/__tests__/helpers/fake-backend.ts b/ix-cli/src/cli/__tests__/helpers/fake-backend.tsindex eb749ee1..11aa3e63 100644--- a/ix-cli/src/cli/__tests__/helpers/fake-backend.ts+++ b/ix-cli/src/cli/__tests__/helpers/fake-backend.ts@@ -121,6 +121,39 @@ export class FakeBackend {rememberHashes = false;/** Answer `/v1/source-hashes` with a 500, as a backend under load does. */failSourceHashes = false;+ /**+ * Answer the next commits (bulk or single) with these statuses, in order,+ * before any other decision: a proxy or a restarting backend saying "not+ * now" (IN-19C). `retryAfter` is sent as the `Retry-After` header.+ */+ transientCommitFailures: Array<{ status: number; retryAfter?: string }> = [];+ /**+ * ArangoDB is down: `/v1/health` answers 503 `degraded` with+ * `database: "unreachable"` (Ix-memory 1.0.33+), every commit answers 500,+ * and `/v1/revisions/current` answers 500.+ */+ databaseDown = false;+ /**+ * The status commits answer while `databaseDown`. 500 is what the backend+ * sends; 503 is a backend (or a proxy in front of it) that reads a lost+ * database as "not now", which the bulk sites would otherwise retry.+ */+ databaseDownCommitStatus = 500;+ /**+ * Whether `/v1/health` reports `database` at all. Off is a backend older+ * than 1.0.33: health answers 200 `ok` whatever the database is doing, and+ * only a read that needs it (`/v1/revisions/current`) can tell.+ */+ healthReportsDatabase = true;+ /**+ * `/v1/health` reports the database unreachable while it is in fact up and+ * commits land: a wrong or stale health answer.+ */+ healthSaysDatabaseDown = false;+ /** `/v1/health` requests, in order. */+ healthRequests = 0;+ /** `/v1/revisions/current` requests, in order. */+ revisionRequests = 0;/*** Answer `GET /v1/patches/{id}` with a 500, as a backend under load does:* every read when true, or the reads the predicate picks (given the id and@@ -406,7 +439,7 @@ export class FakeBackend {private route(url: string, body: string, res: ServerResponse): void {const path = new URL(url, "http://x").pathname;- const send = (code: number, payload: unknown): void => {+ const send = (code: number, payload: unknown, headers: Record<string, string> = {}): void => {// Stamp the answer onto the request that produced it. Whether a bulk was// ACCEPTED or refused is the difference between two completely different// code paths in `ingestFiles`, and a test that assumes the wrong one is@@ -414,7 +447,7 @@ export class FakeBackend {// went wrong.const last = this.requests[this.requests.length - 1];if (last !== undefined && last.code === undefined) last.code = code;- res.writeHead(code, { "Content-Type": "application/json" });+ res.writeHead(code, { "Content-Type": "application/json", ...headers });res.end(JSON.stringify(payload));};@@ -438,6 +471,19 @@ export class FakeBackend {this.aborter.abort();}+ const transientFailure = this.transientCommitFailures.shift();+ if (transientFailure !== undefined) {+ return send(+ transientFailure.status,+ { error: `${transientFailure.status}: service unavailable` },+ transientFailure.retryAfter !== undefined ? { "Retry-After": transientFailure.retryAfter } : {},+ );+ }+ if (this.databaseDown) {+ const status = this.databaseDownCommitStatus;+ return send(status, { error: `${status}: Connection refused: arangodb:8529` });+ }+if (path === "/v1/patches/bulk") this.bulkBodyBytes.push(Buffer.byteLength(body));if (path === "/v1/patches/bulk" && this.maxBulkBytes !== undefined && patches.length > 1 &&Buffer.byteLength(body) > this.maxBulkBytes) {@@ -530,8 +576,17 @@ export class FakeBackend {}if (path === "/v1/health") {+ this.healthRequests++;const release = this.releaseVersion !== undefined ? this.releaseVersion : this.semantics === "head" ? "1.0.34" : null;- return send(200, { status: "ok", version: "1.0.28", ...(release ? { release_version: release } : {}) });+ const base = { version: "1.0.28", ...(release ? { release_version: release } : {}) };+ if (!this.healthReportsDatabase) return send(200, { status: "ok", ...base });+ if (this.databaseDown || this.healthSaysDatabaseDown) return send(503, { status: "degraded", database: "unreachable", ...base });+ return send(200, { status: "ok", database: "reachable", ...base });+ }+ if (path === "/v1/revisions/current") {+ this.revisionRequests++;+ if (this.databaseDown) return send(500, { error: "500: Connection refused: arangodb:8529" });+ return send(200, { rev: this.rev });}if (path === "/v1/source-hashes") {if (this.failSourceHashes) return send(500, { error: "500: transaction begin timeout" });diff --git a/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts b/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.tsindex 572cbe6e..78b81315 100644--- a/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts+++ b/ix-cli/src/cli/__tests__/ingest-commit-outcome.test.ts@@ -89,6 +89,23 @@ describe("describeCommitOutcome", () => {});});+describe("describeCommitOutcome after `ix reset` (IN-19C)", () => {+ it("is still fatal, and says the graph is still empty rather than unchanged", () => {+ const outcome = describeCommitOutcome(12, 0, "--verbose", false, 0, false, true);+ expect(outcome.kind).toBe("fatal");+ const message = outcome.kind === "fatal" ? outcome.message : "";+ expect(message).not.toContain("unchanged");+ expect(message).toContain("this workspace's graph is still empty");+ expect(message).toContain("--verbose");+ });++ it("keeps the old text without the flag, and warns the same on a partial run", () => {+ const old = describeCommitOutcome(12, 0, "--verbose", false, 0, false);+ expect(old.kind === "fatal" && old.message).toContain("The graph is unchanged");+ expect(describeCommitOutcome(3, 9, "--verbose", false, 0, false, true).kind).toBe("warn");+ });+});+describe("ingestCompletedCleanly", () => {it("is false when patches failed to commit", () => {// The guard the PR calls load-bearing, and the one nothing pinned: itdiff --git a/ix-cli/src/cli/__tests__/ingest-files.test.ts b/ix-cli/src/cli/__tests__/ingest-files.test.tsindex 5aedf344..071d81d1 100644--- a/ix-cli/src/cli/__tests__/ingest-files.test.ts+++ b/ix-cli/src/cli/__tests__/ingest-files.test.ts@@ -193,6 +193,8 @@ describe("ingestFiles against a fake backend", () => {"IX_COMMIT_FAILURE_LIMIT","IX_COMMIT_HTTP_MAX_FILES","IX_COMMIT_CONCURRENCY",+ "IX_COMMIT_CONFLICT_RETRIES",+ "IX_COMMIT_TRANSIENT_RETRY_MS","IX_STITCH_COOLDOWN_MS","IX_STITCH_WAIT_MS","IX_MAP_DEADLINE_MS",@@ -1267,6 +1269,196 @@ describe("ingestFiles against a fake backend", () => {expect(backend.singleCount, "the fan-out is for failures only").toBe(0);});+ describe("commit retries and the database probe (IN-19C)", () => {+ let stderr: ReturnType<typeof vi.spyOn>;+ const written = (): string => stderr.mock.calls.map((call: unknown[]) => String(call[0])).join("");+ const sequence = (): string[] =>+ backend.requests.filter(r => r.path !== "/v1/stitch").map(r => `${r.path} ${r.code}`);+ beforeEach(() => {+ stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true);+ });+ afterEach(() => {+ stderr.mockRestore();+ process.exitCode = undefined;+ });++ it("re-sends a bulk the backend answered 503, and lands everything with no fan-out", async () => {+ fixture(12);+ backend.transientCommitFailures = [{ status: 503, retryAfter: "0" }];++ const summary = await run();++ expect(summary.commitErrors).toBe(0);+ expect(summary.patchesApplied).toBe(12);+ expect(sequence()).toEqual(["/v1/patches/bulk 503", "/v1/patches/bulk 200"]);+ expect(backend.singleCount, "no per-file fallback").toBe(0);+ });++ it("still re-sends a 503 bulk with IX_COMMIT_CONFLICT_RETRIES=0: that knob is for conflicts only", async () => {+ fixture(4);+ process.env.IX_COMMIT_CONFLICT_RETRIES = "0";+ backend.transientCommitFailures = [{ status: 503, retryAfter: "0" }];++ const summary = await run();++ expect(summary.commitErrors).toBe(0);+ expect(summary.patchesApplied).toBe(4);+ expect(sequence()).toEqual(["/v1/patches/bulk 503", "/v1/patches/bulk 200"]);+ });++ it("sends the bulk once with IX_COMMIT_TRANSIENT_RETRY_MS=0, then falls back per file as before", async () => {+ fixture(4);+ process.env.IX_COMMIT_TRANSIENT_RETRY_MS = "0";+ backend.transientCommitFailures = [{ status: 503, retryAfter: "0" }];++ const summary = await run();++ expect(summary.commitErrors).toBe(0);+ expect(summary.patchesApplied).toBe(4);+ expect(sequence()).toEqual([+ "/v1/patches/bulk 503",+ "/v1/patch 200",+ "/v1/patch 200",+ "/v1/patch 200",+ "/v1/patch 200",+ ]);+ });++ it("trips at once when the backend reports its database down and says so, while the commits still decide the exit", async () => {+ fixture(30);+ backend.databaseDown = true;++ const message = await runFatal();++ expect(message).toContain("30 of 30");+ expect(backend.healthRequests, "the probe asked").toBeGreaterThan(0);+ // Tripped on the first per-file failure, so the fan-out holds the rest.+ // The held patches are still sent afterwards, as on main, so the exit+ // code comes from their failures, not from the health answer -- and the+ // drain's failure budget still bounds them below one request per file.+ expect(backend.bulkCount, "the held set was re-sent").toBeGreaterThan(1);+ expect(backend.singleCount).toBeLessThan(30);+ const banner = written();+ expect(banner).toContain("cannot reach its database (ArangoDB)");+ expect(banner).not.toContain("still passes");+ });++ it("asks about the database before waiting out a 503 on the first bulk", async () => {+ fixture(30);+ backend.databaseDown = true;+ backend.databaseDownCommitStatus = 503;+ const started = Date.now();++ await runFatal();++ // Asked after the first 503, not after the backoff: the database is+ // down, so the bulk is not retried and the chunk goes to the cutoff+ // rather than the per-file fallback. The held set is then still sent,+ // so the commits, not the probe, decide the exit.+ expect(Date.now() - started, "no transient backoff").toBeLessThan(5_000);+ expect(backend.healthRequests, "the probe asked").toBeGreaterThan(0);+ expect(backend.singleCount).toBeLessThan(30);+ const banner = written();+ expect(banner).toContain("cannot reach its database (ArangoDB)");+ });++ it("a health answer that wrongly says the database is down does not fail a run whose commits land", async () => {+ fixture(30);+ backend.healthSaysDatabaseDown = true;+ backend.transientCommitFailures = [{ status: 503, retryAfter: "0" }];++ const summary = await run();++ expect(backend.healthRequests, "the probe asked, and was told 'unreachable'").toBeGreaterThan(0);+ expect(summary.commitErrors, "the held patches were sent and landed").toBe(0);+ expect(summary.patchesApplied).toBe(30);+ expect(process.exitCode ?? 0).toBe(0);+ // Every patch landed, so no cutoff banner: it would tell the user the+ // graph is "partly updated" and to re-run, which is false.+ expect(written()).not.toContain("kept failing");+ expect(written()).not.toContain("partly updated");+ });++ it("does not trip on a 5xx from the current-revision fallback (an older backend proves nothing)", async () => {+ fixture(30);+ backend.databaseDown = true;+ backend.healthReportsDatabase = false;++ await runFatal();++ expect(backend.revisionRequests).toBeGreaterThan(0);+ expect(written()).not.toContain("cannot reach its database (ArangoDB)");+ });++ it("changes nothing on a backend that refuses every commit while its database is reachable", async () => {+ fixture(30);+ backend.refuseEverything = true;++ const message = await runFatal();++ expect(backend.healthRequests, "the probe asked").toBeGreaterThan(0);+ expect(message).toContain("30 of 30");+ // The streak, not the probe, tripped it: five per-file failures, as before.+ expect(backend.singleCount).toBeGreaterThanOrEqual(5);+ expect(backend.commitCount, "well under main's 31").toBeLessThan(25);+ expect(written()).toContain("`ix doctor` still passes");+ });++ it("does not call a graph wiped under it 'unchanged' when nothing lands", async () => {+ // A baseline from an earlier map, then a backend that holds none of it:+ // the DB-reset guard fires, as it does after `ix reset` from another dir.+ fixture(8);+ const incremental = () =>+ ingestFiles(repo, { format: "text", suppressOutput: true, printSummary: false });+ await incremental();+ backend.refuseEverything = true;++ let message = "";+ try {+ await incremental();+ } catch (err) {+ message = String(err);+ }++ expect(message, "still fatal").toContain("8 of 8");+ const out = `${written()}\n${message}`;+ expect(out).toContain("kept failing");+ expect(out).not.toContain("unchanged");+ expect(out).toContain("this workspace's graph is still empty");+ });++ it("does not call it 'unchanged' after a first map that committed nothing, below the cutoff", async () => {+ fixture(2);+ backend.refuseEverything = true;+ let message = "";+ try {+ await ingestFiles(repo, { format: "text", suppressOutput: true, printSummary: false });+ } catch (err) {+ message = String(err);+ }+ expect(message).toContain("committed nothing");+ expect(message).not.toContain("unchanged");+ expect(message).toContain("still empty");+ });++ it("does not call the workspace's graph empty when a scoped run with no local baseline commits nothing", async () => {+ // `ix ingest <new file>` in a repo mapped from another HOME or machine:+ // no local baseline, and the lookup covers that one file only, so an+ // empty answer says nothing about the rest of the graph.+ fixture(2);+ backend.refuseEverything = true;+ let message = "";+ try {+ await ingestFiles(join(repo, "src", "m000.ts"), { format: "text", suppressOutput: true, printSummary: false });+ } catch (err) {+ message = String(err);+ }+ expect(message).toContain("committed nothing");+ expect(message).not.toContain("still empty");+ expect(message).toContain("may now be stale");+ });+ });+describe("a patch that deletes (IN-04)", () => {const incremental = () =>ingestFiles(repo, { format: "text", suppressOutput: true, printSummary: false });diff --git a/ix-cli/src/cli/__tests__/ingest-retry.test.ts b/ix-cli/src/cli/__tests__/ingest-retry.test.tsindex 496d3af7..f7015000 100644--- a/ix-cli/src/cli/__tests__/ingest-retry.test.ts+++ b/ix-cli/src/cli/__tests__/ingest-retry.test.ts@@ -1,6 +1,6 @@// Copyright 2026 Ix Infrastructure Inc.-import { describe, expect, it, vi } from 'vitest';+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';import {commitBulkWithPayloadSplit,commitFailureIndictsBackend,@@ -14,6 +14,8 @@ import {parseBulkCommittedPatchIds,parseSuggestedMaxPatches,} from '../commands/ingest.js';+import { retryCommit } from '../commit-retry.js';+import { httpError } from '../../client/api.js';// Verbatim from a backend that refused a bulk save of a >1,000-file repo (Ix#516).// A bulk commit runs as one exclusive Arango transaction, and Arango caps that at@@ -637,3 +639,74 @@ describe('bulk request bounds (IN-05)', () => {expect(cut.map(c => c.map(item => item.i))).toEqual([[0, 1], [2], [3, 4], [5]]);});});++// IN-19C wraps the bulk commit in `retryCommit` with transient retries on. A+// 413 must reach the payload split exactly as before: no retry of the same+// oversized body, and no wait.+describe('commitBulkWithPayloadSplit under retryCommit', () => {+ it('splits a 413 without retrying it, with the same requests as before', async () => {+ const bulkCalls: number[][] = [];+ const committed: number[] = [];+ const sleep = vi.fn(async () => {});+ const commitIndividually = vi.fn(async () => {});++ await commitBulkWithPayloadSplit([1, 2, 3, 4, 5], {+ commitBulk: batch => retryCommit(async () => {+ bulkCalls.push([...batch]);+ if (batch.length > 2) {+ throw httpError(413, '{"error":"payload_too_large"}', new Headers({ 'Retry-After': '1' }));+ }+ return batch.length;+ }, { maxRetries: 6, transient: true, sleep }),+ onBulkCommitted: batch => committed.push(...batch),+ commitIndividually,+ });++ expect(bulkCalls).toEqual([+ [1, 2, 3, 4, 5],+ [1, 2, 3],+ [1, 2],+ [3],+ [4, 5],+ ]);+ expect(committed).toEqual([1, 2, 3, 4, 5]);+ expect(sleep, 'a 413 is never waited on').not.toHaveBeenCalled();+ expect(commitIndividually).not.toHaveBeenCalled();+ });+});++// F1 of the #845 review: a 504 on a long bulk used to buy up to three more+// full re-sends before the per-file fallback. Now the bulk is sent once, and+// the fallback gets the group at once.+describe('a slow 504 bulk under retryCommit', () => {+ beforeEach(() => { vi.useFakeTimers(); });+ afterEach(() => { vi.useRealTimers(); });++ it('is sent once and falls back to the per-file path, without three full re-sends', async () => {+ const bulkCalls: number[][] = [];+ const commitIndividually = vi.fn(async (_items: number[], _err?: unknown) => {});+ const start = Date.now();++ const done = commitBulkWithPayloadSplit([1, 2, 3], {+ commitBulk: batch => retryCommit(+ () => new Promise<number>((_resolve, reject) => {+ bulkCalls.push([...batch]);+ // The gateway gives up after two minutes; the backend may still be at it.+ setTimeout(() => reject(httpError(504, '<html>Gateway Timeout</html>', new Headers())), 120_000);+ }),+ { maxRetries: 6, transient: true },+ ),+ onBulkCommitted: vi.fn(),+ commitIndividually,+ }).then(() => Date.now() - start);+ await vi.advanceTimersByTimeAsync(30 * 60_000);+ const elapsed = await done;++ expect(bulkCalls).toEqual([[1, 2, 3]]);+ expect(elapsed, 'one request, no waits').toBe(120_000);+ expect(commitIndividually).toHaveBeenCalledOnce();+ const [items, err] = commitIndividually.mock.calls[0];+ expect(items).toEqual([1, 2, 3]);+ expect(String(err)).toMatch(/^Error: 504:/);+ });+});diff --git a/ix-cli/src/cli/__tests__/map-cloud-no-retry.test.ts b/ix-cli/src/cli/__tests__/map-cloud-no-retry.test.tsnew file mode 100644index 00000000..2ac537a6--- /dev/null+++ b/ix-cli/src/cli/__tests__/map-cloud-no-retry.test.ts@@ -0,0 +1,80 @@+// Copyright 2026 Ix Infrastructure Inc.++// The cloud ingest path is not the bulk commit path, and the transient-status+// retry added for bulk commits (IN-19C, commit-retry.ts) must never reach it.+// `ix map` hands a cloud-ready workspace to the registered RemoteRunner in one+// call. A long cloud upload that fails, even with a 502/503/504/429 and a+// `Retry-After`, is reported once and never sent again from here, and nothing+// on that path goes through a commit route.++import * as fs from "node:fs";+import * as os from "node:os";+import * as path from "node:path";+import { Command } from "commander";+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";+import { httpError, IxClient } from "../../client/api.js";+import { registerMapCommand } from "../commands/map.js";+import { registerRemoteRunner, type RemoteIngestOptions } from "../remote.js";++describe("ix map on the cloud path", () => {+ let dir: string;+ const saved: Record<string, string | undefined> = {};+ const envKeys = ["IX_LOCK_DIR", "IX_ENDPOINT", "IX_HOME", "IX_AUTO_MAP", "IX_MAP_DEADLINE_MS"];++ beforeEach(() => {+ dir = fs.mkdtempSync(path.join(os.tmpdir(), "ix-map-cloud-"));+ for (const key of envKeys) saved[key] = process.env[key];+ process.env.IX_LOCK_DIR = path.join(dir, "locks");+ fs.mkdirSync(process.env.IX_LOCK_DIR);+ process.env.IX_HOME = path.join(dir, "home");+ // Nothing here may reach a real backend: a stray request fails fast.+ process.env.IX_ENDPOINT = "http://127.0.0.1:9";+ delete process.env.IX_AUTO_MAP;+ delete process.env.IX_MAP_DEADLINE_MS;+ process.exitCode = undefined;+ });++ afterEach(() => {+ vi.restoreAllMocks();+ // No unregister: a runner that is never ready routes everything local again.+ registerRemoteRunner({ runIngestion: async () => {}, isReady: async () => false });+ for (const key of envKeys) {+ if (saved[key] === undefined) delete process.env[key];+ else process.env[key] = saved[key];+ }+ process.exitCode = undefined;+ fs.rmSync(dir, { recursive: true, force: true });+ });++ it.each([+ ["a gateway timeout", () => httpError(504, "<html>Gateway Timeout</html>", new Headers())],+ ["a 503 asking to retry at once", () => httpError(503, '{"error":"unavailable"}', new Headers({ "Retry-After": "0" }))],+ ["a 429 asking to retry", () => httpError(429, '{"error":"slow down"}', new Headers({ "Retry-After": "1" }))],+ ])("sends a failed cloud upload once, after %s, and never through a commit route", async (_label, failure) => {+ const root = path.join(dir, "workspace");+ fs.mkdirSync(root);+ const calls: RemoteIngestOptions[] = [];+ registerRemoteRunner({+ isReady: async () => true,+ runIngestion: async opts => {+ calls.push(opts);+ throw failure();+ },+ });+ const bulk = vi.spyOn(IxClient.prototype, "commitPatchBulk");+ const single = vi.spyOn(IxClient.prototype, "commitPatch");+ vi.spyOn(console, "error").mockImplementation(() => {});++ const program = new Command();+ registerMapCommand(program);+ await program.parseAsync(["node", "ix", "map", root, "--silent"]);++ expect(calls, "one upload, no re-send").toHaveLength(1);+ // `ix map` uploads the canonical root (realpath): /var -> /private/var on+ // macOS, 8.3 short names expanded on Windows.+ expect(calls[0]).toMatchObject({ cwd: fs.realpathSync.native(root) });+ expect(bulk).not.toHaveBeenCalled();+ expect(single).not.toHaveBeenCalled();+ expect(process.exitCode).toBe(1);+ });+});diff --git a/ix-cli/src/cli/backend-version.ts b/ix-cli/src/cli/backend-version.tsindex c802014b..e9dc006f 100644--- a/ix-cli/src/cli/backend-version.ts+++ b/ix-cli/src/cli/backend-version.ts@@ -176,8 +176,10 @@ export async function fetchBackendHealth(client: IxClient,knownLatest: string | null | undefined,isNewer: (a: string, b: string) => boolean,+ /** Bounds a probe that must answer in seconds; the client's default otherwise. */+ timeoutMs?: number,): Promise<HealthResponse> {- const health = await client.health();+ const health = await client.health(timeoutMs);// From the client the request actually went to. Taking it as a second// argument reintroduced exactly the "a call site can get it wrong" mistake// this module exists to make impossible.diff --git a/ix-cli/src/cli/commands/doctor.ts b/ix-cli/src/cli/commands/doctor.tsindex c178b3ff..fcaef0c0 100644--- a/ix-cli/src/cli/commands/doctor.ts+++ b/ix-cli/src/cli/commands/doctor.ts@@ -41,6 +41,7 @@ import { loadIngestBaseline } from "../ingest-baseline.js";import { isCloudReady } from "../remote.js";import { hasCompletedMapBaseline } from "../stale.js";import { printJson } from "../format.js";+import { healthFromError } from "../database-probe.js";import type { CapabilitiesResponse, HealthResponse } from "../../client/types.js";interface CheckResult {@@ -250,21 +251,8 @@ export function assessLocalAuth(};}-/**- * A health answer, including the 503 a backend sends while it cannot reach its- * database: that is a server that answered, and the body says what is wrong.- * Thrown for anything else.- */-export function healthFromError(err: unknown): HealthResponse | null {- const m = /^503:\s*(\{.*\})\s*$/s.exec((err as Error)?.message ?? "");- if (!m) return null;- try {- const body = JSON.parse(m[1]!) as HealthResponse;- return typeof body?.status === "string" ? body : null;- } catch {- return null;- }-}+// Shared with the ingest's database probe; re-exported for existing importers.+export { healthFromError };/*** "Database reachable" from the health answer. A backend that reportsdiff --git a/ix-cli/src/cli/commands/ingest.ts b/ix-cli/src/cli/commands/ingest.tsindex 9ce32893..405b7d8c 100644--- a/ix-cli/src/cli/commands/ingest.ts+++ b/ix-cli/src/cli/commands/ingest.ts@@ -44,6 +44,11 @@ import {drainInPasses,perFileAction,} from '../commit-breaker.js';+import { retryCommit, TRANSIENT_RETRY_BUDGET_MS } from '../commit-retry.js';+import { createDatabaseProbe, probeDatabase } from '../database-probe.js';+// Moved to commit-retry.ts with the retry loop that uses them; re-exported so+// existing importers keep working.+export { isAbortError, isRetryableCommitConflict } from '../commit-retry.js';import { readBackendHealth } from './upgrade.js';import {createIgnoreMatcher, parseIgnoreFile, parseIgnorePattern, type IgnorePattern,@@ -135,6 +140,19 @@ export function parsePositiveIntEnv(name: string, fallback: number): number {return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;}+/**+ * `parsePositiveIntEnv`, but 0 is a value rather than "unset". For a count of+ * RETRIES, where 0 means "do not retry" and reading it as the default made the+ * setting impossible to turn off (IX_COMMIT_CONFLICT_RETRIES=0 retried 6 times).+ */+export function parseNonNegativeIntEnv(name: string, fallback: number): number {+ const raw = process.env[name]?.trim();+ if (!raw) return fallback;+ if (!/^\d+$/.test(raw)) return fallback;+ const parsed = Number.parseInt(raw, 10);+ return Number.isSafeInteger(parsed) ? parsed : fallback;+}+/*** The patterns in force for this ingest: `--exclude` flags first, then the@@ -950,32 +968,6 @@ export function planDeletedFileRecovery(return { previousDeletedFiles, nextDeletedFiles, recreatedPaths, forceReingestPaths };}-/** Lock contention, as opposed to the transport failures below. Kept separate- * only so each group's reason for being retryable stays readable. */-const COMMIT_CONFLICT_RETRY_PATTERNS = [- // ArangoDB lock conflicts. These were briefly split into their own constant- // so `commitFailureIndictsBackend` could exclude them; that exclusion was- // reverted in round 3 -- it made the cutoff inert in a real saturation mode --- // and the split had no consumer left, so it is inlined.- 'write-write conflict',- 'timeout waiting to lock key',- 'error: 1200',- // Transport-level failures (k8s ingress reset, socket drop under load).- //- // These are retried because the request usually never reached the server --- // but "usually" is the whole of Ix#495. A bulk commit is chunked and writes- // its idempotency keys as each chunk lands, so a socket that drops after the- // server started work leaves part of the group committed. Retrying then- // re-sends a set the server has partly seen, and because the bulk group id is- // a hash of the patch ids, the retry reproduces the same group and is refused- // with a 409 that no further retry can clear.- //- // The retry stays (it is right far more often than not); what changed is that- // the resulting 409 is now recoverable — see isBulkPartiallyCommittedError.- 'fetch failed',- 'econnreset',- 'econnrefused',-];/*** A bulk commit the server accepted only in part, or whose patches all landed@@ -1027,23 +1019,6 @@ export function parseBulkCommittedPatchIds(err: unknown): Set<string> | undefine}}-// An abort from the shared wall-clock deadline (or a per-request timeout) must-// NOT be retried — the whole point of the deadline is to stop work. Retrying-// here would defeat it and add 6 more backed-off requests to an already-late-// run. AbortSignal.timeout raises a TimeoutError; an aborted deadline raises an-// AbortError; both also stringify with "aborted".-export function isAbortError(err: unknown): boolean {- const name = (err as { name?: string } | null)?.name;- if (name === "AbortError" || name === "TimeoutError") return true;- return String(err).toLowerCase().includes("aborted");-}--export function isRetryableCommitConflict(err: unknown): boolean {- if (isAbortError(err)) return false;- const message = String(err).toLowerCase();- return COMMIT_CONFLICT_RETRY_PATTERNS.some(pattern => message.includes(pattern));-}-/*** Does this commit failure say anything about the BACKEND? (Ix#560)*@@ -1080,18 +1055,14 @@ export function commitFailureIndictsBackend(err: unknown, runDeadlineExpired: boreturn true;}+/**+ * The per-file commit's retry: lock conflicts and transport drops only.+ * Transient statuses (502/503/504/429) are retried at the BULK sites, which+ * call `retryCommit` themselves; retrying them per patch would multiply a+ * per-bulk wait budget by the number of patches.+ */async function retryOnConflict<T>(fn: () => Promise<T>, maxRetries: number): Promise<T> {- for (let attempt = 0; ; attempt++) {- try {- return await fn();- } catch (err) {- if (attempt >= maxRetries || !isRetryableCommitConflict(err)) throw err;- // ArangoDB lock conflicts usually clear as soon as the competing transaction- // commits. Transport errors may need a bit more breathing room under k8s load.- const delay = 10 * (1 << attempt) + Math.random() * 10;- await new Promise<void>(r => setTimeout(r, delay));- }- }+ return retryCommit(fn, { maxRetries, transient: false });}/**@@ -2007,7 +1978,13 @@ export function describeCommitOutcome(* patches? When it did, the banner above has already given the reason, and* this line must not send the reader back to a flag that adds nothing.*/- cutoffFired: boolean = cutoffSkipped > 0+ cutoffFired: boolean = cutoffSkipped > 0,+ /**+ * The backend held none of this run's files before it started (a first map,+ * or one after `ix reset`). With nothing landed, "the graph is unchanged"+ * would read as "the old graph is still there", and there is none.+ */+ graphStartedEmpty: boolean = false): CommitOutcome {if (commitErrors <= 0) return { kind: "ok" };const attempted = commitErrors + patchesApplied;@@ -2112,7 +2089,10 @@ export function describeCommitOutcome(kind: "fatal",message:`Ingest committed nothing: all ${commitErrors} ${patches} failed to commit. ` +- `The graph is unchanged and may now be stale. Re-run with ${debugFlag} to see why.`,+ (graphStartedEmpty+ ? `The backend held none of these files before this run, so this workspace's graph is still empty. `+ : `The graph is unchanged and may now be stale. `) ++ `Re-run with ${debugFlag} to see why.`,};}return {@@ -2507,6 +2487,14 @@ export async function ingestFiles(// a way this run can have written to the graph, or have found it not where// the last run left it, without a patch being counted.let dbResetDetected = false;+ /**+ * The backend held none of this run's files when it started: the DB-reset+ * guard fired, or a lookup over the whole file set found no hashes (a first+ * map, or one after `ix reset` / `reset --ingest`). Only the failure messages+ * read it: "the graph is unchanged" is misleading when there was no graph.+ * Unset on any path that ends before the lookup.+ */+ let graphStartedEmpty: boolean | undefined;let extractorRebuild = false;let deletionsFound = false;let stitchSent = false;@@ -2614,6 +2602,60 @@ export async function ingestFiles(* raises TimeoutError too, and that is a saturation signal, not our clock.*/const runDeadlineExpired = (): boolean => opts.deadlineSignal?.aborted === true;+ /** The same deadline, for the commit retries' waits to abort on. */+ const runDeadlineSignal = opts.deadlineSignal;+ /**+ * The backend's own word on its database, asked only once commits are+ * failing (IN-19C). The cutoff judges by a streak, which with ArangoDB down+ * costs a streak of doomed requests here and a failure budget per drain+ * pass; the backend can simply say so. One probe answers every caller for a+ * few seconds, under a 2 s timeout, and only `unreachable` changes anything+ * -- see `database-probe.ts`.+ */+ const newDatabaseProbe = () => createDatabaseProbe(() => probeDatabase({+ // Through the one place health is fetched and its release recorded.+ readHealth: timeoutMs => readBackendHealth(client, timeoutMs),+ currentRevision: timeoutMs => client.currentRevision(timeoutMs),+ }), 3_000);+ let askDatabase = newDatabaseProbe();+ /** `patchesTheBackendTook` when `askDatabase`'s cache was started; a landed patch is newer evidence than any cached verdict. */+ let askedAtPatches = 0;+ /** `patchesTheBackendTook` when the backend last reported its database down; undefined since it reported it up. */+ let databaseDownAsOf: number | undefined;+ /** The last answer was "unreachable", and nothing has landed since to say otherwise. */+ const databaseKnownDown = (): boolean =>+ databaseDownAsOf !== undefined && databaseDownAsOf === patchesTheBackendTook;+ /** Ask (or reuse a recent answer) and record it. True when the database is down. */+ const checkDatabase = async (): Promise<boolean> => {+ // IX_COMMIT_FAILURE_LIMIT=0 opts out of giving up; the probe never stops a run then.+ if (commitBreaker.limit <= 0) return false;+ if (runDeadlineExpired()) return databaseKnownDown();+ if (askedAtPatches !== patchesTheBackendTook) {+ askDatabase = newDatabaseProbe();+ askedAtPatches = patchesTheBackendTook;+ }+ const verdict = await askDatabase();+ if (verdict === 'unreachable') databaseDownAsOf = patchesTheBackendTook;+ else if (verdict === 'reachable') databaseDownAsOf = undefined;+ return verdict === 'unreachable';+ };+ /**+ * After a commit failure that indicts the backend: if its database is down,+ * give up now rather than after a streak. A later commit landing clears the+ * trip, as it clears one a streak set.+ */+ const tripIfDatabaseDown = async (err: unknown): Promise<void> => {+ if (commitBreaker.tripped()) return;+ if ((await checkDatabase()) && !commitBreaker.tripped()) commitBreaker.trip(err);+ };+ const noteCommitRetry = (where: string) =>+ (info: { attempt: number; delayMs: number; error: unknown; kind: string }): void => {+ if (!debug) return;+ const msg = String(info.error);+ process.stderr.write(+ `\n [${where} retry ${info.attempt}, ${info.kind}] waiting ${Math.round(info.delayMs)} ms: ${msg.length > 200 ? `${msg.slice(0, 200)}…` : msg}\n`,+ );+ };let tooLarge = 0;/** The stat loop's record of empty and oversized files, once it exists; `tooLarge` counts its oversized ones. */let skippedForSize: ReadonlyMap<string, 'empty' | 'tooLarge'> | undefined;@@ -2974,6 +3016,10 @@ export async function ingestFiles(// file, so the next run's baseline did not name them, they were never// found deleted again, and their nodes stayed in the graph for good.const deferredDeletions: string[] = [];+ // Whether the lookup ANSWERED. Thrown every time now, rather than only with+ // deletions pending, so an empty answer can be told from a failed one; the+ // catch below does what `loadExistingHashes` did with a swallowed failure.+ let hashLookupAnswered = false;if (hashLookupPaths.length > 0) {try {knownHashes = await loadExistingHashes(@@ -2982,10 +3028,11 @@ export async function ingestFiles(toWorkspaceRelative,sourceWorkspaceIdOf,debug,- deletedPaths.length > 0,+ true,);+ hashLookupAnswered = true;} catch (err) {- if (debug) process.stderr.write(`\n [deletion cleanup skipped] hash lookup failed: ${err}\n`);+ if (debug && deletedPaths.length > 0) process.stderr.write(`\n [deletion cleanup skipped] hash lookup failed: ${err}\n`);deferredDeletions.push(...deletedPaths);deletedPaths.length = 0;knownHashes = new Map();@@ -2996,6 +3043,19 @@ export async function ingestFiles(knownHashes = new Map();if (debug) process.stderr.write(`\n All ${filePaths.length} files mtime-clean — skipping hash lookup\n`);}+ // With no local baseline every file was looked up, so an answer naming+ // none of them means the backend holds none. Not under --force, which+ // says nothing about the baseline, and not on a failed lookup, which+ // says nothing about the backend. Not on a workspace-id migration either:+ // the old id's graph is still there until the cleanup after this run.+ // And not on a partial run (a file or directory scope, or --lang): its+ // lookup covered only its own files, so an empty answer says nothing about+ // the rest of the workspace -- `ix ingest <new file>` in a repo mapped from+ // another machine has no local baseline but a full graph.+ graphStartedEmpty =+ dbResetDetected ||+ (!opts.force && !workspaceMigrated && !scopedRun && !opts.lang &&+ mtimeCache.size === 0 && hashLookupAnswered && knownHashes.size === 0);const hashed = performance.now();timings.hashMs = Math.round(hashed - discovered);@@ -3229,7 +3289,12 @@ export async function ingestFiles(return cutToLimits(items);};const COMMIT_CONCURRENCY = parsePositiveIntEnv('IX_COMMIT_CONCURRENCY', 8); // parallel HTTP save requests- const COMMIT_CONFLICT_RETRIES = parsePositiveIntEnv('IX_COMMIT_CONFLICT_RETRIES', 6); // retry transient Arango lock conflicts+ // Retries of one commit after a lock conflict or a transport drop. 0: none.+ const COMMIT_CONFLICT_RETRIES = parseNonNegativeIntEnv('IX_COMMIT_CONFLICT_RETRIES', 6);+ // Total wait for re-sending a BULK the backend answered 502/503/504/429+ // (at most three retries, `Retry-After` cut to it). 0: no such retries.+ // Its own knob, so turning conflict retries off leaves these alone.+ const COMMIT_TRANSIENT_RETRY_MS = parseNonNegativeIntEnv('IX_COMMIT_TRANSIENT_RETRY_MS', TRANSIENT_RETRY_BUDGET_MS);// Re-sends of a commit that lost the base-rev race to another writer; see retryOnBaseRevRace.const COMMIT_BASE_REV_RETRIES = parsePositiveIntEnv('IX_COMMIT_BASE_REV_RETRIES', 8);let baseRevRetries = 0;@@ -3542,7 +3607,13 @@ export async function ingestFiles(}} else {commitErrors++;- if (indicts) commitBreaker.recordFailure(commitErr);+ if (indicts) {+ commitBreaker.recordFailure(commitErr);+ // Not in the drain (the only caller with a failure+ // budget): it ignores `tripped`, so a probe there would+ // only add health requests and latency.+ if (fallbackOpts?.failureBudget === undefined) await tripIfDatabaseDown(commitErr);+ }}if (debug) {const errMsg = String(commitErr);@@ -3586,6 +3657,10 @@ export async function ingestFiles(/** Re-send the held set. What it cannot place is left on `deferredByCutoff`. */const commitIndividuallyAfterCutoff = async (itemsIn: PreparedPatch[]): Promise<void> => {let items = itemsIn;+ // Re-sent even when the backend has reported its database down. The+ // probe may hold patches back during the fan-out, but whether the run+ // fails is decided by commits, as on main: a wrong or stale health+ // answer must not turn a run whose patches would land into an exit 1.if (debug) {process.stderr.write(`\n [cutoff] re-sending ${items.length} held-back patches with a fresh streak\n`,@@ -3629,9 +3704,19 @@ export async function ingestFiles(const bulkStart = performance.now();try {const result = await retryOnBaseRevRace(- () => retryOnConflict(+ () => retryCommit(() => client.commitPatchBulk(chunk.map(item => item.patch)),- COMMIT_CONFLICT_RETRIES,+ {+ maxRetries: COMMIT_CONFLICT_RETRIES,+ transient: COMMIT_TRANSIENT_RETRY_MS > 0,+ transientBudgetMs: COMMIT_TRANSIENT_RETRY_MS,+ signal: runDeadlineSignal,+ // Not the breaker: this bulk exists because the breaker+ // tripped, so it would always say give up. Asked, not just+ // read: a 503 here may be the database going down.+ shouldGiveUp: () => checkDatabase(),+ onRetry: noteCommitRetry('cutoff bulk'),+ },),COMMIT_BASE_REV_RETRIES,noteBaseRevRetry,@@ -3776,6 +3861,9 @@ export async function ingestFiles(// tests, which previously asserted the stranding guarantee at a budget// the CLI never runs at and so could not see it fail.const leftover = await drainInPasses(items, async pending => {+ // No database probe here (the per-file loop skips it when given a+ // failure budget): the drain sends, as on main, so the exit code+ // follows what the commits did, never a health answer.commitBreaker.reset();const tookBefore = patchesTheBackendTook;await commitItemsSerially(pending, recordDrainMs, undefined, {@@ -3823,9 +3911,31 @@ export async function ingestFiles();const commitStart = performance.now();const result = await retryOnBaseRevRace(- () => retryOnConflict(+ () => retryCommit(() => client.commitPatchBulk(items.map(item => item.patch)),- COMMIT_CONFLICT_RETRIES,+ {+ maxRetries: COMMIT_CONFLICT_RETRIES,+ // 502/503/504/429 too, honouring Retry-After up to the+ // budget (5 s per bulk by default, requests included; a+ // slow bulk is not sent again). A 413 and a 409 are+ // thrown at once, to the payload split and the+ // partial-commit handling below.+ transient: COMMIT_TRANSIENT_RETRY_MS > 0,+ transientBudgetMs: COMMIT_TRANSIENT_RETRY_MS,+ signal: runDeadlineSignal,+ // Asked, not just read. On the first bulk of a run nothing+ // has probed yet, and a 503 from a backend whose database+ // is down would otherwise spend the whole budget+ // before anything asked it. A database reported down trips+ // the breaker, so `shouldStop` hands the chunk to the+ // cutoff instead of the per-file fallback.+ shouldGiveUp: async err => {+ if (commitBreaker.tripped()) return true;+ await tripIfDatabaseDown(err);+ return commitBreaker.tripped() || databaseKnownDown();+ },+ onRetry: noteCommitRetry('bulk'),+ },),COMMIT_BASE_REV_RETRIES,noteBaseRevRetry,@@ -5049,6 +5159,14 @@ export async function ingestFiles(// left the agent believing the graph was current while `ix search` and// `ix impact` answered from a stale one. A wrong answer nobody can see is// worse than a failed command.+ // A trip alone is not enough: the database probe can trip the breaker on a+ // bulk failure with no `commitErrors++` (a health answer of "unreachable"+ // that turns out wrong), and the cutoff and drain may then land every patch.+ // Printing the cutoff banner then would say the graph is "partly updated"+ // and ask for a re-run when nothing is missing. A skipped patch is counted+ // in `commitErrors`, so this still fires whenever the cutoff held anything+ // back that did not land.+ const cutoffFired = commitBreaker.everTripped() && commitErrors > 0;const commitReport = describeCommitOutcome(commitErrors,patchesApplied,@@ -5058,7 +5176,8 @@ export async function ingestFiles(commitBreaker.skipped(),// The same expression the banner is gated on, so this can never say "See// the cutoff above" with no cutoff above it.- commitBreaker.everTripped()+ cutoffFired,+ graphStartedEmpty === true);const summary: IngestFilesSummary = {filesDiscovered,@@ -5096,15 +5215,7 @@ export async function ingestFiles(commitErrors === 0 &&stitchErrors === 0,};- // `everTripped()` alone, and it is not a weakening: `recordFailure` has one- // call site, in the per-file catch immediately after `commitErrors++`, so a- // trip already implies a real commit failure. A `commitLoopErrors` counter- // guarded this for two rounds against a scenario that no longer exists --- // replay failures were moved off `recordFailure` onto a local counter, and a- // deletion patch that fails to BUILD never reaches the breaker at all. A dead- // conjunct defended by a comment describing an impossible case is the exact- // shape this PR has spent rounds removing.- if (commitBreaker.everTripped()) {+ if (cutoffFired) {// Gated on the cutoff having FIRED, not on what was left unsent. A patch// that was never sent produces no `[commit error]` line even under// --verbose, so this has to print for those -- but gating on them alone@@ -5116,7 +5227,10 @@ export async function ingestFiles(//// Before the stitch line and before the commit report: this is the cause,// and everything else printed about this run is its consequence.- process.stderr.write(`${describeCommitCutoff(commitBreaker, client.endpoint, patchesApplied, opts.deadlineSignal?.aborted === true)}\n`);+ process.stderr.write(`${describeCommitCutoff(commitBreaker, client.endpoint, patchesApplied, opts.deadlineSignal?.aborted === true, {+ graphStartedEmpty: graphStartedEmpty === true,+ databaseUnreachable: databaseKnownDown(),+ })}\n`);}if (stitchErrors > 0) {process.stderr.write(` ${describeStitchFailure(stitchError)}\n`);diff --git a/ix-cli/src/cli/commands/upgrade.ts b/ix-cli/src/cli/commands/upgrade.tsindex a19ddbe2..0ed5e9d2 100644--- a/ix-cli/src/cli/commands/upgrade.ts+++ b/ix-cli/src/cli/commands/upgrade.ts@@ -39,11 +39,11 @@ export function backendCeiling(): string | null {return readCache()?.backendLatest ?? null;}-export async function readBackendHealth(client: IxClient) {+export async function readBackendHealth(client: IxClient, timeoutMs?: number) {// backendLatest, NOT latest: `latest` is the CLI's own release series, so a// ceiling taken from it would refuse every legitimate container report for// ever — silently reinstating the bug this exists to fix.- return fetchBackendHealth(client, backendCeiling(), isNewer);+ return fetchBackendHealth(client, backendCeiling(), isNewer, timeoutMs);}const __filename = fileURLToPath(import.meta.url);diff --git a/ix-cli/src/cli/commit-breaker.ts b/ix-cli/src/cli/commit-breaker.tsindex bcbbe3af..35d4de8c 100644--- a/ix-cli/src/cli/commit-breaker.ts+++ b/ix-cli/src/cli/commit-breaker.ts@@ -98,6 +98,15 @@ export interface CommitBreaker {recordSuccess(): void;/** A commit failed in a way that sending it again cannot fix. */recordFailure(error: unknown): void;+ /**+ * Give up now, without waiting for a streak: the backend has said itself+ * that it cannot reach its database (see `database-probe.ts`).+ *+ * The same latch a streak sets, and cleared the same way: a commit that lands+ * is direct evidence the probe was stale, and `recordSuccess` un-trips.+ * `everTripped()` stays set, so the banner still explains the run.+ */+ trip(error: unknown): void;/*** A failure worth QUOTING, but not worth counting.*@@ -166,6 +175,13 @@ export function createCommitBreaker(limit = commitFailureLimit()): CommitBreakereverTripped: () => everLatched,recordSuccess: () => { consecutive = 0; latched = false; },noteError: (error) => { last = error; },+ trip: (error) => {+ last = error;+ // limit 0 is the documented opt-out: this breaker never trips.+ if (limit <= 0) return;+ latched = true;+ everLatched = true;+ },recordFailure: (error) => {consecutive++;last = error;@@ -474,6 +490,17 @@ export function describeCommitCutoff(* attribution of the count is not.*/deadlineHit = false,+ context: {+ /**+ * The backend held none of this run's files before it started: a first+ * map, or one after `ix reset` (or `reset --ingest`). "The graph is+ * unchanged" is then technically true and practically misleading -- it+ * reads as "your old graph is still there", and there is none.+ */+ graphStartedEmpty?: boolean;+ /** The backend reported during this run that it cannot reach its database. */+ databaseUnreachable?: boolean;+ } = {},): string {const detail = String(breaker.lastError() ?? "unknown error");const trimmed = detail.length > 300 ? `${detail.slice(0, 300)}…` : detail;@@ -488,8 +515,20 @@ export function describeCommitCutoff(// be true, which is also one less piece of run-scoped state to keep correct.const state =patchesApplied === 0- ? "The graph is unchanged."+ ? context.graphStartedEmpty+ ? "Nothing landed, and the backend held none of these files before this run (a first map, or a map after `ix reset`), so this workspace's graph is still empty."+ : "The graph is unchanged.": `${patchesApplied} other ${patchesApplied === 1 ? "patch" : "patches"} did land, so the graph is partly updated.`;+ // When the backend has SAID its database is unreachable there is nothing+ // left to tell apart, and "`ix doctor` still passes" would be false: doctor+ // reads the same health answer and fails its database check on it.+ const cause = context.databaseUnreachable+ ? [` The backend reports that it cannot reach its database (ArangoDB); \`ix doctor\` shows it.`]+ : [+ ` Either the database cannot keep up — \`ix doctor\` still passes, since an ArangoDB that`,+ ` cannot begin a transaction is reachable and consistent — or the backend is rejecting these`,+ ` particular patches. The error above is the backend's own answer and says which.`,+ ];// No failure count in this message at all, deliberately, and neither// `breaker.limit` nor the live streak is read here. The fan-out stops on a// streak, the drain on a per-pass budget, and the batch gate on two empty@@ -529,9 +568,7 @@ export function describeCommitCutoff(// to `docker stats` and `/_api/query/current`, which is the wrong// subsystem entirely. The backend's own words are quoted above; let them// say which.- ` Either the database cannot keep up — \`ix doctor\` still passes, since an ArangoDB that`,- ` cannot begin a transaction is reachable and consistent — or the backend is rejecting these`,- ` particular patches. The error above is the backend's own answer and says which.`,+ ...cause,` Set IX_COMMIT_FAILURE_LIMIT=0 to send every patch regardless.`,].join("\n");}diff --git a/ix-cli/src/cli/commit-retry.ts b/ix-cli/src/cli/commit-retry.tsnew file mode 100644index 00000000..aef32cc5--- /dev/null+++ b/ix-cli/src/cli/commit-retry.ts@@ -0,0 +1,297 @@+// Copyright 2026 Ix Infrastructure Inc.++// ---------------------------------------------------------------------------+// Retrying one commit request (IN-19C).+//+// Two classes of failure are worth sending the same commit again for, and they+// want different waits:+//+// - lock conflicts and transport drops, which clear in milliseconds. These+// have always been retried, everywhere a commit is sent, on a 10 ms * 2^n+// backoff.+// - a backend or proxy saying "not now": 502, 503, 504 and 429. These clear+// in seconds, and a 429 or 503 may say how many in `Retry-After`. They are+// retried only where the caller asks (`transient: true`), which is the+// ingest's BULK commits: one request per chunk, at most three retries in+// a 5-second budget per bulk (`IX_COMMIT_TRANSIENT_RETRY_MS`). A+// `Retry-After` longer than what is left of the budget is cut to it. Hook+// runners give `ix map` about 30 s in all, so a long wait here would cost+// more than the retry can win back.+//+// The budget bounds TOTAL time, not just the sleeps: every attempt's own+// duration and every give-up check (a health probe) is charged to it, and a re-send is made only when the time spent so+// far, the wait, and one more attempt as long as the one that just failed all+// fit. A bulk that took longer than the budget to fail is therefore never sent+// again here; it goes straight to the caller's per-file fallback. A 504 gets at+// most ONE re-send on top of that: a gateway timeout means the request was+// slow, and the backend may still be applying it. Re-sending is safe either+// way -- the backend keys every patch by its id, so a bulk it already applied+// answers `Idempotent`, and one it partly applied answers a 409 naming the ids+// that landed -- but it is not free, so it is kept rare.+//+// Everything else is thrown at once. In particular 413 (the payload split owns+// that), 409 (the partial-commit and plan-mismatch handling owns that) and 500+// (a backend that answered and refused; the cutoff owns that). A 504 that the+// backend did apply comes back on the retry as a 409 naming the ids or as+// `Idempotent`, and both are already handled by the caller.+//+// Each class has its own bound: `IX_COMMIT_CONFLICT_RETRIES` counts conflict+// and transport retries only (0: none), and `IX_COMMIT_TRANSIENT_RETRY_MS`+// bounds the transient waits (0: no transient retries).+// ---------------------------------------------------------------------------++/** Lock contention, as opposed to the transport failures below. Kept separate+ * only so each group's reason for being retryable stays readable. */+const COMMIT_CONFLICT_RETRY_PATTERNS = [+ // ArangoDB lock conflicts. These were briefly split into their own constant+ // so `commitFailureIndictsBackend` could exclude them; that exclusion was+ // reverted in round 3 -- it made the cutoff inert in a real saturation mode --+ // and the split had no consumer left, so it is inlined.+ 'write-write conflict',+ 'timeout waiting to lock key',+ 'error: 1200',+ // Transport-level failures (k8s ingress reset, socket drop under load).+ //+ // These are retried because the request usually never reached the server --+ // but "usually" is the whole of Ix#495. A bulk commit is chunked and writes+ // its idempotency keys as each chunk lands, so a socket that drops after the+ // server started work leaves part of the group committed. Retrying then+ // re-sends a set the server has partly seen, and because the bulk group id is+ // a hash of the patch ids, the retry reproduces the same group and is refused+ // with a 409 that no further retry can clear.+ //+ // The retry stays (it is right far more often than not); what changed is that+ // the resulting 409 is now recoverable — see isBulkPartiallyCommittedError.+ 'fetch failed',+ 'econnreset',+ 'econnrefused',+];++// An abort from the shared wall-clock deadline (or a per-request timeout) must+// NOT be retried — the whole point of the deadline is to stop work. Retrying+// here would defeat it and add 6 more backed-off requests to an already-late+// run. AbortSignal.timeout raises a TimeoutError; an aborted deadline raises an+// AbortError; both also stringify with "aborted".+export function isAbortError(err: unknown): boolean {+ const name = (err as { name?: string } | null)?.name;+ if (name === "AbortError" || name === "TimeoutError") return true;+ return String(err).toLowerCase().includes("aborted");+}++export function isRetryableCommitConflict(err: unknown): boolean {+ if (isAbortError(err)) return false;+ const message = String(err).toLowerCase();+ return COMMIT_CONFLICT_RETRY_PATTERNS.some(pattern => message.includes(pattern));+}++/** Statuses that mean "not now" rather than "no". */+const TRANSIENT_COMMIT_STATUSES = new Set([429, 502, 503, 504]);++/**+ * The HTTP status of a failed request: the `status` `httpError` puts on the+ * error when it has the headers, else the `<status>:` its message starts with.+ */+function statusOf(err: unknown): number | undefined {+ const status = (err as { status?: unknown } | null)?.status;+ if (typeof status === "number") return status;+ const m = /^(\d{3}):/.exec((err as Error | null)?.message ?? "");+ return m ? Number(m[1]) : undefined;+}++/** A 502, 503, 504 or 429: worth one more try after a wait. Never an abort. */+export function isTransientCommitStatus(err: unknown): boolean {+ if (isAbortError(err)) return false;+ const status = statusOf(err);+ return status !== undefined && TRANSIENT_COMMIT_STATUSES.has(status);+}++/**+ * Milliseconds a `Retry-After` value asks for: delta-seconds or an HTTP-date+ * (IMF-fixdate), never below 0. Undefined when absent or unreadable, in which+ * case the caller backs off on its own schedule.+ */+export function retryAfterMs(value: string | undefined | null, now: number = Date.now()): number | undefined {+ const raw = value?.trim();+ if (!raw) return undefined;+ if (/^\d+$/.test(raw)) return Number(raw) * 1000;+ // Every HTTP-date form starts with the day name. Without this, `Date.parse`+ // reads "-1" as the year -1 and a malformed delay became "retry now".+ if (!/^[A-Za-z]/.test(raw)) return undefined;+ const at = Date.parse(raw);+ if (Number.isNaN(at)) return undefined;+ return Math.max(0, at - now);+}++function retryAfterOf(err: unknown): string | undefined {+ const value = (err as { retryAfter?: unknown } | null)?.retryAfter;+ return typeof value === "string" ? value : undefined;+}++/** A wait that ends early, rejecting, when `signal` aborts. */+function abortableSleep(ms: number, signal?: AbortSignal): Promise<void> {+ return new Promise<void>((resolve, reject) => {+ if (signal?.aborted) {+ reject(signal.reason);+ return;+ }+ const onAbort = (): void => {+ clearTimeout(timer);+ reject(signal!.reason);+ };+ const timer = setTimeout(() => {+ signal?.removeEventListener("abort", onAbort);+ resolve();+ }, ms);+ signal?.addEventListener("abort", onAbort, { once: true });+ });+}++/** The caller's answer, with a broken check read as "do not give up". */+async function giveUpSafely(+ shouldGiveUp: (error: unknown) => boolean | Promise<boolean>,+ error: unknown,+): Promise<boolean> {+ try {+ return (await shouldGiveUp(error)) === true;+ } catch {+ return false;+ }+}++/**+ * `promise`'s answer, or undefined if it has not answered within `ms`. The+ * promise is left to settle on its own; `giveUpSafely` never rejects.+ */+function withinBudget<T>(promise: Promise<T>, ms: number): Promise<T | undefined> {+ let timer: ReturnType<typeof setTimeout> | undefined;+ const timeout = new Promise<undefined>(resolve => { timer = setTimeout(() => resolve(undefined), ms); });+ return Promise.race([promise, timeout]).finally(() => clearTimeout(timer));+}++/** Total time one call may spend on transient statuses, attempts and waits together (the default of `IX_COMMIT_TRANSIENT_RETRY_MS`). */+export const TRANSIENT_RETRY_BUDGET_MS = 5_000;+/** Most transient retries one call makes, whatever the budget: a `Retry-After: 0` must not loop. */+export const TRANSIENT_MAX_RETRIES = 3;+/** Most re-sends after a 504: the backend may still be working on the first one. */+export const GATEWAY_TIMEOUT_MAX_RETRIES = 1;+/** First backoff for a transient status without `Retry-After`; doubles each time. */+const TRANSIENT_BASE_DELAY_MS = 500;++export interface RetryCommitOptions {+ /** Conflict and transport retries after the first attempt. 0 means none. Transient statuses are bounded by the budget instead. */+ maxRetries: number;+ /** Retry 502/503/504/429 too. Only the bulk commit sites set this. */+ transient: boolean;+ /**+ * Total time allowed for transient retries: every attempt's duration, every+ * give-up check and the waits. A re-send is made only if one more attempt as long as the last one+ * still fits; a longer wait is cut to what is left. 0 means no transient retries.+ */+ transientBudgetMs?: number;+ /** The run's deadline. A wait it interrupts ends the loop with the last error. */+ signal?: AbortSignal;+ /**+ * Has the caller given up on the backend? Consulted before a TRANSIENT retry+ * only: a run whose cutoff has tripped, or whose backend reports its database+ * down, should not spend its transient budget waiting on it. Conflict and transport+ * retries keep their existing behaviour.+ *+ * It may be async, so a caller can ASK the backend (a short health probe)+ * rather than only read what it already knows: on the first bulk of a run+ * nothing has asked yet. It gets the failure that prompted it. A rejection+ * counts as "keep going", so a probe that breaks never ends the loop.+ */+ shouldGiveUp?: (error: unknown) => boolean | Promise<boolean>;+ sleep?: (ms: number, signal?: AbortSignal) => Promise<void>;+ random?: () => number;+ now?: () => number;+ onRetry?: (info: { attempt: number; delayMs: number; error: unknown; kind: "conflict" | "transient" }) => void;+}++/**+ * Send a commit, and send it again while the failure is one a wait can fix.+ * The last error is rethrown when the retries, the budget or the caller's+ * patience run out, and at once for anything else.+ */+export async function retryCommit<T>(fn: () => Promise<T>, options: RetryCommitOptions): Promise<T> {+ const {+ maxRetries,+ transient,+ transientBudgetMs = TRANSIENT_RETRY_BUDGET_MS,+ signal,+ shouldGiveUp,+ sleep = abortableSleep,+ random = Math.random,+ now = Date.now,+ onRetry,+ } = options;+ // Time charged to the transient budget: every attempt's own duration, every give-up check and+ // every transient wait.+ let transientSpentMs = 0;+ let transientRetries = 0;+ let gatewayTimeoutRetries = 0;+ let conflictRetries = 0;+ for (;;) {+ const attemptStart = now();+ try {+ return await fn();+ } catch (err) {+ const attemptMs = Math.max(0, now() - attemptStart);+ transientSpentMs += attemptMs;+ if (signal?.aborted) throw err;+ let delay: number;+ let kind: "conflict" | "transient";+ if (isRetryableCommitConflict(err)) {+ if (conflictRetries >= maxRetries) throw err;+ // ArangoDB lock conflicts usually clear as soon as the competing transaction+ // commits. Transport errors may need a bit more breathing room under k8s load.+ delay = 10 * (1 << conflictRetries) + random() * 10;+ conflictRetries++;+ kind = "conflict";+ } else if (transient && isTransientCommitStatus(err)) {+ // What is left once the re-send itself is paid for, estimated as+ // long as the attempt that just failed. Nothing left: no re-send.+ let remaining = transientBudgetMs - transientSpentMs - attemptMs;+ if (transientRetries >= TRANSIENT_MAX_RETRIES || remaining <= 0) throw err;+ const gatewayTimeout = statusOf(err) === 504;+ if (gatewayTimeout && gatewayTimeoutRetries >= GATEWAY_TIMEOUT_MAX_RETRIES) throw err;+ if (shouldGiveUp) {+ // The give-up check may be a health probe that takes seconds. Its+ // time is charged to the budget like an attempt's, it may take no+ // longer than what is left (a check that has not answered by then+ // leaves no room for a re-send anyway), and what is left is worked+ // out again after it.+ const probeStart = now();+ const giveUp = await withinBudget(giveUpSafely(shouldGiveUp, err), remaining);+ transientSpentMs += Math.max(0, now() - probeStart);+ if (giveUp !== false) throw err;+ if (signal?.aborted) throw err;+ remaining = transientBudgetMs - transientSpentMs - attemptMs;+ if (remaining <= 0) throw err;+ }+ const asked = retryAfterMs(retryAfterOf(err), now());+ // Jittered by up to a quarter, so the chunks of one run that all met+ // the same restart do not come back in lockstep. Either wait is cut to+ // what is left of the budget, so `Retry-After` is honoured only up to it.+ delay = Math.min(+ remaining,+ asked ?? TRANSIENT_BASE_DELAY_MS * 2 ** transientRetries * (1 + random() * 0.25),+ );+ transientSpentMs += delay;+ transientRetries++;+ if (gatewayTimeout) gatewayTimeoutRetries++;+ kind = "transient";+ } else {+ throw err;+ }+ onRetry?.({ attempt: conflictRetries + transientRetries, delayMs: delay, error: err, kind });+ try {+ await sleep(delay, signal);+ } catch {+ // The deadline fired during the wait. The commit's own failure is the+ // one worth reporting, and the caller reads the deadline separately.+ throw err;+ }+ }+ }+}diff --git a/ix-cli/src/cli/database-probe.ts b/ix-cli/src/cli/database-probe.tsnew file mode 100644index 00000000..a57322fb--- /dev/null+++ b/ix-cli/src/cli/database-probe.ts@@ -0,0 +1,121 @@+// Copyright 2026 Ix Infrastructure Inc.++// ---------------------------------------------------------------------------+// "Can the backend reach its database?", asked quickly, for a run whose+// commits are failing (IN-19C, the CL-17 breaker follow-up).+//+// The commit cutoff (commit-breaker.ts) judges the backend by behaviour: a+// streak of refused commits. That is right when the cause is unknown, and slow+// when it is not: with ArangoDB down, every commit waits out its own failure+// and the drain spends its whole failure budget, three passes over, before the+// run stops. The backend knows the answer already. Since Ix-memory 1.0.33,+// `/v1/health` reports `database: "reachable" | "unreachable"`, and answers 503+// `degraded` while the database is down. An older backend never touched the+// database for health, so a read that does (`/v1/revisions/current`) stands in.+//+// Only an explicit answer counts. A probe that fails, times out, or is refused+// for want of the local token is `unknown`, and the caller then does exactly+// what it would have done without one.+//+// The answer only shortens the fan-out (the breaker trips, the rest is held)+// and the transient waits. It never decides the exit code: the held patches+// are still sent by the cutoff and the drain, as on main, so a wrong or stale+// "unreachable" cannot fail a run whose commits would land.+// ---------------------------------------------------------------------------++import type { HealthResponse } from "../client/types.js";++/**+ * The two reads a probe makes. `readHealth` is the caller's way to fetch+ * health, which in the CLI is `readBackendHealth`: health is fetched, and the+ * release it reports recorded, in exactly one place (backend-version.ts), and+ * this probe goes through it like every other reader.+ */+export interface DatabaseProbeReads {+ readHealth(timeoutMs: number): Promise<HealthResponse>;+ currentRevision(timeoutMs: number): Promise<unknown>;+}++/**+ * A health answer, including the 503 a backend sends while it cannot reach its+ * database: that is a server that answered, and the body says what is wrong.+ * Thrown for anything else.+ */+export function healthFromError(err: unknown): HealthResponse | null {+ const m = /^503:\s*(\{.*\})\s*$/s.exec((err as Error)?.message ?? "");+ if (!m) return null;+ try {+ const body = JSON.parse(m[1]!) as HealthResponse;+ return typeof body?.status === "string" ? body : null;+ } catch {+ return null;+ }+}++export type DatabaseVerdict = "reachable" | "unreachable" | "unknown";++/** How long a probe may take. It runs on a failing run's critical path. */+export const DATABASE_PROBE_TIMEOUT_MS = 2_000;++/**+ * Ask the backend whether it can reach its database.+ *+ * `unreachable` only on the backend's own word: health reporting+ * `database: "unreachable"` (with a 200, or the 503 `degraded` answer);+ * anything else, including a failed fallback read on an older backend, is+ * `unknown`.+ */+export async function probeDatabase(+ reads: DatabaseProbeReads,+ opts: { timeoutMs?: number } = {},+): Promise<DatabaseVerdict> {+ const timeoutMs = opts.timeoutMs ?? DATABASE_PROBE_TIMEOUT_MS;+ let health: HealthResponse;+ try {+ health = await reads.readHealth(timeoutMs);+ } catch (err) {+ const degraded = healthFromError(err);+ if (!degraded) return "unknown";+ health = degraded;+ }+ if (health.database === "unreachable") return "unreachable";+ if (health.database === "reachable") return "reachable";+ // A backend older than 1.0.33: health says nothing about the database.+ try {+ await reads.currentRevision(timeoutMs);+ return "reachable";+ } catch {+ // A failed read proves nothing about the database: a 5xx may be a saturated+ // Arango, a proxy or a restarting pod. Without health's own word, stay quiet.+ return "unknown";+ }+}++/**+ * One probe shared by every caller that asks within `minIntervalMs` of the+ * last answer, and by everyone who asks while one is in flight. A failing run+ * reaches the probe once per failed commit, and concurrency 8 means eight at+ * a time; the backend should be asked once.+ */+export function createDatabaseProbe(+ probe: () => Promise<DatabaseVerdict>,+ minIntervalMs = 5_000,+ now: () => number = Date.now,+): () => Promise<DatabaseVerdict> {+ let pending: Promise<DatabaseVerdict> | undefined;+ let last: DatabaseVerdict | undefined;+ let answeredAt = 0;+ return () => {+ if (pending) return pending;+ if (last !== undefined && now() - answeredAt < minIntervalMs) return Promise.resolve(last);+ pending = probe()+ .catch((): DatabaseVerdict => "unknown")+ .then(verdict => {+ last = verdict;+ answeredAt = now();+ pending = undefined;+ return verdict;+ });+ return pending;+ };+}diff --git a/ix-cli/src/client/__tests__/http-error.test.ts b/ix-cli/src/client/__tests__/http-error.test.tsnew file mode 100644index 00000000..2a95cee0--- /dev/null+++ b/ix-cli/src/client/__tests__/http-error.test.ts@@ -0,0 +1,42 @@+// Copyright 2026 Ix Infrastructure Inc.++import { describe, expect, it } from "vitest";+import { httpError, LocalTokenRequiredError } from "../api.js";++// `httpError`'s message is read back by many callers (status prefixes, error+// codes in the JSON body), so the headers it now takes for the commit retry+// (IN-19C) may add properties and must not touch the message.+describe("httpError", () => {+ const body = '{"error":"service unavailable"}';++ it("keeps the message byte-identical and carries status and Retry-After when given headers", () => {+ const err = httpError(503, body, new Headers({ "Retry-After": "2" })) as Error & { status?: number; retryAfter?: string };+ expect(err.message).toBe(`503: ${body}`);+ expect(err.message).toBe(httpError(503, body).message);+ expect(err.status).toBe(503);+ expect(err.retryAfter).toBe("2");+ // Non-enumerable: nothing that serialises or spreads an error picks them up.+ expect(Object.keys(err)).toEqual([]);+ });++ it("has no Retry-After when the response had none", () => {+ const err = httpError(502, "bad gateway", new Headers()) as Error & { status?: number; retryAfter?: string };+ expect(err.status).toBe(502);+ expect("retryAfter" in err).toBe(false);+ });++ it("adds nothing without headers", () => {+ const err = httpError(503, body);+ expect(err.message).toBe(`503: ${body}`);+ expect("status" in err).toBe(false);+ expect("retryAfter" in err).toBe(false);+ });++ it("still raises LocalTokenRequiredError for the loopback guard's 401", () => {+ const guard = '{"code":"local_token_required","error":"token required"}';+ const err = httpError(401, guard, new Headers({ "Retry-After": "1" }));+ expect(err).toBeInstanceOf(LocalTokenRequiredError);+ expect(err.message).toBe(`401: ${guard}`);+ expect("retryAfter" in err).toBe(false);+ });+});diff --git a/ix-cli/src/client/api.ts b/ix-cli/src/client/api.tsindex a9104f7f..f2a50d09 100644--- a/ix-cli/src/client/api.ts+++ b/ix-cli/src/client/api.ts@@ -367,8 +367,8 @@ export class IxClient {* The body is the head's revision record when the backend has one, and the* bare revision number when it does not; both are returned as-is.*/- async currentRevision(): Promise<unknown> {- return this.get("/v1/revisions/current");+ async currentRevision(timeoutMs?: number): Promise<unknown> {+ return this.get("/v1/revisions/current", timeoutMs);}async diff(@@ -405,7 +405,7 @@ export class IxClient {});if (!resp.ok) {const text = await resp.text();- throw httpError(resp.status, text);+ throw httpError(resp.status, text, resp.headers);}return resp.json() as Promise<PatchCommitResult>;}@@ -487,7 +487,7 @@ export class IxClient {});if (!resp.ok) {const text = await resp.text();- throw httpError(resp.status, text);+ throw httpError(resp.status, text, resp.headers);}return resp.json() as Promise<PatchCommitResult>;}@@ -779,8 +779,9 @@ export class IxClient {return this.get(`/v1/stats${qs ? `?${qs}` : ""}`, opts?.timeoutMs);}- async health(): Promise<HealthResponse> {- return this.get("/v1/health");+ /** `timeoutMs` bounds a probe that must answer in seconds; the default is unchanged. */+ async health(timeoutMs?: number): Promise<HealthResponse> {+ return this.get("/v1/health", timeoutMs);}async capabilities(): Promise<CapabilitiesResponse> {@@ -898,15 +899,37 @@ export class LocalTokenRequiredError extends Error {}}-/** The error for a non-2xx response: `<status>: <body>`, or a typed one we can explain. */-export function httpError(status: number, body: string): Error {+/** What `httpError` adds to its Error when it is handed the response headers. */+export interface HttpErrorDetail {+ /** The HTTP status, the same number the message starts with. */+ readonly status?: number;+ /** The raw `Retry-After` header, when the response carried one. */+ readonly retryAfter?: string;+}++/**+ * The error for a non-2xx response: `<status>: <body>`, or a typed one we can explain.+ *+ * With `headers`, the Error also carries `status` and, when the response had+ * one, `retryAfter` (see `HttpErrorDetail`), both non-enumerable. The message+ * is the same either way: many callers read the status and the error code back+ * out of it, so it must not change shape. Only the commit routes pass headers,+ * for the ingest's transient-status retry (`cli/commit-retry.ts`).+ */+export function httpError(status: number, body: string, headers?: Pick<Headers, "get">): Error {if (status === 401) {try {const parsed = JSON.parse(body) as { code?: unknown } | null;if (parsed?.code === "local_token_required") return new LocalTokenRequiredError(body);} catch { /* not the guard's JSON: a plain 401 */ }}- return new Error(`${status}: ${errorBodyForMessage(body)}`);+ const err = new Error(`${status}: ${errorBodyForMessage(body)}`);+ if (headers !== undefined) {+ Object.defineProperty(err, "status", { value: status, enumerable: false });+ const retryAfter = headers.get("retry-after");+ if (retryAfter !== null) Object.defineProperty(err, "retryAfter", { value: retryAfter, enumerable: false });+ }+ return err;}/**