Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

Model catalogs never hold startup or graff/models past a budget

Rach Pradhandf0155bc3f12
diff --git a/docs/adr/0306-model-catalogs-never-hold-startup.md b/docs/adr/0306-model-catalogs-never-hold-startup.md
new file mode 100644
index 00000000..6c2014b9
--- /dev/null
+++ b/docs/adr/0306-model-catalogs-never-hold-startup.md
@@ -0,0 +1,39 @@
+# 0306. Model catalogs never hold startup or `graff/models`
+
+Status: accepted 2026-10-28
+
+## Context
+
+ACP clients discover models by spawning `graff acp` and sending
+`initialize`, `session/new`, and `graff/models` under one short timeout.
+Boot awaited every live catalog GET: Kimi's account catalog on every start
+(it had no disk cache), and every router catalog whose cache was older than
+its TTL. The `graff/models` gateway refresh had a 500ms deadline, but on
+timeout it cancelled and joined its HTTP worker, and a worker parked in DNS
+or TLS cannot be cancelled. A cold DNS lookup or a slow host held
+`initialize` for 10 to 40 seconds and the client reported "graff model
+discovery timed out". The fleet elite pull used `io.async`, which runs
+inline when the pool is busy, so boot also waited on its 3s deadline.
+
+## Decision
+
+Catalog GETs run on detached threads (`catalog_bounded.zig`) with their own
+`Io` and allocator, and their only output is the disk cache.
+
+- A cached catalog, fresh or stale, activates immediately. A stale one (and
+ xAI's, always) refreshes behind it; nothing waits.
+- Only catalogs with no cache wait, together, for at most
+ `startup_budget` (800ms). A fetch that misses it is abandoned, not
+ cancelled, and still writes its cache for the next process.
+- Kimi's catalog is cached at `~/.codegraff/kimi-models.json` with a 6h TTL.
+- The `graff/models` gateway refresh uses the same worker, so its 500ms
+ deadline holds.
+- The fleet elite pull starts with `io.concurrent`, falling back to `async`.
+
+`/models` and `graff models` remain the blocking freshness moments.
+
+## Consequences
+
+The first start on a slow network, with no cache, may show baked rows for a
+provider until the next process. An abandoned worker keeps running until its
+own HTTP timeout; it touches no session state, so exit does not wait for it.
diff --git a/docs/adr/README.md b/docs/adr/README.md
index 7e7cc17a..364bf6ba 100644
--- a/docs/adr/README.md
+++ b/docs/adr/README.md
@@ -330,6 +330,7 @@ record only when you need the evidence or the edge cases.
| [0303](0303-terminal-markdown-and-worker-inspection.md) | The line REPL renders inline emphasis and offers a temporary mouse/keyboard `/workers` inspector over the existing bounded ACP child feed, without replacing the editor or changing engine behavior. |
| [0304](0304-file-tools-allow-in-root-symlinks.md) | File tools allow canonical in-root symlinks, reject escaping or unresolved links, and retain lexical confinement and missing write targets; the check is not an atomic sandbox. |
| [0305](0305-cancelled-checklist-items-carry-a-reason.md) | A `cancelled` todo needs a short reason or the write is rejected whole; it stays listed with that reason, stops blocking completion and never counts as verified. A verification item can be cancelled after a newer user message or an answered `ask_user`. |
+| [0306](0306-model-catalogs-never-hold-startup.md) | Boot and `graff/models` never wait on a catalog GET past a budget: cached catalogs activate now and refresh behind on detached workers whose only output is the disk cache; only uncached ones wait, together, up to 800ms. |
## When to write one
diff --git a/src/acp.zig b/src/acp.zig
index a58cecc5..ba3262f4 100644
--- a/src/acp.zig
+++ b/src/acp.zig
@@ -126,7 +126,7 @@ fn liveModels(ctx: *anyopaque, arena: Allocator, w: *Io.Writer, req: proto.Reque
// Hydrate deferred local metadata without network refresh or MCP startup.
@import("acp_model_option.zig").hydrate(live);
const er = @import("effort_route.zig");
- if (@import("gateway_picker_catalog.zig").requested(req.params)) @import("gateway_picker_catalog.zig").refresh(root.gpa, root.io, root.arena, keys.*);
+ if (@import("gateway_picker_catalog.zig").requested(req.params)) @import("gateway_picker_catalog.zig").refresh(root.io, root.arena, root.home, keys.*);
const catalog = pricing.models();
const Row = struct {
name: []const u8,
diff --git a/src/catalog_bounded.zig b/src/catalog_bounded.zig
new file mode 100644
index 00000000..9c2a72f6
--- /dev/null
+++ b/src/catalog_bounded.zig
@@ -0,0 +1,178 @@
+//! Live model catalogs never hold startup or `graff/models` past a budget.
+//!
+//! A cold DNS lookup or a slow TLS host could hold `initialize` for 10-40s,
+//! because boot awaited every live catalog GET (Kimi always, router specs
+//! whenever their cache was stale) and the `graff/models` gateway refresh
+//! joined its HTTP worker on timeout. ACP clients then gave up on model
+//! discovery. Catalog GETs now run on detached threads whose only output is
+//! the disk cache; the caller waits at most a budget, then activates whatever
+//! cache exists (fresh, stale, or none: the baked rows stay).
+//!
+//! A worker that misses the budget is abandoned, not cancelled: cancelling
+//! still joins a thread parked in getaddrinfo (mcp_teardown.zig has the same
+//! finding). It finishes on its own and leaves a fresh cache for the next
+//! process. Each worker owns a private Io and allocator, so it never touches
+//! the session Io or gpa after the caller moves on, and exit never waits for
+//! it or tears down an Io it is still inside.
+
+const std = @import("std");
+const builtin = @import("builtin");
+const Io = std.Io;
+
+const provider = @import("provider.zig");
+
+/// Boot waits this long for catalogs with no disk cache at all. A cached
+/// catalog, fresh or stale, never waits: it activates now and refreshes behind.
+pub const startup_budget: Io.Duration = .fromMilliseconds(800);
+
+pub const Job = union(enum) {
+ router: struct { spec: provider.ProviderSpec, source: provider.Keys.CredentialSource },
+ kimi,
+};
+
+/// Thread-safe and independent of every caller's lifetime: an abandoned task
+/// frees itself here, never into a session allocator's leak report.
+const alloc = std.heap.smp_allocator;
+
+const running: u8 = 0;
+const finished: u8 = 1;
+const abandoned: u8 = 2;
+
+const Task = struct {
+ state: std.atomic.Value(u8) = .init(running),
+ ok: bool = false,
+ job: Job,
+ home: []u8,
+ key: []u8,
+
+ fn destroy(t: *Task) void {
+ alloc.free(t.home);
+ alloc.free(t.key);
+ alloc.destroy(t);
+ }
+
+ fn run(t: *Task) void {
+ var threaded: Io.Threaded = .init(alloc, .{});
+ defer threaded.deinit();
+ const io = threaded.io();
+ t.ok = switch (t.job) {
+ .router => |r| @import("router_catalog.zig").fetchToCache(io, alloc, t.home, r.spec, t.key, r.source),
+ .kimi => @import("kimi_catalog.zig").fetchToCache(io, alloc, t.home, t.key),
+ };
+ // The waiter gave up: nobody else will free this task.
+ if (t.state.cmpxchgStrong(running, finished, .acq_rel, .acquire) != null) t.destroy();
+ }
+};
+
+fn start(job: Job, home: []const u8, key: []const u8) ?*Task {
+ if (builtin.single_threaded or home.len == 0 or key.len == 0) return null;
+ const t = alloc.create(Task) catch return null;
+ const home_d = alloc.dupe(u8, home) catch {
+ alloc.destroy(t);
+ return null;
+ };
+ const key_d = alloc.dupe(u8, key) catch {
+ alloc.free(home_d);
+ alloc.destroy(t);
+ return null;
+ };
+ t.* = .{ .job = job, .home = home_d, .key = key_d };
+ const thread = std.Thread.spawn(.{}, Task.run, .{t}) catch {
+ t.destroy();
+ return null;
+ };
+ thread.detach();
+ return t;
+}
+
+/// Refresh a cache behind the caller's back; nothing waits for it.
+pub fn detach(job: Job, home: []const u8, key: []const u8) void {
+ const t = start(job, home, key) orelse return;
+ if (t.state.cmpxchgStrong(running, abandoned, .acq_rel, .acquire) != null) t.destroy();
+}
+
+pub const max_tasks = provider.provider_specs.len + 2;
+
+/// Fetches that share one deadline: boot pays the slowest host, capped.
+pub const Batch = struct {
+ tasks: [max_tasks]?*Task = @splat(null),
+ ok: [max_tasks]bool = @splat(false),
+ len: usize = 0,
+
+ /// Start a fetch; returns its slot, or null when it could not start (no
+ /// home for a cache, no key, no thread). Its result is the cache file.
+ pub fn spawn(b: *Batch, job: Job, home: []const u8, key: []const u8) ?usize {
+ if (b.len == max_tasks) return null;
+ const t = start(job, home, key) orelse return null;
+ b.tasks[b.len] = t;
+ b.len += 1;
+ return b.len - 1;
+ }
+
+ /// Wait until every fetch finished or `budget` passed, then settle each:
+ /// a finished one is reclaimed and its outcome kept, a late one abandoned.
+ pub fn wait(b: *Batch, io: Io, budget: Io.Duration) void {
+ const step: Io.Duration = .fromMilliseconds(5);
+ var left = budget.nanoseconds;
+ while (left > 0 and !b.allFinished()) {
+ const nap: Io.Duration = .fromNanoseconds(@min(left, step.nanoseconds));
+ io.sleep(nap, .awake) catch break;
+ left -= nap.nanoseconds;
+ }
+ for (b.tasks[0..b.len], 0..) |maybe, i| {
+ const t = maybe orelse continue;
+ b.tasks[i] = null;
+ if (t.state.cmpxchgStrong(running, abandoned, .acq_rel, .acquire) == null) continue;
+ b.ok[i] = t.ok; // finished: the worker is done with `t`
+ t.destroy();
+ }
+ }
+
+ fn allFinished(b: *const Batch) bool {
+ for (b.tasks[0..b.len]) |maybe| {
+ const t = maybe orelse continue;
+ if (t.state.load(.acquire) == running) return false;
+ }
+ return true;
+ }
+
+ /// Whether the fetch in `slot` finished in time and wrote its cache.
+ pub fn fetched(b: *const Batch, slot: ?usize) bool {
+ const i = slot orelse return false;
+ return b.ok[i];
+ }
+};
+
+/// The abandoned worker keeps reading its spec after the test returns.
+var test_url_buf: [64]u8 = undefined;
+
+test "a batch that misses its budget returns on time and abandons the fetch" {
+ if (builtin.single_threaded) return error.SkipZigTest;
+ const io = std.testing.io;
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ // Accept the connection but never answer: the worker parks in its read.
+ var address = try Io.net.IpAddress.parseLiteral("127.0.0.1:0");
+ var server = try Io.net.IpAddress.listen(&address, io, .{});
+ defer server.deinit(io);
+ var spec = provider.specFor("codegraff").?;
+ spec.models_url = try std.fmt.bufPrint(&test_url_buf, "http://127.0.0.1:{d}/v1/models", .{server.socket.address.getPort()});
+ const home = try tmp.dir.realPathFileAlloc(io, ".", std.testing.allocator);
+ defer std.testing.allocator.free(home);
+
+ var batch: Batch = .{};
+ const slot = batch.spawn(.{ .router = .{ .spec = spec, .source = .environment } }, home, "test-key");
+ try std.testing.expect(slot != null);
+ const started = Io.Timestamp.now(io, .awake).nanoseconds;
+ batch.wait(io, .fromMilliseconds(100));
+ try std.testing.expect(Io.Timestamp.now(io, .awake).nanoseconds - started < std.time.ns_per_s);
+ try std.testing.expect(!batch.fetched(slot));
+}
+
+test "a batch reports nothing it could not start" {
+ var batch: Batch = .{};
+ try std.testing.expectEqual(@as(?usize, null), batch.spawn(.kimi, "", "key"));
+ try std.testing.expectEqual(@as(?usize, null), batch.spawn(.kimi, "/home", ""));
+ batch.wait(std.testing.io, .fromMilliseconds(1));
+ try std.testing.expect(!batch.fetched(null));
+}
diff --git a/src/commands_model.zig b/src/commands_model.zig
index 22e5cd08..322d04f4 100644
--- a/src/commands_model.zig
+++ b/src/commands_model.zig
@@ -112,7 +112,7 @@ pub fn tryHandle(root: *Agent, keys: *Keys, arena: Allocator, line: []const u8,
const arg = std.mem.trim(u8, line["/model".len..], " \t");
providers.ensureModelQueryCatalogs(root, keys.*, arg);
if (arg.len == 0) {
- @import("gateway_picker_catalog.zig").refresh(root.gpa, root.io, root.arena, keys.*);
+ @import("gateway_picker_catalog.zig").refresh(root.io, root.arena, root.home, keys.*);
if (main_mod.use_color) { // interactive TTY → fuzzy picker
if (modelPicker(root, keys, arena, out)) |idx| {
const m = pricing.models()[idx];
diff --git a/src/gateway_picker_catalog.zig b/src/gateway_picker_catalog.zig
index 7c539ba4..280b226c 100644
--- a/src/gateway_picker_catalog.zig
+++ b/src/gateway_picker_catalog.zig
@@ -2,6 +2,7 @@
const std = @import("std");
const Io = std.Io;
const Allocator = std.mem.Allocator;
+const catalog_bounded = @import("catalog_bounded.zig");
const router = @import("router_catalog.zig");
const provider = @import("provider.zig");
const util = @import("util.zig");
@@ -29,54 +30,36 @@ test "gateway refresh is skipped only by explicit false" {
try std.testing.expect(!requested(parsed.value));
}
-pub fn refresh(gpa: Allocator, io: Io, arena: Allocator, keys: provider.Keys) void {
+pub fn refresh(io: Io, arena: Allocator, home: []const u8, keys: provider.Keys) void {
const key = keys.get("codegraff") orelse return;
const now = util.unixMs(io);
if (last_attempt) |previous| if (now >= previous and now - previous < 30_000) return;
last_attempt = now;
const spec = provider.specFor("codegraff") orelse return;
- _ = refreshAt(gpa, io, arena, spec, key, keys.source("codegraff"), .fromMilliseconds(500));
+ _ = refreshAt(io, arena, home, spec, key, keys.source("codegraff"), .fromMilliseconds(500));
}
-pub fn refreshAt(gpa: Allocator, io: Io, arena: Allocator, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource, deadline: Io.Duration) bool {
- const buffer = gpa.alloc(u8, 2 * 1024 * 1024) catch return false;
- defer gpa.free(buffer);
- var writer: Io.Writer = .fixed(buffer);
- var client: std.http.Client = .{ .allocator = gpa, .io = io };
- defer client.deinit();
- var headers_buffer: [4]std.http.Header = undefined;
- var scratch = std.heap.ArenaAllocator.init(gpa);
- defer scratch.deinit();
- const headers = router.catalogHeaders(scratch.allocator(), spec, key, source, &headers_buffer) orelse return false;
- const Done = union(enum) { fetched: bool, timeout: void };
- var done_buffer: [2]Done = undefined;
- var select: Io.Select(Done) = .init(io, &done_buffer);
- select.concurrent(.timeout, delay, .{ io, deadline }) catch return false;
- select.concurrent(.fetched, fetch, .{ &client, spec.models_url, headers, &writer }) catch {
- select.cancelDiscard();
- return false;
- };
- const first = select.await() catch {
- select.cancelDiscard();
- return false;
- };
- select.cancelDiscard(); // join the HTTP worker before releasing its storage
- if (first != .fetched or !first.fetched) return false;
- const snapshot = router.parseModels(arena, spec.id, writer.buffered()) orelse return false;
- return router.activate(arena, spec, snapshot.models);
+/// Fetch into the disk cache on a detached worker and activate it only if it
+/// lands within `deadline`. Cancelling an HTTP worker joins it, and one parked
+/// in DNS or TLS held `graff/models` for seconds past this deadline; a late
+/// worker is abandoned instead and leaves the cache for the next caller.
+pub fn refreshAt(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource, deadline: Io.Duration) bool {
+ var batch: catalog_bounded.Batch = .{};
+ const slot = batch.spawn(.{ .router = .{ .spec = spec, .source = source } }, home, key);
+ batch.wait(io, deadline);
+ if (!batch.fetched(slot)) return false;
+ return router.activateCached(io, arena, home, spec);
}
-fn delay(io: Io, duration: Io.Duration) void {
- io.sleep(duration, .awake) catch {};
-}
-
-fn fetch(client: *std.http.Client, url: []const u8, headers: []const std.http.Header, writer: *Io.Writer) bool {
- const result = client.fetch(.{ .location = .{ .url = url }, .method = .GET, .extra_headers = headers, .response_writer = writer }) catch return false;
- return result.status == .ok;
-}
+/// The 50ms call's worker is abandoned and keeps reading its spec after the test.
+var test_url_buf: [64]u8 = undefined;
test "gateway picker refresh uses live aliases and preserves its snapshot on timeout" {
const gpa = std.testing.allocator;
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ const home = try tmp.dir.realPathFileAlloc(std.testing.io, ".", gpa);
+ defer gpa.free(home);
const io = std.testing.io;
const pricing = @import("pricing.zig");
const saved = pricing.active_model_table;
@@ -90,16 +73,16 @@ test "gateway picker refresh uses live aliases and preserves its snapshot on tim
defer group.cancel(io);
try group.concurrent(io, answer, .{ io, &server });
var spec = provider.specFor("codegraff").?;
- spec.models_url = try std.fmt.allocPrint(arena.allocator(), "http://127.0.0.1:{d}/v1/models", .{server.socket.address.getPort()});
- try std.testing.expect(refreshAt(gpa, io, arena.allocator(), spec, "test-key", .environment, .fromSeconds(1)));
+ spec.models_url = try std.fmt.bufPrint(&test_url_buf, "http://127.0.0.1:{d}/v1/models", .{server.socket.address.getPort()});
+ try std.testing.expect(refreshAt(io, arena.allocator(), home, spec, "test-key", .environment, .fromSeconds(1)));
try std.testing.expect(pricing.providerModelInTable("codegraff", "new-gateway-alias"));
const before = pricing.active_model_table;
for (0..2) |_| {
- try std.testing.expect(!refreshAt(gpa, io, arena.allocator(), spec, "test-key", .environment, .fromSeconds(1)));
+ try std.testing.expect(!refreshAt(io, arena.allocator(), home, spec, "test-key", .environment, .fromSeconds(1)));
try std.testing.expectEqual(before.ptr, pricing.active_model_table.ptr);
}
const started = Io.Timestamp.now(io, .awake).nanoseconds;
- try std.testing.expect(!refreshAt(gpa, io, arena.allocator(), spec, "test-key", .environment, .fromMilliseconds(50)));
+ try std.testing.expect(!refreshAt(io, arena.allocator(), home, spec, "test-key", .environment, .fromMilliseconds(50)));
try std.testing.expect(Io.Timestamp.now(io, .awake).nanoseconds - started < 2 * std.time.ns_per_s);
try std.testing.expectEqual(before.ptr, pricing.active_model_table.ptr);
}
diff --git a/src/kimi_catalog.zig b/src/kimi_catalog.zig
index dde0245a..d2aac481 100644
--- a/src/kimi_catalog.zig
+++ b/src/kimi_catalog.zig
@@ -114,10 +114,17 @@ pub const Fetch = struct {
pub fn fetch(io: Io, gpa: Allocator, arena: Allocator, access: []const u8) Fetch {
if (access.len == 0) return .{ .source = "baked fallback — no Kimi login" };
+ const body = fetchBody(io, gpa, arena, access) orelse return .{};
+ const rows = parseModels(arena, body) orelse
+ return .{ .source = "baked fallback — invalid live catalog" };
+ return .{ .rows = rows, .source = "live account catalog" };
+}
+
+fn fetchBody(io: Io, gpa: Allocator, arena: Allocator, access: []const u8) ?[]const u8 {
var client: std.http.Client = .{ .allocator = gpa, .io = io };
defer client.deinit();
var aw: Io.Writer.Allocating = .init(arena);
- const bearer = std.fmt.allocPrint(arena, "Bearer {s}", .{access}) catch return .{};
+ const bearer = std.fmt.allocPrint(arena, "Bearer {s}", .{access}) catch return null;
var extra: [8]std.http.Header = undefined;
extra[0] = .{ .name = "Authorization", .value = bearer };
extra[1] = .{ .name = "Accept", .value = "application/json" };
@@ -128,11 +135,51 @@ pub fn fetch(io: Io, gpa: Allocator, arena: Allocator, access: []const u8) Fetch
.response_writer = &aw.writer,
.headers = .{ .user_agent = .{ .override = user_agent } },
.extra_headers = extra[0..sent],
- }) catch return .{};
- if (@backingInt(response.status) != 200) return .{};
- const rows = parseModels(arena, aw.writer.buffered()) orelse
- return .{ .source = "baked fallback — invalid live catalog" };
- return .{ .rows = rows, .source = "live account catalog" };
+ }) catch return null;
+ if (@backingInt(response.status) != 200) return null;
+ return aw.writer.buffered();
+}
+
+/// The account catalog as last fetched, so boot activates it instead of
+/// waiting on the network; `catalog_bounded` rewrites it in the background.
+pub const cache_ttl_ms: i64 = 6 * 60 * 60 * 1000;
+
+fn cachePaths(arena: Allocator, home: []const u8) [2][]const u8 {
+ if (home.len == 0) return .{ "", "" };
+ return .{
+ std.fmt.allocPrint(arena, "{s}/.codegraff/kimi-models.json", .{home}) catch "",
+ std.fmt.allocPrint(arena, "{s}/.codegraff-kimi-models.json", .{home}) catch "",
+ };
+}
+
+/// Fetch the catalog and store the raw body; true when a valid catalog was
+/// written. Runs on a `catalog_bounded` worker with its own Io and allocator.
+pub fn fetchToCache(io: Io, gpa: Allocator, home: []const u8, access: []const u8) bool {
+ var arena_state = std.heap.ArenaAllocator.init(gpa);
+ defer arena_state.deinit();
+ const a = arena_state.allocator();
+ const body = fetchBody(io, gpa, a, access) orelse return false;
+ if (parseModels(a, body) == null) return false;
+ for (cachePaths(a, home)) |path| {
+ if (path.len == 0) continue;
+ @import("credential_store.zig").replaceFile(io, Io.Dir.cwd(), path, body, .default_file) catch continue;
+ return true;
+ }
+ return false;
+}
+
+pub const Cached = struct { rows: []const pricing.ModelInfo, fresh: bool };
+
+pub fn cached(io: Io, arena: Allocator, home: []const u8) ?Cached {
+ for (cachePaths(arena, home)) |path| {
+ if (path.len == 0) continue;
+ const data = Io.Dir.cwd().readFileAlloc(io, path, arena, .limited(1024 * 1024)) catch continue;
+ const rows = parseModels(arena, data) orelse continue;
+ const stat = Io.Dir.cwd().statFile(io, path, .{}) catch return .{ .rows = rows, .fresh = false };
+ const age = util.unixMs(io) - @as(i64, @intCast(@divTrunc(stat.mtime.nanoseconds, std.time.ns_per_ms)));
+ return .{ .rows = rows, .fresh = age >= 0 and age <= cache_ttl_ms };
+ }
+ return null;
}
fn retain(arena: Allocator, rows: []const pricing.ModelInfo) ?[]const pricing.ModelInfo {
@@ -163,7 +210,7 @@ pub fn commit(arena: Allocator, rows: []const pricing.ModelInfo, source: []const
catalog_source = source;
return;
};
- catalog_source = if (pricing.activateKimiModels(arena, kept)) "live account catalog" else source;
+ catalog_source = if (pricing.activateKimiModels(arena, kept)) source else "baked fallback — catalog rows rejected";
}
/// Claim the once-per-process fetch. Startup's fan-out calls this before
@@ -237,3 +284,24 @@ test "commit installs fetched Kimi rows into the active table" {
try std.testing.expect(pricing.providerModelInTable("kimi", "k9-fanout"));
try std.testing.expect(pricing.providerModelInTable("anthropic", "claude-opus-4-8"));
}
+
+test "the cached account catalog activates at boot and reports its age" {
+ const io = std.testing.io;
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var state = std.heap.ArenaAllocator.init(std.testing.allocator);
+ defer state.deinit();
+ const a = state.allocator();
+ const home = try tmp.dir.realPathFileAlloc(io, ".", a);
+ try std.testing.expect(cached(io, a, home) == null);
+ try tmp.dir.createDir(io, ".codegraff", .default_dir);
+ const path = try std.fmt.allocPrint(a, "{s}/.codegraff/kimi-models.json", .{home});
+ try Io.Dir.cwd().writeFile(io, .{ .sub_path = path, .data =
+ \\{"object":"list","data":[{"id":"k9-cached","context_length":262144}]}
+ });
+ const hit = cached(io, a, home) orelse return error.TestUnexpectedResult;
+ try std.testing.expect(hit.fresh);
+ try std.testing.expectEqualStrings("k9-cached", hit.rows[0].name);
+ try Io.Dir.cwd().writeFile(io, .{ .sub_path = path, .data = "not json" });
+ try std.testing.expect(cached(io, a, home) == null);
+}
diff --git a/src/router_catalog.zig b/src/router_catalog.zig
index db0d5abd..37f71f81 100644
--- a/src/router_catalog.zig
+++ b/src/router_catalog.zig
@@ -1,9 +1,10 @@
//! Model discovery for Codegraff, Anthropic, xAI, and the optional workspace router.
//!
-//! Workspace router cache: `.graff/.config.router`. xAI prefers a live GET but
-//! boot activates any disk cache, waits at most 500ms to recache, else
-//! refreshes in the background. `/models` is the blocking freshness moment.
-//! Catalog failures never prevent startup. Baked pricing.zig rows are offline fallback.
+//! Workspace router cache: `.graff/.config.router`. Boot activates any disk
+//! cache now (fresh, or stale and refreshed behind it; xAI always refreshes)
+//! and waits at most `catalog_bounded.startup_budget` only when there is none.
+//! `/models` is the blocking freshness moment. Catalog failures never prevent
+//! startup. Baked pricing.zig rows are offline fallback.
const std = @import("std");
const Io = std.Io;
@@ -13,6 +14,7 @@ const Value = std.json.Value;
const root = @import("main.zig");
const catalog_selection = @import("catalog_selection.zig");
const credential_store = @import("credential_store.zig");
+const catalog_bounded = @import("catalog_bounded.zig");
const kimi_catalog = @import("kimi_catalog.zig");
const pricing = @import("pricing.zig");
const pricing_db = @import("pricing_db.zig"); // #557: LiteLLM price/context overlay, hydrated on the same beats as these catalogs
@@ -27,43 +29,18 @@ const catalog_page_limit = 16 * 1024 * 1024; // Full provider lists may be much
var attempted: [provider.provider_specs.len]bool = @splat(false);
var additional_attempted = false;
-/// xAI boot uses any disk cache; live GET is budgeted, else background. `/models` still blocks.
-pub const live_budget_ms: i64 = 500;
+/// xAI refreshes behind even a fresh cache; `/models` still blocks.
pub fn alwaysLive(spec: provider.ProviderSpec) bool {
return std.mem.eql(u8, spec.id, "xai");
}
-var bg_refresh: Io.Future(void) = undefined;
-var bg_refresh_live = false;
-var bg_done: Io.Event = .unset;
-
-fn refreshCacheTask(io: Io, gpa: Allocator, home: []u8, spec: provider.ProviderSpec, key: []u8, source: provider.Keys.CredentialSource) void {
- defer gpa.free(home);
- defer gpa.free(key);
- defer bg_done.set(io);
+/// Fetch the live list into the disk cache; true when it was written. Runs
+/// on a `catalog_bounded` worker with its own Io and allocator.
+pub fn fetchToCache(io: Io, gpa: Allocator, home: []const u8, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource) bool {
var arena_state = std.heap.ArenaAllocator.init(gpa);
defer arena_state.deinit();
- const snapshot = fetch(io, gpa, arena_state.allocator(), spec, key, source) orelse return;
- writeCache(io, arena_state.allocator(), home, spec, snapshot.models);
-}
-
-fn startBackgroundRefresh(io: Io, gpa: Allocator, home: []const u8, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource) void {
- if (bg_refresh_live) return;
- const home_d = gpa.dupe(u8, home) catch return;
- const key_d = gpa.dupe(u8, key) catch {
- gpa.free(home_d);
- return;
- };
- bg_done.reset();
- const args = .{ io, gpa, home_d, spec, key_d, source };
- bg_refresh = io.concurrent(refreshCacheTask, args) catch io.async(refreshCacheTask, args);
- bg_refresh_live = true;
-}
-
-pub fn shutdown(io: Io) void {
- if (!bg_refresh_live) return;
- bg_refresh.await(io);
- bg_refresh_live = false;
+ const snapshot = fetch(io, gpa, arena_state.allocator(), spec, key, source) orelse return false;
+ return writeCache(io, arena_state.allocator(), home, spec, snapshot.models);
}
pub const Snapshot = struct {
@@ -323,13 +300,20 @@ pub fn cacheDocument(io: Io, arena: Allocator, spec: provider.ProviderSpec, mode
return aw.writer.buffered();
}
-fn writeCache(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec, models: []const pricing.ModelInfo) void {
- const document = cacheDocument(io, arena, spec, models) orelse return;
+fn writeCache(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec, models: []const pricing.ModelInfo) bool {
+ const document = cacheDocument(io, arena, spec, models) orelse return false;
for ([_][]const u8{ dirPath(arena, home, spec), flatPath(arena, home, spec) }) |path| {
if (path.len == 0) continue;
credential_store.replaceFile(io, Io.Dir.cwd(), path, document, .default_file) catch continue;
- return;
+ return true;
}
+ return false;
+}
+
+/// Activate the spec's disk cache, whatever its age.
+pub fn activateCached(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec) bool {
+ const snapshot = cachedSnapshot(io, arena, home, spec) orelse return false;
+ return activate(arena, spec, snapshot.models);
}
fn cachedSnapshot(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec) ?Snapshot {
@@ -362,7 +346,7 @@ fn loadSpec(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, spec: pr
};
if (fetch(io, gpa, arena, spec, key, source)) |snapshot| {
if (activate(arena, spec, snapshot.models)) {
- if (home.len != 0 or isAdditional(spec)) writeCache(io, arena, home, spec, snapshot.models);
+ if (home.len != 0 or isAdditional(spec)) _ = writeCache(io, arena, home, spec, snapshot.models);
return true;
}
}
@@ -437,7 +421,7 @@ fn finishFetch(io: Io, arena: Allocator, home: []const u8, outcome: FetchOutcome
if (outcome.models.len > 0) {
const kept = retainModels(arena, outcome.models) orelse return;
if (activate(arena, outcome.spec, kept) and outcome.write_cache)
- writeCache(io, arena, home, outcome.spec, kept);
+ _ = writeCache(io, arena, home, outcome.spec, kept);
return;
}
if (cachedSnapshot(io, arena, home, outcome.spec)) |snapshot|
@@ -451,20 +435,9 @@ fn cacheFresh(io: Io, arena: Allocator, home: []const u8, spec: provider.Provide
return age >= 0 and age <= cache_ttl_ms and activate(arena, spec, snapshot.models);
}
-const KimiOutcome = struct {
- fetch: kimi_catalog.Fetch = .{},
- arena_state: ?std.heap.ArenaAllocator = null,
-};
-
-fn kimiFetchTask(io: Io, gpa: Allocator, access: []const u8) KimiOutcome {
- var arena_state = std.heap.ArenaAllocator.init(gpa);
- return .{
- .fetch = kimi_catalog.fetch(io, gpa, arena_state.allocator(), access),
- .arena_state = arena_state,
- };
-}
-
-fn spawnKimi(io: Io, gpa: Allocator, arena: Allocator, keys: provider.Keys, model_flag: ?[]const u8, saved: ?serde.SavedModel) ?Io.Future(KimiOutcome) {
+/// Kimi's account catalog: any cache activates now (a stale one refreshes
+/// behind it); with none, a live fetch joins the boot batch. Returns its slot.
+fn startKimi(io: Io, batch: *catalog_bounded.Batch, arena: Allocator, home: []const u8, keys: provider.Keys, model_flag: ?[]const u8, saved: ?serde.SavedModel) ?usize {
if (!catalog_selection.startupMayUse(keys, "kimi", model_flag, saved)) return null;
if (!kimi_catalog.claimAttempt()) return null;
const access = keys.get("kimi") orelse "";
@@ -472,55 +445,65 @@ fn spawnKimi(io: Io, gpa: Allocator, arena: Allocator, keys: provider.Keys, mode
kimi_catalog.commit(arena, &.{}, "baked fallback — no Kimi login");
return null;
}
- const args = .{ io, gpa, access };
- return io.concurrent(kimiFetchTask, args) catch io.async(kimiFetchTask, args);
+ const cache = kimi_catalog.cached(io, arena, home) orelse return batch.spawn(.kimi, home, access);
+ kimi_catalog.commit(arena, cache.rows, "cached account catalog");
+ if (!cache.fresh) catalog_bounded.detach(.kimi, home, access);
+ return null;
}
-/// Startup catalog load. Fresh disk caches activate inline; live GETs run
-/// concurrently so boot pays the slowest host, not the sum. xAI activates any
-/// cache immediately, waits `live_budget_ms` only when there is none, and
-/// otherwise refreshes the cache file in the background.
+/// Startup catalog load. Never waits on a host that already has a disk
+/// cache: fresh caches activate as they are, stale ones activate and refresh
+/// behind. Hosts with no cache fetch concurrently under one budget, so boot
+/// pays at most `startup_budget` however slow DNS or TLS is; a fetch that
+/// misses it still caches its list for the next process.
pub fn ensureForStartup(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, keys: provider.Keys, model_flag: ?[]const u8, saved: ?serde.SavedModel) void {
- var kimi_task = spawnKimi(io, gpa, arena, keys, model_flag, saved);
- defer if (kimi_task) |*fut| {
- var outcome = fut.await(io);
- defer if (outcome.arena_state) |*ta| ta.deinit();
- kimi_catalog.commit(arena, outcome.fetch.rows, outcome.fetch.source);
- };
-
- var futures: [provider.provider_specs.len + 1]?Io.Future(FetchOutcome) = @splat(null);
+ _ = gpa;
+ var batch: catalog_bounded.Batch = .{};
+ const kimi_slot = startKimi(io, &batch, arena, home, keys, model_flag, saved);
+ var slots: [provider.provider_specs.len + 1]?usize = @splat(null);
for (provider.provider_specs, 0..) |spec, index| {
if (!dynamic(spec) or !catalog_selection.startupMayUse(keys, spec.id, model_flag, saved)) continue;
if (attempted[index]) continue;
const key = keys.get(spec.id) orelse continue;
if (key.len == 0) continue;
attempted[index] = true;
- const source = keys.source(spec.id);
- if (cacheFresh(io, arena, home, spec)) {
- if (alwaysLive(spec)) startBackgroundRefresh(io, gpa, home, spec, key, source);
- continue;
- }
- if (alwaysLive(spec)) {
- startBackgroundRefresh(io, gpa, home, spec, key, source);
- bg_done.waitTimeout(io, .{ .duration = .{ .raw = .fromMilliseconds(live_budget_ms), .clock = .awake } }) catch {};
- if (cachedSnapshot(io, arena, home, spec)) |snapshot| _ = activate(arena, spec, snapshot.models);
- continue;
- }
- futures[index] = spawnFetch(io, gpa, home, spec, key, source);
+ slots[index] = startSpec(io, &batch, arena, home, spec, key, keys.source(spec.id));
}
if (provider.additional_router) |spec| {
if (catalog_selection.startupMayUse(keys, spec.id, model_flag, saved) and !additional_attempted) {
if (keys.get(spec.id)) |key| if (key.len != 0) {
additional_attempted = true;
- if (!cacheFresh(io, arena, home, spec))
- futures[provider.provider_specs.len] = spawnFetch(io, gpa, home, spec, key, keys.source(spec.id));
+ slots[provider.provider_specs.len] = startSpec(io, &batch, arena, home, spec, key, keys.source(spec.id));
};
}
}
- for (&futures) |*maybe| {
- const outcome = if (maybe.*) |*fut| fut.await(io) else continue;
- finishFetch(io, arena, home, outcome);
+ if (batch.len == 0) return;
+ batch.wait(io, catalog_bounded.startup_budget);
+ if (batch.fetched(kimi_slot)) {
+ if (kimi_catalog.cached(io, arena, home)) |fresh| kimi_catalog.commit(arena, fresh.rows, "live account catalog");
+ } else if (kimi_slot != null) kimi_catalog.commit(arena, &.{}, "baked fallback — live catalog still loading");
+ for (provider.provider_specs, 0..) |spec, index|
+ if (batch.fetched(slots[index])) {
+ _ = activateCached(io, arena, home, spec);
+ };
+ if (provider.additional_router) |spec| if (batch.fetched(slots[provider.provider_specs.len])) {
+ _ = activateCached(io, arena, home, spec);
+ };
+}
+
+fn startSpec(io: Io, batch: *catalog_bounded.Batch, arena: Allocator, home: []const u8, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource) ?usize {
+ const job: catalog_bounded.Job = .{ .router = .{ .spec = spec, .source = source } };
+ // The workspace router caches under `.graff`, not home.
+ const cache_home = if (isAdditional(spec)) "." else home;
+ if (cacheFresh(io, arena, home, spec)) {
+ if (alwaysLive(spec)) catalog_bounded.detach(job, cache_home, key);
+ return null;
+ }
+ if (activateCached(io, arena, home, spec)) {
+ catalog_bounded.detach(job, cache_home, key);
+ return null;
}
+ return batch.spawn(job, cache_home, key);
}
/// An explicit /models listing is the freshness moment: hit every keyed
@@ -530,7 +513,6 @@ pub fn ensureForStartup(io: Io, gpa: Allocator, arena: Allocator, home: []const
/// remains the offline fallback, and each live list is re-cached for the
/// next boot. Returns how many providers were pinged.
pub fn refreshForListing(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, keys: provider.Keys) usize {
- shutdown(io);
pricing_db.ensure(io, gpa, arena, home, true); // the same freshness moment covers the price sheet, network and all
var futures: [provider.provider_specs.len + 1]?Io.Future(FetchOutcome) = @splat(null);
var pinged: usize = 0;
diff --git a/src/router_catalog_tests.zig b/src/router_catalog_tests.zig
index 03579ebd..c36ae260 100644
--- a/src/router_catalog_tests.zig
+++ b/src/router_catalog_tests.zig
@@ -282,7 +282,6 @@ test "alwaysLive is xAI-only so other OpenAI routers keep the disk TTL" {
try std.testing.expect(!alwaysLive(provider.specFor("codegraff").?));
try std.testing.expect(!alwaysLive(provider.specFor("anthropic").?));
try std.testing.expect(!alwaysLive(provider.specFor("fireworks").?));
- try std.testing.expectEqual(@as(i64, 500), rc.live_budget_ms);
}
test "xAI /v1/models rows inherit baked context windows" {
@@ -457,14 +456,16 @@ test "router discovery replaces only its provider slice" {
try std.testing.expect(pricing.providerModelInTable("codex", "gpt-6-sol"));
}
-test "startup catalog loads fan out concurrently with Kimi" {
+test "startup catalog loads fan out with Kimi under one bounded wait" {
const src = @embedFile("router_catalog.zig");
- try std.testing.expect(std.mem.indexOf(u8, src, "io.concurrent(fetchSpecTask") != null);
- try std.testing.expect(std.mem.indexOf(u8, src, "io.concurrent(kimiFetchTask") != null);
- try std.testing.expect(std.mem.indexOf(u8, src, "fn spawnKimi") != null);
- try std.testing.expect(std.mem.indexOf(u8, src, "startBackgroundRefresh") != null);
- try std.testing.expect(std.mem.indexOf(u8, src, "live_budget_ms") != null);
- try std.testing.expect(std.mem.indexOf(u8, src, "if (alwaysLive(spec)) return false") == null);
+ const start = std.mem.indexOf(u8, src, "pub fn ensureForStartup") orelse return error.TestUnexpectedResult;
+ const end = std.mem.indexOfPos(u8, src, start, "\nfn startSpec") orelse return error.TestUnexpectedResult;
+ const body = src[start..end];
+ // Boot never awaits a catalog GET: one batch, one budget, then the cache.
+ try std.testing.expect(std.mem.indexOf(u8, body, "startKimi(io, &batch") != null);
+ try std.testing.expect(std.mem.indexOf(u8, body, "batch.wait(io, catalog_bounded.startup_budget)") != null);
+ try std.testing.expect(std.mem.indexOf(u8, body, ".await(io)") == null);
+ try std.testing.expect(std.mem.indexOf(u8, src, "if (alwaysLive(spec)) catalog_bounded.detach(job") != null);
}
// ── provider.zig routing tests parked here under its 600-line ceiling ──────
diff --git a/src/session_globals.zig b/src/session_globals.zig
index 82e66aab..be8b9ff9 100644
--- a/src/session_globals.zig
+++ b/src/session_globals.zig
@@ -13,7 +13,6 @@ pub fn release(gpa: Allocator, io: Io) void {
presence.retire(io);
@import("presence_accord.zig").stop(io);
presence.deinit(gpa);
- @import("router_catalog.zig").shutdown(io);
// #1196: filled by the first successful MCP tool result.
@import("mcp_shapes.zig").shutdown(io);
}
diff --git a/src/session_run.zig b/src/session_run.zig
index f06d852d..de37b010 100644
--- a/src/session_run.zig
+++ b/src/session_run.zig
@@ -291,7 +291,10 @@ pub fn buildRootAgent(
// Background the fleet-champion pull: a ~0.3s TLS round-trip that used to block
// the first prompt. Spawn it now; joinElites() reaps it on the main thread at the
// first turn, so the user's typing hides the fetch (prompt paints ~0.3s sooner).
- fleet.g_elites_future = io.async(fleet.pullElites, .{ io, arena, client, telemetry.g_telem, telem_endpoint, scoring.providerClass(root.provider.model), arena.dupe(u8, pull_esh) catch pull_esh, fleet.g_agent_types });
+ // `concurrent`, not `async`: async runs inline when the pool is busy, as it
+ // is at boot, and then boot waited up to the pull's 3s deadline.
+ const pull_args = .{ io, arena, client, telemetry.g_telem, telem_endpoint, scoring.providerClass(root.provider.model), arena.dupe(u8, pull_esh) catch pull_esh, fleet.g_agent_types };
+ fleet.g_elites_future = io.concurrent(fleet.pullElites, pull_args) catch io.async(fleet.pullElites, pull_args);
return root;
}
diff --git a/src/startup_tests.zig b/src/startup_tests.zig
index d3398a87..9e605bbd 100644
--- a/src/startup_tests.zig
+++ b/src/startup_tests.zig
@@ -42,7 +42,6 @@ test "resolveKeys does not load Kimi/xAI OAuth credentials for an explicit unrel
const saved_models = pricing.active_model_table;
defer pricing.active_model_table = saved_models;
defer @import("http2_pool.zig").shutdown(io);
- defer @import("router_catalog.zig").shutdown(io);
// This test checks credential scope, so seed a fresh local catalog. It
// must not turn a synthetic key into a real provider GET during tier 1.
diff --git a/src/test_hooks.zig b/src/test_hooks.zig
index 8e8ef734..5c48814e 100644
--- a/src/test_hooks.zig
+++ b/src/test_hooks.zig
@@ -339,6 +339,7 @@ test {
_ = @import("piped_paste.zig"); // ADR 0231
_ = @import("mcp_connect_notice.zig"); // ADR 0230
_ = @import("image_sniff.zig"); // #1762: image media type from the bytes
+ _ = @import("catalog_bounded.zig"); // model discovery: catalog GETs never hold boot past a budget
_ = @import("oauth_callback.zig");
_ = @import("cache_key_tests.zig"); // ADR 0223: moved off http_headers.zig (600-line cap)
_ = @import("router_headers_tests.zig"); // moved off http_headers.zig (600-line cap)