graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
Model catalogs never hold startup or graff/models past a budget
Rach Pradhandf0155bc3f12
diff --git a/docs/adr/0306-model-catalogs-never-hold-startup.md b/docs/adr/0306-model-catalogs-never-hold-startup.mdnew file mode 100644index 00000000..6c2014b9--- /dev/null+++ b/docs/adr/0306-model-catalogs-never-hold-startup.md@@ -0,0 +1,39 @@+# 0306. Model catalogs never hold startup or `graff/models`++Status: accepted 2026-10-28++## Context++ACP clients discover models by spawning `graff acp` and sending+`initialize`, `session/new`, and `graff/models` under one short timeout.+Boot awaited every live catalog GET: Kimi's account catalog on every start+(it had no disk cache), and every router catalog whose cache was older than+its TTL. The `graff/models` gateway refresh had a 500ms deadline, but on+timeout it cancelled and joined its HTTP worker, and a worker parked in DNS+or TLS cannot be cancelled. A cold DNS lookup or a slow host held+`initialize` for 10 to 40 seconds and the client reported "graff model+discovery timed out". The fleet elite pull used `io.async`, which runs+inline when the pool is busy, so boot also waited on its 3s deadline.++## Decision++Catalog GETs run on detached threads (`catalog_bounded.zig`) with their own+`Io` and allocator, and their only output is the disk cache.++- A cached catalog, fresh or stale, activates immediately. A stale one (and+ xAI's, always) refreshes behind it; nothing waits.+- Only catalogs with no cache wait, together, for at most+ `startup_budget` (800ms). A fetch that misses it is abandoned, not+ cancelled, and still writes its cache for the next process.+- Kimi's catalog is cached at `~/.codegraff/kimi-models.json` with a 6h TTL.+- The `graff/models` gateway refresh uses the same worker, so its 500ms+ deadline holds.+- The fleet elite pull starts with `io.concurrent`, falling back to `async`.++`/models` and `graff models` remain the blocking freshness moments.++## Consequences++The first start on a slow network, with no cache, may show baked rows for a+provider until the next process. An abandoned worker keeps running until its+own HTTP timeout; it touches no session state, so exit does not wait for it.diff --git a/docs/adr/README.md b/docs/adr/README.mdindex 7e7cc17a..364bf6ba 100644--- a/docs/adr/README.md+++ b/docs/adr/README.md@@ -330,6 +330,7 @@ record only when you need the evidence or the edge cases.| [0303](0303-terminal-markdown-and-worker-inspection.md) | The line REPL renders inline emphasis and offers a temporary mouse/keyboard `/workers` inspector over the existing bounded ACP child feed, without replacing the editor or changing engine behavior. || [0304](0304-file-tools-allow-in-root-symlinks.md) | File tools allow canonical in-root symlinks, reject escaping or unresolved links, and retain lexical confinement and missing write targets; the check is not an atomic sandbox. || [0305](0305-cancelled-checklist-items-carry-a-reason.md) | A `cancelled` todo needs a short reason or the write is rejected whole; it stays listed with that reason, stops blocking completion and never counts as verified. A verification item can be cancelled after a newer user message or an answered `ask_user`. |+| [0306](0306-model-catalogs-never-hold-startup.md) | Boot and `graff/models` never wait on a catalog GET past a budget: cached catalogs activate now and refresh behind on detached workers whose only output is the disk cache; only uncached ones wait, together, up to 800ms. |## When to write onediff --git a/src/acp.zig b/src/acp.zigindex a58cecc5..ba3262f4 100644--- a/src/acp.zig+++ b/src/acp.zig@@ -126,7 +126,7 @@ fn liveModels(ctx: *anyopaque, arena: Allocator, w: *Io.Writer, req: proto.Reque// Hydrate deferred local metadata without network refresh or MCP startup.@import("acp_model_option.zig").hydrate(live);const er = @import("effort_route.zig");- if (@import("gateway_picker_catalog.zig").requested(req.params)) @import("gateway_picker_catalog.zig").refresh(root.gpa, root.io, root.arena, keys.*);+ if (@import("gateway_picker_catalog.zig").requested(req.params)) @import("gateway_picker_catalog.zig").refresh(root.io, root.arena, root.home, keys.*);const catalog = pricing.models();const Row = struct {name: []const u8,diff --git a/src/catalog_bounded.zig b/src/catalog_bounded.zignew file mode 100644index 00000000..9c2a72f6--- /dev/null+++ b/src/catalog_bounded.zig@@ -0,0 +1,178 @@+//! Live model catalogs never hold startup or `graff/models` past a budget.+//!+//! A cold DNS lookup or a slow TLS host could hold `initialize` for 10-40s,+//! because boot awaited every live catalog GET (Kimi always, router specs+//! whenever their cache was stale) and the `graff/models` gateway refresh+//! joined its HTTP worker on timeout. ACP clients then gave up on model+//! discovery. Catalog GETs now run on detached threads whose only output is+//! the disk cache; the caller waits at most a budget, then activates whatever+//! cache exists (fresh, stale, or none: the baked rows stay).+//!+//! A worker that misses the budget is abandoned, not cancelled: cancelling+//! still joins a thread parked in getaddrinfo (mcp_teardown.zig has the same+//! finding). It finishes on its own and leaves a fresh cache for the next+//! process. Each worker owns a private Io and allocator, so it never touches+//! the session Io or gpa after the caller moves on, and exit never waits for+//! it or tears down an Io it is still inside.++const std = @import("std");+const builtin = @import("builtin");+const Io = std.Io;++const provider = @import("provider.zig");++/// Boot waits this long for catalogs with no disk cache at all. A cached+/// catalog, fresh or stale, never waits: it activates now and refreshes behind.+pub const startup_budget: Io.Duration = .fromMilliseconds(800);++pub const Job = union(enum) {+ router: struct { spec: provider.ProviderSpec, source: provider.Keys.CredentialSource },+ kimi,+};++/// Thread-safe and independent of every caller's lifetime: an abandoned task+/// frees itself here, never into a session allocator's leak report.+const alloc = std.heap.smp_allocator;++const running: u8 = 0;+const finished: u8 = 1;+const abandoned: u8 = 2;++const Task = struct {+ state: std.atomic.Value(u8) = .init(running),+ ok: bool = false,+ job: Job,+ home: []u8,+ key: []u8,++ fn destroy(t: *Task) void {+ alloc.free(t.home);+ alloc.free(t.key);+ alloc.destroy(t);+ }++ fn run(t: *Task) void {+ var threaded: Io.Threaded = .init(alloc, .{});+ defer threaded.deinit();+ const io = threaded.io();+ t.ok = switch (t.job) {+ .router => |r| @import("router_catalog.zig").fetchToCache(io, alloc, t.home, r.spec, t.key, r.source),+ .kimi => @import("kimi_catalog.zig").fetchToCache(io, alloc, t.home, t.key),+ };+ // The waiter gave up: nobody else will free this task.+ if (t.state.cmpxchgStrong(running, finished, .acq_rel, .acquire) != null) t.destroy();+ }+};++fn start(job: Job, home: []const u8, key: []const u8) ?*Task {+ if (builtin.single_threaded or home.len == 0 or key.len == 0) return null;+ const t = alloc.create(Task) catch return null;+ const home_d = alloc.dupe(u8, home) catch {+ alloc.destroy(t);+ return null;+ };+ const key_d = alloc.dupe(u8, key) catch {+ alloc.free(home_d);+ alloc.destroy(t);+ return null;+ };+ t.* = .{ .job = job, .home = home_d, .key = key_d };+ const thread = std.Thread.spawn(.{}, Task.run, .{t}) catch {+ t.destroy();+ return null;+ };+ thread.detach();+ return t;+}++/// Refresh a cache behind the caller's back; nothing waits for it.+pub fn detach(job: Job, home: []const u8, key: []const u8) void {+ const t = start(job, home, key) orelse return;+ if (t.state.cmpxchgStrong(running, abandoned, .acq_rel, .acquire) != null) t.destroy();+}++pub const max_tasks = provider.provider_specs.len + 2;++/// Fetches that share one deadline: boot pays the slowest host, capped.+pub const Batch = struct {+ tasks: [max_tasks]?*Task = @splat(null),+ ok: [max_tasks]bool = @splat(false),+ len: usize = 0,++ /// Start a fetch; returns its slot, or null when it could not start (no+ /// home for a cache, no key, no thread). Its result is the cache file.+ pub fn spawn(b: *Batch, job: Job, home: []const u8, key: []const u8) ?usize {+ if (b.len == max_tasks) return null;+ const t = start(job, home, key) orelse return null;+ b.tasks[b.len] = t;+ b.len += 1;+ return b.len - 1;+ }++ /// Wait until every fetch finished or `budget` passed, then settle each:+ /// a finished one is reclaimed and its outcome kept, a late one abandoned.+ pub fn wait(b: *Batch, io: Io, budget: Io.Duration) void {+ const step: Io.Duration = .fromMilliseconds(5);+ var left = budget.nanoseconds;+ while (left > 0 and !b.allFinished()) {+ const nap: Io.Duration = .fromNanoseconds(@min(left, step.nanoseconds));+ io.sleep(nap, .awake) catch break;+ left -= nap.nanoseconds;+ }+ for (b.tasks[0..b.len], 0..) |maybe, i| {+ const t = maybe orelse continue;+ b.tasks[i] = null;+ if (t.state.cmpxchgStrong(running, abandoned, .acq_rel, .acquire) == null) continue;+ b.ok[i] = t.ok; // finished: the worker is done with `t`+ t.destroy();+ }+ }++ fn allFinished(b: *const Batch) bool {+ for (b.tasks[0..b.len]) |maybe| {+ const t = maybe orelse continue;+ if (t.state.load(.acquire) == running) return false;+ }+ return true;+ }++ /// Whether the fetch in `slot` finished in time and wrote its cache.+ pub fn fetched(b: *const Batch, slot: ?usize) bool {+ const i = slot orelse return false;+ return b.ok[i];+ }+};++/// The abandoned worker keeps reading its spec after the test returns.+var test_url_buf: [64]u8 = undefined;++test "a batch that misses its budget returns on time and abandons the fetch" {+ if (builtin.single_threaded) return error.SkipZigTest;+ const io = std.testing.io;+ var tmp = std.testing.tmpDir(.{});+ defer tmp.cleanup();+ // Accept the connection but never answer: the worker parks in its read.+ var address = try Io.net.IpAddress.parseLiteral("127.0.0.1:0");+ var server = try Io.net.IpAddress.listen(&address, io, .{});+ defer server.deinit(io);+ var spec = provider.specFor("codegraff").?;+ spec.models_url = try std.fmt.bufPrint(&test_url_buf, "http://127.0.0.1:{d}/v1/models", .{server.socket.address.getPort()});+ const home = try tmp.dir.realPathFileAlloc(io, ".", std.testing.allocator);+ defer std.testing.allocator.free(home);++ var batch: Batch = .{};+ const slot = batch.spawn(.{ .router = .{ .spec = spec, .source = .environment } }, home, "test-key");+ try std.testing.expect(slot != null);+ const started = Io.Timestamp.now(io, .awake).nanoseconds;+ batch.wait(io, .fromMilliseconds(100));+ try std.testing.expect(Io.Timestamp.now(io, .awake).nanoseconds - started < std.time.ns_per_s);+ try std.testing.expect(!batch.fetched(slot));+}++test "a batch reports nothing it could not start" {+ var batch: Batch = .{};+ try std.testing.expectEqual(@as(?usize, null), batch.spawn(.kimi, "", "key"));+ try std.testing.expectEqual(@as(?usize, null), batch.spawn(.kimi, "/home", ""));+ batch.wait(std.testing.io, .fromMilliseconds(1));+ try std.testing.expect(!batch.fetched(null));+}diff --git a/src/commands_model.zig b/src/commands_model.zigindex 22e5cd08..322d04f4 100644--- a/src/commands_model.zig+++ b/src/commands_model.zig@@ -112,7 +112,7 @@ pub fn tryHandle(root: *Agent, keys: *Keys, arena: Allocator, line: []const u8,const arg = std.mem.trim(u8, line["/model".len..], " \t");providers.ensureModelQueryCatalogs(root, keys.*, arg);if (arg.len == 0) {- @import("gateway_picker_catalog.zig").refresh(root.gpa, root.io, root.arena, keys.*);+ @import("gateway_picker_catalog.zig").refresh(root.io, root.arena, root.home, keys.*);if (main_mod.use_color) { // interactive TTY → fuzzy pickerif (modelPicker(root, keys, arena, out)) |idx| {const m = pricing.models()[idx];diff --git a/src/gateway_picker_catalog.zig b/src/gateway_picker_catalog.zigindex 7c539ba4..280b226c 100644--- a/src/gateway_picker_catalog.zig+++ b/src/gateway_picker_catalog.zig@@ -2,6 +2,7 @@const std = @import("std");const Io = std.Io;const Allocator = std.mem.Allocator;+const catalog_bounded = @import("catalog_bounded.zig");const router = @import("router_catalog.zig");const provider = @import("provider.zig");const util = @import("util.zig");@@ -29,54 +30,36 @@ test "gateway refresh is skipped only by explicit false" {try std.testing.expect(!requested(parsed.value));}-pub fn refresh(gpa: Allocator, io: Io, arena: Allocator, keys: provider.Keys) void {+pub fn refresh(io: Io, arena: Allocator, home: []const u8, keys: provider.Keys) void {const key = keys.get("codegraff") orelse return;const now = util.unixMs(io);if (last_attempt) |previous| if (now >= previous and now - previous < 30_000) return;last_attempt = now;const spec = provider.specFor("codegraff") orelse return;- _ = refreshAt(gpa, io, arena, spec, key, keys.source("codegraff"), .fromMilliseconds(500));+ _ = refreshAt(io, arena, home, spec, key, keys.source("codegraff"), .fromMilliseconds(500));}-pub fn refreshAt(gpa: Allocator, io: Io, arena: Allocator, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource, deadline: Io.Duration) bool {- const buffer = gpa.alloc(u8, 2 * 1024 * 1024) catch return false;- defer gpa.free(buffer);- var writer: Io.Writer = .fixed(buffer);- var client: std.http.Client = .{ .allocator = gpa, .io = io };- defer client.deinit();- var headers_buffer: [4]std.http.Header = undefined;- var scratch = std.heap.ArenaAllocator.init(gpa);- defer scratch.deinit();- const headers = router.catalogHeaders(scratch.allocator(), spec, key, source, &headers_buffer) orelse return false;- const Done = union(enum) { fetched: bool, timeout: void };- var done_buffer: [2]Done = undefined;- var select: Io.Select(Done) = .init(io, &done_buffer);- select.concurrent(.timeout, delay, .{ io, deadline }) catch return false;- select.concurrent(.fetched, fetch, .{ &client, spec.models_url, headers, &writer }) catch {- select.cancelDiscard();- return false;- };- const first = select.await() catch {- select.cancelDiscard();- return false;- };- select.cancelDiscard(); // join the HTTP worker before releasing its storage- if (first != .fetched or !first.fetched) return false;- const snapshot = router.parseModels(arena, spec.id, writer.buffered()) orelse return false;- return router.activate(arena, spec, snapshot.models);+/// Fetch into the disk cache on a detached worker and activate it only if it+/// lands within `deadline`. Cancelling an HTTP worker joins it, and one parked+/// in DNS or TLS held `graff/models` for seconds past this deadline; a late+/// worker is abandoned instead and leaves the cache for the next caller.+pub fn refreshAt(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource, deadline: Io.Duration) bool {+ var batch: catalog_bounded.Batch = .{};+ const slot = batch.spawn(.{ .router = .{ .spec = spec, .source = source } }, home, key);+ batch.wait(io, deadline);+ if (!batch.fetched(slot)) return false;+ return router.activateCached(io, arena, home, spec);}-fn delay(io: Io, duration: Io.Duration) void {- io.sleep(duration, .awake) catch {};-}--fn fetch(client: *std.http.Client, url: []const u8, headers: []const std.http.Header, writer: *Io.Writer) bool {- const result = client.fetch(.{ .location = .{ .url = url }, .method = .GET, .extra_headers = headers, .response_writer = writer }) catch return false;- return result.status == .ok;-}+/// The 50ms call's worker is abandoned and keeps reading its spec after the test.+var test_url_buf: [64]u8 = undefined;test "gateway picker refresh uses live aliases and preserves its snapshot on timeout" {const gpa = std.testing.allocator;+ var tmp = std.testing.tmpDir(.{});+ defer tmp.cleanup();+ const home = try tmp.dir.realPathFileAlloc(std.testing.io, ".", gpa);+ defer gpa.free(home);const io = std.testing.io;const pricing = @import("pricing.zig");const saved = pricing.active_model_table;@@ -90,16 +73,16 @@ test "gateway picker refresh uses live aliases and preserves its snapshot on timdefer group.cancel(io);try group.concurrent(io, answer, .{ io, &server });var spec = provider.specFor("codegraff").?;- spec.models_url = try std.fmt.allocPrint(arena.allocator(), "http://127.0.0.1:{d}/v1/models", .{server.socket.address.getPort()});- try std.testing.expect(refreshAt(gpa, io, arena.allocator(), spec, "test-key", .environment, .fromSeconds(1)));+ spec.models_url = try std.fmt.bufPrint(&test_url_buf, "http://127.0.0.1:{d}/v1/models", .{server.socket.address.getPort()});+ try std.testing.expect(refreshAt(io, arena.allocator(), home, spec, "test-key", .environment, .fromSeconds(1)));try std.testing.expect(pricing.providerModelInTable("codegraff", "new-gateway-alias"));const before = pricing.active_model_table;for (0..2) |_| {- try std.testing.expect(!refreshAt(gpa, io, arena.allocator(), spec, "test-key", .environment, .fromSeconds(1)));+ try std.testing.expect(!refreshAt(io, arena.allocator(), home, spec, "test-key", .environment, .fromSeconds(1)));try std.testing.expectEqual(before.ptr, pricing.active_model_table.ptr);}const started = Io.Timestamp.now(io, .awake).nanoseconds;- try std.testing.expect(!refreshAt(gpa, io, arena.allocator(), spec, "test-key", .environment, .fromMilliseconds(50)));+ try std.testing.expect(!refreshAt(io, arena.allocator(), home, spec, "test-key", .environment, .fromMilliseconds(50)));try std.testing.expect(Io.Timestamp.now(io, .awake).nanoseconds - started < 2 * std.time.ns_per_s);try std.testing.expectEqual(before.ptr, pricing.active_model_table.ptr);}diff --git a/src/kimi_catalog.zig b/src/kimi_catalog.zigindex dde0245a..d2aac481 100644--- a/src/kimi_catalog.zig+++ b/src/kimi_catalog.zig@@ -114,10 +114,17 @@ pub const Fetch = struct {pub fn fetch(io: Io, gpa: Allocator, arena: Allocator, access: []const u8) Fetch {if (access.len == 0) return .{ .source = "baked fallback — no Kimi login" };+ const body = fetchBody(io, gpa, arena, access) orelse return .{};+ const rows = parseModels(arena, body) orelse+ return .{ .source = "baked fallback — invalid live catalog" };+ return .{ .rows = rows, .source = "live account catalog" };+}++fn fetchBody(io: Io, gpa: Allocator, arena: Allocator, access: []const u8) ?[]const u8 {var client: std.http.Client = .{ .allocator = gpa, .io = io };defer client.deinit();var aw: Io.Writer.Allocating = .init(arena);- const bearer = std.fmt.allocPrint(arena, "Bearer {s}", .{access}) catch return .{};+ const bearer = std.fmt.allocPrint(arena, "Bearer {s}", .{access}) catch return null;var extra: [8]std.http.Header = undefined;extra[0] = .{ .name = "Authorization", .value = bearer };extra[1] = .{ .name = "Accept", .value = "application/json" };@@ -128,11 +135,51 @@ pub fn fetch(io: Io, gpa: Allocator, arena: Allocator, access: []const u8) Fetch.response_writer = &aw.writer,.headers = .{ .user_agent = .{ .override = user_agent } },.extra_headers = extra[0..sent],- }) catch return .{};- if (@backingInt(response.status) != 200) return .{};- const rows = parseModels(arena, aw.writer.buffered()) orelse- return .{ .source = "baked fallback — invalid live catalog" };- return .{ .rows = rows, .source = "live account catalog" };+ }) catch return null;+ if (@backingInt(response.status) != 200) return null;+ return aw.writer.buffered();+}++/// The account catalog as last fetched, so boot activates it instead of+/// waiting on the network; `catalog_bounded` rewrites it in the background.+pub const cache_ttl_ms: i64 = 6 * 60 * 60 * 1000;++fn cachePaths(arena: Allocator, home: []const u8) [2][]const u8 {+ if (home.len == 0) return .{ "", "" };+ return .{+ std.fmt.allocPrint(arena, "{s}/.codegraff/kimi-models.json", .{home}) catch "",+ std.fmt.allocPrint(arena, "{s}/.codegraff-kimi-models.json", .{home}) catch "",+ };+}++/// Fetch the catalog and store the raw body; true when a valid catalog was+/// written. Runs on a `catalog_bounded` worker with its own Io and allocator.+pub fn fetchToCache(io: Io, gpa: Allocator, home: []const u8, access: []const u8) bool {+ var arena_state = std.heap.ArenaAllocator.init(gpa);+ defer arena_state.deinit();+ const a = arena_state.allocator();+ const body = fetchBody(io, gpa, a, access) orelse return false;+ if (parseModels(a, body) == null) return false;+ for (cachePaths(a, home)) |path| {+ if (path.len == 0) continue;+ @import("credential_store.zig").replaceFile(io, Io.Dir.cwd(), path, body, .default_file) catch continue;+ return true;+ }+ return false;+}++pub const Cached = struct { rows: []const pricing.ModelInfo, fresh: bool };++pub fn cached(io: Io, arena: Allocator, home: []const u8) ?Cached {+ for (cachePaths(arena, home)) |path| {+ if (path.len == 0) continue;+ const data = Io.Dir.cwd().readFileAlloc(io, path, arena, .limited(1024 * 1024)) catch continue;+ const rows = parseModels(arena, data) orelse continue;+ const stat = Io.Dir.cwd().statFile(io, path, .{}) catch return .{ .rows = rows, .fresh = false };+ const age = util.unixMs(io) - @as(i64, @intCast(@divTrunc(stat.mtime.nanoseconds, std.time.ns_per_ms)));+ return .{ .rows = rows, .fresh = age >= 0 and age <= cache_ttl_ms };+ }+ return null;}fn retain(arena: Allocator, rows: []const pricing.ModelInfo) ?[]const pricing.ModelInfo {@@ -163,7 +210,7 @@ pub fn commit(arena: Allocator, rows: []const pricing.ModelInfo, source: []constcatalog_source = source;return;};- catalog_source = if (pricing.activateKimiModels(arena, kept)) "live account catalog" else source;+ catalog_source = if (pricing.activateKimiModels(arena, kept)) source else "baked fallback — catalog rows rejected";}/// Claim the once-per-process fetch. Startup's fan-out calls this before@@ -237,3 +284,24 @@ test "commit installs fetched Kimi rows into the active table" {try std.testing.expect(pricing.providerModelInTable("kimi", "k9-fanout"));try std.testing.expect(pricing.providerModelInTable("anthropic", "claude-opus-4-8"));}++test "the cached account catalog activates at boot and reports its age" {+ const io = std.testing.io;+ var tmp = std.testing.tmpDir(.{});+ defer tmp.cleanup();+ var state = std.heap.ArenaAllocator.init(std.testing.allocator);+ defer state.deinit();+ const a = state.allocator();+ const home = try tmp.dir.realPathFileAlloc(io, ".", a);+ try std.testing.expect(cached(io, a, home) == null);+ try tmp.dir.createDir(io, ".codegraff", .default_dir);+ const path = try std.fmt.allocPrint(a, "{s}/.codegraff/kimi-models.json", .{home});+ try Io.Dir.cwd().writeFile(io, .{ .sub_path = path, .data =+ \\{"object":"list","data":[{"id":"k9-cached","context_length":262144}]}+ });+ const hit = cached(io, a, home) orelse return error.TestUnexpectedResult;+ try std.testing.expect(hit.fresh);+ try std.testing.expectEqualStrings("k9-cached", hit.rows[0].name);+ try Io.Dir.cwd().writeFile(io, .{ .sub_path = path, .data = "not json" });+ try std.testing.expect(cached(io, a, home) == null);+}diff --git a/src/router_catalog.zig b/src/router_catalog.zigindex db0d5abd..37f71f81 100644--- a/src/router_catalog.zig+++ b/src/router_catalog.zig@@ -1,9 +1,10 @@//! Model discovery for Codegraff, Anthropic, xAI, and the optional workspace router.//!-//! Workspace router cache: `.graff/.config.router`. xAI prefers a live GET but-//! boot activates any disk cache, waits at most 500ms to recache, else-//! refreshes in the background. `/models` is the blocking freshness moment.-//! Catalog failures never prevent startup. Baked pricing.zig rows are offline fallback.+//! Workspace router cache: `.graff/.config.router`. Boot activates any disk+//! cache now (fresh, or stale and refreshed behind it; xAI always refreshes)+//! and waits at most `catalog_bounded.startup_budget` only when there is none.+//! `/models` is the blocking freshness moment. Catalog failures never prevent+//! startup. Baked pricing.zig rows are offline fallback.const std = @import("std");const Io = std.Io;@@ -13,6 +14,7 @@ const Value = std.json.Value;const root = @import("main.zig");const catalog_selection = @import("catalog_selection.zig");const credential_store = @import("credential_store.zig");+const catalog_bounded = @import("catalog_bounded.zig");const kimi_catalog = @import("kimi_catalog.zig");const pricing = @import("pricing.zig");const pricing_db = @import("pricing_db.zig"); // #557: LiteLLM price/context overlay, hydrated on the same beats as these catalogs@@ -27,43 +29,18 @@ const catalog_page_limit = 16 * 1024 * 1024; // Full provider lists may be muchvar attempted: [provider.provider_specs.len]bool = @splat(false);var additional_attempted = false;-/// xAI boot uses any disk cache; live GET is budgeted, else background. `/models` still blocks.-pub const live_budget_ms: i64 = 500;+/// xAI refreshes behind even a fresh cache; `/models` still blocks.pub fn alwaysLive(spec: provider.ProviderSpec) bool {return std.mem.eql(u8, spec.id, "xai");}-var bg_refresh: Io.Future(void) = undefined;-var bg_refresh_live = false;-var bg_done: Io.Event = .unset;--fn refreshCacheTask(io: Io, gpa: Allocator, home: []u8, spec: provider.ProviderSpec, key: []u8, source: provider.Keys.CredentialSource) void {- defer gpa.free(home);- defer gpa.free(key);- defer bg_done.set(io);+/// Fetch the live list into the disk cache; true when it was written. Runs+/// on a `catalog_bounded` worker with its own Io and allocator.+pub fn fetchToCache(io: Io, gpa: Allocator, home: []const u8, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource) bool {var arena_state = std.heap.ArenaAllocator.init(gpa);defer arena_state.deinit();- const snapshot = fetch(io, gpa, arena_state.allocator(), spec, key, source) orelse return;- writeCache(io, arena_state.allocator(), home, spec, snapshot.models);-}--fn startBackgroundRefresh(io: Io, gpa: Allocator, home: []const u8, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource) void {- if (bg_refresh_live) return;- const home_d = gpa.dupe(u8, home) catch return;- const key_d = gpa.dupe(u8, key) catch {- gpa.free(home_d);- return;- };- bg_done.reset();- const args = .{ io, gpa, home_d, spec, key_d, source };- bg_refresh = io.concurrent(refreshCacheTask, args) catch io.async(refreshCacheTask, args);- bg_refresh_live = true;-}--pub fn shutdown(io: Io) void {- if (!bg_refresh_live) return;- bg_refresh.await(io);- bg_refresh_live = false;+ const snapshot = fetch(io, gpa, arena_state.allocator(), spec, key, source) orelse return false;+ return writeCache(io, arena_state.allocator(), home, spec, snapshot.models);}pub const Snapshot = struct {@@ -323,13 +300,20 @@ pub fn cacheDocument(io: Io, arena: Allocator, spec: provider.ProviderSpec, modereturn aw.writer.buffered();}-fn writeCache(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec, models: []const pricing.ModelInfo) void {- const document = cacheDocument(io, arena, spec, models) orelse return;+fn writeCache(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec, models: []const pricing.ModelInfo) bool {+ const document = cacheDocument(io, arena, spec, models) orelse return false;for ([_][]const u8{ dirPath(arena, home, spec), flatPath(arena, home, spec) }) |path| {if (path.len == 0) continue;credential_store.replaceFile(io, Io.Dir.cwd(), path, document, .default_file) catch continue;- return;+ return true;}+ return false;+}++/// Activate the spec's disk cache, whatever its age.+pub fn activateCached(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec) bool {+ const snapshot = cachedSnapshot(io, arena, home, spec) orelse return false;+ return activate(arena, spec, snapshot.models);}fn cachedSnapshot(io: Io, arena: Allocator, home: []const u8, spec: provider.ProviderSpec) ?Snapshot {@@ -362,7 +346,7 @@ fn loadSpec(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, spec: pr};if (fetch(io, gpa, arena, spec, key, source)) |snapshot| {if (activate(arena, spec, snapshot.models)) {- if (home.len != 0 or isAdditional(spec)) writeCache(io, arena, home, spec, snapshot.models);+ if (home.len != 0 or isAdditional(spec)) _ = writeCache(io, arena, home, spec, snapshot.models);return true;}}@@ -437,7 +421,7 @@ fn finishFetch(io: Io, arena: Allocator, home: []const u8, outcome: FetchOutcomeif (outcome.models.len > 0) {const kept = retainModels(arena, outcome.models) orelse return;if (activate(arena, outcome.spec, kept) and outcome.write_cache)- writeCache(io, arena, home, outcome.spec, kept);+ _ = writeCache(io, arena, home, outcome.spec, kept);return;}if (cachedSnapshot(io, arena, home, outcome.spec)) |snapshot|@@ -451,20 +435,9 @@ fn cacheFresh(io: Io, arena: Allocator, home: []const u8, spec: provider.Providereturn age >= 0 and age <= cache_ttl_ms and activate(arena, spec, snapshot.models);}-const KimiOutcome = struct {- fetch: kimi_catalog.Fetch = .{},- arena_state: ?std.heap.ArenaAllocator = null,-};--fn kimiFetchTask(io: Io, gpa: Allocator, access: []const u8) KimiOutcome {- var arena_state = std.heap.ArenaAllocator.init(gpa);- return .{- .fetch = kimi_catalog.fetch(io, gpa, arena_state.allocator(), access),- .arena_state = arena_state,- };-}--fn spawnKimi(io: Io, gpa: Allocator, arena: Allocator, keys: provider.Keys, model_flag: ?[]const u8, saved: ?serde.SavedModel) ?Io.Future(KimiOutcome) {+/// Kimi's account catalog: any cache activates now (a stale one refreshes+/// behind it); with none, a live fetch joins the boot batch. Returns its slot.+fn startKimi(io: Io, batch: *catalog_bounded.Batch, arena: Allocator, home: []const u8, keys: provider.Keys, model_flag: ?[]const u8, saved: ?serde.SavedModel) ?usize {if (!catalog_selection.startupMayUse(keys, "kimi", model_flag, saved)) return null;if (!kimi_catalog.claimAttempt()) return null;const access = keys.get("kimi") orelse "";@@ -472,55 +445,65 @@ fn spawnKimi(io: Io, gpa: Allocator, arena: Allocator, keys: provider.Keys, modekimi_catalog.commit(arena, &.{}, "baked fallback — no Kimi login");return null;}- const args = .{ io, gpa, access };- return io.concurrent(kimiFetchTask, args) catch io.async(kimiFetchTask, args);+ const cache = kimi_catalog.cached(io, arena, home) orelse return batch.spawn(.kimi, home, access);+ kimi_catalog.commit(arena, cache.rows, "cached account catalog");+ if (!cache.fresh) catalog_bounded.detach(.kimi, home, access);+ return null;}-/// Startup catalog load. Fresh disk caches activate inline; live GETs run-/// concurrently so boot pays the slowest host, not the sum. xAI activates any-/// cache immediately, waits `live_budget_ms` only when there is none, and-/// otherwise refreshes the cache file in the background.+/// Startup catalog load. Never waits on a host that already has a disk+/// cache: fresh caches activate as they are, stale ones activate and refresh+/// behind. Hosts with no cache fetch concurrently under one budget, so boot+/// pays at most `startup_budget` however slow DNS or TLS is; a fetch that+/// misses it still caches its list for the next process.pub fn ensureForStartup(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, keys: provider.Keys, model_flag: ?[]const u8, saved: ?serde.SavedModel) void {- var kimi_task = spawnKimi(io, gpa, arena, keys, model_flag, saved);- defer if (kimi_task) |*fut| {- var outcome = fut.await(io);- defer if (outcome.arena_state) |*ta| ta.deinit();- kimi_catalog.commit(arena, outcome.fetch.rows, outcome.fetch.source);- };-- var futures: [provider.provider_specs.len + 1]?Io.Future(FetchOutcome) = @splat(null);+ _ = gpa;+ var batch: catalog_bounded.Batch = .{};+ const kimi_slot = startKimi(io, &batch, arena, home, keys, model_flag, saved);+ var slots: [provider.provider_specs.len + 1]?usize = @splat(null);for (provider.provider_specs, 0..) |spec, index| {if (!dynamic(spec) or !catalog_selection.startupMayUse(keys, spec.id, model_flag, saved)) continue;if (attempted[index]) continue;const key = keys.get(spec.id) orelse continue;if (key.len == 0) continue;attempted[index] = true;- const source = keys.source(spec.id);- if (cacheFresh(io, arena, home, spec)) {- if (alwaysLive(spec)) startBackgroundRefresh(io, gpa, home, spec, key, source);- continue;- }- if (alwaysLive(spec)) {- startBackgroundRefresh(io, gpa, home, spec, key, source);- bg_done.waitTimeout(io, .{ .duration = .{ .raw = .fromMilliseconds(live_budget_ms), .clock = .awake } }) catch {};- if (cachedSnapshot(io, arena, home, spec)) |snapshot| _ = activate(arena, spec, snapshot.models);- continue;- }- futures[index] = spawnFetch(io, gpa, home, spec, key, source);+ slots[index] = startSpec(io, &batch, arena, home, spec, key, keys.source(spec.id));}if (provider.additional_router) |spec| {if (catalog_selection.startupMayUse(keys, spec.id, model_flag, saved) and !additional_attempted) {if (keys.get(spec.id)) |key| if (key.len != 0) {additional_attempted = true;- if (!cacheFresh(io, arena, home, spec))- futures[provider.provider_specs.len] = spawnFetch(io, gpa, home, spec, key, keys.source(spec.id));+ slots[provider.provider_specs.len] = startSpec(io, &batch, arena, home, spec, key, keys.source(spec.id));};}}- for (&futures) |*maybe| {- const outcome = if (maybe.*) |*fut| fut.await(io) else continue;- finishFetch(io, arena, home, outcome);+ if (batch.len == 0) return;+ batch.wait(io, catalog_bounded.startup_budget);+ if (batch.fetched(kimi_slot)) {+ if (kimi_catalog.cached(io, arena, home)) |fresh| kimi_catalog.commit(arena, fresh.rows, "live account catalog");+ } else if (kimi_slot != null) kimi_catalog.commit(arena, &.{}, "baked fallback — live catalog still loading");+ for (provider.provider_specs, 0..) |spec, index|+ if (batch.fetched(slots[index])) {+ _ = activateCached(io, arena, home, spec);+ };+ if (provider.additional_router) |spec| if (batch.fetched(slots[provider.provider_specs.len])) {+ _ = activateCached(io, arena, home, spec);+ };+}++fn startSpec(io: Io, batch: *catalog_bounded.Batch, arena: Allocator, home: []const u8, spec: provider.ProviderSpec, key: []const u8, source: provider.Keys.CredentialSource) ?usize {+ const job: catalog_bounded.Job = .{ .router = .{ .spec = spec, .source = source } };+ // The workspace router caches under `.graff`, not home.+ const cache_home = if (isAdditional(spec)) "." else home;+ if (cacheFresh(io, arena, home, spec)) {+ if (alwaysLive(spec)) catalog_bounded.detach(job, cache_home, key);+ return null;+ }+ if (activateCached(io, arena, home, spec)) {+ catalog_bounded.detach(job, cache_home, key);+ return null;}+ return batch.spawn(job, cache_home, key);}/// An explicit /models listing is the freshness moment: hit every keyed@@ -530,7 +513,6 @@ pub fn ensureForStartup(io: Io, gpa: Allocator, arena: Allocator, home: []const/// remains the offline fallback, and each live list is re-cached for the/// next boot. Returns how many providers were pinged.pub fn refreshForListing(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, keys: provider.Keys) usize {- shutdown(io);pricing_db.ensure(io, gpa, arena, home, true); // the same freshness moment covers the price sheet, network and allvar futures: [provider.provider_specs.len + 1]?Io.Future(FetchOutcome) = @splat(null);var pinged: usize = 0;diff --git a/src/router_catalog_tests.zig b/src/router_catalog_tests.zigindex 03579ebd..c36ae260 100644--- a/src/router_catalog_tests.zig+++ b/src/router_catalog_tests.zig@@ -282,7 +282,6 @@ test "alwaysLive is xAI-only so other OpenAI routers keep the disk TTL" {try std.testing.expect(!alwaysLive(provider.specFor("codegraff").?));try std.testing.expect(!alwaysLive(provider.specFor("anthropic").?));try std.testing.expect(!alwaysLive(provider.specFor("fireworks").?));- try std.testing.expectEqual(@as(i64, 500), rc.live_budget_ms);}test "xAI /v1/models rows inherit baked context windows" {@@ -457,14 +456,16 @@ test "router discovery replaces only its provider slice" {try std.testing.expect(pricing.providerModelInTable("codex", "gpt-6-sol"));}-test "startup catalog loads fan out concurrently with Kimi" {+test "startup catalog loads fan out with Kimi under one bounded wait" {const src = @embedFile("router_catalog.zig");- try std.testing.expect(std.mem.indexOf(u8, src, "io.concurrent(fetchSpecTask") != null);- try std.testing.expect(std.mem.indexOf(u8, src, "io.concurrent(kimiFetchTask") != null);- try std.testing.expect(std.mem.indexOf(u8, src, "fn spawnKimi") != null);- try std.testing.expect(std.mem.indexOf(u8, src, "startBackgroundRefresh") != null);- try std.testing.expect(std.mem.indexOf(u8, src, "live_budget_ms") != null);- try std.testing.expect(std.mem.indexOf(u8, src, "if (alwaysLive(spec)) return false") == null);+ const start = std.mem.indexOf(u8, src, "pub fn ensureForStartup") orelse return error.TestUnexpectedResult;+ const end = std.mem.indexOfPos(u8, src, start, "\nfn startSpec") orelse return error.TestUnexpectedResult;+ const body = src[start..end];+ // Boot never awaits a catalog GET: one batch, one budget, then the cache.+ try std.testing.expect(std.mem.indexOf(u8, body, "startKimi(io, &batch") != null);+ try std.testing.expect(std.mem.indexOf(u8, body, "batch.wait(io, catalog_bounded.startup_budget)") != null);+ try std.testing.expect(std.mem.indexOf(u8, body, ".await(io)") == null);+ try std.testing.expect(std.mem.indexOf(u8, src, "if (alwaysLive(spec)) catalog_bounded.detach(job") != null);}// ── provider.zig routing tests parked here under its 600-line ceiling ──────diff --git a/src/session_globals.zig b/src/session_globals.zigindex 82e66aab..be8b9ff9 100644--- a/src/session_globals.zig+++ b/src/session_globals.zig@@ -13,7 +13,6 @@ pub fn release(gpa: Allocator, io: Io) void {presence.retire(io);@import("presence_accord.zig").stop(io);presence.deinit(gpa);- @import("router_catalog.zig").shutdown(io);// #1196: filled by the first successful MCP tool result.@import("mcp_shapes.zig").shutdown(io);}diff --git a/src/session_run.zig b/src/session_run.zigindex f06d852d..de37b010 100644--- a/src/session_run.zig+++ b/src/session_run.zig@@ -291,7 +291,10 @@ pub fn buildRootAgent(// Background the fleet-champion pull: a ~0.3s TLS round-trip that used to block// the first prompt. Spawn it now; joinElites() reaps it on the main thread at the// first turn, so the user's typing hides the fetch (prompt paints ~0.3s sooner).- fleet.g_elites_future = io.async(fleet.pullElites, .{ io, arena, client, telemetry.g_telem, telem_endpoint, scoring.providerClass(root.provider.model), arena.dupe(u8, pull_esh) catch pull_esh, fleet.g_agent_types });+ // `concurrent`, not `async`: async runs inline when the pool is busy, as it+ // is at boot, and then boot waited up to the pull's 3s deadline.+ const pull_args = .{ io, arena, client, telemetry.g_telem, telem_endpoint, scoring.providerClass(root.provider.model), arena.dupe(u8, pull_esh) catch pull_esh, fleet.g_agent_types };+ fleet.g_elites_future = io.concurrent(fleet.pullElites, pull_args) catch io.async(fleet.pullElites, pull_args);return root;}diff --git a/src/startup_tests.zig b/src/startup_tests.zigindex d3398a87..9e605bbd 100644--- a/src/startup_tests.zig+++ b/src/startup_tests.zig@@ -42,7 +42,6 @@ test "resolveKeys does not load Kimi/xAI OAuth credentials for an explicit unrelconst saved_models = pricing.active_model_table;defer pricing.active_model_table = saved_models;defer @import("http2_pool.zig").shutdown(io);- defer @import("router_catalog.zig").shutdown(io);// This test checks credential scope, so seed a fresh local catalog. It// must not turn a synthetic key into a real provider GET during tier 1.diff --git a/src/test_hooks.zig b/src/test_hooks.zigindex 8e8ef734..5c48814e 100644--- a/src/test_hooks.zig+++ b/src/test_hooks.zig@@ -339,6 +339,7 @@ test {_ = @import("piped_paste.zig"); // ADR 0231_ = @import("mcp_connect_notice.zig"); // ADR 0230_ = @import("image_sniff.zig"); // #1762: image media type from the bytes+ _ = @import("catalog_bounded.zig"); // model discovery: catalog GETs never hold boot past a budget_ = @import("oauth_callback.zig");_ = @import("cache_key_tests.zig"); // ADR 0223: moved off http_headers.zig (600-line cap)_ = @import("router_headers_tests.zig"); // moved off http_headers.zig (600-line cap)