Multiplayer agent harness for work.
Discover gateway models without an environment allowlist (#1162)
Regan Bell8738eab4d8d5
diff --git a/README.md b/README.mdindex a939d0d7b..15e8ce227 100644--- a/README.md+++ b/README.md@@ -206,6 +206,7 @@ messages, and screenshots for organization identifiers before it pushes. Nothing## Going deeper- [`docs/swarms.md`](./docs/swarms.md) — durable agent pools, scoped messages, and blank Modal workers+- [`docs/model-gateway.md`](./docs/model-gateway.md) — discover and route models through a gateway- [`docs/getting-started.md`](./docs/getting-started.md) — first run, end to end- [`cli/README.md`](./cli/README.md) — the `qm` CLI and the deployment directory contract- [`docs/deploy-directory.md`](./docs/deploy-directory.md) — the deployment directory in fulldiff --git a/docs/model-gateway.md b/docs/model-gateway.mdnew file mode 100644index 000000000..ba6dfab34--- /dev/null+++ b/docs/model-gateway.md@@ -0,0 +1,17 @@+# Model gateways++Pi can route model calls through an authenticated gateway instead of holding provider keys. Set `MODEL_GATEWAY_URL`, `MODEL_GATEWAY_API_KEY`, and `MODEL_GATEWAY_API_KEY_HEADER`. The key stays server-side. No model list is required.++QM discovers models using authenticated `GET /v1/models` and `GET /model_group/info`, the end-user metadata endpoint implemented by LiteLLM. A base URL ending in `/v1` is supported; both endpoints retain any preceding path prefix. Discovery uses the same key as inference, rejects redirects, and ignores upstream credential and endpoint fields.++Only models listed by both endpoints, marked as chat models with tool support, and carrying valid context, output and pricing metadata are offered. Discovered IDs are namespaced as `gateway/<upstream-id>`. Groups whose metadata identifies only Anthropic providers use native Messages, preserving signed thinking blocks across tool calls. Other groups use OpenAI-compatible streaming chat completions. Reasoning controls are enabled for native Anthropic and OpenAI/Azure groups; other provider combinations use standard chat without requesting reasoning effort. They are available in Pi, not process harnesses. The gateway owns provider translation and group capabilities. QM selects the protocol from provider metadata, never model names. Gateways must expose `/v1/messages` for Anthropic groups.++Availability refreshes on demand every five minutes and before inference when expired. Successful refreshes replace the catalog, including an empty catalog. After discovery has succeeded, a failed refresh disables gateway routes until recovery without interrupting independent direct providers, with retries after thirty seconds. In-flight requests are not cancelled; the gateway still enforces its key permissions on every call. Retired selections remain in conversation history but cannot issue a new request through a different provider.++`MODEL_GATEWAY_MODELS` is optional and accepts the existing `local-id=upstream-id` pairs. These provide compatibility aliases using QM's existing native model definitions; they do not limit discovered models. After discovery succeeds, aliases are usable only while their upstream IDs remain advertised. Gateways without the discovery endpoints can continue to use explicit mappings. Before the first successful discovery, failures leave those configured routes available. This compatibility fallback never restores a route removed by a successful discovery.++For LiteLLM deployments, allow the two authenticated GET endpoints alongside inference paths. Do not expose administrative `/model/info` or key-management routes to enable discovery. Gateways implementing only the OpenAI model-list endpoint can use explicit mappings or implement the metadata contract below.++The metadata endpoint returns `{"data": [...]}` with one object per model group. QM uses `model_group`, `mode`, `supports_function_calling`, `max_input_tokens`, `max_output_tokens`, `input_cost_per_token`, and `output_cost_per_token`. Optional fields include `providers`, `supports_vision`, `supports_reasoning`, `supports_adaptive_thinking`, `supported_openai_params`, `cache_read_input_token_cost`, and `cache_creation_input_token_cost`. Prices are per token and converted to per-million-token prices for QM. Missing cache prices use the input rate. Input capacity is used conservatively as the context budget; output capacity must be smaller.++LiteLLM group metadata may combine capabilities and maximum limits from multiple deployments. It is a gateway contract, not a guarantee that every backing deployment has identical capabilities.diff --git a/plugins/web-ui/src/model-options.ts b/plugins/web-ui/src/model-options.tsindex c41d68a7f..1f6c6679b 100644--- a/plugins/web-ui/src/model-options.ts+++ b/plugins/web-ui/src/model-options.ts@@ -23,6 +23,7 @@ const HARNESS_LABELS: Record<string, string> = {};const PROVIDER_LABELS: Record<string, string> = {+ "qm:gateway": "Gateway",anthropic: "Anthropic",openai: "OpenAI",openrouter: "OpenRouter",diff --git a/src/api/runtime-config.ts b/src/api/runtime-config.tsindex 8cf677629..a9d467caf 100644--- a/src/api/runtime-config.ts+++ b/src/api/runtime-config.ts@@ -1,3 +1,4 @@+import { gatewayModelCatalog } from "../model/gateway-models.ts";import type { ServerDeps } from "./deps.ts";import type { ScopeId } from "../types.ts";import { orgScope } from "./routes/shared.ts";@@ -32,16 +33,21 @@ export type RuntimeDeps = Pick<export function runtimeFallback(ctx: { deps: RuntimeDeps }): { harnessId: HarnessId; modelId: string } {const harnessId = isHarnessId(ctx.deps.harnessId) ? ctx.deps.harnessId : "pi";- return { harnessId, modelId: ctx.deps.baseModelDefault ?? defaultModelForHarness(harnessId) };+ const gatewayIds = gatewayModelCatalog().map((model) => model.id);+ const providers =+ gatewayIds.length && ctx.deps.providerKeys+ ? { ...ctx.deps.providerKeys, modelIds: new Set(gatewayIds) }+ : undefined;+ return { harnessId, modelId: ctx.deps.baseModelDefault ?? defaultModelForHarness(harnessId, undefined, providers) };}export async function runtimeConfigBody(ctx: { deps: RuntimeDeps }, scope: ScopeId) {const config = ctx.deps.config!;+ const configuredKeys = ctx.deps.providerKeys ?? ALL_PROVIDERS_AVAILABLE;+ const managedKeys = ctx.deps.modelCredentials ? await ctx.deps.modelCredentials.availability() : configuredKeys;const fallback = runtimeFallback(ctx);const org = orgScope(ctx.deps);const approvedHarnesses = ((await config.getApprovedHarnessesDurable()) ?? [fallback.harnessId]).filter(isHarnessId);- const configuredKeys = ctx.deps.providerKeys ?? ALL_PROVIDERS_AVAILABLE;- const managedKeys = ctx.deps.modelCredentials ? await ctx.deps.modelCredentials.availability() : configuredKeys;const providersFor = (harnessId: string) => modelProviderAvailabilityFor(harnessId, configuredKeys, managedKeys);const catalog =ctx.deps.modelCredentials && managedKeys.openrouterdiff --git a/src/config.ts b/src/config.tsindex cd93404c0..29263b47d 100644--- a/src/config.ts+++ b/src/config.ts@@ -933,7 +933,7 @@ function csvPaths(value: string | undefined): string[] | undefined {function modelGatewayFromEnv(env: NodeJS.ProcessEnv): ModelGatewayTransportConfig | undefined {const names = ["MODEL_GATEWAY_URL", "MODEL_GATEWAY_API_KEY", "MODEL_GATEWAY_API_KEY_HEADER", "MODEL_GATEWAY_MODELS"];if (!names.some((name) => env[name]?.trim())) return undefined;- for (const name of names) {+ for (const name of names.filter((name) => name !== "MODEL_GATEWAY_MODELS")) {if (!env[name]?.trim()) throw new Error(`${name} is required when model gateway routing is configured`);}const apiKeyHeader = env.MODEL_GATEWAY_API_KEY_HEADER!.trim();@@ -941,7 +941,7 @@ function modelGatewayFromEnv(env: NodeJS.ProcessEnv): ModelGatewayTransportConfithrow new Error("MODEL_GATEWAY_API_KEY_HEADER must be a valid HTTP header name");}const models: Record<string, string> = {};- for (const mapping of env.MODEL_GATEWAY_MODELS!.split(",")) {+ for (const mapping of env.MODEL_GATEWAY_MODELS?.trim() ? env.MODEL_GATEWAY_MODELS.split(",") : []) {const separator = mapping.indexOf("=");const source = mapping.slice(0, separator).trim();const target = mapping.slice(separator + 1).trim();diff --git a/src/harness/pi-harness.ts b/src/harness/pi-harness.tsindex 696f9ad1c..f511cc6d0 100644--- a/src/harness/pi-harness.ts+++ b/src/harness/pi-harness.ts@@ -1,3 +1,4 @@+import { gatewayModelsJson, gatewayModelsVersion } from "../model/gateway-models.ts";import { Type } from "typebox";import { mkdirSync, mkdtempSync, rmSync, statSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";@@ -387,6 +388,7 @@ async function directAnthropicJson(.includes("anthropic"))return undefined;+ await modelGateway?.refresh?.();const gateway = modelGatewayRequest(modelGateway, model);const requestModel = gateway?.model ?? model;const requestKey = gateway?.apiKey ?? apiKey;@@ -1148,9 +1150,10 @@ export interface ProviderKeys {// version instead of leaking a temp dir per turn.let cachedCustomModels: { version: number; path: string | null } | null = null;function customModelsPath(): string | null {- const version = customProvidersVersion();+ const version = customProvidersVersion() + gatewayModelsVersion();if (cachedCustomModels?.version === version) return cachedCustomModels.path;- const custom = customModelsJson();+ const providers = { ...customModelsJson()?.providers, ...gatewayModelsJson() };+ const custom = Object.keys(providers).length ? { providers } : undefined;let path: string | null = null;if (custom) {path = join(mkdtempSync(join(tmpdir(), "pi-custom-models-")), "models.json");@@ -1165,6 +1168,7 @@ async function buildModelRuntime(modelGateway?: ModelGatewayTransportConfig,cacheRetention?: "long",): Promise<ModelRuntime> {+ await modelGateway?.refresh?.();const k: ProviderKeys = typeof keys === "string" ? { anthropic: keys } : keys;// Custom providers must exist in the runtime's own registry — a runtime// API key alone is invisible to its availability checks. models.json is@@ -1221,7 +1225,10 @@ async function buildModelRuntime(if (!body || typeof body !== "object" || Array.isArray(body)) {throw new Error("model gateway request payload must be an object");}- return { ...body, model: request.target };+ await modelGateway?.refresh?.();+ const current = modelGatewayRequest(modelGateway, model);+ if (!current) throw new Error(`Gateway model is unavailable: ${model.id}`);+ return { ...body, model: current.target };},} as unknown as ModelsApiStreamOptions<TApi>;return stream(request.model, context, routedOptions);@@ -1243,7 +1250,10 @@ async function buildModelRuntime(if (!body || typeof body !== "object" || Array.isArray(body)) {throw new Error("model gateway request payload must be an object");}- return { ...body, model: request.target };+ await modelGateway?.refresh?.();+ const current = modelGatewayRequest(modelGateway, model);+ if (!current) throw new Error(`Gateway model is unavailable: ${model.id}`);+ return { ...body, model: current.target };},} as ModelsSimpleStreamOptions;return streamSimple(request.model, context, routedOptions);diff --git a/src/model/gateway-catalog.ts b/src/model/gateway-catalog.tsnew file mode 100644index 000000000..ebd0e9a3f--- /dev/null+++ b/src/model/gateway-catalog.ts@@ -0,0 +1,178 @@+import type { Api, Model } from "@earendil-works/pi-ai";+import type { ModelGatewayTransportConfig } from "./provider-endpoints.ts";+import { GATEWAY_MODEL_PREFIX, GATEWAY_PROVIDER, setGatewayModels } from "./gateway-models.ts";++const MAX_BYTES = 2 * 1024 * 1024;+const MAX_MODELS = 1_000;+const TTL_MS = 5 * 60_000;+const RETRY_MS = 30_000;++function record(value: unknown): Record<string, unknown> | undefined {+ return value !== null && typeof value === "object" && !Array.isArray(value)+ ? (value as Record<string, unknown>)+ : undefined;+}++function modelId(value: unknown): value is string {+ return typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9._:/+-]{0,191}$/.test(value);+}++function tokens(value: unknown): number | undefined {+ return typeof value === "number" && Number.isSafeInteger(value) && value > 0 ? value : undefined;+}++function price(value: unknown): number | undefined {+ return typeof value === "number" && Number.isFinite(value) && value >= 0 && Number.isFinite(value * 1_000_000)+ ? value * 1_000_000+ : undefined;+}++function parseModel(value: unknown, allowed: Set<string>, baseUrl: string): Model<Api> | undefined {+ const info = record(value);+ if (!info || !modelId(info.model_group) || !allowed.has(info.model_group)) return undefined;+ if (info.mode !== "chat" || info.supports_function_calling !== true) return undefined;+ const contextWindow = tokens(info.max_input_tokens);+ const maxTokens = tokens(info.max_output_tokens);+ const input = price(info.input_cost_per_token);+ const output = price(info.output_cost_per_token);+ if (!contextWindow || !maxTokens || maxTokens >= contextWindow || input === undefined || output === undefined)+ return undefined;+ const providers = Array.isArray(info.providers) ? info.providers : [];+ const anthropic = providers.length > 0 && providers.every((provider) => provider === "anthropic");+ const openai = providers.length > 0 && providers.every((provider) => provider === "openai" || provider === "azure");+ return {+ id: GATEWAY_MODEL_PREFIX + info.model_group,+ name: info.model_group,+ provider: GATEWAY_PROVIDER,+ api: anthropic ? "anthropic-messages" : "openai-completions",+ baseUrl: anthropic ? baseUrl.replace(/\/v1$/, "") : baseUrl,+ reasoning: (anthropic || openai) && info.supports_reasoning === true,+ input: info.supports_vision === true ? ["text", "image"] : ["text"],+ contextWindow,+ maxTokens,+ cost: {+ input,+ output,+ cacheRead: price(info.cache_read_input_token_cost) ?? input,+ cacheWrite: price(info.cache_creation_input_token_cost) ?? input,+ },+ compat: {+ supportsDeveloperRole: false,+ supportsStore: false,+ supportsStrictMode: false,+ supportsStrictTools: false,+ forceAdaptiveThinking: anthropic && info.supports_adaptive_thinking === true,+ supportsLongCacheRetention: false,+ supportsReasoningEffort:+ Array.isArray(info.supported_openai_params) && info.supported_openai_params.includes("reasoning_effort"),+ supportsUsageInStreaming: true,+ maxTokensField:+ Array.isArray(info.supported_openai_params) && info.supported_openai_params.includes("max_completion_tokens")+ ? "max_completion_tokens"+ : "max_tokens",+ },+ };+}++async function readCatalog(+ url: string,+ config: ModelGatewayTransportConfig,+ fetcher: typeof fetch,+): Promise<unknown[]> {+ const response = await fetcher(url, {+ headers: { [config.apiKeyHeader]: config.apiKey },+ signal: AbortSignal.timeout(5_000),+ redirect: "error",+ });+ if (!response.ok) throw new Error(`Gateway catalog returned HTTP ${response.status}`);+ if (!response.body || Number(response.headers.get("content-length")) > MAX_BYTES)+ throw new Error("Gateway catalog exceeds size limit");+ const reader = response.body.getReader();+ const decoder = new TextDecoder();+ let bytes = 0;+ let body = "";+ try {+ while (true) {+ const chunk = await reader.read();+ if (chunk.done) break;+ bytes += chunk.value.byteLength;+ if (bytes > MAX_BYTES) throw new Error("Gateway catalog exceeds size limit");+ body += decoder.decode(chunk.value, { stream: true });+ }+ body += decoder.decode();+ } finally {+ void reader.cancel().catch(() => {});+ reader.releaseLock();+ }+ const data = record(JSON.parse(body))?.data;+ if (!Array.isArray(data) || data.length > MAX_MODELS) throw new Error("Invalid gateway catalog");+ return data;+}++export function createGatewayCatalog(+ config: ModelGatewayTransportConfig,+ fetcher: typeof fetch = fetch,+ now: () => number = Date.now,+) {+ const root = config.url.replace(/\/v1$/, "");+ const baseUrl = `${root}/v1`;+ let routes: Readonly<Record<string, string>> = { ...config.models };+ let discovered = false;+ let expiresAt = 0;+ let retryAt = 0;+ let inFlight: Promise<void> | undefined;+ const transport: ModelGatewayTransportConfig = {+ ...config,+ reservedModelIds: new Set(Object.keys(config.models)),+ refresh,+ get models() {+ return routes;+ },+ };+ async function refresh(): Promise<void> {+ if (inFlight) return inFlight;+ if (now() < retryAt) return;+ inFlight = (async () => {+ try {+ const [listing, metadata] = await Promise.all([+ readCatalog(`${baseUrl}/models`, config, fetcher),+ readCatalog(`${root}/model_group/info`, config, fetcher),+ ]);+ const allowed = new Set(+ listing.flatMap((entry) => {+ const id = record(entry)?.id;+ if (!modelId(id)) throw new Error("Invalid gateway model id");+ return [id];+ }),+ );+ const models = metadata.flatMap((entry) => {+ const model = parseModel(entry, allowed, baseUrl);+ return model ? [model] : [];+ });+ const next: Record<string, string> = Object.fromEntries(+ models.map((model) => [model.id, model.id.slice(GATEWAY_MODEL_PREFIX.length)]),+ );+ for (const [id, target] of Object.entries(config.models)) {+ if (allowed.has(target)) next[id] = target;+ }+ routes = next;+ discovered = true;+ expiresAt = now() + TTL_MS;+ retryAt = expiresAt;+ setGatewayModels(models);+ } catch {+ retryAt = now() + RETRY_MS;+ if (discovered || Object.keys(config.models).length === 0) {+ routes = {};+ expiresAt = 0;+ setGatewayModels([]);+ console.warn("[model] Gateway model discovery unavailable; retrying in 30 seconds");+ }+ }+ })().finally(() => {+ inFlight = undefined;+ });+ return inFlight;+ }+ return { transport, refresh };+}diff --git a/src/model/gateway-models.ts b/src/model/gateway-models.tsnew file mode 100644index 000000000..e0aef4eb1--- /dev/null+++ b/src/model/gateway-models.ts@@ -0,0 +1,58 @@+import type { Api, Model } from "@earendil-works/pi-ai";++export const GATEWAY_PROVIDER = "qm:gateway";+export const GATEWAY_MODEL_PREFIX = "gateway/";++let models = new Map<string, Model<Api>>();+let version = 0;+let snapshot = "[]";++export function isGatewayModelId(id: string): boolean {+ return id.startsWith(GATEWAY_MODEL_PREFIX);+}++export function gatewayModelsVersion(): number {+ return version;+}++export function setGatewayModels(next: readonly Model<Api>[]): void {+ const serialized = JSON.stringify(next);+ if (serialized === snapshot) return;+ models = new Map(next.map((model) => [model.id, model]));+ snapshot = serialized;+ version += 1;+}++export function resolveGatewayModel(id: string): Model<Api> | undefined {+ return models.get(id);+}++export function gatewayModelCatalog(): Array<{ id: string; name: string; provider: string }> {+ return [...models.values()].map(({ id, name, provider }) => ({ id, name, provider }));+}++export function gatewayModelsJson(): Record<string, unknown> {+ const first = models.values().next().value;+ if (!first) return {};+ return {+ [GATEWAY_PROVIDER]: {+ name: "Model gateway",+ baseUrl: first.baseUrl,+ api: "openai-completions",+ models: [...models.values()].map(+ ({ id, name, contextWindow, maxTokens, cost, reasoning, input, compat, api, baseUrl }) => ({+ id,+ name,+ contextWindow,+ maxTokens,+ cost,+ reasoning,+ input,+ compat,+ api,+ baseUrl,+ }),+ ),+ },+ };+}diff --git a/src/model/model-catalog.ts b/src/model/model-catalog.tsindex 9d0c68132..dde58971f 100644--- a/src/model/model-catalog.ts+++ b/src/model/model-catalog.ts@@ -1,3 +1,4 @@+import { gatewayModelCatalog } from "./gateway-models.ts";import {modelSupportedByHarness,registerOpenRouterCatalogModel,@@ -42,7 +43,12 @@ export function builtInModelCatalog(): ModelCatalogEntry[] {: [];});const known = new Set(builtIns.map((model) => model.id));- return [...builtIns, ...[...overlayModelCatalog(), ...customModelCatalog()].filter((model) => !known.has(model.id))];+ return [+ ...builtIns,+ ...[...overlayModelCatalog(), ...customModelCatalog(), ...gatewayModelCatalog()].filter(+ (model) => !known.has(model.id),+ ),+ ];}async function boundedJson(response: Response): Promise<unknown> {diff --git a/src/model/pi-models.ts b/src/model/pi-models.tsindex 23b10dc64..26332f1d3 100644--- a/src/model/pi-models.ts+++ b/src/model/pi-models.ts@@ -1,3 +1,4 @@+import { gatewayModelCatalog, gatewayModelsVersion, isGatewayModelId, resolveGatewayModel } from "./gateway-models.ts";import { getBuiltinModel } from "@earendil-works/pi-ai/providers/all";import type { Api, Model } from "@earendil-works/pi-ai";import { parseModelOverlay, type ModelOverlay } from "./model-overlay.ts";@@ -186,7 +187,7 @@ let overlayVersion = 0;let overlaySnapshot = JSON.stringify([[], [], []]);export function modelOverlayVersion(): number {- return overlayVersion;+ return overlayVersion + gatewayModelsVersion();}export function isOverlayModel(id: string): boolean {@@ -198,11 +199,14 @@ export function modelOfferedInWebui(id: string): boolean {}export function modelUnavailableReason(id: string): string | undefined {+ if (isGatewayModelId(id) && !resolveGatewayModel(id))+ return "Gateway model is unavailable; select another model or retry after discovery recovers";return unavailableOverlays.get(id);}export function modelIdReserved(id: string): boolean {return (+ isGatewayModelId(id) ||id.startsWith(CODEX_SUBSCRIPTION_PREFIX) ||REGISTRY_BY_ID.has(id) ||Boolean(builtinModel(id)) ||@@ -270,6 +274,7 @@ export function setModelOverlays(export function selectableBaseModels(): ReadonlyArray<{ id: string; name: string }> {return [+ ...gatewayModelCatalog(),...SELECTABLE_BASE_MODELS.filter(({ id }) => resolveModel(id)),...[...overlays.values()].filter((m) => m.base).map(({ id, name }) => ({ id, name })),];@@ -282,7 +287,11 @@ export function overlayModelCatalog(): Array<{ id: string; name: string; provide}export function defaultWebuiModelIds(): readonly string[] {- return [...DEFAULT_WEBUI_MODEL_IDS, ...[...overlays.values()].filter((m) => m.webui).map((m) => m.id)];+ return [+ ...DEFAULT_WEBUI_MODEL_IDS,+ ...gatewayModelCatalog().map((m) => m.id),+ ...[...overlays.values()].filter((m) => m.webui).map((m) => m.id),+ ];}export function fastModeModelIds(): readonly string[] {@@ -388,6 +397,7 @@ export function resolveBuiltinModel(id: string): PiModel | undefined {}function resolveBaseModel(id: string): PiModel | undefined {+ if (isGatewayModelId(id)) return resolveGatewayModel(id);if (unavailableOverlays.has(id)) return undefined;const builtin = resolveBuiltinModel(id);if (builtin) return builtin;@@ -441,6 +451,7 @@ export function contextTokenBudgetForModel(id: string): number | undefined {}export function modelSupportedByHarness(id: string | undefined, harness: string): boolean {+ if (id && isGatewayModelId(id)) return (harness === "pi" || harness === "mock") && Boolean(resolveGatewayModel(id));if (!id || unavailableOverlays.has(id)) return false;if (overlays.has(id)) return harness === "pi" || harness === "mock";if (isCustomModelId(id) && !REGISTRY_BY_ID.has(id))@@ -480,6 +491,7 @@ function providerFlags(value: ModelProviderAvailability): ModelProviderAvailabil}export function modelServiceable(id: string, providers: ModelProviderAvailability): boolean {+ if (isGatewayModelId(id)) return Boolean(resolveGatewayModel(id) && providers.modelIds?.has(id));const provider = resolveModel(id)?.provider;if (!provider) return false;if (isCustomModelId(id) && !REGISTRY_BY_ID.has(id)) return true;diff --git a/src/model/provider-endpoints.ts b/src/model/provider-endpoints.tsindex 0187ed6d8..f13064776 100644--- a/src/model/provider-endpoints.ts+++ b/src/model/provider-endpoints.ts@@ -1,3 +1,4 @@+import { isGatewayModelId } from "./gateway-models.ts";/*** Provider endpoint overrides.*@@ -25,6 +26,8 @@ export interface ModelGatewayTransportConfig {apiKey: string;apiKeyHeader: string;models: Readonly<Record<string, string>>;+ reservedModelIds?: ReadonlySet<string>;+ refresh?: () => Promise<void>;}/**@@ -71,14 +74,23 @@ export function providerBaseUrl(provider: string): string | undefined {return (PROVIDER_IDS as readonly string[]).includes(provider) ? configured[provider as ProviderId] : undefined;}-export function modelGatewayRequest<T extends { id: string; baseUrl: string }>(+export function modelGatewayRequest<T extends { id: string; baseUrl: string; api?: string }>(config: ModelGatewayTransportConfig | undefined,model: T,): { model: T; target: string; apiKey: string; headers: Record<string, string> } | undefined {const target = config?.models[model.id];- if (!target || !config) return undefined;+ if (!target || !config) {+ if (isGatewayModelId(model.id) || config?.reservedModelIds?.has(model.id))+ throw new Error(`Gateway model is unavailable: ${model.id}`);+ return undefined;+ }return {- model: { ...model, baseUrl: config.url },+ model: {+ ...model,+ baseUrl: isGatewayModelId(model.id)+ ? `${config.url.replace(/\/v1$/, "")}${model.api === "anthropic-messages" ? "" : "/v1"}`+ : config.url,+ },target,apiKey: config.apiKey,headers: { [config.apiKeyHeader]: config.apiKey },diff --git a/src/wiring.ts b/src/wiring.tsindex ff7a539d5..ba41ac325 100644--- a/src/wiring.ts+++ b/src/wiring.ts@@ -1,3 +1,4 @@+import { createGatewayCatalog } from "./model/gateway-catalog.ts";import { createRuntimeService } from "./harness/runtime-control.ts";import { createPostgresBrokerSessions, type BrokerSessionStore } from "./auth/broker-sessions.ts";import { createDirectFileUploads, type DirectFileUploads } from "./files/direct-file-upload.ts";@@ -535,7 +536,10 @@ export function buildApp(if (unknownGatewayModels.length) {throw new Error(`MODEL_GATEWAY_MODELS contains unsupported models: ${unknownGatewayModels.join(", ")}`);}- const gatewayModels = config.modelGateway?.models ?? {};+ const gatewayCatalog = config.modelGateway+ ? createGatewayCatalog(config.modelGateway, overrides.modelCredentialFetch)+ : undefined;+ const gatewayTransport = gatewayCatalog?.transport;const directProviderAvailability = providerKeysPresent(config);const directModelCredentials = createModelCredentialStore({backing: artifactMap("model_credentials"),@@ -549,10 +553,11 @@ export function buildApp(const modelCredentials: ModelCredentialStore = {...directModelCredentials,async availability() {+ await gatewayCatalog?.refresh();const direct = await directModelCredentials.availability();return {...direct,- modelIds: new Set(Object.keys(gatewayModels)),+ modelIds: new Set(Object.keys(gatewayTransport?.models ?? {})),};},};@@ -1096,7 +1101,7 @@ export function buildApp(const modelVerifier = createModelVerifier({credentials: modelCredentials,keyMaterial: config.connectorSecretKey ?? randomBytes(32),- modelGateway: config.modelGateway,+ modelGateway: gatewayTransport,probe: overrides.modelVerificationProbe,});const modelRegistry = createModelOverlayStore(artifactMap("model_registry"), writeModelRegistry, modelVerifier);@@ -1104,6 +1109,7 @@ export function buildApp(setCustomProviders(await customProviders.enabled());};const refreshModels = async () => {+ await gatewayCatalog?.refresh();await refreshCustomProviders();await modelRegistry.refresh();};@@ -1141,12 +1147,22 @@ export function buildApp(const runtimeOrgScope = scopeId("org", config.orgId);const orgBaseModelId = (): string | undefined =>configStore.getRuntimeSelection(runtimeOrgScope)?.modelId ?? configStore.getBaseModel(runtimeOrgScope) ?? undefined;+ const defaultForHarness = (harness: string) =>+ defaultModelForHarness(+ harness,+ configuredModelForHarness(config, harness),+ baseModelProviders(config) ??+ (harness === "pi" && gatewayTransport+ ? { ...directProviderAvailability, modelIds: new Set(Object.keys(gatewayTransport.models)) }+ : undefined),+ );const adapters = new Map<HarnessId, Harness>([["pi",createPiHarness({...piHarnessConfigOptions(config),- resolveBaseModelId: orgBaseModelId,+ modelGateway: gatewayTransport,+ resolveBaseModelId: () => orgBaseModelId() ?? defaultForHarness("pi"),resolveProviderKeys: resolveModelProviderKeys,signals: runSignals,mcpTools,@@ -1217,11 +1233,7 @@ export function buildApp(const fallback = {harnessId: fallbackHarness,get modelId() {- return defaultModelForHarness(- fallbackHarness,- configuredModelForHarness(config, fallbackHarness),- baseModelProviders(config),- );+ return defaultForHarness(fallbackHarness);},};const judgeModelId = (): string => config.judgeModelId ?? auxiliaryModelFor(orgBaseModelId() ?? fallback.modelId);@@ -1937,7 +1949,9 @@ export function buildApp({config: configStore,harnessId: fallbackHarness,- baseModelDefault: fallback.modelId,+ get baseModelDefault() {+ return fallback.modelId;+ },providerKeys: providerKeysPresent(config),modelCredentials,modelCredentialFetch: overrides.modelCredentialFetch,diff --git a/test/gateway-catalog.test.ts b/test/gateway-catalog.test.tsnew file mode 100644index 000000000..3e7f9f594--- /dev/null+++ b/test/gateway-catalog.test.ts@@ -0,0 +1,231 @@+import { validateWebTurnModelOptions } from "../src/core/turn-options.ts";+import { createMemoryConfigStore } from "../src/resolution/config-store.ts";+import { resolveRuntimeChoice } from "../src/harness/harness-router.ts";+import { afterEach, test } from "node:test";+import assert from "node:assert/strict";+import { createGatewayCatalog } from "../src/model/gateway-catalog.ts";+import { setGatewayModels } from "../src/model/gateway-models.ts";+import { modelGatewayRequest } from "../src/model/provider-endpoints.ts";+import {+ modelIdReserved,+ modelUnavailableReason,+ modelServiceable,+ modelSupportedByHarness,+ resolveModel,+ safeModelMetadata,+} from "../src/model/pi-models.ts";+import { builtInModelCatalog } from "../src/model/model-catalog.ts";+import { validateCustomProviderSpec } from "../src/model/custom-providers.ts";+import { loadConfig } from "../src/config.ts";++const config = {+ url: "https://gateway.example/prefix/v1",+ apiKey: "private-key",+ apiKeyHeader: "x-gateway-key",+ models: {},+};+const group = (id = "vendor/new-model", extra: object = {}) => ({+ model_group: id,+ mode: "chat",+ supports_function_calling: true,+ max_input_tokens: 128000,+ max_output_tokens: 8192,+ input_cost_per_token: 0.000002,+ output_cost_per_token: 0.000008,+ supports_vision: true,+ supported_openai_params: ["tools", "max_completion_tokens"],+ ...extra,+});+function fixture(models = [group()], aliases = {}) {+ let clock = 0;+ let status = 200;+ let metadata = models;+ let listing = models.map((m) => ({ id: m.model_group }));+ const calls: Array<{ url: string; options?: RequestInit }> = [];+ const fetcher: typeof fetch = async (url, options) => {+ calls.push({ url: String(url), options });+ return Response.json({ data: String(url).endsWith("/models") ? listing : metadata }, { status });+ };+ const catalog = createGatewayCatalog({ ...config, models: aliases }, fetcher, () => clock);+ return {+ catalog,+ calls,+ tick: () => {+ clock += 300001;+ },+ status: (next: number) => {+ status = next;+ },+ metadata: (next: typeof models) => {+ metadata = next;+ },+ listing: (next: string[]) => {+ listing = next.map((id) => ({ id }));+ },+ };+}+afterEach(() => setGatewayModels([]));++test("gateway configuration no longer requires a model list", () => {+ const parsed = loadConfig({+ HARNESS: "mock",+ MODEL_GATEWAY_URL: config.url,+ MODEL_GATEWAY_API_KEY: config.apiKey,+ MODEL_GATEWAY_API_KEY_HEADER: config.apiKeyHeader,+ });+ assert.deepEqual(parsed.modelGateway?.models, {});+});++test("discovers unknown tool models through the company key without leaking transport metadata", async () => {+ const f = fixture([group("vendor/new-model", { api_base: "https://evil.example", api_key: "metadata-secret" })]);+ await Promise.all([f.catalog.refresh(), f.catalog.refresh()]);+ assert.equal(f.calls.length, 2);+ assert.deepEqual(+ f.calls.map((c) => c.url),+ ["https://gateway.example/prefix/v1/models", "https://gateway.example/prefix/model_group/info"],+ );+ for (const call of f.calls) {+ assert.deepEqual(call.options?.headers, { "x-gateway-key": "private-key" });+ assert.equal(call.options?.redirect, "error");+ }+ await f.catalog.refresh();+ assert.equal(f.calls.length, 2);+ const id = "gateway/vendor/new-model";+ const model = resolveModel(id)!;+ assert.equal(model.api, "openai-completions");+ assert.equal(model.baseUrl, config.url);+ assert.deepEqual(model.cost, { input: 2, output: 8, cacheRead: 2, cacheWrite: 2 });+ assert.equal(modelGatewayRequest(f.catalog.transport, model)?.target, "vendor/new-model");+ assert.ok(builtInModelCatalog().some((m) => m.id === id));+ const availability = { anthropic: false, openai: false, openrouter: false, modelIds: new Set([id]) };+ assert.equal(validateWebTurnModelOptions({ model: id }, null, availability), null);+ assert.match(validateWebTurnModelOptions({ model: id }, [], availability) ?? "", /not enabled/);++ assert.equal(modelSupportedByHarness(id, "pi"), true);+ for (const harness of ["opencode", "claude", "codex"]) assert.equal(modelSupportedByHarness(id, harness), false);+ assert.equal(modelServiceable(id, { anthropic: true, openai: true, openrouter: true }), false);+ assert.equal(+ modelServiceable(id, { anthropic: false, openai: false, openrouter: false, modelIds: new Set([id]) }),+ true,+ );+ assert.doesNotMatch(+ JSON.stringify(safeModelMetadata(id)),+ /private-key|metadata-secret|evil.example|gateway.example/,+ );+});++test("only models present in both responses with valid chat capabilities are selectable", async () => {+ const f = fixture([+ group(),+ group("embed", { mode: "embedding" }),+ group("no-tools", { supports_function_calling: false }),+ group("no-context", { max_input_tokens: null }),+ group("bad-price", { input_cost_per_token: -1 }),+ group("bad-output", { max_output_tokens: 200000 }),+ group("not-authorized"),+ ]);+ f.listing(["vendor/new-model", "embed", "no-tools", "no-context", "bad-price", "bad-output"]);+ await f.catalog.refresh();+ assert.deepEqual(Object.keys(f.catalog.transport.models), ["gateway/vendor/new-model"]);+});++test("successful revocation clears legacy aliases and cached models and cannot be undone by a failure", async () => {+ const f = fixture([group()], { "claude-opus-5": "vendor/new-model" });+ await f.catalog.refresh();+ const stale = resolveModel("gateway/vendor/new-model")!;+ const native = resolveModel("claude-opus-5")!;+ assert.ok(f.catalog.transport.models["claude-opus-5"]);+ f.tick();+ f.listing([]);+ await f.catalog.refresh();+ assert.deepEqual(f.catalog.transport.models, {});+ assert.equal(resolveModel(stale.id), undefined);+ assert.match(modelUnavailableReason(stale.id) ?? "", /unavailable/);+ assert.throws(() => modelGatewayRequest(f.catalog.transport, stale), /unavailable/);+ assert.throws(() => modelGatewayRequest(f.catalog.transport, native), /unavailable/);+ f.tick();+ f.status(503);+ await f.catalog.refresh();+ assert.deepEqual(f.catalog.transport.models, {});+ assert.throws(() => modelGatewayRequest(undefined, stale), /unavailable/);+});++test("legacy routes work without discovery support, then discovery adds models and governs access", async () => {+ const f = fixture([group()], { "claude-opus-5": "vendor/new-model" });+ f.status(404);+ await f.catalog.refresh();+ assert.equal(f.catalog.transport.models["claude-opus-5"], "vendor/new-model");+ f.tick();+ f.status(200);+ await f.catalog.refresh();+ assert.ok(f.catalog.transport.models["gateway/vendor/new-model"]);+ f.tick();+ f.status(401);+ await f.catalog.refresh();+ assert.deepEqual(f.catalog.transport.models, {});+});++test("discovery outage hides stale models; later successful refresh recovers", async () => {+ const f = fixture();+ await f.catalog.refresh();+ f.tick();+ f.status(503);+ await f.catalog.refresh();+ assert.equal(resolveModel("gateway/vendor/new-model"), undefined);+ f.tick();+ f.status(200);+ f.metadata([group("new")]);+ f.listing(["new"]);+ await f.catalog.refresh();+ assert.ok(resolveModel("gateway/new"));+});++test("malformed and oversized catalogs cannot grant access", async () => {+ for (const response of [+ Response.json({ nope: [] }),+ Response.json({ data: Array(1001).fill({ id: "m" }) }),+ new Response("x".repeat(2 * 1024 * 1024 + 1)),+ new Response("{}", { headers: { "content-length": "99999999" } }),+ ]) {+ const catalog = createGatewayCatalog(config, async () => response.clone());+ await catalog.refresh();+ assert.deepEqual(catalog.transport.models, {});+ }+});++test("gateway namespace cannot be registered by custom providers even before discovery", () => {+ assert.doesNotThrow(() =>+ validateCustomProviderSpec({+ id: "gateway",+ name: "Existing gateway",+ protocol: "openai",+ baseUrl: config.url,+ models: [{ id: "old-model" }],+ }),+ );+ assert.equal(modelIdReserved("gateway/future"), true);+ const spec = {+ id: "custom",+ name: "Custom",+ protocol: "openai" as const,+ baseUrl: config.url,+ models: [{ id: "gateway/future" }],+ };+ assert.throws(() => validateCustomProviderSpec(spec), /already registered/);+ assert.throws(+ () => validateCustomProviderSpec({ ...spec, id: "qm:gateway", models: [{ id: "other" }] }),+ /must match/,+ );+});++test("removed gateway selections never fall back to a different org or direct model", () => {+ const store = createMemoryConfigStore("default-org");+ const org = "org:default-org" as const;+ const personal = "personal:alice" as const;+ const fallback = { harnessId: "pi" as const, modelId: "claude-opus-5" };+ store.setRuntimeSelection(org, { harnessId: "pi", modelId: "gateway/removed" });+ assert.throws(() => resolveRuntimeChoice(store, org, personal, fallback), /Gateway model is unavailable/);+ store.setRuntimeSelection(org, fallback);+ store.setRuntimeSelection(personal, { harnessId: "pi", modelId: "gateway/removed" });+ assert.throws(() => resolveRuntimeChoice(store, org, personal, fallback), /Gateway model is unavailable/);+});diff --git a/test/gateway-discovery-e2e.test.ts b/test/gateway-discovery-e2e.test.tsnew file mode 100644index 000000000..24945611d--- /dev/null+++ b/test/gateway-discovery-e2e.test.ts@@ -0,0 +1,260 @@+import "./support/auto-fake-sprites.ts";+import assert from "node:assert/strict";+import { createServer } from "node:http";+import type { AddressInfo } from "node:net";+import { test } from "node:test";+import { createGatewayCatalog } from "../src/model/gateway-catalog.ts";+import { setGatewayModels } from "../src/model/gateway-models.ts";+import { resolveModel } from "../src/model/pi-models.ts";+import { oneShot } from "../src/harness/pi-harness.ts";+import { buildApp } from "../src/wiring.ts";+import { testConfig } from "./support/test-config.ts";+import { runtimeConfigBody } from "../src/api/runtime-config.ts";+import { scopeId } from "../src/types.ts";++const metadata = {+ model_group: "acme/future",+ mode: "chat",+ supports_function_calling: true,+ max_input_tokens: 64000,+ max_output_tokens: 4096,+ input_cost_per_token: 0.000001,+ output_cost_per_token: 0.000002,+};++for (const protocol of ["openai", "anthropic"])+ test(`live ${protocol} gateway discovery, picker and tool continuation`, async (t) => {+ let useTool = false;+ let clockOffset = 0;+ const requests: Array<{ path: string; body: Record<string, unknown>; key: unknown }> = [];+ const upstream = createServer(async (req, res) => {+ const chunks = [];+ for await (const chunk of req) chunks.push(chunk);+ const raw = Buffer.concat(chunks).toString();+ const body = raw ? JSON.parse(raw) : {};+ requests.push({ path: req.url!, body, key: req.headers["x-gateway-key"] });+ if (req.headers["x-gateway-key"] !== "company-key") {+ res.writeHead(401);+ return res.end();+ }+ if (req.url === "/v1/models" || req.url === "/model_group/info") {+ res.writeHead(200, { "content-type": "application/json" });+ return res.end(+ JSON.stringify({+ data:+ req.url === "/v1/models"+ ? [{ id: "acme/future" }]+ : [+ {+ ...metadata,+ providers: [protocol],+ supports_reasoning: protocol === "anthropic",+ supports_adaptive_thinking: protocol === "anthropic",+ },+ ],+ }),+ );+ }+ assert.equal(req.url, protocol === "anthropic" ? "/v1/messages" : "/v1/chat/completions");+ assert.equal(body.model, "acme/future");+ assert.equal(body.prompt_cache_retention, undefined);+ assert.equal(body.store, undefined);+ res.writeHead(200, { "content-type": "text/event-stream" });+ if (protocol === "anthropic") {+ if (body.thinking) assert.equal(body.thinking.type, "adaptive");+ const toolResult = body.messages.some(+ (m: { content: unknown }) =>+ Array.isArray(m.content) && m.content.some((c: { type: string }) => c.type === "tool_result"),+ );+ if (toolResult) {+ assert.ok(+ body.messages.some(+ (m: { content: unknown }) =>+ Array.isArray(m.content) &&+ m.content.some(+ (c: { type: string; signature?: string }) =>+ c.type === "thinking" && c.signature === "signed-thinking",+ ),+ ),+ );+ }+ const send = (type: string, data: object) =>+ res.write(`event: ${type}\ndata: ${JSON.stringify({ type, ...data })}\n\n`);+ send("message_start", {+ message: {+ id: "msg_test",+ type: "message",+ role: "assistant",+ content: [],+ model: body.model,+ stop_reason: null,+ usage: { input_tokens: 10, output_tokens: 0 },+ },+ });+ send("content_block_start", { index: 0, content_block: { type: "thinking", thinking: "" } });+ send("content_block_delta", { index: 0, delta: { type: "thinking_delta", thinking: "Check the tool." } });+ send("content_block_delta", { index: 0, delta: { type: "signature_delta", signature: "signed-thinking" } });+ send("content_block_stop", { index: 0 });+ if (useTool && !toolResult) {+ clockOffset += 300001;+ send("content_block_start", {+ index: 1,+ content_block: { type: "tool_use", id: "call_discovery", name: "execute", input: {} },+ });+ send("content_block_delta", {+ index: 1,+ delta: { type: "input_json_delta", partial_json: JSON.stringify({ command: "echo discovery-tool-ok" }) },+ });+ } else {+ send("content_block_start", { index: 1, content_block: { type: "text", text: "" } });+ send("content_block_delta", { index: 1, delta: { type: "text_delta", text: "Discovered model replied" } });+ }+ send("content_block_stop", { index: 1 });+ send("message_delta", {+ delta: { stop_reason: useTool && !toolResult ? "tool_use" : "end_turn" },+ usage: { output_tokens: 20 },+ });+ send("message_stop", {});+ return res.end();+ }+ if (useTool && !body.messages.some((m: { role: string }) => m.role === "tool")) {+ clockOffset += 300001;+ const chunk = {+ id: "test",+ object: "chat.completion.chunk",+ model: body.model,+ choices: [+ {+ index: 0,+ delta: {+ role: "assistant",+ tool_calls: [+ {+ index: 0,+ id: "call_discovery",+ type: "function",+ function: { name: "execute", arguments: JSON.stringify({ command: "echo discovery-tool-ok" }) },+ },+ ],+ },+ finish_reason: null,+ },+ ],+ };+ res.write(`data: ${JSON.stringify(chunk)}\n\n`);+ res.write(+ `data: ${JSON.stringify({ ...chunk, choices: [{ index: 0, delta: {}, finish_reason: "tool_calls" }] })}\n\n`,+ );+ return res.end("data: [DONE]\n\n");+ }++ for (const [delta, finish] of [+ [{ role: "assistant", content: "Discovered model replied" }, null],+ [{}, "stop"],+ ]) {+ res.write(+ `data: ${JSON.stringify({+ id: "test",+ object: "chat.completion.chunk",+ model: body.model,+ choices: [{ index: 0, delta, finish_reason: finish }],+ usage: finish ? { prompt_tokens: 5, completion_tokens: 3, total_tokens: 8 } : undefined,+ })}\n\n`,+ );+ }+ res.end("data: [DONE]\n\n");+ });+ await new Promise<void>((resolve) => upstream.listen(0, "127.0.0.1", resolve));+ t.after(() => {+ upstream.closeAllConnections();+ upstream.close();+ setGatewayModels([]);+ });+ const config = {+ url: `http://127.0.0.1:${(upstream.address() as AddressInfo).port}`,+ apiKey: "company-key",+ apiKeyHeader: "x-gateway-key",+ models: {},+ };+ const built = buildApp(+ testConfig({+ modelGateway: config,+ harness: "pi",+ piSystemCacheSplit: true,+ modelId: "gateway/acme/future",+ }),+ );+ await built.refreshModels();+ const runtime = await runtimeConfigBody(+ {+ deps: {+ config: built.config,+ harnessId: "pi",+ baseModelDefault: "gateway/acme/future",+ modelCredentials: built.modelCredentials,+ refreshModels: built.refreshModels,+ },+ },+ scopeId("org", "default-org"),+ );+ assert.deepEqual(runtime.modelsByHarness.pi, ["gateway/acme/future"]);+ assert.equal(+ runtime.modelCatalog["gateway/acme/future"]?.api,+ protocol === "anthropic" ? "anthropic-messages" : "openai-completions",+ );+ assert.doesNotMatch(JSON.stringify(runtime), /company-key|127\.0\.0\.1/);+ const catalog = createGatewayCatalog(config, fetch, () => Date.now() + clockOffset);+ await catalog.refresh();+ const model = resolveModel("gateway/acme/future")!;+ const reply = await oneShot("gateway-e2e", model, {}, "Reply briefly", "Hello", {+ modelGateway: catalog.transport,+ });+ assert.equal(reply, "Discovered model replied");+ assert.ok(requests.some((r) => r.path === (protocol === "anthropic" ? "/v1/messages" : "/v1/chat/completions")));+ assert.ok(requests.every((r) => r.key === "company-key"));+ useTool = true;+ const beforeRefresh = requests.filter((r) => r.path === "/v1/models").length;+ assert.equal(+ await oneShot("gateway-ttl-e2e", model, {}, "Reply briefly", "Use a tool", { modelGateway: catalog.transport }),+ "Discovered model replied",+ );+ assert.ok(requests.filter((r) => r.path === "/v1/models").length > beforeRefresh);+ const result = await built.app.turn({+ surface: "test",+ actor: { externalId: "U1" },+ conversation: { kind: "dm", threadRef: "discovery-tool-loop" },+ text: "Use the execute tool, then reply.",+ });+ assert.equal(result.status, "ok");+ assert.match(result.reply ?? "", /Discovered model replied/);+ assert.ok(+ requests.some(+ (r) =>+ Array.isArray(r.body.messages) &&+ r.body.messages.some(+ (m: { role: string; content: unknown }) =>+ m.role === "tool" ||+ (Array.isArray(m.content) && m.content.some((c: { type: string }) => c.type === "tool_result")),+ ),+ ),+ );+ });++test("gateway outage leaves direct providers available", async () => {+ const built = buildApp(+ testConfig({+ anthropicApiKey: "independent-key",+ modelGateway: {+ url: "https://gateway.invalid",+ apiKey: "company-key",+ apiKeyHeader: "x-api-key",+ models: {},+ },+ }),+ { modelCredentialFetch: async () => new Response(null, { status: 503 }) },+ );+ const providers = await built.modelCredentials.availability();+ assert.equal(providers.anthropic, true);+ assert.deepEqual([...providers.modelIds!], []);+ await built.refreshModels();+});