Multiplayer agent harness for work.
feat: simplify admin model setup with metadata lookup
Josh Francec7b1f814b363
diff --git a/docs/admin-model-verification.md b/docs/admin-model-verification.mdindex 606771a6c..c73c75ab5 100644--- a/docs/admin-model-verification.md+++ b/docs/admin-model-verification.md@@ -17,3 +17,11 @@ Slow probes run outside the registry write lock. Before saving, the store rechec- A small probe does not certify prices, the maximum advertised context/output limits, image support, every reasoning level, or correct tool execution. Check provider documentation for those fields.- Configuring a model does not add support for a new protocol or native harness. Admin-managed API models remain Pi-only in normal use.- Live provider compatibility requires running the check against that provider. Development tests use local streaming provider fixtures.++## Find a model before entering metadata++Start with **Provider** and **Model ID**, then **Look up model**. Exact matches use the bundled, maintained model catalog, including models not currently offered in the picker. The result shows its source and catalog generation date when available. Existing models can be verified and added to the web picker without cloning them or changing the organization default. This uses existing built-in model policy; it does not create an admin-managed override or persistent overlay verification record.++For a new ID, lookup requests that exact record from the selected provider using organization credentials and configured endpoints. Redirects are refused. Lookup is bounded to five seconds and 64 KiB. Available name and token-limit fields are imported; missing prices, limits, and the compatible protocol template are requested explicitly. Provider APIs do not consistently publish pricing or complete metadata. Unknown values are never replaced with values from a similar model, and no third-party pricing source is queried.++The common path requires only the initial two inputs. New-model details that are already known remain under **Advanced overrides**, while missing fields are shown separately. All new or edited definitions still require **Verify and enable**. Metadata lookup is not a generation-access test and cannot certify a personal credential. Editing the provider or model ID clears a prior lookup result. The existing full-definition PUT remains supported.diff --git a/plugins/admin/public/index.html b/plugins/admin/public/index.htmlindex dee8ea46d..87463177e 100644--- a/plugins/admin/public/index.html+++ b/plugins/admin/public/index.html@@ -4986,8 +4986,8 @@<div class="head"><h2>Models on existing providers</h2><p>- Add an API model using an existing provider and a compatible builtin template. Endpoint and- credentials are inherited. Available through Pi; subscription support is not implied.+ Start with a provider and model ID. We look up exact catalog or provider metadata before asking for+ any missing details. Organization credentials are inherited; subscription support is not implied.</p></div><div class="body">@@ -5005,8 +5005,6 @@<p class="muted" id="model-registry-empty">No additional models.</p></div><div class="body setup-form">- <label>Model id <input type="text" id="model-registry-id" autocomplete="off" required /></label>- <label>Display name <input type="text" id="model-registry-name" autocomplete="off" required /></label><label>Provider<select id="model-registry-provider">@@ -5015,51 +5013,88 @@</select></label><label- >Compatible builtin template- <select id="model-registry-template"></select- ></label>- <p class="muted" id="model-registry-capabilities"></p>- <label- >Context window (tokens)- <input id="model-registry-contextWindow" type="number" min="2" step="1" required- /></label>- <label- >Maximum output (tokens) <input id="model-registry-maxTokens" type="number" min="1" step="1" required- /></label>- <label- >Input ($ / million tokens)- <input id="model-registry-input" type="number" min="0" step="any" required- /></label>- <label- >Output ($ / million tokens)- <input id="model-registry-output" type="number" min="0" step="any" required- /></label>- <label- >Cache read ($ / million tokens)- <input id="model-registry-cacheRead" type="number" min="0" step="any" required- /></label>- <label- >Cache write ($ / million tokens)- <input id="model-registry-cacheWrite" type="number" min="0" step="any" required+ >Model ID+ <input+ type="text"+ id="model-registry-id"+ autocomplete="off"+ placeholder="e.g. claude-opus-4-6"+ required/></label>- <label class="check"- ><input id="model-registry-base" type="checkbox" checked /> Offer in admin model choices</label- >- <label class="check"- ><input id="model-registry-webui" type="checkbox" checked /> Offer in web picker by default</label- >- <label class="check"- ><input id="model-registry-fastMode" type="checkbox" /> Model supports the Anthropic fast tier</label- >- <label class="check"- ><input id="model-registry-auxiliary" type="checkbox" /> Eligible for auxiliary tasks</label- >- <details>- <summary>Advanced pricing tiers</summary>- <label- >JSON array of inputTokensAbove, input, output, cacheRead and cacheWrite- <textarea id="model-registry-tiers" rows="4" placeholder="[]"></textarea>- </label>+ </div>+ <div class="foot"><button id="model-registry-lookup" class="primary">Look up model</button></div>+ <div class="body" id="model-registry-result" hidden>+ <p id="model-registry-source" class="muted"></p>+ <p id="model-registry-summary"></p>+ <p id="model-registry-lookup-note" class="muted"></p>+ <div id="model-registry-missing" hidden>+ <h3>Complete missing information</h3>+ <p class="muted">+ These fields were not available from the exact model record. Use the provider's documentation;+ choose a compatible template explicitly.+ </p>+ <div class="setup-form" id="model-registry-missing-fields"></div>+ </div>+ <details id="model-registry-advanced">+ <summary>Advanced overrides</summary>+ <p class="muted">+ Review imported values or override them. Pricing is per million tokens. Overrides require+ verification again.+ </p>+ <div class="setup-form" id="model-registry-advanced-fields">+ <label+ >Display name <input type="text" id="model-registry-name" autocomplete="off" required+ /></label>+ <label+ >Compatible builtin template+ <select id="model-registry-template"></select+ ></label>+ <p class="muted" id="model-registry-capabilities"></p>+ <label+ >Context window (tokens)+ <input id="model-registry-contextWindow" type="number" min="2" step="1" required+ /></label>+ <label+ >Maximum output (tokens)+ <input id="model-registry-maxTokens" type="number" min="1" step="1" required+ /></label>+ <label+ >Input ($ / million tokens)+ <input id="model-registry-input" type="number" min="0" step="any" required+ /></label>+ <label+ >Output ($ / million tokens)+ <input id="model-registry-output" type="number" min="0" step="any" required+ /></label>+ <label+ >Cache read ($ / million tokens)+ <input id="model-registry-cacheRead" type="number" min="0" step="any" required+ /></label>+ <label+ >Cache write ($ / million tokens)+ <input id="model-registry-cacheWrite" type="number" min="0" step="any" required+ /></label>+ <label class="check"+ ><input id="model-registry-base" type="checkbox" checked /> Offer in admin model choices</label+ >+ <label class="check"+ ><input id="model-registry-webui" type="checkbox" checked /> Offer in web picker by default</label+ >+ <label class="check"+ ><input id="model-registry-fastMode" type="checkbox" /> Model supports the Anthropic fast+ tier</label+ >+ <label class="check"+ ><input id="model-registry-auxiliary" type="checkbox" /> Eligible for auxiliary tasks</label+ >+ <details>+ <summary>Advanced pricing tiers</summary>+ <label+ >JSON array of inputTokensAbove, input, output, cacheRead and cacheWrite+ <textarea id="model-registry-tiers" rows="4" placeholder="[]"></textarea>+ </label>+ </details>+ </div></details></div><p class="muted" id="model-registry-verification-notice">@@ -5070,7 +5105,7 @@Existing saved settings remain unchanged. A failed recheck of those settings removes their verification.</p><div class="foot">- <button class="primary" id="model-registry-save">Verify and enable</button+ <button class="primary" id="model-registry-save" disabled>Verify and enable</button><button id="model-registry-new">New model</button><span class="status" id="st-model-registry"></span></div></section>@@ -8963,10 +8998,17 @@setStatus("st-onboarding-model", "Provider disabled.", "ok");};let modelRegistryTemplates = [];+ let modelRegistryLookup = null;+ let modelRegistryLookupRequest = 0;+ let modelRegistryBusy = false;function modelRegistryTemplateOptions() {const select = $("model-registry-template");const previous = select.value;select.replaceChildren();+ const blank = document.createElement("option");+ blank.value = "";+ blank.textContent = "Choose a compatible template";+ select.appendChild(blank);for (const template of modelRegistryTemplates.filter((m) => m.provider === $("model-registry-provider").value,)) {@@ -8975,7 +9017,7 @@option.textContent = template.name;select.appendChild(option);}- if ([...select.options].some((option) => option.value === previous)) select.value = previous;+ select.value = [...select.options].some((option) => option.value === previous) ? previous : "";modelRegistryCapabilities();}function modelRegistryCapabilities() {@@ -8984,21 +9026,101 @@? `Inherited: ${template.input.join(", ")} input; reasoning ${template.reasoning ? "yes" : "no"}.`: "";}+ function modelRegistryIdentity() {+ return { provider: $("model-registry-provider").value, id: $("model-registry-id").value.trim() };+ }+ function resetRegistryLookup() {+ modelRegistryLookupRequest++;+ modelRegistryLookup = null;+ $("model-registry-result").hidden = true;+ $("model-registry-save").disabled = true;+ $("model-registry-save").textContent = "Verify and enable";+ setStatus("st-model-registry", "", "");+ }+ function showRegistryLookup(result) {+ modelRegistryLookup = { ...result, identity: modelRegistryIdentity() };+ const spec = result.spec;+ modelRegistryTemplateOptions();+ for (const field of ["name", "template", "contextWindow", "maxTokens"])+ $("model-registry-" + field).value = spec[field] ?? "";+ for (const field of ["input", "output", "cacheRead", "cacheWrite"])+ $("model-registry-" + field).value = spec.cost?.[field] ?? "";+ for (const field of ["base", "webui", "auxiliary", "fastMode"])+ $("model-registry-" + field).checked = spec[field] ?? (field === "base" || field === "webui");+ $("model-registry-tiers").value = JSON.stringify(spec.cost?.tiers ?? [], null, 2);+ $("model-registry-source").textContent =+ "Source: " ++ result.source ++ (result.catalogGeneratedAt ? " · catalog " + new Date(result.catalogGeneratedAt).toLocaleDateString() : "");+ $("model-registry-summary").textContent =+ result.kind === "builtin"+ ? `${spec.name} · ${spec.contextWindow.toLocaleString()} context tokens · ${spec.maxTokens.toLocaleString()} maximum output · $${spec.cost.input} input / $${spec.cost.output} output per million tokens`+ : spec.name || spec.id;+ $("model-registry-lookup-note").textContent = result.message;+ $("model-registry-result").hidden = false;+ const missing = result.missing || [];+ for (const field of [+ "name",+ "template",+ "contextWindow",+ "maxTokens",+ "input",+ "output",+ "cacheRead",+ "cacheWrite",+ ]) {+ const label = $("model-registry-" + field).closest("label");+ $(missing.includes(field) ? "model-registry-missing-fields" : "model-registry-advanced-fields").appendChild(+ label,+ );+ }+ $("model-registry-missing").hidden = !missing.length;+ $("model-registry-advanced").hidden = result.kind === "builtin";+ $("model-registry-advanced").open = false;+ $("model-registry-save").textContent =+ result.kind === "builtin" ? "Verify and enable existing model" : "Verify and enable";+ $("model-registry-save").disabled = false;+ modelRegistryCapabilities();+ }function editRegistryModel(spec) {+ if (modelRegistryBusy) return;+ resetRegistryLookup();$("model-registry-id").disabled = Boolean(spec);$("model-registry-provider").disabled = Boolean(spec?.provider);+ $("model-registry-id").value = spec?.id || "";$("model-registry-provider").value = spec?.provider || "openai";modelRegistryTemplateOptions();- for (const field of ["id", "name", "contextWindow", "maxTokens", "template"])- $("model-registry-" + field).value =- spec?.[field] ?? (field === "template" ? $("model-registry-template").value : "");- for (const field of ["input", "output", "cacheRead", "cacheWrite"])- $("model-registry-" + field).value = spec?.cost?.[field] ?? "";- for (const field of ["base", "webui", "auxiliary", "fastMode"])- $("model-registry-" + field).checked = spec?.[field] ?? (field === "base" || field === "webui");- $("model-registry-tiers").value = JSON.stringify(spec?.cost?.tiers || [], null, 2);- modelRegistryCapabilities();+ if (spec)+ showRegistryLookup({+ kind: "saved",+ spec,+ missing: [+ ...["name", "template", "contextWindow", "maxTokens"].filter((field) => spec[field] == null),+ ...["input", "output", "cacheRead", "cacheWrite"].filter((field) => spec.cost?.[field] == null),+ ],+ source: "Saved administrator definition",+ message: "Review the saved definition under Advanced overrides. Changes require verification again.",+ });}+ $("model-registry-lookup").onclick = async () => {+ if (modelRegistryBusy) return;+ resetRegistryLookup();+ const request = modelRegistryLookupRequest;+ const identity = modelRegistryIdentity();+ $("model-registry-lookup").disabled = true;+ setStatus("st-model-registry", "Looking up exact model metadata…", "");+ try {+ const response = await api("POST", "/api/model-registry/lookup", identity);+ if (request !== modelRegistryLookupRequest) return;+ if (!response.ok) throw new Error(response.data?.message || "Lookup failed. Try again.");+ showRegistryLookup(response.data);+ setStatus("st-model-registry", "Metadata loaded. Review, then verify access.", "");+ } catch (error) {+ if (request === modelRegistryLookupRequest) setStatus("st-model-registry", error.message, "err");+ } finally {+ $("model-registry-lookup").disabled = false;+ }+ };async function loadModelRegistry() {const response = await api("GET", "/api/model-registry");if (!response.ok) {@@ -9053,51 +9175,79 @@rows.appendChild(row);}}- $("model-registry-provider").onchange = modelRegistryTemplateOptions;+ $("model-registry-provider").onchange = () => {+ resetRegistryLookup();+ modelRegistryTemplateOptions();+ };+ $("model-registry-id").oninput = resetRegistryLookup;$("model-registry-template").onchange = modelRegistryCapabilities;$("model-registry-new").onclick = () => editRegistryModel(null);$("model-registry-save").onclick = async () => {+ if (modelRegistryBusy || !modelRegistryLookup) return;+ const lookup = modelRegistryLookup;+ const identity = modelRegistryIdentity();+ if (JSON.stringify(identity) !== JSON.stringify(lookup.identity)) {+ resetRegistryLookup();+ return;+ }const button = $("model-registry-save");+ let disabledControls = [];try {const value = (field) => $("model-registry-" + field).value.trim();const number = (field) => {if (!value(field)) throw new Error(field + " is required");return Number(value(field));};- const spec = {- id: value("id"),- name: value("name"),- provider: value("provider"),- template: value("template"),- contextWindow: number("contextWindow"),- maxTokens: number("maxTokens"),- cost: Object.fromEntries(- ["input", "output", "cacheRead", "cacheWrite"].map((field) => [field, number(field)]),- ),- };- spec.cost.tiers = JSON.parse(value("tiers") || "[]");- for (const field of ["base", "webui", "auxiliary", "fastMode"])- spec[field] = $("model-registry-" + field).checked;- button.disabled = true;+ const builtin = lookup.kind === "builtin";+ const spec = builtin+ ? null+ : {+ ...identity,+ name: value("name"),+ template: value("template"),+ contextWindow: number("contextWindow"),+ maxTokens: number("maxTokens"),+ cost: Object.fromEntries(+ ["input", "output", "cacheRead", "cacheWrite"].map((field) => [field, number(field)]),+ ),+ };+ if (spec) {+ if (!spec.name || !spec.template) throw new Error("Display name and a compatible template are required.");+ spec.cost.tiers = JSON.parse(value("tiers") || "[]");+ for (const field of ["base", "webui", "auxiliary", "fastMode"])+ spec[field] = $("model-registry-" + field).checked;+ }+ modelRegistryBusy = true;+ disabledControls = [...button.closest("section").querySelectorAll("input,select,textarea,button")].map(+ (el) => [el, el.disabled],+ );+ disabledControls.forEach(([el]) => {+ el.disabled = true;+ });button.textContent = "Verifying…";setStatus("st-model-registry", "Checking the model with organization credentials…", "");- const result = await api("PUT", "/api/model-registry/" + encodeURIComponent(spec.id), {- ...spec,- verify: true,- });- if (!result.ok) throw new Error(result.data?.message || "Could not save model.");+ const result = await api(+ builtin ? "POST" : "PUT",+ "/api/model-registry/" + encodeURIComponent(identity.id) + (builtin ? "/enable" : ""),+ builtin ? { provider: identity.provider, verify: true } : { ...spec, verify: true },+ );+ if (!result.ok) throw new Error(result.data?.message || "Could not verify model.");await loadOnboarding();- editRegistryModel(spec);setStatus("st-model-registry",- "Model verified and enabled with organization credentials. Personal-key access may differ.",+ builtin+ ? result.data.message+ : "Model verified and enabled with organization credentials. Personal-key access may differ.","ok",);} catch (error) {setStatus("st-model-registry", error.message, "err");} finally {- button.disabled = false;- button.textContent = "Verify and enable";+ modelRegistryBusy = false;+ disabledControls.forEach(([el, disabled]) => {+ el.disabled = disabled;+ });+ button.textContent = lookup.kind === "builtin" ? "Verify and enable existing model" : "Verify and enable";}};diff --git a/plugins/admin/src/index.ts b/plugins/admin/src/index.tsindex 66391c479..23b43fe72 100644--- a/plugins/admin/src/index.ts+++ b/plugins/admin/src/index.ts@@ -288,7 +288,7 @@ const WRITES = new Map<string, string[]>([["users", ["PUT", "POST"]],["slack-installation", ["PUT", "DELETE"]],["model-providers", ["PUT", "DELETE"]],- ["model-registry", ["PUT", "DELETE"]],+ ["model-registry", ["POST", "PUT", "DELETE"]],["custom-providers", ["PUT", "DELETE"]],]);diff --git a/plugins/admin/test/onboarding-view.test.ts b/plugins/admin/test/onboarding-view.test.tsindex c22285976..be3afb348 100644--- a/plugins/admin/test/onboarding-view.test.ts+++ b/plugins/admin/test/onboarding-view.test.ts@@ -135,15 +135,32 @@ test("unknown views still fall back to the default view", () => {});test("model registry verification makes charges and credential scope explicit", () => {- assert.match(html, /id="model-registry-save">Verify and enable/);+ assert.match(html, /id="model-registry-save" disabled>Verify and enable/);const notice = slice('id="model-registry-verification-notice"', "</p>");assert.match(notice, /provider charge/);assert.match(notice, /personal-key access/);const save = slice('$("model-registry-save").onclick', "let customProvidersLoaded");assert.match(save, /verify: true/);- assert.match(save, /button.disabled = true/);+ assert.match(save, /el.disabled = true/);assert.match(save, /Verifying…/);assert.match(save, /finally/);- assert.match(save, /button.disabled = false/);+ assert.match(save, /el.disabled = disabled/);assert.match(html, /Verified with organization credentials/);});++test("model setup starts with two inputs, separates missing fields and discards stale lookups", () => {+ const form = slice(+ '<label\n >Provider\n <select id="model-registry-provider">',+ 'id="model-registry-result"',+ );+ assert.match(form, /model-registry-id/);+ assert.doesNotMatch(form, /model-registry-contextWindow|model-registry-input/);+ assert.match(html, /<summary>Advanced overrides<\/summary>/);+ assert.match(html, /id="model-registry-missing-fields"/);+ const code = slice("let modelRegistryTemplates = []", "let customProvidersLoaded");+ assert.match(code, /request !== modelRegistryLookupRequest/);+ assert.match(code, /\$\("model-registry-id"\)\.oninput = resetRegistryLookup/);+ assert.match(code, /JSON\.stringify\(identity\) !== JSON\.stringify\(lookup.identity\)/);+ assert.match(code, /Choose a compatible template/);+ assert.match(code, /source/);+});diff --git a/src/api/deps.ts b/src/api/deps.tsindex a01ed044e..8e38d4f82 100644--- a/src/api/deps.ts+++ b/src/api/deps.ts@@ -1,3 +1,4 @@+import type { ModelVerifier } from "../model/model-verification.ts";import type { DurableByteStore } from "../files/durable-byte-store.ts";import type { SessionShareStore } from "../sessions/session-share.ts";import type { ModelOverlayStore } from "../model/model-overlay-store.ts";@@ -106,6 +107,7 @@ export interface ServerDeps {mcpToolService?: McpToolService;modelCredentialFetch?: typeof fetch;modelRegistry?: ModelOverlayStore;+ modelVerifier?: ModelVerifier;refreshModels?: () => Promise<void>;customProviders?: CustomProviderStore;refreshCustomProviders?: () => Promise<void>;diff --git a/src/api/routes/admin.ts b/src/api/routes/admin.tsindex 74f0e88f8..03579484a 100644--- a/src/api/routes/admin.ts+++ b/src/api/routes/admin.ts@@ -1,4 +1,4 @@-import { modelRegistry } from "./admin/model-registry.ts";+import { modelRegistry, lookupRegistryModel, enableBuiltinModel } from "./admin/model-registry.ts";import { type ApiCtx, type Route } from "./route.ts";import {getAdminResources,@@ -74,6 +74,8 @@ const routes: ReadonlyArray<Route<ApiCtx>> = [{ method: "PUT", path: "/v1/admin/mcp-servers/:id", auth: "either", handle: putMcpServer },{ method: "DELETE", path: "/v1/admin/mcp-servers/:id", auth: "either", handle: deleteMcpServer },{ method: "DELETE", path: "/v1/admin/model-providers/:provider", auth: "either", handle: deleteModelProvider },+ { method: "POST", path: "/v1/admin/model-registry/lookup", auth: "either", handle: lookupRegistryModel },+ { method: "POST", path: "/v1/admin/model-registry/:model/enable", auth: "either", handle: enableBuiltinModel },{ method: "GET", path: "/v1/admin/model-registry", auth: "either", handle: modelRegistry },{ method: "PUT", path: "/v1/admin/model-registry/:model", auth: "either", handle: modelRegistry },{ method: "DELETE", path: "/v1/admin/model-registry/:model", auth: "either", handle: modelRegistry },diff --git a/src/api/routes/admin/model-registry.ts b/src/api/routes/admin/model-registry.tsindex 545a1f74a..632ba743d 100644--- a/src/api/routes/admin/model-registry.ts+++ b/src/api/routes/admin/model-registry.ts@@ -1,3 +1,4 @@+import { getBuiltinModels } from "@earendil-works/pi-ai/providers/all";import { ModelVerificationError } from "../../../model/model-verification.ts";import { MODEL_REGISTRY, safeModelMetadata } from "../../../model/pi-models.ts";import { errMessage } from "../../../util/errors.ts";@@ -13,7 +14,12 @@ export async function modelRegistry(ctx: ApiCtx): Promise<void> {if (!store) return sendJson(ctx.res, 404, { error: "not_found" });if (ctx.method === "GET") {await ctx.deps.refreshModels?.();- const templates = MODEL_REGISTRY.flatMap(({ id }) => {+ const ids = new Set([+ ...MODEL_REGISTRY.map((model) => model.id),+ ...getBuiltinModels("openai").map((model) => model.id),+ ...getBuiltinModels("anthropic").map((model) => model.id),+ ]);+ const templates = [...ids].flatMap((id) => {const model = safeModelMetadata(id);return model && (model.provider === "openai" || model.provider === "anthropic") ? [model] : [];});@@ -63,3 +69,124 @@ export async function modelRegistry(ctx: ApiCtx): Promise<void> {await store.refresh();return sendJson(ctx.res, 200, { ok: true, ...verification });}++import { builtinModelSpec, lookupModel, modelLookupInput } from "../../../model/model-lookup.ts";+import { builtInModelCatalog, selectableModelCatalog } from "../../../model/model-catalog.ts";+import { verificationFailure } from "../../../model/model-verification.ts";++export async function lookupRegistryModel(ctx: ApiCtx): Promise<void> {+ if (!(await authorizeAdmin(ctx, orgScope(ctx.deps)))) return;+ const input = modelLookupInput.safeParse(ctx.body);+ if (!input.success)+ return sendJson(ctx.res, 400, { error: "bad_request", message: "Choose a provider and enter a valid model ID." });+ await ctx.deps.refreshModels?.();+ try {+ const saved = (await ctx.deps.modelRegistry?.statuses())?.find(+ (row) => row.spec.id === input.data.id && !row.disabled,+ );+ if (saved?.spec.provider && saved.spec.provider !== input.data.provider)+ return sendJson(ctx.res, 400, {+ error: "provider_mismatch",+ message: "The saved model uses a different provider.",+ });+ if (saved && saved.spec.provider)+ return sendJson(ctx.res, 200, {+ kind: "saved",+ spec: saved.spec,+ source: "Saved administrator definition",+ missing: [+ ...["name", "template", "contextWindow", "maxTokens"].filter(+ (field) => saved.spec[field as keyof typeof saved.spec] == null,+ ),+ ...["input", "output", "cacheRead", "cacheWrite"].filter(+ (field) => saved.spec.cost?.[field as "input" | "output" | "cacheRead" | "cacheWrite"] == null,+ ),+ ],+ message: "Review or edit this existing definition, then verify it again.",+ });+ return sendJson(+ ctx.res,+ 200,+ await lookupModel(input.data, ctx.deps.modelCredentials, ctx.deps.modelCredentialFetch),+ );+ } catch {+ return sendJson(ctx.res, 400, {+ error: "lookup_failed",+ message: "The model ID does not match the selected provider.",+ });+ }+}++export async function enableBuiltinModel(ctx: ApiCtx): Promise<void> {+ const scope = orgScope(ctx.deps);+ const actor = await authorizeAdmin(ctx, scope);+ if (!actor) return;+ if (!isObj(ctx.body)) return sendJson(ctx.res, 400, { error: "bad_request" });+ const input = modelLookupInput.safeParse({ provider: ctx.body.provider, id: ctx.params.model });+ if (!input.success || ctx.body.verify !== true)+ return sendJson(ctx.res, 400, {+ error: "bad_request",+ message: "Choose a provider and explicitly consent to verification with verify: true.",+ });+ const { config, modelVerifier } = ctx.deps;+ if (!config || !modelVerifier) return sendJson(ctx.res, 503, { error: "verification_unavailable" });+ await ctx.deps.refreshModels?.();+ const spec = builtinModelSpec(input.data);+ if (!spec)+ return sendJson(ctx.res, 400, {+ error: "not_builtin",+ message: "Use Verify and enable on the model definition instead.",+ });+ const approved = (await config.getApprovedHarnessesDurable()) ?? [ctx.deps.harnessId ?? "pi"];+ if (!approved.includes("pi"))+ return sendJson(ctx.res, 400, {+ error: "pi_not_approved",+ message: "Enable the Pi harness in Models settings first.",+ });+ try {+ const context = await modelVerifier(spec);+ const signal = AbortSignal.timeout(15_000);+ await context.probe(signal);+ const apply = async () => {+ const stillApproved = (await config.getApprovedHarnessesDurable()) ?? [ctx.deps.harnessId ?? "pi"];+ if (!stillApproved.includes("pi"))+ throw new ModelVerificationError(+ "pi_not_approved",+ "Pi was disabled during verification. Review Models settings.",+ );+ if ((await modelVerifier(spec)).fingerprint !== context.fingerprint)+ throw new ModelVerificationError("changed_during_verification", "Serving credentials changed. Verify again.");+ const managed = await ctx.deps.modelCredentials?.availability();+ const catalog = managed?.openrouter+ ? await selectableModelCatalog(ctx.deps.modelCredentialFetch)+ : builtInModelCatalog();+ const configured = await config.getWebuiModelsDurable(scope);+ const ids = configured ?? catalog.map((model) => model.id);+ config.setWebuiModels(scope, [...new Set([...ids, spec.id])]);+ await config.flushScope(scope);+ };+ if (ctx.deps.advisoryLock) await ctx.deps.advisoryLock.withLock("admin-model-picker", apply);+ else await apply();+ } catch (error) {+ const failure = verificationFailure(error);+ audit(ctx.deps, {+ principalId: actor.id,+ action: "model-registry.verification-failed",+ resource: spec.id,+ scopeLabel: scope,+ });+ return sendJson(ctx.res, 422, { error: failure.code, message: failure.message });+ }+ audit(ctx.deps, {+ principalId: actor.id,+ action: "model-registry.enable-builtin",+ resource: spec.id,+ scopeLabel: scope,+ });+ return sendJson(ctx.res, 200, {+ ok: true,+ verifiedAt: Date.now(),+ verificationScope: "organization",+ message: "Existing model verified and added to the web picker. Organization default unchanged.",+ });+}diff --git a/src/model/model-lookup.ts b/src/model/model-lookup.tsnew file mode 100644index 000000000..d845f52c2--- /dev/null+++ b/src/model/model-lookup.ts@@ -0,0 +1,127 @@+import { getBuiltinModelDataGeneratedAt } from "@earendil-works/pi-ai/providers/all";+import { z } from "zod";+import { modelSupportsFastMode, resolveBuiltinModel } from "./pi-models.ts";+import type { ModelOverlay } from "./model-overlay.ts";+import type { ModelCredentialStore } from "./model-credential-store.ts";+import { providerBaseUrl } from "./provider-endpoints.ts";++export const modelLookupInput = z.strictObject({+ provider: z.enum(["openai", "anthropic"]),+ id: z+ .string()+ .trim()+ .regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,199}$/),+});+export type ModelLookupInput = z.infer<typeof modelLookupInput>;++export function builtinModelSpec({ provider, id }: ModelLookupInput): ModelOverlay | undefined {+ const model = resolveBuiltinModel(id);+ if (!model || model.provider !== provider) return undefined;+ return {+ id,+ provider,+ name: model.name,+ template: id,+ contextWindow: model.contextWindow,+ maxTokens: model.maxTokens,+ cost: structuredClone(model.cost),+ fastMode: modelSupportsFastMode(id),+ base: true,+ webui: true,+ auxiliary: false,+ };+}++async function providerJson(response: Response): Promise<unknown> {+ if (!response.body) throw new Error("Empty response");+ const reader = response.body.getReader();+ let bytes = 0;+ const chunks: Uint8Array[] = [];+ try {+ for (;;) {+ const { value, done } = await reader.read();+ if (done) break;+ bytes += value.byteLength;+ if (bytes > 64 * 1024) throw new Error("Metadata too large");+ chunks.push(value);+ }+ return JSON.parse(Buffer.concat(chunks).toString("utf8"));+ } finally {+ await reader.cancel();+ }+}+const providerMetadata = z.object({+ id: z.string(),+ display_name: z.string().trim().min(1).max(200).optional(),+ max_input_tokens: z.number().int().positive().safe().optional(),+ max_tokens: z.number().int().positive().safe().optional(),+});++export async function lookupModel(+ input: ModelLookupInput,+ credentials?: ModelCredentialStore,+ fetcher: typeof fetch = fetch,+) {+ const builtin = builtinModelSpec(input);+ if (builtin)+ return {+ kind: "builtin" as const,+ spec: builtin,+ missing: [] as string[],+ source: "Bundled model catalog",+ catalogGeneratedAt: getBuiltinModelDataGeneratedAt(),+ message:+ "This model already exists. Verify access and add it to the web picker; no duplicate definition or default change is needed.",+ };+ if (resolveBuiltinModel(input.id)) throw new Error("This model ID belongs to a different provider.");+ const spec: Partial<ModelOverlay> = { ...input, cost: undefined };+ let source = "Manual entry";+ let message: string;+ try {+ const key = await credentials?.resolve(input.provider);+ if (key) {+ const base =+ providerBaseUrl(input.provider) ??+ (input.provider === "anthropic" ? "https://api.anthropic.com" : "https://api.openai.com/v1");+ const suffix = input.provider === "anthropic" ? "/v1/models/" : "/models/";+ const headers: Record<string, string> =+ input.provider === "anthropic"+ ? { "x-api-key": key, "anthropic-version": "2023-06-01" }+ : { authorization: `Bearer ${key}` };+ const response = await fetcher(base.replace(/\/$/, "") + suffix + encodeURIComponent(input.id), {+ headers,+ signal: AbortSignal.timeout(5000),+ redirect: "error",+ });+ if (response.ok) {+ const metadata = providerMetadata.parse(await providerJson(response));+ if (metadata.id !== input.id) throw new Error("Model ID mismatch");+ source = `${input.provider === "anthropic" ? "Anthropic" : "OpenAI"} model API`;+ spec.name = metadata.display_name ?? input.id;+ if (input.provider === "anthropic") {+ spec.contextWindow = metadata.max_input_tokens;+ spec.maxTokens = metadata.max_tokens;+ if (spec.contextWindow && spec.maxTokens && spec.maxTokens >= spec.contextWindow) {+ spec.contextWindow = undefined;+ spec.maxTokens = undefined;+ }+ }+ message =+ "Exact provider record found. Pricing and protocol compatibility still need confirmation. Lookup does not verify generation access.";+ } else {+ await response.body?.cancel();+ message =+ "The provider could not return metadata for this ID. Check the ID and credential access, or enter documented values manually.";+ }+ } else+ message =+ "No exact catalog match or organization provider key. Configure the key first, or enter documented values manually.";+ } catch {+ message = "Provider metadata could not be loaded. Retry lookup or enter documented values manually.";+ }+ const missing = ["name", "template", "contextWindow", "maxTokens"].filter(+ (key) => spec[key as keyof ModelOverlay] === undefined,+ );+ missing.push("input", "output", "cacheRead", "cacheWrite");+ return { kind: "new" as const, spec, missing, source, message };+}diff --git a/src/model/pi-models.ts b/src/model/pi-models.tsindex 470317f0d..76ec31d1c 100644--- a/src/model/pi-models.ts+++ b/src/model/pi-models.ts@@ -359,7 +359,7 @@ export function registerOpenRouterCatalogModel(definition: OpenRouterCatalogModereturn model;}-function resolveBuiltinModel(id: string): PiModel | undefined {+export function resolveBuiltinModel(id: string): PiModel | undefined {if (id.startsWith(CODEX_SUBSCRIPTION_PREFIX)) {const m = getModel(CODEX_SUBSCRIPTION_PROVIDER, id.slice(CODEX_SUBSCRIPTION_PREFIX.length));// Keep the namespaced id: pi resolves the turn's model by this string,diff --git a/src/wiring.ts b/src/wiring.tsindex 255994c24..0a0608673 100644--- a/src/wiring.ts+++ b/src/wiring.ts@@ -1,4 +1,4 @@-import { createModelVerifier } from "./model/model-verification.ts";+import { createModelVerifier, type ModelVerifier } from "./model/model-verification.ts";import type { probeModel } from "./harness/pi-harness.ts";import { createAwsRoleBroker, type AwsRoleBroker } from "./auth/aws-role-broker.ts";import type { SessionShare, SessionShareStore } from "./sessions/session-share.ts";@@ -409,6 +409,7 @@ export interface BuiltApp {modelCredentials: ModelCredentialStore;userModelCredentials: UserModelCredentialStore;modelRegistry: ModelOverlayStore;+ modelVerifier: ModelVerifier;refreshModels: () => Promise<void>;customProviders: CustomProviderStore;refreshCustomProviders: () => Promise<void>;@@ -995,16 +996,13 @@ export function buildApp(backing: artifactMap("custom_model_providers"),keyMaterial: config.connectorSecretKey ?? randomBytes(32),});- const modelRegistry = createModelOverlayStore(- artifactMap("model_registry"),- writeModelRegistry,- createModelVerifier({- credentials: modelCredentials,- keyMaterial: config.connectorSecretKey ?? randomBytes(32),- modelGateway: config.modelGateway,- probe: overrides.modelVerificationProbe,- }),- );+ const modelVerifier = createModelVerifier({+ credentials: modelCredentials,+ keyMaterial: config.connectorSecretKey ?? randomBytes(32),+ modelGateway: config.modelGateway,+ probe: overrides.modelVerificationProbe,+ });+ const modelRegistry = createModelOverlayStore(artifactMap("model_registry"), writeModelRegistry, modelVerifier);const refreshCustomProviders = async () => {setCustomProviders(await customProviders.enabled());};@@ -1983,6 +1981,7 @@ export function buildApp(modelCredentials,userModelCredentials,modelRegistry,+ modelVerifier,refreshModels,customProviders,refreshCustomProviders,@@ -2069,6 +2068,7 @@ export function serverDeps(modelCredentials: built.modelCredentials,userModelCredentials: built.userModelCredentials,modelRegistry: built.modelRegistry,+ modelVerifier: built.modelVerifier,refreshModels: built.refreshModels,customProviders: built.customProviders,refreshCustomProviders: built.refreshCustomProviders,diff --git a/test/admin-model-lookup.test.ts b/test/admin-model-lookup.test.tsnew file mode 100644index 000000000..932b84cd5--- /dev/null+++ b/test/admin-model-lookup.test.ts@@ -0,0 +1,270 @@+import "./support/auto-fake-sprites.ts";+import assert from "node:assert/strict";+import { test } from "node:test";+import { buildApp, serverDeps } from "../src/wiring.ts";+import { testConfig } from "./support/test-config.ts";+import { createInsecureTestServer } from "../src/api/server.ts";+import type { AddressInfo } from "node:net";++const headers = { "content-type": "application/json", "x-admin-actor": "admin-alice@default-org" };+test("admin lookup recognizes an existing model without asking for a clone template", async () => {+ const config = testConfig({ anthropicApiKey: "test-only" });+ const built = buildApp(config);+ const server = createInsecureTestServer(built.app, serverDeps(config, built));+ await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));+ try {+ const result = await fetch(+ `http://127.0.0.1:${(server.address() as AddressInfo).port}/v1/admin/model-registry/lookup`,+ { method: "POST", headers, body: JSON.stringify({ provider: "anthropic", id: "claude-opus-4-6" }) },+ );+ assert.equal(result.status, 200);+ const body = (await result.json()) as { kind: string; spec: { template: string }; missing: string[] };+ assert.equal(body.kind, "builtin");+ assert.equal(body.spec.template, "claude-opus-4-6");+ assert.deepEqual(body.missing, []);+ } finally {+ await new Promise<void>((r) => server.close(() => r()));+ }+});++import { lookupModel } from "../src/model/model-lookup.ts";+import { createModelCredentialStore } from "../src/model/model-credential-store.ts";+import { createMemoryMap } from "../src/persistence/durable-map.ts";+import { setProviderBaseUrls } from "../src/model/provider-endpoints.ts";+import { verificationUpstream } from "./support/model-verification-upstream.ts";+import { setModelOverlays } from "../src/model/pi-models.ts";+import { afterEach } from "node:test";++afterEach(() => {+ setProviderBaseUrls({});+ setModelOverlays([]);+});+const credentials = () =>+ createModelCredentialStore({+ backing: createMemoryMap(),+ keyMaterial: "test-key",+ fallback: { openai: "test-openai-key", anthropic: "test-anthropic-key" },+ });++test("exact catalog lookup returns documented values without network or other-model guesses", async () => {+ const result = await lookupModel({ provider: "anthropic", id: "claude-opus-4-6" }, credentials(), async () => {+ throw Error("must not fetch");+ });+ assert.equal(result.kind, "builtin");+ assert.equal(result.spec.template, "claude-opus-4-6");+ assert.equal(result.spec.cost?.input, 5);+ assert.ok(result.spec.maxTokens);+ assert.equal(result.source, "Bundled model catalog");+ await assert.rejects(lookupModel({ provider: "openai", id: "claude-opus-4-6" }), /different provider/);+});++test("provider lookup only imports exact bounded metadata, never pricing from a similar model", async () => {+ setProviderBaseUrls({ anthropic: "http://127.0.0.1:19998" });+ let calls = 0;+ const result = await lookupModel(+ { provider: "anthropic", id: "future-lookup-model" },+ credentials(),+ async (url, options) => {+ calls++;+ assert.equal(url, "http://127.0.0.1:19998/v1/models/future-lookup-model");+ assert.equal(new Headers(options?.headers).get("x-api-key"), "test-anthropic-key");+ assert.equal(options?.redirect, "error");+ return Response.json({+ id: "future-lookup-model",+ display_name: "Future exact",+ max_input_tokens: 160000,+ max_tokens: 40000,+ apiKey: "private",+ cost: { input: 100 },+ });+ },+ );+ assert.equal(calls, 1);+ assert.equal(result.spec.name, "Future exact");+ assert.equal(result.spec.contextWindow, 160000);+ assert.equal(result.spec.maxTokens, 40000);+ assert.equal(result.spec.template, undefined);+ assert.equal(result.spec.cost, undefined);+ assert.deepEqual(result.missing, ["template", "input", "output", "cacheRead", "cacheWrite"]);+ assert.doesNotMatch(JSON.stringify(result), /private|test-anthropic-key/);+});++test("OpenAI minimal metadata and missing credentials leave unknown fields visibly missing", async () => {+ const result = await lookupModel({ provider: "openai", id: "future-openai-model" }, credentials(), async () =>+ Response.json({ id: "future-openai-model", object: "model" }),+ );+ assert.equal(result.spec.name, "future-openai-model");+ assert.equal(result.spec.template, undefined);+ assert.ok(result.missing.includes("contextWindow"));+ const absent = await lookupModel({ provider: "openai", id: "future-openai-model" }, undefined, async () => {+ throw Error("must not fetch");+ });+ assert.equal(absent.source, "Manual entry");+ assert.ok(absent.missing.includes("name"));+ assert.match(absent.message, /provider key/);+});++test("bad responses degrade to manual fields with no partial or mismatched metadata", async () => {+ for (const response of [+ Response.json({ id: "wrong-model", display_name: "Wrong", max_input_tokens: 1, max_tokens: 1 }),+ Response.json({ id: "future-model", max_tokens: -1 }),+ new Response("secret", { status: 403 }),+ new Response("x".repeat(65537)),+ ]) {+ const result = await lookupModel(+ { provider: "anthropic", id: "future-model" },+ credentials(),+ async () => response,+ );+ assert.equal(result.source, "Manual entry");+ assert.equal(result.spec.contextWindow, undefined);+ assert.equal(result.spec.name, undefined);+ assert.ok(result.missing.includes("template"));+ assert.doesNotMatch(JSON.stringify(result), /secret|Wrong/);+ }+ const offline = await lookupModel({ provider: "openai", id: "future-model" }, credentials(), async () => {+ throw Error("secret");+ });+ assert.match(offline.message, /Retry lookup/);+});++test("live lookup is admin-only and builtin enable verifies without a duplicate or changed default", async () => {+ const upstream = await verificationUpstream();+ const config = testConfig({+ harness: "pi",+ anthropicApiKey: "lookup-key",+ providerBaseUrls: { anthropic: upstream.url },+ });+ const built = buildApp(config, {+ modelCredentialFetch: async () => {+ throw Error("lookup must not fetch for builtin");+ },+ });+ const server = createInsecureTestServer(built.app, serverDeps(config, built));+ await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));+ const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;+ const post = (path: string, body: object, actor = "admin-alice@default-org") =>+ fetch(base + path, { method: "POST", headers: { ...headers, "x-admin-actor": actor }, body: JSON.stringify(body) });+ const identity = { id: "claude-opus-4-6", provider: "anthropic" };+ const enable = "/v1/admin/model-registry/claude-opus-4-6/enable";+ try {+ assert.equal((await post("/v1/admin/model-registry/lookup", identity, "bob@default-org")).status, 403);+ assert.equal((await post(enable, { provider: "anthropic", verify: true }, "bob@default-org")).status, 403);+ assert.equal((await post(enable, { provider: "anthropic" })).status, 400);+ assert.equal(upstream.requests.length, 0);+ const priorDefault = await built.config.getRuntimeSelectionDurable("org:default-org");+ built.config.setWebuiModels("org:default-org", ["claude-sonnet-4-6"]);+ await built.config.flushScope("org:default-org");+ upstream.behavior.status = 403;+ assert.equal((await post(enable, { provider: "anthropic", verify: true })).status, 422);+ assert.deepEqual(await built.config.getWebuiModelsDurable("org:default-org"), ["claude-sonnet-4-6"]);+ upstream.behavior.status = 200;+ assert.equal((await post(enable, { provider: "anthropic", verify: true })).status, 200);+ assert.equal(upstream.requests.at(-1)?.body.model, "claude-opus-4-6");+ assert.equal(upstream.requests.at(-1)?.body.max_tokens, 128);+ assert.deepEqual(await built.config.getWebuiModelsDurable("org:default-org"), [+ "claude-sonnet-4-6",+ "claude-opus-4-6",+ ]);+ assert.deepEqual(await built.config.getRuntimeSelectionDurable("org:default-org"), priorDefault);+ assert.deepEqual(await built.modelRegistry.statuses(), []);+ const runtime = await fetch(+ base + "/v1/runtime-config?principalId=admin-alice@default-org&scopeId=personal:admin-alice@default-org",+ { headers },+ );+ const body = (await runtime.json()) as { modelsByHarness: { pi: string[] } };+ assert.ok(body.modelsByHarness.pi.includes("claude-opus-4-6"));+ } finally {+ server.closeAllConnections();+ await new Promise<void>((r) => server.close(() => r()));+ await upstream.close();+ }+});++test("enable rechecks serving credentials and harness policy before mutating the picker", async () => {+ let onProbe: () => Promise<void> = async () => {};+ const config = testConfig({ harness: "pi", anthropicApiKey: "test-key" });+ const built = buildApp(config, { modelVerificationProbe: async () => onProbe() });+ const server = createInsecureTestServer(built.app, serverDeps(config, built));+ await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));+ const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}/v1/admin/model-registry/claude-opus-4-6/enable`;+ const enable = () =>+ fetch(url, { method: "POST", headers, body: JSON.stringify({ provider: "anthropic", verify: true }) });+ const org = "org:default-org";+ try {+ onProbe = async () => {+ await built.modelCredentials.set("anthropic", "changed-key", "admin");+ };+ assert.equal((await enable()).status, 422);+ assert.equal(await built.config.getWebuiModelsDurable(org), null);+ onProbe = async () => {+ built.config.setApprovedHarnesses(["claude"]);+ await built.config.flushScope(org);+ };+ assert.equal((await enable()).status, 422);+ assert.equal(await built.config.getWebuiModelsDurable(org), null);+ onProbe = async () => {+ throw Error("must not call");+ };+ assert.equal((await enable()).status, 400);+ } finally {+ server.closeAllConnections();+ await new Promise<void>((r) => server.close(() => r()));+ }+});++test("identity schema rejects URL-like and extra inputs before lookup", async () => {+ const { modelLookupInput } = await import("../src/model/model-lookup.ts");+ for (const body of [+ { provider: "openai", id: "https://example.invalid/model" },+ { provider: "other", id: "model" },+ { provider: "openai", id: "model", baseUrl: "https://example.invalid" },+ { provider: "openai", id: "../models" },+ ])+ assert.equal(modelLookupInput.safeParse(body).success, false);+});++import { createServer } from "node:http";++test("real metadata transport authenticates only the configured endpoint and refuses redirects", async () => {+ let mode = "record";+ let sinkRequests = 0;+ const server = createServer((req, res) => {+ if (req.url === "/sink") {+ sinkRequests++;+ res.end("unexpected");+ return;+ }+ assert.equal(req.headers["x-api-key"], "test-anthropic-key");+ if (mode === "redirect") {+ res.writeHead(302, { location: "/sink" });+ res.end();+ return;+ }+ res.setHeader("content-type", "application/json");+ res.end(+ JSON.stringify({+ id: "live-metadata-model",+ display_name: "Exact network record",+ max_input_tokens: 200000,+ max_tokens: 32000,+ }),+ );+ });+ await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));+ setProviderBaseUrls({ anthropic: `http://127.0.0.1:${(server.address() as AddressInfo).port}` });+ try {+ const value = await lookupModel({ provider: "anthropic", id: "live-metadata-model" }, credentials());+ assert.equal(value.spec.name, "Exact network record");+ assert.equal(value.spec.contextWindow, 200000);+ assert.equal(value.spec.maxTokens, 32000);+ mode = "redirect";+ const redirect = await lookupModel({ provider: "anthropic", id: "live-metadata-model" }, credentials());+ assert.equal(sinkRequests, 0);+ assert.equal(redirect.source, "Manual entry");+ assert.equal(redirect.spec.name, undefined);+ } finally {+ server.closeAllConnections();+ await new Promise<void>((r) => server.close(() => r()));+ }+});